57 lines
2.6 KiB
Markdown
57 lines
2.6 KiB
Markdown
+++
|
|
title = "Index Blocking"
|
|
description = "Define pages to prevent them from appearing in search."
|
|
categories = ["search"]
|
|
tags = ["links", "search", "robots", "metadata"]
|
|
features = ["code highlighter", "related content", "snippets"]
|
|
[[copyright]]
|
|
owner = "Josh Habdas"
|
|
date = "2019"
|
|
license = "agpl-3.0-or-later"
|
|
+++
|
|
|
|
After Dark uses the `noindex` robots meta directive to prevent search engines from crawling and indexing certain parts of your site. It appears in the HTML document `head` and looks like this:
|
|
|
|
```html
|
|
<meta name="robots" content="noindex">
|
|
```
|
|
|
|
Unlike {{< external href="http://www.robotstxt.org" text="robots.txt" />}} meta directives are defined within page content itself and unambiguously indicate which, if any, pages should be blocked from indexing — even if some of those pages appear in your site's {{< external href="https://gohugo.io/templates/sitemap-template/" text="Sitemap" />}}.
|
|
|
|
To facilitate the discovery of index blocked pages [Fuzzy Search](../fuzzy-search) utilizes the very same meta directive exposed to search engines to prevent disclosure of pages in its own result listings. Therefore, if a page can be found in fuzzy search, that page may ultimately appear on a search engine result page.
|
|
|
|
Adjust index blocking per-page using `noindex` {{< external href="https://gohugo.io/content-management/front-matter/" text="Front Matter" />}}:
|
|
|
|
```toml
|
|
noindex = true # set false or remove to unblock
|
|
```
|
|
|
|
Block entire sections using an `_index.md` file with the above setting:
|
|
|
|
```
|
|
├── content
|
|
│ ├── legal
|
|
│ │ ├── _index.md
|
|
│ │ ├── terms.md
|
|
│ │ └── privacy.md
|
|
│ ├── post
|
|
```
|
|
|
|
By default the following page types are blocked automatically:
|
|
|
|
- Section listings automatically linked to from the [Section Menu](../section-menu);
|
|
- [Taxonomy Pages](../taxonomy-pages) such as `Category`, `Tag` and terms listings; and,
|
|
- If enabled, the [Fuzzy Search](../fuzzy-search) page or any deep-linked result within.
|
|
|
|
Adjust defaults using the `noindex_kinds` setting from {{< external href="https://gohugo.io/getting-started/configuration/" text="Site Configuration" />}}. For example, to enable crawling of section pages add the following to the config:
|
|
|
|
```toml
|
|
[params]
|
|
noindex_kinds = [
|
|
"taxonomy",
|
|
"taxonomyTerm"
|
|
] # crawl "section" pages
|
|
```
|
|
|
|
Learn about {{< external href="https://moz.com/learn/seo/robots-meta-directives" text="Robots Meta Directives on Moz" />}} and see how Google uses `noindex` in {{< external href="https://support.google.com/webmasters/answer/93710" text="Block search indexing with 'noindex'" />}}.
|