There are two ways to add meta tag headers to a URL. For a normal web page, you place meta tags inside the HTML <head> section. For files that have no HTML, such as PDFs and images, you send the same robots rules in an HTTP header called X-Robots-Tag, usually set in your server config or .htaccess file.
People often mix up these two ideas, because both are "headers" in some sense. This guide covers each one, shows working code, and explains which tags Google really uses.
What are meta tags and HTTP headers?
A meta tag is a line of HTML that describes a page to browsers and search engines. Visitors do not see it on the page. It lives in the <head> section, near the top of the HTML file.
An HTTP header is a piece of information your server sends before the page or file itself. Browsers and search bots read it first. You cannot see it in the page source; it travels with the server's response.
The useful overlap is robots rules. You can tell search engines "do not index this" with a meta tag or with an X-Robots-Tag header. The header also works for files that cannot hold a meta tag.
Which meta tags are worth adding?
Google lists the meta tags it supports in its Search Central docs. The ones most sites need are:
| Tag | What it does |
|---|---|
title | Main source for the page's title link in results |
meta name="description" | Short summary Google may use as the snippet |
meta name="robots" | Controls indexing and snippets |
meta name="viewport" | Tells Google the page is mobile friendly |
Strictly speaking, title is its own element, not a meta tag, but it sits in the same place. Google says the meta keywords tag has no effect on indexing or ranking, so you can skip it. Open Graph tags such as og:title help social sites build link previews, and Google may also read og:title when it builds a title link.
How to add meta tag headers inside the HTML head
Follow these steps for any page you control.
- Open the page template. On a static site, this is the HTML file. On a PHP site, it is often a shared header include. On WordPress, an SEO plugin usually handles it.
- Find the head section. It starts with
<head>and ends with</head>, before<body>. - Add your tags. Place each tag on its own line between those two markers.
- Save and upload. Then clear any cache so the new version goes live.
- Check the result. View the page source to confirm the tags appear once and only once.
A clean head section might look like this:
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Handmade Oak Tables | Example Shop</title>
<meta name="description" content="Solid oak dining tables made to order in four sizes.">
<meta property="og:title" content="Handmade Oak Tables">
</head>
Google says meta tags must sit in the head section. Letter case does not matter for most meta tags, except the Search Console verification tag. Each page should have its own title and description; copying one description across a whole site helps nobody.
On my own sites, I moved from hand-edited HTML to PHP includes years ago. With a shared header file, you set the tags once in the template and fill the title and description from a variable on each page.
How do you add a noindex rule with the X-Robots-Tag header?
Use the HTTP header when the file has no HTML head, or when you want to set a rule for many files at once. Google supports the same rules in the header as in the robots meta tag, including noindex, nofollow, none, nosnippet, noimageindex and unavailable_after.
On an Apache server, you can add this to your .htaccess file to keep all PDF files out of search results. It needs Apache's mod_headers module:
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>
To keep image files out of the index:
<Files ~ "\.(png|jpe?g|gif)$">
Header set X-Robots-Tag "noindex"
</Files>
On NGINX, the same PDF rule goes in the server block:
location ~* \.pdf$ { add_header X-Robots-Tag "noindex, nofollow"; }
In PHP, you can send the header from the page itself, before any output: header("X-Robots-Tag: noindex");
Be careful with these rules. One wrong pattern in .htaccess can hide your whole site from search. Test on one file first. If you also edit redirects in the same file, our guide to creating a 301 redirect in .htaccess shows the safe way to do it.
Why do robots rules sometimes not work?
The most common reason is robots.txt. Google states that if robots.txt blocks a page from crawling, Google will not see any indexing rules on it. So a page blocked in robots.txt and tagged noindex can still appear in results as a bare URL. To remove a page, let Google crawl it and keep the noindex rule.
Other common causes:
- A cache or CDN serves an old version without the new tag or header.
- A plugin adds a second robots tag that conflicts with yours. When rules conflict, the stricter one applies.
- The header goes to the wrong file type because of a typo in the pattern.
How can you check that it worked?
For meta tags, view the page source or use your browser's developer tools. Our guide on how to find meta tags on a website walks through each method.
For HTTP headers, run curl -I https://example.com/file.pdf in a terminal. Look for a line that starts with X-Robots-Tag. The URL Inspection tool in Google Search Console also shows whether Google sees a noindex rule. Once a page is in good shape, you can follow our search engine listing guide to get the pages you do want found.
To sum up: put meta tag headers for normal pages in the HTML head, and use the X-Robots-Tag HTTP header for PDFs, images and bulk rules. Then check both with the page source, curl and Search Console.
Frequently asked questions
Can I add meta tags to a URL without editing HTML?
For robots rules, yes. You can send an X-Robots-Tag header from your server config, .htaccess or code. Other tags, such as the title and description, must go in the HTML head, often through your CMS or an SEO plugin.
Does the meta keywords tag help SEO?
No. Google says it does not use the meta keywords tag at all for indexing or ranking. You can leave it out and spend the time on a good title and description.
Should I block a page in robots.txt or use noindex?
Use noindex if you want the page out of search results. Google must crawl the page to see the noindex rule, so do not also block it in robots.txt. Use robots.txt to control crawling, not indexing.
