Google explains how to hide a website from search results

Google says the best way to hide a website from search results is to use a password, but you may want to consider other options.

This topic is highlighted in the latest installment of the Ask Googlebot video series on YouTube.

Google’s John Mueller answers a question asking how to prevent content from being indexed in search and if that’s something websites are allowed to do.

“In short, yes you can,†says Mueller.

There are three ways to hide a website from search results:

• Use a password
• Block exploration
• Block indexing

Websites can either turn off indexing completely or be indexed and hide content from Googlebot with a password.

Blocking Googlebot content is not against webmaster guidelines, as long as it is blocked from users at the same time.

For example, if the site is password protected when crawled by Googlebot, it should also be password protected for users.

Alternatively, the site must have guidelines in place to prevent Googlebot from crawling or indexing the site.

Where you may run into issues is if your website offers different content to Googlebot than to users.

This is called “masking” and is against Google guidelines.

With that distinction made, here are some good ways to hide content from search engines.

3 ways to hide content from search engines

1. Password protection

Locking a website with a password is often the best approach if you want to keep your site private.

A password will ensure that neither search engines nor random internet users will be able to see your content.

This is a common practice for developing websites. Publishing the website live is an easy way to share work in progress with customers, while also preventing Google from accessing a website that is not yet ready to be seen.

2. Block exploration

Another way to prevent Googlebot from accessing your site is to block crawling. This is done with the robots.txt file.

With this method, people can access your site with a direct link, but it will not be picked up by “well behaved” search engines.

This isn’t the best option, Mueller says, because search engines can still index the website address without accessing the content.

It is rare for this to happen, but it is a possibility you should be aware of.

3. Block indexing

The third and final option is to block your website from indexing.

To do this, you add a robots noindex meta tag to your pages.

A noindex tag tells search engines not to index this page before after they crawl it.

Users do not see the META tag and can still access the page normally.

Mueller’s final thoughts

Mueller concludes the video by saying that Google’s main recommendation is to go the password route:

“Overall, for private content, our recommendation is to use password protection. It’s easy to verify that it works and it prevents anyone from accessing your content.

Blocking crawling or indexing are good options when the content is not private. Or if there are only parts of a website that you want to prevent from appearing in search. “

See the full video below:

Featured image: Screenshot from: YouTube.com/GoogleSearchCentral, November 2021.