Removing Magento staging or development sites from google

removing staging site from google

As you might know that if your staging or development site has been crawled by search engines then they will index them and show them in Search Engine Result Pages (SERPs) which is not good for your live website because it will affect your live site page ranking. This can lead to dropping your organic traffic which ultimately affects your sales. How can you remove staging site from Google?

There are following ways you can prevent search engines to crawl or index your test or staging Magento site

1) Add the following entry in robots.txt and upload this file on root of your test or staging sites

User-agent: *
Disallow: /

This robots.txt is respected by all search engines and by adding the above entry you are telling them that don’t crawl any URL of this website.

2) Add password to your development or staging sites

Step 1 – Create .htpasswd file with the following entries

username:encrypted password

Encrypted Password for .htpasswd can be generated by using this link http://www.htaccesstools.com/htpasswd-generator/. Once generated update your .htpasswd with the generated line and upload .htpasswd to root folder of your staging or test site.

Step 2 – Update .htaccess file with the following entries

AuthUserFile /www/public_html/mystaging/.htpasswd
AuthType Basic
AuthName "Enter username and password"
Require valid-user

Just make sure you change the path of your .htpasswd file and leave rest of the lines as it is.

3) Set NO INDEX, NO FOLLOW for all of Magento URLs

Step 1 – Login to your Magento Admin

Step 2 – Go to System -> Configuration -> General -> Design -> HTML Head -> Default Robots -> NOINDEX, NOFOLLOW

Step 3 – Go to System -> Cache Management -> Refresh Cache

Please make sure you don’t apply any of the above prevention methods on your live site

The above will help you to prevent your staging site to appear in Google but what if you forgot to put any of the above prevention at the first place. And Google and other search engines have crawled and indexed your staging or development sites.

The most easiest way to remove your staging site from search engines is to add any of the above preventions and request Google to remove your staging site.

To remove your already indexed site from Google search engine pages you need to follow the following steps -:

Step 1 Create your staging site in Google webmaster tool

Step 2 Get your staging site verified

Step 3 Once the verification is done then click on your staging site in Google webmaster tool

Step 4 Select Google Index -> Remove URLs from left hand navigation

Step 5 Click “Create a new removal request” button

Step 6 Enter “/” in the URL text box provided

The above will create “Site removal” entry for Google to remove your site from search result pages. It generally takes 24 hours for Google to remove all your staging site links from search results.

Hope this article helped you in some way. Please leave us your comment and let us know what do you think? Thanks.

Scommerce

One Comment

  1. I would be careful with the nocrawl rule in the robots.txt
    1) it’s often forgotten after launch
    2) Your site can still end up in the index (just with no desc)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.