What is Robots.txt
Robots.txt file is one type of instruction for web crawlers that controls search bots to allow or disallow indexing of your site’s content. Actually it says to search bots which part of your website should index and which part shouldn’t index into its database. This file is simple but very important for a WordPress site. For search engine optimization (SEO), you should have a good Robots.txt file to ensure proper crawling of your WordPress site. But sometimes webmasters do not give importance to Robots.txt, as result different types of problems just like search engines show lots of duplicate contents, index unwanted contents. Sometimes it creates threats to WordPress site by giving insecure access to malware bots and hackers. By using Robots.txt, you can block unwanted contents of a WordPress, Just like admin page, media content, readme.html, plugin list etc. In this article, I am going to share how to add a robots.txt file using Yoast SEO and how to customize that file.
Sample of Robots.txt
Here is a sample of Robots.txt file. Below code is optimized for search engine bots, ads bot, malware bots and unwanted access bots. You can use it in your WordPress site. Just need to modify some area of this text file. Replace all wppundit.com with your site’s URL. You can also add your own Robots.txt file.
User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /archives/ disallow: /*?* Disallow: /recommended/ Disallow: /comments/feed/ Disallow: /trackback/ Disallow: /index.php Disallow: /xmlrpc.php Disallow: /wp-content/plugins/ Disallow: /readme.html User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Mobile Allow: / #Sitemap: http://www.wppundit.com/sitemap.xml #Sitemap: http://www.wppundit.com/sitemap_index.xml
How to add robots.txt file using Yoast SEO plugin
Yoast SEO is one of the most popular plugin for WordPress SEO. I also use this plugin in this site. You can properly optimize your WordPress site for search engines by using Yoast SEO. Some important features of Yoast SEO plugin are article optimization, title optimization, adding meta contents, adding XML sitemap, RSS, adding breadcrumbs navigation, .htaccess file, robots.txt file, social integration and many more. Without any confusion, you can use Yoast SEO.
In this post, I only show you how to add Robots.txt file using Yoast SEO to WordPress site. You can download or search Yoast SEO from Plugin >> Add New >> Search Plugin area. Then install and activate the Yoast SEO.
After activating Yoast SEO, you will get a setting icon in your dashboard menu item. From the menu item area click SEO >> Tools.
From tools area, Click the File Editor option and go to the robots.txt editing area to submit your site’s Robots.txt file.
In the Robots.txt area, you can see a robots.txt as a default. Now customize your site’s Robots.txt in a notepad. After customizing the robots.txt copy paste in the editing area and hit Save changes to Robots.txt. Now your job is done and your WordPress site is fully optimized for search engines. Check the Robots.txt adding /robots.txt after your domain name.
Now check in the Google Analytics, how your submitted Robots.txt is working. If search bots face any problem during access your site then you can see it in the Crawl Error area. If you face any problem then you can share it with us through the comment box below.