What is robots.txt? Have you ever heard of it, if not, today is the news of your good news, so today you are giving some information about Robot.txt to the people?
If you have a blog or website, you must have felt that sometimes we do not want anything. All those information also gets public on the Internet, do you know why such a thing happens.
Why are many good things have not been indexed after too many days? If you want to know about the secret behind all these things, then what are these bobots.txt articles? You will have to study carefully so that you will know about all these things until the end of the article.
To tell the search engine, the files and folders have to show the entire public on the website and no matter what the robots meta tag is used for. But all search engines do not want to read meta tag, so use lots of robots.txt files, which can easily be easily given to search engines about the files and folders of their website or blog.
So today I thought why you should give all the information about what Robots.txt is to people so that you do not have any problem in understanding it further.
What is Robots.txt File?
Robots.txt is known as a text file that contains a few lines of simple code, it is saved on the website or blog server, which allows web crawl to index and crawl your blog in search results.
This means that you can restrict web pages from web crawlers to your blog, so that it cannot be indexed in search blogs like your blog labels page, your demo page or any other pages, to index Not important.
Always remember that the search crawlers scan the robots.txt file before crawling the webpage. Each blog hosted on Blogger has its own default robots.txt file that looks like this.
User-agent: Media partners-Google
Explanation of Blogger robots.txt file?
User-agent: Media partners-Google: We are looking into our default blogger robots.txt file. The first line includes some things like “User-agent”, we use this “User-agent” in our robots.txt file to target a particular robot. You can see that this is targeting the “Mediapartner-Google” bot and telling its access limit on your blog via “Disallow”.
User-agent: * This is for all robots that are marked with asterisks (*) In the default setting, the label links of our blogs are restricted by search crawlers to index, which means that the web crawler will have our label page link due to the code given below Will not show up.
Disallow: /search: This means that links with keyword search only after the domain name are ignored. See the examples given below, which is a link to the label page labelled SEO.
And if we remove Disallow: / search from the above code, then crawl will reach our entire blog index and crawl all its content and web page.
Here Allow: / Refers to the homepage, which means that web crawlers can crawl and index our blog’s homepage.
Disallow Particular Post: Now suppose that if we want to exclude a particular post from the index, then we can add the lines in the code below.
Here yyyy and mm respectively mention the publication year and month of the post. For example, if we published a post in March 2016 in March 2016, then we have to use the format.
To make this work easier, you can simply copy the post URL and remove the blog name from the beginning.
Disallow Particular Page: If we need to reject a particular page, then we can use the same method as the latter. Simply copy the page URL and remove the blog address from it which will look something like this.
This code refers to the Sitemap of our blog. We are just optimizing the crawlers rate of our blog by adding a sitemap link here.
Whenever web crawlers scan our robots.txt file, they can find the path to our Sitemap where all the links in our published posts are present. Web crawlers make it easy to crawl all our posts.
Therefore, there are good prospects that web crawlers crawl all our blog posts without ignoring anyone.
Note: This sitemap will only tell web crawlers about 25 posts if you want to increase the number of links in your sitemap, instead of default sitemap with one below it will work in the first 500 recent posts.
If you have more than 500 published posts on your blog, then you can use two sitemaps below.
So you have a good understanding of what a robots.txt file is. Do you want to add to your blog? So let’s see.
How to add Blogger Custom Robots.txt File?
For self-hosted blogs, we usually create a robots.txt file and upload it to our server’s root directory, but Blogger provides a robots.txt file by default.
The way we want, we can edit this robots.txt file, so if you want to create some custom rules for robots on your blogger blog then follow the steps below.
- First, log in to your Blogger account.
- Go to the “Settings” section of your blog
- Click on the “Search Preferences” section.
- Scroll down and find “Custom Robots.txt“
- Enable it and enter your custom rule text to disallow or allow some of your blogs immediately after the “User-agent: *” line.
- After adding the Robots.txt File, click “Save changes“.
Now your Robots.txt file has been added.
It is a matter of paying attention. If you are submitting your blog to the search engine, then it is necessary to add a Robots.txt file. Without Robots.txt File Search Engine cannot understand what this is. So by adding a Robots.txt file, keep your next step.
To add a custom robots.txt file to blogger, I read the whole article today, I tried my best to make this article as easy and informative as possible.
But still, if you have any doubts or questions, ask me to ask in the empty comment section and do not keep any code without knowing about it in your custom robots.txt setting.
Thank you for reading this article, if you like it, please do not forget to share this post on your social media profile.