bots

Our websites are crawled from different bots and search engines, sometimes it is beneficial or sometime it is creating the issue. Bots and crawlers are consumed lots of bandwidth. To avoid this issue we can upload robots.txt file in our home directory. In this file we can block bots and for that...

How to create Robots.txt file to block bots? If we talk about bots so there are few great bots, for example, Google and Bing that you might need to permit to crawl the site. This is simple process and it may be completed by utilizing file robots.txt in the root directory. The following is a...