{"id":2592,"date":"2025-08-05T13:48:18","date_gmt":"2025-08-05T12:48:18","guid":{"rendered":"https:\/\/www.forestsoftware.co.uk\/blog\/?p=2592"},"modified":"2025-08-05T10:06:24","modified_gmt":"2025-08-05T09:06:24","slug":"what-is-robots-txt-and-do-you-even-need-one-for-your-small-business-website","status":"publish","type":"post","link":"https:\/\/www.forestsoftware.co.uk\/blog\/2025\/08\/what-is-robots-txt-and-do-you-even-need-one-for-your-small-business-website\/","title":{"rendered":"What is robots.txt and Do You Even Need One for Your Small Business Website?"},"content":{"rendered":"Reading Time: <\/span> 7<\/span> minutes : <\/span><\/span>

What is robots.txt and Do You Even Need One for Your Small Business Website?<\/h1>\n

If you\u2019ve ever dived into the geeky side of running a website, chances are you\u2019ve come across something called robots.txt<\/code>. Sounds fancy, right? Like something out of a sci-fi film. But really, it\u2019s just a small text file that can make a big difference to how search engines deal with your site. The thing is, there\u2019s a lot of confusion around what it actually does, whether you really need one, and how it works alongside other stuff like sitemaps.<\/p>\n

So, if you\u2019re running a small business website and trying to figure out if this is one of those things you need to stress about \u2014 good news: this blog is just for you. We\u2019re going to break it all down in simple terms, no tech jargon (well, only the important bits, and we\u2019ll explain those too), and help you understand when to use a robots.txt file \u2014 and when not to bother.<\/p>\n

<\/p>\n

What Even Is a robots.txt File?<\/h2>\n

Alright, let\u2019s start with the basics. A robots.txt<\/code> file is just a plain text file you put in the root of your website \u2014 that means the main folder where everything lives. It gives instructions to bots (mainly search engine bots like Googlebot) about what they can and can\u2019t look at on your website.<\/p>\n

Think of it like a polite sign on a door that says, \u201cOi, please don\u2019t go snooping in here.\u201d Bots don\u2019t have to obey it, but most good ones will. It\u2019s part of a thing called the Robots Exclusion Protocol. Sounds very official, doesn\u2019t it? But it\u2019s just a way of saying, \u201cThese are the rules for crawling my site.\u201d<\/p>\n

What Does a robots.txt File Look Like?<\/h2>\n

Here\u2019s a super basic example of what you might find in a robots.txt<\/code> file:<\/p>\n

User-agent: *\r\nDisallow: \/private-folder\/\r\n<\/pre>\n
That asks all bots (*<\/code> means \u201call of you\u201d) to stay out of the folder called \u201cprivate-folder\u201d. Easy peasy. You can get more specific, but that\u2019s the general idea \u2014 it\u2019s all about telling search engines which pages or folders they shouldn\u2019t poke around in.<\/p>\n
A more advanced file may look like (note that the lines starting with # are comments are are ignored by bots, but are useful when you come back to the file in 6 months time and wonder what those records are for) :<\/p>\n
#To block SemrushBot from crawling your site for different SEO and technical issues:\r\nUser-agent: SiteAuditBot\r\nDisallow: \/\r\n#To block SemrushBot from crawling your site for Backlink Audit tool:\r\nUser-agent: SemrushBot-BA\r\nDisallow: \/\r\n#To block SemrushBot from crawling your site for On Page SEO Checker tool and similar tools:\r\nUser-agent: SemrushBot-SI\r\nDisallow: \/\r\n#To block SemrushBot from checking URLs on your site for SWA tool:\r\nUser-agent: SemrushBot-SWA\r\nDisallow: \/\r\n#To block SplitSignalBot from crawling your site for SplitSignal tool:\r\nUser-agent: SplitSignalBot\r\nDisallow: \/\r\n#To block SemrushBot-OCOB from crawling your site for ContentShake AI tool:\r\nUser-agent: SemrushBot-OCOB\r\nDisallow: \/\r\n#To block SemrushBot-FT from crawling your site for Plagiarism Checker and similar tools:\r\nUser-agent: SemrushBot-FT\r\nDisallow: \/\r\nUser-agent: SemrushBot\r\nDisallow: \/<\/pre>\n
And you can, for some bots ask them to slow down their visits if they are impacting your server by using a record like (like the lines above, anything after the # is treated as a comment and ignored by bots) :<\/p>\n
User-Agent:  FacebookBot                              \r\nCrawl-delay: 10              # 1 page per 10 seconds<\/pre>\n
If you need to do this though, check that the bot follows the Crawl-delay request as not all of them do so.<\/p>\n
Why Would You Want to Stop Search Engines Crawling Your Site?<\/h2>\n
So here\u2019s the thing: not everything on your site needs to be indexed by Google. Maybe you\u2019ve got some pages just for your team, like a staff rota or an admin login. Maybe you\u2019ve got old product pages you don\u2019t want to show up in search anymore. Or maybe you\u2019re running a little experiment behind the scenes and don\u2019t want search engines finding it just yet.<\/p>\n
Using robots.txt<\/code>, you can tell bots, \u201cHey, leave this stuff alone.\u201d It doesn\u2019t delete anything, and it doesn\u2019t block people from seeing it \u2014 it just asks bots to ignore it.<\/p>\n
But Here\u2019s the Big One: You Do Not<\/em> Need a robots.txt File Just to Say \u201cIndex Everything\u201d<\/h2>\n
This is a biggie, and it trips up a lot of people. If you don\u2019t have a robots.txt<\/code> file at all, do you know what happens? Search engines will just assume they\u2019re allowed to crawl and index everything. That\u2019s the default behaviour.<\/p>\n
You don\u2019t need to write a file that says \u201cYes, index all my pages\u201d \u2014 because they already will, as long as nothing else is stopping them (like a noindex tag or password protection).<\/p>\n
Adding a robots.txt<\/code> file that says:<\/p>\n
User-agent: *\r\nDisallow:\r\n<\/pre>\n
\u2026does the same as not having one at all. It\u2019s like saying, \u201cHey bots, you can go wherever you like,\u201d even though they already could. So unless you\u2019ve got something specific you want to block, it\u2019s perfectly fine \u2014 and even preferred \u2014 to just skip the robots.txt file entirely.<\/p>\n
So Why Do So Many People Add One Anyway?<\/h2>\n
Habit, mostly. Or they\u2019ve read online that \u201cevery website needs one,\u201d which isn\u2019t true. Some website platforms create one automatically even if there\u2019s nothing in it. And sometimes people copy what they\u2019ve seen elsewhere without fully understanding it.<\/p>\n
The problem is, if you get it wrong, it can accidentally stop search engines from indexing your whole site. We\u2019ve seen it happen more than once \u2014 a small business owner thinks they\u2019re helping by adding a robots.txt<\/code> file, but ends up adding something like:<\/p>\n
User-agent: *\r\nDisallow: \/\r\n<\/pre>\n
That one little slash means \u201cdon\u2019t index anything on this site.\u201d Total disaster if your business relies on people finding you on Google!\u00a0 Believe it or not, I’ve seen this so many times in my 28 year SEO career that it’s the frst thing I check when a new client says that they are not in the Google results.<\/p>\n
When Should a Small Business Use a robots.txt File?<\/h2>\n
Right, so while you don\u2019t need<\/em> a robots.txt file if you want everything to be indexable, there are a few cases where it might come in handy. For example:<\/p>\n