Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Should You Block AI Crawlers from Your Website?
15 Plays8 months ago
In today's episode, Andrew Laws dives into a highly pertinent topic: whether or not to block AI crawlers from indexing your website. Starting off with a fundamental explanation of what crawling involves—essentially, search engines sending out bots to gather data from websites—Andrew discusses the conventional reasons why some might choose to restrict these digital explorers, including privacy concerns and control over content dissemination.
Find out more about ActionCoach - https://yeseo.io/go/actioncoach
However, Andrew argues against blocking these bots, particularly AI crawlers. He explains that allowing AI to index your site could bolster brand awareness and drive traffic, as AI's role in digital search evolves. Despite the fear that AI might misuse the gathered content, Andrew sees little evidence of such risks and suggests that enabling AI indexing might contribute positively to the internet's future.
Highlighting the significance of staying current with technological advances, he reflects on insights from a recent conference, emphasising the potential pitfalls of ignoring AI developments.
Conclusion: While the decision remains personal, Andrew leans towards keeping your website open to AI crawlers, viewing it as a step forward in harnessing AI's potential for positive impact.
Further Reading: Check the show notes for links to a detailed Moz article on this topic and more insights from the BizEx conference.
Connect: Join the discussion or reach out via yeseo.io or Andrew Laws on LinkedIn.
Further reading - https://moz.com/blog/who-should-block-ai-bots
Recommended
Transcript
Introduction to SEO Daily Podcast
00:00:00
Speaker
Hello, welcome back to SEO Daily. I'm your host, Andrew Laws. SEO Daily is a part of the Untitled SEO Podcast. Today, I am going to talk to you about something that was going to come up sooner or later.
Understanding Web Crawling
00:00:13
Speaker
Should you block AI from calling your website?
00:00:17
Speaker
Now, if I've already lost you with crawling, I'm gonna give you a very quick explanation of that. Google, Bing, and other search engines send a little fella, or a gender neutral bit of code to your website that reads it, then it follows links on it. It's called crawling, it's just gathering information, and then it will find a link to somebody else's website and go shoot and off down that, also known as the spider.
Why Block Web Crawlers?
00:00:42
Speaker
Now, some people have been blocking
00:00:45
Speaker
bots, these kind of crawlers from their website for a number of reasons. Some people even block the Google crawler, which means Google won't be able to find out about you. But more traditional reasons for blocking other bots might include because you're not wanting to have your information gathered.
00:01:04
Speaker
So you can block any of the major SEO tools from your website quite easily. There's Mars, Ahrefs, Semrush. You can block them, usually in robots.txt.
Debating AI Crawlers: To Block or Not?
00:01:14
Speaker
And the big flaw of this is that blocking crawlers from your website relies on the people who operate the crawlers from adhering to their scouts on a promise that they won't call your website if you ask them not to.
00:01:29
Speaker
This is a problem because the bad actors, the people who are up to no good in the world of crawling the web, ain't going to pay any attention to your robots.txt. So there's just a brief intro into crawling, not crawling. My personal opinion is just open up your website, let anyone who wants to crawl it. But that's a topic for another time, kind of, because I'm now going to talk about whether you should block AI from crawling your website.
Risks of AI Repurposing Content
00:01:56
Speaker
So again, another bit of background, chat models. Sorry, camera's going wonky now. Chat models, most notably chat AI, learn by looking around the web, seeing what's out there. That's how they gather their data. Although famously, if you're on the free version of chat GPT 3.5, you're potentially looking at data that's now out of date. So why would you want to block them?
00:02:26
Speaker
Well, maybe you don't want to risk your content being repurposed somewhere else because if Google, if I'm going to stop having to pick on open AI here, but if a AI crawler gathers information from your website, there's potentially a chance that a user somewhere else in the world who asks the chat bot to write an article, they might be regurgitating some of yours. Incidentally,
00:02:54
Speaker
Here at ESEO, every article we write, we run through copy scapes as a plagiarism checker. So even if we write it ourselves, we just want to make really sure we're not just regurgitating someone else's. So that's something that could be happening. Your work could be repurposed and churned out somewhere else.
Arguments for Blocking AI Crawlers
00:03:13
Speaker
I mean, I think it's pretty unlikely the web is a very big place and
00:03:19
Speaker
If you're really concerned about people nicking your content, you might need to get rid of yourself a bit because unless you genuinely are the leading expert or the most authoritative genius on any topic you're writing about,
00:03:35
Speaker
you're probably fairly safe from having your content nicked because it'll be nicked from other people. But I'm also not saying be rubbish online. That's not a good defense. So what other reasons might you block AI? Well, just privacy if you just don't want to be a part of the AI revolution.
Benefits of Allowing AI Crawling
00:03:53
Speaker
And at this stage, I respect the fact that some people really don't want to be. But let's look at the reasons not to block it. The reason is just to leave your web open to being crawled.
00:04:05
Speaker
One thing is brand awareness. If you are providing the answers to problems that people have or have good, helpful information on your website, then why not let chat AI tell people where they found the answer? Incidentally, that is a really good thing to do if you're using chat GPT or any of the others. If you're given an answer, just type, prove it.
00:04:28
Speaker
or click the funny little arrow link that comes up in chat results, you know, chat outputs. So it could become, allowing your content to be crawled could become a good source of traffic because at some point
00:04:48
Speaker
all the chat models are gonna start acting as search engines.
Future of AI: Search Engines and Beyond
00:04:52
Speaker
None of them claim to be search engines yet, apart from Bing, I guess, which is just using chat GPT, but at some point, there's gonna become chat models that are marketing themselves as search engines. The closest we've had so far is, I announced a couple of days ago, that Brave, which is a browser, Brave browser,
00:05:16
Speaker
is implementing AI search functionality. I mean, it's almost inevitable.
AI Crawling and Internet Evolution
00:05:23
Speaker
So if you block your website and all these AIs aren't gathering information about your site, when they start acting more overtly as a search engine, they could potentially leave you out. It could be like being left out of Google now. So there's one reason. Another reason is that
00:05:46
Speaker
At the moment, the threat of your information being sucked into AI and given out to somebody else, I've just not really seen any evidence for that. It's a fear and it's legitimate to look into these things, but I've not seen any evidence that that is definitely a thing. Another argument for allowing
00:06:09
Speaker
AI to scam your site and crawl it is just that if it is going to be a very positive force for mankind then perhaps allowing your data to be crawled is just the little small part you have in building a better version of the internet. I'm going to link to a Moz article that partly inspired this episode that goes into a lot more pros and cons and I'm going to link to that in the show notes and I recommend reading it. For now
00:06:37
Speaker
I'm still not sure I'm the right person to add this much comment on AI, but we need to discuss
Conclusion: To Block AI or Not?
00:06:43
Speaker
these things. These things need to be talked about. I was at a conference on Thursday and Friday last week in Liverpool, BizEx conference, which I'll probably also link to. It's from Action Coach, so I'll drop that link in. We saw
00:06:57
Speaker
a very good two hour long sort of presentation workshop question answer session from someone called Piers Linney. And he raised some very interesting things about AI and why if you're not adopting now, potentially you could be like those companies that it took until about 2005, 2010 for them to build a website, you could just be left behind a blank catch up. There was a lot to learn here and we are cutting edge. We are,
00:07:26
Speaker
As Piers Linney said, and many other people have said, we are at the point in history where AI is at its worst, is at its nappest. This is just the start. So, should you block AI from crawling your website? Probably not. It's up to you,
Listener Engagement and Contact
00:07:42
Speaker
though. At the moment, we're just making up the rules and adapting as we go along.
00:07:47
Speaker
I'd really like to know what you think about blocking AI. So please do comment, please email me through yessio.io or find me on LinkedIn. Thank you for listening. I shall see you and speak to you tomorrow.