@tjrobertson52

There's a new file you might have to start including on your website if you want large language models like ChatGPT to crawl you. txt, and it could be very important for ranking in these large language models. So I just wanna talk about everything we know so far and also what we don't know.

txt file that tells web crawlers like Google whether or not they're allowed to crawl certain pages of your website. txt is like that for large language models, except instead of telling them what pages they're allowed to crawl, it shows them where the important information on your website is. So this is a markdown file you place in the root directory of your website, and it just has details, but all the most important pages and what information can be found there.

And this might be helpful because large language models aren't gonna crawl every page of your website. And so you wanna make sure you're directing them to the pages that are most important. Talk about how to add this to your website in a little bit.

But first let's talk about if it actually helps. txt is just a proposed web standard. One of the biggest proponents of it is anthropic.

They're the makers of one of my favorite models Claude and so it's a pretty safe bet that they've already adopted it. On the other end of the spectrum is Google, who has just recently come out and said they are not adopting LMS dot txt. That makes sense since they already have the most sophisticated search engine.

Google actually recommended adding a no index header to your LMS dot txt file so it doesn't show up in search results and confuse users. And then in the middle, and perhaps the most important, is ChatGPT. To my knowledge have not made a public statement about it,

lts and confuse users. And then in the middle, and perhaps the most important, is ChatGPT. To my knowledge have not made a public statement about it, but there have been some people reporting that they've seen ChatGPT crawling their lms dot txt in their log files.

Now, just because they're looking at your lms dot txt doesn't mean they're actually adopting the protocol. So we'll just have to wait and see on that one. But if you're wondering if you should add this to your website, you might as well.

It's really easy. If you're on a WordPress site you can just download a plugin. I think the most popular right now is website l 0 m s dot t x t.

The SEO plugin Yoast is now also offering one click LMS TXT creation. I'm sure other website builders like Wix will be following suit soon. But if your website builder doesn't support it yet, it's just a simple markdown file.

Open up your favorite large language model. I would recommend ChatGPT three's O3 model and just have it walk you through the process of creating and adding the llms TXT file to your website. There's also something called an LLMS full TXT, which also includes the important content on all those important pages.

This is all happening incredibly fast, but if you're watching this, it means that you're keeping up. And I think there are going to be huge opportunities for those of us keeping up with it.