Transcript
If there's one thing that I've spent the most time doing in 2026 while building directories, it's definitely data scraping. And I thought it'd be an interesting video to do the five levels of data scraping. Level one being what I started doing back in 2022 with my first directory, level three which is what I think most of us are doing in Cloud Code, and level five which is deploying a bunch of agents doing all sorts of data scraping tasks that turn into productive tasks that can increase traffic or help monetization or just manage really boring tasks and it's the closest thing to what I think is an AI employee but for your directory or directories.
So that's what I'm going to talk about today because data scraping is so ingrained in every single little process of building a directory for me. It doesn't matter if I'm finding an idea, I'm probably scraping Reddit problems, conversations that I could go and build a directory around. I've scraped Mediavine, Ezoic, Raptive, looking at successful websites that have gotten traffic, looking for ideas there. And once I find the idea, it doesn't stop, right? I'm scraping addresses, phone numbers, images sometimes. And then when it comes to SEO, of course I am scraping emails to reach out to site owners to get back links pointed to my directory.
It's really just everywhere all at once, right? So let's talk about it because maybe you've been scraping data one way for a very long time and you realize that there are probably better ways and not just better ways but combining processes and scraping data to achieve more automated workflows effectively. And level one is really just not even scraping data. This is what I did for my first directory back in 2022. I know some people are still literally copying and pasting data which is insane in 2026. You should not be doing that especially for things like business hours, phone numbers, addresses, all the basic information.
But when it comes to getting data that's really hard to get, manual curation is still the best way because you can't scrape data that you have to call or email a business for. And this usually speaks to pricing data. I've about parting dot com and how really they are the only people online who have pricing data for specific funeral homes. And that's because they emailed and called a bunch of them a couple years ago. So I actually think level one of not even data scraping is actually quite valuable still depending on what kind of data you get. So manual curation is still good for the hard stuff.
It's just that the easy stuff, which is like 90% of your directory, especially when you first launch, is scrapable. Level two of data scraping is if you're using Outscraper or Apify. And I know people still use this because sometimes I get some Outscraper credits from people clicking on my affiliate links, which thank you for that. And even though I don't think I've used Outscraper at all this year, I still think there are some solid pros to using it. Namely, if you're still trying to mine Google Maps for any really solid and under tapped directory niches cuz those definitely exist. An example would be if you're trying to create a directory around all the public restrooms in parks in the nation, right?
That's going to be way easier to get just through scraping Google Maps versus just scraping the open web like Google Search or whatever. There are certain directory niches where I think it makes sense. It's just not really what I do anymore. And I don't think for like most directories you even need to pay Outscraper or Apify to scrape data for. With Apify, I used to explore it because I wanted to scrape Reddit and it was really hard to scrape Reddit at the time when I discovered it. And now I'm able to scrape Reddit by just prompting Claude code. So even Apify, I don't really find myself using too often if I'm going to be honest.
And there's probably great scrapers that I'm not taking advantage of that are amazing. It's just not really what I need for the types of directories that I'm personally building. Obviously, another pro with this is that it's super easy to use and you don't really have to do much. It comes at a cost though, right? I've spent probably thousands of dollars to Outscraper just scraping data that I no longer have to do. But going into some of the cons, you have a lot of junk data, which means there's a lot more data cleaning, and that is a little bit more time. It's not that bad, especially if you just let Claude code do your data cleaning.
But I remember back in the day I would have to scrape without scraper, and then I would manually or use Claude code to basically clean all this data. And sometimes it's not that easy to clean data depending on what niche you're you're building in. It's kind of like, you know, an intensive process where you have to go into websites and then look for certain keywords and be like, oh yeah, that is indeed a septic tank repair company and not a waste management company. That's just the nature of using these tools. You're going to get tons of data, and with that is going to be a bunch of junk data that you have to sift through and clean before you feel comfortable adding it to your directory.
And that's just an extra step that I don't think the majority of us even need to go through if I'm going to be honest, unless you are mining Google Maps for those kind of interesting hidden niches where data lives in Google Maps and really nowhere else. Level three of data scraping is really just using Claude code or Codex to start your scrape. And I think this is what most people are doing, saying something like, I want you to scrape all the funeral homes in America, get the name, address, phone number, and I want you to identify these businesses by looking for keywords like cremation, obituary, memorial service, funeral services.
And that's how you can kind of scrape and clean at the same time so that by the time you find them, you pretty much know they're worthy to be included in your directory. This one's pretty straightforward, and I think it's great because you can go really long tail and niche with it. Like if you wanted to find all of the roofing companies that specialize in metal roofs, then you can go and do that and build a niche directory that falls under this broader and more competitive category of just roofing. So there's kind of some cool stuff that you can do where you're saving time, you don't have to clean the data, and you can get really niche with it and relatively low cost because you're really just using the credits that come with your subscription for Claude or Codex.
And as far as data enrichment, pretty much the same thing, right? You can compile a list of all the things that you think are helpful to include in your directories listings and then go and scrape the actual business websites for that data enrichment and then include it on your listings to make it more helpful. Other kind of information that people want to know. Don't think I need to spend that much time on this. I think this is again what most of us are probably doing. Level four of data scraping is really just level three on steroids. So I think this is where I give Claude Code a super mega prompt that turns into a chain of jobs that kind of feeds into one another.
You might be playing around with a VPS at this point for literally day long or week long scrapes. It's the way that I think is how I would go and pump out directories every single week because it's like constantly just scraping data, cleaning it, enriching it, scraping Reddit, identifying how to actually provide useful information on my listings, maybe scraping all of the articles, boring tools, calculators that I can go and build that will attract traffic. This is pretty much how I start a lot of my directories. I will do this all the time and just have it build a directory in the next like five days where yeah, it's taking five days, but I'm really spending like a couple hours at most like in front of the computer prompting to Claude Code while just like working in the background.
So if you want a one shot directories that like are pretty solid and have specific instructions when it comes to anything data related, on page SEO related, maybe even doing things like scraping all the business emails off your websites, running them through some kind of email verifier, and then having any leads that come through automatically get sent to that respective business owner in that area. Those are all things that I will do or have done to essentially combine data scraping with tasks that I think are productive or that actually move the needle in terms of traffic or monetization. This even includes after you build the directory and you have to build back links, I am scraping other website owners within my niche, connecting it to a bunch of external APIs like Serper, maybe Ahrefs if you have Ahrefs, and then identifying the most relevant websites to get a backlink from, pitching it, right?
Automating that whole process. Now level five of data scraping isn't actually harder. I think up until now I've been talking about data scraping a lot in the context of like starting a directory and coming up with ideas and cleaning data and stuff like that. But I think level five is more than that. It's kind of thinking about the future, like the three month, six month, one year old directory that you have and you built before, and all the things that you have to do to keep it in good standing. Like a simple example of this is that I have a directory where a lot of the businesses close down often cuz that's just the nature of the niche.
And so I need to make sure that my directory is up to date, otherwise over time I just lose trust from people who come on the wrong hours cuz I have the wrong store hours or they end up going there when it's permanently closed. And so I end up getting these emails. And so I'm like, wow, you know what would be a lot better is if I could not have to think about this every single month and I just had some kind of agent rescrape data, update the hours, and that way I have an accurate up-to-date directory at all times, right? That's just like one example of just thinking about things in the future.
You can even think about it in the context of distribution. You could scrape all of the Reddit threads in a Reddit forum that's relevant to your niche, get involved in the conversation, answer a question, and at the very least you'll get more awareness. But at the most you might even get some monetization opportunities, some ideas, maybe even meet some people. That's legit happened to me so many times where I left a single comment and my DMs turn into some kind of backlink or something happens. So what I'm trying to say is level five is more than just scrape data. I think it's turning your processes and scrape data for your specific directory, turning them into skills, and then handing them to agents that then handle all the boring stuff that makes running a directory, let alone a portfolio of directories, way way way easier.
So I created this document here just to give you an idea. There's 10 different things that I think about in the future after I've launched my directory that I really don't want to consciously go in and think about and do. This includes refreshing data, right? Checking hours, phone numbers, websites, closures. There's a Reddit researcher. This is more for understanding what customers want and if that changes over time. Managing emails. This one kind of scrapes your email inbox to manage any incoming messages where it's like, hey, can you add my listing? Or hey, this is actually a permanently closed location.
I get those emails on a weekly basis and I really just ignore them if I don't have a system set up. This one's a lead router. So, someone fills out a form on your directory and you don't really know what to do with that lead. But if you had scraped all of the emails and made sure that they were verified, you can start sending these leads over to local businesses, kind of starting that business relationship and that partnership early. If you want to improve your traffic, get a little more fancy, you can connect your data for SEO or Ahrefs API key and scrape all of the boring tools and calculator opportunities within your niche.
So, this is actually pretty underrated. I found one today that I couldn't believe I didn't build, but it's super relevant to one of my directory niches. Gets like 30,000 organic monthly visitors. So, I just built it myself after finding this by scraping a lot of the different boring tools within my niche. And this is probably going to get me more traffic that I can then monetize. So, I'm not going to go through every single one of these. Not all of them have to do with data scraping, but all of these represent small, boring tasks that I'd rather set up once, give to an agent so that it can just do it every month or every week or every 6 months, whatever it is.
By now, you're thinking, okay, we get it, Frey. You're using AI agents. How do I do this for myself and automate all of these boring tasks that I know I need to do? So, I'm using the Hermes agent through Hostinger, which I'm happy to say is partnering up with me for this video. And I was able to document all the nuances, the processes behind all of these boring that I don't really want to do on a regular basis. And I turn those into skills, then I turn those into agents, and now they run all of these tasks for me. And my little gift to you is that I made all of my skills free in a public GitHub repo so that if you want to get set up, you can do this super easily and you can access the very skills that I'm using to manage my own directories.
So let me show you how to get set up and show you what this looks like in action. I'll make this super easy for you. Down below in the description, you'll see this link to this public GitHub repo for all of my skills, and you'll also be able to see the discount code, which will give you an additional 10% off of your Hermes agent. But here you will see a button called Deploy on Hostinger. You can simply click on this and then you'll end up on the Hermes agent page. Here we can see they're having their biggest sale of the year. It's their Black Friday sale at the time of this recording. So up to 70% off of your Hermes agent.
If you want to match me, I chose KVM 2. If you need more firepower, you can choose KVM 4 or KVM 8. Maybe you're managing a ton of websites, but you can go ahead and choose this plan. So from here you can choose your period, which is 1 month, 12 months, or 24 months. And just below you can choose how many Hostinger or Oxylab AI credits you want. Both of these are really useful for managing multiple agents, but whatever you end up choosing, you can head over to the coupon code area, type in the coupon code Fray Hermes in all caps, no spaces. And when you hit apply, you're going to get an additional 10% off of your Hostinger Hermes agent.
Once you do get fully set up, you'll see this interface here and you can go ahead and just start a new conversation over here. And I created this little document here which you will get if you just install the uh contents in the public GitHub repo. But you can just copy this message and paste it directly into the Hermes chat. And this is just going to get you set up fully and get you access to all of the skills. It should say 15 skills installed and now we can go ahead and just onboard a site. So I'm just going to type in onboard the site. And this is kind of what I 'm doing for all of my directories that I haven't yet attached to or have recently built that I need to automate tasks for.
I'm literally doing this. So I'm just going to do this for Discover Plasma, which is a Plasma donation directory that I've talked about a lot on this channel. And so I'm just going to kind of talk to it and answer these questions. The first one being, where are you at with the directory? I'll say it's live making a little money. Now it's asking me, what do I want to do the most? I think definitely more traffic, more accurate listings. It's asking for my website, so I'm just going to give it discoverplasma.com. So it just did a quick review of my site. It's finding the pages, the plasma locations.
So here's the recommended plan. I'm not going to go through all of them. I'm just going to do a little bit of the first one, which is basically a skill that helps keeps my listings accurate because truthfully, I think this data is pretty old. I know I have probably a lot of store hours that are wrong. So I do want to check on that really quick, but you can see all of the different skills that are available as well, even beyond the recommended game plan. And these are all the ones that I created as well. And I'm just going to say, can we just do a little like tester for keeping the listings accurate?
I just want to do that and I just want to see what you find when it comes to that skill. So it's going to look through 10 to 20 plasma centers. Everything it's doing right now is just gathering information so that once I like what it's doing and agree with the process and check off that it is in fact accurate, I'm just going to make sure that it just remembers to do this every month. And I'm going to set that schedule. I can customize it to whatever I'd like, but I'm just doing this sampler to show you what it looks like for the setup. And then after this, I'm just going to make sure it is not my problem anymore.
Before I approve this, I'm actually going to change this model instead of ChatGPT 5.6 Luna, I am going to go over to Claude Sonnet 5.5, which is going to consume a lot more credits, but it's just going to be a lot better. So I do recommend Claude Sonnet for this. So we'll just let this finish. So as this works, I do just want to mention you can obviously make more specific skills that are tailored to your directory. I think there's so many other skills that I could create to make my life easier cuz a lot of my directories are very specific to that directory. Like for example, even Discover Plasma, I have to like manually approve crowdsourced data that gets submitted and that's something that I think I could just let Hermes do with an agent.
And so like stuff like that you can always add on to the existing skills that I created. But going back to this, it's doing exactly what I would do manually. It's basically taking each listing, looking up a query, making sure that it's finding the correct source of truth, not some third-party data or a competitor directory, which also could be wrong. And it's actually polling the right hours from Google Maps listings as well if it needs to. Um same with like the name, address, whether it's permanently closed, all that kind of stuff. And so I don't have to do that manually anymore. It just runs that and it's going to run it on schedule based on the timeline that I wanted to.
So it did finish checking the 15 random listings and it found a lot of room for improvement. You can see all of the hours that my site currently has and the recommended changes based on the center's own page. And I also found a bunch of other issues like um I think like dead links or zip codes missing the leading zero. So there's actually a lot of things that it also checked for me that I would then basically just approve and then connect my Superbase database with and then it would just go ahead and make sure that is fixed. I would probably ask it to do it site-wide as well, but this is just like one very small little task that I would have it run every say like three months or so.
And then um I would never really have to worry about stale data ever again. And so a lot of these skills are designed to just remove the headache and the conscious thought of having to go in and do this for my own directories. Super simple to set up and everything you need can be found below in the description. Don't forget to use the coupon code Fray Hermes all caps no spaces for an additional 10% off your hosting your Hermes agent. Like I said, I think level five was a lot more than just data scraping, but in my opinion is a lot more productive and something that actually moves the needle when it comes to managing your directory, getting more traffic, getting more ideas, and maybe even moving monetization forward.
So if you found this valuable, let me know in the comments below. Always love to hear what you guys think. But with that, I appreciate you guys watching. Thanks so much for the support and I'll catch you guys in the next video. See you.