How We Built an AI-Powered Web Scraping Bot Without Knowing How to Code
In 2024, when GreyFox Ads was just getting started, we wanted to build something that would make our small French programmatic ad network stand out. The problem? Our idea required collecting technical data from hundreds of websites. And we did not know how to code. 😅
1. Why we needed our own Green Score
At the start of the GreyFox Ads adventure in 2024, we were looking for a way to differentiate our programmatic offering beyond the usual targeting and inventory selection promises.
We looked at solutions addressing the environmental impact of digital advertising. Many focused on variables such as connection type, creative weight, delivery conditions and supply-chain characteristics. We wanted to ask another question: how efficiently is the publisher's website itself built and delivered?
That became the starting point for our proprietary Green Score—a practical signal designed to complement, not replace, other quality and sustainability criteria.
2. Why we started with Google Lighthouse
Google Lighthouse is a widely used auditing tool for web performance and other aspects of website quality. It can produce separate results for desktop and mobile conditions, making it a useful starting point for comparing technical efficiency across sites (provided the test configuration and methodology remain consistent).
Our hypothesis was simple: a site with a more efficient technical implementation might be a better candidate for digital sobriety than a similarly situated site with heavy, poorly optimized pages. That is a hypothesis about technical efficiency, not a direct measurement of carbon emissions.
But there was a logistical problem: manually auditing the top 500 French websites would have taken days. We needed automation.
3. Three hours, thirty errors and one AI coding assistant
There was just one tiny detail: we did not know how to code. So we asked ChatGPT to help us build a scraping and automation bot.
What followed was not a magical one-prompt success story. We spent roughly three hours working through around thirty errors, often sharing screenshots of error messages and asking the assistant to explain what had gone wrong and how to fix it.
Eventually, we had a working first-generation tool capable of collecting desktop and mobile Lighthouse scores for approximately 500 French sites. Not bad for an afternoon of trial, error and coffee. ☕
The important lesson was not that AI had replaced software engineers. It was that an industry specialist could use an AI assistant to prototype a useful technical workflow without starting from a traditional development background.
4. What the workflow actually does
At a high level, our workflow turns a list of websites into a structured set of technical indicators:
| Step | Action | Why it matters |
|---|---|---|
| 1. Input | Prepare the list of French publisher domains or URLs. | Establish a consistent analysis perimeter. |
| 2. Audit | Run or retrieve Lighthouse results for mobile and desktop. | Capture comparable technical performance signals. |
| 3. Collect | Extract the scores into a structured file. | Avoid hundreds of manual copy-and-paste operations. |
| 4. Review | Check failures, outliers and inconsistent measurements. | Prevent a temporary error from becoming a misleading ranking. |
| 5. Use | Feed the results into a broader Green Score methodology. | Turn technical information into a usable curation signal. |
This is an explanation of the workflow, not a claim that our original prototype used a particular software library or API. There are several ways to automate Lighthouse, and the appropriate implementation depends on scale, access rules and available infrastructure.
5. A Lighthouse score is not a carbon footprint
This distinction matters. Lighthouse does not directly measure the CO₂ emitted by a website or an advertising impression. Its performance metrics can serve as indirect indicators of technical efficiency, but they do not capture the entire environmental impact of hosting, networks, devices or ad delivery.
Scores can also change with test conditions, page selection, network simulation and website updates. For that reason, a meaningful methodology needs repeatable tests and clear assumptions (a single homepage score is not necessarily representative of every article page).
For an advertiser, a sustainability-oriented score should therefore be treated as one decision signal among several, alongside brand safety, attention, supply-path quality and actual campaign outcomes.
6. From Lighthouse scores to a broader Green Score
Our original bot was only the beginning. We subsequently evolved the GreyFox Ads Green Score to incorporate the bid requests per page-view ratio, helping us identify potentially excessive ad-auction activity and flag inventory that may warrant closer examination for Made-for-Advertising (MFA) characteristics.
A high bid-request-to-page-view ratio is a useful warning signal, but it is not proof that a website is MFA (the figure depends on refresh behavior, header bidding configuration, ad slots and measurement scope).
This work fits a wider question we regularly explore: how much of the Open Web actually meets demanding quality criteria? Better signals can help us be more selective without confusing quantity with value.
7. What non-developers can learn from this experiment
If you work in media, product or AdTech and have a repetitive data task, you do not necessarily need to wait for a fully staffed engineering project to test your idea.
- Start with a real business question. We were not trying to “do AI”; we needed website-performance data at scale.
- Build the smallest useful prototype. A basic script that works is more valuable than an ambitious system that never ships.
- Use errors as feedback. Screenshots and precise descriptions of failures can help an AI assistant debug step by step.
- Validate the output. Automation can multiply mistakes as quickly as it multiplies productivity.
- Respect websites and services. Check terms, rate limits, robots directives where applicable, privacy requirements and service permissions before collecting data at scale.
And no, the lesson is not “anyone can build anything in three hours.” Our experience was a relatively simple, first-generation automation task. Production-grade tooling still requires maintenance, monitoring and technical judgment.
8. At GreyFox Ads, ingenuity is part of the product
We are a small French programmatic ad network, and that makes resourcefulness especially important. Our approach combines local knowledge of the French Open Web with hands-on curation, practical quality signals and transparent supply selection.
We also explore how Curator Seats can make these choices actionable for media buyers, and how better inventory signals complement more meaningful campaign KPIs than viewability alone.
For media traders outside France, our Master Curated Deal brings French-market expertise into a single activation route. And yes, we still like building clever little tools when they solve a real problem. 🤘
Activate quality French Open Web inventory
Looking for a partner combining French publisher knowledge, practical technology and curated programmatic activation?
Discover the Master DealContact GreyFox AdsOriginal story: Read the original LinkedIn post. The figures about development time, debugging and site coverage reflect the author's firsthand account.