I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).
Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.
I think it's inevitable that eventually one of the big AI companies makes something like this. It's such an obvious consumer win. "I am not a bot" checkboxes are a string and peg tethering the elephant.
Thanks for letting us know that we need a new layer of detection systems.
Also it’s great(!) to see that we’re going from “but ethics” to “I got mine, who cares”.
Humans are interesting creatures.
Edit: Please before assuming that I'm assuming things, this is an observation I'm making over time. It's possible that I'm in a bubble, but it's not a sample size of 1 (i.e. The comment I replied only).
I'm becoming more and more convinced that a big source of outrage on the internet is caused by people assuming that all other people are a homogenous blob.
It's not that "we" are going from one thing to another. It's that these are two different people, with different ethical boundaries.
There is no detection method that will prevent AI from accessing systems without also blocking humans. The only thing we can do at this point is throttling.
Agreed on inevitable collateral blockage of real people using real "headed" browser. I'm getting a ton of that already, personally.
Throttling is poor help though. Mass scrapers are using "residential proxy" loophole + rotating UA and other attributes. You can't throttle somebody without identifying them. Unless you're talking about a global rate-limit.
throttling based on sessions kind of works; we're headed in the direction that sites like Reddit will probably prevent logged-out users from viewing threads (as they already do with mobile devices)
Once the LLMs create sockpuppets to get around that, the web services will need to resort to profiling users more aggressively so that they know which actual human an account corresponds to.
If someone has a malicious browser extension that uses their session to scrape Reddit then, they're probably going to see significant usage obstacles.
You're framing this as if people are deliberately making decisions that they believe are unethical. The reality is that people have different ethical frameworks. For example, I believe that there is no ethical distinction between whether a web request originates from a browser or from an LLM on my behalf.
It is, because `--remote-debugging-port` is detected via the root DOM object (and that can't be changed unless you want to recompile Chromium), so for example, if you try doing a Google search with debugging enabled, you'll get blocked (usually just by being served a blank page).
I think it's the latter, and that's why I think using a "normal" non-debug browser is the best idea because it's essentially undetectable. And for the record, you can technically control Chromium using IPC if you feel like adding that feature & fully rebuilding it from scratch.
`navigator.webdriver` afaik, but I think there's a couple more, I did some research on it a few months ago. If using Playwright/Puppeteer, they also have a few special things injected that you have to strip.
I built the POC by hand (just had like 3 things it could do: scroll, click, type), and then Codex used my established patterns to make it slightly more general and cleaned it up.
Yep, it's super similar to this! I think his is a bit overengineered, but tbh I haven't built Firefox plugins in forever so that might be the way you have to do stuff on FF. All you need is the websocket plugin to have "full access" to all visited webpages, and then your driver just hooks into the DOM like normal (with the extra ws layer—and even the websocket layer can be simplified away I think).
Are you saying they're blocking the `User-Agent`? Because that's trivial to bypass. I'm not exactly sure what you mean by "Chromium" because that's a whole class of browsers (Edge, Brave, Chrome, etc. are all forked "Chromium" browsers).
Aliexpress specifically uses a javascript blob to do detection, and quite a sophisticated one. There was a recent story about that blob messing with bluetooth headphones.
Here's my agentic Chromium browser browsing Aliexpress[1]. Usually I'd run it in headless mode, but wanted to test and make sure I'm getting the website proper.
It searched for "Aliexpress" on Google and clicked the link, hence the Google UTM params.
It's sad, but this train has left the station a quarter of a century ago imo. Many people have since become billionaires scraping the web without anyone agreeing (Google, Yahoo, and now possibly Anthropic and OpenAI).
There is a huge difference between a clearly identifiable, robots.txt-respecting, largely symbiotic search engine spider versus this new AI-driven casual sociopathy of firing shotguns of deliberately masked bots at sites in ways which benefit no one except for, maybe, the bot-owner.
I agree with you, though, that the future has almost certainly already been decided, and see the eventual outcome of all this selfishness to be mandatory device attestation to access most services on the internet—which will of course never be allowed on any open platform. AI engineers who believed in freedom to compute (or just individual freedom more generally) have committed perhaps the greatest self-own in the history of our species to date.
anyone can spin these kind of side projects and do easy talk, but the moment you actually try to use this on signed-in Linkedin or Amazon its going to fail
the only solution is to drive your regular browser with all your sessions/cookies via an extension
> the only solution is to drive your regular browser with all your sessions/cookies via an extension
This is exactly what I'm doing, but mocking/randomizing all the sessions/cookies/params (like resolution, OS, WebGL , etc.) in a separate Chromium binary. It's popular these days, but imo using your normal browser for agentic stuff is a very bad idea. These models do dumb stuff all the time.
no it doesn't fail. Agents are very good at comparing traffic characteristics from a real browser and a headless/automation browser and getting it to behave in the same manner. It's a cat and mouse game for sites stopping unauthorized access but right now llm agents are ahead.
For testing proxies should be used to avoid IP ban issues but given enough time modern agents can figure out how to bypass most of the modern scraping/automation prevention mechanisms.
I have built and used a lot of different automations and web scraping implementations for my business and it's never got permanently stuck yet, some take a bit longer, some shorter, but all within a reasonable time with little external help they have succeeded in their tasks.
As someone having to fight Meta's bots everyday to keep websites accessible to actual customers, I'm not surprised to read that it can be seen as something positive on the other side of the fence.
The end game will be that either your site is fully open to bots and humans alike, or your site is open to humans only(×) and requires Airport security style identity verification.
I think this is the end of sites where it is expected that only humans may interact with them. It's been cat and mouse for a while, and some places like Reddit sell API access, but these agents are for all intents and purposes, humans interacting with the site. They're going to need to figure out new business models.
Every time a new tool launches there's a good window where it can act as a scraping proxy. Back in the 2010s I used Google Translate for years to scrape hard targets like LinkedIn but these windows are much shorter these days as scraping is so much bigger.
One thing with Muse though is that you can scrape Meta's own sites which are currently all going under login walls and restricting discovery/search entirely.
similarly, gemini via google ai studio will happily run workloads over youtube that would be very annoying to run at scale. Especially if you needed to download the video.
Most residential proxies are already far more blocked and rate limited than any Meta IP. The internet is becoming a very weird place, where individual and "trusted" personal IPs are becoming a kind of commodity. Some sites are already scoring IPs based on usage activity - like a credit score. It's only a matter of time until this data is collated and commoditised. AI analysis is turning this up to 11.
> Some sites are already scoring IPs based on usage activity - like a credit score. It's only a matter of time until this data is collated and commoditised.
Spamhaus is nearly thirty years old and the notion of electronic distribution of IP and domain "reputation" lists is at least that old.
I'll bet my hat that the Internet "advertising" [0] industry has been calculating and determining the reputation of individual households (if not individual users) for at least a decade.
[0] The scare quotes are because its primary purpose these days is for dragnet private-sector surveillance.
Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.
Also it’s great(!) to see that we’re going from “but ethics” to “I got mine, who cares”.
Humans are interesting creatures.
Edit: Please before assuming that I'm assuming things, this is an observation I'm making over time. It's possible that I'm in a bubble, but it's not a sample size of 1 (i.e. The comment I replied only).
It's not that "we" are going from one thing to another. It's that these are two different people, with different ethical boundaries.
Throttling is poor help though. Mass scrapers are using "residential proxy" loophole + rotating UA and other attributes. You can't throttle somebody without identifying them. Unless you're talking about a global rate-limit.
Once the LLMs create sockpuppets to get around that, the web services will need to resort to profiling users more aggressively so that they know which actual human an account corresponds to.
If someone has a malicious browser extension that uses their session to scrape Reddit then, they're probably going to see significant usage obstacles.
We are headed to a very user-hostile place.
Is defeating captchas unethical? I don’t think so. Not on its own.
Is scraping unethical? I don’t think so. Not on its own.
Are there tons of uses for both of those that are sketchy or outright wrong? Yeah, absolutely.
Is that necessary? You could start Chromium with an open debug port and use Chrome Devtools Protocol to send commands.
I've had success launching the browser and using dumb dumb methods to get around the captcha before attaching playwright.
Dumb dumb methods = wmctrl, xdotool, bash (work fine for Cloudflare's "are you human" check)
Assuming "more people catching onto" this, expect Cloudflare and most everyone else to follow the suite.
> not sure what you mean by "Chromium"
I meant that literally. This thing: https://www.chromium.org/getting-involved/download-chromium/...
It searched for "Aliexpress" on Google and clicked the link, hence the Google UTM params.
[1] https://imgur.com/a/eNulg11
I agree with you, though, that the future has almost certainly already been decided, and see the eventual outcome of all this selfishness to be mandatory device attestation to access most services on the internet—which will of course never be allowed on any open platform. AI engineers who believed in freedom to compute (or just individual freedom more generally) have committed perhaps the greatest self-own in the history of our species to date.
the only solution is to drive your regular browser with all your sessions/cookies via an extension
This is exactly what I'm doing, but mocking/randomizing all the sessions/cookies/params (like resolution, OS, WebGL , etc.) in a separate Chromium binary. It's popular these days, but imo using your normal browser for agentic stuff is a very bad idea. These models do dumb stuff all the time.
For testing proxies should be used to avoid IP ban issues but given enough time modern agents can figure out how to bypass most of the modern scraping/automation prevention mechanisms.
I have built and used a lot of different automations and web scraping implementations for my business and it's never got permanently stuck yet, some take a bit longer, some shorter, but all within a reasonable time with little external help they have succeeded in their tasks.
But I'm wondering: at what cost?
(×) and their AI delegates
So of course I clicked yes and it dutifully convinced the site that it was not a bot.
Muse's utility has significantly dropped with the blockages.
To become truly useful again they will need to use residential proxies, but I can't see them use those due to the risks and reputational damage.
One thing with Muse though is that you can scrape Meta's own sites which are currently all going under login walls and restricting discovery/search entirely.
Spamhaus is nearly thirty years old and the notion of electronic distribution of IP and domain "reputation" lists is at least that old.
I'll bet my hat that the Internet "advertising" [0] industry has been calculating and determining the reputation of individual households (if not individual users) for at least a decade.
[0] The scare quotes are because its primary purpose these days is for dragnet private-sector surveillance.