As a user that gets the Cloudflare thingy, unless it is a website that I really need, I just close the tab as soon as I their captacha starts the loading stuff. If the site admin/masters do not care about the end user, I do not care about their site either. Users should vote with their feet (or clicks?).
We care, we just can't spend the resources self-hosting business critical infrastructure. It'll break, get AI-DoSed, or will have an expired cert. It's a boring and risky problem to solve. Cloudflare provides a really good service for basically free.
If you add additional piece to chain, chain becomes weaker, not stronger
> get AI-DoSed
Thats not that common. There are specific industries prone to DDoS, like gaming, but your average site don't get DDoS-ed. Then again CF free service really don't protect your site from DDoS. I have seen several times CF becoming source of DoS (not caching or denying malicious requests) and if back-end is on shared infra, CF goes to firewall.
It's not free at all. You give them access to your unencrypted traffic and they give you protection from attacks they finance themselves. You're being ripped off
If it is a site you use a lot (which is happening more and more) it's important to make sure the site owner is aware of the problem. I usually do this by email. Then they can ask cloudflare to fix it and get rebuffed and sour their opinion of cloudflare as they start having to maintain long UA-string whitelists.
Show site owners cloudflare isn't doing the job they are paid to do.
Designing a site to not be affected by bots is fixing the problem. Blocking things that are bot-like, poorly, with no technical understanding, is a kludge, not a fix.
Building a WAF that has zero false positives and zero false negatives is impossible. All we can ask is that the companies that build WAFs be responsive, but they also need accurate bug reports with sufficient information to identify the variable.
That's already been established. My point is that there exists a tradeoff. Blocking a small number of legitimate users can be worth the benefits blocking the bots.
Except the Cloudflare firewall is just so goddamn dumb. It considers all of Asia a "bot", anybody who uses Linux or has a tracker blocker. I've had this complaint from several regular people who don't even have an adblocker.
Be honest about your User-Agent and get you blocked, pretend to be Chrome and all the problems go away. It creates a strong incentive for people to lie about User-Agent.
user-agent had and will never be useful for the client, and therefore, cannot ever be useful for the server in trying to identify someone.
If it is advantagious to lie about the client, the client will lie, and the server will know this and not trust the client's headers.
Blocking or restricting based on other metrics should be the way to go - for example, shape traffic based on the speed and frequency, such that bots gets shaped but users' dont (because they're slower etc). It aligns properly with the desire to stop bots.
> user-agent had and will never be useful for the client
In the past it was definitely useful for some things. To use as an example something I was involved with, many years ago: if you were deploying software via ClickOnce and needed a particular version of .NET installed, that information was exposed in the user-agent string, so you could tell the user “you need to go install this other thing first”, rather than saying “try this, and if it doesn’t work in one of these ways, check if you’ve installed this”.
These days, it’s not often useful, and feature detection is highly preferable where possible, but platform detection (whether via user-agent string or properties like navigator.platform) is still definitely useful, for things that should be tied to what the platform actually is. For example, on a download page you want to know what the OS and architecture are in order to recommend the appropriate file to download for that machine. Or, when implementing keyboard shortcuts, Apple platforms use metaKey (⌘) instead of ctrlKey (⌃), and visually present the shortcuts differently too, such as ⌘⌥⇧⌫ or Command-Option-Shift-Delete, compared with Ctrl+Alt+Shift+Backspace for everyone else.
Even Chrome User-Agent string is full of lies. In HTTP/3 they should have just get rid of this travesty with: "Client MUST NOT send an User-Agent header".
Most anti-bots will send other browser characteristics as well and can detect if one is lying about which browser, and even which version they're using based on which APIs and properties are/aren't available.
UA strings, just like any other client information forwarded to the server (see: android/iOS data accessible by JS & the server) only exists to build a profile to better filter who you want to serve ads to. It's always ads. Anything else is an excuse, period.
That's what'll trigger the CF captcha as well: if you don't give up client data. Run a site on Chrome bundled with your Anroid? Never going to get captcha, since Google will gladly serve any of your data when it's asked.
Even when using Firefox (with RFP) I often get captchas that say "You have been blocked" after you solve it. I don't bother with sites asking for captchas anymore. If I see a captcha, I close the tab, since there is no guarantee I'll be getting the page if I solve it anyways. Sorry, I'm not training your AI for free. I hope someone will sue captcha providers for accessibility, which if it succeeds, will break captchas forever.
Will Cloudflare people try to get us to believe that they "didn't know" or say that it only affects a tiny number of people, or somehow try to gaslight us in to thinking that non-mainstream browsers are a threat to something or another?
For a company that has so many of their employees on this site, they sure do seem to be clueless when their supposedly amazing tech marginalizes people or otherwise creates issues. Even when they respond in these threads, it takes ages for them to address things, and often the problems are never permanently fixed and come up again after some time.
It reminds me of this fortune(6):
As far as we know, our computer has never had an undetected error.
-- Weisert
Perhaps fans of Cloudflare will downvote instead of engage, but that's almost a given these days. Let's be real - Cloudflare has been given a free pass for far too long. If you disagree and REALLY believe that there're technically valid reasons for punishing people who use non-mainstream browsers, try to actually engage and discuss. I'm truly interested in someone's take about why we're doing something wrong by not using what everyone else is using.
There's a fundamental tension here between scale and openness. Cloudflare's bot detection likely relies on statistical models[1] trained on mainstream browser behavior. When you operate at their scale, even a 0.1% false positive rate affects millions (billions?) of users, but the economic incentives push toward minimizing false negatives (actual bots getting through to prevent leaking) rather than false positives (real users getting blocked).
The real issue isn't just Cloudflare - it's the centralization of web infrastructure that creates these single points of failure for browser diversity. Alternative browsers aren't just about personal preference - they're essential for innovation and preventing technological monoculture. We should be demanding transparent criteria for what constitutes "suspicious" behavior rather than accepting black-box filtering.
This has been going on for so long, other parties feel justified in their similar decisions. E.g. try this accessible-for-some-browsers search result page behind the fastly "Client Challenge": https://pypi.org/search/?q=pip
Does Cloudflare’s almost completely single point control switch of a bottleneck, on a lot of the Internet, get discussed enough? At this point I just give up before trying if I see their captcha and already start to write that site off, if it starts showing that transient “verifying your humaneness” splash screen.
Hey, HN mentioned on general news! Is that a first?
"According to some in the Hacker News discussion of the problem, something else that can count as suspicious – other than using niche browsers or OSes – is something as simple as asking for a URL unaccompanied by any referrer IDs. To us, that sounds like a user with good security measures that block tracking, but it seems that, to the CDN merchant, this looks like an alert to an action that isn't operated by a human."
The Register is a pretty tech advanced journalist outfit from the UK, AFAIK. We also see occasional HN references in tech articles from Ars Technica, etc.
Major issue is that reaching their support is next to impossible. Payment for my domain is not going through and now I am waiting for my domain to get expired and be available in market again, so that I can buy it from other vendor.
Cloudflare is gathering the filler for the masses so we can get to the content faster and easier. Next milestone is automating and search result filter. God speed.
The forum link in the first paragraph is a broken link.
The one about Pale Moon from 2015 suggests the user did something custom and it was seen by CloudFlare (which is acting as a WAF) as something like an SQL Injection[1].
All of the CloudFlare hate on this site is tiresome and borders on Crying Wolf.
Websites aren’t forced to use it. It’s affordable and gives DDoS protection. If reducing false positives for bot/malicious traffic detection were more reliable, this would already be solved.
What a disgrace. People from Cloudflare have been trying to attract the open web community for years now, including posting on HN (hi, jgrahamc!), but they're interfering with a major part of the open web. Now all alternative browsers have to play the user agent faking game.
At what point did anyone think cloudflare were the good guys? It’s a facade. They have shown multiple times over the years they are not ok the side of the users. They will try sweep things under the rug unless it ends up on HN or something.
It has been always necessary to fake the user agent for browsers. Too many sites would complain otherwise.
25 years ago it was necessary to pretend to be MSIE. Then IceBrowser, a Java based browser, was doing that. But one of the customers for the company asked to provide a way to test for IceBrowser as the browser did not emulated MSIE properly in some cases so they wanted to provide a workaround. So a JavaScript-based test was provided. Then in couple of years a common JS library for browser detection had started to include that test and the test has to be disabled.
Then there was a story about Vivaldi browser from 5 years ago. Due to a bug in keyboard-only navigation there was an issue on google.com and apparently Google has implemented a workaround based on user-agent sniffing before the bug was fixed. Then, when Vivaldi fixed the issue, the fix broke google.com as Google was unaware that it should disable their workaround. That was the last straw forcing Vivaldi to fake the user agent by default.
Unfortunately I think it's just statistics killing browser diversity, not conspiracy or collusion.
The math doesn't care about our freedom of choice. Tech savvy users making alternative choices on their web experience are an extreme minority in the sum total of HTTP requests. But the outcome is the same: a narrowing web where only mainstream options function reliably.
The ironic part, as everyone here understands, is that those who actually understand technology enough to use alternative browsers or privacy tools are the ones getting locked out. We're punishing ourselves for our technical literacy by implementing these strategies at these companies. And it really does help the average person who does not think about their browser choices.
It is like the drunk man looking for his lost keys under the lamp because that is where the light is (https://en.wikipedia.org/wiki/Streetlight_effect). For-profit companies are only able to perceive bleeding edge javascript running browsers because that is their "light". They can't see you and you don't exist and certainly don't matter if you don't execute their code properly.
What I'm saying is that there are more people (who are not tech savvy) not using Chrome than you may think. But you'd never know from the statistics. The collection method for the statistics is inherently biased.
Hate to be that guy, but it’s about time to stop advertising the User Agent. Where needed the same could be done exposing capabilities, and it wouldn’t have to be on every single http request. Benefits: less bandwidth, blocking by capabilities would require JS. Negatives: revisit screen readers, which could keep using UA, no more stats by browser (don’t care).
It's worth noting that Ladybird, the most promising and growing independent browser implementation, is not surprisingly getting hit by this too: https://github.com/LadybirdBrowser/ladybird/issues/226
Seeing https://news.ycombinator.com/item?id=43321145 and https://news.ycombinator.com/item?id=43322922 also showing up in a close timespan to this makes me really suspicious that there was some part of a hidden plan to close off the Internet which suddenly took a significant step.
I thought that Cloudflare et al. just had a whitelist based on the SSL fingerprint.
As a user that gets the Cloudflare thingy, unless it is a website that I really need, I just close the tab as soon as I their captacha starts the loading stuff. If the site admin/masters do not care about the end user, I do not care about their site either. Users should vote with their feet (or clicks?).
We care, we just can't spend the resources self-hosting business critical infrastructure. It'll break, get AI-DoSed, or will have an expired cert. It's a boring and risky problem to solve. Cloudflare provides a really good service for basically free.
> It'll break
If you add additional piece to chain, chain becomes weaker, not stronger
> get AI-DoSed
Thats not that common. There are specific industries prone to DDoS, like gaming, but your average site don't get DDoS-ed. Then again CF free service really don't protect your site from DDoS. I have seen several times CF becoming source of DoS (not caching or denying malicious requests) and if back-end is on shared infra, CF goes to firewall.
> will have an expired cert.
Your back-end still needs certificate
It's not free at all. You give them access to your unencrypted traffic and they give you protection from attacks they finance themselves. You're being ripped off
> attacks they finance themselves
Care to provide details or source?
Cloudflare provides free hosting to booters and stressers.
If it is a site you use a lot (which is happening more and more) it's important to make sure the site owner is aware of the problem. I usually do this by email. Then they can ask cloudflare to fix it and get rebuffed and sour their opinion of cloudflare as they start having to maintain long UA-string whitelists.
Show site owners cloudflare isn't doing the job they are paid to do.
Blocking bots can improve the user experience of the end users.
Designing a site to not be affected by bots is fixing the problem. Blocking things that are bot-like, poorly, with no technical understanding, is a kludge, not a fix.
That is not always possible, may not be affordable, or may degrade the overall user experience.
> with no technical understanding
WTF?
Building a WAF that has zero false positives and zero false negatives is impossible. All we can ask is that the companies that build WAFs be responsive, but they also need accurate bug reports with sufficient information to identify the variable.
I would ask that the companies that build WAFs please stop, honestly.
I don't think you understood the parent.
> Designing a site to not be affected by bots is fixing the problem.
Bots footprint is minimal and shouldn't affect your site performance.
Bots have negative impacts beyond potential performance impact:
- scraping & content theft
- spam & fake engagement
- DDoS attacks
- Ad fraud
> Blocking things that are bot-like, poorly, with no technical understanding, is a kludge, not a fix.
a kludge might be better than none. And it's an easy kludge (for the site owner - a checkbox in cloudflare).
The fault lies with cloudflare implementing a lazy bot detector.
And blocking users can make the user experience worse.
That's already been established. My point is that there exists a tradeoff. Blocking a small number of legitimate users can be worth the benefits blocking the bots.
Maybe, but it has to be a really sharp ratio.
Except the Cloudflare firewall is just so goddamn dumb. It considers all of Asia a "bot", anybody who uses Linux or has a tracker blocker. I've had this complaint from several regular people who don't even have an adblocker.
I just put my content behind a paywall. If the bots want to pay $50/mo I guess I don't mind. No cloudflare!
Be honest about your User-Agent and get you blocked, pretend to be Chrome and all the problems go away. It creates a strong incentive for people to lie about User-Agent.
user-agent had and will never be useful for the client, and therefore, cannot ever be useful for the server in trying to identify someone.
If it is advantagious to lie about the client, the client will lie, and the server will know this and not trust the client's headers.
Blocking or restricting based on other metrics should be the way to go - for example, shape traffic based on the speed and frequency, such that bots gets shaped but users' dont (because they're slower etc). It aligns properly with the desire to stop bots.
> user-agent had and will never be useful for the client
In the past it was definitely useful for some things. To use as an example something I was involved with, many years ago: if you were deploying software via ClickOnce and needed a particular version of .NET installed, that information was exposed in the user-agent string, so you could tell the user “you need to go install this other thing first”, rather than saying “try this, and if it doesn’t work in one of these ways, check if you’ve installed this”.
These days, it’s not often useful, and feature detection is highly preferable where possible, but platform detection (whether via user-agent string or properties like navigator.platform) is still definitely useful, for things that should be tied to what the platform actually is. For example, on a download page you want to know what the OS and architecture are in order to recommend the appropriate file to download for that machine. Or, when implementing keyboard shortcuts, Apple platforms use metaKey (⌘) instead of ctrlKey (⌃), and visually present the shortcuts differently too, such as ⌘⌥⇧⌫ or Command-Option-Shift-Delete, compared with Ctrl+Alt+Shift+Backspace for everyone else.
Even Chrome User-Agent string is full of lies. In HTTP/3 they should have just get rid of this travesty with: "Client MUST NOT send an User-Agent header".
Most anti-bots will send other browser characteristics as well and can detect if one is lying about which browser, and even which version they're using based on which APIs and properties are/aren't available.
see: https://blog.castle.io/anti-detect-browser-analysis-how-to-d...
Just be honest like Chrome which pretends to be Netscape, Apple Webkit and Safari
UA strings, just like any other client information forwarded to the server (see: android/iOS data accessible by JS & the server) only exists to build a profile to better filter who you want to serve ads to. It's always ads. Anything else is an excuse, period.
That's what'll trigger the CF captcha as well: if you don't give up client data. Run a site on Chrome bundled with your Anroid? Never going to get captcha, since Google will gladly serve any of your data when it's asked.
Even when using Firefox (with RFP) I often get captchas that say "You have been blocked" after you solve it. I don't bother with sites asking for captchas anymore. If I see a captcha, I close the tab, since there is no guarantee I'll be getting the page if I solve it anyways. Sorry, I'm not training your AI for free. I hope someone will sue captcha providers for accessibility, which if it succeeds, will break captchas forever.
Captchas have become so annoying to solve. I've spent minutes resolving captchas over and over without success so many times in recent memory.
Not saying MITM is a bad term but Cloudflare is the single biggest MITM.
Will Cloudflare people try to get us to believe that they "didn't know" or say that it only affects a tiny number of people, or somehow try to gaslight us in to thinking that non-mainstream browsers are a threat to something or another?
For a company that has so many of their employees on this site, they sure do seem to be clueless when their supposedly amazing tech marginalizes people or otherwise creates issues. Even when they respond in these threads, it takes ages for them to address things, and often the problems are never permanently fixed and come up again after some time.
It reminds me of this fortune(6):
Perhaps fans of Cloudflare will downvote instead of engage, but that's almost a given these days. Let's be real - Cloudflare has been given a free pass for far too long. If you disagree and REALLY believe that there're technically valid reasons for punishing people who use non-mainstream browsers, try to actually engage and discuss. I'm truly interested in someone's take about why we're doing something wrong by not using what everyone else is using.There's a fundamental tension here between scale and openness. Cloudflare's bot detection likely relies on statistical models[1] trained on mainstream browser behavior. When you operate at their scale, even a 0.1% false positive rate affects millions (billions?) of users, but the economic incentives push toward minimizing false negatives (actual bots getting through to prevent leaking) rather than false positives (real users getting blocked).
The real issue isn't just Cloudflare - it's the centralization of web infrastructure that creates these single points of failure for browser diversity. Alternative browsers aren't just about personal preference - they're essential for innovation and preventing technological monoculture. We should be demanding transparent criteria for what constitutes "suspicious" behavior rather than accepting black-box filtering.
[1] https://developers.cloudflare.com/bots/plans/biz-and-ent/#he...
This has been going on for so long, other parties feel justified in their similar decisions. E.g. try this accessible-for-some-browsers search result page behind the fastly "Client Challenge": https://pypi.org/search/?q=pip
Does Cloudflare’s almost completely single point control switch of a bottleneck, on a lot of the Internet, get discussed enough? At this point I just give up before trying if I see their captcha and already start to write that site off, if it starts showing that transient “verifying your humaneness” splash screen.
PyPI uses (depends on, to be more precise) fastly.
I get the fastly transient every time I use it.
Hey, HN mentioned on general news! Is that a first?
"According to some in the Hacker News discussion of the problem, something else that can count as suspicious – other than using niche browsers or OSes – is something as simple as asking for a URL unaccompanied by any referrer IDs. To us, that sounds like a user with good security measures that block tracking, but it seems that, to the CDN merchant, this looks like an alert to an action that isn't operated by a human."
The Register is a pretty tech advanced journalist outfit from the UK, AFAIK. We also see occasional HN references in tech articles from Ars Technica, etc.
Major issue is that reaching their support is next to impossible. Payment for my domain is not going through and now I am waiting for my domain to get expired and be available in market again, so that I can buy it from other vendor.
Cloudflare is gathering the filler for the masses so we can get to the content faster and easier. Next milestone is automating and search result filter. God speed.
The forum link in the first paragraph is a broken link.
The one about Pale Moon from 2015 suggests the user did something custom and it was seen by CloudFlare (which is acting as a WAF) as something like an SQL Injection[1].
All of the CloudFlare hate on this site is tiresome and borders on Crying Wolf.
Websites aren’t forced to use it. It’s affordable and gives DDoS protection. If reducing false positives for bot/malicious traffic detection were more reliable, this would already be solved.
[1] https://forum.palemoon.org/viewtopic.php?t=7096#p45805
Looks like it's broken link because Cloudflare deleted the forum post.
Google's still got it in their search results:
https://www.google.com/search?q=https%3A%2F%2Fcommunity.clou...
What a disgrace. People from Cloudflare have been trying to attract the open web community for years now, including posting on HN (hi, jgrahamc!), but they're interfering with a major part of the open web. Now all alternative browsers have to play the user agent faking game.
At what point did anyone think cloudflare were the good guys? It’s a facade. They have shown multiple times over the years they are not ok the side of the users. They will try sweep things under the rug unless it ends up on HN or something.
It has been always necessary to fake the user agent for browsers. Too many sites would complain otherwise.
25 years ago it was necessary to pretend to be MSIE. Then IceBrowser, a Java based browser, was doing that. But one of the customers for the company asked to provide a way to test for IceBrowser as the browser did not emulated MSIE properly in some cases so they wanted to provide a workaround. So a JavaScript-based test was provided. Then in couple of years a common JS library for browser detection had started to include that test and the test has to be disabled.
Then there was a story about Vivaldi browser from 5 years ago. Due to a bug in keyboard-only navigation there was an issue on google.com and apparently Google has implemented a workaround based on user-agent sniffing before the bug was fixed. Then, when Vivaldi fixed the issue, the fix broke google.com as Google was unaware that it should disable their workaround. That was the last straw forcing Vivaldi to fake the user agent by default.
> Now all alternative browsers have to play the user agent faking game.
Wait, what? Were they not submitting a fake agent before? What's the benefit to the user or the browser of submitting a novel user agent string?
Even the mainstream ones were doing it. This is about referrers, which seem also fakeable?
Unfortunately I think it's just statistics killing browser diversity, not conspiracy or collusion.
The math doesn't care about our freedom of choice. Tech savvy users making alternative choices on their web experience are an extreme minority in the sum total of HTTP requests. But the outcome is the same: a narrowing web where only mainstream options function reliably.
The ironic part, as everyone here understands, is that those who actually understand technology enough to use alternative browsers or privacy tools are the ones getting locked out. We're punishing ourselves for our technical literacy by implementing these strategies at these companies. And it really does help the average person who does not think about their browser choices.
It is like the drunk man looking for his lost keys under the lamp because that is where the light is (https://en.wikipedia.org/wiki/Streetlight_effect). For-profit companies are only able to perceive bleeding edge javascript running browsers because that is their "light". They can't see you and you don't exist and certainly don't matter if you don't execute their code properly.
What I'm saying is that there are more people (who are not tech savvy) not using Chrome than you may think. But you'd never know from the statistics. The collection method for the statistics is inherently biased.
Hate to be that guy, but it’s about time to stop advertising the User Agent. Where needed the same could be done exposing capabilities, and it wouldn’t have to be on every single http request. Benefits: less bandwidth, blocking by capabilities would require JS. Negatives: revisit screen readers, which could keep using UA, no more stats by browser (don’t care).
[flagged]