WebEngine Nanobrowser not able to pass Antibot Validation

[RicTa]

While trying Nano Browser, the QT WebEngine demo project, I discovered that it does not pass any of the most used Antibot checks, for example the cloudflare or Akamai ones.
This is an extremely worrying issue because just this week Cloudflare released to public the new Antibot called Turnstile and it has great advantages compared to Google captcha. Hence people will migrate soon.
Now, below an example of failure while attempting the login at the same Cloudflare main website.

Also https://web.whatsapp.com/ , Digital Ocean and others are not working neither.
I have correlated some behaviour from multiple browsers (using https://browserleaks.com/client-hints as way to check the browser transmitted headers) to try to understand why Nano is triggering the antibot response, although webengine and Nano browser are not at all bot neither scraping tools.

as we can see from the values in RED, Nano QT Web Engine browser reply with some very weird values, while all the other browsers (still based on chromium), they transmit legitimate or undefined values and they can pass the antibot checks without any problem.
My question is, how can we change these values? or even better, by when QT will fix this problem that if now is relatively small, will become huge once the website starts migrating to Cloudflare Turnstile.

[JKSH]

Hi @RicTa, and welcome!

If you set a custom user-agent string (see https://doc.qt.io/qt-6/qwebengineprofile.html#setHttpUserAgent ) are you able to get it to pass?

[RicTa]

Hi @JKSH thanks for the prompt reply.
unfortunately the UA seems not to be the core issue for this problem.

I have tested dozen of UA in the recent months and the results are the following:
Original QT UA: test failed
Chromium UA: test failed or passed but after few hours/days the test fails again
Completely fake UA: initially passed but after few hours/days the test fails again
Firefox UA: initially passed but after few hours/days the test fails again

The antibot system is testing hundreds of Browser information and compared with the baseline of other most used browsers.

I have adapted all the HTTP Request Headers to be equal to chromium and other browser but this was not enough neither.

The core issue I feel it's in the Browser details that are queried using the Chromium generic API, because Webengine is returning completely empty either unrealistic values. (see red fields in the attached picture in my post).

[RicTa]

An example of HTTP Request header i also had to add is the "Accept-Language" one that was not filled by Webengine but largely needed for the user experience and to pass some antibot checks.
Below the comparison of some of the HTTP Request headers in different Browsers as per the website:
https://httpbin.org/headers
https://browserspy.dk/headers.php

[RicTa]

@JKSH
I might have found something interesting.

Visiting the page https://51degrees.com/client-hints it's clear how webegine does not make any UA-CH headers visible to the server, hence these "empty" values.

So the problem could be relatively easy solved if we find a way to let webengine pass the UA-CH values as a normal browser does.

Any idea?

[JKSH]

@RicTa Good detective work! Would you be willing to submit a bug report for this? https://bugreports.qt.io/

[RicTa]

Hi @JKSH ,
indeed in the meantime I have open this bug here:
https://bugreports.qt.io/browse/QTBUG-107451
please let me know in case I made some mistakes in creating it.
thanks a lot for you support

[JKSH]

@RicTa said in WebEngine Nanobrowser not able to pass Antibot Validation:

indeed in the meantime I have open this bug here:
https://bugreports.qt.io/browse/QTBUG-107451
please let me know in case I made some mistakes in creating it.
thanks a lot for you support

Thanks! Your bug report looks fine to me, and I see that you are having a productive discussion with an engineer who is working on Qt WebEngine.