GeezerVison is showing github.com — Dismiss cookie permission, GDPR, and subscription popups before archiving · Issue #175 · ArchiveBox/ArchiveBox
Dismiss cookie permission, GDPR, and subscription popups before archiving #175DescriptionReacted by Atarii, hankman13, André Kelpe, Johannes, reepje123, Andreas Lämmlein, Sophie, Murat Çorlu, DMW007, MisterrrX and 1 more Activitypirate commented on Mar 15, 2019 pirate commented on Nov 18, 2022 datoslabs commented on Jan 7, 2025 Hi,
I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue.
Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up!
The test I choose was artnet.com, it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio:
In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control. GlassedSilver commented on Jan 21, 2025
Hi,
I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue.
Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up!
The test I choose was artnet.com, it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio:
In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control.
This has major privacy implications, maybe optionally this could be fine, but I think a local AI model would suit this purpose better if you want to involve AI at all.
Personally I use Consent-o-Matic in my browsers and it not only auto okays all cookies, it even accepts or denies based on what the site lets you control and what you desire. For example allowing functional cookies for things like embeds but keeping trackers out.
It's truly set and forget, rarely misses and could be a fantastic consideration to source from. It even clicks your choices in a hidden manner.
https://youtu.be/LtgyVYwjlGE pirate commented on Jan 21, 2025 Yes for paid clients I'm already using a mixture of local LLMs and hosted models to detect and solve CAPTCHAs, hide cookie popups, and do QA analysis of captures to determine if there's any obsctructions to seeing the content.
None of it is open source right now because it's the only thing I can charge for to sustain the rest of the project. Reacted by Glassed Silver and Mela Eckenfels pirate commented on Dec 29, 2025 ok this is implemented on dev using archivebox/plugins/modalcloser and archivebox/plugins/istilldontcareaboutcookies, it's a simple approach for now but we can extend it in the future with something like stagehand/browser-use if needed. Reacted by Glassed Silver GlassedSilver commented on Jan 7, 2026
ok this is implemented on dev using archivebox/plugins/modalcloser and archivebox/plugins/istilldontcareaboutcookies, it's a simple approach for now but we can extend it in the future with something like stagehand/browser-use if needed.
This is great news and makes me very happy to see!
Very thankful for this, just one note that if anyone else extends on this in the future or maybe you yourself might I suggest using Consent-o-Matic? pirate commented on Jan 27, 2026 I have tested both extensively and unfortunately consent-o-matic breaks a lot of browser automations and makes the headless browsers flakyer in general (e.g. sometimes it'll open a new features/config page on first browser launch that steals foreground tab focus away from the page we're trying to archive). I've stuck with istilldontcareaboutcookies for now but if there is enough demand we could add content-o-matic as an opt-in choice as well.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
