GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Add a scraper check utility · Issue #124 · openzim/python-scraperlib

Add a scraper check utility #124

New issue
Copy link
New issue
Copy link
Open
Enhancement
Open
Add a scraper check utility#124
Enhancement
Copy link
Labels
pending clarificationWaiting for a clarification and/or a way to reproduce the problemWaiting for a clarification and/or a way to reproduce the problem
Milestone
backlog

Description

@benoit74
benoit74
opened on Feb 5, 2024
Issue body actions

Currently, we rely on various objects in scraperlib to:

  • create the ZIM
  • re-encode videos and images
  • cache these assets on the optimization cache

We might consider to have a mechanism to perform sanity checks on scraper behavior:

  • did we cached all re-encoded images / videos when a cache is present?
  • did we removed temporary files from the filesystem as they are added to the ZIM? (we know that while we prefer in-memory/streaming approaches, there are still many scrapers which are using the temporary file approach, and even some situation which have to rely on it)

What I do not yet know:

  • should we make the scraper fails if these checks fails?
  • is there any chance we automate these checks? (i.e. no need to modify the scrapers, or as little as possible - at least not make a call to "check_i_m_ok" mandatory, because the scraper developers might forget about it as well ; I doubt about this because there are many kind of situations)
  • can we do these checks early? (so that we fail the scraper asap instead of wasting time and resources)

Activity

  1. benoit74
    added
    enhancementDeprecated: use issue type EnhancementDeprecated: use issue type Enhancement
    questionDeprecated: use issue type QuestionDeprecated: use issue type Question
    on Feb 5, 2024
  2. benoit74
    added this to the backlog milestone on Nov 14, 2024
  3. benoit74
    removed
    enhancementDeprecated: use issue type EnhancementDeprecated: use issue type Enhancement
    questionDeprecated: use issue type QuestionDeprecated: use issue type Question
    on Oct 1, 2026
  4. benoit74
    added the
    Enhancement
    issue type on Oct 1, 2026
  5. pawank925 commented on Oct 19, 2026

    @pawank925
    pawank925
    on Oct 19, 2026
    Contributor
    More actions

    @benoit74 can i get assigned to this please!

  6. benoit74
    added
    pending clarificationWaiting for a clarification and/or a way to reproduce the problemWaiting for a clarification and/or a way to reproduce the problem
    on Oct 19, 2026
  7. benoit74 commented on Oct 19, 2026

    @benoit74
    benoit74
    on Oct 19, 2026
    CollaboratorAuthor
    More actions

    This issue is not ready to be worked on unfortunately

    Reacted by pawank925
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pending clarificationWaiting for a clarification and/or a way to reproduce the problemWaiting for a clarification and/or a way to reproduce the problem

    Type

    Enhancement

    Projects

    No projects

      Milestone

      • backlogNo due date

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402503 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.