GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GitHub - code402/warc-benchmark: Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.

BranchesTags
Open more actions menu

Latest commit

 

History

1 Commit
1 Commit

Folders and files

NameName
Last commit message
Last commit date
bash
bash
 
 
code402
code402
 
 
go
go
 
 
java
java
 
 
node
node
 
 
python
python
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 

Repository files navigation

wat-benchmark

This repository acts as a Hello World for working with WARC files.

Its subfolders contain implementations that fetch a WARC file and search all captures from .com domains for a regex that detects YouTube links.

See also the blog post.

This is not bulletproof, production-ready code - I/O retries, closing resources and robust character decoding is omitted to focus on the WARC aspect of the code.

About

Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.

code402.com/hello-warc-common-crawl-code-samples

Topics

common-crawlcommoncrawlwarc

Resources

Readme
Apache-2.0 license
Activity
Custom properties

Stars

8 stars

Watchers

2 watching

Forks

6 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.