简体中文 | English
WebMainBench is a specialized benchmark tool for end-to-end evaluation of web main content extraction quality.
- Multiple Extractor Support: Supports various extraction tools such as trafilatura, resiliparse, and more
- Comprehensive Evaluation Metrics: Includes multi-dimensional metrics such as text edit distance, table structure similarity (TEDS), formula extraction quality, etc.
- Manual Annotation Support: 100% manually annotated evaluation dataset
| Metric Name | Calculation Method | Value Range | Description |
|---|---|---|---|
overall |
Average of all successful metrics | 0.0-1.0 | Comprehensive quality score, higher is better |
text_edit |
1 - (edit distance / max text length) |
0.0-1.0 | Plain text similarity, higher is better |
code_edit |
1 - (edit distance / max code length) |
0.0-1.0 | Code content similarity, higher is better |
table_TEDS |
1 - (tree edit distance / max nodes) |
0.0-1.0 | Table structure similarity, higher is better |
table_edit |
1 - (edit distance / max table length) |
0.0-1.0 | Table content similarity, higher is better |
formula_edit |
1 - (edit distance / max formula length) |
0.0-1.0 | Formula content similarity, higher is better |
- data module: Read/write management of evaluation sets and results
- extractors module: Unified interface for various extraction tools
- metrics module: Implementation of evaluation metrics calculation
- evaluator module: Execution and result output of evaluation tasks
# Basic installation pip install webmainbench # Install with all optional dependencies pip install webmainbench[all] # Development environment installation pip install webmainbench[dev]
from webmainbench import DataLoader, Evaluator, ExtractorFactory # 1. Load evaluation dataset dataset = DataLoader.load_jsonl("your_dataset.jsonl") # 2. Create extractor extractor = ExtractorFactory.create("trafilatura") # 3. Run evaluation evaluator = Evaluator() result = evaluator.evaluate(dataset, extractor) # 4. View results print(f"Overall Score: {result.overall_metrics['overall']:.4f}")
Evaluation datasets should contain the following fields:
{
"track_id": "0b7f2636-d35f-40bf-9b7f-94be4bcbb396",
"html": "<html><body><h1 cc-select=\"true\">This is a title</h1></body></html>", # Manually annotated with cc-select="true" attribute
"url": "https://orderyourbooks.com/product-category/college-books-p-u/?products-per-page=all",
"main_html": "<h1 cc-select=\"true\">This is a title</h1>", # Main content HTML pruned from html
"convert_main_content": "# This is a title", # Converted from main_html + html2text
"groundtruth_content": "# This is a title", # Manually calibrated markdown (partially provided)
"meta": {
"language": "en", # Web page language
"style": "artical", # Web page style
"table": [], # [], ["layout"], ["data"], ["layout", "data"]
"equation": [], # [], ["inline"], ["interline"], ["inline", "interline"]
"code": [], # [], ["inline"], ["interline"], ["inline", "interline"]
"level": "mid" # simple, mid, hard
}
}- trafilatura: trafilatura extractor
- resiliparse: resiliparse extractor
- llm-webkit: llm-webkit extractor
- magic-html: magic-html extractor
- Custom extractors: Implement by inheriting from
BaseExtractor
| extractor | extractor_version | dataset | total_samples | overall (macro avg) | code_edit | formula_edit | table_TEDS | table_edit | text_edit |
|---|---|---|---|---|---|---|---|---|---|
| llm-webkit | 4.1.1 | WebMainBench1.0 | 545 | 0.8256 | 0.9093 | 0.9399 | 0.7388 | 0.678 | 0.8621 |
| magic-html | 0.1.5 | WebMainBench1.0 | 545 | 0.5141 | 0.4117 | 0.7204 | 0.3984 | 0.2611 | 0.7791 |
| trafilatura_md | 2.0.0 | WebMainBench1.0 | 545 | 0.3858 | 0.1305 | 0.6242 | 0.3203 | 0.1653 | 0.6887 |
| trafilatura_txt | 2.0.0 | WebMainBench1.0 | 545 | 0.2657 | 0 | 0.6162 | 0 | 0 | 0.7126 |
| resiliparse | 0.14.5 | WebMainBench1.0 | 545 | 0.2954 | 0.0641 | 0.6747 | 0 | 0 | 0.7381 |
# Compare multiple extractors extractors = ["trafilatura", "resiliparse"] results = evaluator.compare_extractors(dataset, extractors) for name, result in results.items(): print(f"{name}: {result.overall_metrics['overall']:.4f}")
python examples/multi_extractor_compare.py
This example demonstrates how to:
- Load test dataset: Use sample data containing multiple content types such as code, formulas, tables, text, etc.
- Create multiple extractors:
magic-html: Extractor based on magic-html librarytrafilatura: Extractor based on trafilatura libraryresiliparse: Extractor based on resiliparse library
- Batch evaluation comparison: Use
evaluator.compare_extractors()to evaluate all extractors simultaneously - Generate comparison report: Automatically save evaluation results in multiple formats
After evaluation is complete, three important files will be generated in the results/ directory:
| File Name | Format | Content Description |
|---|---|---|
leaderboard.csv |
CSV | Leaderboard file: Contains overall rankings and sub-metric comparisons for each extractor, for quick performance comparison |
evaluation_results.json |
JSON | Detailed evaluation results: Contains complete evaluation data, metric details and metadata for each extractor |
dataset_with_results.jsonl |
JSONL | Enhanced dataset: Original test data plus extraction results from all extractors, for manual inspection and analysis |
leaderboard.csv content example:
extractor,dataset,total_samples,success_rate,overall,code_edit,formula_edit,table_TEDS,table_edit,text_edit magic-html,sample_dataset,4,1.0,0.1526,0.1007,0.0,0.0,0.0,0.6624 resiliparse,sample_dataset,4,1.0,0.1379,0.0,0.0,0.0,0.0,0.6897 trafilatura,sample_dataset,4,1.0,0.1151,0.1007,0.0,0.0,0.0,0.4746
from webmainbench.metrics import BaseMetric, MetricResult class CustomMetric(BaseMetric): def _setup(self): pass def _calculate_score(self, predicted, groundtruth, **kwargs): # Implement custom evaluation logic score = your_calculation(predicted, groundtruth) return MetricResult( metric_name=self.name, score=score, details={"custom_info": "value"} ) # Add to evaluator evaluator.metric_calculator.add_metric("custom", CustomMetric("custom"))
from webmainbench.extractors import BaseExtractor, ExtractionResult class MyExtractor(BaseExtractor): def _setup(self): # Initialize extractor pass def _extract_content(self, html, url=None): # Implement extraction logic content = your_extraction_logic(html) return ExtractionResult( content=content, content_list=[...], success=True ) # Register custom extractor ExtractorFactory.register("my-extractor", MyExtractor)
webmainbench/
├── data/ # Data processing module
│ ├── dataset.py # Dataset class
│ ├── loader.py # Data loader
│ └── saver.py # Data saver
├── extractors/ # Extractor module
│ ├── base.py # Base interface
│ ├── factory.py # Factory pattern
│ └── ... # Specific implementations
├── metrics/ # Metrics module
│ ├── base.py # Base interface
│ ├── text_metrics.py # Text metrics
│ ├── table_metrics.py # Table metrics
│ └── calculator.py # Metric calculator
├── evaluator/ # Evaluator module
│ └── evaluator.py # Main evaluator
└── utils/ # Utility module
└── helpers.py # Helper functions
This project is licensed under the MIT License - see the LICENSE file for details.