🤖 Botview

✨ Features

📋 Table of Contents

🚀 Quick Start

Using Docker (Recommended)

# Run in foreground (for testing)
docker run -it --rm -p 3000:3000 mtsrus/botview

# Run in background (for production)
docker run -d --restart always -p 3000:3000 --name botview mtsrus/botview

Test the Service

curl http://localhost:3000/render/https://example.com/

📖 API Reference

Render Endpoint

GET /render/{url}

# Render a page
curl http://localhost:3000/render/https://spa-website.com/products

# URL with query parameters (must be URL-encoded)
curl "http://localhost:3000/render/https://spa-website.com/search?q=test"

Custom Status Codes

<head>
  <!-- Return 301 redirect status -->
  <meta name="prerender-status" content="301" />

  <!-- Return 404 not found -->
  <meta name="prerender-status" content="404" />
</head>

Metrics Endpoint

GET /metrics

⚙️ Configuration

Timeout Settings

Viewport Settings

docker run -d -p 3000:3000 \
  -e BOTVIEW_VIEWPORT="1920x1080" \
  mtsrus/botview

Resource Blocking

docker run -d -p 3000:3000 \
  -e BOTVIEW_BLOCK_URLS="https://google-analytics.com,https://facebook.com/tr" \
  -e BOTVIEW_BLOCK_URLS_REGEX=".*\.ads\..*,.*tracking.*" \
  mtsrus/botview

Authentication

docker run -d -p 3000:3000 \
  -e BOTVIEW_BASIC_AUTHS="https://protected.example.com:admin:secret123" \
  mtsrus/botview

Logging

🚀 Performance Optimization

Default Optimizations

Custom Blocking Rules

# Block by URL prefix
-e BOTVIEW_BLOCK_URLS="https://slow-cdn.com,https://heavy-analytics.com"

# Block by regex pattern
-e BOTVIEW_BLOCK_URLS_REGEX=".*facebook.*widget.*,.*twitter.*embed.*"

Recommended Production Settings

docker run -d --restart always -p 3000:3000 \
  --memory="512m" \
  --cpus="1" \
  -e BOTVIEW_NAV_TIMEOUT=20000 \
  -e BOTVIEW_DEFAULT_TIMEOUT=10000 \
  -e BOTVIEW_LOG_LEVEL=warn \
  --name botview \
  mtsrus/botview

📊 Monitoring

Prometheus Metrics

curl http://localhost:3000/metrics

Prometheus Configuration

# prometheus.yml
scrape_configs:
  - job_name: 'botview'
    static_configs:
      - targets: ['botview:3000']
    metrics_path: '/metrics'

Security Note

location /metrics {
    deny all;
    # or allow only internal IPs
    # allow 10.0.0.0/8;
    # deny all;
}

🔧 Proxy Setup

Nginx Configuration

location / {
    set $prerender 0;

    # Detect search engine and social media bots
    if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|telegrambot|linkedinbot|whatsapp|slackbot|discordbot|pinterest|applebot") {
        set $prerender 1;
    }

    # Support for escaped fragment (legacy)
    if ($args ~ "_escaped_fragment_") {
        set $prerender 1;
    }

    # Don't prerender for Botview's own requests
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }

    # Don't prerender static files
    if ($uri ~ "\.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|pdf|zip)$") {
        set $prerender 0;
    }

    # Route to Botview for bots
    if ($prerender = 1) {
        rewrite (.*) /render/$scheme://$host$1 break;
        proxy_pass http://botview:3000;
    }

    # Route to application for regular users
    if ($prerender = 0) {
        proxy_pass http://app:80;
    }
}

Docker Compose Example

version: '3.8'
services:
  app:
    image: your-spa-app
    ports:
      - "80:80"

  botview:
    image: mtsrus/botview
    restart: always
    environment:
      - BOTVIEW_LOG_LEVEL=info
    ports:
      - "3000:3000"

  nginx:
    image: nginx:alpine
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    ports:
      - "443:443"
    depends_on:
      - app
      - botview

🔍 Troubleshooting

Timeout Errors

Memory Issues

docker run -d --memory="512m" \
  -e BOTVIEW_BLOCK_IMAGES=true \
  -e BOTVIEW_BLOCK_STYLESHEETS=true \
  mtsrus/botview

Debug Mode

docker run -it --rm -p 3000:3000 \
  -e BOTVIEW_LOG_LEVEL=debug \
  mtsrus/botview

🛠️ Development

Prerequisites

Local Setup

# Clone repository
git clone https://github.com/MobileTeleSystems/botview.git
cd botview

# Install dependencies
npm install

# Run in development mode
npm run start:dev

# Run tests
npm test

# Run e2e tests
npm run test:e2e

# Build for production
npm run build

Project Structure

src/
├── main.ts                          # Application entry point
├── app.module.ts                    # Root NestJS module
├── config.ts                        # Configuration management
├── LogLevels.ts                     # Log level definitions
├── controllers/
│   ├── render-controller/           # Main render endpoint
│   └── metrics/                     # Prometheus metrics endpoint
├── services/
│   ├── prerender.service.ts         # Core prerendering logic
│   └── json-logger.service.ts       # Structured JSON logging
├── middleware/
│   └── RequestLoggerMiddleware.ts   # Request logging
└── models/
    └── LeakedRequests.ts            # Request tracking model

🤝 Contributing

📄 License

🙏 Acknowledgments