๐ค Botview
High-performance prerendering microservice for Single Page Applications (SPA)
Botview makes your client-side rendered web applications visible to search engine crawlers and social media bots by providing fully rendered HTML content. Built with NestJS and Playwright, it delivers fast, reliable prerendering with minimal resource usage.
โจ Features
| Feature |
Description |
| ๐ SEO Optimization |
Makes SPA content visible to search engine crawlers (Google, Bing, Yandex, etc.) |
| ๐ฑ Social Media Ready |
Enables proper link previews for Facebook, Twitter, Telegram, and other platforms |
| ๐ Framework Agnostic |
Works with React, Vue, Angular, Svelte, jQuery, StencilJS, and any JavaScript framework |
| โก High Performance |
Lightweight Chromium engine with configurable resource blocking |
| ๐ Basic Auth Support |
Prerender pages behind HTTP Basic Authentication |
| ๐ Prometheus Metrics |
Built-in monitoring with /metrics endpoint |
| ๐ฏ Custom Status Codes |
Control HTTP response codes via meta tags |
| ๐ณ Docker Ready |
Production-ready container with multi-stage build |
๐ Table of Contents
๐ Quick Start
Using Docker (Recommended)
# Run in foreground (for testing)
docker run -it --rm -p 3000:3000 mtsrus/botview
# Run in background (for production)
docker run -d --restart always -p 3000:3000 --name botview mtsrus/botview
Test the Service
Open your browser or use curl:
curl http://localhost:3000/render/https://example.com/
The response will contain the fully rendered HTML of the target page.
๐ API Reference
Render Endpoint
GET /render/{url}
| Parameter |
Type |
Description |
url |
string |
URL-encoded target page to render |
Example:
# Render a page
curl http://localhost:3000/render/https://spa-website.com/products
# URL with query parameters (must be URL-encoded)
curl "http://localhost:3000/render/https://spa-website.com/search?q=test"
Response:
- Content-Type:
text/html
- Status Code:
200 (or custom via meta tag)
Custom Status Codes
Add a meta tag to your HTML to control the response status code:
<head>
<!-- Return 301 redirect status -->
<meta name="prerender-status" content="301" />
<!-- Return 404 not found -->
<meta name="prerender-status" content="404" />
</head>
Metrics Endpoint
GET /metrics
Returns Prometheus-formatted metrics for monitoring.
โ๏ธ Configuration
All configuration is done via environment variables:
Timeout Settings
| Variable |
Default |
Description |
BOTVIEW_NAV_TIMEOUT |
30000 |
Maximum navigation timeout in milliseconds (docs) |
BOTVIEW_DEFAULT_TIMEOUT |
15000 |
Default timeout for page operations in milliseconds (docs) |
BOTVIEW_WAIT_UNTIL |
networkidle |
Wait condition: load, domcontentloaded, networkidle, or commit (docs) |
Viewport Settings
| Variable |
Default |
Description |
BOTVIEW_VIEWPORT |
360x640 |
Browser viewport size in WIDTHxHEIGHT format |
Example:
docker run -d -p 3000:3000 \
-e BOTVIEW_VIEWPORT="1920x1080" \
mtsrus/botview
Resource Blocking
Improve performance by blocking unnecessary resources:
| Variable |
Default |
Description |
BOTVIEW_BLOCK_IMAGES |
true |
Block image loading |
BOTVIEW_BLOCK_STYLESHEETS |
true |
Block CSS loading |
BOTVIEW_BLOCK_FONTS |
true |
Block font loading |
BOTVIEW_BLOCK_MEDIA |
true |
Block audio/video loading |
BOTVIEW_BLOCK_URLS |
https://an.yandex.ru, https://mc.yandex.ru |
URL prefixes to block (comma/space separated) |
BOTVIEW_BLOCK_URLS_REGEX |
`` |
Regex patterns to block URLs (comma/space separated) |
Example:
docker run -d -p 3000:3000 \
-e BOTVIEW_BLOCK_URLS="https://google-analytics.com,https://facebook.com/tr" \
-e BOTVIEW_BLOCK_URLS_REGEX=".*\.ads\..*,.*tracking.*" \
mtsrus/botview
Authentication
| Variable |
Default |
Description |
BOTVIEW_BASIC_AUTHS |
`` |
HTTP Basic Auth credentials |
Format: url:username:password (comma or space separated for multiple entries)
docker run -d -p 3000:3000 \
-e BOTVIEW_BASIC_AUTHS="https://protected.example.com:admin:secret123" \
mtsrus/botview
Logging
| Variable |
Default |
Description |
BOTVIEW_LOG_LEVEL |
info |
Log verbosity: trace, debug, info, warn, error, fatal |
๐ Performance Optimization
Default Optimizations
Botview comes pre-configured for optimal performance:
- โ
Images blocked (reduces bandwidth)
- โ
Stylesheets blocked (not needed for SEO)
- โ
Fonts blocked (reduces requests)
- โ
Media files blocked (reduces load time)
- โ
Yandex analytics blocked (prevents timeout issues)
Custom Blocking Rules
Block specific URLs causing slowdowns:
# Block by URL prefix
-e BOTVIEW_BLOCK_URLS="https://slow-cdn.com,https://heavy-analytics.com"
# Block by regex pattern
-e BOTVIEW_BLOCK_URLS_REGEX=".*facebook.*widget.*,.*twitter.*embed.*"
Recommended Production Settings
docker run -d --restart always -p 3000:3000 \
--memory="512m" \
--cpus="1" \
-e BOTVIEW_NAV_TIMEOUT=20000 \
-e BOTVIEW_DEFAULT_TIMEOUT=10000 \
-e BOTVIEW_LOG_LEVEL=warn \
--name botview \
mtsrus/botview
๐ Monitoring
Prometheus Metrics
Access metrics at http://localhost:3000/metrics:
curl http://localhost:3000/metrics
Available metrics include:
- Node.js process metrics (memory, CPU, event loop)
- HTTP request metrics
- Garbage collection metrics
Prometheus Configuration
# prometheus.yml
scrape_configs:
- job_name: 'botview'
static_configs:
- targets: ['botview:3000']
metrics_path: '/metrics'
Security Note
Block the /metrics endpoint from external access in production:
location /metrics {
deny all;
# or allow only internal IPs
# allow 10.0.0.0/8;
# deny all;
}
๐ง Proxy Setup
Configure your reverse proxy to route bot traffic to Botview.
Nginx Configuration
location / {
set $prerender 0;
# Detect search engine and social media bots
if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|telegrambot|linkedinbot|whatsapp|slackbot|discordbot|pinterest|applebot") {
set $prerender 1;
}
# Support for escaped fragment (legacy)
if ($args ~ "_escaped_fragment_") {
set $prerender 1;
}
# Don't prerender for Botview's own requests
if ($http_user_agent ~ "Prerender") {
set $prerender 0;
}
# Don't prerender static files
if ($uri ~ "\.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|pdf|zip)$") {
set $prerender 0;
}
# Route to Botview for bots
if ($prerender = 1) {
rewrite (.*) /render/$scheme://$host$1 break;
proxy_pass http://botview:3000;
}
# Route to application for regular users
if ($prerender = 0) {
proxy_pass http://app:80;
}
}
Docker Compose Example
version: '3.8'
services:
app:
image: your-spa-app
ports:
- "80:80"
botview:
image: mtsrus/botview
restart: always
environment:
- BOTVIEW_LOG_LEVEL=info
ports:
- "3000:3000"
nginx:
image: nginx:alpine
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
ports:
- "443:443"
depends_on:
- app
- botview
๐ Troubleshooting
Timeout Errors
Symptom: Requests timeout with "Leaked requests" in logs.
Cause: Some network requests don't complete, blocking page load.
Solution:
Check logs for leaked request URLs:
docker logs botview | grep "Leaked requests"
Block problematic URLs:
docker run -d -p 3000:3000 \
-e BOTVIEW_BLOCK_URLS="https://problematic-url.com" \
mtsrus/botview
Memory Issues
Solution: Limit container memory and enable resource blocking:
docker run -d --memory="512m" \
-e BOTVIEW_BLOCK_IMAGES=true \
-e BOTVIEW_BLOCK_STYLESHEETS=true \
mtsrus/botview
Debug Mode
Enable verbose logging:
docker run -it --rm -p 3000:3000 \
-e BOTVIEW_LOG_LEVEL=debug \
mtsrus/botview
๐ ๏ธ Development
Prerequisites
- Node.js 20+ (LTS recommended)
- npm 10+
Local Setup
# Clone repository
git clone https://github.com/MobileTeleSystems/botview.git
cd botview
# Install dependencies
npm install
# Run in development mode
npm run start:dev
# Run tests
npm test
# Run e2e tests
npm run test:e2e
# Build for production
npm run build
Project Structure
src/
โโโ main.ts # Application entry point
โโโ app.module.ts # Root NestJS module
โโโ config.ts # Configuration management
โโโ LogLevels.ts # Log level definitions
โโโ controllers/
โ โโโ render-controller/ # Main render endpoint
โ โโโ metrics/ # Prometheus metrics endpoint
โโโ services/
โ โโโ prerender.service.ts # Core prerendering logic
โ โโโ json-logger.service.ts # Structured JSON logging
โโโ middleware/
โ โโโ RequestLoggerMiddleware.ts # Request logging
โโโ models/
โโโ LeakedRequests.ts # Request tracking model
๐ค Contributing
We welcome contributions! Please see our Contributing Guide for details.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature)
- Commit your changes (
git commit -m 'feat: add amazing feature')
- Push to the branch (
git push origin feature/amazing-feature)
- Open a Pull Request
๐ License
This project is licensed under the MIT License - see the LICENSE file for details.
๐ Acknowledgments
Made with โค๏ธ by MobileTeleSystems