A robust Chrome extension that scrapes webpage content and converts it to Markdown format for data analysis.
- Single Page Scraping: Extract content from the current webpage
- Multi-Page Crawling: Automatically crawl and scrape multiple pages within the same domain
- Robust Error Handling: Continues crawling even when individual pages fail
- Multiple Output Methods:
- Download as Markdown file
- Copy to clipboard
- Smart Content Extraction: Uses Mozilla's Readability.js for intelligent content parsing
- Configurable Settings:
- Max pages limit (up to 1000)
- Concurrent tabs (1-10)
- Request delay
- Crawl mode toggle
- Clone or download this repository
- Install dependencies:
npm install
- Build the extension:
npm run build
- Open Chrome and navigate to
chrome://extensions/ - Enable "Developer mode" in the top right
- Click "Load unpacked"
- Select the
dist/folder from this project
- Navigate to the webpage you want to scrape
- Click the extension icon in Chrome toolbar
- Configure settings:
- Crawl sub-pages: Enable to scrape linked pages from the same domain
- Max pages: Maximum number of pages to scrape (1-1000)
- Concurrency: Number of simultaneous tabs (1-10)
- Delay: Milliseconds to wait between requests
- Copy to clipboard: Enable to copy output to clipboard
- Download as file: Enable to download as .md file
- Click "Start" to begin scraping
- Monitor progress in the popup
- Output will be downloaded/copied when complete
src/
├── background/ # Service worker and core logic
├── content/ # Content script for page extraction
├── popup/ # Extension popup UI
├── lib/ # Third-party libraries (Readability.js)
└── shared/ # Shared utilities and configuration
npm test- Run unit testsnpm run build- Build extension to dist/npm run lint- Run ESLintnpm run format- Format code with Prettier
The extension generates a Markdown file with:
- Table of Contents with links to each page
- Full content of each scraped page
- Page URLs and titles
Example output structure:
# Table of Contents
1. [Page Title 1](https://example.com/page1)
2. [Page Title 2](https://example.com/page2)
---
# Page Title 1
**URL:** htt
4DEB
ps://example.com/page1
[Page content...]
---
# Page Title 2
**URL:** https://example.com/page2
[Page content...]