n8n-nodes-free-web-scrapping/README.md

Node n8n for searching and extracting content from web pages via DuckDuckGo and free scraping.

Web Scraper (Free)

A **no-API-key**, DuckDuckGo-based search & scraping node for **n8n**.  
Perform searches and extract page content—all without paying for an API or maintaining credentials.

---

## 🚀 Features

1. **Search URLs only**  
   Perform a DuckDuckGo search and return the top N result URLs.

2. **Extract single page**  
   Fetch any web page and return its title, H1 headers, and paragraphs.

3. **Filter page content**  
   Fetch a page and return only the parts you care about:  
   - Headings (H1, H2, H3)  
   - Paragraphs  
   - Bold text  
   - **All** of the above

4. **Query + Full Scrape**  
   Combines “Search URLs only” and “Extract single page”:  
   - Search DuckDuckGo for a query  
   - Fetch and parse each resulting URL  

5. **Browser-like HTTP requests**  
   - Custom **User-Agent** header  
   - Configurable **timeout** to prevent hanging workflows  

6. **Built-in blacklist**  
   Automatically skips links from social, video, and other noise domains (e.g. YouTube, Facebook, Instagram, TikTok, Reddit, etc.).

7. **Zero dependencies**  
   No external APIs or credentials required—just install once and start scraping.

---

## 📦 Installation

1. In your n8n instance, open **Settings → Community Nodes → Install**  
2. In **npm Package Name** enter:
```

n8n-nodes-free-web-scrapping

````
3. Click **Install** and restart n8n if prompted.

---

## 🔧 Configuration

When you add the **Web Scraper (Free)** node, you’ll see:

### 1. Operation
| Value           | Description                                           |
|-----------------|-------------------------------------------------------|
| `searchUrls`    | DuckDuckGo search → return URLs only                  |
| `extractPage`   | Fetch a single page → extract headers & paragraphs    |
| `filterContent` | Fetch a single page → return only filtered content    |
| `fullScrape`    | DuckDuckGo search → fetch & extract content of URLs   |

### 2. Parameters by Operation

#### **Search URLs only** & **Full Scrape**
- **Search Query** (string)  
Text to search for on DuckDuckGo.
- **Maximum URLs** (number)  
How many top links to process (1–20).

#### **Extract single page** & **Filter page content**
- **Page URL** (string)  
The URL of the page you want to scrape.

#### **Filter page content** only
- **Filter Type** (options)  
- `headings` → H1, H2, H3  
- `paragraphs` → `<p>` elements  
- `bolds` → `<b>` and `<strong>`  
- `all` → combinations of the above  

---

## 📖 Examples

### 1. Search URLs only
```json
[
{ "url": "https://openai.com/gpt-4" },
{ "url": "https://en.wikipedia.org/wiki/GPT-4" }
]
````

### 2. Extract single page

```json
[
  {
    "url": "https://en.wikipedia.org/wiki/Web_scraping",
    "title": "Web scraping - Wikipedia",
    "h1": ["Web scraping"],
    "paragraphs": [
      "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites..."
    ]
  }
]
```

### 3. Filter page content

```json
[
  {
    "url": "https://example.com",
    "filterType": "headings",
    "data": {
      "h1": ["Main Heading"],
      "h2": ["Section Title", "Another Section"],
      "h3": ["Subsection"]
    }
  }
]
```

### 4. Query + Full Scrape

```json
[
  {
    "url": "https://example.com/page1",
    "title": "Example Page",
    "h1": ["Example Heading"],
    "paragraphs": ["First paragraph...", "Second paragraph..."]
  }
]
```

---

## 🔍 Blacklist

```
youtube.com, instagram.com, facebook.com, twitter.com,
tiktok.com, snapchat.com, linkedin.com, pinterest.com,
reddit.com, twitch.tv, vimeo.com, dailymotion.com,
bilibili.com, youku.com
```

---

## ⚙️ Under the Hood

* Uses **got** for HTTP requests.
* Parses DuckDuckGo’s HTML endpoint:
  `https://html.duckduckgo.com/html?q=<query>`
* Extracts links via regex on the `result__a` class.
* Cleans HTML with regex-based strip functions (scripts, styles, comments, tags).
* All operations **return JSON**—ready for downstream nodes.