scrapegraph-js
Version:
Scrape and extract structured data from a webpage using ScrapeGraphAI's APIs.
282 lines (207 loc) • 6.92 kB
Markdown
[](https://badge.fury.io/js/scrapegraph-js) [](https://opensource.org/licenses/MIT) [](https://docs.scrapegraphai.com)
<p align="left">
<img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png" alt="ScrapeGraph API Banner" style="width: 70%;">
</p>
Official JavaScript/TypeScript SDK for the ScrapeGraph AI API - Smart web scraping powered by AI.
- ✨ Smart web scraping with AI
- 🔄 Fully asynchronous design
- 🔍 Detailed error handling
- ⚡ Automatic retries and logging
- 🔐 Secure API authentication
Install the package using npm or yarn:
```bash
npm i scrapegraph-js
yarn add scrapegraph-js
```
> **Note**: Store your API keys securely in environment variables. Use `.env` files and libraries like `dotenv` to load them into your app.
```javascript
import { smartScraper } from 'scrapegraph-js';
import 'dotenv/config';
// Initialize variables
const apiKey = process.env.SGAI_APIKEY; // Set your API key as an environment variable
const websiteUrl = 'https://example.com';
const prompt = 'What does the company do?';
(async () => {
try {
const response = await smartScraper(apiKey, websiteUrl, prompt);
console.log(response.result);
} catch (error) {
console.error('Error:', error);
}
})();
```
```javascript
import { smartScraper } from 'scrapegraph-js';
const apiKey = 'your-api-key';
const url = 'https://example.com';
const prompt = 'Extract the main heading and description.';
(async () => {
try {
const response = await smartScraper(apiKey, url, prompt);
console.log(response.result);
} catch (error) {
console.error('Error:', error);
}
})();
```
> [!NOTE]
> To use this feature, it is necessary to employ the [Zod](https://www.npmjs.com/package/zod) package for schema creation.
Here is a real-world example:
```javascript
import { smartScraper } from 'scrapegraph-js';
import { z } from 'zod';
const apiKey = 'your-api-key';
const url = 'https://scrapegraphai.com/';
const prompt = 'What does the company do? and ';
const schema = z.object({
title: z.string().describe('The title of the webpage'),
description: z.string().describe('The description of the webpage'),
summary: z.string().describe('A brief summary of the webpage'),
});
(async () => {
try {
const response = await smartScraper(apiKey, url, prompt, schema);
console.log(response.result);
} catch (error) {
console.error('Error:', error);
}
})();
```
Search and extract information from multiple web sources using AI.
```javascript
import { searchScraper } from 'scrapegraph-js';
const apiKey = 'your-api-key';
const prompt = 'What is the latest version of Python and what are its main features?';
(async () => {
try {
const response = await searchScraper(apiKey, prompt);
console.log(response.result);
} catch (error) {
console.error('Error:', error);
}
})();
```
Extract structured data from local HTML content
```javascript
import { localScraper } from 'scrapegraph-js';
const apiKey = 'your_api_key';
const prompt = 'What does the company do?';
const websiteHtml = `<html>
<body>
<h1>Company Name</h1>
<p>We are a technology company focused on AI solutions.</p>
<div class="contact">
<p>Email: contact@example.com</p>
</div>
</body>
</html>`;
(async () => {
try {
const response = await localScraper(apiKey, websiteHtml, prompt);
console.log(response);
} catch (error) {
console.error(error);
}
})();
```
Converts a webpage into clean, well-structured markdown format.
```javascript
import { smartScraper } from 'scrapegraph-js';
const apiKey = 'your_api_key';
const url = 'https://scrapegraphai.com/';
(async () => {
try {
const response = await markdownify(apiKey, url);
console.log(response);
} catch (error) {
console.error(error);
}
})();
```
```javascript
import { getCredits } from 'scrapegraph-js';
const apiKey = 'your-api-key';
(async () => {
try {
const credits = await getCredits(apiKey);
console.log('Available credits:', credits);
} catch (error) {
console.error('Error fetching credits:', error);
}
})();
```
```javascript
import { sendFeedback } from 'scrapegraph-js';
const apiKey = 'your-api-key';
const requestId = '16a63a80-c87f-4cde-b005-e6c3ecda278b';
const rating = 5;
const feedbackText = 'This is a test feedback message.';
(async () => {
try {
const response = await sendFeedback(apiKey, requestId, rating, feedbackText);
console.log('Feedback response:', response);
} catch (error) {
console.error('Error sending feedback:', error);
}
})();
```
For detailed documentation, visit [docs.scrapegraphai.com](https://docs.scrapegraphai.com)
1. Clone the repository:
```bash
git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git
cd scrapegraph-sdk/scrapegraph-js
```
2. Install dependencies:
```bash
npm install
```
3. Run linting and testing:
```bash
npm run lint
npm test
```
```bash
npm test
npm run test:coverage
```
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
## 🔗 Links
- [Website](https://scrapegraphai.com)
- [Documentation](https://docs.scrapegraphai.com)
- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)
## 💬 Support
- 📧 Email: support@scrapegraphai.com
- 💻 GitHub Issues: [Create an issue](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues)
- 🌟 Feature Requests: [Request a feature](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues/new)
---
Made with ❤️ by [ScrapeGraph AI](https://scrapegraphai.com)