flipkart-scraper
Version:
[](https://nodei.co/npm/flipkart-scraper/)
247 lines (186 loc) • 5.4 kB
Markdown
[](https://nodei.co/npm/flipkart-scraper/)
# Flipkart Scraper 
This package will help us to scrape all Flipkart products through Flipkart affiliate API.
Please refer [API Documentation](https://hi-imcodeman.github.io/flipkart-scraper) here.
See the [Examples](https://github.com/hi-imcodeman/flipkart-scraper/tree/master/examples) here
## Installation
Install using 'npm'
```sh
npm i flipkart-scraper
```
Install using 'yarn'
```sh
yarn add flipkart-scraper
```
## Usage
```javascript
import { FlipkartScraper } from "flipkart-scraper";
const scraper = new FlipkartScraper(
"<Affiliate-Id-Here>",
"<Affiliate-Token-Here>"
);
// 'data' event handler
scraper.on("data", (data) => {
console.log(data.products.length);
});
// Start the scraper
scraper.start();
```
## Class: FlipkartScraper
- extends [EventEmitter](https://nodejs.org/api/events.html#events_class_eventemitter)
This module will help us to scrape all the Flipkart products.
### constructor(affiliateId, affiliateToken, [options])
This will create instance of the `FlipkartScraper` class with required authentication params.
- options {object}
- `concurrency?` {number} Number for parallel processing in the queue, Default set to `2`
- `maxRequest?` {number} Maximum request to Flipkart affliate server, Default set to `0` - means unlimited
- `maxPage?` {number} Maximum number of pages to scrape per category, Default set to `0` - means unlimited
**Example**
```javascript
const scraper = new FlipkartScraper(
"<Affiliate-Id-Here>",
"<Affiliate-Token-Here>",
{
/**
* It will make 5 parallel request to Flipkart.
* This is optional param, default is set to 2
**/
concurrency: 5,
/**
* It will make only 500 request to Flipkart. After that program ends.
* This is optional param, default is set to 0 means unlimited
**/
maxRequest: 500,
/**
* Maximum 3 request per category
**/
maxPage: 3,
}
);
```
### scraper.start([categoriesToScrape])
This method will start scraping through Flipkart affiliate API.
- `categoriesToScrape?` {string[]} Pass the list of categories that you want to scrape. Default set to `[]` which means all categories.
**Example**
```javascript
scraper.start(["telivision", "mobiles"]); // It will scrape only specified categories
```
### scraper.stats(showAsNumbers?=false)
This method will show the stats of scraper. By default stats will come as numerals like (3.1k, 1.45GB)
- `showAsNumbers?` {boolean} Stats will return as number instead of numerals.
**Sample Stats**
```javascript
/*
{
"startTime": "2021-02-21T08:14:29.445Z",
"endTime": undefined, // endTime will be available once scraping finished
"status": "inprogress",
"concurrency": 30,
"waitingRequests": 0,
"productsCount": "19.54k",
"elapsed": "0:00:06 10ms",
"durationPerMillionProducts": "0:05:08 558ms",
"productsPerSec": "3.25k products/sec",
"avgResponseTime": "147ms",
"requestPerSec": "7/sec",
"requestedCount": "43.00",
"processedCount": "41.00",
"errorCount": "0.0",
"retryCount": "0.0",
"retryHaltCount": "0.0",
"pendingCategory": 2,
"completedCategory": 1,
"downloadSize": "84.78MB",
"downloadedSpeed": "14.11MB/sec",
"info": {
"pendingCategories": [
{
"category": "mens_clothing",
"startTime": "2021-02-21T08:14:29.602Z",
"noOfPages": 18,
"elapsed": 5668,
"totalProducts": 9000
},
{
"category": "mobiles",
"startTime": "2021-02-21T08:14:29.603Z",
"noOfPages": 13,
"elapsed": 5771,
"totalProducts": 6500
}
],
"completedCategories": [
{
"category": "laptops",
"noOfPages": 10,
"totalProducts": 4500,
"elapsed": 3878
}
]
}
}
*/
```
### events: 'response'
Emitted when successful HTTP response from the Flipkart Affiliate server.
**Example**
```javascript
// 'response' event handler
scraper.on("response", (response) => {
console.log(response);
});
```
### events: 'data'
Emitted when products returned from Flipkart affiliate API.
**Example**
```javascript
// 'data' event handler
scraper.on("data", (data) => {
console.log(data.apiData.products);
});
```
### events: 'categoryCompleted'
Emitted when all products scraped for the category.
**Example**
```javascript
// 'categoryCompleted' event handler
scraper.on("categoryCompleted", (completedCategoryInfo) => {
console.log(completedCategoryInfo);
});
```
### events: 'finished'
Emitted when scraper finished.
**Example**
```javascript
// 'finished' event handler
scraper.on("finished", (info) => {
console.log(info);
});
```
### events: 'error'
Emitted if any errors occured.
**Example**
```javascript
// Triggered if any error occured
scraper.on("error", (error) => {
console.error(error);
});
```
### events: 'retry'
Emitted if any retry occured.
**Example**
```javascript
// Triggered if any retry occured
scraper.on("retry", (retryInfo) => {
console.log(retryInfo);
});
```
### events: 'retryHalted'
Emitted if any retries failed for 10 times.
**Example**
```javascript
// Triggerd when retry failed 10 times
scraper.on("retryHalted", (retryHaltInfo) => {
console.error(retryHaltInfo);
});
```