@blocklet/crawler/README.md

blocklet crawler lib

# @blocklet/crawler

**@blocklet/crawler** is a package crawler for Blocklet.

## Package Structure

The package is composed of both frontend and backend components. The backend code can be found in the `middlewares` folder.

## Development

### Install In Blocklet

```
# You can use npm / yarn
pnpm add @blocklet/crawler
```

### Install Dependencies

To install the required dependencies, run the following command:

```
pnpm i
```

### Build Packages

To build the packages, execute the following command:

```
pnpm build
```

### Build, Watch, and Run Development Server

For building, watching changes, and running the development server, use the following command:

```
pnpm run dev
```

## Backend Example

```javascript
const { initSEOMiddleware, initCronCrawlBlocklet } = require('@blocklet/crawler/middlewares');

// init cron job to crawl blocklet that generate SEO pages
initCronCrawlBlocklet({
  time: '0 0 */12 * * *',
});

const router = express.Router();
// init SEO middleware, when request come, we will return the SEO static page
router.use(initSEOMiddleware());
```

## License

This package is licensed under the MIT license.