@tricoteuses/senat/README.md

Handle French Sénat's open data

# Tricoteuses-Senat

## _Handle French Sénat's open data_

## Installation

```bash
git clone https://git.en-root.org/tricoteuses/tricoteuses-senat
cd tricoteuses-senat/
```

Create a `.env` file to set PostgreSQL database informations and other configuration variables (you can use `example.env` as a template). Then

```bash
npm install
```

### Database creation

#### Using Docker

```bash
docker run --name local-postgres -d -p 5432:5432 -e POSTGRES_PASSWORD=$YOUR_CUSTOM_DB_PASSWORD postgres
# Default Postgres user is postgres
# But scripts require an "opendata" role
docker exec -it local-postgres psql -U postgres -c "CREATE ROLE opendata;"
```

## Usage

### Retrieval, cleaning & import of open data from Sénat

```bash
mkdir ../senat-data/
npx babel-node --extensions ".ts" --max-old-space-size=4096 -- src/scripts/retrieve_open_data.ts --fetch --unzip --repair-encoding ../senat-data/
npx babel-node --extensions ".ts" -- src/scripts/retrieve_open_data.ts --import ../senat-data/
npx babel-node --extensions ".ts" -- src/scripts/fix_db.ts --updateSchema
npx babel-node --extensions ".ts" -- src/scripts/convert_data.ts ../senat-data/ [--categories DosLeg]
```

## Retrieval of sénateurs' pictures from Sénat's website

```bash
npx babel-node --extensions ".ts" -- src/scripts/retrieve_senateurs_photos.ts --fetch ../senat-data/
```

### Retrieval & cleaning of bills from Sénat HTML pages

```bash
npx babel-node --extensions ".ts" -- src/scripts/retrieve_textes.ts ../senat-data/
```

### Generation of raw types from SQL schema.

To be run after database has been fixed (`src/scripts/fix_db.ts --updateSchema`).
```bash
npx babel-node --extensions ".ts" -- src/scripts/retrieve_open_data.ts --schema ../senat-data/
```