node-firestore-import-export
Version:
Firestore data import and export
287 lines (226 loc) • 9.95 kB
Markdown
# node-firestore-import-export
Firestore data importing, exporting, and data clearing tool.
[](https://codebeat.co/projects/github-com-jloosli-node-firestore-import-export-master)
[](https://www.codacy.com/app/jloosli/node-firestore-import-export?utm_source=github.com&utm_medium=referral&utm_content=jloosli/node-firestore-import-export&utm_campaign=Badge_Grade)
[](https://david-dm.org/jloosli/node-firestore-import-export)
[](https://snyk.io/test/github/jloosli/node-firestore-import-export)
[](https://circleci.com/gh/jloosli/node-firestore-import-export)
Export a Firestore database, including collections and documents, while keeping the structure
intact.
## Table of Contents
- [Data format](#data-format)
- [Installation](#installation)
- [Retrieving credentials](#retrieving-google-cloud-account-credentials)
- [Usage](#usage)
- [Command line](#command-line)
- [Export](#export)
- [Import](#import)
- [Clear](#clear)
- [Library](#library)
- [Export](#exporting)
- [Import](#importing)
- [Clear](#clearing)
- [Contributions](#contributions)
### Data Format
Exports a json file with the following format:
```json
{
"__collections__": {
"companies": {
"docA": {
"name": "Big Co",
"employee_count": 2012,
"created": {
"__datatype__": "timestamp",
"value": {
"_seconds": 12343456,
"_nanoseconds": 7890
}
},
"location": {
"__datatype__": "geopoint",
"value": {
"_latitude": -123.456789,
"_longitude": 34.5678
}
},
"AdministratorRef": {
"__datatype__": "documentReference",
"value": "path/to/the/document"
},
"__collections__": {
"employees": ...,
"products": ...
}
},
"docB": ...,
"docC": ...
},
"administrators": {
"docA": ...,
"docB": ...
}
}
}
```
where `__collections__` holds the collections for a given document (or the root of the database).
Imports need to be from a file with the same structure (e.g. from an exported file).
__Be careful!__ This can easily overwrite or mess up your data if you import
to the wrong location.
#### Special Datatypes
Three types of data are serialized in the export:
* Timestamps
* Geopoints
* DocumentReferences
They each are serialized in the following format:
```json
{
"__datatype__": "timestamp|geopoint|documentReference",
"value": "The serialized value"
}
```
## Installation
Install using [__npm__](https://www.npmjs.com/).
```bash
npm install -g node-firestore-import-export
```
or [__yarn__](https://yarnpkg.com/en/)
```bash
yarn global add node-firestore-import-export
```
Alternatively download the source.
```bash
git clone https://github.com/jloosli/node-firestore-import-export.git
```
### Retrieving Google Cloud Account Credentials
1. Visit the [Firebase Console](https://console.firebase.google.com)
1. Select your project
1. Navigate to __Project Settings__ (at the time of writing the __gear__ icon button at the top left of the page).
1. Navigate to __Service Accounts__
1. Click _Generate New Private Key_
This downloaded json file contains the proper credentials needed for __node-firestore-import-export__ to authenticate.
### Using Firebase Firestore Emulator
If using [Firebase Emulators](https://firebase.google.com/docs/rules/emulator-setup), all commands
([Export](#export),[Import](#import), and [Clear](#clear)) will override the account credentials
setting if the `FIRESTORE_EMULATOR_HOST` environment variable is set.
## Usage
### Command Line
The path to the account credentials can either be passed with the `-a/--accountCredentials` flag, or placed in the
`GOOGLE_APPLICATION_CREDENTIALS` environment variable. For example:
```bash
export GOOGLE_APPLICATION_CREDENTIALS=path/to/my/credentials.json
firestore-export -p
```
#### Export
* `-a`, `--accountCredentials` `<path>` - path to Google Cloud account credentials JSON file.
If missing, will look at the `GOOGLE_APPLICATION_CREDENTIALS` environment variable for the path.
* `-b`, `--backupFile` `<path>`- Filename to store backup. (e.g. backups/full-backup.json).
Defaults to `firestore-export.json` if missing.
* `-n`, `--nodePath` `<path>`- Path to database node to start (e.g. collectionA/docB/collectionC).
Backs up full database if empty or missing.
* `-p`, `--prettyPrint` - JSON backups done with pretty-printing.
##### Examples
###### Export full database
```bash
firestore-export --accountCredentials path/to/credentials/file.json --backupFile /backups/myDatabase.json
```
###### Export with pretty printing
```bash
firestore-export --accountCredentials path/to/credentials/file.json --backupFile /backups/myDatabase.json --prettyPrint
```
###### Export from a specific path (and all its children/collections)
```bash
firestore-export --accountCredentials path/to/credentials/file.json --backupFile /backups/myDatabase.json --nodePath collectionA/document1/collectionCC
```
#### Import
* `-a`, `--accountCredentials` `<path>` - path to Google Cloud account credentials JSON file.
If missing, will look at the `GOOGLE_APPLICATION_CREDENTIALS` environment variable for the path.
* `-b`, `--backupFile` `<path>`- Filename with backup data. (e.g. backups/full-backup.json).
* `-n`, `--nodePath` `<path>`- Path to database node to start (e.g. collectionA/docB/collectionC).
* `-y`, `--yes` - Unattended import without confirmation (like hitting "y" from the command line).
##### Examples
###### Import full database
```bash
firestore-import --accountCredentials path/to/credentials/file.json --backupFile /backups/myDatabase.json
```
###### Import to a specific path
```bash
firestore-import --accountCredentials path/to/credentials/file.json --backupFile /backups/myDatabase.json --nodePath collectionA/document1/collectionCC
```
#### Clear
* `-a`, `--accountCredentials` `<path>` - path to Google Cloud account credentials JSON file.
If missing, will look at the `GOOGLE_APPLICATION_CREDENTIALS` environment variable for the path.
* `-n`, `--nodePath` `<path>`- Path to database node to start (e.g. collectionA/docB/collectionC).
* `-y`, `--yes` - Unattended clear without confirmation (like hitting "y" from the command line). Command will wait 5
seconds so you can `Ctrl-C` to stop.
* `-w`, `--noWait` - Combine this with the `--yes` confirmation to not wait 5 seconds
##### Example
###### Clear everything under a specific node
```bash
firestore-clear --accountCredentials path/to/credentials/file.json --yes
```
### Library
The underlying library can be used in a node or web application for importing and exporting data
in a similar fashion
#### Exporting
```typescript
import {firestoreExport} from 'node-firestore-import-export';
import * as firebase from 'firebase-admin';
firebase.initializeApp({
apiKey: "AIza....",
authDomain: "YOUR_APP.firebaseapp.com",
databaseURL: "https://YOUR_APP.firebaseio.com",
storageBucket: "YOUR_APP.appspot.com",
messagingSenderId: "123456789"
});
const collectionRef = firebase.firestore().collection('collectionA/docB/collectionC');
firestoreExport(collectionRef)
.then(data=>console.log(data));
```
#### Importing
```typescript
import {firestoreImport} from 'node-firestore-import-export';
import * as firebase from 'firebase-admin';
firebase.initializeApp({
apiKey: "AIza....",
authDomain: "YOUR_APP.firebaseapp.com",
databaseURL: "https://YOUR_APP.firebaseio.com",
storageBucket: "YOUR_APP.appspot.com",
messagingSenderId: "123456789"
});
const data = {
docA: {
name: 'bob',
__collections__: {}
},
docB: {
name: 'jill',
__collections__: {}
}
};
const collectionRef = firebase.firestore().collection('collectionA/docB/collectionC');
firestoreImport(data, collectionRef)
.then(()=>console.log('Data was imported.'));
```
#### Clearing
```typescript
import {firestoreClear} from 'node-firestore-import-export';
import * as firebase from 'firebase-admin';
firebase.initializeApp({
apiKey: "AIza....",
authDomain: "YOUR_APP.firebaseapp.com",
databaseURL: "https://YOUR_APP.firebaseio.com",
storageBucket: "YOUR_APP.appspot.com",
messagingSenderId: "123456789"
});
const collectionRef = firebase.firestore().collection('collectionA/docB/collectionC');
firestoreClear(collectionRef)
.then(()=>console.log('Everything under collectionA/docB/collectionC was removed.'));
```
## Contributions
Feel free to report bugs and make feature requests in the [Issue Tracker](https://github.com/jloosli/node-firestore-import-export/issues), fork and create pull requests!
### Inspiration
The command line was inspired heavily by SteadyEquipment's [node-firestore-backup](https://github.com/steadyequipment/node-firestore-backup)
## Support on Beerpay
Hey dude! Help me out for a couple of :beers:!
[](https://beerpay.io/jloosli/node-firestore-import-export) [](https://beerpay.io/jloosli/node-firestore-import-export?focus=wish)