mashr
Version:
Simple data pipeline framework for GCP's BigQuery
181 lines (134 loc) • 6.17 kB
Markdown
## Overview
Mashr is an easy to use data pipeline framework that orchestrates moving data
from external sources into BigQuery. It’s meant for small software applications
using Google Cloud Platform (GCP) who have their data spread out in a variety
of external sources that they want to be able to aggregate and perform
analysis on.
## Mashr Architecture
[TODO: Diagram TBD]
## The Team
**[Jacob Coker-Dukowitz](https://)** *Software Engineer* San
Francisco, CA
**[Linus Phan](https://)** *Software Engineer* Los Angeles, CA
**[Mat Sachs](https://matsachs.com)** *Software Engineer* Portland, OR
## Getting Started
### Prerequisites
* GCP (Google Cloud Platform) account
* GCP project, service account email, and json keyfile
* [GCP Cloud SDK](https://cloud.google.com/appengine/docs/standard/go/download)
* Node.js >= 8
* NPM
Mashr requires that users have a project with a service account on GCP and
set up the Cloud SDK CLI on their local machine. If you have not already done
so, please [Download the Cloud
SDK](https://cloud.google.com/appengine/docs/standard/go/download) and use the
["console" to create a
project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
and a [service
account](https://cloud.google.com/iam/docs/creating-managing-service-accounts)
with a role of "owner". Mashr will use the project id, service account email,
and service account json keyfile to to interact with GCP services.
### Install Mashr
```
npm install -g mashr
```
-------------------------------------------------------------------------------
## Documentation
### The Mashr Process
[TODO: paragraph for the init process]
Starting with a mashr_config.yml file and the terminal command `mashr deploy`,
Mashr creates a Google Compute Engine (GCE) instance, Google Cloud Storage
(GCS) staging and archive buckets, BigQuery dataset and table, and a Google
Cloud Function (GCF) to automate the porting of data between the GCS buckets
and BigQuery table.
The GCE instance hosts a docker container with the
[Embulk](https://www.embulk.org/docs/) data loader running on it. A cron job
runs an embulk job every 10 minutes. The embulk job pulls data from an external
source, like Salesforce or a postgres database, and puts it into a GCS staging
bucket. When data is loaded to the GCS staging bucket, the GCF is triggered.
The GCF moves the data to a coldline storage bucket for archiving and failover,
and then loads the data into the appropriate BigQuery table.
### GCP Project and Service Account Setup
* Make sure you have a Google Cloud Platform (GCP) account
* Create a new project in your Google Cloud account
* After creating a new project, you will need to enable the Cloud Functions
API from the web console.
- Go to the main menu and choose "APIs & Services"
- Click the "+ Enable APIs and Services" button at the top of the page
- Search for "Cloud Function" (no 's')
- Click "Cloud Functions API"
- Click "enable" to enable the API
* Download and install the [gcloud CLI
SDK](https://cloud.google.com/sdk/docs/quickstarts) from Google.
### Commands
```
mashr init [--template <template_name>]
```
Initializes your current working directory with a template mashr_config.yml
file necessary for running `mashr deploy`. Optionally include the `--template`
flag and name of the template. Template names include `http`, `psql`,
`random`.
-------------------------------------------------------------------------------
```
mashr deploy
```
Deploys the integration: adds it to the list of mashr integrations and creates
related GCP resources including staging and archive GCS buckets, a cloud
function and a GCE instance.
Creates a 'function' folder that stores the code the cloud function uses in
this integration. You can edit and redeploy the cloud function with `gcloud`.
A `mashr_config.yml` file in the user's working directory is required. Run
`mashr init` to see a template file you can use.
-------------------------------------------------------------------------------
```
mashr list
```
Lists all integrations that the user has deployed.
-------------------------------------------------------------------------------
```
mashr destroy <integration name>
```
Destroys the integration: removes it from the list of mashr integrations and
destroys related GCP resources including the staging and archive GCS buckets,
the cloud function and the GCE instance.
-------------------------------------------------------------------------------
```
mashr help
```
Documentation of commands.
-------------------------------------------------------------------------------
### Accessing the BigQuery Table
[ TODO ]
### Accessing the Compute Engine Instance and Docker Container
[ TODO ]
-------------------------------------------------------------------------------
## Helpful Tips
### Location considerations for your GCP Services
Consider colocating as many as your services as possible. For example, it's
required that your GCS (Google Cloud Storage) and GBQ (Google Big Query) be
located in the same regions. See the [Locations Considerations
document](./docs/gcp_locations_considerations.md) for more information
-------------------------------------------------------------------------------
### Details / Notes / Delete
* happens on the users machine:
- create the install_gems.sh
- create the embulk_config file
* nodejs create a compute instance
- with a startup script in the nodejs create compute instance, script.
- cp the install_gems.sh
- cp the keyfile
- cp the embulk_config file
- cp Dockerfile from and build the image
OR
pull image from public directory on dockerhub
- run dockerfile
* when Docker container first runs:
- copies the install_gems, keyfile, and embulk_config.yml into container
- installs the embulk gems
- sets volume for persisting data like a diff file, etc.
- starts cron job
- embulk run embulk_config.yml
* need another .sh file for the cron job to run
- based on what user gives us from mashr_config
Dockerfile > build > image > start a container from the image