3.0.6 • Published 10 months ago

@openfn/language-bigquery v3.0.6

Weekly downloads
-
License
LGPLv3
Repository
github
Last release
10 months ago

Language BigQuery

Language Pack for building expressions and operations to make HTTP calls.

Documentation

state.json

The configuration key must be a valid GCP credential as JSON.

{
  "configuration": {
    "type": "service_account",
    "project_id": "some-project",
    "private_key_id": "670b9e3c8c366e83aa569dd57cbfc5c575b72e42",
    "private_key": "-----BEGIN PRIVATE KEY-----\nblah\nmoreblah=\n-----END PRIVATE KEY-----\n",
    "client_email": "bigquery@some-project.iam.gserviceaccount.com",
    "client_id": "someid",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/bigquery%40some-project.iam.gserviceaccount.com"
  },
  "data": { "a": 1 }
}

operations (expression.js)

load(...)

Load data to a table in bigquery from a CSV. See full options here.

load(
  '/home/taylor/Desktop/type-C_r-ALL_ps-201012_freq-M_px-HS_pub-20200731_fmt-csv_ex-20200818.csv',
  'awesome-solutions-project', // project
  'test01', // dataset
  'mytable', // table
  {
    schema:
      'classification:STRING,year:STRING,period:STRING,period_desc:STRING,aggregate_level:STRING,is_leaf_code:STRING,trade_flow_code:STRING,trade_flow:STRING,reporter_code:STRING,reporter:STRING,reporter_iso:STRING,partner_code:STRING,partner:STRING,partner_iso:STRING,commodity_code:STRING,commodity:STRING,qty_unit_code:STRING,qty_unit:STRING,qty:INTEGER,netweight_kg:INTEGER,trade_value:INTEGER,flag:STRING',
    schemaUpdateOptions: ['ALLOW_FIELD_ADDITION'],
    // createDisposition: 'CREATE_IF_NEEDED',
    writeDisposition: 'WRITE_APPEND',
    skipLeadingRows: 1,
  } // loadOptions
);

Development

Clone the adaptors monorepo. Follow the Getting Started guide inside to get set up.

Run tests using pnpm run test or pnpm run test:watch

Build the project using pnpm build.

To just build the docs run pnpm build docs

3.0.6

10 months ago

3.0.5

10 months ago

3.0.4

10 months ago

3.0.3

11 months ago

3.0.2

11 months ago

2.1.2

1 year ago

2.1.1

1 year ago

2.0.10

1 year ago

2.1.0

1 year ago

3.0.1

1 year ago

3.0.0

1 year ago

2.0.9

1 year ago

2.0.7

1 year ago

2.0.8

1 year ago

2.0.5

1 year ago

2.0.6

1 year ago

1.2.3

2 years ago

2.0.3

2 years ago

2.0.2

2 years ago

2.0.4

2 years ago

2.0.1

2 years ago

2.0.0

2 years ago

1.2.2

2 years ago

1.2.1

2 years ago

1.1.5

2 years ago

1.1.4

2 years ago

1.1.3

2 years ago

1.1.2

3 years ago

1.1.1

3 years ago

1.1.0

3 years ago