clippd-ingestion v1.0.0

Clippd Data Ingestion Scripts

Quick start:

1. npm install
2. npm start

Each lambda has a HTTP trigger configured at infrastructure/serverless/serverless.yml:

GET | https://localhost:3000/development/api/golf-data-labs
GET | https://localhost:3000/development/api/golf-stats-lab
GET | https://localhost:3000/development/api/trackman
GET | https://localhost:3000/development/api/arccos
GET | https://localhost:3000/development/api/world-handicap-system

Each invocation will run relative script in src/server/handler.

Each script uses getCredentials function in src/server/service/credientals to get the credentials stored in DynamoDB. The table is simply a PK that's the name for the service and the rest is usually api_key (if applicable), password, url (the base URL), and username

If the response is JSON, they are automatically parsed into POJOs. If the response is HTML, we use cheerio to parse it in a jQuery like format and traverse the document to get the data.