# headlinebot > This is a tool that can be used to scrape news website content and provide alternate means for reading it (currently Slack and RSS). Latest version **0.0.4** (published 2023-05-06) · CC0-1.0 license · 0 weekly downloads ## Install ```sh npm install headlinebot pnpm add headlinebot yarn add headlinebot bun add headlinebot ``` Provides the command `headlinebot`. ## Health **Score 15/100 (F)** — status: abandoned. Positive: no vulnerabilities. Warnings: low downloads; no types; no esm support; pre 1.0. Negative: abandoned; low maintenance score. ## Facts | | | |---|---| | Version | 0.0.4 | | Published | 2023-05-06 | | First published | 2023-05-06 | | Weekly downloads | 0 | | License | CC0-1.0 | | TypeScript types | none | | Module format | CommonJS | | Node | >= 16 | | Dependencies | 15 | | Unpacked size | 55.4 KB | | Known vulnerabilities | 0 (+1 in 1 direct dependencies) | | Install scripts | no | | Author | Matt Hinz | | Maintainers | matthinz | ## Links - npm: https://www.npmjs.com/package/headlinebot - npm.io page: https://npm.io/package/headlinebot ## Dependencies (15) - [zod](https://npm.io/package/zod.md) ^3.21.4 - [feed](https://npm.io/package/feed.md) ^4.2.2 - [dotenv](https://npm.io/package/dotenv.md) ^16.0.3 - [openai](https://npm.io/package/openai.md) ^3.2.1 - [sqlite](https://npm.io/package/sqlite.md) ^4.1.2 - [aws-sdk](https://npm.io/package/aws-sdk.md) ^2.1336.0 - [cheerio](https://npm.io/package/cheerio.md) ^1.0.0-rc.12 - [sqlite3](https://npm.io/package/sqlite3.md) ^5.1.6 - [date-fns](https://npm.io/package/date-fns.md) ^2.29.3 - [prettier](https://npm.io/package/prettier.md) ^2.8.4 - [puppeteer](https://npm.io/package/puppeteer.md) ^19.7.3 - [sanitize-html](https://npm.io/package/sanitize-html.md) ^2.10.0 - [@slack/web-api](https://npm.io/package/@slack/web-api.md) ^6.8.1 - [puppeteer-extra](https://npm.io/package/puppeteer-extra.md) ^3.3.6 - [puppeteer-extra-plugin-stealth](https://npm.io/package/puppeteer-extra-plugin-stealth.md) ^2.11.2 ## Recent versions - 0.0.4 (latest) — 2023-05-06 - 0.0.3 — 2023-05-06 - 0.0.2 — 2023-05-06 - 0.0.1 — 2023-05-06 ## README # headlinebot This is a tool that can be used to scrape news website content and provide alternate means for reading it (currently Slack and RSS). To work around common techniques used to block automated scraping of website content, it drives a real instance of Google Chrome (using puppeteer). That said, scraping is inherently fragile. Expect this thing to break. Regularly. ## Requirements - Node.js (see `.nvmrc` for exact version) - Yarn ## Getting started You'll need to set a number of environment variables for this tool to work. Once you've done that, you can execute it like so: ```shell yarn && yarn start ``` ## Environment variables | Variable | Example | Description | | ------------------ | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `ALLOWED_HOSTS` | `"example.org,account.example.org"` | During scraping, requests made to any hosts _not_ in this list (for example, to load third-party Javascript) will be blocked. **It may take some trial and error to get this list right.** | | `CHROME_PATH` | `"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"` | Path to the Google Chrome executable. | | `HEADLINES_URL` | `"https://example.org/local-news"` | URL to scrape news headlines from. | | `WEBSITE_PASSWORD` | `"trustno1"` | Password used to log into the news website when a paywall is hit. | | `WEBSITE_USERNAME` | `"my-email@example.org"` | Username used to log into the news website when a paywall is hit. | ### Summarization Articles can be automatically summarized using ChatGPT. | Variable | Example | Description | | ---------------- | ------------------- | ------------------------------------------------------------------- | | `OPENAI_API_KEY` | `"sk-sldkjflsdkjf"` | Key used to access the OpenAI API (used for article summarization). | ### Slack integration When configured, new articles can be periodically posted to a Slack channel. | Variable | Example | Description | | --------------- | ------------- | ------------------------------------------------------------------------------ | | `SLACK_CHANNEL` | `"#the-news"` | When integrated with Slack, the channel that new articles should be posted in. | | `SLACK_TOKEN` | `"xoxb-foo"` | Bot token used to access the Slack API to post. | ### RSS feed generation Each run can generate an RSS feed .xml file and upload it to S3 (or a compatible service). | Variable | Example | Description | | -------------------------- | --------------------------------- | ------------------------------------------------------- | | `S3_BUCKET` | `"my-bucket"` | S3 bucket to upload RSS XML to. | | `S3_REGION` | `"us-east-1"` | S3 region to use. | | `S3_ENDPOINT` | `"https://example.org/my-bucket"` | Alternate endpoint (allows using an S3-compatible API). | | `AWS_ACCESS_KEY_ID` | | (AWS credential used for RSS upload.) | | `AWS_SECRET_ACCESS_KEY_ID` | | (AWS credential used for RSS upload.) | --- _Source: https://npm.io/package/headlinebot · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._