# newsagent > Monitor data sources, get alerted when they change. Latest version **1.5.0** (published 2021-07-23) · Apache-2.0 license · 0 weekly downloads ## Install ```sh npm install newsagent pnpm add newsagent yarn add newsagent bun add newsagent ``` Provides the command `newsagent`. ## Health **Score 20/100 (F)** — status: abandoned. Positive: esm support; no vulnerabilities. Warnings: low downloads; no types. Negative: abandoned; low maintenance score. ## Facts | | | |---|---| | Version | 1.5.0 | | Published | 2021-07-23 | | First published | 2020-11-07 | | Weekly downloads | 0 | | License | Apache-2.0 | | TypeScript types | none | | Module format | ESM | | Dependencies | 15 | | Unpacked size | 31.9 KB | | Known vulnerabilities | 0 (+37 in 4 direct dependencies) | | Install scripts | no | | GitHub stars | 13 | | Author | Max Harlow | | Maintainers | maxharlow | ## Links - npm: https://www.npmjs.com/package/newsagent - Repository: https://github.com/maxharlow/newsagent - Homepage: https://github.com/maxharlow/newsagent#readme - Issues: https://github.com/maxharlow/newsagent/issues - npm.io page: https://npm.io/package/newsagent ## Dependencies (15) - [zod](https://npm.io/package/zod.md) 3.5.x - [bree](https://npm.io/package/bree.md) 6.2.x - [open](https://npm.io/package/open.md) 8.2.x - [yaml](https://npm.io/package/yaml.md) 1.10.x - [axios](https://npm.io/package/axios.md) 0.21.x - [chalk](https://npm.io/package/chalk.md) 4.1.x - [yargs](https://npm.io/package/yargs.md) 17.0.x - [cheerio](https://npm.io/package/cheerio.md) 1.0.0-rc.10 - [fs-extra](https://npm.io/package/fs-extra.md) 10.0.x - [jmespath](https://npm.io/package/jmespath.md) 0.15.x - [papaparse](https://npm.io/package/papaparse.md) 5.3.x - [nodemailer](https://npm.io/package/nodemailer.md) 6.6.x - [playwright](https://npm.io/package/playwright.md) 1.13.x - [object-hash](https://npm.io/package/object-hash.md) 2.2.x - [node-notifier](https://npm.io/package/node-notifier.md) 10.0.x ## Recent versions - 1.5.0 (latest) — 2021-07-23 - 1.4.0 — 2021-04-07 - 1.3.0 — 2020-11-13 - 1.2.0 — 2020-11-12 - 1.1.0 — 2020-11-08 - 1.0.0 — 2020-11-07 ## README Newsagent ========= Monitor data sources, get alerted when they change. Installing ---------- $ npm install -g newsagent Alternatively, don't install it and just prepend the command with `npx`. Usage ----- $ newsagent Where `` are names of one or more watchfiles. ### Watchfiles A watchfile defines how a data source should be monitored, what transforms to apply to those changes, and when to fire alerts. They are written in Yaml. An example: name: AP Politics Twitter source: method: fetch-html url: 'https://twitter.com/search?f=live&q=from:AP_Politics' browser: chromium selection: 'article div[lang="en"]' schedule: 10s monitor: additions-only processes: - method: match-transform match: '/BREAKING: ([A-Za-z]+) ([A-Za-z ]+) wins (.+) to (.+) in (.+)\. #APracecall at (.+)\./' transform: '\\4: \\5 -- \\3 of \\2 (\\1) at \\6' - method: find-and-replace find: 'U.S.' replace: 'US' alerts: - method: log A watchfile can also have a top-level array with multiple watches. The sections:
#### `name` A unique name for this watchfile.
#### `source` Where the data is going to come from. The `method` field should be one of the following source methods. The other source fields are specified by the method. ##### ► `fetch-html` Fetch a HTML page, and extract text from it. * `url` The URL to fetch * `browser` (optional) Specify either `chromium`, `webkit`, or `firefox` to fetch the page using that browser, if not specified simply fetches the HTML * `selection` A CSS selector for one or more text elements, or the more advanced [Playwright format](https://playwright.dev/#path=docs/selectors.md) if a browser is specified; to get the content of an attribute use a pipe (`|`) followed by the attribute name * `subselection` (optional) Fields within that selection, each with their own selector ##### ► `fetch-json` Fetch a Json file, and extract data from it. * `url` The URL to fetch * `selection` A [JmesPath selector](https://jmespath.org/tutorial.html) ##### ► `fetch-csv` Fetch a CSV file. * `url` The URL to fetch
#### `schedule` How often to check for changes. Supports [various human-readable formats](https://github.com/breejs/bree#job-interval-and-timeout-values).
#### `monitor` What kinds of changes do you want to be alerted on? Set to either `additions-and-removals`, `additions-only`, or `removals-only`.
#### `processes` An optional section. A list of processes which the changes are pushed through, one after another. They can modify or filter out what goes through to fire alerts. ##### ► `select` Select specific fields to retain from the data. Expects data input and outputs data. * `fields` Specify fields, each with their own [JmesPath selector](https://jmespath.org/tutorial.html) ##### ► `filter` Filters out anything not matching a regular expression. Expects text/text-array input unless `field` is specified, and outputs the same. * `match` A regular expression * `field` (optional) The field within the input data to manipulate. ##### ► `trim` Remove whitespace at the start and end. Expects text/text-array input unless `field` is specified, and outputs the same. * `field` (optional) The field within the input data to manipulate. ##### ► `split` Splits out new fields from text. Expects text/text-array input unless `field` is specified, and outputs the same. * `fields` Specify fields to output, each with their own regular expression. * `field` (optional) The field within the input data to extract data from ##### ► `combine` Combine different fields together. Expects data input and outputs text unless `field` is specified. * `combination` The combined text, with fields included using double braces, eg. `{{field-name}}` * `field` (optional) The name of a new output field. ##### ► `match-transform` Select parts of text using a regular expression and transform them. Expects text/text-array input and outputs text, unless `field` is specified. * `match` A regular expression * `transform` Text to output, using groups from the match, eg. `\\1` * `field` (optional) The field within the input data to manipulate. ##### ► `find-and-replace` Looks for text and replaces it with something else. Expects text/text-array input unless `field` is specified, and outputs the same. * `find` The text to look for. * `replace` What to replace it with. * `field` (optional) The field within the input data to manipulate.
#### `alerts` A list of alerts to fire for all changes that have made it this far. ##### ► `log` Just prints to the console. ##### ► `email` Sends an email. * `to` The email address to send to. * `smtpHost` The host name of the SMTP server (for a Gmail account, use `smtp.gmail.com`) * `smtpUsername` Your SMTP username (for a Gmail account, use your email address) * `smtpPassword` Your SMTP password (for a Gmail account, use [an app password](https://myaccount.google.com/apppasswords)) ##### ► `macos-notification` Shows a MacOS notification. Expects text input unless `bodyField` is specified. * `bodyField` (optional) A specific field to use for the notification text. * `url` (optional) A field containing a URL, which will be opened if the notification is clicked. ##### ► `webhook` Call a webhook, such as a [Slack Webhook Workflow](https://slack.com/intl/en-gb/help/articles/360041352714-Create-more-advanced-workflows-using-webhooks). * `url` A webhook URL to call * `bodyField` (optional) A specific field to use for the message text.
Inspiration ----------- * [Some unnamed tool built by the LA Times](https://www.youtube.com/watch?v=iP-On8PzEy8) * [Workbench](https://workbenchdata.com/) * [Pipedream](https://pipedream.com/) * [Morph](https://morph.io/) * [Huginn](https://github.com/huginn/huginn) * [Easymorph](https://easymorph.com/) * [Visualping](https://visualping.io/), nee Change Detection * [Stakeout](https://github.com/veltman/stakeout) * [Datawire](https://github.com/arc64/datawi.re) * [Datastringer](https://github.com/BBC-News-Labs/datastringer) * [Yahoo Pipes](https://en.wikipedia.org/wiki/Yahoo!_Pipes) --- _Source: https://npm.io/package/newsagent · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._