# @polymath-ai/ingest

> An ingest library for Polymath

Latest version **1.0.1** (published 2023-04-09) · MIT license · 0 weekly downloads

## Install

```sh
npm install @polymath-ai/ingest
pnpm add @polymath-ai/ingest
yarn add @polymath-ai/ingest
bun add @polymath-ai/ingest
```

## Health

**Score 30/100 (F)** — status: abandoned.

Positive: has types; esm support; no vulnerabilities; high quality score.

Warnings: low downloads.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.1 |
| Published | 2023-04-09 |
| First published | 2023-04-01 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Dependencies | 10 |
| Unpacked size | 108.1 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 6 |
| Author | The Polymath Crew |
| Maintainers | komorama, dimitriglazkov, dalmaer |
| Keywords | polymath, genai, llm |

## Links

- npm: https://www.npmjs.com/package/@polymath-ai/ingest
- Repository: https://github.com/polymath-ai/cli
- Homepage: https://github.com/polymath-ai//polymath-ai#readme
- Issues: https://github.com/polymath-ai/polymath-ai/issues
- npm.io page: https://npm.io/package/@polymath-ai/ingest

## Dependencies (10)

- [i](https://npm.io/package/i.md) ^0.3.7
- [esm](https://npm.io/package/esm.md) ^3.2.25
- [npm](https://npm.io/package/npm.md) ^9.6.2
- [chalk](https://npm.io/package/chalk.md) ^5.2.0
- [inquirer](https://npm.io/package/inquirer.md) ^9.1.4
- [commander](https://npm.io/package/commander.md) ^10.0.0
- [rss-parser](https://npm.io/package/rss-parser.md) ^3.12.0
- [html-to-text](https://npm.io/package/html-to-text.md) ^9.0.5
- [@polymath-ai/types](https://npm.io/package/@polymath-ai/types.md) *
- [@polymath-ai/client](https://npm.io/package/@polymath-ai/client.md) ^2.0.3

## Alternatives

- [random-seedable](https://npm.io/package/random-seedable.md) — 27.9K weekly downloads
- [n2words](https://npm.io/package/n2words.md) — 22.2K weekly downloads
- [@stdlib/math-base-special-factorialln](https://npm.io/package/@stdlib/math-base-special-factorialln.md) — 5.7K weekly downloads
- [@stdlib/math-base-special-abs2](https://npm.io/package/@stdlib/math-base-special-abs2.md) — 1.7K weekly downloads
- [commons-math-interpolation](https://npm.io/package/commons-math-interpolation.md) — 1.4K weekly downloads

## Recent versions

- 1.0.1 (latest) — 2023-04-09
- 1.0.0 — 2023-04-01

## README

# Polymath Ingest

This is a libraries that can be used to ingest data from a source and convert it into a format that can be used by the Polymath system.

## Standalone ingestion tool

See: @polymath/cli

## API

### Extend `Ingester` for use in the CLI

You are able to extend the `Ingester` class to create your own importer.

1. Create a new class that extends `Ingester`
2. Implement the `getStringsFromSource` method
   a. getStringsFromSource should return an AsyncGenerator of partially filled out bits that is the *full string* of data from the source.
   b. The `text` property of the bit should be the full string of data from the source. The `info` property of the bit should be an object that contains any additional information about the bit. The `info` property is optional.

```
class MYRSS extends Ingester {

  constructor() {
    super();
  }

  async *getStringsFromSource(source: string): AsyncGenerator<Bit> {
    const feed = await (new RSSParser).parseURL(source);
    console.log(feed.title); // feed will have a `foo` property, type as a string

    for (const item of feed.items) {
      yield {
        text: item.content || "",
        info: {
          url: item.link || "",
          title: item.title || "",
        }
      };
    }
  }

}
```

E.g:
`> polymath ingest rss https://paul.kinlan.me/index.xml`

### Embed the `Ingest` process

It is possible to use the `Ingest` process directly in your own code, for example if you want to run a custom export as the embeddings are generated. The process will load the correct plugin and start processing it.

```
import { Ingest } from "@polymath/ingest";

const ingest = new Ingest({ /* options */ });

for await(const bit of ingest.run({ args, options, command })) {
  console.log(`${bit.url}: Embedding: ${bit.embedding}`);)  
}

```

---
_Source: https://npm.io/package/@polymath-ai/ingest · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._