# @cjpais/inference

> Trying to wrap a bunch of different inference providers models and rate limit them. As well as getting them to support typescript more natively.

Latest version **0.0.18** (published 2024-10-18) · 0 weekly downloads

## Install

```sh
npm install @cjpais/inference
pnpm add @cjpais/inference
yarn add @cjpais/inference
bun add @cjpais/inference
```

## Health

**Score 25/100 (F)** — status: abandoned.

Positive: has types; esm support; no vulnerabilities.

Warnings: low downloads; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.0.18 |
| Published | 2024-10-18 |
| First published | 2024-02-29 |
| Weekly downloads | 0 |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Dependencies | 5 |
| Unpacked size | 193.6 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Maintainers | cjpais |

## Links

- npm: https://www.npmjs.com/package/@cjpais/inference
- npm.io page: https://npm.io/package/@cjpais/inference

## Dependencies (5)

- [openai](https://npm.io/package/openai.md) ^4.68.0
- [form-data](https://npm.io/package/form-data.md) ^4.0.1
- [bottleneck](https://npm.io/package/bottleneck.md) ^2.19.5
- [@anthropic-ai/sdk](https://npm.io/package/@anthropic-ai/sdk.md) ^0.29.2
- [@mistralai/mistralai](https://npm.io/package/@mistralai/mistralai.md) ^0.1.3

## Recent versions

- 0.0.18 (latest) — 2024-10-18
- 0.0.17 — 2024-10-18
- 0.0.16 — 2024-10-18
- 0.0.15 — 2024-10-18
- 0.0.14 — 2024-10-18
- 0.0.13 — 2024-03-18
- 0.0.12 — 2024-03-14
- 0.0.11 — 2024-03-14
- 0.0.9 — 2024-03-14
- 0.0.8 — 2024-03-14
- 0.0.7 — 2024-03-14
- 0.0.6 — 2024-03-14
- 0.0.5 — 2024-03-06
- 0.0.4 — 2024-03-01
- 0.0.3 — 2024-02-29
- … 2 more at https://npm.io/package/@cjpais/inference/versions

## README

# inference

Trying to wrap a bunch of different inference providers models and rate limit them. As well as getting them to support typescript more natively.

My specific application may send many parallel requests to inference models and I need to rate limit these requests across the application per provider. This effectively solves that problem

This is a major WIP so a bunch of things are left unimplmented for the time being. However the basic functionality should be there

Supported providers:
* OpenAI (for chat, audio, image, embedding)
* Together (for chat)
* Mistral (for chat)
* Whisper.cpp (for audio)

WIP Stuff:
* consistent JSON mode
* error handling
* more rate limiting options
* more providers (llama.cpp for chat, image and embedding)
* move to config file & code gen for better typing?

## Usage

Check out `test/index.test.ts` for usage examples.

Generally speaking

1. Instantiate a provider

```typescript
const oai = new OpenAIProvider({
  apiKey: process.env.OPENAI_API_KEY!,
});
```

2. Create a rate limiter based on your own usage (this is in requests per second)

```typescript
const oaiLimiter = createRateLimiter(2);
```

3. Define what models you want to use and their alias

```typescript
const CHAT_MODELS: Record<string, ChatModel> = {
  "gpt-3.5": {
    provider: oai,
    name: "gpt-3.5",
    providerModel: "gpt-3.5-turbo-0125",
    rateLimiter: oaiLimiter,
  },
  "gpt-4": {
    provider: oai,
    name: "gpt-4",
    providerModel: "gpt-4-0125-preview",
    rateLimiter: oaiLimiter,
  }
}
```

4. Create inference with the models you want

```typescript
const inference = new Inference({chatModels: CHAT_MODELS});
```

5. Call the inference with the model you want to use

```typescript
const result = await inference.chat({model: "gpt-3.5", prompt: "Hello, world!"});
```

To install dependencies:

```bash
bun install
```

To run:

```bash
bun run index.ts
```

---
_Source: https://npm.io/package/@cjpais/inference · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._