0.18.0 • Published 5 months ago

@empiricalrun/llm v0.18.0

Weekly downloads
-
License
-
Repository
github
Last release
5 months ago

llm

Package to connect and trace LLM calls.

Usage

import { LLM } from "@empiricalrun/llm";

const llm = new LLM({
  provider: "openai",
  defaultModel: "gpt-4o",
});
const llmResponse = await llm.createChatCompletion({ ... });

Vision utilities

This package also contains utilities for vision.

Query

Ask a question against the image (e.g. to extract some info, make a decision) and get the answer.

import { query } from "@empiricalrun/llm/vision";

// With Appium
const data = await driver.saveScreenshot("dummy.png");
const instruction =
  "Extract number of ATOM tokens from the image. Return only the number.";

const text = await query(data.toString("base64"), instruction);
// Example response: "0.01"

Get bounding boxes

import { getBoundingBox } from "@empiricalrun/llm/vision/bbox";

// With Appium
const data = await driver.saveScreenshot("dummy.png");
// Give a line describing the screen and then the element that you want to find
const instruction =
  "This screenshot shows a screen to send crypto tokens. What is the bounding box for the dropdown to select the token?";

const bbox = await getBoundingBox(data.toString("base64"), instruction);
const centerToTap = bbox.center; // { x: 342, y: 450 }

// **Note**: These coordinates are relative to the image dimensions, and actions like
// tap require scaling the coordinates to Appium coordinates

Bounding box can require some prompt iterations, and you can do that with a debug flag. This flag returns a base64 image that has the bounding box drawn on top of the original image.

const bbox = await getBoundingBox(data.toString("base64"), instruction, {
  debug: true,
});
console.log(bbox.annotatedImage);
0.17.2

5 months ago

0.15.4

6 months ago

0.13.6

7 months ago

0.17.3

5 months ago

0.13.8

7 months ago

0.11.0

7 months ago

0.11.1

7 months ago

0.13.0

7 months ago

0.11.2

7 months ago

0.13.1

7 months ago

0.11.3

7 months ago

0.15.0

6 months ago

0.13.2

7 months ago

0.11.4

7 months ago

0.15.1

6 months ago

0.13.3

7 months ago

0.11.5

7 months ago

0.17.0

5 months ago

0.15.2

6 months ago

0.13.4

7 months ago

0.17.1

5 months ago

0.15.3

6 months ago

0.13.5

7 months ago

0.9.34

8 months ago

0.9.35

8 months ago

0.9.36

8 months ago

0.9.30

9 months ago

0.9.31

9 months ago

0.9.32

9 months ago

0.9.33

8 months ago

0.14.5

6 months ago

0.14.6

6 months ago

0.14.7

6 months ago

0.14.8

6 months ago

0.10.1

7 months ago

0.12.0

7 months ago

0.10.2

7 months ago

0.10.3

7 months ago

0.14.0

7 months ago

0.14.1

7 months ago

0.16.0

6 months ago

0.14.2

7 months ago

0.16.1

5 months ago

0.14.3

7 months ago

0.18.0

5 months ago

0.14.4

7 months ago

0.9.23

12 months ago

0.9.24

12 months ago

0.9.25

12 months ago

0.9.26

11 months ago

0.10.0

7 months ago

0.9.27

11 months ago

0.9.28

10 months ago

0.9.29

10 months ago

0.9.21

12 months ago

0.9.22

12 months ago

0.9.16

1 year ago

0.9.18

1 year ago

0.9.19

1 year ago

0.9.20

12 months ago

0.9.14

1 year ago

0.9.15

1 year ago

0.9.13

1 year ago

0.9.12

1 year ago

0.9.9

1 year ago

0.9.10

1 year ago

0.9.11

1 year ago

0.9.6

1 year ago

0.9.5

1 year ago

0.9.4

1 year ago

0.9.3

1 year ago

0.9.2

1 year ago

0.9.1

1 year ago

0.9.0

1 year ago

0.8.3

1 year ago

0.8.2

1 year ago

0.8.1

1 year ago

0.8.0

1 year ago

0.7.4

1 year ago

0.7.3

1 year ago

0.7.2

1 year ago

0.7.1

1 year ago

0.7.0

1 year ago

0.5.0

1 year ago

0.4.5

1 year ago

0.4.4

1 year ago

0.4.3

1 year ago

0.4.2

1 year ago

0.4.1

1 year ago

0.4.0

1 year ago

0.3.0

1 year ago

0.2.0

1 year ago