rag-aiquest v1.4.1
π AIQuest
Aiquest is an npm package that streamlines the process of parsing websites, splitting content into manageable chunks, embedding these chunks into machine-friendly vectors, and subsequently storing and retrieving these embeddings from AWS. This documentation outlines its design and possibilities.
π Table of Contents
- π§ Design
- π Installation
- π οΈ Usage
- πΈοΈ Parsing
- βοΈ Chunking
- 𧬠Embedding
- βοΈ Storing on AWS
- π Retrieval
- π Examples
- π Future Enhancements
- π€ Contribution
- π Bug Reporting
π§ Design
rag-aiquest integrates several utilities under one package:
UnifiedParser: For parsing content from URLs,PDF or Text File.
ChunkUtility: To split the parsed content into chunks.
EmbeddingUtility: Utilizes the OpenAI API to embed the chunks into vectors.
VectorStoreAWS: A utility for AWS operations related to embedding storage.
Retrival: Provides functionality to retrieve knowledge and run QnA.
π Installation
npm install rag-aiquest
π οΈ Usage
πΈοΈ Parsing
Use the UnifiedParser to parse content from a URL.
const parser = new UnifiedParser();
const parsedValue = await parser.parse('YOUR_URL_HERE');
βοΈ Chunking
To split the parsed content into chunks:
const chunks = ChunkUtility.splitIntoChunks(parsedValue, chunkSize, overlapSize);
𧬠Embedding
Embed chunks using OpenAI API.
const embedding = new EmbeddingUtility('YOUR_OPENAI_API_KEY');
const embedded = await embedding.createEmbedding(chunks);
βοΈ Storing on AWS
To upload the embedded model to AWS:
const aws = new VectorStoreAWS(AWS_ACCESS_KEY_ID, AWS_ACCESS_SECRET, AWS_BUCKET_NAME);
await aws.uploadEmbededModeltoAWS(embedded, 'YOUR_FILE_NAME');
π Retrieval
To retrieve and query the knowledge:
const knowledge = await aws.getKnowledgeData('YOUR_FILE_NAME');
const retrive = new Retrival('YOUR_OPENAI_API_KEY');
const search = await retrive.QnARetrival(knowledge, 'YOUR_QUERY');
console.log(search.choices[0].message);
π Examples
As given in the provided code, you can easily integrate the utilities to parse, chunk, embed, store, and retrieve knowledge.
π Future Enhancements
Compression: Improve storage efficiency by compressing embedded vectors.
Batch Processing: Enhance the library to handle batch processing of URLs.
Support for More Embeddings: Plan to add support for other embedding APIs.
π€ Contribution
If you wish to contribute to rag-aiquest, please refer to the CONTRIBUTING.md file.
π Bug Reporting
Feel free to open an issue on GitHub if you find any bug.
β Feature Request
- Feel free to Open an issue on GitHub to request any additional features you might need for your use case.
- Connect with me on LinkedIn. I'd love β€οΈοΈ to hear where you are using this library.
π Release Notes
Check here for release notes.
π Prerequisites
- Active OpenAI API Key
- AWS BUCKET SECRET_KEY,Bucket Name, AUTH KEY