Llm-pack NPM | npm.io

██╗     ██╗     ███╗   ███╗            ██████╗  █████╗  ██████╗██╗  ██╗
██║     ██║     ████╗ ████║            ██╔══██╗██╔══██╗██╔════╝██║ ██╔╝
██║     ██║     ██╔████╔██║            ██████╔╝███████║██║     █████╔╝
██║     ██║     ██║╚██╔╝██║            ██╔═══╝ ██╔══██║██║     ██╔═██╗
███████╗███████╗██║ ╚═╝ ██║            ██║     ██║  ██║╚██████╗██║  ██╗
╚══════╝╚══════╝╚═╝     ╚═╝            ╚═╝     ╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝
          Project Content Aggregator for Large Language Models

LLM-Pack is a command-line tool designed to aggregate and optimize project content for use with Large Language Models (LLMs). It recursively scans your project directory, respects standard ignore patterns, and generates a single structured file that's optimized for LLM processing.

Features

📁 Recursive directory scanning with intelligent filtering
🚫 Respects .gitignore, .dockerignore, and other standard ignore files
📝 Markdown and JSON output formats with syntax highlighting
🎨 Smart content compaction for large files
📑 Automatic table of contents generation
🔧 Configurable file size limits and file count constraints
🎯 Custom ignore patterns support
🚀 Built-in common directory exclusions (node_modules, dist, etc.)
⚡ Streaming support for large file processing
📊 Progress tracking with real-time updates
🔄 Parallel processing with worker threads
💾 Memory-optimized processing for large projects
🧠 Intelligent code compaction preserving important sections
🔍 Advanced pattern detection for various file types
🌈 Rich syntax highlighting with multiple themes

Installation

# Global installation
npm install -g llm-pack

# Or using npx directly
npx llm-pack [options]

Usage

Basic Usage

Run in your project directory:

llm-pack

This will: 1. Scan your project directory 2. Process files according to configured limits and patterns 3. Generate a formatted output file with all relevant content 4. Include a table of contents and syntax-highlighted code blocks

Command Line Options

llm-pack [options]

Basic Options:
  -f, --format         Output format (markdown or json)     [default: "markdown"]
  -o, --output        Output file path                     [default: "llm-pack-output.md" or "llm-pack-output.json"]
  -i, --ignore        Custom ignore files                  [array]
  -e, --extensions    Additional file extensions to include [array]
  -c, --config        Path to configuration file           [string]
  -h, --help         Show help                            [boolean]
  -v, --version      Show version number                  [boolean]

Content Processing:
  --max-files         Maximum number of files to include   [number]
  --max-file-size     Maximum file size in kilobytes      [number]
  --use-compactor     Enable code compaction              [boolean] [default: false]
  --compact-lines     Maximum lines before compaction      [number] [default: 100]
  --context-lines     Context lines to preserve           [number] [default: 3]
  --importance        Importance threshold (0-1)          [number] [default: 0.6]

Performance Options:
  --enable-workers              Enable parallel processing     [boolean] [default: false]
  --max-workers                Maximum worker threads          [number] [default: 4]
  --enable-memory-monitoring   Monitor memory usage            [boolean] [default: false]
  --chunk-size                Streaming chunk size (bytes)     [number] [default: 65536]
  --progress                  Show detailed progress           [boolean] [default: false]

Configuration

Create llm-pack.config.json in your project root:

{
  "output": {
    "format": "markdown",
    "path": "./llm-pack-output",
    "createDirectory": true
  },
  "limits": {
    "maxFileSize": 1024,
    "maxFiles": null,
    "maxTotalSize": null
  },
  "processing": {
    "batchSize": 100,
    "streamingThreshold": 512,
    "compactLargeFiles": true,
    "enableWorkers": true,
    "maxWorkers": 4
  },
  "extensions": {
    "include": [".js", ".jsx", ".ts", ".tsx", ".md", ".json"],
    "exclude": [".min.js", ".map"]
  },
  "ignore": {
    "customPatterns": [],
    "extendGitignore": true,
    "defaultIgnores": true
  },
  "normalization": {
    "normalizeLineEndings": true,
    "normalizeWhitespace": true,
    "removeHtmlTags": false,
    "preserveCodeBlocks": true
  }
}

Default Behavior

LLM-Pack comes with sensible defaults:

Excluded Directories

node_modules
dist
coverage
.git
build
temp
cache

Supported File Extensions

JavaScript: .js, .jsx
TypeScript: .ts, .tsx
Documentation: .md
Data: .json, .yaml, .yml
Web: .html, .css
Other: .py, .java, .c, .cpp, .rb, .go, .php, .sh

Ignore Files Support

.gitignore
.npmignore
.dockerignore
.eslintignore
.prettierignore
.hgignore
.svnignore

Advanced Features

Intelligent Code Compaction

When using --use-compactor, LLM-Pack intelligently preserves:

Class and function definitions
Important method implementations
Configuration objects
API operations
Error handling
State management
Documentation comments

Memory Management

Streaming support for large files
Automatic chunk size optimization
Memory usage monitoring
Garbage collection optimization
Configurable memory limits

Worker Thread Processing

Parallel file processing
Automatic work distribution
Configurable worker pool
Memory-aware task scheduling

Content Normalization

Line ending standardization
Whitespace optimization
HTML tag handling
Code block preservation
Consistent formatting

Performance Features

Streaming support for large file processing
Real-time progress tracking
Memory-efficient file handling
Asynchronous file processing
Smart content buffering
Parallel processing capabilities
Optimized memory management

LLM Use Cases

1. Code Understanding & Analysis

Generate a concise overview for LLMs to understand your codebase:

llm-pack \
  --use-compactor \
  --compact-lines 100 \
  --importance 0.8 \
  --normalize-whitespace \
  --output ./llm/codebase-overview.md

Best for: Asking LLMs to analyze architecture, explain code patterns, or suggest improvements

2. Technical Documentation Generation

Create comprehensive documentation from your codebase:

llm-pack \
  --format markdown \
  --extensions .js,.ts,.md \
  --preserve-code-blocks \
  --output ./llm/documentation-input.md

Best for: Having LLMs generate documentation, API guides, or technical specifications

3. Code Migration Planning

Prepare codebase analysis for migration projects:

llm-pack \
  --use-compactor \
  --importance 0.9 \
  --preserve-structure \
  --normalize-line-endings \
  --output ./llm/migration-analysis.md

Best for: Asking LLMs to plan migrations, suggest refactoring strategies, or identify dependencies

4. Bug Analysis

Collect relevant code context for bug investigation:

llm-pack \
  --format markdown \
  --context-lines 5 \
  --importance 0.8 \
  --output ./llm/bug-context.md

Best for: Having LLMs analyze bugs, suggest fixes, or explain error patterns

5. Code Review Assistance

Generate focused content for code review:

llm-pack \
  --use-compactor \
  --compact-lines 50 \
  --importance 0.9 \
  --extensions .js,.ts,.jsx,.tsx \
  --output ./llm/code-review.md

Best for: Getting LLMs to review code, suggest improvements, or identify potential issues

Best Practices for LLM Interaction

Content Optimization
- Use --use-compactor to focus on important code sections
- Set --importance threshold to 0.8+ for critical code understanding
- Include --context-lines to maintain code comprehension
Format Selection
- Use markdown format for better code structure preservation
- Enable --preserve-code-blocks for accurate syntax interpretation
- Use --normalize-whitespace for consistent formatting
Context Management
- Keep file sizes manageable (use --max-file-size)
- Include necessary context with --context-lines
- Use --compact-lines to focus on essential code

Input Size Optimization

For GPT-3.5 (4K context)

llm-pack \
  --use-compactor \
  --compact-lines 50 \
  --max-file-size 100 \
  --importance 0.9

For GPT-4 (8K context)

llm-pack \
  --use-compactor \
  --compact-lines 100 \
  --max-file-size 200 \
  --importance 0.8

For Claude (100K context)

llm-pack \
  --use-compactor \
  --compact-lines 500 \
  --max-file-size 1000 \
  --importance 0.7

LLM Prompt Examples

Code Understanding

I'm providing a codebase overview generated by llm-pack. Please:
1. Analyze the overall architecture
2. Identify key design patterns
3. Suggest potential improvements
<paste llm-pack output>

Documentation Generation

I've used llm-pack to collect my project's code. Please:
1. Generate comprehensive documentation
2. Include API endpoints and usage
3. Document key functions and components
<paste llm-pack output>

Migration Planning

Here's my codebase exported by llm-pack. Please:
1. Analyze the current structure
2. Suggest a migration strategy
3. Identify potential challenges
<paste llm-pack output>

Bug Analysis

I've collected relevant code context using llm-pack. Please:
1. Analyze the code for potential issues
2. Suggest possible bug fixes
3. Recommend prevention strategies
<paste llm-pack output>

Common LLM Tasks

Architecture Review

llm-pack \
  --use-compactor \
  --importance 0.9 \
  --preserve-structure \
  --output ./llm/architecture.md

For getting LLM insights on system design and architecture

Security Analysis

llm-pack \
  --use-compactor \
  --importance 0.95 \
  --context-lines 10 \
  --output ./llm/security-review.md

For having LLMs identify potential security issues

API Documentation

llm-pack \
  --extensions .js,.ts \
  --preserve-code-blocks \
  --normalize-whitespace \
  --output ./llm/api-docs.md

For generating API documentation with LLM assistance

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.