Mix Task: LLM Ingest

A powerful Mix task for generating LLM-friendly code ingestion files from Elixir projects. Creates well-structured markdown files containing your source code, making it easy to share context with Large Language Models for code analysis, debugging, or feature development.

✨ Features

• 🎯 Feature-based filtering - Focus on specific parts of your codebase
• 📁 Smart directory traversal - ASCII tree structure with proper organization
• 🎨 Markdown output - Clean formatting with syntax highlighting
• 🚫 Intelligent exclusions - Built-in patterns for common artifacts
• 📝 Gitignore integration - Automatically respects your .gitignore
• 🔧 Highly configurable - Custom include/exclude patterns
• 🤖 LLM-optimized - Includes context notes for better AI understanding

🚀 Quick Start

1. Install the Task

Copy llm_ingest.ex to your project:

mkdir -p lib/mix/tasks
# Copy llm_ingest.ex to lib/mix/tasks/

2. Create Feature Configuration

Create llm_features.exs in your project root:

%{
  "auth" => %{
    include: "lib/auth/**,test/auth/**,priv/repo/migrations/*_auth_*",
    exclude: "**/*_test.exs"
  },
  "api" => %{
    include: "lib/api/**,lib/schemas/**,test/api/**",
    exclude: "lib/api/legacy/**"
  },
  "core" => %{
    include: "lib/application.ex,lib/core/**,mix.exs"
  }
}

3. Generate LLM-Ready Files

# Generate for specific feature
mix llm_ingest --feature=auth

# Generate for entire project
mix llm_ingest

# Custom patterns
mix llm_ingest --include="lib/specific/**" --exclude="**/*_old.ex"

📖 Usage Examples

Basic Commands

# Feature-based ingestion (recommended)
mix llm_ingest --feature=auth
# → Creates: doc/llm-ingest-auth.md

# Full project analysis
mix llm_ingest
# → Creates: doc/llm-ingest.md

# Custom patterns
mix llm_ingest --include="lib/payments/**,test/payments/**"
# → Creates: doc/llm-ingest.md

# Custom output location
mix llm_ingest --feature=api --output=analysis/api-deep-dive.md

Advanced Usage

# Combine feature with additional excludes
mix llm_ingest --feature=auth --exclude="**/legacy_auth.ex"

# Disable gitignore integration
mix llm_ingest --no-gitignore

# Different root directory
mix llm_ingest /path/to/other/project --feature=core

⚙️ Configuration

Feature Configuration File (llm_features.exs)

%{
  "auth" => %{
    include: "lib/auth/**,test/auth/**,priv/repo/migrations/*_create_users*",
    exclude: "**/*_test.exs,lib/auth/legacy/**"
  },
  "api" => %{
    include: "lib/api/**,lib/schemas/**,lib/*_web/controllers/api/**",
    exclude: "lib/api/v1/**"  # Exclude old API versions
  },
  "frontend" => %{
    include: "assets/**,lib/*_web/**,test/*_web/**",
    exclude: "assets/node_modules/**"
  },
  "payments" => %{
    include: "lib/payments/**,lib/billing/**,test/payments/**",
    exclude: "lib/payments/legacy/**,**/*_test.exs"
  },
  "complete" => %{
    include: "lib/**,config/**,mix.exs",
    exclude: "**/*_test.exs,lib/legacy/**"
  }
}

Pattern Syntax

• ** - Matches any number of directories (recursive)
• * - Matches any characters except /
• lib/auth/** - All files under lib/auth/ recursively
• *.ex - All .ex files
• **/*_test.exs - All test files recursively

📋 Command Line Options

Option	Description	Example
--feature	Use predefined feature configuration	--feature=auth
--include	Include files matching patterns (comma-separated)	--include="lib/auth/**,test/auth/**"
--exclude	Exclude files matching patterns (comma-separated)	--exclude="**/*_test.exs"
--output	Specify output file path	--output=analysis/deep-dive.md
--no-gitignore	Disable automatic gitignore integration	--no-gitignore

📄 Output Structure

The generated markdown file includes:

# ProjectName
Project description from mix.exs

---

## Feature: auth
**Include patterns:** `lib/auth/**`, `test/auth/**`
**Exclude patterns:** `**/*_test.exs`

---

## Notes for AI Analysis
**Authentication Feature Context:**
- Look for authentication pipelines, plugs, and middleware
- Check for token generation, validation, and refresh logic
...

**General Elixir Project Reading Guide:**
- Application structure explanations
- Module conventions and patterns
- Key Elixir idioms and practices

---

## Project Structure

./lib/ ├── auth/ │ ├── guardian.ex │ ├── pipeline.ex │ └── user_manager.ex └── auth.ex

---

## Files

---

### lib/auth/guardian.ex
```elixir
defmodule MyApp.Auth.Guardian do
  # Your code with syntax highlighting
end

## 🎯 LLM Integration Tips

### Best Practices

1. **Feature-Focused Analysis**
   ```bash
   # For authentication work
   mix llm_ingest --feature=auth
   
   # For API development  
   mix llm_ingest --feature=api

2. Include Relevant Tests

   "auth" => %{
     include: "lib/auth/**,test/auth/**",  # Include tests for context
     exclude: "test/fixtures/**"           # But exclude fixtures
   }

3. Exclude Legacy Code

   "payments" => %{
     include: "lib/payments/**",
     exclude: "lib/payments/legacy/**,lib/payments/deprecated/**"
   }

Effective Prompts

When sharing generated files with LLMs:

I'm working on the authentication feature of my Elixir app. 
Here's the relevant code structure and implementation:

[Paste generated markdown content]

Please help me:
1. Review the authentication pipeline for security issues
2. Suggest improvements for token handling
3. Identify any missing error cases

🔧 Project Setup

Add to Your Project

1. Copy the task file:

   mkdir -p lib/mix/tasks
   curl -o lib/mix/tasks/llm_ingest.ex https://gist.github.com/[gist-id]/raw/llm_ingest.ex

2. Create feature configuration:

   # Copy example and customize
   curl -o llm_features.exs https://gist.github.com/[gist-id]/raw/llm_features.example.exs

3. Update .gitignore:

   # LLM ingest files
   doc/llm-ingest*.md

4. Compile and use:

   mix compile
   mix llm_ingest --feature=auth

Project Requirements

• Elixir: 1.14+ (uses modern Path and File functions)
• Mix: Standard Elixir installation
• Dependencies: None (pure Elixir/OTP)

🛠️ Customization

Adding New File Types

Edit the write_files_content function to add syntax highlighting for new extensions:

language = case Path.extname(file) do
  ".ex" -> "elixir"
  ".exs" -> "elixir"
  ".heex" -> "html"      # Add Phoenix templates
  ".leex" -> "html"      # Add LiveView templates
  ".eex" -> "html"       # Add EEx templates
  # ... your custom extensions
  _ -> ""
end

Custom Default Exclusions

Modify the @default_excludes list for your needs:

@default_excludes [
  # Add your custom exclusions
  "priv/static/**",
  "coverage/**",
  # ... existing exclusions
]

Feature-Specific Notes

Add custom notes in the get_feature_specific_notes function:

defp get_feature_specific_notes(feature) do
  case feature do
    "my_custom_feature" -> """
**My Custom Feature Context:**
- Look for specific patterns in your domain
- Check for custom implementations
"""
    # ... existing features
  end
end

🤝 Contributing

This is a community-driven tool! Ways to contribute:

1. ⭐ Star this gist if you find it useful
2. 💬 Comment with suggestions, improvements, or issues
3. 🍴 Fork and customize for your specific needs
4. 📢 Share with the Elixir community

Common Improvements

• Add support for more file types
• Enhance pattern matching capabilities
• Improve LLM context notes
• Add integration with other tools

📚 Real-World Examples

Phoenix Web Application

%{
  "auth" => %{
    include: "lib/my_app/accounts/**,lib/my_app_web/controllers/auth_*,test/my_app/accounts/**",
    exclude: "**/*_test.exs"
  },
  "live_views" => %{
    include: "lib/my_app_web/live/**,assets/js/**",
    exclude: "assets/node_modules/**"
  },
  "api" => %{
    include: "lib/my_app_web/controllers/api/**,lib/my_app_web/views/api/**",
    exclude: "lib/my_app_web/controllers/api/v1/**"  # Old version
  }
}

GenServer/OTP Application

%{
  "workers" => %{
    include: "lib/my_app/workers/**,lib/my_app/supervisor.ex",
    exclude: "**/*_test.exs"
  },
  "core" => %{
    include: "lib/my_app/application.ex,lib/my_app/core/**,config/**"
  }
}

Umbrella Project

%{
  "web_app" => %{
    include: "apps/my_app_web/**",
    exclude: "apps/my_app_web/assets/node_modules/**"
  },
  "business_logic" => %{
    include: "apps/my_app/**",
    exclude: "**/*_test.exs"
  }
}

🐛 Troubleshooting

Empty Output

Problem: Generated file only contains headers, no files

Solutions:

1. Check include patterns match actual file paths
2. Verify files aren't excluded by default patterns
3. Test with simpler patterns first: --include="lib/**"

Pattern Not Matching

Problem: Expected files not included

Solutions:

1. Use forward slashes / even on Windows
2. Remember ** for recursive, * for single level
3. Test patterns: --include="lib/auth/**" --exclude=""

Permission Errors

Problem: Cannot read files or write output

Solutions:

1. Check file permissions on source directories
2. Ensure doc/ directory is writable
3. Run with appropriate permissions

Large Output Files

Problem: Generated files are too large for LLM context

Solutions:

1. Use more specific include patterns
2. Exclude test files: --exclude="**/*_test.exs"
3. Focus on specific features rather than entire project

📜 License

This Mix task is provided as-is for educational and development purposes. Feel free to modify and distribute according to your project's needs.

---

Made with ❤️ for the Elixir community

Star this gist if it helps you build better software with AI assistance!