lingest

# lingest `lingest` (Local Ingest) is a command-line tool that generates a single Markdown file named `lingest_output.md`. It recursively finds all text files in the current directory and its subdirectories and concatenates their content. Each file's content is preceded by a header indicating its relative path. This is particularly useful for creating a single context file for Large Language Models (LLMs) from your local projects, enabling them to understand the structure and content of your codebase or documentation. ## Features - **Directory Tree Visualization:** Automatically generates a visual directory tree structure at the beginning of the output, showing the project layout with standard tree characters. - **Recursive Scanning:** Traverses directories deep to find all relevant files. - **Content Concatenation:** Combines the content of all found text files into one output. - **Clear File Headers:** Each file's content is demarcated with `FILE: path/to/file.ext` for easy identification. - **Customizable Output:** Specify the name of the generated Markdown file. - **Flexible Filtering:** - **Ignore Patterns:** Use glob patterns to exclude specific files or directories (e.g., build artifacts, temporary files). - **Include Patterns:** Focus the tool on specific file types or directories using glob patterns. - **Overwrite Protection:** Prevents accidental overwriting of existing output files unless explicitly forced. - **Quiet Mode:** Suppresses informational logs, ideal for scripting or CI/CD pipelines. - **Dry Run Mode:** Preview which files would be included and where the output would be saved, without writing anything. - **Standard CLI Interface:** Includes `--help` and `--version` flags. ## Installation The easiest way to use `lingest` is with `npx`: ```bash npx lingest [options] ``` Alternatively, install it globally using npm, Yarn, or pnpm: ```bash npm install -g lingest # or yarn global add lingest # or pnpm add -g lingest ``` Once installed globally, run it directly: ```bash lingest [options] ``` ## Usage Navigate to the root directory of the project you want to process, then run: ```bash lingest [options] ``` ### Options | Option | Alias | Description | Default | | ----------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | | `--output` | `-o` | Output file name for the generated Markdown. | `lingest_output.md` | | `--ignore` | `-i` | Comma-separated list of glob patterns to ignore (files or directories). These are added to default ignores. | `""` | | `--include` | `-n` | Comma-separated list of glob patterns to include. If specified, only files matching these patterns will be processed (still respects ignore patterns). | `""` (include all) | | `--force` | `-f` | Force overwrite of the output file if it already exists. | `false` | | `--quiet` | `-q` | Suppress informational messages; only errors will be shown. | `false` | | `--dry-run` | | List files that would be processed and the final output path, but don't actually write the file. | `false` | | `--no-tree` | | Skip generating the directory tree structure at the beginning of the output. | `false` | | `--help` | `-h` | Show this help message and exit. | | | `--version` | `-v` | Show the program's version number and exit. | | ### Default Ignored Items By default, `lingest` ignores common files and directories that are typically not needed in code analysis: **Directories:** - `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `.tox`, `.nox`, `.mypy_cache`, `.ruff_cache`, `.hypothesis` - `bower_components`, `.npm`, `.yarn`, `.pnpm-store`, `.gradle`, `build`, `.settings`, `.build`, `.bundle` - `vendor/bundle`, `target`, `pkg`, `obj`, `packages`, `bin`, `.svn`, `.hg` - `venv`, `.venv`, `env`, `.env`, `virtualenv`, `.idea`, `.vscode`, `.vs` - `.cache`, `.sass-cache`, `dist`, `out`, `site-packages`, `.docusaurus`, `.next`, `.nuxt` - `.terraform`, `vendor`, `xcuserdata` **Files:** - Python: `*.pyc`, `*.pyo`, `*.pyd`, `.coverage`, `poetry.lock`, `Pipfile.lock` - JavaScript/Node: `package-lock.json`, `yarn.lock`, `bun.lock`, `bun.lockb` - Java: `*.class`, `*.jar`, `*.war`, `*.ear`, `*.nar`, `.classpath`, `*.gradle`, `.project` - C/C++: `*.o`, `*.obj`, `*.dll`, `*.dylib`, `*.exe`, `*.lib`, `*.out`, `*.a`, `*.pdb` - Swift/Xcode: `*.xcodeproj`, `*.xcworkspace`, `*.xcuserstate`, `.swiftpm` - Ruby: `*.gem`, `Gemfile.lock`, `.ruby-version`, `.rvmrc` - Rust: `Cargo.lock`, `**/*.rs.bk` - .NET/C#: `*.suo`, `*.user`, `*.nupkg` - Version control: `.gitignore`, `.gitattributes`, `.gitmodules` - Media files: `*.svg`, `*.png`, `*.jpg`, `*.jpeg`, `*.gif`, `*.ico`, `*.pdf`, `*.mov`, `*.mp4`, `*.mp3`, `*.wav` - Temporary/cache: `*.log`, `*.bak`, `*.swp`, `*.tmp`, `*.temp`, `.DS_Store`, `Thumbs.db` - Build artifacts: `*.egg-info`, `*.egg`, `*.whl`, `*.so` - Minified files: `*.min.js`, `*.min.css`, `*.map` - Terraform: `*.tfstate*` - The output file itself (e.g., `lingest_output.md`, or whatever is specified by `--output`) ### Glob Patterns `lingest` uses [micromatch](https://github.com/micromatch/micromatch) for glob pattern matching. Common examples: - `**/*.js`: Matches all `.js` files in any directory. - `src/**`: Matches all files and folders within the `src` directory. - `*.log`: Matches all `.log` files in the current directory. - `!src/important.js`: (Negation not supported directly but illustrates how to exclude files conceptually.) ### Examples 1. **Basic usage (generates `lingest_output.md` in the current directory):** ```bash lingest ``` Or using npx: ```bash npx lingest ``` 2. **Specify a custom output file name:** ```bash lingest --output project_snapshot.md ``` Or: ```bash lingest -o project_snapshot.md ``` 3. **Add custom ignore patterns (e.g., all `dist` folders and `.log` files):** ```bash lingest --ignore "**/dist/**,**/*.log" ``` 4. **Only include JavaScript and TypeScript files:** ```bash lingest --include "**/*.js,**/*.ts" ``` 5. **Include only `.md` files from the `docs` folder, excluding `drafts`:** ```bash lingest --include "docs/**/*.md" --ignore "docs/drafts/**" ``` 6. **Force overwrite an existing output file:** ```bash lingest --force ``` Or: ```bash lingest -f ``` 7. **Dry run for Python files:** ```bash lingest --dry-run --include "**/*.py" ``` 8. **Quiet mode for scripting:** ```bash lingest -q -o context.md ``` 9. **Generate output without directory tree:** ```bash lingest --no-tree ``` ## Output Format Each file’s content in the generated Markdown file will be structured like this: ```markdown # FILE: path/to/your/file.ext (Content of file.ext) ================================================ # FILE: another/path/file.js ================================================ (Content of file.js) ``` ## Handling Non-Text Files If `lingest` encounters a non-UTF-8 or unreadable file, it will: 1. Log a warning (unless in quiet mode). 2. Include the file’s header. 3. Add `[Content not included: Could not be read as UTF-8 text...]`. This preserves file structure and alerts you about skipped files. ## Contributing Contributions are welcome! 1. Fork the repository. 2. Create a branch. 3. Make your changes. 4. Submit a pull request with details. Issues and suggestions? Open one on [GitHub](https://github.com/chiragasarpota/lingest) (replace with actual link). ## License MIT License. See `LICENSE` file for details.