pulp Open the mill

Local-first · written in Rust · MIT

Grind a folder into one LLM-ready dump.

Pulp walks a tree, pulls text out of code, PDFs, Office files, and notebooks in parallel, and writes one file you can hand a model.

cargo install --git https://github.com/BeeGass/pulp --locked

The mill after pulping its built-in sample. Nothing is uploaded: open the browser mill and press Try a sample to run it in your tab.

Reference

What goes in, what comes out.

KindNotes
SourceRust, Lean, Python, TypeScript, TOML, Markdown, and other text
NumPy.npy and .npz metadata and a small preview (not treated as binary)
HTML, XML, JSONReadable text, or decoded source with --source
CSV, TSVTabular text
PDF, Word, PowerPoint, Excel, OpenDocument, EPUB, RTFExtracted text
JupyterCells; --notebook-outputs keeps outputs
Zip, tarA root archive always expands; nested members need --archives

Binary media (images, audio, wasm, …) is skipped unless --binaries. Noisy trees stay out even when they are tracked: node_modules/, target/, dist/, virtualenvs, lockfiles, .env, keys, and similar. Inside a git repository, .gitignore is honored unless --no-gitignore.

Formats

The same two files, three ways.

Plain text · -f txt
Directory structure: testdata/ ├── Hello.lean └── hello.rs ================================================ FILE: Hello.lean ================================================ def hello : String := "hello from pulp" ================================================ FILE: hello.rs ================================================ pub fn hello() -> &'static str { "hello from pulp" }
Markdown · -f md
# Directory structure ``` testdata/ ├── Hello.lean └── hello.rs ``` ## Hello.lean ```lean def hello : String := "hello from pulp" ``` ## hello.rs ```rust pub fn hello() -> &'static str { "hello from pulp" } ```
XML · -f xml
<documents> <document_tree> testdata/ ├── Hello.lean └── hello.rs </document_tree> <document index="1"> <source>Hello.lean</source> <document_content> def hello : String := "hello from pulp" </document_content> </document> <document index="2"> <source>hello.rs</source> <document_content> pub fn hello() -&gt; &amp;'static str { "hello from pulp" } </document_content> </document> </documents>

Real output of pulp -f txt|md|xml testdata from this repo. Without -f, the layout follows the -o extension and otherwise defaults to plain text.

Options

Flags you will reach for.

FlagMeaning
-o, --output FILEWrite the dump here (default: stdout)
-f, --format FMTtxt, md, or xml
--tree MODEselected (default), full, or none
-j, --jobs NExtraction threads; 0 uses every available core
--include GLOBRepeatable allow-list
--exclude GLOBRepeatable extra deny-list
--hiddenInclude hidden files (.git is still skipped)
--no-gitignoreIgnore .gitignore and .ignore
--archivesExpand nested zip and tar members
--sourceKeep HTML, XML, and JSON as source
--notebook-outputsInclude Jupyter cell outputs
--listPrint the paths that would be pulped

Every flag is in the README.

Two mills

Same core, two places to run it.

CapabilityBrowser mill · /millLocal mill · pulp ui
Files leave the deviceNeverNever
Choosing filesFolder or file picker, or dropYour OS file manager, or any typed path
.gitignoreNot availableHonored
Default excludesSame core matcherSame core matcher
PDF, Office, zipA worker of its own, with a time limitA child process with a time limit
CancelStops the workerBetween files
SetupNoneOne cargo line

Install

One cargo line.

cargo install --git https://github.com/BeeGass/pulp --locked
$ pulp ui              # local mill on 127.0.0.1:8747
$ pulp -o dump.xml .   # this folder, one XML dump
$ pulp -f md src       # Markdown to stdout

Requires Rust 1.88 or newer. Browse opens Finder on macOS, zenity or kdialog on Linux, and Explorer on Windows; pulp ui --no-open suits headless boxes.