Taming Rust’s target/ Directory

My disk hit 96% last week. I went looking for old Docker images or a forgotten VM, and found this instead:

$ find ~ -name CACHEDIR.TAG -path '*/target/*' | wc -l
38
$ # ...sum their sizes
150.6 GiB

150.6 GiB of Rust build artifacts, in 38 target/ directories. Three of them were over 30 GiB each. The source code of those three projects together is a few megabytes.

cargo clean fixes this for about two weeks. I wanted to understand where the bytes actually go, try the usual fixes, and see if something better is possible. This post follows that order: the problems, the usual solutions, what I added, and where I ended up.

Ground rules

Smaller is easy if you give up enough. So before changing anything, I decided what must survive:

• A panic must still give a backtrace with file:line in my own code.
• I must still be able to stop in a debugger in my own code and see variables.
• All tests must pass.
• The edit, build, test loop must not get slower. That is the thing I do a hundred times a day.

The problems

I started with the three biggest directories. They told the same story:

Release builds do not matter; everything is in the dev profile. Inside it, I found three problems.

1. The build I use is too big. Most of deps/ is executables, and they are test binaries. Every file in tests/ is compiled into its own executable, and each one links every dependency it uses, statically, together with the full debug info of all of them. These projects have 8 to 42 integration test files, and each test binary is 130 to 380 MB.

2. The builds I stopped using are never deleted. Cargo names artifacts like libtokio-<hash>.rlib, and the hash covers the compiler version, the profile, the features and the versions of all dependencies. Update Rust, try a different profile setting, bump one crate deep in the tree, and a new copy appears next to the old one. Nothing ever deletes the old one. Project B has 419 incremental directories for its 34 crates: about twelve generations of each.

3. Every project builds the same dependencies again. My services use almost the same crates, and each one compiles tokio, hyper and the rest into its own target/.

The usual solutions

Searching for this gives roughly the same list every time:

• cargo clean, now and then. Frees everything; the next build starts from zero.
• Less debug info: debug = false, or debug = "line-tables-only", which keeps just enough for backtraces.
• Compress the debug info when linking: -C link-arg=-Wl,--compress-debug-sections=zstd.
• Turn off incremental compilation: incremental = false.
• Clean old artifacts with a tool: cargo-sweep deletes artifacts by age or by toolchain, and cargo-target-gc cleans stale caches by age.
• Share work between projects: one CARGO_TARGET_DIR for all of them, or sccache, the Rust version of ccache.
• On nightly: stop storing crate metadata twice, with -Zembed-metadata=no.

Cargo itself cleans its download cache since 1.88, but doing the same for build artifacts is still an open issue. So I tried all of the above.

A test bench

I did not want to experiment on my real projects. A full build of one of them takes minutes, and I could not compare two settings fairly. So I built a demo workspace with the dependencies most of my projects end up with anyway: axum, tokio, reqwest, tonic and OpenTelemetry, async-nats, jsonwebtoken, utoipa, clap, tracing. At this point it is less a dependency list and more a personality. Writing a whole service just to measure it used to be a lot of work for one blog post, but code is cheap to write now; the measuring was the expensive part.

It has 358 crates in the lockfile and 24 integration test files, and every test calls into the full stack, so every test binary links what a real one would: 194 MB each, right in the range of my real projects. Each configuration runs the same session from an empty target directory (build, build the tests, a rust-analyzer check, three rounds of edit and rebuild, run the tests) and then checks the ground rules. The details are at the end. With default settings:

target/ after the session          8.19 GiB
  executables (31 files)           5.27 GiB
  rlib + rmeta                     1.51 GiB
  incremental                      0.93 GiB
edit, rebuild all tests             7.0 s

Trying the usual solutions

Less debug info

debug = false gives 2.58 GiB, 68% less. And the first ground rule fails immediately. The panic message still says where it happened (that location is compiled in either way), and the frames still have function names, but none of them has a file:line.

debug = "line-tables-only" keeps exactly what backtraces need and drops the rest. 4.19 GiB, 49% less, and backtraces work. But gdb can no longer show local variables, so the second rule fails.

Compressing the debug info

This one asks the linker to compress the debug info inside each binary. Same information, smaller file:

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-arg=-Wl,--compress-debug-sections=zstd"]

It looked like a win. The target directory went down to 4.70 GiB, all tests passed, and gdb could still stop on a line and show variables. Then I looked at a backtrace. This is what a panic in my code prints with RUST_BACKTRACE=1 and the default settings:

   2: svc_core::explode
             at ./crates/svc-core/src/lib.rs:79:9
   3: svc_core::explode
             at ./crates/svc-core/src/lib.rs:81:5

And this is the same panic with zstd compressed debug info:

   2: svc_core::explode
   3: svc_core::explode

The at file:line lines are gone. When a Rust program panics, it reads its own debug info to print them, and that code in the standard library understands zlib compression but not zstd. When it finds zstd, it quietly prints the function names and nothing else. gdb is a separate program that understands both, so the debugger keeps working and nothing points at the problem. You find out the day you need a backtrace.

It was also slower: rebuilding the tests after an edit took 14.0 seconds instead of 7.0, because the linker now compresses 24 large binaries every time. With zlib instead of zstd, backtraces work and edits are not slower, at 5.19 GiB. But every test binary still carries its own copy of the debug info, only compressed.

Turning off incremental

incremental = false removes the incremental/ directory, which sounds like a lot. It saved 14%, and the edit loop got slower, 8.4 seconds instead of 7.0. That is the right setting for CI and the wrong one for a laptop.

Not storing metadata twice

When cargo builds a library, it asks rustc for two files: an .rlib with the code and an .rmeta with the crate metadata, so that dependent crates can start compiling earlier. By default the metadata is written into both. Nightly Rust can leave it out of the .rlib:

$ cargo +nightly -Zembed-metadata=no build

Or per project:

[unstable]
embed-metadata = false

(The value is a boolean. "no" is rejected, which surprised me after typing it on the command line.)

The rlibs became 26% smaller, 1.08 to 0.80 GiB. For the whole directory that is only 4% with default settings, because the executables are so much bigger. It works, and it keeps every ground rule, but it does not touch the real problem yet. I come back to it later. Current nightly cargo already does this by default, so it should reach stable without any setting.

On stable it does not work. I tried passing the rustc flag directly with RUSTFLAGS="-Zembed-metadata=no" and RUSTC_BOOTSTRAP=1. The flag reaches rustc, and then the build fails:

error: only metadata stub found for `rlib` dependency `proc_macro2` please provide path to the corresponding .rmeta file with full metadata

Only nightly cargo knows that it has to point dependent crates to the .rmeta files.

Cleaning old builds

To test the cleaners, I let a target directory age the way mine do, on a realistic timeline: I set the file times of each step back to when it would really have happened. No line of code changed.

Building everything I currently use from scratch, release included, takes 7.97 GiB. So 21.2 GiB of this directory is garbage. For each cleaner I started from an identical copy of it, ran the cleaner, and then ran my normal session to see what had to be rebuilt:

None of them gets it right. cargo clean deletes the build I am using. Cleaning by age deletes it too, because the build I am using is 30 days old: cargo does not touch files it reuses, so a live artifact and a dead one look the same. Cleaning by toolchain is safe, but it only finds the toolchain part. cargo-target-gc saw that the directory was used today and left all of it alone.

Sharing between projects

For the third problem, I built two different projects with the same dependency stack, one after the other:

The shared directory built the second project three times faster, but saved only 10% of disk, because test binaries are never shared and they are most of the size.

sccache surprised me. Only 412 of 716 cacheable compilations were hits, and the second project built slower than with no cache at all. It also cannot cache anything that is linked (binaries, tests, proc macros) or compiled incrementally (your own crates), which was 101 of 817 compiler calls here. Where it does well is the same project after cargo clean: every cacheable compilation was a hit, and the rebuild took 22.3 seconds instead of 42.8. So it makes cargo clean cheap, and it is useful in CI. It does not save disk; it adds some.

A shared target directory has its own costs. Builds lock it, so two projects (or two editors running rust-analyzer) wait for each other. Binaries with the same name overwrite each other in target/debug/. And cargo clean in one project deletes everything.

So far: some of the usual solutions break a ground rule, some help a little, and none of them solves a problem.

Adding new things

Debug info, but only where I need it

Turning off debug info failed the ground rules because it removed it everywhere. But I debug my own code; I almost never step into tokio. Cargo can set profile options for all dependencies at once, and "*" means every package that is not a member of the workspace:

[profile.dev.package."*"]
debug = false

All ground rules pass. My crates keep full debug info. Frames inside dependencies still show function names in a backtrace, just without file:line. If I ever need to step into one dependency, I can turn it back on for that crate only:

[profile.dev.package.hyper]
debug = true

A side effect I did not expect: proc macros and build scripts shrank from 0.39 to 0.11 GiB. Cargo already skips debug info for those, but the documentation says "when possible", and apparently it is often not possible.

This setting is in my global config now, so it applies to every project on the machine. One warning: a profile setting is part of every artifact’s hash, so changing it means the next build is a full rebuild, and the old artifacts stay where they are. That is the second problem again; I come back to it below.

So far: 3.84 GiB (from 8.19), edit loop 3.4 s (from 7.0).

One copy of the debug info

Compression made 24 smaller copies of the debug info. On Linux, the linker copies debug info into every executable by default. split-debuginfo = "unpacked" leaves it in .dwo files next to the object files instead, and the executables only point to them:

[profile.dev]
split-debuginfo = "unpacked"

Backtraces and gdb work as before. The only catch is that the binaries now refer to the .dwo files by absolute path, so you should not copy dev builds to another machine. You probably were not doing that anyway. (macOS already does this by default.)

So far: 3.14 GiB, edit loop 2.9 s. Both settings are in my global config; no project has changed yet.

One test binary

I still had 24 test binaries for one crate, and each one is a separate link of the whole dependency tree. Cargo makes every file in tests/ its own crate, but it also accepts a directory with a main.rs:

tests/
├── health.rs
├── publish_ok.rs
└── ... 22 more

tests/
└── it/
    ├── main.rs        // mod health; mod publish_ok; ...
    ├── health.rs
    └── ...

No Cargo.toml change is needed; cargo finds tests/it/main.rs by itself. Move the files, write the mod lines, done.

I expected a smaller directory. I did not expect the edit loop to get almost three times faster on top of what I already had. After a small edit, most of the remaining time went into linking 24 almost identical binaries. Now it links one.

The cost is that integration tests now share one process. They already shared threads inside each file, so anything that is safe in parallel is safe here. Look out for tests that change environment variables or the working directory, set process-wide state that can only be set once (for a global tracing subscriber, use try_init()), or need a different panic strategy. Keep those in their own file. If you want one process per test anyway, cargo nextest gives you that on top of a single binary. A subset runs with cargo test --test it publish_.

So far: 1.76 GiB, edit loop 1.0 s. 78% less than where I started, the edit loop is seven times faster, and all ground rules still hold.

Metadata, again

Now the executables are no longer the problem. The 1.76 GiB looks like this:

rlib + rmeta (libraries)          1.01 GiB
incremental                       0.30 GiB
executables (8 files)             0.18 GiB
proc macros, build scripts        0.14 GiB
.dwo (split debug info)           0.12 GiB

More than half is libraries now, which is exactly what -Zembed-metadata=no shrinks. On nightly, with everything above: 1.38 GiB instead of 1.76, with the rlibs going from 0.61 to 0.35 GiB and the edit loop still at 1.0 second. That is the smallest directory in this post that keeps every ground rule. On stable, it has to wait.

Asking cargo what is still in use

That leaves the old builds. The cleaners failed because they guess from the outside: by age, or by toolchain. But cargo knows exactly which files it needs. Run any cargo command with --message-format=json, and it reports every file the build uses, including the ones that are already up to date. Each of those files has a hash in its name. Any file in target/ whose hash is not on that list belongs to a build I no longer use.

So I wrote a small Python script, target-gc, that does just that:

1. Mark: run the cargo commands I normally use (build, build the tests, check) and collect the hashes of every file cargo reports.
2. Sweep: delete every file and directory in target/ whose hash is not in that list. Profiles that none of the commands use, like release/, are not touched.

On the same aged directory, next to the two reference points from before:

It found 99% of the garbage in under two seconds, and the next session compiled nothing. If you regularly build with other features or profiles, give it those commands too, for example target-gc -- "test --no-run --features foo" "build --release", and try --dry-run first. For a shared target directory, it must see every project that uses it: target-gc --manifest-path ~/a/Cargo.toml --manifest-path ~/b/Cargo.toml.

Outcome

This is what I ended up with. In my global ~/.cargo/config.toml:

[profile.dev]
split-debuginfo = "unpacked"

[profile.dev.package."*"]
debug = false

plus one integration test binary per crate, and target-gc once a week and after every rustup update. On the demo, that is 1.76 GiB instead of 8.19, and 1.0 second instead of 7.0 to rebuild the tests after an edit, with every ground rule still holding. Run target-gc (or cargo clean) once after changing the config, because the change leaves a full set of old artifacts behind.

If your constraints are different, these are the variants I measured:

For many projects with the same dependencies, a shared target directory on top of this made the second project build three times faster and saved 26% of disk (3.48 to 2.58 GiB).

CI

The CI row deserves a word. CI caches the target directory and pays for every byte twice per run, once to download and once to upload. With incremental off, the defaults leave 6.72 GiB; with the CI settings it is 1.12 GiB, and building the tests took 10.1 seconds instead of 15.9. The Cargo documentation says incremental is off by default when CI is set. I checked, and on Rust 1.98 it is not; it starts with 1.99. On older versions, set CARGO_INCREMENTAL=0 yourself. Package-specific profile settings cannot be set through environment variables, so [profile.dev.package."*"] has to go into the repository’s .cargo/config.toml. And run target-gc with your CI’s cargo commands before the cache is saved, so old artifacts never get into it.

What I learned

• Most of a dev target/ is test binaries, and most of a test binary is debug info. Fix those two, and the edit loop gets faster too.
• Turning things off globally is what breaks debugging. Turning them off where you do not need them does not.
• The popular tips are not wrong, but each solves a smaller problem than it seems, and one of them quietly breaks backtraces.
• Old builds are the other half of the problem, and only cargo knows which files are still in use. Guessing by age deletes the build you are using.

A note on what these numbers mean

The session for every configuration: cargo build --workspace, cargo test --workspace --no-run, cargo check --workspace --all-targets (what rust-analyzer runs), three rounds of editing a source file and rebuilding all tests (I report the median), and cargo test --workspace, which must pass. Then the ground rules: a panic inside one of my crates must show file:line in the backtrace frames, and gdb must stop on a line and print the function’s arguments.

Sizes are disk usage: allocated blocks, with hardlinked files counted once. One honest detail: my first round of measurements counted the hardlinked copies twice. That made the configurations look 9 to 23% bigger, and it hid the fact that split-debuginfo still helps on top of the other changes. I found it while reviewing the scripts, measured everything again, and the numbers in this post are from the second round.

Sizes do not depend on the machine: the same toolchain and settings produce the same bytes anywhere. Times do. They are from one machine with 20 cores and an NVMe disk, and most of the edit loop is linking, so a smaller machine will show larger numbers. The ratios between configurations are what I would trust.

Scope

Everything here was measured on one demo workspace, with Rust 1.98 stable (1.99 for the toolchain update, nightly 1.101 for the metadata tests), on x86_64 Linux, linking with rust-lld, which is the default there now. On macOS split-debuginfo already defaults to unpacked, and on Windows with MSVC debug info goes into PDB files, so those two will look different.

target-gc, the demo workspace and every measurement script are on GitHub.