Repository navigation
Consider deprecating forge-specific fetchTree inputs #16740
Description
xokdvium with fetchTree being experimental, is there any efforts, pun intended, nix the forge-specific fetcher code? This seems like an unfun, moving-target maintenance burden to try to track all of their APIs (while users often ask to expand to more types of forges).
Originally posted by @toastal in #16588 (comment)
The forge-specific fetchTree inputs have long been criticized, for various reasons. As they comprise most of the subject-to-change fetchTree types, it would be a good idea to reevaluate supporting them at some point on the road to stabilization.
One of the biggest reasons that they've been criticized is for binding Nix itself to the shifting implementations of external/proprietary APIs. This has caused breakage in the past, e.g. in #16588, and will risk incompatibility issues between different Nix versions post-stabilization.
Activity
xokdvium commentedon Nov 1, 2026
Ultimately it might end up being that the end state of things that we want is that the fetchers are pluggable at executable level. Even the current forge-specific fetchers don't do anything beyond downloading/unpacking a tarball, while evaluation reads the files from a git repo backing the tarball cache. If that could just be a snix-castore that'd be nice. Then you can populate it however you want and the only thing that nix itself would need is being able to do a tiny bit of IPC to the fetcher hookables and read the blob/tree storage in-process.
I generally gravitate a bit more towards this architecture more. It's unavoidable that nix needs forge-specific fetchers that can do incremental fetching (and general freedom to fetch stuff however they want as long as it hands nix a content-addressed tree to read). Certain existing features might make such a split hard (looking at you, LFS support).
Otherwise, it seems to me like a no-brainer that you do want to be able to update forge-specific fetchers at short notice to account for infrastructure changes without that being nix's responsibility.
I think @toastal had good success with that in nixtamal?
toastal commentedon Nov 3, 2026
I don’t know about “success” 😅 as the Nixtamal user pool is still small, but the novel
fresh-cmdconcept (where the maintainer specifies an arbitrary command to be ran on their end) gets around this entirely since you just usegit ls-remoteorTOKEN=x curl https://api.exampleor whatever—to which an arbitrary command is agnostic to the code forge (& VCS). The onus is now on the user, not our project, while allowing the ability to say “filter on all stable or RC tags, to pick the freshest” which can’t be achieved withfetchTree’s query parameters. The “No forge-specific rules” was motivated by:nix run gitlab:…overnix run https://minor.forge/…even if it’s not many more characters & hides nothing. It’s not just about convenience but an implicit endorsement towards those platform leaving the rest to feel less supported. I prefer alternatives being on an even playing field.fetchTreehavegitweb:,allyu:,cgit:,gogs:,allura:,got:,heptapod:, orrgit:? …I think you get the point, & we haven’t even considered the non-Git VCS options. Under this list is all Git & almost all offer a/archive/*.tar.*-like archive.My follow-up question would be “why does this need to be in the Nix binary?”. The biggest reason it seems to be here is to support Flakes, & its input pinning being baked into the binary under the
experimental-features = flakes. Which should prompt the next follow-up question, “does input fetching need to be in the binary?”. If it’s just to keep Flakes together, then it would be worthwhile looking at deprecating flakes as well from the binary—putting it to into a plugin or separate project where that project can build their forge-specific support layer on top, iterating on upstream API churn independent of NixCpp, if they choose to do so.xokdvium commentedon Nov 3, 2026
My point was that "nix, the entire systems" needs such capability - nothing saying that it should be embedded in the binary. For one, I'd be immensely happy if we didn't run most of the fetching code in the main binary address space and instead let people implement proper sandboxing out-of-tree.
Note the emphasis on incremental. My desire is of course to move the actual fetching into external executables that are not strictly dependent on being embedded in the binary (although for practical reasons some of those should probably get included by default in the installation, maybe a symlink farm in libexec/ ).
One unavoidable reality would of course be that nix binary does need to be able to read the fetched tree somehow – and copying to the store is not a viable long-term solution.
So my vision would be something along these lines:
We bless certain blob/tree storage backends by having nix binary be capable of reading from those trees. The only example we have of it now is via libgit2, which also happens to be used for the tarball cache.
Another example of such content-addressed storage could be either something homegrown or the more established snix machinery (I'm not sure it has a pack file implementation for storing objects currently).
So in the smallest approximation, the external fetcher project could be responsible for accepting a description of an input and producing a pack file (or several) and pointing nix at the directory that has those packfiles with the tree hash that represents the tree.
We might even consider allowing to read loose objects and combining multiple storage backends to make local repository workflows a bit saner – not the current duality where dirty repos are read straight from disk and not the git object backends.
Of course blessing only git object stores isn't the only option, though I can't think of another universal content-addressed storage layer that's also used for source code.
toastal commentedon Nov 3, 2026
@P-E-Meunier does Sanakirja fit the description?
P-E-Meunier commentedon Nov 3, 2026
I don't see the relation with Sanakirja, as it's just a low-level key-value store. Pijul does fit that description though.
TBH I never understood why a package manager has to be tied to a particular implementation of version control or forges. The way I've done this in Elpe is by merging hashes, locks and update scripts under a single concept for example, and it works well.
eureka-cpu commentedon Nov 3, 2026
Was just looking at elpe, very cool project
xokdvium commentedon Nov 3, 2026
6AC3
It really doesn't, what matters of course is a way to efficiently and natively represent the content addressed store (not as in the Nix store - which stores everything as flat files) of the backend of the version control system.
P-E-Meunier commentedon Nov 4, 2026
Absolutely. I certainly understand how, my question was about the why.
My point being, if Git is the only choice, then at least do it all the way, like these nice folks at Basel University: https://www.ephraimsiegfried.ch/posts/nix-binary-cache-backed-by-git