Nix and remote building a single system package

Today I had a rather unusual issue with my nixos setup. I tried to rebuild the system for one of my hosts, lets call it “H”. The system rebuild started a build from source for the waybar package, which is C++ application. Because H is a rather old host and I don't want to have a hot lap, I figured that this package should be built remotely.

But there's an issue. It is configured via home-manager deep inside of the derivation of the host! Also, I don't want to rebuild my whole host on that remote builder. I could, but there's another issue: All my other hosts are currently turned off and I am sitting on the couch. I don't want to get up, because I am a lazy fellow.

Luckily, I have access to the amazing nix-community.org builders (or one of them rather).

But ... I don't want to rebuild my system on a host that I share with others, that's essentially public. That's because I bake stuff into my system that's private (don't ask).

So here was the challenge: Rebuild one package from one of my nixosConfigurations on a remote host. Not baking in the host as a remote builder, of course.

Finding the attribute

Finding the right attribute is rather simple, actually. After some digging with nix repl, I found the right one:

nixosConfigurations.$(hostname).config.home-manager.users.$(whoami).programs.waybar.package

To break it down:

  1. We access the current host within the nixosConfigurations attribute set – easy
  2. We access the config of that host, where we go into the home-manager config bits
  3. We access the right user
  4. ...where the “waybar” program is configured and this one exposes a package attribute that points to the package in use

That's the one we want to build.

If we wouldn't need another host, that would be the commandline now:

nix build .#nixosConfigurations.$(hostname).config.home-manager.users.$(whoami).programs.waybar.package

Doing the thing remotely

But of course we want to --max-jobs 0 here, so that the local host does not build anything.

So what do we do? Well, there's the --builders attribute, that can be used to tell nix about additional builders it may use for this single invokation. According to the documentation we can pass something like

  --builders 'ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd'

for example, to give it two additional builders. So for the problem at hand that would be --builders "ssh://build01.nix-community.org" – but that's not nearly enough.

See, the nix daemon runs as root. Even though I have a mapping from build01.nix-community.org to the right user and SSH identity in my .ssh/config, root does not know anything about that. So we must tell the nix invokation which user it should use to try to log into that sweet builder. Besides that, we must also tell it which SSH key to use (because again: nix runs as root, it doesn't know about my users ssh-agent instance, so it cannot query it for the appropriate ssh key and use that).

Luckily, we can also pass that. I did not fully figure out whether the ?ssh-key=... setting in the URL did the trick, or the setting within the --builders argument... but here's the thing: You can pass more options to the --builders option! You can pass platform identifiers to tell nix “hey, this machine can build aarch64” for example. You can tell nix about the relative speed of the machine (useful for large builds and if you have multiple machines that you pass), you can tell it the maximum number of builds it can execute on there.

All that is really nice, but not relevant for us with the rather small (although too much for my notebook) “waybar” package. Luckily, passing - instead of a real value uses the defaults.

Last but not least, we must tell nix about the host key identification of the host we're connecting to. Again, you can probably set this via programs.ssh.knownHosts in your nixos configuration, but the whole point here was that I don't want to rebuild my system before doing the package rebuild. Extracting the host key was a strange endeavour, because nix wants you to base64 that information, but not only the whole line from the ~/.ssh/known_hosts, but only the last two bits of it.

After I figured the whole thing out (with a bit of help from the nixos matrix channel – thanks for that folks and especially to the user named n4ch723hr3r!) – my commandline looked like this:

nix build -L .#nixosConfigurations.$(hostname).config.home-manager.users.$(whoami).programs.waybar.package --max-jobs 0 --builders "ssh://$NIX_COMMUNITY_USER@build01.nix-community.org?ssh-key=$NIX_COMMUNITY_KEY_PATH - $NIX_COMMUNITY_KEY_PATH - - - - $(cat ~/.ssh/known_hosts | grep nix-community | awk '{print $2,$3}' | base64 -w0)"

and I was able to build the package on the remote host.