Nix in a Cluster Environment

● ●

Kevin Cox https://kevincox.ca

● ●

Been using Nix for ~2 years. Have been running a personal cluster for almost that long. Working as an SRE at Google. Opinions expressed are my own, no one else’s.

Overview ● ● ● ●

What is a cluster good for? Why are clusters hard? How Nix makes clusters easier. Introduction to my personal cluster as an example implementation.

What I Want out of a Cluster ● Abstract individual machines. ● Keep track of resource usage. ● Isolate difference tasks/services.

Challenges in a cluster environment. ● ● ● ●

Base system consistency. Getting data where it needs to be. Base system still needs to be updated. You don’t know your neighbours. ○ You could even be a neighbour with a different version of yourself!

How Nix Helps ● Isolation ○ Nix packages depend only on other store paths. ○ Store paths are named based on hashes so they don’t conflict with each other.

● Deployment ○ Easy dependencies. ○ No naming issues.

More Nix Advantages ● Packages can be installed by any user. ● Packages are file/directory trees. ● Built in distribution infrastructure.

Putting it Together 1. 2. 3. 4.

Developer commits new version. Packages are built in CI and tested. Artifacts are uploaded to “binary cache”. On success new version is sent to the cluster scheduler.

Developer commits new version. ● Works just as you are used to. ● You can also use nix-env to give your developers an environment just like production.

Packages are built in CI and Tested ● This build “locks in” exact package versions. ○ I generally follow a nixpkgs release. ○ Could also pin to a specific version.

● Tests are run on exact set of dependencies. ● Results are uploaded to package server. ● If tests fail the exact packages can be fetched for debugging.

Service Updated ● Contains exact version information. ● Optionally contain config and data. ○ Recommended, depends on your use case.

● Your tooling should handle rollouts and rollbacks.

Questions?

My Cluster Always a work in progress, many options selected for simplicity.

● ● ● ● ●

Mesos/Marathon Semaphore CI B2 Cloud Storage Github and Bitbucket etcd + Cloudflare DNS DNS

Semaphore CI ● Free and hosted. ● Allows custom commands. ● Easy to switch.

Semaphore CI # (Heavily) simplified. See https://github.com/kevincox/nix-ci/blob/master/nix-ci.sh curl -fsS https://nixos.org/nix/install | bash # Build/test nix-build --fallback --show-trace # Upload nix-env -i /nix/store/jm3pazmq42x9lqz9dsvwrj16f74bwj2g-b2-nix-cache \ /nix/store/p5rdp7zlgfqryqnk80jjwnbc7b2f0x8v-backblaze-b2-0.6.2 b2-nix-cache $(cat secrets/b2-bucket) secrets/nix-cache-key # Deploy nix-build -A marathon -o result-marathon wget "${args[@]}" \ --method PUT \ --header 'Content-Type: application/json' --body-file result-marathon "$(cat secrets/marathon)/v2/apps"

Mesos/Marathon ● Provide cluster scheduling. ● Marathon has basic deployment tools. ● Marathon provides a nice API ○ Used for monitoring/alerting on task health. ○ Used for updating DNS to point at healthy tasks.

● Mesos is incredibly flexible.

Mesos/Marathon marathon = klib.marathon.config [{ id = "/api.dontsayit.kevincox.ca"; dns = [{ name = "api.dontsayit.kevincox.ca" }]; mem = 50; ports = 1; user = "api-dontsayit"; env-pass = [ "PORT0" ]; env-files = [ "/etc/dontsayit" ]; env = { RUST_BACKTRACE = "1"; DSI_WORDS = "${./data/words.csv}"; }; exec = '' export DSI_BIND=0.0.0.0:$PORT0 exec "${out}/bin/dontsayit-api" ''; healthChecks = [{ … }]; }];

Mesos/Marathon # Stage 1 args = [ "/run/current-system/sw/bin/sh" "-c" '' set -eaux . /etc/kevincox-environment nix-store -r ${stage2f} --add-root klib-marathon-stage-2 --indirect exec env -i ${env-pass} ${stage2f} "$@" '' "--" ]

Mesos/Marathon # Stage 2 stage2 = '' #! ${pkgs.dash}/bin/dash set -eaux ${ concatMapStringsSep "\n" (f: ". '${f}'") r.env-files } ${ env } exec "$@" # Not executed, included to make a dependency. ${concatStrings user-cmd} '';

Questions?