NixCluster

Simple framework to manage, control and orchestrate multiple Linux devices to run parallelized programs. Heavily Nixed.

How it works:

1. Clients connect to the server and wait for a new task.
2. User sends an URL to a Nix Flake to the server. This Flake contains two programs:

• Server-side - prepares tasks for the clients and sends them to the daemon.
• Client-side - the task itself, compiled for client's architecture and with client's own input.

3. Server builds server-side part of the Flake and runs it.
4. Server-side part generates input for client-side part of the Flake and sends it to the daemon.
5. Daemon prepares tasks using generated inputs and sends them to free clients as Flake URL + input + architecture config.
6. Clients receive the tasks and builds them. (Using the server as a binary cache so the server can help the clients)
7. Clients run tasks, get the results from them, and send them back to the server.
8. Clients wait for the next task to run.

Task Nix Flake

{
  packages.${system} = {
    server = derivation; # Server-side part of the task
    client = derivation; # Client-side part taking additional arguments as task input
  };
}

Framework takes the generated input, puts it raw into .nix file and runs:

packages.${system}.client.override ((import ./taskInput.nix pkgs) // (import ./clientConfig.nix))

Input generated by server-side should be a nix function taking pkgs as an argument.

In reality it does not import this .nix file this way, but uses it as a non-flakey flake input.

Example flake:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs";
    flake-parts.url = "github:hercules-ci/flake-parts";
  };

  outputs = inputs @ {flake-parts, ...}:
    flake-parts.lib.mkFlake {inherit inputs;} {
      systems = [ "x86_64-linux" "aarch64-linux" ];

      perSystem = {pkgs, ...}: {
        packages = {
          server = pkgs.callPackage ./server.nix {}; # Nothing weird here
          client = pkgs.callPackage (
            {stdenv, foobar, clientConfig, ...}:
              stdenv.mkDerivation {
                pname = "client";
                version = "1.0.0";

                src = ./.;

                CLIENT_FLAGS = clientConfig; # -march flag

                FOOBAR = foobar; # puts the server-generated input
                                 # into the FOOBAR environment variable during the building
              };
          ) {};
        };
      };
    };
}

Server would generate the input in the form of

pkgs: {
  foobar = "123";
}

And then FOOBAR env var would have a value of 123 during the building