Typhon

Typhon is a Nix-based continuous integration software inspired by Hydra.

Prerequisites

This book assumes that you are familiar with Nix and Nix flakes. Being familiar with Hydra is not necessary.

Disclaimer

Typhon is still in early development and is merely a proof of concept. A lot of core features are still missing and it is full of bugs. Please do not use it for any serious purpose.

Concepts

Note

Core concepts are described with a flake workflow in mind. But Typhon also supports a more traditional workflow, see at the end of this section for details.

Projects are the central abstraction of Typhon. A project typically corresponds to an under CI repository. Projects define jobsets, which in turn spawn jobs. Jobsets typically correspond to branches of the repository. They are evaluated periodically, typically on push events. These evaluations produce the Nix jobs associated with a commit.

On top of these concepts, taken from Hydra, Typhon adds actions. Actions are user-defined scripts, triggered by Typhon on certain occasions. They can have different purposes, like triggering evaluations, creating new jobsets, setting statuses or deploying something.

Projects

Projects are defined declaratively. This means that almost no configuration is made in Typhon, everything is done externally via a Nix flake. Concretely, a project is defined by a flake URL. The referenced flake must expose an output typhonProject defining the project settings.

typhonProject contains two attributes: meta and actions. meta is an attribute set which defines metadata about the project: a title, a description and a homepage. actions is an attribute set of derivations that build actions for the project and holds encrypted secrets for use by the actions.

A project typically configures CI for a repository, but the declaration can exist in a separate repository. In fact, the declaration of a project is quite sensitive since it defines the way the project's unencrypted secrets are handled. Malicious edits to the declaration can potentially leak these secrets.

Jobsets

A jobset is also a flake URL, referencing a flake that exposes an output typhonJobs. typhonJobs is an attribute set of derivations, called jobs, that are built by Typhon. Jobsets typically correspond to the branches of the repository. Their flake URL is locked periodically, creating an evaluation.

Jobsets updates and evaluations are meant to be triggered automatically by the webhook action.

Evaluations

An evaluation locks the flake URL of a jobset. It typically corresponds to a commit on the repository. Once the jobset is locked, the output typhonJobs is evaluated and the corresponding jobs are spawned.

Jobs

Jobs are the result of an evaluation, there is one for each derivation defined in the jobset. A job run consists of the build of the derivation and the execution of two actions, one at the beginning and one at the end. These actions are typically used to set statuses on the commit or to do deployment.

Actions

Actions are scripts run by Typhon in isolation from the system, but connected to the internet. They play different roles in Typhon. At the moment there are four actions a project can define:

The jobsets action is responsible for declaring the jobsets of a project. It is triggered periodically by the webhook action, typically when a branch is created on the repository.
The begin and end actions are run at the beginning and end of all jobs of your project. They are typically used to set statuses on your repository, but can also be used for deployment.
The webhook action is triggered by calls to a specific endpoint of the API. It outputs commands for Typhon to update or evaluate jobsets. It is meant to trigger jobs automatically.

Actions can also expose a secrets file. This is an age encrypted JSON file that typically contains tokens for the actions. It must be encrypted with the project's public key and is decrypted at runtime and passed as input to the actions.

Thanks to the use of actions, Typhon is forge-agnostic: it has no code specific to any forge. Instead, it is the actions' job to plug Typhon to the user's workflow. The actions can be built using the Nix library that comes with Typhon.

Legacy mode

In legacy mode, flake URLs are still used to declare projects and jobsets, but the underlying expressions do not need to be flakes. Instead of the output typhonProject, a legacy project must expose the expression nix/typhon.nix, that will produce the same content as typhonProject. Similarly, a legacy jobset must expose nix/jobs.nix instead of typhonJobs. These expressions are functions called without any arguments, and must evaluate purely.

Installation

Nix requirements

Typhon requires Nix >= 2.18 with experimental features "nix-command" and "flakes" enabled.

NixOS

At the moment the preferred way to install Typhon is on NixOS via the exposed module.

Example

Here is a sample NixOS module that deploys a Typhon instance:

{ pkgs, ... }:

let typhon = builtins.getFlake "github:typhon-ci/typhon";
in {
  imports = [ typhon.nixosModules.default ];

  # enable experimental features
  nix.settings.experimental-features = [ "nix-command" "flakes" ];

  # install Nix >= 2.18 if necessary
  nix.package = pkgs.nixVersions.nix_2_18;

  # enable Typhon
  services.typhon = {
    enable = true;

    # path to the argon2id hash of the admin password
    # $ SALT=$(cat /dev/urandom | head -c 16 | base64)
    # $ echo -n password | argon2 "$SALT" -id -e > /etc/secrets/password.txt
    hashedPasswordFile = "/etc/secrets/password.txt";
  };

  # configure nginx
  services.nginx = {
    enable = true;
    forceSSL = true;
    enableACME = true;
    virtualHosts."example.com" = {
      locations."/" = {
        proxyPass = "http://localhost:3000";
        recommendedProxySettings = true;
      };
    };
  };
}

Options

Here is a list of options exposed by the NixOS module.

Mandatory:

services.typhon.enable: a boolean to activate the Typhon instance.
services.typhon.hashedPasswordFile or services.typhon.hashedPassword: the Argon2id hash of the admin password in the PHC string format.

Optional:

services.typhon.home: a string containing the home directory of the Typhon instance.
services.typhon.package: a derivation to override the package used for the Typhon instance.

Usage

This section gives an example of how to use Typhon with a GitHub project. Let's assume your username is $user and you have two repositories, github.com/$user/$project and github.com/$user/$config. $project is the repository you want to put under CI, $config is going to contain the Typhon declaration. These two repositories can actually be the same, but separating the two can mitigate security concerns. Finally, let's assume your Typhon instance URL is $typhon_url (you must have https enabled).

Creating a new Typhon project

Log in to your Typhon instance and create a new project, with an identifier $id (typically $id == $project). Set the declaration to use the flake URL github:$user/$config. Once the project is created, a public key is associated to it, let's call it $pk.

GitHub settings

We need to generate a token on GitHub and make sure it has permission to update statuses on $project, let's call it $token. Then let's generate a random string $secret and add a webhook to $project with the following settings:

payload URL: $typhon_url/api/projects/$id/webhook
content type: application/json
secret: $secret
events: Just the push event

The configuration flake

Let's create a flake in the $config repository, then add an output typhonProject. We are going to import typhon as a flake input and use the github.mkProject helper function from the library:

{
  inputs = {typhon.url = "github:typhon-ci/typhon";};

  outputs = {
    self,
    typhon,
  }: {
    typhonProject = typhon.lib.github.mkProject {
      owner = "$user";
      repo = "$project";
      secrets = ./secrets.age;
      typhonUrl = "$typhon_url";
    };
  };
}

We need to generate the secrets.age file. First let's write a secrets.json file containing the secrets you generated (don't commit it!):

{
  "github_token": "$token",
  "github_webhook_secret": "$secret"
}

Then, we encrypt the JSON file with age, using the public key of the project:

nix run nixpkgs#age -- --encrypt -r "$pk" -o secrets.age secrets.json

We also need to generate the lock file:

nix flake lock

Finally, we commit secrets.age, flake.nix and flake.lock.

The project flake

In the $project repository, we create a flake with a typhonJobs attribute. For instance, let's declare GNU hello as your only job:

{
  inputs = {nixpkgs.url = "nixpkgs";};

  outputs = {
    self,
    nixpkgs,
  }: let
    system = "x86_64-linux";
    pkgs = nixpkgs.legacyPackages.${system};
  in {
    typhonJobs.${system} = {
      inherit (pkgs) hello;
    };
  };
}

We need to generate a lock file and commit flake.nix and flake.lock.

Refreshing the project declaration

Let's go to your project's page on Typhon and refresh the declaration. This is not done automatically on purpose. Always be careful before refreshing: if a malicious commit was made on $config, your secrets could be compromised. Once this is done, your Typhon project is using the settings declared in $config.

Verifying everything is working

We can now update the jobsets of your project from the project interface. A list of jobsets should appear, one for each branch of your repository. Now, any push to the repository should generate an evaluation in the corresponding jobset and statuses should appear on your repository.

Deployment

Now, let's add a deployment action to push your store paths to Cachix. We will assume you have a cache named $cache already set up and an authentication token that comes with it. First add the token to your secrets as an attribute called cachix_token. Then edit $config as follows:

typhonProject = typhon.lib.github.mkProject {
  deploy = [
    {
      name = "Push to Cachix";
      value = typhon.lib.cachix.mkPush {name = "$cache";};
    }
  ];
  ...
};

Refresh your project on Typhon, then your binaries should be pushed to your cache at the end of every job.

Finally, you can use typhon.lib.compose.match to run your deployments only on certain jobsets or jobs.

Hacking

Typhon is written in Rust. It consists of four packages:

typhon-core is the core logic of Typhon
typhon-webapp is the frontend application
typhon-types is a common library shared between the two
typhon is the server and the main package

Development environment

This documentation assumes that you are using Nix, so you can simply run nix-shell at the root of the project to enter the development environment. Experimental features "nix-command" and "flakes" need to be enabled in your Nix configuration for the server to run properly. Nix >= 2.18 is also required but it is provided by the Nix shell.

The following instructions assume that you are inside the Nix development environment.

Dependencies

Typhon uses Actix for the web server and Diesel for the database management. The webapp is written with Leptos. Typhon is built with cargo-leptos.

Building

If you are building Typhon for the first time, first go to typhon-webapp/assets and run npm install.

Then, to build Typhon, go to the root of the project and run:

build

Testing

To run Typhon, create /nix/var/nix/gcroots/typhon/ and make sure that you have write access to the directory. Then go to the root of the project and run:

serve

The server will be available at http://localhost:3000, the admin password is password.

You can also run watch to re-compile the server automatically at each modification of the code.

Formatting

Before submitting changes to Typhon, be sure to format the code using the format command.

Typhon documentation