Recently I've been collaborating with friends on Lyceum, an MMO game with an Erlang backend and a Zig + Raylib client (as if erlang wasn't a weird enough of a choice). Now, this is an unusual combination, but that's the whole reason our pesky group exists in the first place (if you want to know more check my friend's Lemos post).
There is also a couple of stadards we try to follow when doing this project, all of the team works with microservices all day in their normal jobs, so whenever we want to do something we try follow some simple rules:
- Can we develop all of the project parts locally? Preferably with no networking as well (besides pulling dependencies).
- Can we do so by leveraging a couple handy tools to their limit?
One can imagine that setting up such a development environment might be
nightmarish, but thankfully the 21st century brought us some interesting tools
that make Unix less of a mess to deal with, and yes, I'm talking about Nix. My
goal here is to show people what our development experience looks like and maybe
convince a few souls dealing with more normal tools (brew, asdf, <insert random linux package manager>, ...) to at least give Nix a try.
Devenv {#devenv}
We use devenv to setup our development shell, think of it as your favorite
programming language's envinroment and dependency manager (pip, poetry, nvm,
rvm, ...) but capable of installinng anything availiable on nixpkgs and
much more.
A unified development shell for Erlang and Zig {#a-unified-development-shell-for-erlang-and-zig}
No one is expected to have Erlang, Zig and Postgres installed, nor are they
expected to have any of the environment variables needed for this project to
work, the development shell already does all of that boring stuff for
you. Here's a snippet of what it lookups like:
# (...)
devShells = forAllSystems (
system:
let
pkgs = nixpkgs.legacyPackages.${system};
# Erlang shit
erlangLatest = pkgs.erlang_27;
erlangLibs = getErlangLibs erlangLatest;
# Zig shit
raylib = pkgs.raylib;
zigLatest = pkgs.zig;
linuxPkgs = with pkgs; [
inotify-tools
xorg.libX11
xorg.libXrandr
xorg.libXinerama
xorg.libXcursor
xorg.libXi
xorg.libXi
libGL
];
darwinPkgs = with pkgs.darwin.apple_sdk.frameworks; [
CoreFoundation
CoreServices
];
in
{
# (...) This will be shown in the next section
# `nix develop`
default = devenv.lib.mkShell {
inherit inputs pkgs;
modules = [
(
{ pkgs, lib, ... }:
{
packages =
with pkgs;
[
erlang-ls
erlfmt
just
rebar3
dbeaver-bin
]
++ lib.optionals stdenv.isLinux (linuxPkgs)
++ lib.optionals stdenv.isDarwin darwinPkgs;
languages.erlang = {
enable = true;
package = erlangLatest;
};
languages.zig = {
enable = true;
package = zigLatest;
};
env = mkEnvVars pkgs erlangLatest erlangLibs raylib;
scripts = {
build.exec = "just build";
server.exec = "just server";
};
enterShell = ''
echo "Starting Development Environment..."
just deps
'';
services.postgres = {
package = pkgs.postgresql_16.withPackages (p: with p; [ p.periods ]);
enable = true;
initialDatabases = [ { name = "mmo"; } ];
port = 5432;
listen_addresses = "127.0.0.1";
initialScript = ''
CREATE USER admin SUPERUSER;
ALTER USER admin PASSWORD 'admin';
GRANT ALL PRIVILEGES ON DATABASE mmo to admin;
'';
};
}
)
];
};
# (...)
Let's try building the Zig client:
$ just client-build
$ just client
Running Postgres {#running-postgres}
As you may have noticed, not only are we installing Erlang and Zig, some
madlad even put dbeaver there for God knows what reason, but hey, that's the dev
shell, just do whatever you want. We also have a local postgres setup and the
workflow mimics what you usually have with docker-compose or podman. By running:
devenv up
inside the shell, a local Postgres 16 with custom extensions will be
spinned. The list of services supported by devenv keeps growing and you can take
a you can check them here.
Direnv {#direnv}
As if thigs weren't awesome enough, I need to talk about direnv, a simple tool
that can make wonders (and it comes with nix integrations for free), with a
single .envrc in your project's repo you can jump inside a certain development
shell just by cd-ing into the project's directory. Here's an example of my
.envrc:
use flake . --impure
followed by a direnv allow in my shell:
$ direnv allow
direnv: loading ~/Code/Personal/lyceum/.envrc
direnv: using flake . --impure
direnv: nix-direnv: Renewed cache
Starting Development Environment...
rebar3 get-deps
===> Verifying dependencies...
rebar3 nix lock
===> Verifying dependencies...
# (...)
That's it. Now every time I cd <lyceum-directory>, I'll immediatly load the
whole development shell and be ready to work on it. This section is optional but
it really simplifies my.
The CI environment {#the-ci-environment}
Since we are already went to the trouble of setting up a whole dev environmet for Erlang and Zig, we should just make another one for when we need to run builds and testing suites on CI.
# `nix develop .#ci`
# reduce the number of packages to the bare minimum needed for CI
ci = pkgs.mkShell {
env = mkEnvVars pkgs erlangLatest erlangLibs raylib;
buildInputs = with pkgs; [
erlangLatest
heroku
just
rebar3
zigLatest
];
};
If you use Github Actions, now you can leverage both the Install Nix and Magic Nix Cache actions.
The full devshell {#the-full-devshell}
You can check what the full devshell looks like here.
Nix Build {#nix-build}
In the previous section I've showed you our impure environment, there's no way (as of now) to make things 100% pure while developing, specially because we need to have a postgres service running to debug and test locally. However, when we talk about releases, things change, we need to find a way to properly build the server.
A pure build of the Erlang server {#a-pure-build-of-the-erlang-server}
This is the original reason I've decided to write this, it took me some time to go through the NixOS BEAM manual and I've yet to know how to properly build this project with the buildRebar3 Tools (it seems it's used more inside Nixpkgs itself than to integrate with Erlang projects). Nevertheless, you can properly package this with the derivations Nix already gives you:
# Leverages nix to build the erlang backend release
# nix build .#server
server =
let
deps = import ./rebar-deps.nix { inherit (pkgs) fetchHex fetchFromGitHub fetchgit; };
in
pkgs.stdenv.mkDerivation {
name = "server";
version = "0.0.1";
src = pkgs.lib.cleanSource ./.;
buildInputs = with pkgs; [
erlangLatest
pkgs.stdenv.cc.cc.lib
rebar3
just
gnutar
];
nativeBuildInputs = with pkgs; [
autoPatchelfHook
coreutils
gawk
gnugrep
libz
ncurses
openssl
systemdLibs
];
buildPhase = ''
mkdir -p _checkouts
# https://github.com/NixOS/nix/issues/670#issuecomment-1211700127
export HOME=$(pwd)
${toString (
pkgs.lib.mapAttrsToList (k: v: ''
cp -R --no-preserve=mode ${v} _checkouts/${k}
'') deps
)}
just release-nix
'';
installPhase = ''
mkdir -p $out
mkdir -p $out/database
# Add migrations to the output as well, otherwise the server
# breaks at runtime.
cp -r database/migrations $out/database
tar -xzf _build/prod/rel/*/*.tar.gz -C $out/
'';
};
This is a derivation, a meta-package, a recipe containing every step and every
dependecy I need to satisfy and properly build our server. Now, as for the
deps.nix file, it was auto-generated with rebar3-nix, which itself has a rebar3
plugin. So everytime someone adds a BEAM dependency in our current flow, we
automatically generate a nix lockfile to match as well. Here's what we needed to
add in our rebar3 config to benefit from the Nix integration:
{plugins, [
{ rebar3_nix, ".*", {git, "https://github.com/erlang-nix/rebar3_nix.git", {tag, "v0.1.1"}}}
]}.
now let's see if this really works:
$ nix build .#server
# (...)
# We now have a `result` directory in the project's root...
$ ls ./result/
bin database erts-13.2.2.10 lib releases
# Now try running the server we've just build and...
$ ./result/bin/server foreground
Exec: /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/erts-13.2.2.10/bin/erlexec -noinput +Bd -boot /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/releases/0.0.1/start -mode embedded -boot_var SYSTEM_LIB_DIR /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/lib -config /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/releases/0.0.1/sys.config -args_file /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/releases/0.0.1/vm.args -- foreground
Root: /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server
/nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server
Connecting to: "127.0.0.1"
Connected to "127.0.0.1" with USER = "admin"
Finding migration scripts...
Migration Path: "/nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/database/migrations"
Running DB migrations.
Migrations completed successfully.
# (...) it works
Containers {#containers}
There is a treasure trove of examples in Nixpkgs, I've decided to go with the simplest one. This what a container for the backend looks like in Nix:
# nix build .#dockerImage
dockerImage = pkgs.dockerTools.buildLayeredImage {
name = "lyceum";
tag = "latest";
created = "now";
# This will copy the erlang release derivation from the
# previous step into to the image
contents = [ server pkgs.coreutils pkgs.gawk pkgs.gnugrep ];
config = {
Cmd = [
"${server}/bin/server"
"foreground"
];
Env = [
"ERL_DIST_PORT=8001"
];
ExposedPorts = {
"8080/tcp" = { };
};
};
};
It doesn't really look like most Dockerfiles you see around the net. Notice that
I'm using the server derivation from the previous step, the hard work required
to make it work the first time is immediatly rewarded because now we can keep
composing the previous solutions into more complex flows. To test this, let's
build the image:
$ nix build .#dockerImage
# Now load the build image in docker (or podman)
$ docker load < ./result
# Make sure you have `devenv up` running
$ docker container run --network=host --rm lyceum:latest
Exec: /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/erts-13.2.2.10/bin/erlexec -noinput +Bd -boot /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/releases/0.0.1/start -mode embedded -boot_var SYSTEM_LIB_DIR /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/lib -config /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/releases/0.0.1/sys.config -args_file /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/releases/0.0.1/vm.args -- foreground
Root: /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server
/nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server
server[1] Starting up
Connecting to: "127.0.0.1"
Connected to "127.0.0.1" with USER = "admin"
Finding migration scripts...
Migration Path: "/nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/database/migrations"
Running DB migrations.
Migrations completed successfully.
# (...)
Conclusion {#conclusion}
As I wanted to show here, we've used Nix all the way from defining a common development environment for the developers, to re-using some of the stuff in CI, to later repurpose some of the flows for pure builds, that later got repursed into our containers, all by leveraging the same tool. I wish modern devops was more about that, but it seems it'll take time for people to realize that immutability, composition and functional programming can go hand in hand and give us a better experience than one can find in most other solution (built by trillion dollar companies). Luckilly, Nix is gaining some traction and more people are talking about it.
I've been using it for the past 6 years in my workstations and don't regret doing so, its a tool worth learning (and there's still so much to learn about it), it makes my life dealing with Unix systems less painfull.
TODO
There is still much to do, and it can be left for a part II later.
- <input disabled="" type="checkbox"> I have yet to learn how to deploy a production-ready erlang system. Add (Cesarini and Vinoski 2016) to my readlist.
- <input disabled="" type="checkbox"> Properly build the client, it seems that non2nix breaks with the format for zon files, I'm not familiar with Zig toolig and ill take a look at this later
- <input disabled="" type="checkbox"> We are still unsure where to deploy, but I really want to move away from Heroku and check what Nix has to offer to manage a fleet of VMs.