Getting Siril 1.2.0-beta1 to work in NixOS: A story of pain and suffering
I recently took up Astrophotography as a hobby. It burnt a huge hole in my wallet, but I managed to get a setup up and running.
Image processing is a critical part of Astrophotography. I don't want to bore with all the details, but you need image processing softwares to bring out all the details in the image you capture with your telescope.
There are many software for Windows which can be used. Adobe Photoshop, Pixinsight, and Deep Sky Stacker are some of them. As always, only a fraction of these are available for Linux - Siril being one of them.
Now Siril is extremely powerful and a breeze to work with. The version of Siril available in NixOS is 1.0.6, which works fine for most cases, but they recently released 1.2.0-beta1 with a truckload of new features.
"I gotta try it!" - Aniket exclaimed! After all, how hard can it be to modify the derivation to build 1.2.0-beta1 instead of 1.0.6? Turns out extremely hard!
NOTE: I'm a NixOS noob. So the things I'm doing may not be "correct." If you think there are easier ways, let me know in the comments.
The Start
The starting point is obviously the existing derivation in nixpkgs:
{ lib, stdenv, fetchFromGitLab, pkg-config, meson, ninja
, git, criterion, gtk3, libconfig, gnuplot, opencv, json-glib
, fftwFloat, cfitsio, gsl, exiv2, librtprocess, wcslib, ffmpeg
, libraw, libtiff, libpng, libjpeg, libheif, ffms, wrapGAppsHook
}:
stdenv.mkDerivation rec {
pname = "siril";
version = "1.0.6";
src = fetchFromGitLab {
owner = "free-astro";
repo = pname;
rev = version;
sha256 = "sha256-KFCA3fUMVFHmh1BdKed5/dkq0EeYcmoWec97WX9ZHUc=";
};
nativeBuildInputs = [
meson ninja pkg-config git criterion wrapGAppsHook
];
buildInputs = [
gtk3 cfitsio gsl exiv2 gnuplot opencv fftwFloat librtprocess wcslib
libconfig libraw libtiff libpng libjpeg libheif ffms ffmpeg json-glib
];
# Necessary because project uses default build dir for flatpaks/snaps
dontUseMesonConfigure = true;
configureScript = ''
${meson}/bin/meson --buildtype release nixbld .
'';
postConfigure = ''
cd nixbld
'';
meta = with lib; {
homepage = "https://www.siril.org/";
description = "Astrophotographic image processing tool";
license = licenses.gpl3Plus;
changelog = "https://gitlab.com/free-astro/siril/-/blob/HEAD/ChangeLog";
maintainers = with maintainers; [ hjones2199 ];
platforms = platforms.linux;
};
}I decided to create an overlay to modify the version. I'm not going into too many details about an overlay or how I have set up overlays in my system, as that is irrelevant here.
In most cases, you need to change the version and the SHASUM, and you should be ready. So I started with a basic overlay:
final: prev: {
siril = prev.siril.overrideAttrs (o: rec {
version = "1.2.0-beta1";
src = prev.fetchFromGitLab {
owner = "free-astro";
repo = o.pname;
rev = "f3611ada24853cb3870b604621b355d70d06719d";
sha256 = "";
};
});
}The rev argument tells nix which commit to build. From the GitLab repo, you can see that it is the release of 1.2.0-beta1. The sha256 argument is kept empty so that when you try to build it for the first time, it fails and outputs the actual sha256, which I can put there and rebuild. Here's the overlay after putting the sha256:
final: prev: {
siril = prev.siril.overrideAttrs (o: rec {
version = "1.2.0-beta1";
src = prev.fetchFromGitLab {
owner = "free-astro";
repo = o.pname;
rev = "f3611ada24853cb3870b604621b355d70d06719d";
sha256 = "hJNycHZtI9UAiA5s/5MVU2zsGWjkkTxOj2F/VadlzaM=";
};
});
}Now, this actually works if the build steps stay the same between the two versions. It turns out that from upgrading to 1.2.0-beta1, they added CMake as a build dependency to build the htmesh subproject. And my pain started here.
Pain 1: CMake
The build failed looking for CMake:
error: builder for '/nix/store/2wxcgrfbmrqx4cs1myvq1bqlpq6m7j2h-siril-1.2.0-beta1.drv' failed with exit code 1;
last 10 log lines:
>
> Executing subproject htmesh method cmake
>
> htmesh| Did not find CMake 'cmake'
> htmesh| Found CMake: NO
>
> meson.build:268:2: ERROR: Unable to find CMake
>
> A full log can be found at /build/source/nixbld/meson-logs/meson-log.txt
> WARNING: Running the setup command as `meson [options]` instead of `meson setup [options]` is ambiguous and deprecated.
For full logs, run 'nix log /nix/store/2wxcgrfbmrqx4cs1myvq1bqlpq6m7j2h-siril-1.2.0-beta1.drv'.This fix should be easy. I just need to add cmake to the nativeBuildInputs.
final: prev: {
siril = prev.siril.overrideAttrs (o: rec {
version = "1.2.0-beta1";
src = prev.fetchFromGitLab {
owner = "free-astro";
repo = o.pname;
rev = "f3611ada24853cb3870b604621b355d70d06719d";
sha256 = "hJNycHZtI9UAiA5s/5MVU2zsGWjkkTxOj2F/VadlzaM=";
};
nativeBuildInputs = o.nativeBuildInputs ++ [prev.cmake ];
});
}Here I take the original nativeBuildInputs and add cmake to it. This should be enough to make CMake available during the build. Well, turns out that it does more than what's needed.
Pain 2: CMake takes over
Nix has the concept of setup hooks. Most often, projects built with common build managers like autotools, make, cmake etc., have the same steps. For example, a project built with autotools is likely to be compiled with the following:
./configure
make
make installTo prevent having to write the same things over and over again, Nix provides setup hooks where adding the package to nativeBuildInputs automatically runs the appropriate commands during the build. For autotools, you simply need to include autoreconfHook in nativeBuildInputs. You can also see in the original derivation that simply including meson and ninja is enough, and there isn't any build step specified.
But now that CMake is added to nativeBuildInputs, its setup hooks take over and try to build the entire project with CMake and fail because there's no CMakeLists.txt:
error: builder for '/nix/store/rnrlz5dfbr1vgjm6q26yr6cg5xv0vcj2-siril-1.2.0-beta1.drv' failed with exit code 1;
last 10 log lines:
> fixing cmake files...
> cmake flags: -GNinja -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/share/locale -DCMAKE_INSTALL_LIBEXECDIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/lib -DCMAKE_INSTALL_DOCDIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/share/doc/siril -DCMAKE_INSTALL_INFODIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/share/info -DCMAKE_INSTALL_MANDIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/share/man -DCMAKE_INSTALL_OLDINCLUDEDIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/include -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/include -DCMAKE_INSTALL_SBINDIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/sbin -DCMAKE_INSTALL_BINDIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_OSX_SYSROOT= -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/store/bfbp3ypd9nm3fapz634gvvs738blrl0y-gcc-wrapper-12.2.0/bin/strip -DCMAKE_RANLIB=/nix/store/bfbp3ypd9nm3fapz634gvvs738blrl0y-gcc-wrapper-12.2.0/bin/ranlib -DCMAKE_AR=/nix/store/bfbp3ypd9nm3fapz634gvvs738blrl0y-gcc-wrapper-12.2.0/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=/nix/store/szl107hvmkzrhar8l6fsmm8v9vid4gav-siril-1.2.0-beta1
> CMake Warning:
> Ignoring extra path from command line:
>
> ".."
>
>
> CMake Error: The source directory "/build/source" does not appear to contain CMakeLists.txt.
> Specify --help for usage, or press the help button on the CMake GUI.
For full logs, run 'nix log /nix/store/rnrlz5dfbr1vgjm6q26yr6cg5xv0vcj2-siril-1.2.0-beta1.drv'.The fix was to add dontUseCmakeConfigure = true; to the overlay. This skips running CMake.
final: prev: {
siril = prev.siril.overrideAttrs (o: rec {
version = "1.2.0-beta1";
src = prev.fetchFromGitLab {
owner = "free-astro";
repo = o.pname;
rev = "f3611ada24853cb3870b604621b355d70d06719d";
sha256 = "hJNycHZtI9UAiA5s/5MVU2zsGWjkkTxOj2F/VadlzaM=";
};
nativeBuildInputs = o.nativeBuildInputs ++ [prev.cmake ];
dontUseCmakeConfigure = true;
});
}Pain 3: sleef.h
The build now failed with a new error:
FAILED: src/libsiril.a.p/rt_rt_algo.cc.o
g++ -Isrc/libsiril.a.p -Isrc -I../src -Isubprojects/htmesh/__CMake_build -I../subprojects/htmesh/__CMake_build -Isubprojects>
../src/rt/rt_algo.cc:34:10: fatal error: sleef.h: No such file or directory
34 | #include "sleef.h"
| ^~~~~~~~~
compilation terminated.The error is in src/rt_algo.c. Looking at the previous version, there used to be a core/sleef.h which has now been removed, and instead, sleef.h from librtprocess is used. Now, librtprocess is a subproject, and if it isn't available, it'll be built automatically, Looking back at the original derivation, librtprocess is being included in buildInputs. Good, so we just need to remove it.
final: prev: {
siril = prev.siril.overrideAttrs (o: rec {
version = "1.2.0-beta1";
src = prev.fetchFromGitLab {
owner = "free-astro";
repo = o.pname;
rev = "f3611ada24853cb3870b604621b355d70d06719d";
sha256 = "hJNycHZtI9UAiA5s/5MVU2zsGWjkkTxOj2F/VadlzaM=";
};
nativeBuildInputs = o.nativeBuildInputs ++ [prev.cmake ];
dontUseCmakeConfigure = true;
buildInputs = prev.lib.lists.remove prev.librtprocess o.buildInputs;
});
}I'm not sure if this is the best way to remove a buildInput. Let me know in the comments.
Now librtprocess should be built, but ...
Pain 4: Git Submodule
This time I was faced with a new error:
error: builder for '/nix/store/99xfym7x88qagqjiza5f2s0vyd13rvig-siril-1.2.0-beta1.drv' failed with exit code 1;
last 10 log lines:
> kplot| Subproject kplot finished.
>
> Dependency kplot from subproject subprojects/kplot found: YES 0.1.14
> Run-time dependency opencv4 found: YES 4.7.0
> Run-time dependency rtprocess found: NO (tried pkgconfig and cmake)
>
> meson.build:251:2: ERROR: Subproject exists but has no CMakeLists.txt file
>
> A full log can be found at /build/source/nixbld/meson-logs/meson-log.txt
> WARNING: Running the setup command as `meson [options]` instead of `meson setup [options]` is ambiguous and deprecated.
For full logs, run 'nix log /nix/store/99xfym7x88qagqjiza5f2s0vyd13rvig-siril-1.2.0-beta1.drv'.This one is easy to understand. The subprojects/librtprocess directory is a Git submodule, which isn't fetched by default. The solution is to add fetchSubmodules = true; in fetchFromGitLab.
final: prev: {
siril = prev.siril.overrideAttrs (o: rec {
version = "1.2.0-beta1";
src = prev.fetchFromGitLab {
owner = "free-astro";
repo = o.pname;
rev = "f3611ada24853cb3870b604621b355d70d06719d";
sha256 = "hJNycHZtI9UAiA5s/5MVU2zsGWjkkTxOj2F/VadlzaM=";
fetchSubmodules = true;
};
nativeBuildInputs = o.nativeBuildInputs ++ [prev.cmake ];
dontUseCmakeConfigure = true;
buildInputs = prev.lib.lists.remove prev.librtprocess o.buildInputs;
});
}Adding that line will still reuse the previously downloaded source from the cache. To force Nix to redownload, you need to invalidate the hash. The way I do it is to remove the sha256 argument and let it fail. Then put it back and rebuild.
Pain 5: Starnet++
At this point, Siril builds and works fine. But I wasn't done yet. I mainly wanted to try 1.2.0-beta1 to use the Starnet++ integration.
To put things simply, Starnet++ uses a neural network to remove stars from a picture. It's handy when you want to work on a faint object such as a nebula so that you can bring out the details in the nebula without over-exposing the stars. usually, you remove the stars, work on the starless picture, and then put the stars back.
Starnet provides a CLI tool for Linux. Downloading it gives you a starnet++ binary, a few libraries, and the neural net weights.
The starnet++ executable (well, it isn't executable yet, and needs to be chmod-ed) is dynamically linked to the libraries and must be in the same directory. It's not a big deal, as you need to put the directory containing starnet++ in Siril, and it does the rest. But I should probably run it and make sure it works, right?
Running ./starnet++ throws an error:
exec: Failed to execute process './starnet++': The file exists and is executable. Check the interpreter or linker?This is expected. NixOS has a different filesystem structure; whatever it was compiled/linked against doesn't exist now.
$ file ./starnet++
./starnet++: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=be43e27d0517f13391054abd07d685f09d6b4527, for GNU/Linux 3.2.0, not stripped/lib64/ld-linux-x86-64.so.2 does not exist in NixOS. Thankfully, we have a solution: steam-run. No, it has nothing to do with Steam. steam-run wraps many standard libraries and provides a compatibility layer for running executables compiled in regular Linux. I started a shell with steam-run installed.
nix-shell -p steam-runThen it's as easy as running the executable with steam-run
steam-run ./starnet++And it works!
StarNet++ v2.0
StarNet++ removes stars from images using neural network.
Only RGB/Greyscale STRETCHED tif files with 16bit per channel
are supported by this executable.
Modes of usage:
starnet++ INPUT
starnet++ INPUT OUTPUT
starnet++ INPUT OUTPUT STRIDE
Stride is set to 256 if not specified
OUTPUT is set to 'starless.tif' if not specified
Example:
starnet++.exe mysuperimage.tif mystarlesssuperimage.tif 256
Dependencies:
Make sure that libtensorflow.so.2, libtensorflow_framework.so.2
and starnet2_weights.pb are in the same directory as starnet++
or somewhere in $PATH
More info:
https://www.astrobin.com/users/nekitmm/
(c) 2022 Nikita Misiura
So, I moved the starnet++ executable to starnet++-orig and created a bash file starnet++ which simply wraps the starnet++ executable with steam-run:
#!/usr/bin/env nix-shell
#!nix-shell -i bash -p steam-run
exec steam-run ./starnet++-orig "$@"
Here, nix-shell is used as a shebang interpreter to launch a bash shell with steam-run and then starnet++ is launched.
Great! I ran it, and it works fine. Let's try it from Siril. So I launched Siril and put the directory containing starnet++ in the settings and went to launch my first star removal.
Pain 6: nixpkgs
Launching starnet++ from Siril threw an error:
23:21:39: Error: external command starnet++ failed...
23:21:39: Error: StarNet did not execute correctly...I launched Siril from the console and looked for the error:
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
at «string»:1:25:
1| {...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (steam-run) ]; } ""
Turns out that, for some reason nixpkgs is not being found when nix-shellis being launched. I'm unsure why; it works fine when I launch it manually. Probably because of the way Siril launches Starnet (more on this later).
After a bunch of Googling, I changed the starnet++ script to explicitly add a reference to nixpkgs:
#!/usr/bin/env nix-shell
#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-21.11.tar.gz -i bash -p steam-run
exec steam-run ./starnet++-orig "$@"Again, this works when launched from the terminal. Let's try it within Siril.
Pain 7: Unfree
Again, it didn't work within Siril. This time with a new error:
error: Package ‘steam-runtime-0.20211102.0’ in /nix/store/9ks8076lanzwp09dmwy5r3racihsc1ip-source/pkgs/games/steam/runtime.nix:32 has an unfree license (‘unfreeRedistributable’), refusing to evaluate.
a) To temporarily allow unfree packages, you can use an environment variable
for a single invocation of the nix tools.
$ export NIXPKGS_ALLOW_UNFREE=1
Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
(Flake) command, `--impure` must be passed in order to read this
environment variable.
b) For `nixos-rebuild` you can set
{ nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.
Alternatively you can configure a predicate to allow specific packages:
{ nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
"steam-runtime"
];
}
c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
{ allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.
It seems like the nixpkgs I added is not being evaluated because of unfree software. At this point, I spent almost one hour banging my head. I already have nixpkgs.config.allowUnfree = true in my configuration, and allowUnfree = true; in ~/.config/nixpkgs/config.nix.
After many desperate attempts, I found it easier to modify the source code (haha). Looking through the source code of Siril, this is where Starnet is launched. They are using execve to launch starnet, which is fantastic. Because the third argument of execve can be used to pass environment variables.
So I created a patch file patch-unfree.patch:
diff --git a/src/filters/starnet.c b/src/filters/starnet.c
index 8b04ebe6..f1e6512c 100644
--- a/src/filters/starnet.c
+++ b/src/filters/starnet.c
@@ -81,7 +81,10 @@ static int exec_prog(const char **argv) {
int pipe_fds[2];
int n;
char buf[0x100] = {0};
-
+ char *envp[] = {
+ "NIXPKGS_ALLOW_UNFREE=1",
+ 0
+ };
if (pipe(pipe_fds)) {
perror("pipe creation error");
forkerrors = 1;
@@ -93,7 +96,7 @@ static int exec_prog(const char **argv) {
close(pipe_fds[0]);
dup2(pipe_fds[1], 1);
fprintf(stdout, "pid:%d\n",child_pid);
- if (-1 == execve(argv[0], (char **)argv , NULL)) {
+ if (-1 == execve(argv[0], (char **)argv , envp)) {
perror("child process execve failed");
forkerrors = 1;
return 0;
What is it doing? First, it creates an array of strings:
char *envp[] = {
"NIXPKGS_ALLOW_UNFREE=1",
0
};Then, envp is passed to execve. Finally, this patch needs to be applied in the overlay:
final: prev: {
siril = prev.siril.overrideAttrs (o: rec {
version = "1.2.0-beta1";
src = prev.fetchFromGitLab {
owner = "free-astro";
repo = o.pname;
rev = "f3611ada24853cb3870b604621b355d70d06719d";
sha256 = "UOtm58Jf/cbjjaw2CIVjrQ3IbCiR3rMxFAEXXrW5mHs=";
fetchSubmodules = true;
};
nativeBuildInputs = o.nativeBuildInputs ++ [prev.cmake ];
dontUseCmakeConfigure = true;
buildInputs = prev.lib.lists.remove prev.librtprocess o.buildInputs;
patches = [ ./patch-unfree.patch ];
});
}I'm pretty sure there are other "correct" ways of doing this. Let me know in the comments below.
And VOILA! It works! Finally! That wasn't time-consuming at all. Only five hours of my time :)
