Build a Haskell project alongside its Rust dependency

Introduction
Objective
Build using Cargo from Cabal
Build using Haskell.nix and Nix Flakes
What about other existing Rust libraries?
Similar techniques
Portability
Reusability
Conclusion

Introduction

This post presents a way of building both a Haskell project and its required Rust dependency library or code automatically. There are several reasons for this:

There is an existing Rust library that does exactly what we want and there is no Haskell equivalent (either less advanced or non-existent)
The Rust library can do exactly what we want within our constraints (e.g. technical or performance constraints) while it would be more cumbersome to achieve with Haskell only
We may find some logic easier to code in Rust than Haskell

Whatever the reasons, they tend to be related to making the most of both languages and ecosystem.

Everything here is the result of a personal experiment playing with Haskell and Rust code. This post focuses on the essential points with some example code. The full code is available in the dedicated GitHub repository.

Note that how to call Rust from Haskell code (e.g. through Foreign Function Interface or FFI), is a subject on its own and is out of scope of this post. There is already plenty of documentation on that out there. This post also does not apply for projects for which the Rust library binaries are already and systematically available. Those use cases are much simpler as we don’t need to concern ourselves about how to compile the foreign dependencies. It is still on us however to provide access to them to the compiling GHC.

Objective

Our objective is to easily mix Haskell and Rust code within the same project.

Some issues that we need to overcome are:

Rust compilation. What if we want to build the Haskell project on a machine without the Rust library’s binary? We would need to retrieve it and make it available to GHC for linking somehow. What if the binary is unavailable? We would need to compile the Rust code with an available Rust toolchain.
Portability. How can we guarantee that the Rust library’s binary is usable by the Haskell project? How can we guarantee that compilation will succeed on our target platforms?
Reusability. What if we want to use the Haskell project in yet another Haskell project? If we want to distribute the project on Hackage, we also need to make sure that the client project depending on our Haskell project works properly.

One way to solve these issues is to have our Haskell toolchain automatically trigger the compilation of Rust code using the available Rust toolchain and use the generated binaries to build the Haskell project. To achieve this, we will customize our Cabal project’s setup tasks using hooks.

Build using Cargo from Cabal

Refresher

As a building and packaging system for Haskell libraries and programs, Cabal uses configuration files to track dependencies between projects and compatible versions in a declarative way. The ultimate goal is to provide reproducible builds by resolving Haskell dependencies and installing them automatically. External dependencies are also listed for use, though the user is responsible for installing them on the building machine. It is possible to build portable packages on different platforms (OS and CPU), so long as all their dependencies are available or can be built on those.

Cabal can also make use of custom Setup.hs files to change various aspects of the build process through hooks. This may however affect our project’s portability to different platforms if we’re not careful (e.g. non-portable code).

On the other side, we have Cargo as Rust’s build system and package manager. It can download the dependencies of a Rust project and build it entirely using a simple command line tool.

Objective

We will make use of a custom Setup.hs file to:

Build the Rust code using cargo during the initial phases of our Haskell package build
Add the built Rust library to the Haskell package configuration for GHC to find
Clean the built Rust code when the package is cleaned

Note though that the clean logic (3) will only be executed when running cabal v1-clean.

In the sections, we will assume that Cabal and Cargo are already installed.

Configuring our project

Our project’s folder hierarchy must contain both the Haskell code and Rust code. We will use the following layout:

.
├── LICENSE
├── README.md
├── Setup.hs (1)
├── hello.cabal (3)
├── rust (2)
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── src
│   │   └── lib.rs
└── src
    └── Main.hs

This is the hierarchy for a standard Haskell project, though we have included our custom Setup.hs file (1) and the Rust code in a rust folder (2) for easy access and explicit Haskell dependence on the Rust code. The Rust code can actually be located anywhere, as long as the hooks in Setup.hs (1) can find it.

The Cabal configuration file (3) will look like the following:

cabal-version: 3.0
name: hello
version: 1.0.0.0
license-file: LICENSE

-- (1)
build-type: Custom

-- (2)
custom-setup
  setup-depends:
      base >= 4.7 && < 5
    , Cabal >= 3.0.0.0
    , directory >= 1.3.6.0
    , filepath >= 1.3.0.2

executable hello
  hs-source-dirs: src
  main-is: Main.hs
  build-depends: base >= 4.14 && < 5
  -- (3)
  extra-libraries: rs_hello_world
  default-language: Haskell2010

In this configuration, we tell Cabal that we will be using a customized setup (1) found in Setup.hs. The setup module has its own dependencies (2) which differs from the project’s dependencies. Also, we will explicitly mention the Rust library (3) within our project.

It is entirely possible to not mention the Rust library in the Cabal configuration file and have the custom setup module add it programmatically when setting up the project. However, explicitly declaring the Rust dependency this way is preferable for several reasons:

No hidden dependencies. Any developer will have a clearer picture of the project by reading its configuration file only.
Use system dependency as fallback. GHC can use the existing libraries in the system without changes if we remove the custom Setup. We will exploit this in the second part of the post.

Once that is done, we can instruct Cabal how to build our project with a custom Setup.

Creating our custom Setup

The Setup module is just like any regular main Haskell module. It is called by Cabal to handle most of the project’s build process. We usually don’t need anything more than the default build process. In that case, we could use the build-type: Simple Cabal property and omit the Setup.hs file entirely (which is the same as defining it with a call to defaultMain). But as we want to customize the build process, we can actually use the defaultMainWithHooks function which basically does the same as the default build process but allows us to place hooks on various actions of the build to customize them:

main :: IO ()
main = defaultMainWithHooks hooks
  where
    hooks =
        simpleUserHooks
            { preConf = \_ flags -> do -- (1)
                rsMake (fromFlag $ configVerbosity flags)
                pure emptyHookedBuildInfo
            , confHook = \a flags -> -- (2)
                confHook simpleUserHooks a flags
                    >>= rsAddDirs
            , postClean = \_ flags _ _ -> -- (3)
                rsClean (fromFlag $ cleanVerbosity flags)
            }

So we have placed the following hooks:

Before proceeding to the Haskell project’s configuration (1), we will actually build the Rust library. In doing so, we will have all the information we will need to properly configure the Haskell project for our building machine. We will also make sure that if the Rust library fails to compile, the Haskell project’s compilation stops.
During the actual configuration of the build (2), we will indicate the path to the built Rust library to Cabal. In order to do that, we will call the default Cabal implementation for the hook which will return the project’s default Cabal configuration. We will then change it to add the path to the directories containing the artifacts of the built Rust library. This will allow GHC to find the generated Rust library files when it needs it.
During the legacy cleaning, we will also clean up the built Rust project. Here, “legacy cleaning” means that this code will only be used if the developer runs the cabal v1-clean command. The newer v2-clean just cleans up the Haskell project, not running any hooks in the setup module.

Building the Rust project is rather simple using cargo: just run cargo build --lib within the rust folder. We still have to make sure that it ran successfully and tell Cabal where it can find the compiled library. The following is an example of such a code:

-- | Path to the folder containing the Rust project.
rsFolder :: FilePath
rsFolder = "rust"

-- | Runs the specified cargo command with arguments.
-- Will stop the build and exit if the cargo command fails.
execCargo :: Verbosity -> String -> [String] -> IO ()
execCargo verbosity command args = do
    cargoPath <- findProgramOnSearchPath Verbosity.silent defaultProgramSearchPath "cargo"
    dir <- getCurrentDirectory
    let cargoExec = case cargoPath of
            Just (p, _) -> p
            Nothing -> "cargo"
        cargoArgs = command : args
        workingDir = Just (dir </> rsFolder)
        thirdComponent (_, _, c) = c
    maybeExit . fmap thirdComponent $ rawSystemStdInOut verbosity cargoExec cargoArgs workingDir Nothing Nothing IODataModeBinary

-- | Build the library for release.
rsMake :: Verbosity -> IO ()
rsMake verbosity = execCargo verbosity "build" ["--release", "--lib"]

-- | Adds the Rust library information to the '--extra-lib-dirs' and '--extra-include-dirs' Cabal properties.
-- Those will be used by GHC when compiling and linking.
rsAddDirs :: LocalBuildInfo -> IO LocalBuildInfo
rsAddDirs lbi' = do
    dir <- getCurrentDirectory
    let rustIncludeDir = dir </> rsFolder
        rustLibDir = dir </> rsFolder </> "target/release"
        updateLbi lbi = lbi{localPkgDescr = updatePkgDescr (localPkgDescr lbi)}
        updatePkgDescr pkgDescr = pkgDescr{library = updateLib <$> library pkgDescr}
        updateLib lib = lib{libBuildInfo = updateLibBi (libBuildInfo lib)}
        updateLibBi libBuild =
            libBuild
                { includeDirs = rustIncludeDir : includeDirs libBuild
                , extraLibDirs = rustLibDir : extraLibDirs libBuild
                }
    pure $ updateLbi lbi'

-- | Delete the Rust library's artifacts.
rsClean :: Verbosity -> IO ()
rsClean verbosity = execCargo verbosity "clean" []

Once that is done, we can check the result by running cabal build. If everything goes as expected, we should see the target folder created inside the rust folder containing the Rust project’s artifacts and note that Cabal ended successfully. cabal run should allow us to run the Haskell executable without issues.

So we know have a working Haskell project which compilation automatically triggers the build of a Rust library if it is required.

Things to keep in mind

This technique works and allows Cabal to build any Rust library automatically. There is one obvious requirement though: it requires a fully functional Rust build environment on the building machine. If we are actively developing Rust code on that machine, it should already be the case anyway. But things may be different if other users want to use our Haskell code. Documenting can only go so far, especially if our Haskell library is itself a dependency. To try and solve this, we can try and use Nix (more specifically Haskell.nix and Nix Flakes).

Build using Haskell.nix and Nix Flakes

Refresher

In a nutshell, Nix is both a language and a build system enabling reproducible builds in a declarative way. Nixpkgs is a collection of packages defined using this tool. Nix also provides a feature named Nix flakes that enables declaration of a component’s dependencies and their versions.

Haskell.nix is an infrastructure based on Nix to build Haskell packages. It is an alternative to the one defined in Nixpkgs.

Objective

We will use Nix to:

Set up a reproducible building environment
Produce binaries for both the Haskell project and Rust library

The following sections will assume that Nix is already installed and Nix Flakes support is enabled.

Defining our Nix Flake

Nix (through Nixpkgs) supports building Rust projects using a simple declaration. For Haskell projects though, we won’t be using Nixpkgs but Haskell.nix instead: Haskell.nix makes it easier to customize the Haskell configuration, versions etc.

So we will define our standard Haskell.nix flake with the following additions:

Define the Rust library
Define the Haskell project
Provide the Haskell project with the Nix-built Rust library
Tell Cabal not to build the Rust library itself as it is be provided externally (i.e. Setup.hs module doesn’t have to run cargo to build the library)

Thanks to Nix, we can easily build Haskell-with-Rust projects in a reproducible manner.

The folder hierarchy will now look like the following:

.
├── LICENSE
├── README.md
├── Setup.hs
├── flake.lock (2)
├── flake.nix (1)
├── hello.cabal
├── rust
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── src
│   │   └── lib.rs
└── src
    └── Main.hs

The flake.nix file (1) will describe our global project, while the generated flake.lock file (2) is used to track our Nix dependencies’ versions. The flake.nix file will look like the following:

{
  # We use Haskell.nix to compile the Haskell project.
  inputs.haskellNix.url = "github:input-output-hk/haskell.nix";
  inputs.nixpkgs.follows = "haskellNix/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  outputs = { self, nixpkgs, flake-utils, haskellNix }:
    let
      supportedSystems = [ /* ... our system here ... */ ];
    in
    flake-utils.lib.eachSystem supportedSystems (system:
      let
        overlays = [
          haskellNix.overlay

          # (1) We define our Rust project here.
          (final: prev: {
            rs_hello_world = final.rustPlatform.buildRustPackage {
              name = "rs_hello_world";
              src = ./rust;

              cargoLock = {
                lockFile = ./rust/Cargo.lock;
              };
            };
          })

          # (2) We define our Haskell project here.
          (final: prev: {
            hs_hello_world =
              final.haskell-nix.project' {
                src = ./.;
                compiler-nix-name = "ghc8107";
                modules = [{
                  # (3) We tell Cabal (our custom Setup actually) to use the Rust library built by Nix.
                  packages.hello.flags.external_lib = true;
                }];
                shell.tools = {
                  cabal = { };
                };
                shell.buildInputs = with pkgs; [
                  # (4) Allow Cabal to build the Rust library itself when in the development shell.
                  cargo
                ];
              };
          })
        ];
        pkgs = import nixpkgs { inherit system overlays; inherit (haskellNix) config; };
        flake = pkgs.hs_hello_world.flake { };
      in
      flake // rec {
        packages = {
          inherit (pkgs) rs_hello_world;
          hs_hello_world = flake.packages."hello:exe:hello";
        };
      });
}

As mentioned earlier, this flake file is a regular Haskell.nix flake file for any Haskell project, but with the following additions:

We have added our Rust library as an available Nix package (1). Here, we have chosen to use Nixpkgs’s buildRustPackage function to build the derivation corresponding to the Rust library. No matter how we choose to produce this derivation, its output should have a lib and include directories for respectively the library’s binaries and include files (if any). Those will be used by GHC during compilation and linking.
We have defined our Haskell project (2). We make use of the haskell-nix.project' function to build the derivation. The definition is rather standard, except that we’re setting a project flag named external_lib (3) indicating to Cabal that Nix will provide the Rust library. We will see how in a moment.
We are also adding the cargo command to the development shell (4). It is not used when building the project using Nix. This declaration is only meant to help us with development by adding the cargo command to the shell we get from running nix develop.

As usual, we need to run nix flake update to initially generate the flake.lock file or to update the our Nix dependencies.

Is the Rust dependency already available?

So what is this external_lib flag and why do we need it? This flag is defined as such in the Cabal configuration file:

flag external_lib
  description:       Use external library (don't compile automatically)
  default:           False

We don’t necessarily need it. We can have Cabal build the Rust library using cargo no matter what. But that would mean that we need to provide the Haskell.nix project with all the required Rust building tools (cargo) and the Rust library’s dependencies. And things can get complicated quickly. For example, when we add those dependencies to the Haskell.nix project, we won’t be able to know whether those dependencies belong to the Haskell project itself, or are required to build the Rust project.

So we are going another way and define a flag, named in our case external_lib, which allows for two distinct scenarios:

When it’s True, it will indicate to Cabal that the Rust library has already been built and is made available to GHC as a system package. Cabal then doesn’t need to run cargo to build the Rust project, nor does it need to run the hook adding the extra include and lib directories to GHC. As a system package, GHC will already be provided with the necessary information to access any extra-libraries (through the NIX_LDFLAGS environment variable and/or pkg-config).
When it’s False, it will indicate to Cabal that the Rust library must be built and GHC configured through the use of our custom hooks.

So basically, external_lib will be False by default in the development shell or whenever a user without Nix is trying to compile the Haskell project. In that case, Cabal will try and compile the Rust library. external_lib will be True when Nix is compiling the Haskell and Rust project itself, so Cabal will reuse the Nix-built Rust library and use the default Haskell build process.

Having the Rust library defined separately in Nix is also more convenient:

We have a dedicated description of the library, its dependencies etc.
We can customize its compilation as we see fit, through the standard Nix ways.

Only trigger the Rust library build when required

Here’s how we will modify our setup code to accommodate this new external_lib flag:

main :: IO ()
main = defaultMainWithHooks hooks
  where
    hooks =
        simpleUserHooks
            { preConf = \_ flags -> do
                unlessFlagM externalLibFlag flags $ rsMake (fromFlag $ configVerbosity flags) -- (1)
                pure emptyHookedBuildInfo
            , confHook = \a flags ->
                confHook simpleUserHooks a flags
                    >>= applyUnlessM externalLibFlag flags rsAddDirs -- (2)
            , postClean = \_ flags _ _ ->
                rsClean (fromFlag $ cleanVerbosity flags) -- (3)
            }

cabalFlag :: FlagName -> ConfigFlags -> Bool
cabalFlag name = fromMaybe False . lookupFlagAssignment name . configConfigurationsFlags

unlessFlagM :: FlagName -> ConfigFlags -> IO () -> IO ()
unlessFlagM name flags action
    | cabalFlag name flags = pure ()
    | otherwise = action

applyUnlessM :: FlagName -> ConfigFlags -> (a -> IO a) -> a -> IO a
applyUnlessM name flags apply a
    | cabalFlag name flags = pure a
    | otherwise = apply a

In the updated setup code, we only try to compile the Rust library if the external_lib flag is False (1), meaning that it is not provided by the system. In that case, we will also add the required compilation options for GHC to use the generated artifacts (2).

Note that no matter what, we will still clean up the Rust artifacts when the Cabal project is “legacy cleaned” (3) for the following reasons:

Cabal is not technically allowing us to know whether the flag is True or not when cleaning.
In any case, cleaning the Rust artifacts is not dangerous as they can be regenerated from code.

We now have two working ways of compiling a Rust library and embedding them into a Haskell project:

By having both of them automatically compiled by Nix
By having Cabal automatically compiling the Rust library through cargo

What about other existing Rust libraries?

So far, we have only made the assumption that the Rust source code is accessible by Cabal. This is most likely the case if both the Rust library and Haskell library are part of the same overall project. But it may not always be the case e.g. if the library is an external project out of our control.

To work around this issue when not using Nix, we can import the Rust library source code and point Cabal to it. There are many ways to do that:

Download the source code automatically as a separate step
Use Git submodule for Git repositories

Synchronizing the Rust source code will then however be our responsibility.

When using Nix, things are simpler as Nixpkgs is already allowing us to import Rust projects and/or crates.

Similar techniques

This technique is also applicable to languages other than Rust, so long as the hook code is adapted accordingly.

Hooking into the build process is already used in different packages:

protolens: use hooks to automatically generate Haskell code from Protobuf definition files
hlibsass: use hooks to build the libsass library via Make

We could also have used the Nix integration feature from Stack. But it feels like it would only be easier if the library we want to use is already in Nixpkgs. Otherwise, we would need to create a dedicated Nix configuration file, at which point Nix Flakes may be a better choice for customization and reusability.

Portability

The building machine needs to be able to run GHC and Cargo. The hooks in place will merely invoke Cargo to build the Rust project. This method does not target any specific platform, so whether our overall project is portable or not only depend on our code (Haskell and Rust).

Ultimately, the supported platforms will be the intersection of the platforms supported by GHC, Rust, Nix (if we decide to use it) and our code.

Reusability

Our overall project is reusable so long as the clients have the Rust toolchain installed or the developers are using Nix. In both cases, the required binaries will automatically be generated for the target platform.

Conclusion

We have seen how to build some Rust code and use them within a Haskell project:

Declare the Rust dependencies in the Haskell Cabal configuration file
Install setup hooks to automatically build them before building the Haskell project itself
Use Nix to build them and provide the generated binaries to the Haskell project

This method has solved our initial issues:

Automatically Rust code compilation, triggered by Haskell code compilation as required and handled by the available Rust toolchain
Portable code, or rather as portable as GHC, the Rust toolchain and overall project code can be
Reusability of the project’s code

What we will actually end up implementing will ultimately depend on our projects, its constraints and its distribution method. But it should be sufficient if we just want to play with Haskell and Rust code, as was my original intent.

Working code can be found on the dedicated GitHub repository.

Build a Haskell project alongside its Rust dependency

Table of contents

Introduction

Objective

Build using Cargo from Cabal

Refresher

Objective

Configuring our project

Creating our custom Setup

Things to keep in mind

Build using Haskell.nix and Nix Flakes

Refresher

Objective

Defining our Nix Flake

Is the Rust dependency already available?

Only trigger the Rust library build when required

What about other existing Rust libraries?

Similar techniques

Portability

Reusability

Conclusion