My professional life to now has always been a balancing act between the arts and technology (perhaps a rollercoaster would be a better analogy; balancing act somehow evokes the elegance and skill of a trapeze artist). I’ve mused before on some of the common threads that drive me; more recently I’ve been musing on more fundamental similarities. Many years ago, I remember in some Cathedral or other being told of the various carvings which exist in hidden parts of these buildings; art created not to be seen, at least by human eyes. Amongst others, restorations at Salisbury Cathedral uncovered such carvings. Effort expended not for human reward, but for its own worth or, one could say, ad maiorem Dei gloriam. Or, in technology, The Right Thing™. The right thing is what instantly drew me to functional programming back way before it was cool and shortly after the “27” had been added to “dra”. The pragmatic approach of OCaml trying to balance the safety, correctness, and Right Thing of functional programming with the need to write performant programs in a less Right Thing-like world made it a natural choice for a young professional singer writing and maintaining small systems written on trains, planes and hotels around the world! It’s continued to draw me in over the last 9 years.

But for me the art is made to be seen. During the COVID-19 pandemic, when live performance became impossible, I remember spending many months at home unable, or at least unwilling, to sing. Without even the colleagues to perform with, let alone the audience to consume the result, there was no purpose. And so too the perfect software is without purpose without users⁴. In championing and furthering Windows OCaml, I chose the niche of a niche, but I am hugely proud that today every Windows user of OCaml benefits (hopefully!) from the work I both did, and spearheaded others to do too. Likewise, every Windows user running winget install opam begins their journey in OCaml following my vision of how it should work, thanks to the seemingly boundless patience and efforts of my opam co-maintainers!

Behind all this, though, are the companies which allowed this to happen: first at OCaml Labs at the University of Cambridge and then spinning out into Tarides. And behind all that is Jane Street. After I started at OCaml Labs back in 2016, I explained to (mainly musical) colleagues that I was carrying on doing the open source work I’d been doing for the previous 10 years, but that somehow that had become work one could be paid to do (I can’t underscore enough how inconceivable the idea of that would have felt in 2006). That inevitably led to the question “so what do they get out of it?” - and the inevitable surprise that the answer was, directly at least, nothing. Windows opam represented work on a platform with no business case on an unused tool. The OCaml community was - is - the benefit.

Which makes this week feel less a change of course and more a continuation. OxCaml represents to me the evolution of the pragmatism that drew me to OCaml in the first place. And we’ve definitely got users! This year, like last year, is clearly going to be interesting. I think as we continue to navigate the whirlwind of agentic engineering, we’re going to care more and more about the foundation it’s all sat on, especially the compilers. Let’s hope they continue to strive to do the Right Thing™. And let’s see how this looks next year! So, here goes…

open Core

Wind forwards 3¼ years, and OCaml 4.08 is certainly a distant memory, but the features in it means it is very much a baseline at the moment. So, partly because the work is mostly already done, partly because it’s the last piece of the Relocatable puzzle, and mainly because I think it will be useful, I’ve been working towards proposed branches to allow us to make some exceptional releases in these old branches.

OCaml 5.x means that we have already been maintaining OCaml 4.14 as an LTS release of OCaml (although the changes coming in OCaml 5.5 mean that we really will be looking to end that long-term support before too much longer). I’ve previously talked about synchronising the GitHub Actions workflows. and the crazy shell script that manages the backporting.

So, how do you maintain some 20-or-so in-flight PR branches being occasionally rebased forwards onto updated versions of OCaml and simultaneously backported on to, initially, 7 compilers and, by the end, 12 compilers? The answer is: carefully, and with a lot of automation! ‘Carefully’ entailed setting some important ground rules: for example, the golden rule was that either the work was rebased or it was being amended. Never both simultaneously. This rule meant that while effort may be required to rebase forwards, the backports remained fixed, which allowed the patch resolutions to re-settle. In order to ease the automation, it also occasionally required over-backporting, i.e. taking some changes back to old compilers which were not strictly necessary, but which meant that the main branches did not need to be extensively re-written.

Anyway, included in this back-porting was enough work to be able to get CI running, since once we’ve finished support for a given version, the branch is mostly left to gather dust.

Image from Pinterest.

Like getting a stately home ready for visitors, the task I spent this week accomplishing was removing the dust sheets from the CI configurations on these old branches (4.08-4.13) so that the final backported changes could be properly tested. The stacks for Relocatable OCaml had these patches ordered as:

1. Unify CI workflows (affecting 4.08-4.13). We started switching over to GitHub Actions during the 4.12 dev cycle, but only completed the migration during the 4.13 dev cycle, so there were long-dead Travis CI scripts in the 4.08-4.12 branches, and the 4.13 branch’s GitHub Actions scripts hadn’t of course received the updates which the 4.14 ones had.
2. macOS arm64 support (4.08, 4.09 and 4.11). The curious gap is because Apple released the M1 during the 4.11 dev cycle, and it was decided that it was too late to alter the 4.11.0 release, so a re-release of 4.10 was done with Apple Silicon support, and then 4.12.0 “naturally” had the support, as the version of trunk to which it had originally been merged. As it happens, the 4.12 patches apply completely trivially to 4.11 and the 4.10 version of the patches apply trivially to 4.08 and 4.09, which means that you can get the complete support matrix when working on Apple Silicon.
3. Windows FlexDLL bootstrap overhaul (affecting 4.08-4.12). The main thing here is ocaml/ocaml#10135, which made the ability to build the Windows ports of OCaml somewhat easier. The lack of this branch is the reason why the support for Windows in opam-repository at the moment from ocaml/opam-repository#25861 only goes back as far OCaml 4.13. It’s not because it’s impossible, it just requires a lot of hardening in the opam packages to work around various deficiencies, and it’s actually just a configuration/build system change which backports really very easily as far as 4.08 (getting 4.07 and earlier working in opam on Windows is a daftness for another day).
4. Required backports for Relocatable OCaml (actual changes in the runtime/compiler which must be present to support the main PRs).
5. runtime-launch-info backport (from OCaml 5.2; ocaml/ocaml#12751)
6. compiled-primitives backport (from OCaml 5.3; ocaml/ocaml#12896)
7. in-prefix-test harness (from OCaml 5.5; ocaml/ocaml#14014)

Number 1 is exactly where I wanted it, but unfortunately the test harness sits on top of a large number of other changes, mainly because it was actually written well after all of them! So I got out my Git hammer and began to re-forge the commit series. It’s quite a set, but at the end of it, the difference between 4.08 and 4.14 in appveyor.yml, .github/workflows and tools/ci is a small set of trivially justifiable changes relating to actual differences between those two releases (for example, OCaml 4.08 still had the graphics library to test!). Ignoring changes that affect the build, configure and test systems only, it requires just three PRs to resurrect that branch:

• ocaml/ocaml#9557 - a necessary compilation change on IBM POWER
• ocaml/ocaml#9981 - a necessary compilation change for the BSDs
• ocaml/ocaml#12577 - a series of C17 compatibility changes needed for compilation with recent GCC/Clang

… and that’s it!

On top of that unified CI, it’s then possible to backport the test harness, which then provides reliable installation testing for all the relocatable features. I’m almost certainly strange, but this was kind of fun to backport. By the time it gets back to 4.08, the commit message for the harness itself reads:

Deviations from original:
  - C stub added to the harness instead of backporting
    caml_sys_proc_self_exe
  - Compmisc.reinit_path used instead of updating Compmisc.init_path
  - Config.c_compiler_vendor avoided by instead exposing the constant as
    Toolchain.c_compiler_vendor
  - Config.asm_dwarf_version folded into toolchain.ml.in
  - Config.shebangscripts avoided by instead adding an additional
    --with- option to the harness and threading $(SHEBANGSCRIPTS) from
    the build system
  - Required parts of Bytelink.read_runtime_launch_info directly added
    to the test harness, instead of exposing the machinery in Bytelink.
  - Misc.Stdlib.String.to_utf_8_seq added directly to TestRelocation.
  - ocamlmklib -M exits with code 0
  - Standard Library functions added:
    - Bytes.{get_utf_8_uchar,set_utf_16le_uchar} (4.14)
    - Char.Ascii.is_letter (5.4)
    - In_channel.{fold,input}_lines (5.1)
    - In_channel.input_all (4.14)
    - In_channel.really_input_bigarray (5.2)
    - In_channel.{set_binary_mode,with_open_bin} (4.14)
    - Int.max (4.13)
    - List.{drop,take}_while (5.3)
    - List.find_map (4.10)
    - List.fold_left_map (4.11)
    - Option.exists (5.5)
    - Out_channel.{with_open_bin,with_open_text} (4.14)
    - Result.Syntax (5.4)
    - String.{starts,ends}_with (4.13)
    - Sys.{mkdir,rmdir} (4.12)
    - Sys.signal_to_string (5.4)
    - Uchar.utf_{decode_uchar,decode_length,16_byte_length} (4.14)
    - Unix.realpath (4.13)
  - OCaml 5.4 labelled tuples removed
  - Added the ability for Build/Prefix/Relative to be optional in
    TestRelocation
  - Prior to 5.4.0, ocamlrun -config didn't include entries in the
    search path coming from CAML_LD_LIBRARY_PATH
  - Prior to 5.3.0, the compiler added +flexdll to the search path even
    when -nostdlib is specified, which affects the relocation test when
    bootstrapping FlexDLL.
  - Prior to 5.3.0, ocamlcmt and ocamlobjinfo didn't support -vnum. The
    simplest thing in this instance is simply to skip the test of their
    binaries.
  - Prior to 5.3.0, the dynlink library included a complete copy of the
    Config module, which affects the relocation test.
  - Prior to 5.2.0, RNTM isn't always written to bytecode images when
    compiling with -custom. Heuristic altered to analyse the DLLS
    section instead.
  - Prior to 4.14.0, errors in #load don't create detectable script
    errors: toplevel test adapted to cope with this
  - Prior to 4.13.0, the MSVC port didn't translate -lfoo automatically
  - Adapt the toplevel test to deal with the presence of compiler
    plugins in ocamlnat
  - Link test updated to deal with different runtime behaviour for
    errors and the lack of -output-complete-exe, both of which changed
    in 4.10.0
  - Build system adapted (Makefile.common macros didn't support C files
    until OCaml 5.2; VPATH wasn't used until OCaml 5.1; win32unix/unix
    distinction until OCaml 5.0)
  - Misc.Style changed to Misc.Color (changed in OCaml 5.2) and uses of
    the inline_code and hint formatting changed to loc (since
    inline_code was also added in OCaml 5.2 and hint OCaml 5.1)
  - Removed compressed marshalling support, since that was added in
    OCaml 5.1
  - Read camlheader file instead of runtime-launch-info, and adapt the
    relocation test to deal with incorrect naming of two of the header
    files (fixed in 4.09.0)
  - OCAML_FLEXLINK scrubbed from the environment (removed in 5.2)
  - Updated to deal with all bytecode binaries being installed with
    debug information
  - Updated to deal with all bytecode/native versions of tools being
    installed (changed in 5.1)
  - Updated to deal with old installation layout for otherlibs
  - Updated handling for the bigarray library
  - Updated relocation test to deal with different use of debug flags
    from OCaml 4.12+
  - Order of rules in testsuite/in_prefix/Makefile.test updated for
    compatibility with the aged version of GNU make shipped on macOS
    (pattern rules triggered in the wrong order otherwise)

Next-up, having removed all those dust sheets: finalising the actual backports!

Back in September, as part of Relocatable OCaml, I needed to update the ocaml-config package, which houses one of the plumbing scripts used for the ocaml package in opam. Separately, for opam’s CI systems, we wanted to be able to test against trunk OCaml, which implied some updates to the plumbing for ocaml-system as well. The right thing to do with these scripts which had lived in opam-respository is to push them back upstream, which was what I did with a cute piece of Git spelunking in ocaml/ocaml#14351, which contains commits with files cherry-picked from ocaml/opam-repository PRs. Each commit contains a reference to an opam-repository commit, which in turn leads to the original PR. For example, ocaml/ocaml#d0272f8 copies the original files from ocaml/opam-repository#1bab453. The neat thing with putting them into a series under a single path in OCaml now is what then happens with subsequent changes. For example, ocaml/opam-repository#17541 introduced the ocaml-config.2 package, which was a completely fresh script, but ocaml/ocaml#749a918 is instead able to show the actual diff of the script, allowing for much easier review and so forth. Of course, git doesn’t store patches, so the really useful part is that although the history is different, the file in each commit is exactly as in the original commit, which allowed ocaml/opam-repository#29080 just to update the URLs to point to these authoritative upstream sources.

So far, so good - that had all been merged before Christmas. The support for explicit-relative paths in ld.conf added in ocaml/ocaml#14243 required an update to this script, as noted in ocaml/opam-repository#29085, since opam var ocaml:stubsdir for OCaml 5.5 and onwards was giving an erroneous ../stublibs:./stublibs:. as it didn’t know to translate the paths read from ld.conf as being relative to ld.conf itself. That an update was required anyway gave me opportunity to fix three other oddities in that script:

1. The script gets installed to users’ opam switches, rather than used directly as part of the build.
2. There were several versions of it, and there didn’t need to be.
3. The use of opam’s substs mechanism made it difficult to debug outside of opam.

This work had all been left in ocaml/ocaml#14250 in December and was straightforward enough. The ocaml-config package had only come into existence in ocaml/opam-repository#11928 in order to stop the same script being stored multiple times in the same repository. Once ocaml/opam-repository#25960 reduced that single copy to zero copies, it made more sense for each ocaml package just to download the script download, and do away with the ocaml-config package completely. Unifying the scripts, and turning it into a regular OCaml script flows logically from that. Although, the final lines amuse me somewhat:

let oc = open_out package_config_file in
  (* Quoted strings need OCaml 4.02; "\ " needs OCaml 3.09! *)
  Printf.fprintf oc "\
    opam-version: \"2.0\"\n\
    variables {\n  \

This week, I updated that commit series to do the same kind of thing to the ocaml-system script. These are now just regular OCaml scripts (in tools/opam in ocaml/ocaml) which can be run directly and that allowed me to update ocaml/ocaml#14354 which should allow us to be offer an ocaml-system package during development and release cycles. The autoconf stuff might get revisited in that: cute, but potentially annoying (the joy of higher order meta-programming… autoconf generates and updates those files as part of the process of generating configure, rather than as part of running configure itself… what could possibly go wrong!). There’s also ocaml/ocaml#14355 which allows a similar trick to be done with custom compilers. The point here is that you build a non-standard compiler and install it outside of opam and this then provides a relatively simple mechanism for opam to be able to use it. Although “system” compilers are a bit awkward to use from distributions, because opam interacts very badly with system-installed OCaml packages, if the only thing you’re installing at a system-level is the compiler, the support is very good.

The final thing to deal with was a very silly ToDo item I’d added:

- [ ] `tools/opam/Layout.md`

In the excitement of getting the other Relocatable OCaml PRs opened in September, in a brief from sanity, I had decided some kind of documentation ought to be in place before ocaml/ocaml#14250 was considered for merging 😱 Days turned into weeks, weeks turned into months. And then, one not-so-very special day, I went to my laptop, I sat down, and I wrote our compiler packaging story. A story about package names, a story about compiler variants, a story about ancient libraries, long forgotten. But above all things, a story about OCaml.

Which is mercifully now merged.

I am particularly thrilled, and just a little surprised, that Relocatable OCaml has happened as I envisioned, with nothing needing to be dropped. There were of course many useful suggestions for re-working to improve clarity and so forth during review, but the test harnesses, RFCs, talks, and so forth appear to have provided me with the required knife to work out correctly what was necessary to achieve all this!

For posterity, here’s the timeline! On 4 May, ocaml/ocaml#14014 introduced a comprehensive test harness for the Relocatable OCaml RFC, which was reviewed by Nicolás Ojeda Bär and Olivier Nicole and merged on 21 June.

Finalising the PRs took me a little while longer than I’d expected, and the set of 4 PRs, along with a fifth “combined” demonstration were opened on 15 September. Over the coming weeks, Jonah Beckford, Antonin Décimo, Hugo Heuzard, Samuel Hym and Vincent Laviron pored over the details. Their invaluable review then set the scene for a marathon synchronous defence of the branches in Paris with Damien Doligez on 25 November. I’m very grateful to everyone involved in these reviews: a lot of code; and a lot of gnarly details!

That left a small todo list for each PR. I managed to coerce a different core maintainer for each, with Florian Angeletti merging ocaml/ocaml#14243 on 27 November, Gabriel Scherer merging ocaml/ocaml#14244 on 8 December, Nicolás merging ocaml/ocaml#14245 on 12 December and then finally Nick Barnes merging ocaml/ocaml#14246 today.

That means that OCaml’s trunk branch now matches the bulk of ocaml/ocaml#14247. What’s next? There’s a small amount of work still to do on the scripts which plumb the ocaml opam package together. Then, having now got everything merged, it’ll be time to finalise the backports for proposed re-releases of the older compilers.

But, for now, the future’s very definitely relocatable! 🥳

10 days ago, a new user popped up on our Discourse forum with a failure of some core packages to work on Windows. The OP was using MSYS2, which at the moment has slightly less support than the recommended Cygwin, but it looked to me like either my ocaml/opam-repository#28897 or ocaml/ocamlfind#112 ought to fix things.

Except it didn’t, and it wasn’t until today that I was able to reproduce the problem, and figure out what was going on. As it happens, once I could reproduce it, this was easy for me to work out, but it shines an interesting light on to a subtle bit of opam’s Windows internals and consequently something that either the OP or a packager has misunderstood and it also finally goads me into standing on my soapbox and shouting from the rafters:

Code should be changed if and only if a bug is being fixed or an actual user-facing feature is being implemented! The risk of regression ALWAYS outweighs the benefit of refactoring UNLESS you’re actually changing something else.

The target of my particular ire is the wonderful ShellCheck tool. If you don’t use it already, then:

1. You don’t have to write shell scripts, and should continue not having to write shell scripts for as long as possible! However, you may have to one day, so:
2. You should absolutely start using it on the next shell script you have to write. You should also use it to check changes you make to existing scripts, too.

But what I’m finally going to come out and just say, is that you should never ever change a shell script just to please ShellCheck unless you actually have a bug report. If ShellCheck grumbles and warns something might cause a bug, then take it as a challenge - figure out the scenario in which that could be triggered, test it, fix it. If, as is often the case, it’s warning about an unhygienic construct which could go wrong but actually never will because, for example, the file names concerned don’t contain spaces and never will, then leave that script alone!

What was actually going on here? The OP was seeing this effect when trying to install topkg:

C:\Users\DRA>opam install topkg
The following actions will be performed:
=== install 1 package
  ∗ topkg      1.1.1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
⬇ retrieved topkg.1.1.1  (https://opam.ocaml.org/cache)
[ERROR] The compilation of topkg.1.1.1 failed at "ocaml pkg/pkg.ml build --pkg-name topkg --dev-pkg false".

#=== ERROR while compiling topkg.1.1.1 ========================================#
# context     2.5.0 | win32/x86_64 | ocaml.4.14.2 | https://opam.ocaml.org#9bc1691da5d727ede095f83c019b92087a7da33e
# path        C:\Devel\Roots\msys2-testing-native\default\.opam-switch\build\topkg.1.1.1
# command     C:\Devel\Roots\msys2-testing-native\default\bin\ocaml.exe pkg/pkg.ml build --pkg-name topkg --dev-pkg false
# exit-code   125
# env-file    C:\Devel\Roots\msys2-testing-native\log\topkg-8308-179b77.env
# output-file C:\Devel\Roots\msys2-testing-native\log\topkg-8308-179b77.out
### output ###
# Exception: Fl_package_base.No_such_package ("findlib", "").

This error gets readily seen when using OCaml 5.x on MSYS2. It’s not a problem with topkg, it’s actually that the ocamlfind library manager is not installed correctly. There are fixes pending for that in ocaml/ocamlfind#112, but the issue there is to do with OCaml 5.x - this build is OCaml 4.14.2.

The actual issue is readily apparent if we rebuild ocamlfind:

C:\Users\DRA>opam reinstall ocamlfind -v
The following actions will be performed:
=== recompile 1 package
  ↻ ocamlfind 1.9.8

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
⬇ retrieved ocamlfind.1.9.8  (cached)
+ C:\Devel\Roots\msys2-testing-native\default\.opam-switch\build\ocamlfind.1.9.8\./configure "-bindir" "C:\\Devel\\Roots\\msys2-testing-native\\default\\bin" "-sitelib" "C:\\Devel\\Roots\\msys2-testing-native\\default\\lib" "-mandir" "C:\\Devel\\Roots\\msys2-testing-native\\default\\man" "-config" "C:\\Devel\\Roots\\msys2-testing-native\\default\\lib/findlib.conf" "-no-custom" "-no-camlp4" (CWD=C:\Devel\Roots\msys2-testing-native\default\.opam-switch\build\ocamlfind.1.9.8)
- Welcome to findlib version 1.9.8
- Configuring core...
<snip snip snip>
- Configuration for threads written to site-lib-src/threads/META
- Configuration for str written to site-lib-src/str/META
- Configuration for bytes written to site-lib-src/bytes/META
- Access denied - SRC
- File not found - -NAME
- File not found - META.IN
- File not found - -TYPE
- File not found - F
- File not found - -EXEC
- File not found - SH
- File not found - -C
- File not found - SED -E 'S/@VERSION@/1.9.8/G'         -E 'S/@REQUIRES@//G'    \$1\
- File not found -  > \${1%.IN}\
- File not found - SH
- File not found - {}
- File not found - ;
- Detecting compiler arguments: (extractor built) ok

What’s going on at the end of that snippet? It turns out ocamlfind’s configure script contains a call to what it expects to be POSIX find, but what it’s actually getting is the Windows find utility. This problem is as old as the hills, and affects sort as well:

C:\Users\DRA>find /?
Searches for a text string in a file or files.

FIND [/V] [/C] [/N] [/I] [/OFF[LINE]] "string" [[drive:][path]filename[ ...]]

  /V         Displays all lines NOT containing the specified string.
  /C         Displays only the count of lines containing the string.
  /N         Displays line numbers with the displayed lines.
  /I         Ignores the case of characters when searching for the string.
  /OFF[LINE] Do not skip files with offline attribute set.
  "string"   Specifies the text string to find.
  [drive:][path]filename
             Specifies a file or files to search.

If a path is not specified, FIND searches the text typed at the prompt
or piped from another command.

versus:

C:\Users\DRA>C:\msys64\usr\bin\find --version
find (GNU findutils) 4.10.0
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Eric B. Decker, James Youngman, and Kevin Dalley.
Features enabled: D_TYPE O_NOFOLLOW(enabled) LEAF_OPTIMISATION FTS(FTS_CWDFD) CBO(level=1)

Now, it turns out that the OP’s system is misconfigured - which is part of the reason why Cygwin, MSYS2 and Git for Windows don’t put their Unix shell utilities into PATH by default. The Unix utilities are in C:\msys64\usr\bin, but this entry in PATH appears after C:\WINDOWS\system32, so a few utilities get shadowed. For things like curl and tar, this is usually benign, but for find and sort it’s pretty terminal!

In its default mode, where opam manages the Cygwin or MSYS2 environment for you - it’s part of a mildly complex process in ocaml/opam#5832, as opam ensures that either the Cygwin or MSYS2 bin directory is just “left” enough in PATH to ensure that some key utilities are not shadowed, but not so far to the left that it shadows things like Git for Windows!

However, if you’ve set-up PATH yourself, as the OP had, you’re on your own, I’m afraid!

So why is all this making me rant about ShellCheck? Well, the problem is that downgrading ocamlfind to a previous version works just fine, and so it’s the regression in ocamlfind 1.9.8 that I want to focus on. The failing part of the script is:

# create META from META.in in POSIX-compatible & safe way
# see: https://www.shellcheck.net/wiki/SC2044
meta_subst="sed -e 's/@VERSION@/$version/g' \
        -e 's/@REQUIRES@/${req_bytes}/g' \
        \"\$1\" > \"\${1%.in}\""
find src -name 'META.in' -type f -exec sh -c "$meta_subst" sh {} \;

but which was originally (from cbc4a7d):

for part in `cd src; echo *`; do
    if [ -f "src/$part/META.in" ]; then
        sed -e "s/@VERSION@/$version/g" \
            -e "s/@REQUIRES@/${req_bytes}/g" \
            src/$part/META.in >src/$part/META
    fi
done

ShellCheck reports two issues with that original loop:

1. It dislikes the use of backticks rather than $(...)
2. It worries that the exit code of cd is unchecked

The first is because this script is very old, and it was written at a time when there were not only versions of sh which didn’t support $(...) notation, there were people using OCaml on them! In fact, it’s only comparatively recently that the default version of sh in (Open)Solaris/Illumos, etc. actually supports it.

The second is true, but:

1. It’s never going to happen (the src directory be missing from the tarball?!)
2. Even if it it does, the loop will correctly do nothing, and the build will fail later

So, while for part in `cd src; echo *`; do is somewhat showing its age, in the context of this specific fragment of this specific shell script, that will never ever cause a problem. Writing a new script? Don’t write that! Adding a new part to an existing script? Don’t write that!

However, it is now broken, and my proposed fix is that actually the use of the wildcard was completely unnecessary, as we already compute the relevant list of directories which can possibly contain a META.in file slightly later in the script (taken from 1291259):

for part in $parts; do
  if [ -f src/"$part"/META.in ]; then
    sed -e "s/@VERSION@/$version/g" \
        -e "s/@REQUIRES@/${req_bytes}/g" \
        src/"$part"/META.in > src/"$part"/META
  fi
done

A piece of code got changed to fix neither a bug nor add a feature, and something which worked before became broken. It took a non-trivial amount of (wallclock) time to diagnose and fix the issue. That’s a lot of negative consequences, and so far at least, there are no positive consequences.

If it ain’t broke, don’t fix it!

I’ve been involved from both a Tarides perspective and from general day-to-day upstream work on the OCaml runtime with some of the experiences Jane Street had switching to OCaml 5 (because they’re using OxCaml, you’ll more often hear this referred to as “Runtime 5”, but it essentially means Multicore OCaml). It’s interesting to reflect on the decisions we made when merging Multicore OCaml in the light of these subsequent experiences not, of course, as a navel-gazing exercise in the benefits of hindsight, but in terms of what we can potentially learn for the road towards OxCaml becoming OCaml.

Unsurprisingly, there was tons of preparation, planning, and hard work leading up to opening and merging of Multicore in ocaml/ocaml#10831 in January 2022. One of the early ideas was to merge just the runtime changes as a separate runtime, leaving all the language changes to a subsequent update. The main thing here would have been to upstream the immense changes to the allocator and garbage collector along with the domains and fibers machinery, while not yet exposing it. I remember the concern being that having essentially a runtime variant (not unlike the debug runtime) might lead to very slow uptake at actually testing it and possibly a maintenance burden. i.e. we were concerned at maintaining two runtimes. This would probably have resulted in something like OCaml 4.15.0, with an experimental official multicore-aware runtime.

The decisions moved on to being more, “no, it’s all or nothing - let’s take it”. From there gradually we moved more towards this being OCaml 5.00.0; a major version bump. The previous rebase of Multicore had been on 4.12, and that had intentionally separated off the concept of effects to a separate version, as this had the benefit that the surface syntax (especially where keywords were concerned) was unaltered. During the various discussions, there was (slightly unexpected!) enthusiasm to go the whole hog, and bring the effect system in as well, but leaving the changing of the surface syntax to a subsequent release. That this would be OCaml 5 was cemented, and we had a plan.

As a side-note, bumping the major release comes with a few other things. In particular, it was suggested we should deal with the accumulation of deprecations in the Standard Library, finally removing the unsafe string mode, and so forth. Potential risks, but the practical idea was that programs which were being updated to work with OCaml 5 which still hadn’t updated to these, in some cases, very old deprecations would have bigger worries.

During 2022, until the release of OCaml 5.0 on 15 December, I had the role of release manager for all of Tarides’ efforts towards OCaml 5. The simile between OCaml 4 / OCaml 5 and Python 2 / Python 3 weighed heavily on a lot of what we did, coupled with the very real fear that we might still find a fundamental problem that could cause the multicore change to need to be reverted, with more work required. A key thing which was introduced was that apart from those deprecations, and a few bits of installation tidying, we froze all other work on OCaml during the OCaml 5.0 dev cycle. This was such a big deal, we named it the “Sequential Glaciation”, and it meant an unprecedented level of compatibility for OCaml 4.14 programs when running on OCaml 5.0. Essentially, it meant that any program which compiled without warnings on OCaml 4.14 should compile without change on OCaml 5.0 (with some corner cases for Standard Library replacements, etc.), an unusual level of commitment.

This gave an escape hatch: it meant that any code could be being tested on OCaml 5 to check for performance regressions.

So we invested a phenomenal amount of engineering effort into ensuring that the public OCaml ecosystem was compatible at launch - that so-called “sequential” code (i.e. written for OCaml 4) would successfully execute single-domain on OCaml 5.

And then follows an inadvertent mistake, to bring this back round to Jane Street’s experience.

OCaml 5.1’s Changelog includes 25 Standard Library entries of which 17 add new functions, 5 are non-trivial performance improvements and, crucially, three breaking changes and non-trivial bug-fixes. That’s the Standard Library alone!

No such escape hatch now: if you’re only now, as Jane Street were, only just able to think about investigating OCaml 5 (also bear in mind that by OCaml 5.1 we still hadn’t restored feature parity with OCaml 4.14), there’s no easy way back if you do hit a problem. Even if you just upgrade a codebase to OCaml 5.1 without using domains and effects, it’s very hard, bordering on impossible, to go back to 4.14.

It’s important not to put this down to the benefits of hindsight. I don’t remember us ever discussing the idea of actively supporting OCaml 5 (i.e. with all the front-end changes included) still being able to run on the OCaml 4.14 runtime. It’s the kind of suggestion I would have expected in a core developers’ meeting to say, “yeah, I can do that”, but to have met with some resistance from others at the idea of us all having to maintain it. 🙂

However, from a technical perspective, it really wasn’t as difficult as I’d have expected. In the middle of 2023, I prepared a branch of OCaml 5 for Jane Street which replayed the OCaml 5.x changes, skipping all of the alterations to the runtime (a snapshot of it sits in branch backported-trunk-to-5.1-20230929 on my OCaml fork). The work in this formed the basis for Jane Street’s OxCaml-on-runtime4 changes (it wasn’t called OxCaml back then, of course) - and when compared with its basis commit of 5000b93cad, there’s no OCaml 5 runtime changes in there, just various minor things added to the 4.x runtime up to its release and a smattering of non-multicore related things.

With this work, Jane Street managed to get their internal compiler to being the OCaml 5.1 frontend but still running on their altered version of “Runtime 4”. As they started investigating switching all the exciting things in OxCaml to be available on “Runtime 5”, they also considered trying the runtime-variant approach we’d ruled out earlier in the merge. Turns out our ruling it out was correct - it didn’t work easily for them, as it requires too many extra flags and things to be begin plumbed through build systems and so forth.

The trick was to make it a configuration option (which still exists in OxCaml at the moment, although it’s on notice!). That provides a lot of simplification in the deployment (we never have to build both runtimes at once, for example), and making that “escape hatch” slightly more awkward to get to is probably no bad thing, either. For a very large codebase, I can certainly see the value it could have provided.

Maybe that’s also something we should look to as OxCaml’s development continues: focusing in the ecosystem on being able to have that escape hatch back to OCaml itself, possibly more for benchmarking comparison, than anything else. Although so much of OxCaml is visible through the front-end (the modes systems, et al), stack allocation, unboxed types, SIMD, and so forth are all also runtime changes, and perhaps we need to be considering these escape hatches, before as well as during any upstreaming effort. If nothing else, it would help benchmarking. It might also be interesting for things like data-race freedom, which don’t affect the runtime, to be able to have programs which are data-race free on OCaml 5.x, even if the final form of DRF in OxCaml isn’t yet known. And maybe it’ll be a bit too hard to maintain, but I’m always up for a challenge, and musing is musing!

Sticking with OxCaml, I was particularly interested, both from hallway conversations and from talks, to muse on how the modes system in OxCaml might make its way upstream. It’s a known design decision of ocamlopt, which Richard alluded to in his Haskell (!!) keynote, that it prefers predictability and a certain measure of simplicity over a kitchen-sink of optimisations. I want to be able to write high-performance program in OCaml, and I think others should be able to as well, but here are three possible takes on why we might end up with a lot of front-end “complexity” (I put it in quotes, because I tire of “complexity” often begin used as an attack on solutions to problems) of OxCaml even without performance in mind:

1. Last year at ICFP, we saw in Oxidizing OCaml with Modal Memory Management the introduction of modes to increase performance by reducing heap allocation. There’s another neat use of locals, and the regions that come with them: the same technique allows us to stop accidentally programs like

   let uh_oh file = In_channel.with_open_text file In_channel.really_input_string
   

   Performance or no, I think we can all agree we’d like to write programs where the type checker assures us that not only are we well-typed but also that we don’t leak resources!

2. Thanks to Bounding Data Races in Space and Time, OCaml has a much nicer approach to data-races than many languages (certainly than the C language its runtime sits on!). We can say that OCaml programs with data races do not “catch fire”, but it’s still chaos! Atomics give us one way of being able to manage this, but we have to actually do it. The work presented earlier this year in Data Race Freedom à la Mode gives us OxCaml’s capsules… now we can declare we want to “do it”, and have the type checker ensure we actually did.

   Performance or no, I think we can all agree we’d like to write programs where the type checker assures us that not only are we well-typed but also data-race free!

3. Thanks to Modal Effect Types this year (SPLASH and ICFP ran at the same time this year), we can look towards typed effects in OCaml.

   Performance or no, I think we can all agree we’d like to write programs where the type checker assures us that not are we well-typed but also that we handle all our effects!

There’re other approaches to each of these problems as well (indeed, even in OxCaml, where modal effects are concerned), but I think it’s interesting to see what’s happening here as not just about high-performance.

That’s enough musing for now… I’ll endeavour to write some more and maybe in a less musy way about other talks I went to another day!

I expected this to be a fairly simple application of the C API features added in dra27/ocaml#file_descr-interop. It’s not quite what happened, though - as I don’t actually think that any of the changes I propose for the Unix library in OCaml are needed for ocaml-uring. What piqued my interest in this rather deep rabbit-hole was that what I would contend are slightly poor design decisions stem from the corrupting notion in C that file descriptors are integers - i.e. because file descriptors in C are integers, it becomes dangerously tempting to keep them as that in OCaml, because OCaml treats integers specially.

ocaml-uring’s documentation states that it “aims to provide a thin type-safe layer for use in higher-level interfaces”. I think adopting Xavier’s veto on manipulating file descriptors as integers in OCaml (i.e. never doing it) leads to greater type safety, even in thin bindings, without compromising performance. Let’s see! 🤓

The sources for ocaml-uring use Obj.magic both in the library itself in one place, and at various points in both the tests and the documentation, exclusively for duping the type system into believing an int is a Unix.file_descr.

Consider mkdirat(2). This is part of a family of *at syscalls which accept a file descriptor as an argument and which will use the directory name of the file referred to by that file descriptor as the base for a relative path. There is a special value AT_FDCWD which can be passed instead of an open file descriptor (it instructs mkdirat instead to interpret paths relative to the current working directory), and it’s the representation of this AT_FDCWD which causes the Obj.magic. When used in C, AT_FDCWD is an illegal file descriptor value (it’s -100, as it happens). Continuing with mkdirat as an example, we have this C declaration:

void io_uring_prep_mkdirat(struct io_uring_sqe *sqe,
                           int dirfd, const char *path, mode_t mode);

and this corresponding primitive in ocaml-uring:

external submit_mkdirat : t -> id
                          -> Unix.file_descr -> Sketch.ptr -> int
                          -> bool = "ocaml_uring_submit_mkdirat" [@@noalloc]

There’s a close correspondence between the two - the OCaml t gets passed to io_uring_get_sqe which gives us the first parameter for the prep call. The id is separately passed to io_uring_set_sqe_data (that’s part of a mechanism in liburing to be able to identify which syscalls have completed - more on that later). The remaining arguments in this primitive (the Unix.file_descr, the Sketch.ptr and the int) correspond to the dirfd, path and mode arguments to the syscall (the liveness of buffers is complicated here - the Sketch.ptr type is essentially allowing a string to be passed from OCaml to C without subsequently having to worry about it being moved by the GC).

It looks like a good thin layer over the syscall. Every one of these prep calls will necessarily have a io_uring_set_sqe_data call, and it’s always good to minimise the number of crossings between C and OCaml when writing C bindings.

It naturally becomes tempting to have a Unix.file_descr value for AT_FDCWD, and that’s indeed the implementation:

let at_fdcwd : Unix.file_descr = Obj.magic Config.at_fdcwd

However, there really is no need for this, and if we step back from the fact it’s an int, and instead consider what AT_FDCWD is modelling, a more obvious solution emerges which doesn’t require magic. AT_FDCWD is essentially be used to mean “no file descriptor” (and there’s then an interpretation for what that means mkdirat should do). i.e. AT_FDCWD means None, which is then being modelled by -100. An actual file descriptor is Some fd, which is being modelled by its number (which can never be -100). In other words, we could pass a Unix.file_descr option to the C stub, instead of a Unix.file_descr. This falls out even more naturally when we look at the actual Uring.mkdirat exposed to the library user:

let mkdirat t ~mode ?(fd=at_fdcwd) path user_data =

It actually already has a Unix.file_descr option because it’s using an optional argument to convey it! It’s quite an easy adjustment just to allow the C stub to resolve this, it has the benefit that the value of AT_FDCWD no longer has to be determined from OCaml, and it doesn’t involve any additional allocations of memory on the OCaml side (because the optional argument meant there was already one).

So far, so hopefully good. Now on to the one in the README:

# let fd =
    if result < 0 then failwith ("Error: " ^ string_of_int result);
    (Obj.magic result : Unix.file_descr);;
val fd : Unix.file_descr = <abstr>

Well, it wasn’t my intention, but the proposed Unix.fdopen could deal with that. Except that this is not “fdopen” in the spirit it was added for - in fact, it’s precisely what was not supposed to be done. We’ve got an int in OCaml that is actually a file descriptor, and we’re trying to coerce it to be a Unix.file_descr. We’ve got a layering violation - we’ve allowed an abstract value from the world of C that is represented in C as an int to leak up to the world of OCaml as an OCaml int, which should be a number!

Funnily enough, ocaml-uring has another minor layering violation like this in the translation of Unix error values to Unix.error values in the Uring.error_of_errno function:

let error_of_errno e =
  Uring.error_of_errno (abs e)

I didn’t want to abuse my proposed Unix.fdopen this way, so soon after conceiving it for a more noble and even slightly-typed purpose, so instead I opted for bouncing these numbers back to C by instead adding Uring.file_descr_of_result:

value ocaml_uring_file_descr_of_result(value v_result)
{
  CAMLparam0();
  CAMLlocal2(result, val);
  if (Int_val(v_result) < 0) {
    val = unix_error_of_code(-Int_val(v_result));
    result = caml_alloc_small(1, 1);
  } else {
    val = caml_unix_file_descr_of_fd(Int_val(v_result));
    result = caml_alloc_small(1, 0);
  }
  Field(result, 0) = val;
  CAMLreturn(result);
}

which handles the pattern in C, where it belongs. The result certainly looks nicer in some of the test code, for example, the numeric matching in one of the tests:

if result >= 0 then
  let fd = Unix.fdopen result in
  Unix.close fd
else
  let error = Uring.error_of_errno fd in
  raise (Unix.Unix_error(error, (* ... *)))

becomes a slightly more idiomatic:

match Uring.file_descr_of_result result with
| Ok fd -> Unix.close fd
| Error error -> raise (Unix.Unix_error(error, (* ... *)))

but there are two things that still feel bad - those ints are still there in OCaml! Worse, we’re now incurring an extra C call to process them.

The veto tells us that the processing belongs in C. Is there a way to do this where we don’t ever have the general int result exposed in OCaml, but we’re still a thin wrapper around the API and at no cost?

Not without breaking the API, but I think we can have our API cake and also eat it in this case. A first stab sits on my fork. The result of an io_uring operation is always a (C) int. Each of the syscalls uses one of three encodings of that integer. In all three cases, a negative number corresponds to an error, where the absolute value of the number is the Unix error number (from errno.h). Some functions then just return 0 on success. The other functions return a number ≥ 0 and for some of those, that number is a file descriptor.

Now, the choice of encoding is known when we prep the call. So the trick I tried instead was to encode that in the user-data attached to the ring entry. The ocaml-uring implementation is using an array index as the user-data, so even on a 32-bit system we can comfortably steal two bits to encode how to treat the result. The three wait functions now instead of returning:

type cqe_option = private
| Cqe_none
| Cqe_some of { user_data_id : id; res: int }

return a slightly richer:

type cqe_option = private
| Cqe_none
| Cqe_unit of { user_data_id : id; result: unit }
| Cqe_int of { user_data_id : id; result: int }
| Cqe_fd of { user_data_id : id; result: Unix.file_descr }
| Cqe_error of { user_data_id : id; result: Unix.error }

A little bit of additional plumbing is needed for cancellation. I opted to have the C functions themselves return the amended user-data value, as it’s only needed for cancellation. That keeps the considerations of C in C-land, but it might be better to push that into the Heap implementation which backs the IDs and allow both the job ID and heap pointer to be the same number.

I didn’t push this all the way back to Eio, but I did update all the tests and documentation, and I have to say the results looked nice to me (and still low-level) - in particular, the copying tests seem to get a clearer delineation of error path. I’m not sure if the separation of unit/int was strictly buying much, and given that various of the error codes which may want to come aren’t Posix and so map to Unix.UNKNOWNERR (which allocates), it would possibly be worth looking at a richer error type which could itself be converted to a Unix.error expensively, but on-demand. However, what was nice was that the return from the wait from functions still allocates just one two-field OCaml word, and the file descriptor - if there is one - incurs no further conversion, the value comes from C with the correct types already.

Entertainingly, the final version of the patch, while breaking the API, does not actually need any alterations to OCaml’s Unix library at all.

There’s one last possibly fun OxCaml follow-up to look at. The value which comes back from the C wait functions is a cqe_option (see above) whose shape is a single constant constructor and four 2-field constructors. The first field in these cases is the io_uring user-data, which we then translate back to the OCaml value to yield one of these:

type 'a completion_option =
  | None
  | Unit of { result: unit; data: 'a }
  | Int of { result: int; data: 'a }
  | FD of { result: Unix.file_descr; data: 'a }
  | Error of { result: Unix.error; data: 'a }

Which has the same shape! Now, we could do an evil Obj.magic trick here to re-use the block and simply replace the field which contained the io_uring user data with the OCaml value (we’d need to swap the fields around in the types, but that’s a minor detail). However, this kind of lying can be dangerous when the OCaml optimiser comes into play… but can the uniqueness modes be used here to allow the compiler to determine that the cqe_option block is now available and actually compile it down to just that field assignment? We’d then halve the allocations arising from a ring wait operation 🤔

Anyway, file descriptors are not integers. I’m a new disciple. I think we should make T-Shirts.

Various of the syscalls available in io_uring return Unix file descriptors, and the design of ocaml-uring as a low-level interface to it leads to some slightly unfortunate recommendations in its instructions:

# let fd =
    if result < 0 then failwith ("Error: " ^ string_of_int result);
    (Obj.magic result : Unix.file_descr);;
val fd : Unix.file_descr = <abstr>

Anil was wondering if we could dust off some of the code in an old failed OCaml pull request of mine (ocaml/ocaml#1990) to get rid of some of the magic. Pulling out the C code from that PR wasn’t entirely mechanical, and Anil wondered if we might put it in a library. As it happened, there’d been some discussions about the PR at developer meetings, and there was a consensus that we ought to have an official C API for converting a C file descriptor (i.e. an int) to an OCaml Unix.file_descr. I wasn’t that keen on producing an external library without sorting out the compiler’s library at the same time, so I figured I’d dust that change off and see where it went.

That original PR attempted to add primitives to the Unix module to allow OCaml code to convert an OCaml int to Unix.file_descr, so the ocaml-uring example would instead just be Unix.descr_of_fd result with no Obj.magic. However, that approach had an absolute veto from Xavier:

“File descriptors are not integers”

I started off on this little rabbit-hole of changes agreeing philosophically, but not really agreeing, but ended up in total agreement, and ending with the feeling that the fact they are integers - and that OCaml treats integers specially - encourages possibly poorer library design.

On Unix, a Unix.file_descr is just the OCaml representation of the C int, but it’s absolutely not that on Windows, where the implementation is much more complicated. So the Obj.magic “trick” is a route to a segfault on Windows. That’s obviously not important for a Linux-only library like ocaml-uring, but Obj.magic is a wart. The Windows complexity exists because we have both CRT file descriptors (which are just C ints, the same as on Unix) and also OS file handles (which are Win32 HANDLEs - a pointer). There’s some added book-keeping complexity needed, but that’s not important right now. The key thing is that we have a notion of an “Operating System” file descriptor and a “C Runtime Library” file descriptor. On Unix, they happen to be the same thing; on Windows, they’re not. On Windows, given one, it is always possible to obtain the other (the functions _get_osfhandle and _open_osfhandle are provided for this.

That gives a straightforward portable C API: caml_unix_file_descr_of_os which takes an int (on Unix) or a HANDLE (on Windows) and returns an OCaml Unix.file_descr representing it. It’s similarly straightforward to provide caml_unix_file_descr_of_fd and caml_unix_fd_of_file_descr for getting the CRT file descriptor on both, and we then have the portable primitives required (caml_unix_os_of_file_descr is not necessary, but the reasons are left as an exercise for the avid portable C stub code author).

To this, I then added a quick commit to solve another issue in this area, which is tracked in ocaml/ocaml#9052. While there’s the veto on converting a Unix.file_descr to an int, being able to debug the values in logs and so forth is useful.

At this point, the scab of this old PR well and truly picked, I paused and thought a bit about the original problem I’d been trying to solve in #1990, and which remained not-entirely-satisfactorily solved. It’s also described in ocaml/ocaml#6948, and while looking through it, I realised some of my previous work was related to this. In particular, that passing specific file descriptors from one process to another is not a Unix-specific operation, and you can do it on Windows as well, it’s just less common (i.e. you can start a Windows process with file descriptor 3 connected to a control channel if you want, just as you can on Unix).

Which got me thinking some more about having an OCaml function for this, and about “file descriptors are not integers”, and I realised that while this is mostly true, it’s not always true. For a start, there are three well-known values, 0 for “input”, 1 for “output”, and 2 for “logging”, known as the “Standard Input, Output and Error” handles. These are exposed in the Unix library as Unix.stdin, Unix.stdout and Unix.stderr (and in the Standard Library itself, for the channels API), and you can also specify them when spawning processes. Stepping back, let’s consider if instead of treating file descriptors as int, Thompson and Ritchie had instead used opaque pointers¹ for open, read, and so forth in the first version of Unix.

In this hypothetical scenario, virtually all C programs would be just fine: STDIN_FILENO and so forth would still be there, and virtually all C code using file descriptors doesn’t actually care about the precise value. However, the implementation would face the same issue as we have in OCaml when trying to spawn other processes if we needed to pass file descriptors to a process. At that point, it dawned on me: “file descriptors are indeed not integers, but at program startup, there is a partial map from integers to file descriptors”. Most processes are created with at least 0, 1 and 2 added to that map, and they can then be retrieved with Unix.stdin/STDIN_FILENO, etc. Given that Windows does support the notion of passing additional file descriptors, that means that this operation is portable, and the inability to pass abstract file descriptors using integers seems like a gap. This fits with a TODO item I noted in April: once Unix.create_process et al are correctly inheriting file descriptors on Windows, it makes even more sense to be able to do this too.

#6948 proposed adding a file_descr list argument to functions like create_process, but that’s the wrong API here. It assumes that each descriptor is being passed using its current file descriptor number, which is wrong for two reasons. Firstly, it breaks the abstraction (we have just treated a Unix.file_descr as an int). Secondly, as noted in the C API above, if the Unix.file_descr was created from an OS file descriptor on Windows, it doesn’t necessarily have a CRT descriptor associated with it. The fix is easy: pass a (int * file_descr) list. We preserve the abstraction, and specify the mapping.

With this extra list, when we spawn a process, we now have the ability to pass, say, the input portion of a pipe (created with Unix.pipe) as file descriptor 3. How can we pick that up on the OCaml side? As it happens, on Windows we really could trivially enumerate them, and so have Unix.startup_file_descrs : (int * file_descr) list or some such, but there’s no way to do that portably on Unix. For this specific case, there is a concrete reason to have a function int -> Unix.file_descr (indeed, this is exactly what the Unix library does internally to create stdin, stdout and stderr). However, there is no case for a function Unix.file_descr -> int.

What to call this function, therefore? Having file_descr_of_fd (as in C) but not fd_of_file_descr seems very strange. But what does the type tell us? We’re taking an int - a low-level descriptor and returning a file_descr - a high-level descriptor. C already has two levels - the stream API (“FILE *”) and provides a function to go from the low level to the stream API. It’s fdopen. The C streams API isn’t really about abstraction, so C also has a function to go the other way, fileno, but the nice property is that fdopen and fileno do not sound like reverse operations.

Fixing Unix.create_process on Windows is definitely a job for another day, but adding Unix.fdopen is easy, so I did that. The three commits are sat in dra27/ocaml#fdopen. Next up, applying all this to ocaml-uring to get rid of the magic…

I was still (very) curious at the latent remaining bug in Lucas’s excellent work. There were some corners which had been cut in the prototype, and I had a brief foray into this problem, with a view this time to ensuring artefact equivalence between what OCaml’s build system would produce and what our altered driver program was doing.

If you have a pre-built compiler and a clean (of binary artefacts) OCaml source tree, you can actually build the bytecode compiler in just three, ahem, short commands (I’m intentionally glossing over all the generated source files):

$ ocamlc -I utils -I parsing -I typing -I bytecomp -I file_formats -I lambda -I middle_end -I middle_end/closure -I middle_end/flambda -I middle_end/flambda/base_types -I driver -I runtime -g -strict-sequence -principal -absname -w +a-4-9-40-41-42-44-45-48 -warn-error +a -bin-annot -strict-formats -linkall -a -o compilerlibs/ocamlcommon.cma utils/config.mli utils/build_path_prefix_map.mli utils/format_doc.mli utils/misc.mli utils/identifiable.mli utils/numbers.mli utils/arg_helper.mli utils/local_store.mli utils/load_path.mli utils/profile.mli utils/clflags.mli utils/terminfo.mli utils/ccomp.mli utils/warnings.mli utils/consistbl.mli utils/linkdeps.mli utils/strongly_connected_components.mli utils/targetint.mli utils/int_replace_polymorphic_compare.mli utils/domainstate.mli utils/binutils.mli utils/lazy_backtrack.mli utils/diffing.mli utils/diffing_with_keys.mli utils/compression.mli parsing/location.mli parsing/unit_info.mli parsing/asttypes.mli parsing/longident.mli parsing/parsetree.mli parsing/docstrings.mli parsing/syntaxerr.mli parsing/ast_helper.mli parsing/ast_iterator.mli parsing/builtin_attributes.mli parsing/camlinternalMenhirLib.mli parsing/parser.mli parsing/pprintast.mli parsing/parse.mli parsing/printast.mli parsing/ast_mapper.mli parsing/attr_helper.mli parsing/ast_invariants.mli parsing/depend.mli typing/annot.mli typing/value_rec_types.mli typing/ident.mli typing/path.mli typing/type_immediacy.mli typing/outcometree.mli typing/primitive.mli typing/shape.mli typing/types.mli typing/data_types.mli typing/rawprinttyp.mli typing/gprinttyp.mli typing/btype.mli typing/oprint.mli typing/subst.mli typing/predef.mli typing/datarepr.mli file_formats/cmi_format.mli typing/persistent_env.mli typing/env.mli typing/errortrace.mli typing/typedtree.mli typing/signature_group.mli typing/printtyped.mli typing/ctype.mli typing/out_type.mli typing/printtyp.mli typing/errortrace_report.mli typing/includeclass.mli typing/mtype.mli typing/envaux.mli typing/includecore.mli typing/tast_iterator.mli typing/tast_mapper.mli typing/stypes.mli typing/shape_reduce.mli file_formats/cmt_format.mli typing/cmt2annot.mli typing/untypeast.mli typing/includemod.mli typing/includemod_errorprinter.mli typing/typetexp.mli typing/printpat.mli typing/patterns.mli typing/parmatch.mli typing/typedecl_properties.mli typing/typedecl_variance.mli typing/typedecl_unboxed.mli typing/typedecl_immediacy.mli typing/typedecl_separability.mli lambda/debuginfo.mli lambda/lambda.mli typing/typeopt.mli typing/typedecl.mli typing/value_rec_check.mli typing/typecore.mli typing/typeclass.mli typing/typemod.mli lambda/printlambda.mli lambda/switch.mli lambda/matching.mli lambda/value_rec_compiler.mli lambda/translobj.mli lambda/translattribute.mli lambda/translprim.mli lambda/translcore.mli lambda/translclass.mli lambda/translmod.mli lambda/tmc.mli lambda/simplif.mli lambda/runtimedef.mli file_formats/cmo_format.mli middle_end/internal_variable_names.mli middle_end/linkage_name.mli middle_end/compilation_unit.mli middle_end/variable.mli middle_end/flambda/base_types/closure_element.mli middle_end/flambda/base_types/var_within_closure.mli middle_end/flambda/base_types/tag.mli middle_end/symbol.mli middle_end/flambda/base_types/set_of_closures_id.mli middle_end/flambda/base_types/set_of_closures_origin.mli middle_end/flambda/parameter.mli middle_end/flambda/base_types/static_exception.mli middle_end/flambda/base_types/mutable_variable.mli middle_end/flambda/base_types/closure_id.mli middle_end/flambda/projection.mli middle_end/flambda/base_types/closure_origin.mli middle_end/clambda_primitives.mli middle_end/flambda/allocated_const.mli middle_end/flambda/flambda.mli middle_end/flambda/freshening.mli middle_end/flambda/base_types/export_id.mli middle_end/flambda/simple_value_approx.mli middle_end/flambda/export_info.mli middle_end/backend_var.mli middle_end/clambda.mli file_formats/cmx_format.mli file_formats/cmxs_format.mli bytecomp/instruct.mli bytecomp/meta.mli bytecomp/opcodes.mli bytecomp/bytesections.mli bytecomp/dll.mli bytecomp/symtable.mli driver/pparse.mli driver/compenv.mli driver/main_args.mli driver/compmisc.mli driver/makedepend.mli driver/compile_common.mli utils/config.ml utils/build_path_prefix_map.ml utils/format_doc.ml utils/misc.ml utils/identifiable.ml utils/numbers.ml utils/arg_helper.ml utils/local_store.ml utils/load_path.ml utils/clflags.ml utils/profile.ml utils/terminfo.ml utils/ccomp.ml utils/warnings.ml utils/consistbl.ml utils/linkdeps.ml utils/strongly_connected_components.ml utils/targetint.ml utils/int_replace_polymorphic_compare.ml utils/domainstate.ml utils/binutils.ml utils/lazy_backtrack.ml utils/diffing.ml utils/diffing_with_keys.ml utils/compression.ml parsing/location.ml parsing/unit_info.ml parsing/asttypes.ml parsing/longident.ml parsing/docstrings.ml parsing/syntaxerr.ml parsing/ast_helper.ml parsing/ast_iterator.ml parsing/builtin_attributes.ml parsing/camlinternalMenhirLib.ml parsing/parser.ml parsing/lexer.mli parsing/lexer.ml parsing/pprintast.ml parsing/parse.ml parsing/printast.ml parsing/ast_mapper.ml parsing/attr_helper.ml parsing/ast_invariants.ml parsing/depend.ml typing/ident.ml typing/path.ml typing/primitive.ml typing/type_immediacy.ml typing/shape.ml typing/types.ml typing/data_types.ml typing/rawprinttyp.ml typing/gprinttyp.ml typing/btype.ml typing/oprint.ml typing/subst.ml typing/predef.ml typing/datarepr.ml file_formats/cmi_format.ml typing/persistent_env.ml typing/env.ml typing/errortrace.ml typing/typedtree.ml typing/signature_group.ml typing/printtyped.ml typing/ctype.ml typing/out_type.ml typing/printtyp.ml typing/errortrace_report.ml typing/includeclass.ml typing/mtype.ml typing/envaux.ml typing/includecore.ml typing/tast_iterator.ml typing/tast_mapper.ml typing/stypes.ml typing/shape_reduce.ml file_formats/cmt_format.ml typing/cmt2annot.ml typing/untypeast.ml typing/includemod.ml typing/includemod_errorprinter.ml typing/typetexp.ml typing/printpat.ml typing/patterns.ml typing/parmatch.ml typing/typedecl_properties.ml typing/typedecl_variance.ml typing/typedecl_unboxed.ml typing/typedecl_immediacy.ml typing/typedecl_separability.ml typing/typeopt.ml typing/typedecl.ml typing/value_rec_check.ml typing/typecore.ml typing/typeclass.ml typing/typemod.ml lambda/debuginfo.ml lambda/lambda.ml lambda/printlambda.ml lambda/switch.ml lambda/matching.ml lambda/value_rec_compiler.ml lambda/translobj.ml lambda/translattribute.ml lambda/translprim.ml lambda/translcore.ml lambda/translclass.ml lambda/translmod.ml lambda/tmc.ml lambda/simplif.ml lambda/runtimedef.ml bytecomp/meta.ml bytecomp/opcodes.ml bytecomp/bytesections.ml bytecomp/dll.ml bytecomp/symtable.ml driver/pparse.ml driver/compenv.ml driver/main_args.ml driver/compmisc.ml driver/makedepend.ml driver/compile_common.ml
$ ocamlc -I utils -I parsing -I typing -I bytecomp -I file_formats -I lambda -I middle_end -I middle_end/closure -I middle_end/flambda -I middle_end/flambda/base_types -I driver -I runtime -g -strict-sequence -principal -absname -w +a-4-9-40-41-42-44-45-48 -warn-error +a -bin-annot -strict-formats -a -o compilerlibs/ocamlbytecomp.cma bytecomp/bytegen.mli bytecomp/printinstr.mli bytecomp/emitcode.mli bytecomp/bytelink.mli bytecomp/bytelibrarian.mli bytecomp/bytepackager.mli driver/errors.mli driver/compile.mli driver/maindriver.mli bytecomp/instruct.ml bytecomp/bytegen.ml bytecomp/printinstr.ml bytecomp/emitcode.ml bytecomp/bytelink.ml bytecomp/bytelibrarian.ml bytecomp/bytepackager.ml driver/errors.ml driver/compile.ml driver/maindriver.ml
$ ocamlc -I utils -I parsing -I typing -I bytecomp -I file_formats -I lambda -I middle_end -I middle_end/closure -I middle_end/flambda -I middle_end/flambda/base_types -I driver -I runtime -g -compat-32 -o ocamlc -strict-sequence -principal -absname -w +a-4-9-40-41-42-44-45-48 -warn-error +a -bin-annot -strict-formats compilerlibs/ocamlcommon.cma compilerlibs/ocamlbytecomp.cma driver/main.mli driver/main.ml

I wanted to try a different angle on the Load_path, and this time produced a function which predicts the files in the tree. The rules for this were pretty easy for me to define, and I wasn’t sure I could face watching Claude special-case everything. 130 lines of verifiably correct hacked OCaml later, I had my load path function. A little bit more code later, those three commands above were translated into an OCaml script (based on the ocamlcommon and ocamlbytecomp libraries) which should exactly the same build. It ran - and it built the compiler.

ocamlc was, pleasingly, exactly the same. The .cma files, however, were not. For ocamlcommon.cma, that turned out to be me being sloppy with my commands. ocamlcommon.cma is linked with -linkall, but ocamlc -a foo.cma -linkall bar.cmo is not the same as ocamlc -a foo.cma -linkall bar.ml, because -linkall gets recorded in the .cmo file as well. Easy fix - but the files were still different. A bit more tweaking and I could see that actually the .cmo files were different.

A bit more poking and checking with ocamlobjinfo and a few other flags and tricks, and I observed that:

$ ocamlc -g -c utils/config.ml

resulted in slightly different debug information from:

$ console -g -c utils/config.mli utils/config.ml

(it’s observably to do with the debug information - omit the -g and they’re all identical). Lots to suspect here, but time for…

$ claude
╭───────────────────────────────────────────────────╮
│ ✻ Welcome to Claude Code!                         │

The problem was easy to state, but not quite so quick to come up with a conclusive explanation. Claude, like most of these models, appears not to have been trained on this old cartoon, and very merrily buzzes along for a few rounds of investigation, followed by a highly dubious explanation for how it was probably something to do with marshalling and, mumble mumble, the final binaries are the same so this bug is probably OK.

Hmm. A few rounds of, “no, this needs to be equivalent as otherwise it’s not reproducible” (“You’re so right!”), and we had a lot of test programs, a frequent need for reminders that debugging OCaml’s Marshalling format was possibly not going to help, but we weren’t very much closer to an answer.

Stepping back, I re-framed the problem, instead asking Claude to produce a program which would give a textual dump of the debug information in each file, so we could compare it. This was interesting - especially the occasional hallucinations at having analysed “all the fields”, but we got there.

What was interesting was that we were struggling to perceive differences between anything. Claude at this point was desperate to delve into the runtime code and start doing hex-dumps of the marshal format to see what was actually different. I appear to be a little older than Claude, and was more reticent about this approach. I suggested we look at the polymorphic hash of some of these fields instead. At this point, we started to see some differences - Claude’s inferences at this point were working well, and there was a strong suggestion to add all sorts of accessor functions into the Types module to be able to introspect some of the values in more detail than normally intended (i.e. polymorphic hash was telling that us that some abstract values were different, but we wanted to see what the differences really were).

Reader, I told it to use Obj.magic instead 🫣

However, what happened next was truly fascinating and definitely very efficient. The value being returned for one of the type IDs was simply not believable. It was far too high. Claude also correctly observed that it was in fact a block, and not an integer, which was what we were expecting. The human brain at this point cuts in, and looks at the type: Types.get_id: t -> int. No, that accessor looks right. Brain slowly whirring; look at the code:

let get_id t = (repr t).id

Oh - it’s not an accessor (in another life, I could possibly have performed Claude’s responses…).

All I had to point out was that Types.get_id was not an accessor, it was normalising the result (to walk Tlink members of the type representation), and Claude was on it, replacing semi-elegant OCaml code with a sea of calls to Obj functions.

But we had our answer - the type chain was different, if semantically equivalent and, more importantly, Claude then leaped to the problem.

The internal Types.new_id reference isn’t reset between compilations 💥

A quick rebuild later, and the same debug information was given regardless of whether utils/config.mli was compiled at the same time as utils/config.ml. Go Claude. My contribution was keeping the explorations looking at relevant parts of the system, and not disappearing off on sometimes ridiculous and unbelievable tangents. Maybe it would have got there on its own, but who knows the tokens required and the GPUs scorched…

Plug that back into my little script. ocamlcommon.cma still different. At this point, a line from Four Weddings and a Funeral could be heard loud and clear in the human mind. It’s the one which follows “Dear Lord, forgive me for what I am about to, ah, say in this magnificent place of worship…”.

The fix was definitely working. But a quick bit of further experimentation revealed that including other .mli files before utils/config.ml (and there are a lot) was causing the information to change.

So:

$ claude -c

As a human of hopefully normal emotional response to situations, the feeling of being back at square one would normally have meant I’d have at least needed a coffee before being able to face dusting off all the tools and scripts which had been constructed in the previous investigations. But here of course the LLM doesn’t care and was straight into using the tools previously constructed to look at the revised problem. A lot more Obj.magic-like investigations later looking at the shape of some debugging information, and Claude found another bit to reset, this time in Ctype. All the level information in the type-checker isn’t reset between compilations. Not a semantic issue, because the type checker uses those numbers relatively, but again they leak into the representation of some of the debugging information.

And it was working 🥳

Next up was trying to put those fixes into something resembling a commit series that might one day be an acceptable PR. What I really wanted was a test. Claude was great for this, although it lacks anything approximating taste (and this is me writing…!). However, with no feelings to be hurt, the pointers were easy to issue and the results impressive - especially constructing a non-trivial ocamltest block. The result is previewable in dra27/ocaml#237 on my GitHub fork, and the test is entirely Claude’s.

Having got to this stage, I extended the compiler with some of Lucas’s patches, and started passing just the .ml files for compilation, allowing the compiler to compile the .mli files on demand, as before. With some idle tinkering, I got to the end of “coreall”, which is the point in OCaml’s build process where ocamlc, the bytecode versions of everything in tools/ and ocamllex have all been compiled, along with the Standard Library. That was all being done from a single compiler process, where the OCaml script driving the compiler consisted mostly of the list of .ml files. Coupled with the predictive load path I’d already put together, at this stage the “plumbing” needed in the scheduler is just:

let compile_file source_file () =
  Compenv.readenv Format.std_formatter (Before_compile source_file);
  let output_prefix = Compenv.output_prefix source_file in
  if Filename.extension source_file = ".mli" then
    Compile.interface ~source_file ~output_prefix
  else
    let start_from = Clflags.Compiler_pass.Parsing in
    Compile.implementation ~start_from ~source_file ~output_prefix

let rec execute task =
  try task ()
  with effect (Load_path.Missing path), k ->
    let file = Filename.chop_extension path ^ ".mli" in
    execute (compile_file file);
    execute (Effect.Deep.continue k)

(as an aside, when it goes to being done with Domains I’ll possibly switch it to a shallow handler, because the call stack with the deep handlers isn’t as reasonable as I’d hoped for, but to be honest I just wanted to see it work!)

Fascinatingly, all the artefacts (.cma and binaries) being produced were identical except for the Lazy module in the Standard Library!

$ claude -c

Claude was simultaneously amazing and useless at this. Amazing, because I was prompting some of this while cooking a meal, so being able to bark an instruction (actually, I hadn’t set it up for voice - I was just quickly typing) and then leave it to think for a minute or two was strangely efficient, because investigating this on my own would have taken too much continuous concentration. It was useless because we didn’t get anywhere near a believable explanation, despite various efforts at resetting things. Sometimes you just have to say /exit (and eat a meal…).

However, after the aforementioned meal, I dug into it a bit further. The issue here was clearly to do with some state in the compiler - if ocamlcommon.cma or ocamlmiddleend.cma were compiled, then the Lazy module differed. Incidentally, at this point this wasn’t debug information which varied, it was the actual module, but it was still semantically the same. Claude had correctly identified that it was to do with the marshalling, and we had identified that there was a difference in string sharing (so not entirely useless, in fairness). I carried on poking and, with a little bit of jerry-rigging, managed to determine the relatively small set of files in flambda and in ocamlcommon whose compilation caused the change in Lazy. I was highly suspicious it was to do with compilation of lazy values.

$ claude -c

Feeding this information to Claude was a much better trick - the reasoning at this point would contradict its own tangents (“I should look at … but wait, the user has given me the list of affected files”). Impressively, we did hone in on the much more complex explanation for this third issue, which is to do with lazy values used in globals in the Matching module. In this particular case, if the compiler has compiled a file which matched on a lazy, causing Matching.code_force_lazy_block to be forced in the compiler and thus the CamlinternalLazy identified to be added to the current persistent environment, then a subsequent module (in this case lazy.ml in the Standard Library) which both pattern matches on a lazy and which also refers to CamlinternalLazy ends up with two extern’d string representations of CamlinternalLazy instead of one. The reason is that the forced code block in Matching still refers to a string used in a previous persistent environment. It’s not a semantic issue at all, but it manifests itself because the string is not shared when the subsequent file looks up the CamlinternalLazy identifier.

It was a battle to update the test to show this behaviour, but in fairness that would have been a battle anyway! However, we got there too.

Three reproducibility issues identified, and a viable PR produced - with tests!

Lucas got to grips pretty swiftly with OCaml’s build system, and initially looked at generalising a core internal part of the compiler called the Load_path. This is used by the compiler for scanning the various “include” directories for files, principally typing information. For example, if your code contains a call to Unix.stat, then the type checker needs the typing information for a module called Unix which will cause it to request unix.cmi from the Load_path and which will then hopefully resolve that to, say, ~/.opam/switch/lib/ocaml/unix/unix.cmi.

Effects provide an elegant way of inverting the control for this lookup, as the program calling the compiler can then change the way these files are looked up. It also provides the opportunity to “lie” to the compiler about the files which are actually present, and this was the first thing Lucas started to do with this change. In particular, it allows us to ignore the dependency graph. When compiling a module, OCaml requires all the type information that a module refers to have been compiled beforehand. If you have a module in bar.ml with interface in bar.mli and where the code refers to Foo.value, then OCaml requires foo.mli and bar.mli both to have been compiled before bar.ml is compiled. However, thanks to this effectful trick, Lucas could instead allow the compiler to start with just bar.ml. When Foo.value is encountered, there’s a request made for foo.cmi, at which point, in the first prototype, the compiler then quickly spawned another instance of itself to compile foo.mli and then resumed compilation for bar.ml, with the same trick then happening at the end of the compilation with bar.cmi. i.e. three files (foo.mli, bar.mli and bar.ml) all compiled just from ocamlc -c bar.ml.

Possibly neat for being able to remove monstrosities like this from OCaml’s source tree one day, but so far not so exciting. However, effects give us more than just hooks into the compiler’s operations. We’ve got an entire suspended compilation packaged up in a continuation… which means that that same compiler “process” can now do something else. The next trick was to have it that instead of spawning a new compiler, the current process itself returned back into the compiler and itself compiled the required interface file and then simply resumed the continuation of the previous filke. At this point, the 30-year-old codebase rears its head again. For reasons of speed and space, many parts of the compiler, especially in the type checker, feature a lot of global mutable state. In particular, the compilation pipeline is not re-entrant. Luckily, thanks to the Merlin project, there is a mechanism in the type-checker for taking snapshots of all this global state. Lucas was able to piggy-back on this so that, just before the compiler performs an effect to request a .cmi file (that doesn’t yet exist), it snapshots all its global state, performs the effect and then, when resumed, restores that state again.

Using this to interrupt type-checking and start on something else isn’t quite what this Local_store mechanism was originally intended for, and there was a bit of debugging to find a few more pieces global state which weren’t being “registered”, but Lucas was able to get a means of building the OCaml bytecode compiler with nothing pre-compiled where all the compiler had to be given was the list of .ml files required. From a toolchain perspective, we’re essentially retiring ocamldep.

So far, still mostly just so neat: one single compiler process (just about) successfully recompiling the compiler. However, that’s equivalent to compiling with make -j1 - a sequential, and therefore slow, build. The awesome part came next - Domains. In the final version Lucas was working on, multiple domains were started up, each one beginning compilation of one of the .ml files required for the compiler in parallel, with a scheduler handling effects coming from each of these in turn when .mli files needed compiling, and despatching those. The Local_store mechanism in the came in handy here - Lucas extended it to use Domain Local Storage, combined with the snapshotting. The prototype - for simplicity - featured no sharing between these domains.

By the end of the summer, this was very nearly working, which is a result consiserably further than I’d expected in the time available! As is so often the case with these investigations, Lucas’s work had revealed some new facets to this area that weren’t clear to me before. I had previously been wondering how we would be exposing this kind of multi-threaded compiler to the user via the driver programs, but it became increasingly clear that this wasn’t something that would be necessary - the program that we were working on to build the compiler itself was of course not the compiler driver, but a build system. To me, there are two particularly exciting things about that:

1. It’s a really simple build system. Hopefully when the last few kinks in the parallel type checker are ironed out (read on…), we may be able to add that it’s really simple and performant.
2. It’s fundamental portable. It leads to the possibility of bootstrapping OCaml trivially with itself. This has been done before with ocamlbuild, but the result was a maintenance disaster. However, the sheer simplicity of the multi-domain effect-scheduling approach is making this perennial build system hacker tinker…