dryad

An almost-parallel, semi-functioning, dynamic linker experiment, written in Rust

2 years after

Welcome

Build Status Floobits Status https://gitter.im/m4b/dryad

dryad

dryad is the first and only parallel, 64-bit ELF dynamic linker for GNU/Linux, written from scratch in Rust, and is:

  1. not parallel
  2. not ready for production
  3. a prototype
  4. doesn't really work
  5. in a massive state of flux
  6. parallel might be a) impossible, b) not performant, but it will be interesting to try

but all most of these things will disappear in time!

Build

NOTE: I am early adopting the new rustup tool, and also now use a Makefile + experimental Cargo.

Of course, you will need your typical build tools on a linux system, essentially:

  • gcc (or clang)
  • ld (or ld.gold)
  • curl
  • an internet connection
  • an x86-64 GNU/Linux box

Unfortunately, I currently do not support cross compiling at the moment (which is an unusual use case anyway), so you will need an x86-64 GNU/Linux machine, otherwise it will fail.

Once that's settled, you have two options, both fairly easy.

Setup - Easy

Just run ./setup.sh, which will download and install the latest rust nightly and musl target into ~/.multirust.

You can then proceed as normal:

  1. ./gen_tests.sh - builds the test binaries (do this once) (will add this as a make target soon)
  2. make - compiles dryad.so.1 and copies it to /tmp
  3. test/test - runs the test binary test, whose PT_INTERPRETER is /tmp/dryad.so.1
  4. make run - runs ./dryad.so.1, this should run correctly without segfaulting, please file a bug if it does not.

Setup - Use Nightly Rust, slightly less easier

If you don't want to use rustup, then setting up a build environment will require a little work: you will need a nightly rustc, a x86_64-unknown-linux-musl target, and the ability to edit a Makefile to change the environment variables.

If the latest rustc is not installed, just download and use the latest version of rustup script to install the latest nightly with the musl target enabled, i.e. something like:

curl -sSf https://static.rust-lang.org/rustup.sh > rustup.sh
chmod +x rustup.sh
sudo ./rustup.sh --channel=nightly --with-target=x86_64-unknown-linux-musl

And then edit the appropriate variables in the Makefile to use /usr/local as the $PREFIX instead of rust, and probably some other things I'm forgetting about.

You should be able to then compile and run as normal.

Compilation and Linking Requirements

The Makefile does four things:

  1. compiles the x86-64 asm stubs which dryad needs (change the gcc call to clang here if you like) gcc -fPIC -c -o start.o src/arch/x86/asm.s
  2. compiles dryad into an object file: rustc --target=x86_64-unknown-linux-musl src/lib.rs -g --emit obj -o dryad.o
  3. links the asm stubs with dryad and then the rust standard libs, and pthreads and libc and etc., and provides the very important linker flags such as -pie, -Bsymbolic, -I/tmp/dryad.so.1, -soname dryad.so.1, etc.
  4. copies the resulting binary, dryad.so.1, into /tmp/dryad.so.1 because that's what PT_INTERPRETER is set to in the test binaries. In the future we'll obviously make this /usr/lib/dryad.so.1, or wherever the appropriate place for the dynamic linker is (GNU's is called ld-linux-x86-64.so.2 btw).

Really, stage 1 and 3 from above is the problem in the cargo pipeline, which is why I still need to manually link. Additionally, rustc doesn't like to compile a musl binary as a shared object.

I believe some of these issues will go away if I transfer the start assembly into inline assembly in Rust source code (thereby potentially eliminating step 1), but the musl issue could be a problem.

Running

The last step, running test/test (or any of the other test binaries in test), will output a ton of information and then segfault your machine, or perhaps not run at all, or really do any number of things --- I really can't say, since I've only tested on a single machine so far.

NOTE: if you're on Ubuntu or another linux distro which doesn't place libc in /usr/lib, you'll need to pass LD_LIBRARY_PATH=/path/to/libc to your test/test, i.e.: LD_LIBRARY_PATH=/path/to/libc test/test. Furthermore, if libc doesn't have symbolic links for the soname pointing to the actual binary, or the actual binary is installed as the soname, then it also won't work. We need ld.so.cache reader and parser for this - feel free to work on it!

However, dryad is almost capable of interpreting a (simple) binary (like test/test) which uses libc.so.6.

Specifically, this means is that dryad at a high level does the following:

  1. relocates itself
  2. loads and mmap's all binaries in the flattened dependency list
  3. relocates every loaded binary (technically, relocates a subset of the most common relocation symbols)
  4. sets up each binary's GOT with its runtime symbol resolution function (_dryad_resolve_symbol), and its "rendezvous" data structure
  5. resolves GNU ifuncs, and if LD_BIND_NOW is set, prebinds all function symbols.
  6. passes control to the executable
  7. (optionally, if LD_BIND_NOW is not set) lazily binds function calls
  8. segfaults

There are several major, and many minor tasks that need to be finished to be even remotely "complete". The first and most major one is properly setting up TLS. Currently, it hacks it about by just calling the musl symbol __init_tls so we don't segfault on fs:0 accesses and their ilk.

But it really needs to be properly setup, as it's a delicate procedure.

This is easily the least documented part of the entire dynamic linking process I have come across, so work is slow going. Also there are some questions about how this will work exactly, which I'll detail at some other time, or in a blog post.

Lastly, dryad should be capable of interpreting itself, which you can verify by invoking ./dryad.so.1 (yes, dryad is it's own program interpreter).

Project Goals

1. Documenting a Dynamic Linker

The primary goal of this project is to completely document:

  1. the dynamic linking process on an GNU/Linux ELF x86-64 system
  2. an implementation of such a process

The current state of documentation and information on this subject is an embarassment, and I'm continually appalled at the lack of materials, documentation, etc. I've jokingly told people I'm worried what will happen when all the old C programmers die - but I'm not really joking.

Code is not documentation. If it were, then this project would have been easy and finished some time ago.

As such, I hope to thoroughly document the implementation, the process, and maybe even my experiences.

I will be updating this section with more content shortly, please bear with me.

2. Implementing a Dynamic Linker

The current target implementation for dryad is an ELF x86-64 GNU/Linux system.

This is important to note:

  1. The ELF loader only supports the 64-bit variant
  2. The asm assumes an x86-64 instruction set
  3. The linker currently targets Linux, although this need not be set in stone.

I would like to have a working implementation for an ELF x86-64 GNU/Linux target before/if beginning work on other architectures or systems.

That being said, in particular I'm not very interested in porting dryad to work on 32-bit Linux systems, because:

  1. 32-bit systems are in obsolescence in my opinion
  2. Will significantly complicate the ELF target in the source code, as cfg flags would be needed depending on what target we want to switch at build time, etc.
  3. 32-bit ELF dynamic linking is much better documented, and I want to document a 64-bit dynamic linker

Contributing

Contributions wholeheartedly welcome! Let's build a production dynamic linker in Rust for use in x86-64 GNU/Linux systems (and beyond)! Or not, that's cool too.

If you don't know anything about dynamic linking on x86-64 GNU systems for ELF, that's totally OK, because as far as I can tell, no one really does anymore. Here are some random resources if you're curious:

  1. The ELF specification
  2. x86-64 System V Application Binary Interface
  3. ELF TLS spec
  4. google's bionic dynamic linker source code
  5. glibc dynamic linker source code
  6. musl dynlink.c code
  7. sco dynamic linking document
  8. iecc dynamic linking article
  9. ELF loading tutorial
  10. Info on the GOT[0] - GOT[2] values
  11. man ld-so for dynamic linking basics
  12. man dlopen for runtime dynamic linking basics
  13. man 3 getauxval for information on the auxiliary vector passed by the kernel to programs
  14. I'll also hopefully add a couple articles on some of my _mis_adventures on my essentially defunct blog

TODOs

Here are some major todos off the top of my head

  1. MAJOR: properly init dynamic linker's TLS: it's the final countdown.
  2. MAJOR: dlfcn.h implementation and shared object bindings for runtime dynamic loading support
  3. MINOR: /etc/ld.so.cache loader and parser
  4. better documentation
  5. fix any number of the todos littered across the code
  6. make unsafe code safer with rust best practices; rust experts definitely needed!
  7. add profiling configs
  8. add tests
  9. actually implement dynamic linking without segfaulting
  10. x all the things

Coda

Always remember:

Be excellent to each other

Related Repositories

tensorflow-white-paper-notes

tensorflow-white-paper-notes

Annotated notes and summaries of the TensorFlow white paper, along with SVG figu ...

awesome-research

awesome-research

:seedling: a curated list of tools to help you with the research/life ...

style-tiles

style-tiles

Create HTML and CSS Style Tiles with Sass & Compass ...

zika-data

zika-data

Data — and pointers to data — related to the 2015–16 Zika virus outbreak. ...

ensembler

ensembler

Software pipeline for automating omics-scale protein modeling and simulation set ...


Top Contributors

m4b thepowersgang tbelaire