Shadow Shadow Versions Save

Shadow is a discrete-event network simulator that directly executes real application code, enabling you to simulate distributed systems with thousands of network-connected processes in realistic and scalable private network experiments using your laptop, desktop, or server running Linux.

v3.1.0

4 months ago

Summary

The primary user-facing changes that we've made in this release include:

  • New support for spawning processes inside of a Shadow simulation. Technically, this means it now supports fork, vfork, execve, and other related syscalls.
  • A new experimental TCP stack written in Rust, can be enabled with the experimental command-line flag --use-new-tcp. The stack is not yet recommended for default use because it is still missing important TCP features such as congestion control, but work on it continues.
  • Substantial progress in migrating the shim's C code to no_std Rust code.
  • Numerous Socket API improvements, so that Shadow more accurately follows the behaviour of Linux.
  • Additional migration of C code to Rust (we're now below 20% C code remaining).

More details about specific changes are below. We also have a much more detailed writeup of many of these changes in our most recent discussion post https://github.com/shadow/shadow/discussions/3187

MAJOR changes (breaking):

  • No breaking changes in this release

MINOR changes (backwards-compatible):

  • ERROR-level log lines are now logged to stderr in addition to stdout if stdout is not a tty but stderr is. This helps make errors more visible in the common case that stdout is redirected to a log file but stderr is not. This can currently be disabled via the (unstable) option log-errors-to-tty.
  • Added support for subprocess creation and management.
    • The fork syscall and fork-like invocations of the clone and clone3 syscalls.
    • Process parent pid's, process group IDs, process session IDs, and related syscalls.
    • Child exit signals (e.g. SIGCHLD)
    • The execve syscall.
  • Added Debian 12 (Bookworm) to our supported platforms.
  • Added support for sendmsg, recvmsg, and shutdown for UDP sockets.
  • Added support for MSG_TRUNC and MSG_PEEK as recv syscall argument flags for UDP sockets.
  • Added support for MSG_TRUNC as a recv syscall return flag for UDP and Unix sockets.
  • Added support for the SO_DOMAIN, SO_PROTOCOL, and SO_ACCEPTCONN socket options for TCP and UDP sockets.
  • Added support for the SIOCGSTAMP ioctl for TCP and UDP sockets.
  • Improved the simulation run time performance when there are a large number of active sockets on a single host. (#3238)

PATCH changes (bugfixes):

  • Updated documentation and tests to reflect that shadow no longer requires /dev/shm to be executable. (This requirement was actually removed in v3.0.0)
  • Removed several incorrect libc syscall wrappers. These wrappers are a "fast path" for intercepting syscalls at the library level instead of via seccomp. The removed wrappers were for syscalls whose glibc functions have different semantics than the underlying syscall.
  • Fixed a bug in sched_getaffinity. This bug was previously mostly latent due to an incorrectly generated libc syscall wrapper, though would have affected managed programs that made the syscall without going through libc.
  • Fixed #2681: shadow can now escape spin loops that use an inlined syscall instruction to make sched_yield syscalls.
  • Fixed a deadlock when the managed process calls recv (or similar syscalls) on a TCP or UDP socket with an invalid memory address.
  • Fixed a bug that would allow UDP sockets to accept packets from addresses that aren't the peer address.
  • Fixed an incorrect return value from the FIONREAD ioctl for UDP sockets.
  • Fixed the behaviour of the read and recv syscalls when called with 0-length buffers.
  • Fixed incorrect behaviour (incorrect return value or panic) when connect is called on a listening unix or tcp socket. (#3191)

Full changelog since v3.0.0:

Thanks to @stevenengler, @sporksmith, @robgjansen, @RWails for their contributions to this release!

v3.0.0

11 months ago

Summary

The dev team had accumulated a large set of breaking changes that would require a major version bump. In this release, we have focused on clearing our breaking changes queue and merging those improvements. Because these are breaking changes, this release has bumped our major version from 2 to 3. This release also significantly improves the runtime performance compared to Shadow 2.5.0.

Configuration format

  • Shadow no longer implicitly searches its working directory for executables to be run under the simulation. If you wish to specify a process path relative to Shadow's working directory, prefix that path with ./.

  • Shadow now supports YAML merge keys and extension fields. This allows you to combine YAML maps using the << key.

    Example:

    # an "extension field" that we use to store common host options
x-host-client: &host-client
  bandwidth_up: 10Mbps
  bandwidth_down: 10Mbps
hosts:
  client1:
    # merge the fields from the extension field above
    <<: *host-client
    processes: ...
  client2:
    <<: *host-client
    processes: ...

  • Removed the quantity options for hosts and processes. It's now recommended to use YAML anchors and merge keys instead.

    Shadow 2.x:

    hosts:
  client:
    quantity: 3
    processes: ...

    Shadow 3.x:

    hosts:
  client1: &client
    processes: ...
  # copy all fields from 'client1'
  client2: *client
  # copy all fields from 'client1' and add additional fields
  client3:
    <<: *client
    ip_addr: 152.21.4.24

  • Renamed the host_defaults field to host_option_defaults and renamed the host's options field to host_options.

    Shadow 2.x:

    host_defaults:
  ...
hosts:
  client:
    options:
      ...

    Shadow 3.x:

    host_option_defaults:
  ...
hosts:
  client:
    host_options:
      ...

  • Removed the host pcap_directory configuration option and replaced it with a new pcap_enabled option.

    Shadow 2.x:

    hosts:
  client:
    options:
      pcap_directory: ./

    Shadow 3.x:

    hosts:
  client:
    host_options:
      pcap_enabled: true

  • Host names are restricted to the patterns documented in hostname(7).

  • The process environment configuration option now takes a map instead of a semicolon-delimited string.

    Shadow 2.x:

    hosts:
  client:
    processes:
    - path: curl
      environment: ENV_A=1;ENV_B=foo

    Shadow 3.x:

    hosts:
  client:
    processes:
    - path: curl
      environment:
      - ENV_A: "1"
      - ENV_B: foo

  • The per-process option stop_time has been replaced with shutdown_time. When set, the signal specified by shutdown_signal (a new option) will be sent to the process at the specified time. While shadow previously sent SIGKILL at a process's stop_time, the default shutdown_signal is SIGTERM to better support graceful shutdown.

    Shadow 2.x:

    hosts:
  client:
    processes:
    - path: curl
      stop_time: 10s

    Shadow 3.x:

    hosts:
  client:
    processes:
    - path: curl
      shutdown_time: 10s
      shutdown_signal: SIGKILL

  • A new expected_final_state allows you to specify the expected state of the process at the end of the simulation. The supported states are exited, signaled, or running. If any process is not in the correct state at the end of the simulation, Shadow will return a non-zero exit code. The default expected_final_state is exited with code 0.

    In Shadow 2.x the behaviour was to consider any processes which exited with code 0, OR which were still running at the end of the simulation, as a success. Shadow 3.x does not support this specific behaviour, and you must choose a single state.

    Example:

    hosts:
  server:
    processes:
    - path: nginx
      # we expect nginx to run until the end of the simulation
      expected_final_state: running

  • Added support for a parallelism value of 0, which allows Shadow to choose a reasonable parallelism (we currently use the number of physical cores in Shadow's affinity/cgroup). The default value for parallelism has also been changed from 1 to 0.

  • It is now an error to set a process' shutdown_time or start_time to be after the simulation's stop_time.

  • Sub-second configuration values are now allowed for all time-related options, including start_time, stop_time, etc.

  • Removed and updated various experimental options including use_shim_syscall_handler, interface_qdisc, and use_extended_yaml.

File structure

  • A host's data files (files in <data-dir>/hosts/<hostname>/) are no longer prefixed with the hostname. For example a file that was previously named shadow.data/hosts/server/server.curl.1000.stdout is now named shadow.data/hosts/server/curl.1000.stdout.
  • The per-process .exitcode file has been removed due to its confusing semantics, and the new expected_final_state attribute replacing its primary use-case.
  • Generated pcap files are now named using their interface name instead of their IP address. For example "lo.pcap" and "eth0.pcap" instead of "127.0.0.1.pcap" and "11.0.0.1.pcap".

Performance

Shadow's scheduler is very performance-sensitive and needs to run tasks on worker threads with low latency. We added a spinloop in the scheduler that significantly improves Shadow's runtime performance. Some simulations see more than a 2x runtime performance improvement (for example 160 minutes to 47 minutes in a 5% Tor network simulation).

drawing

Supported platforms

We have removed several of our supported platforms. Specifically, we've dropped support for Ubuntu 18.04, Fedora 34/35/36, and CentOS Stream 8. We've also dropped support for Clang, and set a minimum-supported Linux kernel version of 5.4, which requires installing a backports kernel on Debian 10.

Stability guarantees

We've updated our "stability guarantees" document with the following changes:

  • Updated the filenames in Shadow's host-data directories to reflect the removal of the hostname prefix.
  • Added the ability to drop supported platforms in minor releases if the platforms no longer receive free updates and support from the distribution's developer.
  • Shadow no longer guarantees the order in which simulated process IDs (PIDs) are assigned.
  • Shadow will not change the criteria for the minimum supported Linux kernel version as documented in our supported platforms. This still allows us to increase the minimum kernel version as a result of dropping support for a platform.

Additional changes

Minor changes

  • Support the MSG_TRUNC flag for unix sockets. https://github.com/shadow/shadow/pull/2841
  • Support the TIMER_ABSTIME flag for clock_nanosleep. https://github.com/shadow/shadow/pull/2854
  • Removed the --profile, --include, and --library setup script options.
  • Added partial support for the epoll_pwait2 syscall.
  • Implemented the clone3 syscall. Thread libraries we're aware of that use clone3 were gracefully falling back to clone, but eventually they may not do so. This also reduces noise in shadow's log about an unimplemented syscall being attempted.
  • Shadow no longer requires /dev/shm to be executable.

Bug fixes

  • Fixed a memory leak of about 16 bytes per thread due to failing to unregister exited threads with a watchdog thread. This is unlikely to have been noticeable effect in typical simulations. In particular the per-thread data was already getting freed when the whole process exited, so it would only affect a process that created and terminated many threads over its lifetime.
  • Simulated Processes are now reaped and deallocated after the exit, reducing run-time memory usage when processes exit over the course of the simulation. This was unlikely to have affected most users, since Shadow currently doesn't support fork, so any simulation has a fixed number of processes, all of which are explicitly specified in shadow's config.
  • Fixed a potential race condition when exiting managed threads that did not have the clear_child_tid attribute set. This is unlikely to have affected most software running under Shadow, since most thread APIs use this attribute.
  • Changed an error value in clock_nanosleep and nanosleep from ENOSYS to ENOTSUP.
  • A managed process that tries to call the execve syscall will now get an error instead of escaping the Shadow simulation. https://github.com/shadow/shadow/issues/2718
  • Stopped overriding libc's getcwd with an incorrect wrapper that was returning -1 instead of NULL on errors.
  • A call to epoll_ctl with an unknown operation will return EINVAL.
  • Fixed a bug that caused Shadow to panic in some cases when a simulated thread exits. https://github.com/shadow/shadow/pull/2913
  • Fixed a bug causing host_options to undo any changes made to host_option_defaults.

Full changelog

Thanks to contributions from @robgjansen, @stevenengler, @sporksmith, @jtracey, @dependabot

v3.0.0-pre

1 year ago

Summary

The dev team had accumulated a large set of breaking changes that would require a major version bump. In this release, we have focused on clearing our breaking changes queue and merging those improvements. Because these are breaking changes, this release has bumped our major version from 2 to 3.

This release is marked as a pre-release because, although our CI tests are passing, we haven't had as long of a testing period as we usually do. Additionally, we have some additional internal improvements we intend to make prior to the full 3.0.0 release. We believe this pre-release should be stable, but please file issues for any bugs you find. Thanks!

Primary user-facing changes since v2.5.0

MAJOR changes (breaking):

  • Removed deprecated python scripts that only worked on Shadow 1.x config files and topologies.
  • Shadow no longer implicitly searches its working directory for executables to be run under the simulation. If you wish to specify a path relative to Shadow's working directory, prefix that path with ./.
  • Shadow now always enables support for YAML merge keys and extension fields. The experimental configuration option that previously enabled this support, use_extended_yaml, has been removed.
  • Removed the host pcap_directory configuration option and replaced it with a new pcap_enabled option.
  • A host's data files (files in <data-dir>/hosts/<hostname>/) are no longer prefixed with the hostname. For example a file that was previously named shadow.data/hosts/server/server.curl.1000.stdout is now named shadow.data/hosts/server/curl.1000.stdout.
  • The clang C compiler is no longer supported.
  • Host names are restricted to the patterns documented in hostname(7). https://github.com/shadow/shadow/pull/2856
  • The per-process option stop_time has been replaced with shutdown_time. When set, the signal specified by shutdown_signal (a new option) will be sent to the process at the specified time. While shadow previously sent SIGKILL at a process's stop_time, the default shutdown_signal is SIGTERM to better support graceful shutdown.
  • The minimum version of cmake has been bumped from 3.2 to 3.13.4.
  • The minimum version of glib has been bumped from 2.32 to 2.58.
  • Generated pcap files are now named using their interface name instead of their IP address. For example "lo.pcap" and "eth0.pcap" instead of "127.0.0.1.pcap" and "11.0.0.1.pcap".
  • The process environment configuration option now takes a map instead of a semicolon-delimited string.
  • Removed the quantity options for hosts and processes. It's now recommended to use YAML anchors and merge keys instead.
  • Renamed the host_defaults configuration field to host_option_defaults and renamed the host's options field to host_options.
  • Shadow now interprets a process still running at the end of the simulation as an error by default. This can be overridden by the new per-process option expected_final_state. https://github.com/shadow/shadow/pull/2886
  • The per-process .exitcode file has been removed due to its confusing semantics, and the new expected_final_state attribute replacing its primary use-case. https://github.com/shadow/shadow/pull/2906
  • Shadow no longer guarantees the order in which simulated process IDs (PIDs) are assigned. https://github.com/shadow/shadow/pull/2908

MINOR changes (backwards-compatible):

  • Support the MSG_TRUNC flag for unix sockets. https://github.com/shadow/shadow/pull/2841
  • Support the TIMER_ABSTIME flag for clock_nanosleep. https://github.com/shadow/shadow/pull/2854
  • The experimental config option use_shim_syscall_handler has been removed. This optimization is now always enabled.
  • It is now an error to set a process's stop_time or start_time to be after the simulation's stop_time.
  • Sub-second configuration values are now allowed for all time-related options, including start_time, stop_time, etc.
  • Removed the --profile, --include, and --library setup script options.
  • Added partial support for the epoll_pwait2 syscall.
  • Enabled CPU spinning in Shadow's scheduler. This significantly improves Shadow's runtime performance, but may have higher CPU and power/battery usage. https://github.com/shadow/shadow/issues/2877

PATCH changes (bugfixes):

  • Fixed a memory leak of about 16 bytes per thread due to failing to unregister exited threads with a watchdog thread. This is unlikely to have been noticeable effect in typical simulations. In particular the per-thread data was already getting freed when the whole process exited, so it would only affect a process that created and terminated many threads over its lifetime.
  • Fixed a potential race condition when exiting managed threads that did not have the clear_child_tid attribute set. This is unlikely to have affected most software running under Shadow, since most thread APIs use this attribute.
  • Changed an error value in clock_nanosleep and nanosleep from ENOSYS to ENOTSUP.
  • A managed process that tries to call the execve syscall will now get an error instead of escaping the Shadow simulation. https://github.com/shadow/shadow/issues/2718
  • Stopped overriding libc's getcwd with an incorrect wrapper that was returning -1 instead of NULL on errors.
  • A call to epoll_ctl with an unknown operation will return EINVAL.
  • Simulated Processes are now reaped and deallocated after the exit, reducing run-time memory usage when processes exit over the course of the simulation. This was unlikely to have affected most users, since Shadow currently doesn't support fork, so any simulation has a fixed number of processes, all of which are explicitly specified in shadow's config.

All Merged Pull Requests

Full Changelog

v2.5.0

1 year ago

Summary

In this release, we continue our transition from C to Rust. Most of the changes included in the release are backend changes that support our continued Rust migration. In particular, we've made important progress on migrating some of Shadow's core components, including Host, Process, Thread, and networking code. We also fixed some bugs and made some other changes to improve the experience for users as described below.

This release is intended to be the last stable release in the v2.x series. We have accumulated a fair number of issues that require a major version bump to complete as described in this discussion post, so we intend to take care of these issues within the next couple of weeks. As a result, the next stable release will mark the start of the v3.x series.

Primary user-facing changes since v2.4.0

  • Added a contributor code of conduct. https://github.com/shadow/shadow/blob/8003656d94fe781902f8b09420d994963a81c62c/CODE_OF_CONDUCT.md
  • Set the shim library's stdout/stderr to the shim log file. This should only affect simulations that use experimental features to disable interposition. https://github.com/shadow/shadow/pull/2725
  • Removed the experimental options preload_spin_max and use_explicit_block_message. These options were to support an execution model where Shadow workers ran on different CPU cores than the managed threads they were controlling, and each side would "spin" while waiting for a message from the other side. After extensive benchmarking we found that this was rarely a significant win, and dropped support for this behavior while migrating the core IPC functionality to Rust.
  • Changed the order that events are processed in Shadow. Some simulations may see improved runtime performance. https://github.com/shadow/shadow/pull/2522
  • Removed the experimental Dockerfile and related documentation. This is unrelated to running Shadow in Docker following the existing supported documentation, and we continue to support running Shadow in Docker.
  • Fixed the offset calculation in preadv/preadv2/pwritev/pwritev2 to correctly handle negative offsets and large offsets. https://github.com/shadow/shadow/pull/2802

All Merged Pull Requests

Full Changelog

v2.4.0

1 year ago

Summary

In this release, we continue our transition from C to Rust. Most of the changes included in the release are backend changes that support our continued Rust migration. However, we also fixed many bugs and made some other changes to improve the experience for users as described below.

We intend additional work following this release to focus on changes to some of Shadow's core networking components, including the TCP stack and other facilities for forwarding packets between nodes. This is somewhat higher risk work that could result in bugs that affect Shadow's network performance and stability. We are issuing this v2.4.0 release now to ensure that users have a stable version of Shadow that they can use while we work on the high risk networking code.

Primary user-facing changes since v2.3.0

All Merged Pull Requests

New Contributors

Full Changelog: https://github.com/shadow/shadow/compare/v2.3.0...v2.4.0

v2.3.0

1 year ago

Summary

Shadow v2.3.0 is a minor release that contains many bug fixes as well as a large push to convert more code from C to Rust; ~54% of our code is now written in Rust compared to just 39% in C. We have incorporated many improvements to Shadow's design as we migrate to Rust, making the code easier to understand, better tested, and easier to maintain. We plan to continue our focus on migrating code to Rust in our next release.

Primary user-facing changes since v2.2.0:

  • If running Shadow in Docker, you should use --tmpfs /dev/shm:rw,nosuid,nodev,exec,size=1024g rather than --shm-size=1024g to mount /dev/shm as executable. This fixes errors when the managed process maps executable pages. https://github.com/shadow/shadow/issues/2400
  • Added latency modeling and potential thread-yield to rdtsc emulation, allowing managed code to avoid deadlock in busy-loops that use only the rdtsc instruction and no syscalls. https://github.com/shadow/shadow/pull/2314
  • The build now internally uses pkg-config to locate glib, instead of a custom cmake module. This is the recommended way of getting the appropriate glib compile flags, and works better in non-standard layouts such as in a guix environment.
  • The setup script now has a --search option, which can be used to add additional directories to search for pkg-config files, C headers, and libraries. It obsoletes the options --library and --include.
  • Fixed a bug causing mmap to fail when called on a file descriptor that was opened with O_NOFOLLOW. https://github.com/shadow/shadow/pull/2353
  • Bare executable names are now resolved by searching shadow's PATH. Previously these were interpreted as relative to the current directory. For backwards compatibility, Shadow will currently prefer a binary in that location if one is found but log a warning. Such cases should be disambiguated by using an absolute path or prefixing with ./.
  • Fixed order-of-operations bug in CoDel control law that could lead to an unexpected packet drop schedule. We think the bug could have caused Shadow to slightly more aggressively drop packets that have already been sitting in the CoDel queue for longer than 110 milliseconds. Based on the results of some Tor network simulations, the bug didn't appear to affect Tor network performance enough to lead us to believe that previous Tor simulations are invalid. https://github.com/shadow/shadow/pull/2479
  • Changed the default scheduler from thread-per-host to thread-per-core, which has better performance on most machines.
  • Experimental host heartbeat log messages are enabled by default (experimental.host_heartbeat_interval defaults to "1 sec"), but the format of these messages is not stable.
  • Some of Shadow's emulated syscalls and object allocations are counted and written to a shadow.data/sim-stats.json file.
  • Improved experimental strace logging for brk, mmap, munmap, mremap, mprotect, open, and openat syscalls.
  • Several small simulation examples were added to an examples/ directory.
  • Fixed the file access mode for stdin in the managed process (changed from O_WRONLY to O_RDONLY).
  • Fixed support for readv and writev syscalls, and added support for preadv and pwritev.
  • Fixed a rare crash in Shadow's shim while logging. https://github.com/shadow/shadow/pull/2459
  • Set the ifa_netmask field in getifaddrs() to improve compatibility with Node.js applications. https://github.com/shadow/shadow/pull/2456
  • Shadow no longer depends on its absolute installed location, allowing the installation directory to be safely moved. https://github.com/shadow/shadow/pull/2391
  • Shadow now emulated PR_SET_DUMPABLE, allowing it to work for programs that try to disable memory inspection. https://github.com/shadow/shadow/pull/2370
  • Added new test cases to check Shadow's simulated network performance. These new tests help us verify that Shadow's network stack is capable of facilitating high-bandwidth transfers when using a single TCP stream or when using many streams in parallel, and across networks with various latency and bandwidth characteristics. Since we run the tests as part of our CI, it is now much more likely that we will notice when we make changes that significantly reduces Shadow's simulated network performance. We plan to expand the cases that we test in future releases. https://github.com/shadow/shadow/pull/2549

New Contributors

Thanks also to Shadow devs @sporksmith, @stevenengler, and @robgjansen!

Full Changelog

The full changelog can be viewed here: https://github.com/shadow/shadow/compare/v2.2.0...v2.3.0

v2.2.0

1 year ago

Summary

Shadow v2.2.0 is a rather small minor release that contains mostly bug fixes but also some new support for dup()ing file descriptors. We believe that our bug fixes improved Shadow's stability enough to warrant a release.

Rust became the majority language in Shadow in this release, and we plan to focus the next release on continuing our Rust migration.

Here is log of the primary user-facing changes we made since the previous release:

All Merged Pull Requests

Full Changelog: https://github.com/shadow/shadow/compare/v2.1.0...v2.2.0

v2.1.0

1 year ago

Shadow v2.1.0 is a minor release following our significant redesign of Shadow in v2.0.0. See the v2.0.0 release notes for more details about Shadow's new multi-process architecture.

This v2.1.0 release has largely focused on improving support for running various types of applications in Shadow while smoothing some of the rough edges introduced in v2.0.0.

We plan to focus our next release on rust migration, and we expect that Rust will become Shadow's primary programming language in v2.2.0!

New features

  • support for applications that depend on signals
  • support for dynamically linked Go applications
  • support for abstract-named unix sockets
  • VDSO-based function interposition
  • simulation progress messages (general.progress)
  • custom capture sizes for the pcap writer (host_defaults.pcap_capture_size)
  • system call latency modeling (general.model_unblocked_syscall_latency). This feature allows Shadow to escape some "busy loops" it couldn't before, avoiding deadlock in e.g. some versions of curl, iperf, libopenblas, and the golang runtime.
  • a --debug-hosts option to make debugging managed processes easier

New system calls

  • select()
  • getitimer()
  • setitimer()
  • SYS_rseq

New supported platforms

  • Ubuntu 22.04
  • Fedora 35 and 36

Changes

  • significantly improved determinism
  • O_DIRECT flag (packet mode) support for pipes
  • ioctl() support for pipes
  • SYN packets have a sequence number and are now retransmitted if they are not acknowledged
  • TCP sockets have finite-sized accept queues/backlogs
  • listen() can be called more than once for TCP sockets to set the backlog
  • improved error reporting if a process did not start or was killed unexpectedly (for example by the OS)
  • improved ioctl() file flag handling for regular files
  • TCP_NODELAY can be enabled for TCP sockets
  • added limited support for the TCP_CONGESTION socket option
  • pcap writer supports UDP packets
  • improved behaviour when one thread closes a file while another thread is blocked on a syscall that uses that file

Bug fixes

  • fixed broken eventfd and exit syscalls
  • fixed behaviour when reading/writing to a pipe
  • fixed memory leak when calling getservbyname_r()
  • fixed incorrect return value for getaddrinfo()
  • host log level is propagated to the shim log level
  • fixed crash on systems with 64-bit inodes

We've made many other internal improvements, added new test cases, and expanded our documentation.

v2.0.0

2 years ago

Shadow v2.0.0 is released!

Version 2 is the first major version bump since Shadow v1.0.0 was tagged over a decade ago!

In Shadow v2, we completely redesigned the architecture for executing and interacting with applications running in Shadow. To understand the importance of the redesign, let's first look at the previous design and its limitations.

Limitations of the previous version 1 design

In the previous version 1, the underlying architecture was that Shadow would load applications as plugins into the Shadow process space, and as of v1.12.0 the plugins were loaded into independent namespaces in the Shadow process space. This underlying plugin-based architecture had several limitations:

  • Compatibility: The domain of supported applications was limited to those that are compiled as position-independent libraries (PIC) or executables (PIE) that export their symbols to the dynamic symbol table (rdynamic), are dynamically linked to libc, and make all system calls through libc. Rebuilding applications so that they could be loaded into Shadow was tedious, and impossible if the source code was not available (e.g., closed-source software).

  • Correctness: Relying solely on preloading as a mechanism to intercept system calls is unreliable because only dynamically linked functions (e.g., those in libc) can be intercepted using LD_PRELOAD; system calls invoked via statically linked code or assembly instructions could leak outside of the simulation and cause errors.

  • Maintainability: A custom dynamic loader was required to load more than 16 plugin namespaces at once, and a portable threading library was used to support multi-threaded applications (these used to account for 62k LoC in Shadow). libc functions with nontrivial functionality would need to be reimplemented in order to intercept the system calls they make.

All of these issues lead to stability problems in Shadow and limited its use to niche applications like Tor.

Our new v2 design

In version 2, we designed a new architecture for executing and interacting with applications running in Shadow.

In our new design, applications are executed as standard Linux processes and hooked into the simulation through the system call interface using standard kernel facilities (primarily using preloading with seccomp as a backstop). This design overcomes many of the limitations of the previous plugin architecture:

  • the simulator can now execute any existing application without rebuilding it, given just a binary executable and its command line arguments.

  • Linux kernel subsystems guarantee reliable process isolation and correct system call interception.

  • The maintenance of a custom loader, threading libraries, and reimplemented libc functions is no longer required.

The new design allows Shadow to focus on supporting core functionality, i.e. system calls, rather than designing and maintaining code to work around issues brought about by the v1 design limitations. Separating Shadow from applications through a system call interface will enable Shadow to become a much more general purpose tool supporting a large number of use-cases.

Performance

One of our primary concerns while considering a new design was performance: Shadow is designed to run large-scale distributed systems, and we wanted to make sure that any inter-process communication overhead necessitated by the new multi-process design would not significantly detract from Shadow's target use-cases.

We're happy to report that the performance in our new v2 design is in most cases comparable to or faster than the performance of the v1 design! This means the v2 design is an all-around win relative to the v1 design, and it should significantly improve the Shadow user experience.

Rust

During the development of the new architecture, we began the process of migrating Shadow's programming language from C to Rust to further improve stability and correctness. We have made big improvements in this regard, prioritizing user-facing changes into this v2.0.0 release. The most noticeable user-facing change is that we updated our subsystems for specifying command line arguments and Shadow config files. We hope our new yaml-based config files are easier to manually read and edit than the old xml format.

We have further Rust migration tasks planned in future releases. We don't think the remaining migration work will be very noticeable from a user perspective, but please bear with us as we dust some cobwebs and continue our transition to a safer language.

More information

Our new design is the focus of a research paper that will appear at the 2022 USENIX Annual Technical conference. See our design document for more details on the new v2 design and a reference to our published research article.

v1.15.0

2 years ago