IncludeOS Versions Save

Added

• Conan build system
  • Major refactoring of how IncludeOS is built
  • Multiple ARCH is managed by Conan profiles and dependencies
  • 3rd party dependencies are now built and managed in Jenkins. All recipes can be found here
    • Updated to libcxx, libcxxabi 7.0.1
    • Updated to GSL 2.0.0
  • Stable and latest binary packages can be found in bintray
  • A repo to install Conan configs for IncludeOS: conan_config
  • Improvements to Jenkins integration, automatic uploads of latest/stable packages on master-merge/tags
• Experimental IPv6 (WIP) including SLAAC
  • IPv6/IPv4 dual stack integration
  • TCP/UDP client / server
  • Autoconfiguration with SLAAC
  • Configuration with config.json - see #2114
• HAL (work in progress)
  • The OS is now backed by a common Machine structure that makes it easier to create new ports
  • A custom C++ allocator is available very early allowing the use of STL before libc is ready

Changed

• Updates to workflow. All documented in the README
  • No more need for INCLUDEOS_PREFIX in env variables
  • Removed ARCH as part of the path to libraries/drivers/plugins/etc
    • Drivers and Plugins can be created outside includeos
• Moved IncludeOS repository from hioa-cs to includeos organization
• Major breaking changes in the OS API, in particular the OS class is removed, replaced with a smaller os namespace. Much of the code moved to new kernel:: namespace.
• Relocated plugins/libraries/scripts:
  • Hello World example
  • Demos and examples
  • Mana
  • Uplink
  • Vmrunner
  • Diskbuilder
  • Vmbuild
  • MicroLB

Removed / archived

• Cleanup of unused/outdated scripts
  • install.sh is gone as it does no longer work with the Conan workflow
• mender client is archived

Fixes a security issue where the stack protector is not properly initialized. Stack protector is now initialized using a proper rng at boot time and this patch improves how our rng is seeded, now requiring rdrand or rdseed instructions as entropy sources to be present at boot by default, unless this requirement is explicitly disabled.

Huge thanks to Spencer Michaels and Jeff Dileo of NCC Group who first reported this issue and to Jeff who helped us come up with this patch set.

We’ve experienced issues with running out of memory due to lack of a way to limit speed/memory consumption in TCP. This happened when using our load balancer on links with different speed. This release mainly consists of fixes to this issue (and other issues found along the way).

What’s new:

• Our own implementation of std::memory_resource to be used with the C++17 concept polymorphic_allocator
  • Pmr_pool and Pmr_resource keep track and limits the amount of bytes an allocator can allocate.
  • The implementation is very flexible and let us improve and make a more sophisticated memory solution down the road
• Changed our byte vector used around the OS (mainly TCP) to use polmorphic_allocator
  • using buffer = std::pmr::vector<uint8_t>
• Each TCP stack now have its own assigned memory pool (Pmr_pool), and each TCP connection now gets assigned its own resource (Pmr_resource)
  • Assigning memory to a TCP stack is done with set_total_bufsize(const size_t size) (default is set to 64MB)
• TCP connection no longer reports a static receive window, but instead a dynamic one based on how much memory used up in its resource
  • This helps throttling the connection when buffers aren’t released quickly enough, and avoids buffering more than the memory limit
• Added new on_data() callback to TCP connection with support functions read_next() and next_size()
  • Triggered every time new data is ready to be read (use read_next() on connection to retrieve)
  • Solves the issue of buffering having to be implemented on top - this is now done in the connection
  • Use either on_read or on_data depending on use case - as today, on_read has priority over on_data
  • The same functions are now also exposed in the net::Stream interface
• New TLS stream implementation and stream buffer class
  • Now utilizes the on_data() callback instead of on_read
  • Manages congestion and memory allocations for encryption better
• Various fixes and improvements in TCP
• MicroLB hardening, improvements and fixes

Major features:

• Replaced newlib libc with musl libc - which includes POSIX
• Added a linux syscall layer to be compatible with musl
• TCP SACK option support (receiver side)

Read more about our Linux compatibility here.

Minor features:

• Stateful TCP conntrack module
• New memory allocator as backend for mmap
• VFS plugin and config support
  • Now used to mount file descriptors for POSIX support - utilized by RNG and Syslog
• Timestamp plugin for stdout
• Statman and Bufferstore rewritten to use more modern C++
• Many features added to Linux platform (LTO, PGO, callstack generation etc.)

Compatibility:

• Boot and vmrunner now supports solo5

Experimental:

• New IPv6 implementation. Currently a work in progress
  • Including TCP API support
• MLDv2 and NDP (not finished)

Fixes:

• Fixed a flag in vmxnet3 restricting it from receiving VLAN tagged frames
• DNS client hardening

Other:

• TCP buffer sizes is now set per stack (and not per connection)
• DHCP now continues “forever” (but with less frequency)
• net::Socket is now using union address net::Addr which contains either a IPv4 or IPv6 address - many calls has now been replaced with net::Addr
• Print last received packet on panic
• Updated NaCl version
  • Timer type added, which makes it possible to have the service print a timestamp or information about memory, CPU and more at a set interval
  • New implementation under the hood, where each NaCl type can register into the transpiler dynamically
  • Updates to the Iface type which now incorporates the Vlan type (Vlan as its own type has been removed)
  • Iface members added: buffer_limit and send_queue_limit
• Updated solo5 version

IncludeOS v0.12.0-rc.2 with new bundles

Major features:

• LiveUpdate. Example, blog post on liveupdate
• Solo5 / ukvm support. Example, Solo5/uvkm blog post
• WebSockets Example
• Built-in auto configuration of services using config.json. Example

Multiarch / multiplatform:

• 64-bit support, now default. IncludeOS 64-bit blog post
• 32-bit chainloader for use with boot tool
• Reorganized for multiple platforms

Networking:

• Support TCP options/features; Timestamp & Window Scaling (RFC 7323) and Delayed ACK
• Path MTU Discovery, RFC 1191 (IPv4) and partly RFC 1981 (IPv6). Disabled by default until more thoroughly tested in the wild. See the unit test for example
• Kernel hooks for filter chains. Will enable upcoming firewall / NAT functionality.
• Support for Virtual IPs in Inet and TCP
• Support for partial IP/TCP checksumming
• DNS Client caching.
• Inet Domain Name support.

Utilities:

• Fixed size version of MemBitmap - allocates and owns storage.
• CMake now supports additional arguments (see #1404)
• CMake option for what CPU features to compile for.
• CRC64 and base64 implementations by @RicoAntonioFelix
• Script to easily run IncludeOS instances in VMware

Fixes:

• Time is now the same as the hosts time on all supported cloud vendors and hypervisors.
• Port_util now works correctly.

Extra special thanks to @RicoAntonioFelix for innumerable improvements, and to @dfct for help with TCP cleanup.

All dependencies should now be built without redzone

cxxabi is now being built out of the LLVM source tree, invalidating previous bundles. This IncludeOS dependency release adds a new dependency bundle, improved scripts for building the bundle and newlib updated to the latest version.

Update: We've added another bundle which contains 64bit builds of all deps as well, for upcoming x86_64 support.

Partial POSIX support

• Blocking sockets
• Syslog
• Tar/gzip
• Read only file support (read, open, stat, seek etc.)

Replaced old build system with CMake

Extra special thanks to @andreashappe for getting us started!

HTTP 1.1 support

• HTTP server functionality
• Simple, highly usable HTTP client

Monorepo

All IncludeOS subprojects moved into main IncludeOS repo, including:

• Mana Web Application Framework
• Acorn Web Appliance service

Other notable features

• VFS (Virtual file system)
• System RNG (Thanks, @randombit!)
• Boot with Grub (Thanks @sarum90!)

Utility classes

• CRC32 checksumming
• Fixed queue
• Path to regex
• URI support
• Syslogd with RFC5424 over UDP plugin
• TAR support, including gzip support with the help of uzlib.

Dependencies

• http-parser - Efficient http parsing
• rapidjson (optional)
• uzlib
• python-psutil
• python-jsonschema

Tools

• Diskbuilder: Easily create and link a memdisk with the content of a directory
• boot tool: Python program to easily boot IncludeOS services

Platform support

• OpenStack support using IDE boot device
• Injectable support for platform specific protocols, like Unik instance registration

Instrumentation

• Backtrace from ELF symbols
• Context buffer for crashes
• Real-time stack sampling for profiling

Boot

• Stack and heap randomization
• Multiboot support

Kernel features

• ACPI tables, ACPI shutdown
• xAPIC support
• Basic SMP support
• MSI-X support
• APIC timer
• Memory map of named, fixed memory ranges
• Improved timer system
• RTC that is auto-calibrated and inexpensive
• New driver injection system
• Statman: centrally located statistics for anything

Networking

• Inet4 is now driver independent
• TCP modularized with lots of API changes and improvements

Tooling and testing

• Building kernel with minimal code and stripped symbols
• Improvements to install scripts
• Automatic builds using Jenkins for PRs and main branches
• New automatic test suite for unit testing, using LEST

Misc

• IPv4 addresses can now be created from string
• Code refactoring to make use of new C++ features and Core Guidelines
• Lots of bugfixes