wazero 1.2.0 includes 33 days of valiant effort towards performance, interop and debug goals, shared not only in wazero, but WebAssembly in general. We appreciate folks waiting a couple more days than usual and expect you'll enjoy what you see below.

While we haven't set a formal post 1.0 release cadence, you can expect another patch or minor within a month. Meanwhile, this is our most performant and best tested release yet. As always, star all the projects that interest you, and say thanks when you can.

Performance

Performance is something we aim to always improve, release to release. This includes looking at memory usage as well as latency. While there were multiple people involved in efficiency work, @achille-roussel and @lburgazzoli deserve special call outs for leading efforts, and @ncruces deserves a big pat on the back for contributing reviews, cleanups and advice.

@achille-roussel made many changes internal to our compiler, as well linux-only specializations such as using huge pages for the memory mapped regions under wasm functions. These were all profile and benchmark guided changes and proposed in top rigor.

@lburgazzoli tracked down best practice in TinyGo, consolidating advice from various experts, such as the primary developer of TinyGo @aykevl. He worked with @ncruces to make sure our allocation example is not just a code snippet, but an exemplar of good practice, without risk of memory leaks and performance validated with benchmarks.

The combination of backend work (e.g. runtime improvements) and frontend work (e.g. changes to our TinyGo example) combined in a notable holistic gain across the board. This was true teamwork and a job well done!

$ benchstat v1.1.0.txt v1.2.0.txt 
goos: darwin
goarch: arm64
pkg: github.com/tetratelabs/wazero/internal/integration_test/vs/compiler
                          │  v1.1.0.txt  │             v1.2.0.txt             │
                          │    sec/op    │   sec/op     vs base               │
Allocation/Compile-12       3.365m ±  1%   3.174m ± 1%   -5.66% (p=0.002 n=6)
Allocation/Instantiate-12   149.1µ ± 26%   120.4µ ± 7%  -19.23% (p=0.002 n=6)
Allocation/Call-12          1.404µ ±  2%   1.297µ ± 2%   -7.66% (p=0.002 n=6)
geomean                     88.97µ         79.13µ       -11.05%

                          │  v1.1.0.txt  │              v1.2.0.txt               │
                          │     B/op     │     B/op      vs base                 │
Allocation/Compile-12       2.404Mi ± 0%   1.292Mi ± 0%  -46.24% (p=0.002 n=6)
Allocation/Instantiate-12   319.4Ki ± 0%   230.5Ki ± 0%  -27.84% (p=0.002 n=6)
Allocation/Call-12            48.00 ± 0%     48.00 ± 0%        ~ (p=1.000 n=6) ¹
geomean                     33.28Ki        24.27Ki       -27.07%
¹ all samples are equal

                          │ v1.1.0.txt  │              v1.2.0.txt              │
                          │  allocs/op  │  allocs/op   vs base                 │
Allocation/Compile-12       1.830k ± 0%   1.595k ± 0%  -12.84% (p=0.002 n=6)
Allocation/Instantiate-12    803.0 ± 0%    508.0 ± 0%  -36.74% (p=0.002 n=6)
Allocation/Call-12           5.000 ± 0%    5.000 ± 0%        ~ (p=1.000 n=6) ¹
geomean                      194.4         159.4       -18.00%
¹ all samples are equal

Interop

Compatibility is a moving target as both specifications change as well understanding of specifications. For example, the WebAssembly Core Specification 2.0 remains in a draft state, and expectations of the VM change as it changes. Also the d'facto WASI version preview1 (a.k.a. wasip1) had no tests, nor detailed documentation for the first several years of its existence. This left interop as more a quorum of implementation practice vs a spec. While new initiatives such as the wasi-testsuite and wasix aim to stabilize this, WASI compatibility remains a source of work from wazero maintainers and compiler developers. We really appreciate the efforts spent here to keep as many users unaware of these glitches as possible.

On the WebAssembly Core (VM) side, we appreciate @mathetake updating our code and spec suite to pass latest changes there. Also, we appreciate an attempt by @anuraaga with support by @ncruces on the Threads proposal, despite us ending up parking the idea until the proposal finishes.

On the WASI side, we appreciate a lot of work driven by the team working on Go. Specifically, the upcoming GOOS=wasip1 planned for 1.21 helped reveal a number of grey areas that required work to support in Go without breaking other languages. Championing these came from various team members including @Pryz, @achille-roussel and @evacchi on various file rights and non-blocking related glitches, some fixing other language runtimes such as python.

We're also excited that @evacchi began an experiment to support sockets, currently working for blocking requests. As wasm only has one thread to use, libraries often need non-blocking functionality to do anything notable. We'll report more on sockets once non-blocking glitches sort out.

Meanwhile, those using sockets know that the preview1 version of WASI is extremely limited. There are other ABI such as wasmedge_wasi_socket, wasi-sockets and most recently wasix. All of these go beyond the simple TCP sock accept, read/write in wasip1. If you'd like the bleeding edge socket support, please try wasi-go and request the features you want to experiment with. This project is a layer over wazero with an alternate syscall layer. wasi-go can move faster due to less constraints than upstream, such as Windows or virtual files. This makes it a lower risk and ideal playground to develop evolving host function (ABI) specifications. As these functions mature, what makes sense to build-in will land upstream in wazero.

Debug

We are very excited to power the only known out-of-browser CPU and memory profiler for WebAssembly, wzprof. wzprof brings the power of pprof to wasm, regardless of if the source language is Go or not.

wzprof has already served a lot of benefits in its short life so far. For example, the kube-scheduler-wasm-extension used it to isolate a garbage collection problem endemic in large protobuf decoders compiled to wasm.

Implementing this required significant work in wazero, which we are happy went upstream! @pelletier added the source offset to experimental.StackIterator which allows source-mapping in debugging use cases. @achille-roussel polished the experimental.FunctionListener to be more performant, including optimizing its API around errors and removing context propagation. @chriso and @mathetake helped fix some glitches along the way. Finally, @achille-roussel made it easier to develop 3rd party listeners by adding experimental.FunctionListenerFactory to supply them and wazerotest.Module to test them.

While we don't expect a lot of people to implement listeners, it was a great team effort to get the substrate together to the point you can build a profiler on top of it. Kudos especially to the wzprof team on finally giving wasm developers a decent profiler!

What's next?

In the short term, we'll try to close the gaps on non-blocking I/O inside WASI. We still aim to have a fully pluggable filesystem soon, including an in-memory option for those needing to provide something like tmpfs from Go. Of course compatibility issues and user demands will take priority as they always do.

Longer term, @mathetake is taking on the task of an optimizing compiler affectionately named wazevo. This will likely take a year to mature, and will narrow the performance gap on certain libraries like libsodium without adding any platform dependencies whatsoever.

Meanwhile, if you want updates you can always contact the community and ask, or just wait for the next release. Until next time!

wazero 1.1.0 improves debug, reduces memory usage and adds new APIs for advanced users.

1.1 includes every change from prior versions. The below elaborates the main differences, brought to you by many excellent engineers who designed, reviewed, implemented and tested this work. Many also contribute to other areas in Go, TinyGo and other languages, as well specification. If you are happy with the work wazero is doing for Go and Wasm in general, please star our repo as well any projects mentioned!

Now, let's dig in!

Debug

This section is about our debug story, which is better than before thanks to several contributors!

Go stack trace in the face of Go runtime errors

When people had bugs in their host code, you would get a high-level stack trace like below

2023/04/25 10:35:16 runtime error: invalid memory address or nil pointer dereference (recovered by wazero)
wasm stack trace:
        env.hostTalk(i32,i32,i32,i32) i32
        .hello(i32,i32) i64

While helpful, especially to give the wasm context of the error. This didn't point to the specific function that had the bug. Thanks to @mathetake, we now include the Go stack trace in the face of Go runtime errors. Specifically, you can see the line that erred and quickly fix it!

2023/04/25 10:35:16 runtime error: invalid memory address or nil pointer dereference (recovered by wazero)
wasm stack trace:
        env.hostTalk(i32,i32,i32,i32) i32
        .hello(i32,i32) i64

Go runtime stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x64
github.com/tetratelabs/wazero/internal/wasmdebug.(*stackTrace).FromRecovered(0x140001e78d0?, {0x100285760?, 0x100396360?})
        /Users/mathetake/wazero/internal/wasmdebug/debug.go:139 +0xc4
--snip--

experimental.InternalModule

Stealth Rocket are doing a lot of great work for the Go ecosystem, including implementation of GOOS=wasip1 in Go and various TinyGo improvements such as implementing ReadDir in its wasi target.

A part of success is the debug story. wazero already has some pretty excellent logging support built into the command-line interface. For example, you can add -hostlogging=filesystem to see a trace of all sys calls made (e.g. via wasi). This is great for debugging. Under the scenes, this is implemented with an experimental listener.

Recently, @pelletier added StackIterator to the listener, which allows inspection of stack value parameters. This enables folks to build better debugging tools as they can inspect which values caused an exception for example. Since it can walk the stack, propagation isn't required to generate images like this, especially as listeners can access wasm memory.

However, in practice, stack values and memory isn't enough. For example, Go maintains its own stack in the linear memory, instead of using the regular wasm stack. The Go runtime stores the stack pointer in global 0. In order to retrieve arguments from the stack, the listener has to read the value of global 0, then the memory. Notably, this global isn't exported.

To work around this, Thomas exposed an experimental interface experimental.InternalModule which can inspect values given an api.Module. This is experimental as we still aren't quite sure if we should allow custom host code to access unexported globals. However, without this, you can't effectively debug either. If you have an opinion, please join our slack channel and share it with us! Meanwhile, thank @pelletier and Stealth Rocket in general for all the help in the Go ecosystem, not just their help with wazero!

Memory Usage

This section describes an advanced internal change most users don't need to know about. Basically, it makes wazero more memory efficient. If you are interested in details, read on, otherwise thank @mathetake for the constant improvements!

Most users of wazero use the implicit compiler runtime configuration. This memory maps (mmap syscall) the platform-specific code wazero generates from the WebAssembly bytecode. Since it was written, this used mmap once per function in a module.

One problem with this was page size. Basically, mmap can only allocate the boundary of the page size of the underlying os. For example, a very simple function can be as small as several bytes, but would still reserve up a page each (marked as executable and not reusable by the Go runtime). Therefore, we wasted roughly (len(body)%osPageSize)*function.

The new compiler changes the mmap scope to module. Even though we still need to align each function on 16 bytes boundary when mmaping per module, the wasted space is much less than before. Moreover, with the code behind functions managed at module scope, it can be cleaned up with the module. We no longer have to abuse the runtime.Finalizer for cleanup.

Those using Go benchmarks should see improved compilation performance, even if it appears more allocations than before. One tricky thing about Go benchmarks is they can't report what happens via mmap. The net result of wazero will be less wasted memory, even if you see slightly more allocations compiling than before. These allocations are a target of GC and should be ignorable in the long-running program vs the wasted page problem in the prior implementation, as that was persistent until the compiled module closed.

In summary, this is another example of close attention to the big picture, even numbers hard to track. We're grateful for the studious eyes of @mathetake always looking for ways to improve holistic performance.

Advanced APIs

This section can be skipped unless you are really interested in advanced APIs!

Function.CallWithStack

WebAssembly is a stack-based virtual machine. Parameters and results of functions are pushed and popped from the stack, and in Go, the stack is implemented with a []uint64 slice. Since before 1.0, authors of host functions could implement exported functions with a stack-based API, which both avoids reflection and reduces allocation of these []uint64 slices.

For example, the below is verbose, but appropriate for advanced users who are ok with the technical implementation of WebAssembly functions. What you see is 'x' and 'y' being taken off the stack, and the result placed back on it at position zero.

builder.WithGoFunction(api.GoFunc(func(ctx context.Context, stack []uint64) {
	x, y := api.DecodeI32(stack[0]), api.DecodeI32(stack[1])
	sum := x + y
	stack[0] = api.EncodeI32(sum)
}), []api.ValueType{api.ValueTypeI32, api.ValueTypeI32}, []api.ValueType{api.ValueTypeI32})

This solves functions who don't make callbacks. For example, even though the normal api.Function call doesn't use reflection, the below would both allocate a slice for params and also one for the result.

Here's how the current API works to call a calculator function

results, _ := add.Call(ctx, x, y)
sum := results[0]

Specifically, both the size and ptr would be housed by a slice of size one. Code that makes a lot of callbacks from the host spend resources on these slices. @inkeliz began a design of a way out. @ncruces took lead on implementation with a lot of feedback from others. This resulted in several designs, with the below chosen to align to similar semantics of how host functions are defined.

wazero now has a second API for calling an exported function, Function.CallWithStack.

Here's how the stack-based API works to call a calculator function

stack := []uint64{x,y}
_ = add.CallWithStack(ctx, stack)
sum := stack[0]

As you can see above, the caller provides the stack, eliminating implicit allocations. The results of this are more significant on calls that are in a pattern, where the stack is re-used for many calls. For example, like this:

stack := make([]uint64, 4)
for i, search := range searchParams {
	// copy the next params to the stack
	copy(stack, search)
	if err := searchFn.CallWithStack(ctx, stack); err != nil {
		return err
	} else if stack[0] == 1 { // found
		return i // searchParams[i] matched!
	}
}

While most end users won't ever see this API, those using shared libraries can get wins for free. For example, @anuraaga updated go-re2 to use this internally and it significantly improved a regex benchmark, reducing its ns/op from 350 to 311 and eliminating all allocations.

wazero treats core APIs really seriously and don't plan to add any without a lot of consideration. We're happy to have had the help of @inkeliz @ncruces as well the many other participants including @achille-roussel @anuraaga @codefromthecrypt and @mathetake.

emscripten.InstantiateForModule

@jerbob92 has been porting various PDF libraries to compile to wasm. The compilation process uses emscripten, which generates dynamic functions. At first, we hard-coded the set of "invoke_xxx" functions required by PDFium. This was a decision to defer design until we understood how common end users would need these. Recently, Jeroen began porting more PDF libraries, including ghostscript and xpdf . These needed different dynamic functions from eachother. To solve this, @codefromthecrypt implemented a host function generator, emscripten.InstantiateForModule.

This works by analyzing the compiled module for functions needed, so you need to compile your binary first like this:

emscriptenCompiled, _ := r.CompileModule(ctx, emscriptenWasm)
_, err = emscripten.InstantiateForModule(ctx, r, emscriptenCompiled)

If you are doing porting work, you may find this handy. If so, please drop by our #wazero slack channel and tell us about it!

Minor changes

• @Pryz added integrations tests so that the upcoming 1.21 GOOS=wasip1 is tested on each change. Thanks a lot to @mathetake for support on this, too!
• @evacchi improved the implementation of non-blocking stdin on Windows (via select(2))
• @ncruces added internal.WazeroOnly to interfaces not open for implementation. Thanks @achille-roussel for the design support!
• @codefromthecrypt fixed a bug on zero-length reads in WASI
• @mathetake made numerous improvements to the amd64 compiler

wazero v1.0.3 improves optimizes compilation on the amd64 platform and fixes bugs notably in Windows packaging.

A few days ago, we released wazero v1.0.2, which notably improved compilation performance. A lot of folks jumped on that release and we found a few glitches in the process. Meanwhile @mathetake finished up optimization work on amd64 compilation.

Here are some of the notable fixes.

• Windows (MSI and winget) installers didn't set the PATH correctly, so wazero wasn't a one-step install. @evacchi fixed this by updating our infrastructure to the latest native WiX toolchain, scrubbing some other glitches along the way. Thanks to @inliquid for reporting this and verifying the fix.
• @jerbob92 has been porting various PDF libraries to compile to wasm, and in the process noticed we didn't handle invalid file descriptors properly. @codefromthecrypt changed our internal code to treat file descriptors as signed integers and negative ones as EBADF.
• @jerbob92 also noticed compiler cache corruption when you switch on and off experimental function listeners. @mathetake fixed this bug.
• @kevburnsjr tried to copy/paste some of our godoc examples and they had drifted. Thanks for fixing that!

Thanks so much for being such an engaging community, where opportunities and misses can action so quickly! If you haven't already, please thank our contributors with a star!

wazero 1.0.2 improves compiler performance, supports non-blocking stdin and adds a couple new experimental APIs.

Many people were involved in a lot of work in the last 3 weeks. Please reach out and thank them!

Improved compiler performance

wazero has a compile phase (CompileModule) which lowers WebAssembly bytecode into an intermediate representation (IR) and into machine code. This process is CPU and memory intensive and has been optimized significantly since 1.0.1.

We used SQLite wasm, to ensure the encouraging improvements were relevant to real-world use cases.

goos: linux
goarch: amd64
pkg: github.com/tetratelabs/wazero/internal/integration_test/bench
cpu: AMD Ryzen 9 3950X 16-Core Processor
                                       │  v1.0.1.txt   │              new.txt               │
                                       │    sec/op     │   sec/op     vs base               │
Compilation_sqlite3/compiler-32          1001.9m ±  2%   544.0m ± 2%  -45.70% (p=0.001 n=7)
Compilation_sqlite3/interpreter-32       208.57m ±  5%   83.81m ± 5%  -59.82% (p=0.001 n=7)

                                       │  v1.0.1.txt   │               new.txt               │
                                       │     B/op      │     B/op      vs base               │
Compilation_sqlite3/compiler-32          305.10Mi ± 0%   55.31Mi ± 0%  -81.87% (p=0.001 n=7)
Compilation_sqlite3/interpreter-32       142.24Mi ± 0%   51.77Mi ± 0%  -63.60% (p=0.001 n=7)

                                       │  v1.0.1.txt   │              new.txt               │
                                       │   allocs/op   │  allocs/op   vs base               │
Compilation_sqlite3/compiler-32           5217.0k ± 0%   343.2k ± 0%  -93.42% (p=0.001 n=7)
Compilation_sqlite3/interpreter-32       1770.43k ± 0%   14.00k ± 0%  -99.21% (p=0.001 n=7)

The changes to bring the above included a series of refactoring by @evacchi on union types, as well dozens of optimizations by @mathetake, and a couple by @ckaznocha.

All of this was easier due to frequent and thorough advice by @achille-roussel and our latest core maintainer @ncruces. Thanks to all involved for the epic improvement in less than 3 weeks!

Non-blocking stdin

container2wasm is an interesting project that converts containers such that they can run in a webassembly runtime, such as a browser or wazero.

One feature this relies on is non-blocking access to STDIN. Like some other runtimes, wazero didn't handle this properly.

Thanks to a lot of effort by @evacchi with advice from @achille-roussel and support from the container2wasm author @ktock, wazero now handles non-blocking STDIN properly (via the select syscall).

Experimental changes

Code in our "experimental" directory isn't under an API guarantee, so can change even in a patch version. Here are a couple new experiments since last release.

• @pelletier added a StackIterator parameter to listeners, allowing inspection of the stack leading to a function call. Thanks to @Pryz for the initial design and background, as this is used for CPU profiling data.
• @codefromthecrypt added emscripten.InstantiateForModule to dynamically build function imports given a CompiledModule. Thanks to @jerbob92 for the idea and testing with various PDF tools.

Fixes and behavior changes

1.0.2 includes some bug fixes..

• @twilly and @mathetake fixed some concurrency and ordering issues closing modules
• @mathetake fixed a bug in RuntimeConfig.WithMemoryCapacityFromMax
• @ckaznocha fixed a module cleanup related issue.
• @codefromthecrypt fixed a bug in the logging listener

It also includes a couple behavior changes..

• @abraithwaite made it possible to use errors.Is for context-done related error cases.
• @codefromthecrypt made host functions retain insertion order (instead of lexicographic).

wazero v1.0.1 fixes a stdio glitch, improves performance and polishes documentation. We decided to cut an early patch mainly to ensure python works properly.

Python repl hang

Despite trying many things prior to v1.0.0, a glitch escaped us. @evacchi tried the VMware Labs python-wasm, and noticed a repl hang. Edo and @achille-roussel collaborated on a fix, which also ended up deleting tricky code. He verified python-wasm works, and @ncruces verified dcraw still works as well. Thank these folks for the teamwork and rigor!

Optimizations

Due to the nature of our team, you can expect optimizations in every release. A lot of work by @mathetake has been optimization both from line count and performance. There were only several days duration since v1.0.0, the culmination of work by Takeshi and @evacchi (with review support by @achille-roussel) resulted in less code and a slight bump in performance in an end user benchmark:

goos: darwin
goarch: arm64
pkg: github.com/dapr/components-contrib/bindings/wasm
           │   old.txt   │           new.txt            │
           │   sec/op    │   sec/op     vs base         │
Example-12   12.11µ ± 2%   12.02µ ± 1%  ~ (p=0.132 n=6)

pkg: github.com/dapr/components-contrib/middleware/http/wasm
                          │   old.txt   │              new.txt              │
                          │   sec/op    │   sec/op     vs base              │
Native/rewrite/rewrite-12   573.7n ± 0%   575.0n ± 0%       ~ (p=0.240 n=6)
Tinygo/rewrite/rewrite-12   1.161µ ± 1%   1.155µ ± 1%  -0.52% (p=0.026 n=6)
Wat/rewrite/rewrite-12      986.2n ± 0%   988.4n ± 1%       ~ (p=0.485 n=6)
geomean                     869.3n        869.0n       -0.03%

Docs

Our documentation improved in the last few days as well: @jcchavezs fixed some glitches on our home page around trying out wazero, @jerbob92 added PDFium tools to our users page, and @codefromthecrypt implemented @Xe's suggestion to improve our our walltime clock documentation. We really appreciate the pro-activity on user facing documentation!

wazero v1.0.0 completes our six month pre-release period and begins our compatibility promise. We will use semantic versions to label releases, and not break APIs we've exposed as non-experimental.

Those not familiar with wazero can check out this blog which overviews the zero dependency runtime. You can also check out our website especially the community and users pages.

Many of you have been following along with our pre-releases over the last 6 months. We did introduce change since v1.0.0-rc.2 with a particularly notable feature we call "anonymous modules". So, let's talk about that first.

Anonymous modules

There are two main ways wazero is used for high-volume request handling. One way is pooling modules and the other is instantiating per-request.

The pool approach is used for functions designed to be invoked many times, such as http-wasm's handler functions. A host, such a dapr keeps a pool of modules, and checks one out per request.

The re-instantiate approach is where you know you can't re-use a module, because the code is not safe to invoke more than once. For example, WASI commands are not safe to re-invoke. So, you have to instantiate a fresh module per request. You can also re-instantiate for higher security on otherwise safe functions.

The latter case was expensive before, because we had to make sure each request had not just a new module, but also a unique name in the runtime. You would see things like this to do that.

// Currently, concurrent modules can conflict on name. Make sure we have
// a unique one.
instanceNum := out.instanceCounter.Add(1)
instanceName := out.binaryName + "-" + strconv.FormatUint(instanceNum, 10)
moduleConfig := out.moduleConfig.WithName(instanceName)

Both allocating a unique name and also name-based locks have a cost to them, and very high throughput use cases, such as event handling would show some contention around this.

Through a lot of brainstorming and work, @achille-roussel @ckaznocha and @mathetake found a clever way to improve performance. When a module has no name, it has nothing to export to other modules. Most of the lock tension was around things to export, and an unnamed module is basically a leaf node with no consumer except the host. We could avoid a lot of the more expensive locking by special-casing modules instantiated without a name.

In the end, to improve re-instantiation performance (when you can't pool modules), clear your module name!

-       // Currently, concurrent modules can conflict on name. Make sure we have
-       // a unique one.
-       instanceNum := out.instanceCounter.Add(1)
-       instanceName := out.binaryName + "-" + strconv.FormatUint(instanceNum, 10)
-       moduleConfig := out.moduleConfig.WithName(instanceName)
+       // Clear the module name so that instantiations won't conflict.
+       moduleConfig := out.moduleConfig.WithName("")

Other changes

There were a myriad of change from wazero regulars, all of them in the bucket of stabilization, bug fixes or efficiency in general. @achille-roussel @codefromthecrypt and @jerbob92 put a lot of work into triage on WASI edge cases, both discussion and code. @ncruces fixed platform support for solaris/illumos @mathetake optimized wazero performance even more than before. @evacchi fixed a really important poll issue.

These were driven by and thanks to community work. For example, @Pryz led feedback and problem resolution for go compiler tests. Both @ncruces on go-sqlite and @jerbob92 on pdfium shared wins and opportunities for next steps.

In short, there were a lot of exciting relevant work in the short period between rc2 and 1.0.0, and we are lucky for it!

wazero v1.0.0-rc.2 is a stabilizing release, and the last version before 1.0 next week.

wazero 1.0.0 will happen at our release party attended by many contributors present at wasmio in Barcelona. While this is our first community meetup, it won't be our last. Please join us to suggest or help organize subsequent events.

Below are a list of changes, notably new is operating system packaging. Read on to get the full story!

Packaging

@mathetake and @evacchi worked together to publish OS artifacts, you can see attached to this release. Most work was needed around windows as MSI installers need to be signed to avoid warnings. We'll begin distributing wazero via homebrew and winget soon, as well.

Tests

Thanks particularly to @codefromthecrypt and @evacchi, wazero is more tested than we were before, and more than any other runtime we are aware of. We've notably closed gaps not just in WASI, but edge cases around windows and GOOS=js. @mathetake stepped in not just in support of tests, but also adding a test flow so we can code in confidence:

Website

wazero already has extensive code documentation, examples, low-level RATIONALE and language guides.

We have work, yet, on high-level and conceptual documentation. To start, @mathetake added a documentation page, covering architecture and some low-level questions. @evacchi also polished the home page, now that we're focusing a lot more on our CLI user base.

Performance

@mathetake has worked relentlessly to improve performance, especially around initialization of modules. This is analogous to startup time. You can see some of the dramatic improvements below:

$ benchstat v1.0.0-rc.1.txt v1.0.0-rc.2.txt
name                                    old time/op    new time/op    delta
Initialization/interpreter-32             52.9µs ± 2%    36.1µs ± 2%  -31.74%  (p=0.000 n=28+25)
Initialization/interpreter-multiple-32    39.9µs ± 2%    39.7µs ±13%     ~     (p=0.140 n=29+30)
Initialization/compiler-32                32.2µs ± 7%    28.3µs ± 9%  -12.09%  (p=0.000 n=30+30)
Initialization/compiler-multiple-32       25.3µs ± 5%    24.5µs ± 8%   -3.29%  (p=0.000 n=30+26)
Compilation/with_extern_cache-32           206µs ± 2%     200µs ± 2%   -2.78%  (p=0.000 n=29+30)
Compilation/without_extern_cache-32       6.00ms ± 1%    5.94ms ± 1%   -0.95%  (p=0.000 n=29+30)

name                                    old alloc/op   new alloc/op   delta
Initialization/interpreter-32              137kB ± 0%     136kB ± 0%   -0.35%  (p=0.000 n=30+30)
Initialization/interpreter-multiple-32     137kB ± 0%     137kB ± 0%   -0.04%  (p=0.000 n=27+27)
Initialization/compiler-32                 141kB ± 0%     137kB ± 0%   -3.09%  (p=0.000 n=30+23)
Initialization/compiler-multiple-32        142kB ± 0%     142kB ± 0%   -0.03%  (p=0.000 n=27+25)
Compilation/with_extern_cache-32          55.6kB ± 0%    54.6kB ± 0%   -1.79%  (p=0.000 n=29+30)
Compilation/without_extern_cache-32       1.99MB ± 0%    1.99MB ± 0%   -0.12%  (p=0.000 n=30+30)

name                                    old allocs/op  new allocs/op  delta
Initialization/interpreter-32               52.0 ± 0%      38.0 ± 0%  -26.92%  (p=0.000 n=30+30)
Initialization/interpreter-multiple-32      58.0 ± 0%      57.0 ± 0%   -1.72%  (p=0.000 n=30+30)
Initialization/compiler-32                  42.0 ± 0%      38.0 ± 0%   -9.52%  (p=0.000 n=30+30)
Initialization/compiler-multiple-32         48.0 ± 0%      47.0 ± 0%   -2.08%  (p=0.000 n=30+30)
Compilation/with_extern_cache-32           1.10k ± 0%     0.98k ± 0%  -10.86%  (p=0.000 n=27+30)
Compilation/without_extern_cache-32        32.7k ± 0%     32.6k ± 0%   -0.37%  (p=0.000 n=30+30)

To support future improvements, we no longer allow importing unnamed modules (moduleName=""). This is an edge case allowed by spec, but not used in practice. By disallowing this, future versions of wazero can be considerably faster instantiating anonymous modules than today.

Changes in support of wasm compiled by Go

Some of you know Go builds wasm binaries when the environment variables GOARCH=wasm and GOOS=js are set. We include an experimental package gojs which supports this until Go includes a WASI operating system for at least 2 releases. We support gojs in part due to users who want an alternative runtime besides node.js. The other reason is to help support future development in the Go compiler. Our hope is developers can quickly check behavior between JS and WASI, so that problems are solved quicker.

The main change in this version is moving the gojs directory under the experimental folder. It was always experimental, but via documentation. This should help people know that this operating system is temporary until Go supports WASI (GOOS=wasip1) for at least 2 releases.

The other change is exposing gojs.Config with additional go-specific feature toggles, enabled by default in our CLI. This allows things not defined in WASI to pass, for example functionality about the working directory or user IDs of the process. This allows wazero to pass 100pct of the os package tests defined by Go.

wazero v1.0.0-rc.1 starts our journey towards 1.0, with no public API changes. We will have at least one more release candidate between now and 1.0, in a week or two. The next candidate will include CLI binaries, which will allow our non-Go community to use wazero without installing Go, first.

wazero 1.0 will happen March 24th, during the wazero release party at wasmio hosted by Tetrate. Follow Edoardo for updates on on the party, who's doing great job organizing including lightning talks by end users.

Many of you have requested a community page on our website. We added this including a list of users who explicitly opted in. The user list is opt-in, and generally higher signal than the "Used By" list on the GitHub site, as the latter includes transient dependencies.

The biggest internal code change in v1.0.0-rc.1 let to wazero passing the entire wasi-testsuite on linux, darwin (macOS) and windows, with zero exceptions. To do so took tens of days of effort from both @codefromthecrypt and @mathetake, and described at the bottom for the curious. By passing every test defined by WebAssembly, as well stdlib tests in TinyGo and Zig, it is easier for end users to feel confident wazero is a great choice for stability. This is worth the couple weeks of pain.

We intentionally didn't do any other large changes, but there were several people to thank for minor changes. Some of the below took many days of effort each!

• @achille-roussel for refactoring an internal type with generics so we can use it for open directories.
• @ckaznocha for fixing a concurrency issue on context cancellation.
• @mathetake for a mountain of optimizations to reduce compilation overhead
• @dmvolod for removing a redundant wasm decoding validation.
• @evacchi for adding -timeout duration to the CLI which stops runaway processes.
• @evacchi for fixing a corner case around max memory limit
• @mathetake for allowing the CLI to be built with an external version (for packaging)

In closing, thanks very much for sticking with us this last year and a half almost, leading to 1.0. Our fantastic ecosystem is the reason you have a zero dependency runtime, something you can embed without thinking about work or version clashes with other tools. We've done great work together to bring Go forward in the WebAssembly ecosystem, and people are noticing!

Notes on wasi-testsuite compliance

Most people don't need to read this part, so only do if you are interested in low level details, or bored!

wasi-testsuite compliance means passing tests that verify the expected behaviors of wasi_snapshot_preview1, as decided by the WASI W3C subgroup. It does so by running tests compiled in different languages, currently AssemblyScript, C and Rust.

As described at the beginning of the release notes, wazero v1.0.0-rc.1 passes all tests as doing so is least confusing to the ecosystem. This means passing things we don't advise or agree with. Our former release left out a couple tests due to performance overhead, but we pass them now despite it. If you care about this, read on!

The last pre-release skipped a test on dot and dot-dot entries in directory listings. Go throws away these entries before we can read them. To resurrect them costs tens of microseconds. The cost is fixed overhead around inodes, but this same topic has a large performance footprint when multiplied in directory listings.

Recently, a test was added to require inode data (file serial numbers) inside all directory listings. For example, if your directory includes "wazero.txt", a non-zero inode must be returned

inode data isn't typically used except comparing file identity, and even then requires the device to do so properly. For example, Go has a SameFile function which lazily gets this data, as it is well.. expensive. wasi-libc made a change recently to fetch this data regardless of it it is used or not. Not all compilers use wasi-libc, for example Zig has their own directory logic. However, at least C and Rust do, so mitigating this problem became an issue for us, not just for the spectest, but the underlying logic in wasi-libc. Specifically, compilers that use wasi-libc 17+ will perform a guest side stat fan-out when the ino returned is zero. Doing a guest-side fan out is much more expensive than host side.

This problem is roughly analogous to ReadDir vs Readdir problem in Go. The lower-d version says "Most clients are better served by the more efficient ReadDir method", ultimately due to an internal stat fanout in worst case. Unfortunately, the performance impact is worse than upper vs lower d readdir in go. In windows, the inode information isn't in the FileInfo.Sys data, so there's an additional N syscalls to get it from somewhere else.

System performance isn't consistent, but compliance with WASI on this is at least in tens of microseconds additional overhead on directory listings, and it is also linear wrt directory size. Not all users will list directories at after configuration time, neither will all find this overhead intolerable even if they did.

In case you are curious, we did discuss these topics at length with the WASI maintainers. The end of it is that this performance overhead is something they feel as a least bad option of choices before them. That said, progress was made, as they changed the next specification, wasi-filesystem to be performant by default, both not requiring dot directories or eager inodes. We're grateful to have been listened to.

If your performance after initialization time is dominated by this, you may of course file an issue to request a non-strict wasi setting. That said, we'd prefer to not make that setting. We have near term plans, likely this summer, to make the whole filesystem behaviour pluggable. In other words, avoiding this cost will be something you can choose to do on your own later, even for the current WASI specs.

If you'd like to discuss more on this or anything else, jump on gophers slack #wazero channel. Note: You may need an invite to join gophers. If you like what we are doing, please star our repo as folks appreciate it.

wazero v1.0.0-pre.9 integrates Go context to limit execution time running third party code. This is our last API affecting version before 1.0 in March. We'll cut at least one release candidate between now and then.

For those only interested in breaking changes, here's what you need to do:

• Build with minimally Go 1.18
• Rename Runtime.InstantiateModuleFromBinary to Runtime.Instantiate

mod, err := r.InstantiateModuleFromBinary(ctx, guestWasm)
mod, err := r.Instantiate(ctx, guestWasm)

Those of you attending wasmio will be able to meet many contributors and end users in person. This conference is timed almost exactly with our 1.0 release, so quite convenient for the community. If interested about in-person and virtual activities around our release, join gophers slack #wazero channel. Note: You may need an invite to join gophers. Regardless, if you like what we are doing, please star our repo as folks appreciate it. Meanwhile, let's dig into this month's changes!

Stop runaways with Go context!

@mathetake led an exciting development, which allows more control of the third-party wasm you run with wazero. Specifically, a cancel or deadline context can now halt potentially endless loops.

Here's an example:

ctx := context.Background()

r := wazero.NewRuntimeWithConfig(ctx, wazero.NewRuntimeConfig().
	// Integrate go context with the WebAssembly runtime
	.WithCloseOnContextDone(true))
defer r.Close(ctx)

mod, _ := r.Instantiate(ctx, infiniteLoopWasm)

infiniteLoop := mod.ExportedFunction("infinite_loop")

// Add function-scoped timeouts however you like.
ctx, cancel := context.WithTimeout(ctx, time.Second)
defer cancel()

// Pass that context when calling a function to prevent it from looping.
_, err = infiniteLoop.Call(ctx)

When the context is done before wasm returns, you'll get an ExitError with one of these codes: ExitCodeContextCanceled or ExitCodeDeadlineExceeded

It is understood that some would like more features, such as metering or work-based limits. However, we are excited to have the most commonly requested form of function limiting implemented prior to 1.0. Without this last minute spike from @mathetake, it might not have happened, or ended up as an experiment.

Passing all the tests

Besides features, there has been a large amount of effort to pass all available system tests, on darwin, linux and windows. This means that not only do we pass our own integration tests, but also third party ones like so:

wasi-testsuite

wasi-testsuite is an emerging test suite by the custodians of the WASI (system calls for wasm, basically). These include tests compiled from multiple languages including AssemblyScript, C, rust. Runtimes adapt into the suite, so there's one for wazero similar to wasmtime.

With the exception of a minor detail about dot vs dot-dot directory entries (ignored by most wasm compilers), wazero's CLI passes all tests on darwin, linux and windows.

Thanks to @evacchi and @loganek for infrastructure changes on wasi-testsuite which allowed us to be able to test windows. We'd also like to thank @sunfishcode for elaboration on various compatibility points we hit along the way, especially for working these feedback into the next version of WASI.

To pass all the tests meant we had to change from only implementing functions users request, to basically any function accessed by tests. Thanks to @codefromthecrypt and @mathetake for backfilling over ten WASI functions, as well @evacchi @ncruces as well newcomer @egonelbre for helping with various platform support issues.

TinyGo

TinyGo is the de facto Go compiler for WASI. We've had TinyGo tests for a long time, but this is the first time we can execute TinyGo's tests with our CLI using only configuration. These pass on darwin, linux and windows.

@mathetake lead this work, but this also involved support on the TinyGo side, especially to allow another runtime besides wasmtime to execute tests. It is now possible execute tinygo test in a way that it runs WebAssembly tests with our CLI. This is thanks to recent work by @anuraaga and @codefromthecrypt, supported by our TinyGo champions @deadprogram and @dgryski.

Zig

Zig is a very popular language that compiles to wasm, specifically targeting WASI. Part of supporting Zig are issues like ensuring wazero supports all host functions that it might call (such as fd_readdir). Other parts are making sure our CLI can execute their system tests, and those tests pass.

A big milestone happened this this month where @mathetake merged a build check that requires zig system tests to pass on darwin, linux and windows. These passing rides on work similar to wasi-testsuite, but also required changes to Zig both requested by us and those done on their own. Special thanks to @brendandburns @evacchi @jedisct1 @Luukdegram for collaborating to ensure things work both in Zig and in wazero before users notice!

Minor Changes

• Adds Runtime.InstantiateWithConfig to allow configuration without a compile step.
• Adds ModuleConfig.WithOsyield to allow users to control behavior of WASI sched_yield
• Adds a CLI flag -interpreter to force use of the interpreter engine
• Adds a CLI flag -env-inherit to propagate ENV to wasm, useful in Docker.
• Adds concurrent-instantiation example

wazero v1.0.0-pre.8 adds a filesystem configuration API that supports writes, tested by multiple third-party suites. It also brings CompilationCache out of experimental state, obviating a hard to explain Namespace API. Finally, this adds more logging scopes.

We don't expect any API changes next month, as we prepare for wazero 1.0 in March. Most of the scheduled work will be completing WASI and improving tests, so that our first formal release is trustworthy. Please upgrade to this version and give us feedback on how it's going.

The best way to contact us is to join gophers slack #wazero channel. Note: You may need an invite to join gophers. If you like what we are doing, please star our repo as folks appreciate it. Meanwhile, let's dig into this month's changes!

Writable filesystem support

An exciting change for many is the ability to configure writeable filesystems. To do that, we've added ModuleConfig.WithFSConfig which has options to mount directories or a fs.FS such that wasm can access it. Before, we had ModuleConfig.WithFS, and we'll leave that forever. This will help reduce complexity for simple-case configuration.

Here's an example of how to allow read access to the current directory as the root filesystem, while write access to a different directory as "/tmp"

moduleConfig = wazero.NewModuleConfig().
	// Make the current directory read-only accessible to the guest.
	WithReadOnlyDirMount(".", "/").
	// Make "/tmp/wasm" accessible to the guest as "/tmp".
	WithDirMount("/tmp/wasm", "/tmp")

Under the scenes, this maps to appropriate WebAssembly primitives, namely "preopens" for those compiling WASI or a virtual root if GOOS=js.

Those using the wazero CLI can take advantage of this with mount-based syntax, which looks very similar to Docker.

For example, here's the same configuration via the command line:

$ wazero run -mount=.:/:ro -mount=/tmp/wasm:/tmp ...

Under the scenes is more comprehensive than last time. Those compiling source via WASI or GOOS=js can take advantage of newly supported system calls, tested on Linux, MacOS and Windows operating systems. Thanks very much to @codefromthecrypt @evacchi @mathetake and @ncruces for collaborating on these!

• fd_filestat_set_size
• fd_filestat_set_times
• fd_sync
• fd_tell
• fd_pwrite

Some of you may wonder about our progress on a custom filesystem plugin. We have plans to do that, but after version 1.0. This configuration API was designed to be forwards compatible with a raw filesystem plugin once it is ready.

Standard Library Integration Tests

For the first time, wazero change depends on 3rd party integration tests for aspects beyond the WebAssembly Core specification. Specifically, we use multiple tests to ensure WASI not only works based on what the spec leads define, but also work in practice in TinyGo and Zig programming languages. By running multiple tests we are able to get an implicit quorum of what certain functions are expected to do, and reduce the amount of surprise by end users who simply want things to work.

For example, we use wazero instead of wasmtime to run TinyGo wasi target tests. If any fails, our build breaks. We are nearly there with Zig, too, and will be by next month. Both of these are also thanks to the language communities themselves, who have helped champion patches needed to make things portable.

We also run the emerging wasi-testsuite, defined by the spec team. We pass tests they define for the AssemblyScript and C programming languages. We don't yet pass all rust tests recently added from wasmtime: 5 fail mostly due to some edge case functions we've not yet implemented. However, we expect to pass all of them by 1.0 or sooner, or have a very good reason if we don't.

Beyond WASI, we also test the GOARCH=wasm GOOS=js platform baked into the standard Go sdk. This is typically tested via the node.js runtime (which uses V8), but our CLI works in lieu of that also. As Go considers this experimental, we don't require passing all tests, yet. That said we test each function that also exists in WASI. This helps existing users of GOARCH=wasm GOOS=js as well paves an easier transition for those working on the upcoming Go WASI proposal.

Getting these tests understood and integrated into our CI took a lot of effort, with special thanks to @achille-roussel @evacchi and @mathetake for their hard work.

Compilation Cache

In previous versions, we had experimental support for compilation cache. This reduces the first-request penalty on cold starts from the same wasm file by re-using work serialized to disk. We've now exported CompilationCache as a stable API which can be configured by RuntimeConfig. Along the way, we removed the complicated Namespace type as the same isolation+performance benefit can happen by sharing a compilation cache between runtimes. Thanks @mathetake for the hard work simplifying the user experience!

Here's an example of this in action:

// Initializes the new compilation cache with the cache directory.
// This allows the compilation caches to be shared even across multiple OS processes.
cache, err := wazero.NewCompilationCacheWithDir(cacheDir)
if err != nil {
	log.Panicln(err)
}
defer cache.Close(ctx)

// Creates a shared runtime config to share the cache across multiple wazero.Runtime.
runtimeConfig := wazero.NewRuntimeConfig().WithCompilationCache(cache)

// Creates two wazero.Runtimes with the same compilation cache.
runtimeA := wazero.NewRuntimeWithConfig(ctx, runtimeConfig)
runtimeB := wazero.NewRuntimeWithConfig(ctx, runtimeConfig)

HostLogging Scopes

In this version, we've added several log scopes that can compose together, and give you more insight into what's happening in the third-party code you are loading. Thanks @codefromthecrypt for the continued work on making execution easier to understand.

Here's example CLI output that uses the "exit" and "filesystem" logging scopes.

$ wazero run --hostlogging=exit,filesystem --mount=.:/:ro cat.wasm /not_found.txt
==> wasi_snapshot_preview1.fd_prestat_get(fd=3)
<== (prestat={pr_name_len=1},errno=ESUCCESS)
==> wasi_snapshot_preview1.fd_prestat_dir_name(fd=3)
<== (path=/,errno=ESUCCESS)
==> wasi_snapshot_preview1.fd_prestat_get(fd=4)
<== (prestat=,errno=EBADF)
==> wasi_snapshot_preview1.fd_fdstat_get(fd=3)
<== (stat={filetype=DIRECTORY,fdflags=,fs_rights_base=,fs_rights_inheriting=},errno=ESUCCESS)
==> wasi_snapshot_preview1.path_open(fd=3,dirflags=SYMLINK_FOLLOW,path=not_found.txt,oflags=,fs_rights_base=,fs_rights_inheriting=,fdflags=)
<== (opened_fd=,errno=ENOENT)
==> wasi_snapshot_preview1.proc_exit(rval=1)

Those programming in Go can also use this, but note that this API is still experimental, so can cause version compatibility problems. Only compile this API if you are able to change that code when upgrading wazero.

loggingCtx := context.WithValue(testCtx, experimental.FunctionListenerFactoryKey{},
	logging.NewHostLoggingListenerFactory(&log, logging.LogScopeRandom|logging.LogScopeExit))

Other interesting changes

• Thanks @mathetake for various compiler fixes and better fuzzing support
• Thanks @evacchi for adding RuntimeConfig.WithCustomSections(), which allows users to inspect sections wazero doesn't use.
• Thanks @evacchi for starting the Zig language page https://wazero.io/languages/zig/