Summary

This high-priority hot-fix address high CPU usage experienced after the Capella upgrade (April 12 UTC).

We recommend all mainnet users update to this release to prevent high CPU usage from interfering with normal node operations (e.g., staking).

This hot-fix only applies to the Beacon Node. There is no need to upgrade the Validator Client from v4.0.1 (or v4.0.1-rc.0) to this release.

Building from Source

This hot-fix is not yet applied to the unstable or stable branches. To build from source, use the hotfix-exit-verification branch (or checkout the v4.0.2-rc.0 tag).

Breaking Changes

There are no breaking changes in this release.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	High	Low
Non-Staking Users	High	---

See Update Priorities more information about this table.

All Changes

• Bump versions
• Empty commit
• Use head state for exit verification

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.2-rc.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.2-rc.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v4.0.2-rc.0	sigp/lighthouse

Summary

This release is mandatory for all mainnet users (except for those running v4.0.1-rc.0). It enables the Capella/Shanghai ("Shapella") upgrade (#4111), which will occur at April 12, 2023, 10:27:35pm UTC. Any node which is not updated to Lighthouse a v4.x.x release before 10:27pm on April 12th (UTC) will stop following the chain and will need to be resynced. For stakers, this would result in missed rewards and penalties.

For clarity, all mainnet Lighthouse users must be running a v4.x.x release on their BNs and VCs by April 12, 2023, 10:27:35pm UTC.

This release also contains various bug-fixes, optimisations and new features:

• Improve the resilience of the fork choice algorithm (#3962)
• Add a flag to speed up responses to the committees HTTP API (#4081)
• Improve payload reconstruction by utilising a new execution engine API endpoint (#4028)
• Reduce false-positive ERRO logs claiming that builder blocks were published late (#4073)
• Fix a bug that resulted in a harmless ERRO Dialing an already dialing peer log (#4056)
• Add support for IPv6 (#4046)

v4.0.0 Release Retracted

This release replaces the v4.0.0 release which was retracted due to the discovery of a bug. A v4.0.1-rc.0 hot-fix was published to patch that bug. That bug fix is also included in this release.

This release (v4.0.1) replaces both v4.0.0 and v4.0.1-rc.0.

We have the following advice for users:

• Any node running v4.0.0 should immediately upgrade to this release since v4.0.0 is not reliable.
• Any node running v4.0.1-rc.0 may choose to continue on that release or upgrade to this release. This release and the rc.0 release are functionally identical.
• Any node running v3.5.1 or earlier should upgrade to this release at their next convenience (certainly before the Capella upgrade).

These release notes have been written with respect to v3.5.1 rather than the previous retracted release of v4.0.0.

Mainnet Capella/Shanghai ("Shapella") Upgrade

The Capella/Shanghai ("Shapella") upgrade will occur on mainnet at:

• Epoch 194048
• April 12, 2023, 10:27:35pm UTC

All Lighthouse Beacon Nodes and Validator Clients must be upgraded to a reliable v4.x.x release to ensure they follow the correct chain. The following table demonstrates which releases are reliable, Shapella-ready releases:

Release	Is Reliable, Shapella-Ready v4.x.x Release
v3.5.1	❌ No
v4.0.0	❌ No
v4.0.1-rc.0	✅ Yes
v4.0.1	✅ Yes
Any release after v4.0.1	✅ Yes

Preparation for the Shapella upgrade is much simpler than the preparation required for "The Merge" (Bellatrix). To be Shapella ready, users just need to:

• Upgrade their Lighthouse BN(s) to a v4.x.x release.
• Upgrade their Lighthouse VC(s) to v4.x.x release.
• Upgrade their Execution Engine(s) to a Shanghai-ready release:
  • Besu: 23.1.2 or later
  • Geth: v1.11.5 or later
  • Nethermind: v1.17.3 or later
  • Erigon: v2.42.0 or later

If your execution engine does not yet have a Shanghai-ready release then it is safe to upgrade Lighthouse to v4.x.x without also upgrading the execution engine. An up-to-date execution engine will be required before April 12th, though.

There are no new flags to be added or removed for the Shapella upgrade, simply upgrade and wait.

Lighthouse will start periodically emitting the following logs two weeks before the Shapella upgrade (29th of March):

• Not ready for Capella if Lighthouse has detected that the execution engine is too outdated to support Shanghai.
• Ready for Capella if Lighthouse has detected a modern execution engine release.

Just because Lighthouse is logging Ready for Capella does not indicate that your execution engine is on the correct version. There is no way for Lighthouse to determine this exactly and users are responsible for ensuring that their execution engine is using the latest release.

:bug: Known Issues :bug:

Fixed as of Erigon v2.42.0

There was a bug in Erigon v2.41.0 and earlier which caused Erigon to time out when paired with Lighthouse v4.0.1. This was reflected in the Lighthouse logs as an error log containing the phrase TimedOut, like this:

ERRO Error during execution engine upcheck error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec

The issue is resolved in Erigon v2.42.0 and later, see this issue for details: https://github.com/ledgerwatch/erigon/issues/7172.

Breaking Changes

There are no breaking changes between this release and v4.0.0 and v4.0.1-rc.0. The following changes are with reference to v3.5.1.

Breaking Change: Database Schema V16

To support changes to the fork choice algorithm, the database schema has been upgraded to v16. The schema upgrade will be applied automatically upon upgrading and should not take more than a few seconds.

To downgrade, follow the instructions in the book for Database Migrations.

Users may downgrade anytime before the Capella upgrade. Prior Lighthouse releases are not compatible with Capella, so downgrading is fundamentally impossible after the upgrade occurs.

Breaking Change: Minimum Supported Rust Version 1.66

The minimum supported Rust version has been set to 1.66. Users who compile from source (i.e., not Docker or pre-built binary users) will receive the following error if they are using an earlier version of Rust:

lighthouse v4.0.0 (/home/karlm/lighthouse/lighthouse)` cannot be built because it requires rustc 1.66 or newer

Users can typically obtain the latest version of Rust by running rustup update.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	High	High
Non-Staking Users	High	---

See Update Priorities more information about this table.

All Changes

• Release v4.0.1 (#4125)
• Release Candidate v4.0.1-rc.0 (#4123)
• Fix fork choice error message (#4122)

• Release v4.0.0 (#4112)
• Fork choice modifications and cleanup (#3962)
• Set Capella fork epoch for Mainnet (#4111)
• Reduce verbosity of reprocess queue logs (#4101)
• Customisable shuffling cache size (#4081)
• Improve Lighthouse Connectivity Via ENR TCP Update (#4057)
• Ignore self as a bootnode (#4110)
• Reconstruct Payloads using Payload Bodies Methods (#4028)
• Clarify "Ready for Capella" (#4095)
• Reduce false positive logging for late builder blocks (#4073)
• Make more noise when the EL is broken (#3986)
• Siren Ui Lighthouse Version Requirments (#4093)
• Correct a race condition when dialing peers (#4056)
• Remove Router/Processor Code (#4002)
• Complete match for has_context_bytes (#3972)
• Add parent_block_number to payload SSE (#4053)
• Support for Ipv6 (#4046)
• Correct /lighthouse/nat implementation (#4069)
• Added warning when new jwt is generated (#4000)
• Appease Clippy 1.68 and refactor http_api (#4068)
• Fix order of arguments to log_count (#4060)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.1-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.1-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v4.0.1	sigp/lighthouse

🚨 Hot-Fix for v4.0.0 🚨

This release is a hot-fix to patch a bug introduced in v4.0.0, which was published Wednesday, March 22, 2023 11:56:00 PM UTC. The v4.0.0 release has been removed from the Github releases page.

The bug was found by our fuzzers and has not been triggered on any networks. If triggered, it would result in temporary periods of effective downtime of several slots.

It is unlikely that this bug will present itself on mainnet, however we recommend the following actions for all networks (Mainnet, Goerli, Prater, Sepolia, Gnosis, etc):

• If you have not upgraded to v4.0.0 and are on v3.5.1, then take no action. Do not upgrade to v4.0.0 or v4.0.1-rc.0.
• If you have already upgraded to v4.0.0, then upgrade to v4.0.1-rc.0 immediately.

We are not recommending v4.0.0 users downgrade to v3.5.1 since a manual database migration is required. Users familiar with database migrations are safe to downgrade.

If you are not sure if you've updated, you can run lighthouse --version to find out.

The Docker stable flag has been updated to point to the patched version. If you have updated your stable Docker containers after Wednesday, March 22, 2023 11:04:00 PM UTC, we recommend updating again to obtain the patch.

All Changes

• Release Candidate v4.0.1-rc.0 (#4123)
• Fix fork choice error message (#4122)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.1-rc.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.1-rc.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v4.0.1-rc.0	sigp/lighthouse

Summary

This release is a low-priority release for all users except for those using the Goerli test network.

For Goerli (aka Prater) users, this release includes the Capella fork parameters (#4044). Goerli will undergo the Capella upgrade (a.k.a "hardfork") on 14/03/2023 at 10:25:36 pm UTC. Goerli users must update to this version or later before the Capella upgrade. Users that fail to upgrade will cease to follow the chain and will be required to resync.

Users of Ethereum Mainnet or other networks/testnets (e.g., Sepolia, Gnosis Chain) may choose to update to this release to take advantage of various optimisations and bug fixes.

Notable changes in this release include:

• Add a flag to always use payloads from builders (#4052)
• Set Capella fork epoch for Goerli (#4044)
• Optimise signing in the VC to prevent late blocks (#4033)
• Add a BN latency measurement to the VC to help diagnose late blocks (#4024, #4051)
• Fix a bug where Lighthouse rejects an invalid block message from the EE (#4037)
• Delete Kiln and Ropsten configs (#4038)
• Improve compilation flexibility by allowing building without MDBX (#3888)

Breaking Changes

There are no breaking changes that affect Mainnet users.

⚠️ Breaking Change: Removal of Kiln and Ropsten Configs ⚠️

The --network kiln and --network ropsten flags are no longer supported. Kiln was deprecated Q3 2022 and Ropsten was deprecated in Q4 2022. Removing these networks has reduced the size of the Lighthouse binary by approximately 30MiB.

Dedicated Ropsten and Kiln users can still use these networks by manually obtaining the testnet configuration directories and using the --testnet-dir flag.

Goerli Capella Hard Fork

Goerli is upgrading to Capella on 14/03/2023 at 10:25:36 pm UTC! :tada:

Goerli users must update both the Lighthouse beacon node and validator client before March 14. Failure to update will result in missed validator duties and a corrupt beacon node database (requiring a re-sync).

Goerli users must also ensure they are running a compatible execution engine. The Ethereum Foundation will publish a "Goerli Shapella Announcement" in the coming days which will contain more information and a list of compatible execution layer releases.

Lighthouse will emit INFO Ready for Capella logs when both itself and the execution engine are ready for the Capella upgrade. Conversely, Lighthouse will emit WARN Not ready for Capella logs when it detects a misconfiguration.

VC to BN Latency Measurements

In #4024 and #4051 we added new Prometheus metrics for monitoring the round-trip latency between a VC and the BN. The latency measurement is the time it takes the VC to send, receive and parse a call to the BN's eth/v1/node/version endpoint.

There are two new metrics exposed by the VC:

• vc_beacon_node_latency_primary_endpoint: shows the latency for the primary BN.
• vc_beacon_node_latency: shows the latency for all BNs, with the endpoint URL as the endpoint label.

Producing a block requires two HTTP calls between the VC and the BN. Therefore, latency on this connection can contribute significantly to block publishing delays. Blocks which are published late are more likely to be orphaned.

Those using Grafana+Prometheus can use the following query to view the 99th percentile latency for their primary BN:

histogram_quantile(0.99, rate(vc_beacon_node_latency_primary_endpoint_bucket[$__rate_interval]))

Known Issues

The optimised maxperf profile is currently not working on Windows due to a suspected regression in the Rust compiler. The pre-compiled Windows binaries have been built with the release profile instead and may exhibit slightly reduced performance compared to previous versions. See https://github.com/sigp/lighthouse/issues/3964 for details.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	Low Priority	Low Priority
Non-Staking Users	Low Priority	---

The Beacon Node may be updated without the Validator Client, however we recommend updating both components for completeness.

Goerli users must update both the beacon node and the validator client.

See Update Priorities for more information about this table.

All Changes

• Release v3.5.1 (#4049)
• Add a flag to always use payloads from builders (#4052)
• Set Capella fork epoch for Goerli (#4044)
• Add VC metric for primary BN latency (#4051)
• Log a WARN in the VC for a mismatched Capella fork epoch (#4050)
• Update dependencies incl tempfile (#4048)
• Optimise attestation selection proof signing (#4033)
• Optimise payload attributes calculation and add SSE (#4027)
• Add latency measurement service to VC (#4024)
• Permit a null LVH from an INVALID response to newPayload (#4037)
• Log a debug message when a request fails for a beacon node candidate (#4036)
• Cleaner logic for gossip subscriptions for new forks (#4030)
• Delete Kiln and Ropsten configs (#4038)
• Clean capella (#4019)
• Add more logs in the BN HTTP API during block production (#4025)
• Docs for Siren (#4023)
• Use consensus-spec-tests v1.3.0-rc.3 (#4021)
• Add content-type header to metrics server response (#3970)
• Allow compilation with no slasher backend (#3888)
• Execution Integration Tests Correction (#4034)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.5.1-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.5.1-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.5.1	sigp/lighthouse

Summary

This release is a low-priority release for all users (including mainnet), except for those using the Sepolia test network

For Sepolia users, this release includes the Capella fork parameters. Sepolia will undergo the Capella upgrade (a.k.a "hardfork") on 2023/2/28 at 4:04:48 AM UTC. Sepolia users must update to this version or later before the Capella upgrade. Users that fail to upgrade will cease to follow the chain and will be required to resync.

Ethereum mainnet, Ethereum testnet (e.g., Goerli, Ropsten) and Gnosis users may choose to update to this release, but it is low-priority.

Alongside the Sepolia changes, this release includes:

• :warning: Database schema upgrade to v15 :warning:
• HTTP API returns 404 rather than 405 for an unknown route (#3836)
• Add CLI flag to specify the format of logs written to the logfile (#3839)
• Switch allocator to jemalloc (#3697)
• Implement new rewards APIs (#3822, #3903, #3907)
• Add attestation duty slot metric (#2704)
• Reduce some EE and builder related ERRO logs to WARN (#3966)
• Use Local Payload if More Profitable than Builder (#3934)

Breaking Changes

:warning: Breaking Change: Database Schema v15 :warning:

To support Capella the database schema has been upgraded to v15. The schema upgrade will be applied automatically upon upgrading and should not take more than a few seconds.

To downgrade, follow the instructions in the book for Database Migrations.

Users may downgrade anytime before the Capella upgrade. Prior Lighthouse releases are not compatible with Capella, so downgrading is fundamentally impossible after the upgrade occurs.

:warning: Breaking Change: Minimum Supported Rust Vesion :warning:

Users building from source will have to update their Rust compiler version to at least 1.65.0 using a command like:

rustup update stable

Newer versions of the compiler including the latest (v1.67.1 at time of writing) will also work.

Older versions will log an error during compilation.

Breaking Change: HTTP API Unknown Routes

Previously, the Beacon Node HTTP would return 405 (method not allowed) for an unknown route (e.g. http://localhost:5052/not_a_real_route). Lighthouse will now return a 404 (not found) error.

Sepolia Capella Hard Fork

Sepolia is upgrading to Capella on 2023/2/28 at 4:04:48 AM UTC! :tada:

Sepolia users must update both the Lighthouse beacon node and validator client before February 28. Failure to update will result in missed validator duties and a corrupt beacon node database (requiring a re-sync).

Sepolia users must also ensure they are running a compatible execution engine. The Ethereum Foundation Sepolia Shapella Annoucement contains more information and a list of compatible execution layer releases.

Lighthouse will emit INFO Ready for Capella logs when both itself and the execution engine are ready for the Capella upgrade. Conversely, Lighthouse will emit WARN Not ready for Capella logs when it detects a misconfiguration.

Known Issues

The optimised maxperf profile is currently not working on Windows due to a suspected regression in the Rust compiler. The pre-compiled Windows binaries have been built with the release profile instead and may exhibit slightly reduced performance compared to previous versions. See https://github.com/sigp/lighthouse/issues/3964 for details.

Update Priority

This table provides priorities for which classes of mainnet users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	Low Priority	Low Priority
Non-Staking Users	Low Priority	---

Note: this update is high-priority for Sepolia users.

The Beacon Node may be updated without the Validator Client, however we recommend updating both components for completeness. Sepolia users must update both the beacon node and the validator client.

See Update Priorities for more information about this table.

All Changes

This release contains the Capella functionality (#3763), which has been the result of several months of work. To ensure rightful attribution to its contributors, we have elected to perform a merge commit rather than our usual squash-merge. This has created an unusually complex history on the stable branch, so we have omitted the full list of changes in this release.

The full list of commits between this version (v3.5.0) and the previous version (v3.4.0) can be viewed on Github.

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.5.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.5.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.5.0	sigp/lighthouse

Disclaimer

:warning: You should not run this alpha release supporting mainnet validators :warning:

If you are looking for the latest Lighthouse release please go to https://github.com/sigp/lighthouse/releases/latest.

Summary

This is an alpha release of upcoming changes to Lighthouse which improve disk usage and state management.

For more information about the -tree.X family of releases, please see the v3.4.0-tree.1 release notes.

Compared to the previous release the main changes are:

• Fixed a database corruption bug which could occur when reconstructing historic states (dbb0cf7099600ea62e97ca53d1c868ce486469b7, 2f6ffff8d6495d748163eb00c57be9ad9e87c99d).
• Improved interaction between state reconstruction and finalization (481e79289880b75e53cbfea1be07564b1b437323).
• Latest changes from unstable, including new rewards APIs.

Note that there is no v3.4.0-tree.2 release due to a build failure.

:warning: Backwards Compatibility :warning:

This release is not backwards compatible with stable Lighthouse. It uses a different database schema (v20) for which no automatic upgrade or downgrade is implemented. We intend to implement an automatic upgrade process once the new schema is finalized. A re-sync will be required to run future versions of tree-states.

This release is backwards compatible with the prior tree-states release (v3.4.0-tree.1).

Please only run this release if you are willing to re-sync now, and again in several weeks/months.

Known Issues

Expect a few sharp edges. Some things you may run into:

• WARN Parent state is not advanced is logged excessively during sync. This is harmless, albeit annoying.
• Block processing is a bit slower than stable Lighthouse.
• The Windows binary will not compile with the maxperf profile (use release).

Building from source

Build the v3.4.0-tree.3 tag.

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-tree.3-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-tree.3-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.4.0-tree.3	sigp/lighthouse

Disclaimer

:warning: You should not run this alpha release supporting mainnet validators :warning:

If you are looking for the latest Lighthouse release please go to https://github.com/sigp/lighthouse/releases/latest.

Summary

This is an alpha release of upcoming changes to Lighthouse which improve disk usage and state management.

We are making this alpha release so that expert users may help us test these improvements. It is not backwards-compatible and not recommended for mainnet validators.

For the adventurous, the main benefits are:

• Smaller disk footprint for archive nodes, <300GB total. Use checkpoint sync, and set the flags --slots-per-restore-point=32 and --reconstruct-historic-states.
• Smaller disk footprint for regular nodes, <70GB.
• Faster handling of re-orgs.
• Faster restarts.
• Less disk I/O.

We hope that this is useful for running block explorers and supporting other beacon chain analytics. We are using it internally at SigP to run some of our analytics.

Aside from the experimental tree-states changes this release is up-to-date with Lighthouse v3.4.0 and includes the same features.

:warning: Backwards Compatibility :warning:

This release is not backwards compatible with stable Lighthouse. It uses a different database schema (v20) for which no automatic upgrade or downgrade is implemented. We intend to implement an automatic upgrade process once the new schema is finalized. A re-sync will be required to run future versions of tree-states.

Please only run this release if you are willing to re-sync now, and again in several weeks/months.

Known Issues

Expect a few sharp edges. Some things you may run into:

• WARN Parent state is not advanced is logged excessively during sync. This is harmless, albeit annoying.
• The JSON serialisation of the BeaconState does not quote integers for some fields. This will be fixed shortly, but SSZ can be used as a workaround.
• Block processing is a bit slower than stable Lighthouse (see below).

Additional Background

The fundamental change supporting these improvements is a change to the in-memory representation of the BeaconState from flat vectors to copy-on-write trees. This enables many more BeaconStates to be stored in memory, but comes at the cost of increased iteration time. As a result, block and epoch processing times are slower. To mitigate this we are working on new caches and optimisations which bring block processing times back in line with stable Lighthouse. This puts the tree-states branch in a slightly awkward spot, because we cannot responsibly merge it until it is better than stable Lighthouse on every possible metric (a Pareto improvement).

Seeing as tree-states alters the database schema quite drastically, we also decided to roll in a few other improvements. The plan is to get the new schema as close to perfect as possible so that in a future release everyone can upgrade once with minimal fuss.

Building from source

Build the v3.4.0-tree.1 tag.

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-tree.1-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-tree.1-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.4.0-tree.1	sigp/lighthouse

Summary

This high-priority release contains important bug fixes, optimizations and a notable change to fork choice which incentivizes the timely production of blocks.

Other interesting changes include:

• Improved logging for debugging payloads from builders (#3725)
• Significantly improved syncing speeds (#3738, #3794)
• New custom Lighthouse API endpoints (#3756, #3760, #3779)
• Reduced log severity for late blocks and unrevealed builder blocks (#3775)
• Reduced log severity for validator monitor logs (#3727)
• Updated Gnosis bootnodes (#3793, #3855)
• Improved validator monitor for high validator counts (#3728)
• A fix for slow VC API performance with Web3Signer validators (#3801)

Proposer Boost Re-Orgs

Lighthouse nodes running this version (or later) will attempt to re-org blocks which arrive late and subsequently receive very few votes from other validators. Lighthouse will not attempt a re-org unless it is confident that its own block will become the head and the action is in the best interests of the Ethereum network.

It is expected that re-orging a late block will be significantly profitable for the entity which performs the re-org. The loss induced by the validator which produced the late block will incentivize performance improvements.

See Late Block Re-orgs in the Lighthouse Book for more information.

Breaking Changes

This release contains two breaking changes regarding the validator monitor on the BN.

1. Aggregate Validator Monitor Metric

A new flag has been added to the Beacon Node: --validator-monitor-individual-tracking-threshold. The default value is 64, which means that when the validator monitor is tracking more than 64 validators then it will stop tracking per-validator metrics and only track values via the new total label. It will also stop logging per-validator logs and only emit aggregated logs (the exception being that exit and slashing logs are always emitted).

The new total label tracks the aggregated metrics of all validators in the validator monitor (as opposed to each validator being tracking individually using its pubkey as the label). It only tracks values which are easily aggregated (e.g. total count of attestations seen) and omits values which are not easily aggregated (e.g. most recent inclusion distance).

These changes were introduced in #3728 to address issues with untenable Prometheus cardinality and log volume when using the validator monitor with high validator counts (e.g., 1000s of validators). Users with less than 65 validators will see no change in behavior (apart from the added total metric). Users with more than 65 validators who wish to maintain the previous behavior can set something like --validator-monitor-individual-tracking-threshold 999999.

2. Reduced Log Severity for Validator Monitor Logs

The validator monitor will no longer emit WARN and ERRO logs for sub-optimal attestation performance. The logs will now be emitted at INFO level. This change was introduced to avoid cluttering the WARN and ERRO logs with alerts that are frequently triggered by the actions of other network participants (e.g., a missed block) and require no action from the user.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	High Priority	Medium Priority
Non-Staking Users	High Priority	---

The Beacon Node may be updated without the Validator Client, however we recommend updating both components for completeness.

See Update Priorities more information about this table.

All Changes

• Release v3.4.0 (#3862)
• Update dependencies incl Tokio (#3866)
• Web3 signer validator definitions reloading on any request (#3801)
• Improve validator monitor experience for high validator counts (#3728)
• Add more Gnosis bootnodes (#3855)
• Verify execution block hashes during finalized sync (#3794)
• Upgrade to libp2p v0.50.0 (#3764)
• Restructure code for libp2p upgrade (#3850)
• Various CI fixes (#3813)
• docs: remove mention of phases in voluntary exits (#3776)
• Clippy lints for rust 1.66 (#3810)
• send error answering bbrange requests when an error occurrs (#3800)
• Update Gnosis chain bootnodes (#3793)
• Enable proposer boost re-orging (#2860)
• Make all validator monitor logs INFO (#3727)
• Adding light_client gossip topics (#3693)
• Reduce log severity for late and unrevealed blocks (#3775)
• Add API endpoint to get VC graffiti (#3779)
• Update book with missing Lighthouse endpoints (#3769)
• Expose certain validator_monitor metrics to the HTTP API (#3760)
• Delete DB schema migrations for v11 and earlier (#3761)
• Add API endpoint to count statuses of all validators (#3756)
• Prioritise important parts of block processing (#3696)
• Ipv6 bootnodes (#3752)
• Optimize finalized chain sync by skipping newPayload messages (#3738)
• Improve debugging experience for builder proposals (#3725)
• Add Run a Node guide (#3681)
• Gossipsub fast message id change (#3755)
• Add CLI flag for gui requirements (#3731)
• Add CLI flag to opt in to world-readable log files (#3747)
• remove commas from comma-separated kv pairs (#3737)
• Added LightClientBootstrap V1 (#3711)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.4.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.4.0	sigp/lighthouse

Summary

This release is a low-priority release for all users, except those using the Gnosis Chain.

For Gnosis users, this release includes the Gnosis Merge parameters. Gnosis users will need to update to this release before November 30th or they will fail to go through the merge transition and will be left following the wrong chain. Therefore, this release is high-priority for Gnosis users.

Ethereum mainnet and Ethereum testnet (e.g., Goerli, Sepolia) users may choose to update to this release, but it is low-priority.

Alongside the Gnosis changes, this release includes:

• :warning: Database schema upgrade to v13 :warning:
• Support for a new and upcoming improvement to checkpoint sync (#2915)
• Addition of the --checkpoint-sync-url-timeout flag (#3710)
• Improvements to sync committee message signing with optimistic EEs (#3624)

Breaking Changes

:warning: Breaking Change: Database Schema v13 :warning:

To support new features the database schema has been upgraded to v13. The schema upgrade will be applied automatically upon upgrading and should not take more than a few seconds.

To downgrade, follow the instructions in the book for Database Migrations.

Breaking Change: Sync Commmittee Fallback Behaviour

The signing of sync committee messages by the validator client has been improved in a way that is incompatible with old versions of the Lighthouse beacon node. We do not expect many users to be affected because only versions prior to v3.0.0 are incompatible.

For details please see: https://github.com/sigp/lighthouse/pull/3624

Gnosis Merge

Gnosis chain is merging! :tada:

The expected timeline is:

• November 30 2022: Bellatrix hard fork activated on the consensus layer.
• December 5-11 2022: Terminal total difficulty reached, merge completed.

You must update both the Lighthouse beacon node and validator client before November 30. Failure to update will result in missed validator rewards and a corrupt beacon node database (requiring a re-sync).

You must also ensure you're running a compatible verison of Nethermind (v1.14.6 or newer) and that it is correctly connected to your Lighthouse beacon node. For further instructions please see:

• Merge Migration - Lighthouse Book
• Gnosis Chain Merge Guide

Deposit Snapshot Sync

We are pleased to announce the readiness of deposit snapshot sync, which enables nodes to quickly sync the deposit contract.

When setting up a new Lighthouse node using checkpoint sync, a snapshot of the deposit contract will be downloaded from the checkpoint sync provider. Initially the only supported provider is Lighthouse v3.3.0, although we expect checkpointz and the other clients soon will roll out support soon.

If the checkpoint sync provider does not support the feature you'll see a pair of warning logs:

WARN Remote BN does not support EIP-4881 fast deposit sync WARN Failed to finalize deposit cache

These warnings are harmless, and Lighthouse will gracefully fallback to syncing the deposit contract cache from the execution node.

Depsosit snapshot support is the result of many months of work by @ethDreamer, who not only implemented the feature in Lighthouse but also wrote the spec for it, EIP-4811.

For further details on the implementation see: https://github.com/sigp/lighthouse/pull/2915

Update Priority

This table provides priorities for which classes of mainnet users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	Low Priority	Low Priority
Non-Staking Users	Low Priority	---

Note: this update is high-priority for Gnosis chain users.

The Beacon Node may be updated without the Validator Client, however we recommend updating both components for completeness. Gnosis users must update both the beacon node and the validator client.

See Update Priorities for more information about this table.

All Changes

• v3.3.0 (#3741)
• Lower deposit finalization error to warning (#3739)
• Schedule gnosis merge (#3729)
• Super small improvement: Remove unnecessary mut (#3736)
• Add metrics for subnet queries (#3721)
• Simplify GossipTopic -> String conversion (#3722)
• compile with beta compiler on CI (#3717)
• Health Endpoints for UI (#3668)
• CI gardening maintenance (#3706)
• Sync committee sign bn fallback (#3624)
• Add --light-client-server flag and state cache utils (#3714)
• add checkpoint-sync-url-timeout flag (#3710)
• Blinded block and RANDAO APIs (#3571)
• Register blocks in validator monitor (#3635)
• Added Merkle Proof Generation for Beacon State (#3674)
• Blocklookup data inconsistencies (#3677)
• Update stale sections of the book (#3671)
• Clarify error log when registering validators (#3650)
• Fix rust 1.65 lints (#3682)
• Deposit Cache Finalization & Fast WS Sync (#2915)
• Update discv5 (#3171)
• Book spelling and grammar corrections (#3659)
• Added lightclient server side containers (#3655)

Summary

This medium-priority release is a patch release for the recently released v3.2.0, fixing a performance regression.

Please see the release notes for v3.2.0 for a summary of included features and breaking changes:

https://github.com/sigp/lighthouse/releases/tag/v3.2.0

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	Medium Priority	Medium Priority
Non-Staking Users	Low Priority	---

The Beacon Node may be updated without the Validator Client and vice versa, as long as both BN and VC are running a v3.x release.

Beacon nodes and validator clients on pre-v3.0.0 versions have been unable to follow the chain since September 6 2022 when the Bellatrix hard fork was activated.

See Update Priorities for more information about this table.

All Changes

• Release v3.2.1 (#3660)
• Revert "Optimise HTTP validator lookups" (#3658)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.2.1-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.2.1-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.2.1-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.2.1-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.2.1-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.2.1-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.2.1-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.2.1-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.2.1	sigp/lighthouse