🇪🇺 Wire back-end services
Deploy spar before galley (#2543)
Upgrade team-settings version to 4.11.0-v0.31.1-0-9e64150 (#2180)
2nd factor authentication code generation is rate limited now (#2522)
The team member CSV export now fills created_on for SCIM users (#2543)
Add Helm chart for inbucket. Inbucket is a SMTP server that does not relay, but instead display received mail in a webapp and make them accessible via an API. (#2544)
Bump saml2-web-sso (#2545, #2546)
use checkedConnectCluster to avoid dropping requests to Redis when Gundeck reconnects to the Redis cluster (#2542)
Do not log polysemy errors in Galley (#2531)
Remove old crypto-cli tool from the ubuntu image (#2538)
Added new-remote-conversation RPC, used to notify a backend of a remote conversation the first time any user from that backend is added to it. (#2378)
Added federation endpoint send-mls-message used to send messages to remote converesations (#2378)
For users of the (currently alpha) coturn Helm chart: manual intervention may be required when upgrading to this version of the chart from a prior version, due to a bug in Kubernetes which may interfere with applying changes to pod and service port configuration correctly.
If, after updating this chart, the coturn pods do not have both a coturn-udp
port and a coturn-tcp port, then the coturn StatefulSet must be manually
deleted from the cluster, and then recreated by re-running Helm. Similarly, if
the coturn Service does not have both a coturn-udp port and a coturn-tcp
port, this Service must also be deleted and recreated. (#2500)
The nginz{-tcp,-http} services have been unified into a nginz service, and
moved into the nginz chart.
The nginz-ingress-services chart simply targets the nginz service, so there's
no need to set matching service.nginz.external{Http,Tcp}Port inside the
nginx-ingress-services chart anymore.
The config.http.httpPort and config.ws.wsPort values in the nginz chart
still configure the ports the nginz service is listening on.
Metrics were moved from config.http.httpPort to a new http-metrics port.
The nginz chart also gained support for metrics.serviceMonitor.enabled,
creating a ServiceMonitor resource to scrape metrics, like for other wire
services.
(#2476)
Upgrade webapp version to 2022-06-30-production.0-v0.30.5-0-3e2aaf6 (#2302)
In the helm charts, the wireService label has been removed.
In some cases, we were already setting the app label too.
Now we consistently use the app label to label different wire services.
The wireService label was also used in the spec.selector.matchLabels field
on existing Deployment / StatefulSet resources.
As these fields being immutable, changing them isn't possible without recreation.
If you encounter an issue like
field is immutable && cannot patch "*" with kind *
you need to manually delete these StatefulSet and Deployment resources, and apply helm again, which will recreate them.
This means downtime, so plan a maintenance window for it.
The wire-server-metrics chart was previously running some custom
configuration to automatically add all payloads with a wireService label into
metrics scraping.
With the removal of the wireService label, this custom configuration has been
removed.
Instead, all services that expose metrics will now create ServiceMonitor
resources, if their helm chart is applied with metrics.serviceMonitor.enable
set to true.
This prevents scraping agents from querying services that don't expose metrics at /i/metrics unnecessarily.
Additionally, makes it easier to run other metric scraping operators, like
grafana-agent-operator, without the need to also create some custom
wireService label config there.
Generally, if you have any monitoring solution installed in your cluster that
uses the Prometheus CRDs, set metrics.serviceMonitor.enable for the following charts:
The request body of POST /conversations endpoint can now contain an optional creator_client field. The creator_client field is only relevant for MLS conversations, in which case it must be set to the ID of the client making the request. (#2486)
Retire deprecated feature config API endpoints for API version V2 (#2492)
Prevent race conditions in concurrent MLS commit requests. (#2525)
charts/wire-server: Optionally include backoffice (#2490)
The coturn chart has new functionality to enable graceful pod termination, by waiting for all active allocations on a coturn instance to drain first. When combined with a suitable external service discovery mechanism which can steer client traffic away from terminating coturn pods, this can be used to implement graceful rolling restarts of clusters of coturn instances. (#2456)
./deploy/services-demo/create_team_members.sh creates users with given roles now (#2137)
MLS implementation progress:
charts/nginz: Serve swagger-ui for viewing swagger-1.2 docs (#2466)
GET teams/:tid response now contains an optional field splash_screen which contains the asset key of the team's splash screen. PUT teams/:tid now supports updating the splash screen asset key. (#2474)
Missing feature config mapping added (#2494)
Add MLS team feature configuration (#2499)
Team feature API now includes endpoints to get and set the searchVisibilityInbound feature (#2503)
charts/backoffice: Fix version of frontend and auto-bump version of stern on every release (#2490)
The service definitions in the coturn Helm chart were missing the control plane UDP port used by coturn. (#2500)
In nginx-ingress-services chart, when enabling useCertManager, now correctly creates the required issuer by default. (#2532)
Fix handling of creator client in MLS conversations (#2486)
Fix all clients having the same MLS public key (#2501)
A user now cannot delete an identity provider that they are authenticated with any more (#2519)
brig-types: remove all re-exports (#2505)
Fixed flakiness of email update test, related to the test user account being suspended, causing subsequent runs of the test to fail. (#2497)
galley-types: remove all re-exports (#2504)
Enforce some IdP invariants (#2533)
Switch to new MLS test CLI (https://github.com/wireapp/mls-test-cli) (#2508)
Forward /i/users/:uid/features/:feature to brig (#2468)
charts/nginz: Forward /i/legalhold/whitelisted-teams to galley instead of brig (#2460)
make the ldap-scim-bridge chart deployable once per team, and improve docs. (#1843)
Refactored and simplified the feature config API (#2435)
Removed deprecated internal feature config API endpoints (#2496)
Deactivated gundeck's integration tests for local steps (make ci). (#2510)
retry gundeck's Redis connection in case of network errors such as IP changes or network outages (#2512)
Add AWS security token metrics to all services (#2473)
Upgrade team-settings version to 4.10.0-v0.29.7-0-3be8ca3 (#2180)
Upgrade webapp version to 2022-06-13-production.0-v0.29.7-0-2819b90 (#2302)
The .cannon.drainTimeout setting on the wire-server helm chart has been
removed and replaced with .cannon.config.drainOpts. (#2416)
Note for wire.com operators: deploy nginz (#2439)
The back-office (aka stern) team feature API now accenpts an optional TTL parameter (in days), so features can be activated for a limited period. (#2417)
Disable rate limiting for /api-version (#2439)
Drain websockets in a controlled fashion when cannon receives a SIGTERM or SIGINT. Instead of waiting for connections to close on their own, the websockets are now severed at a controlled pace. This allows for quicker rollouts of new versions. (#2416)
Optionally allow to run cannon with its own nginz inside the same pod; and connect to a load balancer directly. This allows the cannon-slow-drain behaviour implemented in #2416 to take effect by not having other intermediate network hops which could break websocket connections all at once. Some (internal) context: https://wearezeta.atlassian.net/wiki/spaces/PS/pages/585564424/How+to+gracefully+drain+cannon+but+not+so+slowly For details on how to configure this, see docs/src/how-to/install/configuration-options.rst (#2421)
Support running brig with GeoIP database when using helm charts (#2406)
charts/nginz: Add upstream configuration for galeb (#2444)
charts/nginz: Allow upstreams to be in other namespaces (#2444)
CSV export in team management now includes the number of devices per user (#2407)
charts/nginz: Resolve collision between brig and galeb endpoints. Ensure /self/consent and /signatures endpoints are configured in all environments (#2457)
When an IdP issuer (aka entity ID) is updated, the old issuer was still marked as "in use". (#2400)
On actions that require re-authentication a password is not required if the user has SAML credentials (#2430, #2434, #2437)
Use SCIM's preferred language as a fallback when privisioning users without a locale. (#2445)
AllFeatureConfigs is now typed (#2403)
Type class for default team feature status (#2404)
charts/{redis-ephemeral,legalhold}: Use old index for bitnami repo as the new index doesn't have old versions of postgresql and redis helm charts (#2448)
Bump haskell/zlib version to 0.6.3.0 (#2431)
New internal brig endpoints for MLS KeyPackage -> Conversation association query/update (#2375)
galley: refactor withSettingsOverrides (#2381)
charts/{nginz,cannon}: Increase map_hash_bucket_size for nginx to 128 (#2443)
charts/{cannon,nginz}: values listed in
nginx_conf.randomport_allowlisted_origins must be full hostnames. Hostnames
listed here will be allowlisted with and without TLS. (#2438)
Remove binding of users to saml idps using saml (this has never been picked up by clients; use scim instead) (#2441)
Remove golden test case generator
(#2442)
Convert Team CSV endpoint to Servant (#2419)
If using cert-manager, you need to have least version 1.0.0 (1.8.0 works at the time of writing) installed. Older cert-manager 0.15.X will no longer work. (#2401)
Upgrade team-settings version to 4.9.0-v0.29.7-0-142a76f (#2180)
Start version 2 of the public API. Main changes:
v3 and v4 suffixes. So for example
/assets/v3 has been replaced by /assets.GET /conversations/:conv/assets/:id and GET /conversations/:conv/otr/assets/:id have been removed.GET /assets/:key/v3 has been removed. Use the qualified endpoint GET /assets/:domain/:key instead.DELETE /assets/:key/v3 has been removed. Use the qualified endpoint
DELETE /assets/:domain/:key instead.GET /connections has been removed. Use POST /list-connections instead.POST /connections has been removed. Use POST /connections/:domain/:user instead.PUT /connections/:domain/:user has been removed: use POST instead.GET /conversations has been removed. Use POST /conversations/list-ids
followed by POST /conversations/list instead.POST /conversations/list/v2 has been replaced by POST /conversations/list.POST /conversations/:domain/:conv/members/v2 has lost its v2 suffix, so
it is now POST /conversations/:domain/:conv/members.GET /users, GET /users/by-handle and GET /users/handles have been
removed. Use POST /search/contacts instead.GET /users/:id has been removed. Use the qualified endpoint GET /users/:domain/:id instead.GET /users/:id/clients has been removed. Use the qualified endpoint GET /users/:domain/:id/clients instead.GET /users/:id/clients/:client has been removed. Use the qualified
endpoint GET /users/:domain/:id/clients/:client instead.Swagger documentation for the previous version of the API can be accessed at
/v1/api/swagger-ui. (#2297)
A new field development has been added to the object returned by GET /api-version. Versions listed there are considered in flux, meaning that the
corresponding API contracts can change arbitrarily over time. Clients are free
to use development versions, as long as they are also listed in supported,
and failures due to incompatibilities are acceptable (e.g. in testing
environments). Backends are the authoritative source on whether a development
version can be used at all. If a development version should not be used, the
backend will not list it among the supported versions at all. (#2297)
charts: Various new values can now be configured and some got changed
Allow new configurations in the brig chart:
config.emailSMS.user.invitationUrl
config.emailSMS.team.tInvitationUrl
config.emailSMS.team.tActivationUrl
config.emailSMS.team.tCreatorWelcomeUrl
config.emailSMS.team.tMemberWelcomeUrl
config.setProviderSearchFilter
config.setWhitelist
config.setFeatureFlags
config.setCustomerExtensions
If any values in config.emailSMS.team are specified, all must be specified.
Allow new configurations in the gundeck chart:
config.perNativePushConcurrency
config.maxConcurrentNativePushes.soft
config.maxConcurrentNativePushes.hard
Other changes:
maxTeamSize changed to 10000 from 500. (#2347)charts/nginx-ingress-services: Allow more fine-grained control over what services are installed. Upgrade Certificate/Issuer resources to 'cert-manager.io/v1' (#2401)
MLS implementation progress:
charts/{brig,cargohold,galley,gundeck}: Allow not configuring AWS credentials and allow using a special service account. This way, when operating wire in AWS cloud either instance profiles or IAM role attached to a service account can be used to communicate with AWS. (#2347)
Implement TURN service discovery using SRV records (#2389)
When config.enablePayment and FEATURE_ENABLE_PAYMENT (envVars) were set,
the team-settings feature flag FEATURE_ENABLE_PAYMENT was rendered two times.
The new behavior is to give the envVars entry priority. I.e. when it's set,
it's used instead of the config.enablePayment value. (#2332)
Modify the nginz access control configuration to prevent clients connecting to listeners with PROXY protocol enabled (such as the websocket listener) from accessing a private metrics endpoint. (#2307)
Verification email is sent when external id is updated via SCIM (#2374)
Move old /docs to /docs/legacy (leaving references). (#2328)
Fixup for #2321 (#2323)
Add pagination docs to POST /list-connections (#2369)
Documentation for the 2nd factor password challenge feature (#2329)
Documentation on how to enforce desktop application only for web app (#2334)
Documentation on how to enforce constant bit rate for all calls (#2336)
Documentation on how to disable media plugins for the web app (#2337)
Documentation on how to extra entropy in the web app (#2338)
Documentation on how to set the instance connection parameters and proxy settings (#2340)
Merged SAML/SCIM docs with its main documentation (#2356)
View and change team feature permissions apply to all features now (#2402)
Add sed to direnv (#2319)
Add python3 to nix development environment. It's needed by hack/bin/serve-charts.sh . (#2333)
Add a target to the Makefile to run ShellCheck. I.e. to run a linter on shell scripts. This will be used in the CI. For now, all scripts with linter issues are excluded from this check. (#2361)
Drop snappy support from bonanza (#2350)
Use cabal in buildah-based builds (#2341)
Fix flakyness of path traversal test (#2387)
Github Actions: disable mac builds (#2355)
Apply versionMiddleware last. This makes sure that every other middleware sees
the rewritten (unversioned) path. In particular, the prometheus middleware will
now only see paths it knows about, which prevents it from reporting "N/A" as the
path. (#2316)
Upgrade version of libzauth dependencies, notably sodiumoxide bindings to libsodium, and fix resulting errors and warnings. (#2327)
libzauth: Update sha256 for source in nix expression (#2354)
Log IO exceptions in Galley and Brig (#2385)
Generalise and move the Logger effect (#2306)
Fix a comment in a Makefile target (#2330)
Fix flaky MLS conversation creation test (#2386)
Fix flaky key package test (#2384)
Fix locale variables in Nix and .envrc (#2393)
Team Member API has been migrated to Servant (#2309)
Integration test for edge case: change external id before account registration (#2396)
Allow specifying 'redisAdditionalWrite' for a secondary redis to which gundeck will write in the context of a redis migration without downtime. (#2304)
Start TURN discovery only when the app starts and not when the Env is created (#2376)
Avoid using IN queries for fetching multiple conversations (#2397)
Remove oromolu GH action (has been moved to concourse https://github.com/zinfra/cailleach/pull/1033) (#2320)
Remove unused data type AllowedUserSearch (#2373)
docs: add latex to docs and publish pdf if exists (#2321)
We now fetch version information from other backends and negotiate a version to use. (#2297)
Fix assertion in testWelcomeNoKey (#2372)
Support remote welcome messages (#2368)
Implement remote admin action: Update receipt mode (#2141)
Note for wire.com operators: deploy nginz (#2270)
Wire cloud operators: Update brig's ES index mapping before deploying. After deploying, run a re-index (#2213, #2220)
Upgrade webapp version to 2022-04-21-production.0 (#2302)
Upgrade team-settings version to 4.7.0-v0.29.7-0-74b81b8 (#2180)
[helm-charts] Allow filtering cassandra nodes by datacenter (#2273)
MLS implementation progress:
POST /mls/key-packages/claim endpoint gained a skip_own query parameter, which can be used to avoid claiming a key package for the requesting client itself (#2287)The user profiles that are returned by a team admin search now contain the additional fields SAML NameID, IdP Issuer, and SCIM externalId (#2213), and unvalidated email address (#2220)
Add missing helm chart mapping for inbound search visibility (#2265)
Fix bug: User search endpoint hides exact handle results in SearchVisibilityNoNameOutsideTeam setting (#2280)
backoffice app (aka stern):
Migrate API routes from wai-route to servant for better Swagger (#2284, #2277, #2266, #2286, #2294, #2244)
Update nginx to latest stable: v1.20.2 (#2289)
Allow additional origins at random ports in nginz Helm chart. This is useful for testing with an HTTP proxy. It should not be used in production. (#2283)
makdeb and bonanza: remove stack-based Makefiles (#2311)
Add skip_reauth param to internal API for creating clients. This is intended to be used in test. (#2260)
Removes an unused function in Brig and relocates another one (#2305)
Print more logs while migrating data in Elasticsearch (#2279)
Replace the base monad in Brig with the Polysemy Sem monad (#2264, #2288)
Move the Random effect from Spar to the polysemy-wire-zoo library (#2303)
Move the Now effect from Spar to a library (#2292)
Improve readability of user search test cases (#2276)
Chart/gundeck's 'bulkpush' optimization is now activated by default (after using it in production for some time) (#2293)
Add an alpha version of a Helm chart for coturn. (#2209)
Document error handling and simplify error logging (#2274)
Improve speed of reindexing by increasing the batch size of processing users. (#2200)
Fix federator integration tests (#2298)
Switch the Haskell driver used in Gundeck to connect to Redis from 'redis-io' to 'hedis', which now supports cluster mode. (#2151)
Various Galley MLS test improvements and cleanups (#2278)
Flag for sending a validation email when updating a user's email address via backoffice/stern (#2301)
Remove stack from all builder docker images (#2312)
Make internal search-visibility endpoint available to staging environments (#2282)
Remove TemplateHaskell as a global default extension (#2291)
Note for wire.com operators: deploy nginz (#2175)
Deploy galley before brig (#2248)
Wire cloud operators: Update brig's ES index mapping before deploying. After deploying run a reindex. (#2241)
Upgrade webapp version to 2022-03-30-production.0-v0.29.2-0-d144552 (#2246)
Cross-team user search (#2208)
restund chart: add dtls support (#2227)
MLS implementation progress:
The bot API will be blocked if the 2nd factor authentication team feature is enabled. Please refer to /docs/reference/config-options.md#2nd-factor-password-challenge. (#2207)
Translations for 2nd factor authentication email templates (#2235)
Script for creating a team with owner via the public API (#2218)
Conversation rename endpoints now return 204 instead of 404 when the conversation name is unchanged (#2239)
Revert temporary sftd bump (#2230)
Remove the MonadMask instance for AppT in Brig (#2259)
Remove the MonadUnliftIO instance for the app monad in Brig (#2233)
Bump hsaml2 version (#2221)
Fix: cabal-install-artefacts.sh fails if not run from root of wire-server (#2236)
Fix: pushing to cachix not working (#2257)
Cannon has been fully migrated to Servant (#2243)
Refactor conversation record and conversation creation functions. This removes a lot of duplication and makes the types of protocol-specific data in a conversation tighter. (#2234)
NewConv
NewConversation record (used as input to the data
function creating a conversation) contain a ConversationMetadata.createConversation
Fix Docs: real-world domain used in examples (#2238)
The CanThrow combinator can now be used to set the corresponding error effects in polysemy handlers. (#2239)
Most error effects in Galley are now defined at the granularity of single error values. For example, a handler throwing ConvNotFound will now directly declare ConvNotFound (as a promoted constructor) among its error effects, instead of the generic ConversationError that was used before. Correspondingly, all such fine-grained Galley errors have been moved to wire-api as constructors of a single enumerated type GalleyError, and similarly for Brig, Cannon and Cargohold. (#2239)
Add a column for MLS clients to the Galley member table (#2245)
Pin direnv version in nix-hls.sh script (#2232)
nginx-ingress-services chart: allow for custom challenge solvers (#2222, #2229)
Remove unused debian Makefile targets (#2237)
Use local serial consistency for Cassandra lightweight transactions (#2251)
The client JSON object now has an additional field mls_public_keys, containing an object mapping signature schemes to public keys, e.g.
{
...
"mls_public_keys": { "ed25519": "GY+t1EQu0Zsm0r/zrm6zz9UpjPcAPyT5i8L1iaY3ypM=" }
...
}
At the moment, ed25519 is the only supported signature scheme, corresponding to MLS ciphersuite 1.
When creating a new client with POST /clients, the field mls_public_keys can be set, and the corresponding public keys are bound to the device identity on the backend, and will be used to verify uploaded key packages with a matching signature scheme.
When updating a client with PUT /clients/:client, the field mls_public_keys can also be set, with a similar effect. If a given signature scheme already has a public key set for that device, the request will fail. (#2147)
Introduce an endpoint for creating an MLS conversation (#2150)
The /billing and /teams/.*/billing endpoints are now available on a versioned path (e.g. /v1/billing)
(#2167)
MLS implementation progress:
2nd factor authentication via 6 digit code, sent by email:
SndFactorPasswordChallenge team feature is locked by default. (#2205)Fix data consistency issue in import of users from TM invitation to SCIM-managed (#2201)
Use the same context string as openmls for key package ref calculation (#2216)
Ensure that only conversation admins can create invite links. (Until now we have relied on clients to enforce this.) (#2211)
account-pages Helm chart: Add a "digest" image option (#2194)
Add more test mappings (#2185)
Internal endpoint for re-authentication (GET "/i/users/:uid/reauthenticate") in brig has changed in a backwards compatible way. Spar depends on this change for creating a SCIM token with 2nd password challenge. (#2149)
Asset keys are now internally validated. (#2162)
Spar debugging; better internal combinators (#2214)
Remove the MonadClient instance of the Brig monad
AppIO r monad. (#2187)