This is the first of three posts on improving type coverage in Sydent. Join us next Friday for the second part!

Sydent is the reference Matrix Identity server. It provides a lookup service, so that you can find a Matrix user via their email address or phone number (if they've chosen to share it).

We recently worked on improving Sydent's type coverage: the proportion of its source code with explicit annotations denoting the kind of data that each variable, expression and return value is expected to hold. These annotations are optional, but if present, they allow tools like mypy to analyze your programs and spot entire classes of bugs before you ship them! In this instance, we aimed to make Sydent pass mypy --strict, which it now does. Here's what the process looked like:

Two lines show two different measures of how well-typed the project is. The grey region covers our two-week sprint towards improving coverage; the earliest data point is from just before previous efforts to improve typing earlier in the year.

In a series of posts, I'd like to reflect on this sprint and share what we've learned. In particular, I aim to:

• explain why we wanted to improve type coverage now;
• work through examples to see how (if?) mypy could have spotted bugs;
• describe the annotation process;
• illustrate common patterns I learned along the way;
• discuss the checks that mypy provides; and finally
• reflect on the state of Python's typing ecosystem.

In this first part, we'll concentrate on the first two topics; the second will cover the middle two; and the third closes with the last two.

Why do this now?

It took us a long time (too long) to notice that the Sydent instance serving matrix.org was failing to send SMS messages for verification. We suspected that something was going wrong with our API call to OpenMarket. Our first step was to improve logging, so we could start to deduce what was going wrong and why. Whilst trawling through logs, we spotted one problem which meant we weren't actually sending off the API request in the first place. Further investigation revealed a strings-versus-bytes confusion which meant that we would always (incorrectly) interpret the API response as having failed.

All in all, phone number verification was unknowingly broken in the 2.4.0 release, to be fixed in 2.4.6 a month later. How could we do better? Better test coverage is (as ever) one answer. But it struck me that the two bugs we'd encountered might be ripe for automatic detection:

• we created an Awaitable but didn't await it or use it in any way, and
• we tried to look up a str key in a dictionary which mapped bytes to bytes.

Could a static analysis tool like mypy detect these? How much work would it take to do so? Are there other bugs and problems we could spot with it? I was curious to answer these questions and learn more about the tools that Python's typing ecosystem provides.

Could typing have spotted these problems?

Let's start with the first of question: what can mypy detect?

The missing await

Instead of writing x = await foo(), we simply had x = foo() and didn't then go on to await x. Mypy doesn't have means to detect this at present. There was interest in this issue on such a feature, with related threads here and here.

Are there other opportunities to spot the error? Here's the relevant bit of source code from before the fix.

            sid = self.sydent.validators.msisdn.requestToken(
                phone_number_object, clientSecret, sendAttempt, brand
            )
            resp = {
                "success": True,
                "sid": str(sid),
                "msisdn": msisdn,
                "intl_fmt": intl_fmt,
            }

The call to requestToken produces a value of type Awaitable[int]. If we tried to assign that to an expression of type int we'd get an error that mypy can spot.

$ cat example.py
async def foo() -> int:
    return 1

async def bar():
    x = foo()      # no error
    y: int = foo() # error: rhs is Awaitable[int], but lhs expects int

$ mypy --check-untyped-defs example.py
example.py:6: error: Incompatible types in assignment (expression has type "Coroutine[Any, Any, int]", variable has type "int")
Found 1 error in 1 file (checked 1 source file)

Note that we have to specifically ask mypy to typecheck the body of bar by passing --check-untyped-defs; by default, mypy will only typecheck annotated code.

We might also have been able to detect the error by looking at how we used sid. Unfortunately, the only use of was in a conversion str(sid), which is a perfectly type-safe call for both sid: int and sid: Awaitable[int]. But let's put that aside for a second—suppose we added "sid": sid directly into the resp dictionary. Could mypy have spotted there was a problem with that?

Unfortunately not. Because resp has no annotation, we have to rely on how it's used to spot any type inconsistencies. There's only one use of resp: as the return value from its enclosing function, render_POST. Mypy's only chance to spot a type problem would be to compare the mypy's inferred type for resp to the return type of render_POST. What are those types? We can use reveal_type to see the former is Dict[str, object]. For the latter:

    @jsonwrap
    def render_POST(self, request: Request) -> JsonDict:

The return type is JsonDict, which is an alias for Dict[str, Any]. This is bad news, because Dict[str, object] and Dict[str, Any] are compatible. Digging a level deeper, this is because sid: Any holds true for both sid: int and sid: Awaitable[int]—so there's no error to spot here. The Any type is compatible with any other type whatsoever! Mypy uses Any as a way to defer all type checking to runtime; mypy won't (and can't!) statically analyse the usage of an expression of type Any. Indeed, mypy's reports will tell you how many Anys you're working with, and offer a variety of options to warn or error on usages of Any.

If we were inserting sid directly into a dictionary, we could do better by annotating the dictionary (or the function's return type) as a TypedDict. This is a way to specify a dictionary with a fixed set of keys, each with a fixed type. It comes in really handy for Sydent, Sygnal and Synapse—all of the Matrix APIs exchange JSON dictionaries, so anything we can do to teach mypy about their shape and types is gold dust.

In short, there were options for detecting this with some code changes, but no magic wand that would have spotted the error in the code as written.

The strings/bytes confusion

Our error was here:

        headers = dict(resp.headers.getAllRawHeaders())
        request_id = None
        if "X-Request-Id" in headers:
            request_id = headers["X-Request-Id"][0]

In this sample, resp.headers is a twisted.web.http_headers.Headers. getAllRawHeaders is documented as returning an iterable of (bytes, Sequence[bytes]) pairs. Even better, mypy can see this because getAllRawHeaders is annotated (many thanks to the twisted authors for this). Mypy should be able to deduce that we build a dictionary headers: Dict[bytes, Sequence[bytes]. We can check this using reveal_type:

        headers = dict(resp.headers.getAllRawHeaders())
        reveal_type(headers)

$ mypy
sydent/sms/openmarket.py:110: note: Revealed type is "builtins.dict[builtins.bytes*, typing.Sequence*[builtins.bytes]]"

(The * in builtins.bytes* here means mypy has inferred that the dictionary's keys are bytes, rather than being told explicitly that they must be bytes.)

That's all fine and dandy. But why didn't we spot this before if the annotations were all in place in twisted? Let's put aside the fact that, erm, we weren't running mypy in Sydent's CI until the recent sprint, unlike our other projects. Checking out the problematic version, we can run mypy on the file we know to contain the bug.

$ git checkout v2.4.0
$ mypy --strict sydent/sms/openmarket.py
sydent/sms/openmarket.py:82: error: Dict entry 0 has incompatible type "str": "int"; expected "str": "str"  [dict-item]

Huh. Mypy spots something, but not the error we were hoping for. What's going on here? We can ask mypy to show its working with reveal_type again.

        resp = await self.http_cli.post_json_get_nothing(
            API_BASE_URL, send_body, {"headers": req_headers}
        )
        reveal_type(resp)
        headers = dict(resp.headers.getAllRawHeaders())
        reveal_type(resp.headers)
        reveal_type(resp.headers.getAllwRawHeaders())
        reveal_type(headers)

This yields:

$ mypy sydent/sms/openmarket.py
sydent/sms/openmarket.py:82: error: Dict entry 0 has incompatible type "str": "int"; expected "str": "str"  [dict-item]
sydent/sms/openmarket.py:102: note: Revealed type is "twisted.web.iweb.IResponse*"
sydent/sms/openmarket.py:104: note: Revealed type is "Any"
sydent/sms/openmarket.py:105: note: Revealed type is "Any"
sydent/sms/openmarket.py:106: note: Revealed type is "builtins.dict[Any, Any]"
Found 1 error in 1 file (checked 1 source file)

Ahh, the Any type. As mentioned above, this represents a value whose type can't be statically determined. We're left to runtime checks to detect the problem. But we won't detect it at runtime, because dictionaries don't enforce any kind of type requirements on their keys and values.

The problem here is that mypy can't see that resp.headers is a twisted Headers object. If we could inform it of this, mypy would spot our bug:

        import twisted.web.http_headers
        raw_headers: twisted.web.http_headers.Headers = resp.headers
        reveal_type(resp)
        headers = dict(raw_headers.getAllRawHeaders())
        reveal_type(raw_headers)
        reveal_type(raw_headers.getAllRawHeaders())
        reveal_type(headers)

$ mypy sydent/sms/openmarket.py
sydent/sms/openmarket.py:82: error: Dict entry 0 has incompatible type "str": "int"; expected "str": "str"  [dict-item]
sydent/sms/openmarket.py:104: note: Revealed type is "twisted.web.iweb.IResponse*"
sydent/sms/openmarket.py:106: note: Revealed type is "twisted.web.http_headers.Headers"
sydent/sms/openmarket.py:107: note: Revealed type is "typing.Iterator[Tuple[builtins.bytes, typing.Sequence[builtins.bytes]]]"
sydent/sms/openmarket.py:108: note: Revealed type is "builtins.dict[builtins.bytes*, typing.Sequence*[builtins.bytes]]"
sydent/sms/openmarket.py:114: error: Invalid index type "str" for "Dict[bytes, Sequence[bytes]]"; expected type "bytes"  [index]
sydent/sms/openmarket.py:114: error: Argument 1 to "split" of "bytes" has incompatible type "str"; expected "Optional[bytes]"  [arg-type]
Found 3 errors in 1 file (checked 1 source file)

There it is, on line 114: Invalid index type "str" for "Dict[bytes, Sequence[bytes]]"; expected type "bytes".

Unfortunately it'd be a pain to annotate our application code to mark every use of IResponse.headers as a Headers object. We'll see a better way to do things in this the next post, which discusses the nitty-gritty details of adding annotations file-by-file.

Many thanks for reading! If you've got any corrections, comments or queries, I'm available on Matrix at @dmrobertson:matrix.org.

A full day of Matrix talks

This year, the Matrix.org Foundation is excited to host the first ever Matrix.org Foundation and Community devroom at FOSDEM. A full day of talks, demos and workshops around Matrix itself and projects built on top of Matrix.

Matrix is the open source project that publishes the Matrix open standard for secure, decentralised, real-time communication, and its Apache licensed reference implementations.

We encourage people working on the Matrix protocol or building on it in an open source project to submit a proposal! Note that companies are welcome to talk about the Matrix details of their open source projects, but marketing talks are not welcome.

We want this devroom to be a space where the Matrix community can show its work, where developers can talk about the challenges they faced and how they overcame them, and where people can get a glimpse of the future of the Matrix protocol and ecosystem.

Talk Details

The talks will be pre-recorded in January. They will be played during FOSDEM, followed by a session of live Q&A depending on the format. During the playback of the talk, people will be able to comment and ask questions in the chat (via Matrix!).

The talks can follow one of three formats:

• 5 min lightning talk, ideal to showcase your project and make people want to have a look at it
• 20 min talk + 10 min Q&A, for topics that can be covered briefly
• 50 min talk + 10 min Q&A for more complex subjects which need more focus

We strongly encourage you to prepare a demo when it makes sense, so people can actually see what your work looks like in practice!

Of course, the proposal must respect the FOSDEM terms as well:

The conference language is English. All content must relate to Free and Open Source Software. By participating in the event you agree to the publication of your recordings, slides and other content provided under the same licence as all FOSDEM content (CC-BY).

Submitting a Proposal

Proposals must be submitted on FOSDEM's conference management system Pentabarf before December 17th 2021. If you are not used to Pentabarf, you can follow this beginners guide to Pentabarf.

We expect to receive more requests than we have slots available. The devroom organisers (two community members and one core team rep) will be reviewing the proposals and accepting them based on the potential positive impact the project has on Matrix (as defined in by the Mission section of https://matrix.org/foundation).

If a project proposal has been turned down, it doesn't mean we don't believe it has good potential. Maintainers are invited to join the #twim:matrix.org Matrix room to give it some visibility.

Synapse 1.48.0 is out now!

NOTE: Synapse 1.49, due out on December 14th, will be the last release of Synapse to support Python 3.6 or PostgreSQL 9.6 per our platform dependency deprecation policy. Accordingly, we will remove support for Ubuntu 18.04 LTS (Bionic) at the same date, as it ships with Python 3.6.

Password resets and identity servers

This release removes the long-deprecated trust_identity_server_for_password_resets configuration option. This option was initially deprecated in Synapse 1.4.0 back in October 2019.

Admins of servers still using this configuration option will need to update their Synapse configuration to send password resets through an SMTP server directly rather than relying on identity servers to send them on their behalf.

New admin APIs and improved alignment with Matrix 1.1

This release also introduces a handful of new admin APIs, allowing administrators to un-shadow-ban users, block a room, and run specific background updates (but we'll talk about this last one a bit later on). The delete room API has also been updated to be able to run in the background or to block a room pre-emptively, even if the server doesn't know about it yet.

This release also brings Synapse into greater alignment with version 1.1 of the Matrix specification by adding support for API paths beginning /_matrix/client/v3 and /_matrix/media/v3.

Background updates

When Synapse updates from one version to another, it might need to run large scale updates on its database. In order to avoid blocking startup for too long while waiting for these updates to run, Synapse runs them in the background after starting.

Lately the Synapse team has been doing some work to improve the performance of these background updates. More specifically, this release includes a performance fix for a background update introduced in Synapse 1.47.0, as well as a new admin API to let admins rerun specific updates.

Future Synapse updates will also include module capabilities and more configuration options for controlling background updates.

Everything else

This release also includes some improved support of MSC3440 to help threading. It also adds support for the stable identifiers from MSC2778, bringing Synapse closer to supporting end-to-end (or end-to-bridge) encryption support for application services.

We also now publish a Docker image, matrixdotorg/synapse:develop, which tracks the development head of Synapse.

Please see the Synapse Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including Dirk Klimpel, Stanislav Motylkov, Tulir Asokan and Neeeflix.

Matrix Live 🎙

Dept of Spec 📜

anoa says

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.

MSC Status

New MSCs:

• MSC3523: Timeboxed/ranged relations endpoint
• MSC3517: "Mention" Pushrule
• MSC3510: Let users kick/ban/demote each other

MSCs with proposed Final Comment Period:

• No MSCs entered proposed FCP state this week.

MSCs in Final Comment Period:

• No MSCs are in FCP.

Merged MSCs:

• MSC2674: Event Relationships

Closed MSCs:

• MSC3006: Bot Interactions

Spec Updates

MSC2675 (serverside aggregations) is getting lots of updates from Bruno in order to align the proposal with what is currently implemented in the wild (as it's easier to iterate on incremental improvements from a starting point grounded in reality). This MSC is a bit of a special case though, as it was implemented with stable prefixes before the MSC landed (in the before times...).

Regardless, thank you very much to Bruno for going through and finally untangling and help land aggregations in the spec! This MSC is one of four which describe how aggregations should work in Matrix, and it's great to see them finally being properly spec'd, especially as further features start to be built on top of them (such as threading!).

And finally, a further thank you to community members @ankur12-1610, @Dominaezzz for an OpenAPI schema fix!

Random Spec of the Week

The random spec of the week is... MSC3395: Synthetic appservice events!

This proposal aims to allow appservices to get ever greater visibility into what is happening on the homeserver, while still maintaining full process separation.

A lot of the time solutions to complex problems require knowing when a user has registered or logged in, with what name/3pids etc, when users change their emails, etc. Hooking into these non-room-based actions can help with developing useful features. Maybe you want an appservice that plays a sound effect whenever a user signs up to your homeserver!

These days, that is often achieved by homeserver implementation-specific solutions, such as modules in Synapse. Being able to notify of these events using a standard API shape would be hugely beneficial to generalisation of projects.

So give the proposal a look over and review if that interests you!

Some exciting MSCs this week, I've been waiting for server-side aggregations for long!

Dept of Servers 🏢

Synapse ↗

Synapse is the reference homeserver for Matrix

callahad announces

As predicted last week, we released Synapse 1.47.1 on Tuesday. This is a security release which fixes an issue with Synapse's built-in media repository. Admins are strongly encouraged to upgrade.

Otherwise, relatively slow week: a handful of of the team have been away (Happy Thanksgiving, Americans!), but we did release Synapse 1.48.0rc1. Most importantly, this release candidate includes changes to improve the efficiency of large background updates from past releases, which should significantly reduce database load when upgrading. It also adds support for the /v3 APIs defined in version 1.1 of the Matrix specification.

We'll talk more about 1.48 when it's formally released next week, but as always, we appreciate folks trying out the release candidates and letting us know how they behave.

Administrators, keep your users safe: update as soon as you can!

Homeserver Deployment 📥️

Helm Chart ↗

Matrix Kubernetes applications packaged into helm charts

Ananace announces

A bit of an earlier update this week, but I wanted to make sure to note that my Hem Charts have been updated to matrix-synapse 1.47.1 for the security fix - and element-web has also been bumped to 1.9.5

Dept of Bridges 🌉

mautrix-googlechat

tulir reports

mautrix-googlechat has seen lots of improvements over the past few days. New features include:

• Bridging edits, deletions, reactions, formatting and read receipts in both directions (even /rainbow somewhat works from Matrix)
• Bridging typing notifications and any types of files from Matrix to Google Chat
• Bridging Google Meet links from Google Chat to Matrix
• Syncing group members from Google Chat

(edits and deletions are only available on Google Workspace accounts, not normal accounts. I have no idea why they did that, but that's just how Google Chat works 🤷)

There's still a bug where it sometimes silently stops receiving messages, which I'm currently trying to solve (or work around). After that I'll make a v0.3.0 release. Backfilling history may also happen in the near future

Matrix Webhook Receiver ↗

An add-on for the matrix-appservice-webhooks bridge. Webhooks are essentially web interfaces for applications to "push" data to. The bridge can receive messages in a certain format, which is nice if the notifying app can be configured. Often it cannot.

kim announces

Matrix Webhook Receiver

Do you like to receive notifications in matrix? Matrix Webhook Receiver (MWR) is an add-on for the matrix-appservice-webhooks bridge. Webhooks are essentially web interfaces for applications to "push" data to. The bridge can receive messages in a certain format, which is nice if the notifying app can be configured. Often it cannot.

This is where MWR comes in: It can receive any (JSON) content, optionally reformat it nicely (customizable!), and forward it to the webhooks bridge which will post it to a room for you. If you are running any software service, there is a good chance it can notify you via webhooks!

Right now, several example configurations exist, ready for you to use:

• GitHub
• GitLab (including Community Edition/self hosted)
• Ansible Tower/AWX
• Grafana Alerts

For example, here is the GitHub webhook as seen in #matrix-webhook-receiver:matrix.org:

More examples of apps currently in development: Prometheus Alertmanager, Jellyfin

It is also easy to use to send messages from the commandline with standard tools (curl), e.g. for your cron jobs! See the README for an example.

Some other webhook matrix things exist, but often require

• ➖ admin access to server (appservices)
• ➖ installing and running and maintaining a specific bot per service
• ➖ create and set up accounts and passwords for each bot to use

In contrast, MWR requires:

• ➖ matrix-appservice-webhooks set up on your server by your admin
• ➖ some knowledge of how to send POST requests for one time setup. I want to improve that, suggestions are welcome!
• ➕ MWR can be installed and run by anyone who can access the bridge
• ➕ one single MWR supports any amount of notifying apps!
• ➕ you don't need admin access to the server to add more apps
• ➕ no need to develop a bot/plugin to support new apps, just write a quick jinja template
• ➕ multiple people can use a single MWR instance by sharing HTTP basic auth credentials

Other features:

• can post into encrypted rooms (see readme)
• automatically generated API docs
• message formatting presets including html markup or m.notice
• arbitrarily customizable webhook URLs
• list and manage currently installed webhooks in your browser (screenshot)

Links:

• Repo
• #matrix-webhook-receiver:matrix.org

What a massive update! At this rate I won't need to leave my Matrix client for anything!

matrix-hookshot ↗

A multi purpose multi platform bridge, formerly known as matrix-github

Half-Shot says

Hey folks! Some exciting new news on the bridge front: I've renamed matrix-github to matrix-hookshot to better reflect it's not-just-GitHub-ness. That's not all though, as there are new features too:

• The bridge now supports Rust as a companion language (we're aiming to rewrite critical sections in rust). Some parts of the formatting code have already been rewritten.
• The bridge now supports JIRA (full puppeting!)
• The bridge now supports generic webhooks too, with the ability to write custom handling code inside the state event to process these hooks into pretty messages.
• Basic support for GitHub discussions.

In the works:

• A provisioning API to hook into integration managers
• More GitLab support
• Better GitHub discussions support

We're not quite ready for a 0.2.0 release, but please check us out at https://github.com/Half-Shot/matrix-hookshot.

An interesting update, and Half-Shot even demoes it in today's Matrix Live!

Dept of Clients 📱

Nheko ↗

Desktop client for Matrix using Qt and C++17.

Nico says

After the 0.9.0 release last week, we have of course been busy fixing all the bugs different people reported. Messing around with the sticker pack editor and then leaving room should not make Nheko crash anymore. The problems where the flatpak has issues starting on Gnome systems are still under investigation. We thought we had a solution, but that seems to have broken other stuff! 💥

Apart from that we have been doing some after release party cleanup. Apart from some refactorings, you can now filter your rooms on whether they are a direct chat or not in the sidebar. This is in addition to the filters we already had for favourites, spaces and your other personal tags. User colors should also now be much less biased towards blue and jdenticons should have more variance. Expect the next release to be a much more colorful experience!

Speaking of colors, Twily made this awesome ZX Spectrum inspired logo after we changed our Gitlab bot to be more colorful! Check it out:

Gorgeous, I love it!

Hydrogen ↗

Hydrogen is a lightweight matrix client with legacy and mobile browser support

Bruno says

We're still distracted with SDK work and other things less visible for users, but this week we've also released 0.2.22 that fixes login on Element One (and other servers using SSO login and not yet supporting the experimental dehydrated devices).

Element ↗

Everything related to Element but not strictly bound to a client

kittykat says

Threads

• On Web we’ve been working on Notifications and Badges, making sure no message goes unread.
• On Mobile we’re building out the new Threads Panel so you can easily see all the Threads in a room.

Polls

• The Polls team is making great progress, focusing their efforts on creating and voting on all platforms.
• User testing sessions are coming up!

Community Testing

• Bug squash session on Web on Tuesday 30 November at 17:00 UTC, join the call at #element-community-testing:matrix.org
• Upcoming plans include testing sessions on Polls and IA.

Element Web/Desktop ↗

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

kittykat announces

• Work continues on Information Architecture: head over to Sidebar settings on develop.element.io to see what the team are currently working on.

Element iOS ↗

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

kittykat announces

• We’re still working hard on replacing Matomo with PostHog to improve how we collect analytics data and making the MatrixKit obsolete.
• Adding more features to Spaces and improvements to Spaces performance is also happening.

Element Android ↗

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

kittykat announces

• Voice message drafts and other improvements to this feature are underway.
• We’re also looking to integrate PostHog on Android.

Dept of SDKs and Frameworks 🧰

simplematrixbotlib ↗

simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio.

krazykirby99999 says

simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio. Version 2.4.0 provides several new features and a fix.

New Features:

• Newlines are now supported when sending markdown messages.
• The msgtype of text and markdown messages can now be specified. Text and markdown messages can now optionally be sent as "m.notice" to avoid alerting everybody of the new message. The default msgtype will continue to be "m.text".

New Fixes:

• Fixed issue where the homeserver was hardcoded in an http request.

Example usage is shown below:

import simplematrixbotlib as botlib

creds = botlib.Creds("https://home.server", "user", "pass")
bot = botlib.Bot(creds)
PREFIX = '!'


@bot.listener.on_message_event
async def echo(room, message):
    match = botlib.MessageMatch(room, message, bot, PREFIX)

    if match.is_not_from_this_bot() and match.prefix() and match.command(
            "echo"):
            
        response = " ".join(arg for arg in match.args())
        await bot.api.send_text_message(room.room_id, response, "m.notice") ## Uses the msgtype of m.notice instead of m.text

bot.run()

A thank you to HarHarLinks for their contributions to version 2.4.0!

Request additional features here.

View source on Github View package on PyPi View docs on readthedocs.io https://matrix.to/#/#simplematrixbotlib:matrix.org

jOlm ↗

Olm bindings for Java

brevilo reports

This week saw three releases of jOlm which fix a native memory management issue, an Olm API (buffer) issue and add a few other improvements. Everyone is strongly encouraged to update to the latest release.

Notes:

• ✅ Bugfix and maintenance releases
• ✅ Up to date with Olm 3.2.6

Changelog:

• Fixed the backing store retention for all Olm instances
• Fixed a buffer issue in InboundGroupSession.decrypt()
• Ensured conversions of variable native strings are trimmed
• Centralized conversions to canonical JSON
• Completed initial set of unit tests (effectively full coverage now)

Cheers!

The only Java bindings of Olm to my knowledge, that's some very valuable work here!

Dept of Ops 🛠

synadm ↗

Command line admin tool for Synapse (Matrix reference homeserver)

jojo reports

synadm v0.32 is out!

My personal favorites of the new features are:

• Dates and times in several subcommands are translate the admin API's UNIX epoch timestamps to a human readable format.
• synadm room list now displays room aliases (#room:your.homeserver)

Read the full release notes here: https://github.com/JOJ0/synadm/releases/tag/v0.32

synadm is very useful as a homeserver administrator. Thanks JOJ0

Dept of Bots 🤖

maubot ↗

A plugin-based Matrix bot system.

tulir reports

maubot v0.2.0 was released last weekend. Highlights:

• Enabling encryption should be much easier: the device ID can be entered in the web UI or you can just do mbc auth --update-client to automatically log in and store the access token and device ID in maubot.
• mbc auth can now log in with SSO.
• The standalone mode for running a single plugin with a static config is now mostly functional and somewhat documented.

Also, I finally took a day to figure out Sphinx/autodoc and made some decent-looking autogenerated docs for mautrix-python. I'll probably extend that to generate maubot-specific API references too eventually.

Dept of Interesting Projects 🛰️

MinesTRIX ↗

A privacy focused social media based on MATRIX

Henri Carnot announces

Quick update on MinesTRIX (a privacy focused social media based on MATRIX). This week was focused on performance and stability.

• Changed database to use Fluffybox, this should greatly improve performances on web (thanks Famedly !)
• Scrolling through the posts of a profile now properly request history.
• Friend suggestions are now sorted according to the sum of user appearance in all rooms. Naïve, but it's the first step.
• Chat page has also been redesigned. Now support replies and reactions. Chat settings now display room avatar and fetch user list from server.
• Bug affecting MinesTRIX profile creation has been fixed. Login process should be way more stable now.
• Minestrix rooms sync has been rewritten to take into account sync events to rebuild the list.
• Debug page now allow forcing sorting rooms.
• Various post display enhancement (links are now clickable, thanks kellya!)

Come chat with us : #minestrix:carnot.cc

That's one exciting project, I can't wait to see how far it's going!

Sign in with Matrix

Mish says

Federated sign-in component for your web app (using Matrix)

This week's update:

• Gained 180 stars on GitHub since release (thanks!)
• Added login states, accessible from the API
• Sign out
• Added CSS styling via variables
• Updated demo

more on https://github.com/mishushakov/signin-with-matrix

As last week, a note to keep in mind that this is a community project and that there is a MSC to make Matrix more OAuth2 friendly. More on that very soon!

Dept of Built on Matrix 🏗️

Matrix Forms

Mish reports

First release of "Matrix Forms", a project which redirects form submissions to designated Matrix rooms

Features:

• Server-side, no additional JavaScript
• Rich formatting
• Many forms on same instance
• File uploads
• Templates
• CORS
• Metadata accessible for bots

Can be installed using NodeJS or Docker

• demo: https://mishushakov.github.io/matrix-forms
• code: https://github.com/mishushakov/matrix-forms

Final Thoughts 💭

Server_Stats

MTRNord says

Today is a good day for those calling me a spy, someone not wanting to care about privacy and for those who did publicly harass me for server_stats.

People using the API likely already noticed it wasn't reachable for a while. Effective immediately I am currently leaving all rooms the bot is part of. This will take days or even months considering this are 6397 rooms at the time of writing. I am not going into the motives of why I am shutting it down. It comes down to personal reasons.

There won't be any dump of the data. The source will be kept public. Note though if anyone ever tries to run it that you need about 600GB of space for synapse, a lot of CPU, a lot of RAM and plenty of workers as this can easily crash synapse.

Server_Stats was an incredibly useful project. It pains me a lot to see it go, but it pains me even further that its author got harrassed. This is not an acceptable behaviour, and we are better than that as a community. Thanks for this incredible project MTRNord, it's been both exciting and useful.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Today we are releasing Synapse 1.47.1, a security update based on last week's release of Synapse 1.47.0. This release patches one high severity issue affecting Synapse installations 1.47.0 and earlier using the media repository. An attacker could cause these Synapses to download a remote file and store it in a directory outside the media repository.

Note that:

• This only affects homeservers using Synapse's built-in media repository, as opposed to synapse-s3-storage-provider or matrix-media-repo.
• Attackers cannot control the exact name or destination of the stored file.

To quote from the advisory:

GHSA-3hfw-x7gx-437c / CVE-2021-41281: Path traversal when downloading remote media.

Impact

Synapse instances with the media repository enabled can be tricked into downloading a file from a remote server into an arbitrary directory, potentially outside the media store directory.

The last two directories and file name of the path are chosen randomly by Synapse and cannot be controlled by an attacker, which limits the impact.

Homeservers with the media repository disabled are unaffected. Homeservers configured with a federation whitelist are also unaffected.

The advisory has full details, including workarounds.

This issue was discovered and fixed by our internal security team.

Please update at your earliest convenience.

Friday already? Did this week already happen? It looked like the spacetime continuum was broken and we didn't know who did it. We needed witnesses to solve the case: did things really happen this week? And the witnesses showed up! A huge thanks to everyone in the Matrix community who reported their progress, and to everyone currently working on making awesome projects around Matrix!

The case is closed: the week has not been stolen from us. Time appears to have wings, and flies faster than we had anticipated.

Matrix Live 🎙

A very very dense and exciting wrap up of what's happening these days in the Matrix space by Matrix Foundation co-founders Matthew & Amandine! Matrix is stepping up a gear with blazing fast Sync v3, Threading Support, VoIP, VR, a new release of the Spec, always more monthly active users, a full security audit and progress on P2P.

Dept of Spec 📜

anoa says

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.

MSC Status

New MSCs:

• MSC3489: Sharing streams of location data
• MSC3488: Extending events with location data

MSCs with proposed Final Comment Period:

• No MSCs entered proposed FCP state this week.

MSCs in Final Comment Period:

• MSC2674: Event Relationships (merge)

Merged MSCs:

• MSC3283: Expose capabilities for profile actions

Spec Updates

This week we finally, finally had one of the aggregation-related MSCs, MSC2674 (event relationships) enter final comment period! This MSC, along with several others, document the stuff that powers message edit, reactions, the upcoming threading and polls MSCs, and much more! So it's really great to see the MSCs start to actually land.

Speaking of threading MSC3440 has had a good amount of review from the Spec Core Team last week. Threading in Matrix has been a long awaited feature for chat applications - as well as helping extend the flexibility of Matrix as a data structure even further. The MSC relies on both MSC2674 and MSC2675 (or a modified version of it), so the latter will be an area of focus for review for next week.

Spec PRs

It seems that the recent Matrix v1.1 has drummed up some more community support for spec PRs. Thank you to everyone who's sent in clarifications, changes and even typo fixes. In particular, @ankur12-1610 for fixes to the OpenAPI fields, and for typo corrections in the spec copy, and @Dominaezzz for helping review them. Thank you both!

Random Spec of the Week

The random spec of the week is... MSC2867: Marking rooms as unread!

This is definitely a feature that I would love to have for chat. Note that this MSC proposes marking a room as unread, rather than a specific point in the room's timeline. This is intentional as noted in the document, as the latter is more complicated, as it intersects with sending out read receipts to other users.

Dept of Servers 🏢

Synapse ↗

Synapse is the reference homeserver for Matrix

dmr announces

Note: we plan to release a security release, Synapse 1.47.1 on the coming Tuesday, 23rd of November; see the predisclosure.

We released Synapse 1.47.0 after squashing a couple of problems related to database migrations remaining in 1.47.0rc2. Briefly, this release includes

• new features for users of the admin and module APIs;
• a number of long-standing bug fixes, including a thorny bug which prevented joining certain old rooms; and
• continuing efforts to prototype new MSCs and improve type coverage.

The blog post has a better summary, and GitHub has the full gory details.

We also released Sydent 2.5.1, a minor release which improves the way we handle and log various error cases.

Thank you as ever to our community contributors, and everyone out there who's using Synapse to communicate!

dmr also reminds us

When we release the fix, the changes will be publicly known and bad actors will have the ability to deduce the vulnerability. Most servers don't run release candidates, so releasing the fix in an RC will mean there's a larger window for an attacker to exploit the problem. Internally, there will be an RC deployed to test homeservers and eventually matrix.org. This means we'll be able to confidently recommend the upgrade to server administrators.

dkasak adds

And that's the very reason we have dedicated security releases, instead of just rolling the security fixes into a feature release. The security release doesn't contain anything new apart from the security fix so it minimizes the chance of things going wrong.

Keep your servers up to date, and your users safe, administrators!

Homeserver Deployment 📥️

Helm Chart ↗

Matrix Kubernetes applications packaged into helm charts

Ananace says

Would you believe it? This week has also seen updates to my Helm Charts, with matrix-synapse having been updated to 1.47.0

Dept of Bridges 🌉

Heisenbridge ↗

Heisenbridge is a bouncer-style Matrix IRC bridge.

hifi says

Release v1.7.0 🥳

• Implement "best effort" basic IRC moderation in plumbed rooms if bot has ops on IRC
• Allow configuring topic sync for plumbs (IRC<->Matrix or one way)
• Allow using forward slash (/) as MXID separator for IRC ghosts
• Bump max mautrix version to <0.12
• Small fixes

Plumb moderation! If the bridge bot has ops on IRC it will do its best to map kicks and bans (regarding IRC users) from Matrix. This definitely isn't perfect and is meant as a convenience.

Topic synchronization is now configurable for plumbs as well to make it possible to share the same topic between an IRC channel and a plumbed Matrix room. Default is still off and it requires the bridge bot to have enough PL to work.

The separator for IRC ghosts can now be changed to forward slash (/) from the default underscore (_). This happens by modifying the regex in the registration file. Only do this for new installations and it will cause all IRC users to duplicate in rooms who you can't remove and probably other bad side effects as well. The default may be changed in the future.

There were lots of refactoring issues so I hope I fixed all of them 🙈.

Best effort your fix from GitHub, PyPI or matrix-docker-ansible-deploy!

Thanks!

Yet another week, yet another great update on Heisenbridge. At this rate Heisenbridge will reach perfection and hifi will run out of things to report!

Dept of Clients 📱

Nheko ↗

Desktop client for Matrix using Qt and C++17.

Nico reports

So, Nheko has a small little release this morning! Okay, that's a lie, it was actually pretty big! You can find the full changelog and some of our binaries here: https://github.com/Nheko-Reborn/nheko/releases/tag/v0.9.0

As always, thank you everone, who contributed. There were over 30 authors this release! If you haven't tried Nheko in a while, give it a whirl. Lots of stuff changed, some things might not even have been mentioned in TWIM! I put the first few lines of the changelog below for your convenience:

Highlights

• Somewhat stable end to end encryption 🔐
  • Show the room verification status
  • Configure Nheko to only send to verified users
  • Store the encryption keys securely in the OS-provided secrets service.
  • Support online keybackup as well as sharing historical session keys.
• Crosssigning bootstrapping 🔄
  • Crosssigning is used to simplify the verification process. In this release Nheko can setup crosssigning on a new account without having to use a different client.
  • Nheko now also prompts you, if there are any unverified devices and asks you to verify them.
• Room directory (Manu) 📂
  • Search for rooms on your server and other servers. (Prezu)
  • If their topic interests you and it has the right amount of members, join the room and the discussion!
• Custom sticker packs 🐈‍⬛
  • Add a custom sticker picker, that allows you to send stickers from MSC2545.
  • Support creating new sticker (and emote) packs.
  • You can share packs in a room and enable them globally or just for that room.
• Token authenticated registration (Callum) 🎫
  • Sign up with a token to servers, that have otherwise disabled registration.
  • This was done as part of GSoC and makes it easier to run private servers for your family and friends!

Features

• Support email in registration (required on matrix.org for example)
• Warn, if an @room would mention the whole room, because some people don't like that.
• Support device removal as well as renaming. (Thulinma)
• Show your devices without encryption support, when showing your profile. (Thulinma)
• Move to the next room with unread messages by pressing Alt-A. (Symphorien)
• Support jdenticons as a placeholder for rooms or users without avatars. (LorenDB)
  • You will need to install https://github.com/Nheko-Reborn/qt-jdenticon
• Properly sign macOS builds.
• Support animated images like GIF and WebP.
  • Optionally just play them on hover.
• Support accepting knocks in the timeline.
• Close a room when clicking it again. (LorenDB)
• Close image overlay with escape.
• Support .well-known discovery during registration.
• Limited spaces support.
  • No nice display of nested spaces.
  • No previews of unjoined rooms.
  • No way to edit a space.
• Render room avatar changes in the timeline. (BShipman)
• Support pulling out the sidebar to make it wider.
• Allow editing pending messages instead of blocking until they are sent. (balsoft)
• Support mnemonics in the context menus. (AppAraat)
• Support TOFU for encryption. (Trust on first use)
• Right click -> copy address location.
• Forward messages. (Jedi18)
• Alt-F to forward messages.
• A new video and audio player, that should look a bit nicer.

As always, come check us out and chat about Nheko in #nheko:nheko.im ♥

That's one massive update for Nheko! Thanks Nheko contributors!

Element ↗

Everything related to Element but not strictly bound to a client

Nad says

Threads

• We’ve been polishing UI, and updating notifications on Web
• On Mobile, we’ve updates iOS & Android to be threads aware, live in the next release
• We also tested Threads more widely in a community testing session (thanks to all who attended!)

Polls

• We’re making good progress on polls on all platforms
• Aiming to get poll creation, voting and the ability to see results working in all platforms in our current sprint, to then move to wider testing

Community Testing

• Two successful testing sessions on VoIP and Threads (first community testing session on iOS!)
• Squashed 26 Android VoIP bugs
• No sessions this coming week, but we’ll be back the week after

Element Web

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

Nad reports

• Work continuing on the Info architecture project. If you haven’t yet, try out the new Sidebar settings and new room header context menu on develop.element.io!

Element iOS

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Nad says

• We fixed regressions reported on our previous release candidates. Sorry for the delay but the current release candidate 1.6.8 should be available on the App Store on Monday
• The work to replace Matomo by PostHog has been resumed
• We are still working on making the MatrixKit obsolete
• Space creation / invites: will start design and code review starting next week
• Start space management integration in rooms

Element Android

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

Nad updates us with

• Element Android 1.3.8 with support for Android 12 has been released on the PlayStore (beta channel) Wednesday. Full changelog: https://github.com/vector-im/element-android/releases/tag/v1.3.8 . The SDK 1.3.8 has also been released. If everything is fine, the app will be promoted to production and to F-Droid on Monday. The README of the project has been updated to clarify the release process: https://github.com/vector-im/element-android#releases-to-app-stores
• Voice message draft is currently under active development. We want the feature to work well before we release it, and it was an opportunity to rework the whole feature, to improve its architecture.
• Still working on the timeline rework.

Dept of Non Clients 🎛️

time-to-matrix ↗

The time command, but it sends the output to a Matrix room

Aine reports

after the Miounne update posted a minute ago here is another one, and it's about time: Time To Matrix (ttm) got v1.4.0 release!

Time To Matrix is a time-like command that will send end of an arbitrary command output and some other info (like exit status) to matrix room.

With new release, following things were added:

• arch linux AUR package
• automatic room alias resolving, so you can use #ttm:etke.cc instead of !XODRhTLplrymaFicdK:etke.cc
• help message and human-readable errors
• option to change message type (m.text or m.notice)
• option to omit plaintext and send only html-formatted message (to get some more space for log)
• option to override message type to m.notice if the command exits with non-zero exit code (by default m.text is sent, so you will get m.notice on failure)

Go check out the source code and say hello in #ttm:etke.cc

Dept of SDKs and Frameworks 🧰

simplematrixbotlib ↗

simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio.

krazykirby99999 reports

Version 2.3.0 Released!

simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio. Version 2.3.0 adds support for additional configuration via config files and other methods. Currently, there is only one setting that can be changed, however many existing and future features will be able to be enabled or disabled via this config.

Example usage is shown below:

"""
random_user
     !echo something
echo_bot
     something
"""

import simplematrixbotlib as botlib

creds = botlib.Creds("https://home.server", "user", "pass")

config = botlib.Config()
config.load_toml("config.toml")

bot = botlib.Bot(creds, config)
PREFIX = '!'

@bot.listener.on_message_event
async def echo(room, message):
     match = botlib.MessageMatch(room, message, bot, PREFIX)

     if match.is_not_from_this_bot() and match.prefix() and match.command("echo"):
          await bot.api.send_text_message(room.room_id,
                                " ".join(arg for arg in match.args()))

bot.run()

An example of a toml config file is shown below.

[simplematrixbotlib.config]
join_on_invite = false

Request additional features here.

View source on Github View package on PyPi View docs on readthedocs.io #simplematrixbotlib:matrix.org

Dept of Ops 🛠

Mother Miounne ↗

The backoffice of etke.cc service

Aine reports

ding-dong Mother Miounne v2.2.0 is here!

Miounne is a backoffice of the etke.cc service.

New release brings notifications for integrated services:

• buymeacoffee notifications on new purchase, supporter, subscription and unsubscription (can be configured independently)
• matrix-registration notifications for new and used invite tokens (can be configured independently)

Go check out the source code and say hello in #miounne:etke.cc

Dept of Bots 🤖

Mjölnir ↗

The moderation bot for Matrix

Yoric announces

Mjölnir v1.2.1 released

• New feature: if a user on your homeserver reports abuse, Mjölnir may now show the abuse report in your moderation room and offer you two-click moderation options. This feature is considered a preview for the time being.
• Performance improvements for protections that need to lock back in the history of a room, decreasing the number of cases in which we could end up timing out.
• Many improvements to testing.

Note: Any rumor of a v1.2.0 Docker image borked by yours truly is sadly true. There should be no risk in 1.2.0 but, to be on the safe side, if you have updated to 1.2.0, please update to 1.2.1.

That's one feature I've wanted for a while, and it's going to make moderation a lot easier! Thanks Mjolnir teams for keeping us safe!

Dept of Interesting Projects 🛰️

MinesTRIX

Henri Carnot says

Hi all !!

Today I want to showcase you MinesTRIX. MinesTRIX is a decentralized social media based on matrix. The goal is to create a privacy respectful social media using the power of matrix while trying to be as simple as possible.

Two Objectives

• Showing that matrix could be used to build such a system.
• Helping find your friends using matrix

Currently supported

• Posting
• Adding and accepting friends
• Basic post management
• Creating groups, posting and adding users to it
• E2EE device verification (thanks FluffyChat !!)
• Cross platform thanks to Flutter (Android, iOS, Linux, Windows, MacOS, WEB)

Now what ?

• Stability fixes
• Finding a logo ;)
• Bring sharing functionality for public groups.
• Adding support for the Circle application.
• Enhance the friends' suggestion algorithm (Currently it's a really naïve one :D)
• Add reactions for chats and posts

🚀 About Demo 🏗️ Gitlab Chat

That's a fascinating client, it looks absolutely fantastic! It reminds me a little of Cerrulean. Good job Henri!

Matrix Login

Paul says

There was the "Sign in with Matrix" project recently

I tried to do something similar with https://matrix-login.lyc.fi / https://gitlab.com/ptman/matrix-login

An important note on the interesting projects using Matrix for the login: those are community projects, and there are MSCs in the works to "do it right" at the Spec level!

As Matthew Hodgson reported in a comment on Hacker News :

The direction we're headed in the Matrix spec core team is instead towards replacing Matrix's current auth mechanisms with normal Open ID Connect (rather than wrapping our own OIDC-like thing, as we do today) - as per https://github.com/sandhose/matrix-doc/blob/msc/sandhose/oauth2-profile/proposals/2964-oauth2-profile.md The common login flow would then be for users to be authed by their server using a trusted OIDC identity provider, rather than ever trusting arbitrary clients with their credentials.

Dept of Guides 🧭

Austin Huang announces

I have compiled a list of public homeservers available for registration, since previous such efforts to make these homeservers more discoverable fell through. This list serves as a sanitized version of the asra.gr list, with only homeservers intended for public consumption included. It is a static list and does not include pings, but rather than focusing on the technical aspect, my list has an emphasis on the written rules of a homeserver, which I believe to play a larger role in the Matrix experience. Hope this can spark other efforts in maintaining a better list!

That's one very useful list of hand curated servers! The transparency about the inclusion criteria is very much appreciated. Good job!

Room of the Week 📆

Timo ⚡️ announces

Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?

This is why I want to share rooms (or spaces) I find interesting.

---

This week's room is: #travel:hacklab.fi

"Discussion about destinations, culture, hotels, flights etc.. English only, be nice to each other."

---

If you want to suggest a room for this section, tell me in #roomoftheweek:fachschaften.org

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!