July 08, 2023

GNOME Web Canary is back

This is a short PSA post announcing the return of the GNOME Web Canary builds. Read on for the crunchy details.

A couple years ago I was blogging about the GNOME Web Canary flavor. In summary this special build of GNOME Web provides a preview of the upcoming version of the underlying WebKitGTK engine, it is potentially unstable, but allows for testing features that have not shipped in a stable release yet.

Unfortunately, Canary broke right after GNOME Web switched to GTK4, because back then the WebKit CI was missing build bots and infrastructure for hosting WebKitGTK4 build artefacts. Recently, thanks to the efforts of my Igalia colleagues, Pablo Abelenda, Lauro Moura, Diego Pino and Carlos López the WebKit CI provides WebKitGTK4 build artefacts, hosted on a server kindly provided by Igalia.

The installation instructions are already mentioned in the introductory post but I’ll just remind them again here:

flatpak --user remote-add --if-not-exists webkit https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
flatpak --user install https://nightly.gnome.org/repo/appstream/org.gnome.Epiphany.Canary.flatpakref

Update:

If you installed the older version of Canary, pre-GTK4, you might see an error related with an expired GPG key. This is due to how I update the WebKit runtime, and I’ll try to avoid it in future updates. For the time being, you can remove the flatpak remote and re-add it:

flatpak --user remote-delete webkit
flatpak --user remote-add webkit https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo

That’s all folks, happy hacking and happy testing.

July 07, 2023

#103 Flowing Information

Update on what happened across the GNOME project in the week from June 30 to July 07.

GNOME Circle Apps and Libraries

Gaphor

A simple UML and SysML modeling tool.

Arjan says

This week Dan Yeaw released Gaphor 2.19.1. The simple modeling tool has received a lot of small improvements and bug fixes. Parts of the UI have been updated to use new GTK 4 elements. Information Flow for Associations has been added, and Object Flows can now connect to Join/Fork Nodes. Get your copy from Gaphor’s download page or directly from Flathub.

Cartridges

Launch all your games

kramo announces

After months of work, Cartridges 2.0 is finally here.

Two new import sources have been added: Legendary and Flatpak.

Alongside several UX improvements, the import backed has been completely rewritten, huge thanks to Geoffrey Coulaud. This allows for faster imports, much better error handling and makes adding new sources much easier, so a lot more are coming soon!

Check it out on Flathub!

Third Party Projects

Swapnil Devesh reports

This week I released Luminance, an app that lets you change brightness of displays including external ones that support DDC/CI (almost all non ancient displays support it). The app is intentionally really simple, and integrates very well with GNOME desktop. It also has a CLI interface for adjusting brightness from scripts or keyboard shortcuts, its available to install for Arch based distros in AUR and deb and rpm packages are available on GitHub releases. Would be looking at porting to GTK4 next, would have loved to make it available as a Flatpak but don’t think that would be possible due to the app needing permissions to access i2c and backlight devices.

Vlad Krupinski says

After two weeks of development I’ve released List 44.6.7. The main feature of this release is implementation of Drag and Drop. Now you can easily move tasks around! Moving sub-tasks between tasks now works too.

This update also includes:

  • Added smooth fade effect on top of the tasks list
  • Tasks list now scrolls when new task added
  • Added button to scroll to the top
  • Minor UI changes and improvements

Gianni Rosato announces

Aviator 0.4.1 is released today with the following changes:

  • The custom SVT-AV1 fork used in Aviator has been updated, promising up to a 40% speed improvement for the slower, higher quality presets
  • A new “Crop” option for switching between cropping & scaling resolution has been added
  • The internal SVT-AV1 parameters have been tweaked for a 0.3-0.8% improvement in perceptual quality

Phosh

A pure wayland shell for mobile devices.

Guido announces

Phosh 0.29.0 is out. This release adds improved audio device selection and notifications for ongoing calls. When phosh has information about a device’s notches it can avoid placing UI elements there and you can enable suspend from the system menu. Notifications can now take more screen space on the lockscreen and we made a bunch of robustness fixes to the emergency info preferences plugin.

Parabolic

Download web video and audio.

Fyodor Sobolev reports

Parabolic V2023.7.1 is here! In this release we fixed 2 bugs and added 1 new feature:

  • Fixed an issue where CPU usage was high during download due to excessive logging
  • When downloading a playlist with enabled metadata embedding, a media’s position will be written to the tag’s track property
  • Fixed an issue where Validate button was sensitive during validation, causing error if the button was pressed multiple times
  • Updated translations (Thanks everyone on Weblate!)

IPlan

Your plan for improving personal life and workflow

Iman Salmani announces

IPlan 1.8.0 is now out! What’s changed:

  • Move up and down option for tasks rows
  • Fade effect for moving between projects
  • Improve date format
  • Popovers aligned and calendar navigation buttons width reduced, to fit in the phone form factor
  • Code refactoring, Bug fixes, and UI improvements

You can get it from Flathub

Fretboard

Look up guitar chords

Brage reports

This week I released Fretboard, an app that lets you find guitar chords by typing their names or plotting them on an interactive guitar neck. No matter if you are a beginner or an advanced guitarist, you can use Fretboard to practice, learn, and master your favorite songs!

I’d appreciate your help with improving the chord data. Please report any weird chord recommendations in the issue tracker, and I’ll have a look at it.

Install Fretboard from Flathub and give it a spin!

GNOME Foundation

Rosanna says

This was a short week because of the holiday, but there still was a lot going on. With GUADEC coming soon at the end of the month, the staff is working towards making sure everything happens as smoothly as possible. If you haven’t signed up for the Latvia social trip yet, there are still some spots open: https://events.gnome.org/event/101/page/164-northern-latvia-trip. I am very much looking forward to that!

I spent a lot of this week catching up on lots of minutiae. There has been an emphasis on taxes and books this past month, as we have been onboarding our bookkeeper and working towards filing our taxes. I also renewed our mailbox, went over our foundation insurance policies, did state-mandated training, etc. In between that, I had meetings on the Executive Director search, travel issues, and finances, along with individual meetings with staff.

I am also getting ready to attend FOSSY next week where I will be giving a talk on creating crosswords with free software. GNOME will also be running a booth there, so please drop on by and say hi if you are going to be in Portland!

That’s all for this week!

See you next week, and be sure to stop by #thisweek:gnome.org with updates on your own projects!

July 06, 2023

Toot toot! Mastodon-powered Blog Comments

If you follow me on Mastodon, you’ve probably heard that I’ve added Mastodon-powered comments to this here blog. If you missed it, check it out!

Read on to dive into my process, the first few iterations of how I did it, and what my hopes are for the future. There are lots of links to the source code on GitHub if you want to follow along, as well. Oh, and throughout this post I’ll be referring to “Mastodon” comments, but technically any Fediverse account that uses ActivityPub can comment on my blog by replying to the the associated Mastodon post—and I do directly use the Mastodon API to get those replies and associated data from my own instance.

Inspiration

At some point, I had seen other folks on Mastodon sharing that they’d added comments to their websites powered by ActivityPub, but it always seemed daunting. After all, I don’t love complex web development and could see rolling my own commenting system as a never-ending web development project. At the same time, I have built a few static websites and spent years iterating on them to feel more like something running on a dynamic server, all while being blazing fast. So the challenge sounded at least vaguely interesting. At some point, WordPress also acquired an ActivityPub plugin, which seemed really cool for people using WordPress. Still, I forgot about it for a while.

I was reminded about it when Jan Wildeboer shared “Client-side comments with Mastodon on a static Jekyll website” on his blog—this seemed perfect! I peeked at it, bookmarked it, and moved on.

Finally, when I was sharing my FreeDesktop Accent Colors proposal for GNOME, I really wanted a better way for people to follow the conversation; that inspired me to finally actually dig in.

First Iteration

As a very first proof-of-concept, I read through Jan’s post again (do so if you are interested in implementing this!), then just took his code verbatim and bolted it onto my own Jekyll site. It actually worked! That allowed me to start really understanding how it works.

Screenshot of comments at first

They’re comments, and they work!

I’ll leave a lot of the details for his post, but the super-simplified gist is:

  1. Write a blog post
  2. Post about it on Mastodon
  3. Grab the resulting post ID and plug it back into the static site
  4. The blog uses the Mastodon API client-side to fetch replies to the given post
  5. Those replies are spit out onto the page and styled

The majority of my work to ship the first iteration was spent on tweaking the DOM that gets spit out and styling it up to fit the style of my site. It was a bit rough, but it worked!

Oops, Emoji and Avatars 🤦️

I excitedly shared a few posts with comments enabled, then someone pointed out: custom emoji (in the :shortcode: format) didn’t work! Womp womp. Luckily, Jan had already solved this for display names, so I was able to copy it for the comment itself pretty easily.

Screenshot of comments with no inline emoji

:oops:, also note the interactions cluttering things up at the bottom

Then it was pointed out they weren’t animated! Well, yeah, because I was requesting the static versions of emoji (and user avatars, actually), so that was expected. I added support for animated emoji and avatars while respecting the prefers-reduced-motion media query that can be set by OSes and browsers. Neat!

Someone pointed out that the avatar styling I was using wasn’t the prettiest with certain profile pictures, so I rewrote that across my whole site—and then I noticed some avatars were provided at a too-small size, so I fixed that, too.

Clean Up

Along the way I noticed a few things that could be cleaned up. First, inline links were included in full, which is pretty ugly. Luckily the Mastodon API returns links with helpful utility classes included, meaning you just have to use a tiny bit of CSS to imitate what Mastodon does on the web and in its apps. Second, I wasn’t handling accounts without an explicit display name as gracefully as I could; I first used some dirty string splitting, then realized I could just use the username from the API.

Screenshot of comments with broken wrapping

Check out that wrapping in the middle comment—that’s no good!

I noticed wrapping was super broken with long names and small screen sizes, so I took the time to finally learn CSS grid; check out the excellent-as-usual CSS-Tricks complete guide if you’ve been putting it off like me. This allowed me to fine-tune the layout of the replies exactly how I wanted in a way that made sense to my brain.

At this point, I was really getting unhappy with the massive blobs of HTML inside of JS inside of HTML—and my text editor’s syntax highlighting wasn’t a fan, either. So I spent some time iterating, rewriting, and simplifying things. First, since I was doing the same thing with emoji twice, I pulled out the emoji-replacing into its own function. Rather than using string replacement on large blobs of HTML, I (re)learned how to directly create and manipulate DOM elements with JavaScript. This was so much cleaner in the resulting code, even if it technically takes a few more lines.

After having a good experience switching to actual JavaScript DOM instead of strings, I took the plunge and rewrote the whole thing in the same fashion. This was my favorite part of the process because it felt like I really finally understood every line of code—and I was in charge of how it worked! Even if it’s not the best way to write JavaScript (I don’t claim to be an expert, here), it really reminded me of whipping up native code in Vala for GTK apps. If you’ve spent any time reading or writing elementary Vala code, you can probably tell I have spent a lot of time doing so as well. 😅️

Design Iteration

Something I noticed when sharing links was that if I had a thread about a blog post, the first post was being omitted because I was only pulling its replies in as comments. I fixed this by hitting the Mastodon API to grab that first post before grabbing its replies.

As a part of the above-mentioned JavaScript rewriting, I spent time thinking about how to clean up the visuals as well. I wanted to make the avatars a bit bigger and just declutter things a bit. I was especially unhappy with having a ton of default-styled links in the header of each reply, including the somewhat daunting full @[email protected] next to every single commenter’s display name.

Literally while I was working on this, Facebook/Meta/Instagram launched their Threads app. I poked at it for a few minutes to see if ActivityPub/federation was working, and it wasn’t enabled yet, so I moved on. While I was poking at it, I noticed something interesting pointing towards that federation, though: on user profiles, Threads includes a little badge with your instance name (currently just threads.net since they’re not federating). I liked this styling, and wondered if I could do something similar to clean up the @[email protected] styling that I didn’t like. So I took a stab at it!

I made the decision to actually heavily deprioritize usernames in favor of display names plus instance badges. This is still a little disjointed as inline @-mentions show usernames, but over all I really like the look. To ensure readers can tell users apart, I added tooltips to the avatar and instance badge to expose the full username—and clicking either will open the commenter’s profile on their own instance. On mobile (at least in the browsers I am able to test), tapping-and-holding either will show the link to the profile which serves the same purpose.

I also wanted a way to visually distinguish my own comments (and more generally, highlight the original poster or “OP” like on Reddit), so I added a sort of “verified” styling to any comments coming from my private instance; I add a check mark badged on my avatar and style the instance badge with a check mark and distinct accent color. I’d like to make this more generalized so if you theoretically had multiple authors on your blog, you could still highlight OP and/or trusted accounts—but that’s a problem for another day.

I went back and forth a few times with if/how to handle post interactions like favorites and boosts; if I was going all-in, I’d figure out a way to use the reader’s Mastodon account to actually directly interact with posts, but that’s… a lot more work. With all my excited sharing about this project on Mastodon, I inspired at least one other person; Julian Fietkau wrote and shared his own implementation. His version also uses JavaScript DOM manipulation, but does a lot more like handling threading, re-ordering replies, stripping leading @-mentions, and more. I still have to dig into it and see if there is more I can pull into my implementation. I did like his styling of favorites, though, so I did something similar for the time being.

Screenshot of comments today

Culmination of these design iterations

What’s Next

To be clear, this whole project was just something I’ve been hacking on for my own site in the evenings and in between my regular work—I don’t promise any future plans! That said, I figure it’s good to actually write up what I’d like to tackle next, if only for my own accountability.

  • Improved loading state: along the way, I made the decision to auto-load the posts instead of requiring user interaction. However, with that change, the comments just pop in once loaded—all with no progress indication. This seems like a clear improvement to make with some sort of placeholder state.

  • Cache at build time then update client-side? I’ve seen some Opinions™ about this since it would require storing others’ data, but this integration intentionally only handles non-private posts, so I think it would be okay? After all, ActivityPub servers cache content in a similar way, edited/deleted posts could be modified/removed client-side after page load, and all the data would be refreshed on the next site build to re-cache the changes. I think this would lead to an improved experience on page load since the comments would be there from the start and only the changes would need to load/shift on the page. Then again, it increases the complexity quite a bit, and basically means I have to maintain code in two places: at build time, and at page load.

  • Threading? At one point I had some CSS to indent replies to other posts or to style them similarly to Twitter/Bluesky/Threads replies, but that quickly turns into a nightmare layout-wise. Naively indenting every reply results in terrible layouts as well. Julian’s implementation does handle some amount of threading, so perhaps I could revisit this and choose to just support one nested level for replies.

  • Reduce links out to make it feel more integrated: instead of just always linking to the commenter’s profile, I could use the data already fetched by the Mastodon API to do something custom like a profile card when tapping or hovering the avatar. This could provide a commenter’s header image, bio, verified links, etc. in a nice digestable format instead of requiring people to follow links. Similarly, I wonder if I could use the Mastodon API to list people who fave’d a post locally/inline, instead of hackily linking to the /favourites URL like now? It’d be neat to have the little overlapping avatars for faves, anyway.

  • Auto-post to Mastodon with the API?? This would be the dream: write a blog post, publish it, then CI actually handles posting to Mastodon on my behalf, logs the resulting post ID, and makes comments appear automatically. Hmm… 🤔️

  • Make reusable! Right now this is somewhat reusable by others with static sites, but I know there are some weird me-isms included (like hardcoding my instance). It would be great to make this even more reusable by others.

Did I Miss Anything?

Okay, you got this far, so you’re dedicated. Let me know if you notice any glaring issues, have a brilliant idea, or have any kind of feedback for me by replying below. After all, dog food is the best food!

July 05, 2023

rpmlint updates (July 2023)

I'm spending some time every week working in the rpmlint project. The tool is very stable and the functionality is well defined, implemented and tested, so there's no crazy development or a lot of new functionalities, but as in all the software, there are always bugs to solve and things to improve.

The recent changes applied now in the main branch include:

  • Update the usage of rpm to not use old API.
  • Fixes for rpmdiff -v, check for NULL char, special macros in comments and spell checking of description in different languages.
  • Move all the metadata from setup.py to pyproject.toml.
  • Releasing rpmlint as pre-commit hook
  • Improvements to the PythonCheck in the dependency checking.

Summer of Code 2023 updates

The first month of the Summer of Code has passed and Afrid is doing a great job there. We've now a draft Pull Request with some initial changes that allow us to mock rpm packages in tests so it's easier to create new tests without the need of creating a binary package.

The first step done was to extend the existing FakePkg class to allow us to define package files and some package metadata.

Now he's working in replacing all of the test_python.py tests that uses binaries rpm to something that doesn't needed.

The idea is to replace as much tests as possible to reduce the number of rpm binaries and after that, provide helper functions, decorators and classes to make it easy to write tests, writing less code.

Roadmap

In any software project there's always room for improvements, fixes and enhancements. If the project is there for enough time, it's even more critical to modernize the code to reduce the technical debt.

My plan for 2023 is to improve the tests around rpmlint as much as possible. First with the GSoC project, making it easier to write more tests, improving the testing tools that we've. And after the summer, improving the test coverage.

There's also a tool that shares some of the ideas with rpmlint, spec-cleaner, it's also written in Python, so the next step, after the tests improvements will be to take a deep look into the code of these two tools and try to integrate in some way. Maybe it's possible to refactor the common code into an external module, maybe we can bring some ideas from spec-cleaner to rpmlint. Not sure yet, but that'll be my next step.

Don't forget that this is free software, so you can participate too! If you find any issue in rpmlint or have an idea to improve it, don't hesitate and create a new issue.

Endless OS’s privacy-preserving metrics system

Endless OS contains an optional anonymous telemetry system, where installations of the OS collect usage data and periodically send it back to us at Endless OS Foundation. Although the system is open-source, has existed in various forms for close to a decade, and individual pieces of it are documented, I don’t think we’ve ever written much about how it all fits together. So here I will try to correct that by describing the current incarnation of this system in its entirety, and why we have it.

Let’s stare directly at the elephant in the room: usage metrics – telemetry, analytics, or whatever other terms one might like to use – is something that free software community members are often opposed to, with some good justification. A lot has changed in the 10 years since the first commit on this project in September 2013. The Facebook–Cambridge Analytica data scandal and other similar events have raised public awareness of the potential for abuse of personal data, and there have been massive regulatory improvements in this space, too. Of course, I have my own opinions about data collection, which you can probably guess from my having spent most of my adult life working on free software! When I joined Endless, I wasn’t comfortable with previous incarnations of our metrics system, but after it was reworked into the form described below, I am confident that it strikes a good balance.

I can’t hope to change every reader’s mind about telemetry, but I do hope to make a more optimistic case for privacy-respecting metrics that are a net positive for free software users and communities alike.

We are only human, so there may be things we have overlooked, whether conceptual or in the implementation. And, we have very limited time to work on this stack at present. If you have feedback or suggestions; if you’re interested in contributing ideas, documentation, code, or analysis; if you want to adopt or adapt these tools in your own distro or projects: please do get in touch!

Goals

Before we go into the details of this system, let’s talk about why an open-source project might want such a system in the first place. After all, the easiest metrics system to build and maintain is no metrics system at all! For Endless OS Foundation, there are two primary reasons to have this system: to demonstrate the impact our work has, and to guide our ongoing work to improve our tools.

Endless OS Foundation is a non-profit, funded by private philanthropic grants. To justify our continued funding, which enables us to keep developing tools like Endless OS and contributing to open-source projects such as Kolibri, GNOME, and Flathub, we need to be able to demonstrate the impact that our work has had in the world. We work with other social-impact organisations who have the same need to justify to their own sponsors why they are collaborating with us and investing their time and money in putting our tools in people’s hands. One way that we do this is through qualitative research, such as end-user interviews; but particularly as a small, geographically-disparate team, we can only do so much of this, so we seek to back this up with quantitative data. Quantity has a quality of its own.

In addition, we are not making software in a vacuum: we are trying to address the digital divide by designing technology for people underserved by other platforms. So at a very basic level, we wish to understand: how do people use our tools? What apps and features of the desktop do people use, or not use? Do people actually use their computer at all, or are they too intimidated by technology to even take the computer out of its box? (This is not a hypothetical example!)

Having such information about if and how people are using our software helps us to make informed decisions about how to improve it, rather than basing our actions on assumption and guesswork. In some scenarios, user testing & interviews are the best way to make such decisions; but when considering usage of a whole desktop over time rather than in a single 1-hour testing session, it’s rarely practical to carry out enough such studies to be confident that you’re not just generalising from a handful of anecdotes.

Commercial organisations have similar needs, which is why basically every website, online store, app and platform today outside the free software space has some kind of analytics capability. In Telemetry Is Not Your Enemy, Glyph Lefkowitz described the FLOSS community’s reflexive rejection of telemetry in any form as “a massive and continuing own-goal”, and imagined how the open-source process could yield telemetry that is “thoughtful, respectful of user privacy, and designed with the principle of least privilege in mind”. I tend to agree. The structure and transparency of FLOSS gives us the opportunity to gather actionable data, transparently, in order to make better decisions and improve our software without betraying our users; and by doing so we could make better use of our limited time and attention.

We also have a very important non-goal: we do not believe people’s individual usage of technology should be tracked. We don’t want or need to know what a particular person does with their computer, or any personal data about them. We are a non-profit organisation with a philanthropic mission to give people access to, and control over, technology. We do not wish to sell data about our users, deliver behaviourally-targeted advertisements to users, and so on.

OK, time to dive in!

Gory details

On Endless OS, applications use a D-Bus API (via a small C library, eos-metrics) to record metrics events locally on the device.

This API is implemented by a system-wide service, named eos-metrics-event-recorder or eos-event-recorder-daemon (no, I don’t know why it has two different names either), which buffers those events in memory, and periodically submits them anonymously to a server, Azafea, which ingests them into a PostgreSQL database (after a short layover in a Redis queue). If the computer is offline – often the case for Endless OS systems! – events are persisted to a size-limited ring buffer on disk, and submitted when the computer is online. (At one point we had a tool for exporting events to external storage on an offline system, and submitting them from a separate online system, but this has bit-rotted.)

In some cases, the events are recorded by individual applications. For example, eos-updater records an event when an OS update fails, and we have patched GNOME Shell in Endless OS to record aggregate app usage duration. There is also another system-wide service, eos-metrics-instrumentation, which records the aggregate duration of user sessions, and information about the system’s hardware.

When stored in the remote database on our server, events carry an event-specific payload (such as the updater error which occurred); a timestamp; the OS version (e.g. “5.0.4”) on which the event occurred; and the channel the system belongs to. The channel is defined by the identifier of the OS image that the system was originally installed from (such as eos-eos5.0-amd64-amd64.230510-144309.fr, which is the downloadable build of 5.0.4 with French content preinstalled), whether this is a live USB, dual-boot install or standalone install, plus an empty-by-default dictionary describing the deployment site (which may be used to distinguish different deployments of the same OS image).

The crucial point here is that the same OS image ID is reported by many systems, so while we can slice the data to a particular cohort of users, we cannot identify the specific computer that any given event came from, or even that two different events came from the same computer. Since we typically have a different OS image for each deployment partner, we are still able to filter and share a summary of only the data that relates to that partner, which might be anywhere between 50 computers and thousands of computers.

There are two types of events. Singular events are buffered and submitted to the server as-is. Aggregate events are, as the name suggests, aggregated on the local system by calendar day and calendar month, and submitted after the current day or month ends. As well as the event payload, they carry an integer counter, which is currently used only for durations (e.g. “Chromium was open for 4 hours on 20th June 2023”), but in principle could also be used to reflect multiple occurrences of the same event. One reason you might want this is to distinguish “OS update failed once due to filesystem corruption on 10,000 different computers” from “factoid actualy just statistical error. Average updater fails 0 times per year. Updaters Georg, who lives in cave & fails to update due to filesystem corruption over 10,000 times each day, is an outlier adn should not have been counted”.

You can view the complete list of events reported by the OS & accepted by the server in the Azafea documentation. (The “Deprecated Events” section documents, for posterity, events that were once recorded at some point in the history of Endless OS, but are not recorded by current OS versions.)

Since no personal data is collected, the metrics system is enabled by default. Defaults are powerful, so having the defaults this way around makes the data more representative of the user base than an opt-in system would; it also means we have a strong need to respect our users’ privacy. A page in Initial Setup allows the user to disable the metrics system; and until Initial Setup is complete, events are buffered locally but not submitted to the server. This means that events from early in the first boot process are not lost if the default is not changed, but that nothing is submitted until the device owner has been informed & given the chance to opt out. (If they do opt out, the buffered events are erased and no further events are buffered unless they later opt back in.)

There is one more component, eos-phone-home, which on the client side is completely separate, though the information flows into the same PostgreSQL database on the server. This is inspired by a system designed for Ubuntu (though I’m not sure if it is in use) and is conceptually somewhat similar to the DNF ‘countme’ system in Fedora. It sends to Endless OS Foundation:

  • A one-time activation message the first time a device goes online, with channel, hardware vendor & model (e.g. “Dell Inc.”, “XPS 13 9380” for my laptop), and the current OS version (e.g. “5.0.4”). The server performs a GeoIP lookup against an offline database, stores the country code (e.g. “GB”) and location rounded to the nearest 1° of latitude & longitude, then discards the IP address without logging it.
  • A ping from each online device (except live USBs), at most once every 24 hours, with channel, hardware vendor & model, current OS version, and the number of previous successful pings from this device. This allows measuring retention over time, without identifying any specific machine. The server determines a country code (and nothing more) via offline GeoIP, then discards the IP address without logging or storing it. The ping also includes a boolean for whether the user has enabled or disabled the system above. Roughly 90% of users have the full metrics system enabled.

For context, 1° of latitude is roughly 111km. 1° of longitude varies between 111km at the equator and 0km at the poles; at the very north of the UK it’s about 55km. This is a 1° by 1° area over Chicago, a city of 2.7 million people:

A rectangle superimposed on a map of Chicago. It covers the bulk of the Chicago metropolitan area, as well as some nearby less-populated areas and a body of water.

And here is São Paulo, where somewhere between 12 and 33 million people live, depending on how you count:

A rectangle superimposed on a map of São Paulo. It completely covers the city and its surrounding metropolitan area, extending almost as far as Campinas, a nearby city.

Here in the UK we sometimes describe large areas in terms of the area of Wales, so here is a 1° by 1° area that is roughly one-third of the size of Wales:

A rectangle superimposed on a map of Wales. It covers a bit less than half of the country.

Every component described above is open-source and auditable. For completeness: it’s not an integral part of the stack, but we use a self-hosted instance of Metabase, an open-core business intelligence platform, to visualise the data stored in the PostgreSQL database. The Endless OS Foundation team has access to Metabase, and thence to the data stored by Azafea; nobody else has access to the raw data. Metabase allows us to make read-only charts and dashboards visible to the public. Here is a visualisation of the total RAM in Endless OS users’ machines in the past 24 hours at the time of writing, rounded to the nearest half-gigabyte (or rather, for my fellow pedants, gibibyte), taken from this event. You can see the live chart here. And if you mouse over one of the segments in the live chart, you can infer from the totals that there are somewhere north of 17,000 computers running Endless OS 4 or newer. (This is an underestimate of our total user count because it only considers the computers in use in a particular 24-hour window, which have not opted out of metrics, and which are not part of the ~20% of systems running an older OS version.)

Chart: “How much RAM do our users have?”  2,048 MB: 1.48% 3.584 MB: 64.23% 5,632 MB: 6.97% 7.680 MB: 23.32% Other: 3.99%

And what have we actually learned?

As you can see above, at the time of writing, a large majority of Endless OS users have around 4GB of RAM. Most of the rest have rather more, but around 1.5% of our users are clinging on with just 2GB. This justifies our periodic work on better handling of low-memory conditions, and reminds us that RAM is not so plentiful as our own devices might have us believe. Perhaps if we could reduce the OS’s RAM usage, we could retain more users on such lower-end hardware.

We have a tool called eos-gates that intercepts users trying to run Windows software or install deb or RPM packages, and (where possible) guides them to the same app, or an equivalent, on Flathub. At the time of writing, by a very wide margin, the most popular Windows app our users try to install is Roblox. Sadly we can’t really recommend a straightforward way to install that particular game, but this data has helped us to find other apps that we should add to the mapping.

After the release of Endless OS 4, a few users came to our forum to report abnormally long boot times. We were able to confirm from our startup-timing metric that this was not an isolated problem, and address the issue in an OS update by moving a repository migration job that was extremely slow on spinning-disk systems with many apps installed to later in the boot process. We saw the improvement in the startup-timing data for subsequent OS releases.

We have also used this data to support decisions outside Endless OS Foundation. For a recent example, the team that develops the freedesktop SDK – used by every one of the 2,000+ apps on Flathub – were considering whether to require a CPU that supports the SSE 4.2 instructions. Because we collect anonymous hardware metrics, we were able to provide a snapshot of the CPUs in use on a given day by Endless OS users, and the freedesktop-sdk team was able to determine what proportion of those systems would be unable to run software from Flathub if this change were made. The other dataset under consideration was the Steam hardware survey, but Endless OS systems are a better representation of “regular people’s computers”, rather than gamers who you would expect to skew towards higher-end systems. (This case demonstrated that recording the CPU model name, such as Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz, is not really adequate. CPU flags would be more useful. In this case, Jordan laboriously mapped the model names back to the relevant data.)

Future work

There is of course more that we could do on this system. In terms of more instrumentation: a common hypothesis is that most desktop Linux systems are single-user, in the sense that they only have one Unix user account (which may or may not be shared between multiple humans). This is supported by anecdote but it would be useful to be able to quantify this. There are many more questions like this: are multi-user systems multi-lingual? Do many users have multiple keyboard layouts configured? How common are different monitor configurations?

It would be interesting to look at ways to further anonymise the reported data. One possible approach is randomised response, where before submitting a boolean data point, you flip a coin. If it comes up heads, you report true. If it comes up tails, you report the actual boolean flag. Then, if 20% of reports are false, you can assume that the true ratio is 40%, without having to know which specific data points are valid. There are related techniques for numerical data where you bias the data with some known probability.

You could use this system as part of an opt-in user studies programme. For example, some tool could explicitly prompt a sample of users for their consent to record identifiable data over (say) a week, to perform some targeted behavioural study, perhaps in combination with a Shell extension implementing some experimental change. The site ID mechanism could be used to tag an individual system for the duration of the study, or which half of an A/B test the system fell into; and further events could be recorded for that week. After the week, the user would be prompted for some written feedback, then the ID would be removed and the additional events disabled.

While we cannot share the raw underlying data publicly – lest someone smarter than me figure out a way to de-anonymise it – I would love to see us publish dashboards of aggregated data, similar to the Steam hardware survey, Mozilla’s Hardware Across The Web, Flathub’s statistics, and so on. This hasn’t happened yet only for want of having time to do it. Please do get in touch if there is specific data of public interest.

And finally: Endless OS is just a moderate slice of the open-source desktop pie. We have somewhere in the low-to-mid tens-of-thousands of active online users – this is not insubstantial, but my guess is that it is one or two orders of magnitude smaller than larger platforms like Fedora, Ubuntu, and Steam OS. I believe the tools Endless has built are pretty good – shout-out to Kurt von Laven, Philip Chimento, Devin Ekins, Cosimo Cecchi, Mathieu Bridon, Guillaume Ayoub, Antoine Michon, and many other contributors over nearly a decade – and it would be great to see them adopted elsewhere in the desktop Linux space. Even more than that, I would love to see privacy-respecting telemetry accepted more widely in the desktop Linux space, so that we as a community can make better-informed, data-driven decisions.

GSoC 2023: Rust and GTK 4 Bustle Rewrite (Week 3 & 4)

Porting to zbus and Diagram Implementation Experimentation

Progress Made

There's unfortunately not a lot to talk about for the past two weeks as I have been really absorbed with finals at my university. However, while preparing for the finals, we got the PR to add from_bytes constructor for zbus::Message merged. This means that we can now construct zbus::Message from raw GDBusMessage bytes and vice versa, which is critical for porting Bustle from GDBus to zbus. On the other hand, the PR to implement Display for Value, Array, Structure, Dict & Maybe is, while the implementation for zbus::Value::Str and zbus::Value::Array is already fixed, still on the review stage as there are still things to polish, such as figuring out a way to simplify the implementation.

Meanwhile, I have also been experimenting with drawing on the diagram using GTK transform functions such as translate_coordinates and compute_point, but nothing is ready enough yet for a screenshot.

Plans for the Following Weeks

With my finals out of the equation, I could focus more on finishing up drawing the signals and methods arrow on the diagram during the last days on GSoC midterm period.

I will also push PR to implement Display for Value, Array, Structure, Dict & Maybe to be merged so that the zbus port MR can also be merged.


That's all for this week. I'm looking forward to writing more in the next weeks. Thanks for reading!





July 04, 2023

CapyPDF 0.4 release and presenter tool

I have just released version 0.4 of CapyPDF. You can get it either via Github or PyPI. The target of this release was to be able to create a pure Python script that can be used to generate PDF slides to be used in presentations. It does not read any input, just always produces the same hardcoded output, but since the point of this was to create a working backend, that does not really matter.

  • The source code for the script is here. It is roughly 200 lines long.
  • The PDF it creates can be accessed here.
  • A screenshot showing all pages looks like the following.

What the screenshot does not tell, however, is that the file uses PDF transition effects. They are used both for between-page transitions as well as within-page transitions, specifically for the bullet points on page 2. This is, as far as I can tell, the first time within-page navigation functionality has been implemented in an open source package, possibly even the first time ever outside of Adobe (dunno if MS Office exports transitions, I don't have it so can't test it). As you can probably tell this took a fair bit of black box debugging because all PDF tools and validators simply ignore transition dictionaries. For example transcoding PDFs with Ghostscript scrubs all transitions from the output.

The only way to see the transitions correctly is to open the PDF in Acrobat Reader and then enable presenter mode. PDF viewer developers who want to add support for presentation effects might want to use that as an example document. I don't guarantee that it's correct, though, only that it makes Acrobat Reader display transition effects.

July 03, 2023

ASG! 2023 CfP Closes Soon

The All Systems Go! 2023 Call for Participation Closes in Three Days!

The Call for Participation (CFP) for All Systems Go! 2023 will close in three days, on 7th of July! We’d like to invite you to submit your proposals for consideration to the CFP submission site quickly!

ASG image

All topics relevant to foundational open-source Linux technologies are welcome. In particular, however, we are looking for proposals including, but not limited to, the following topics:

The CFP will close on July 7th, 2023. A response will be sent to all submitters on or before July 14th, 2023. The conference takes place in 🗺️ Berlin, Germany 🇩🇪 on Sept. 13-14th.

All Systems Go! 2023 is all about foundational open-source Linux technologies. We are primarily looking for deeply technical talks by and for developers, engineers and other technical roles.

We focus on the userspace side of things, so while kernel topics are welcome they must have clear, direct relevance to userspace. The following is a non-comprehensive list of topics encouraged for 2023 submissions:

  • Image-Based Linux 🖼️
  • Secure and Measured Boot 📏
  • TPM-Based Local/Remote Attestation, Encryption, Authentication 🔑
  • Low-level container executors and infrastructure ⚙️.
  • IoT, embedded and server Linux infrastructure
  • Reproducible builds 🔧
  • Package management, OS, container 📦, image delivery and updating
  • Building Linux devices and applications 🏗️
  • Low-level desktop 💻 technologies
  • Networking 🌐
  • System and service management 🚀
  • Tracing and performance measuring 🔍
  • IPC and RPC systems 🦜
  • Security 🔐 and Sandboxing 🏖️

For more information please visit our conference website!

gitlab.freedesktop.org now has a bugbot for automatic issue/merge request processing

As of today, gitlab.freedesktop.org provides easy hooks to invoke the gitlab-triage tool for your project. gitlab-triage allows for the automation of recurring tasks, for example something like

If the label FOO is set, close the issue and add a comment containing ".... blah ..."
Many project have recurring tasks like this, e.g. the wayland project gets a lot of issues that are compositor (not protocol) issues. Being able to just set a label and have things happen is much more convenient than having to type out the same explanations over and over again.

The goal for us was to provide automated handling for these with as little friction as possible. And of course each project must be able to decide what actions should be taken. Usually gitlab-triage is run as part of project-specific scheduled pipelines but since we already have webhook-based spam-fighting tools we figured we could make this even easier.

So, bugbot was born. Any project registered with bugbot can use labels prefixed with "bugbot::" to have gitlab-triage invoked against the project's policies file. These labels thus serve as mini-commands for bugbot, though each project decides what happens for any particular label. bugbot effectively works like this:

sleep 30
for label in {issue|merge_request}.current_labels:
  if label.startswith("bugbot::"):
     wget https://gitlab.freedesktop.org/foo/bar/-/raw/{main|master}/.triage-policies.yml
     run-gitlab-triage --as-user @bugbot --use-file .triage-policies.yml
     break
And this is triggered on every issue/merge request update for any registered project which means that all you need to do is set the label and you're done. The things of note here:
  • bugbot delays by 30 seconds, giving you time to unset an accidentally applied label before it takes effect
  • bugbot doesn't care about the label beyond the (hard-coded) "bugbot::" prefix
  • bugbot always runs your project's triage policies, from your main or master branch (whichever succeeds first)
  • The actions are performed as the bugbot user, not your user
The full documentation of what you can do in a policies file is available at the gitlab-triage documentation but let's look at a simple example that shouldn't even need explanation:
resource_rules:
  issues:
    rules:
      - name: convert bugbot label to other label
        conditions:
          labels:
            - "bugbot::foo"
        actions:
          labels:
            - "foo"
          remove_labels:
            - "bugbot::foo"
          comment: |
            Nice label you have there. Would be a shame 
            if someone removed it
          status: "close"
  merge_requests:
    rules:
      []
And the effect of this file can be seen in this issue here.

Registering a project

Bugbot is part of the damspam project and registering a project can be done with a single command. Note: this can only be done by someone with the Maintainer role or above.

Create a personal access token with API access and save the token value as $XDG_CONFIG_HOME/bugbot/user.token Then run the following commands with your project's full path (e.g. mesa/mesa, pipewire/wireplumber, xorg/lib/libX11):

$ pip install git+https://gitlab.freedesktop.org/freedesktop/damspam
$ bugbot request-webhook foo/bar
After this you may remove the token file and the package
$ pip uninstall damspam
$ rm $XDG_CONFIG_HOME/bugbot/user.token
The bugbot command will file an issue in the freedesktop/fdo-bots repository. This issue will be automatically processed and should be done by the time you finish the above commands, see this issue for an example. Note: the issue processing requires a git push to an internal repo - if you script this for multiple repos please put a sleep(30) in to avoid conflicts.

Adding triage policies

Once registered, the .triage-policies.yml file must be added to the root directory of your project. What bugbot commands you want to respond to (and the actions to take) is up to you, though there are two things of note: you should always remove the bugbot label you are reacting to to avoid duplicate processing and gitlab-triage does not create new labels. So any label in your actions must be manually created in the project first. Beyond that - the sky's your limit.

Remember you can test your policies file with

 $ gitlab-triage --dry-run --token $GITLAB_TOKEN \
   --source-id foo/bar  --resource-reference 1234

As usual, many thanks to Benjamin Tissoires for reviews and the magic of integrating this into infrastructure.

July 02, 2023

Problems With Not Raising Awareness and Protecting Marginalized Groups in FOSS

Introduction

A common problem I notice on the internet is the lack of awareness of many problems in the world. Of which, there are times when someone may be aware of the problem, but not to what extent. One of the common problems is a lack of effort to raise awareness and protect marginalized groups, particularly in the free and open-source software (FOSS) community.

In this article, we’re going to go over the problems associated with not raising awareness and protecting marginalized groups. I’ll also go over the “keep politics out of FOSS” demand from many free software activists who oppose raising awareness.

What Is a “Marginalized Group”?

Before I go over the importance of protecting marginalized groups, allow me to explain the meaning of a marginalized group. “Marginalized” means being excluded in bad faith1 or being viewed as an insignificant. A marginalized group is a group of people that is excluded in bad faith and/or treated as a less important group. Some examples of marginalized groups include but are not limited to:

  • sexuality and gender
  • disability
  • elderly or young (age)
  • immigrant status
  • religion
  • socioeconomic status

For a list of types of discrimination, see the Discrimination sidebar on Wikipedia.

Lack of Awareness May Result in Not Providing Necessary Accommodations

There are many underrepresented groups that are unfortunately easily forgotten, not taken into consideration, never consulted, and/or not prioritized. This may lead to not providing any accommodations until developers are made aware. One group I can think of is people with disabilities, such as blind people.

We all know that people with disabilities exist, but when we have many things to worry about, it may become easy to forget about them. For example, my website’s accessibility (usability for people with disabilities) used to be far from ideal:

  • Media (images/videos) lacked descriptions for screen readers or unstable internet connection.
  • Many decorative images and elements weren’t ignored by screen readers. This means that the screen reader user would have their time wasted on the screen reader explaining unnecessary information.
  • Some areas were low on contrast, like code blocks. This would make it difficult for vision-impaired readers to read the content.
  • A “Skip to main content” button was missing. This button is useful for keyboard users that don’t want to go through the navigation bar.
  • There weren’t any accommodations for people who rely on reduced motion. Buttons were animating no matter the preference.

All these accommodations happened later, because I didn’t take them into consideration when I made the website initially.

When I gained more experience with web development and chatted with people with disabilities, that’s when I was made aware of it. I researched a lot about accessibility and slowly started accommodating more people. I trained myself to write better descriptions for media and faster.

This isn’t only from a maintainer/developer perspective, but from a user perspective as well. Have you ever wondered why buttons on GNOME are large? Not only is it practical for touchscreen users, but it’s also useful for people whose hands constantly fidget, like mine. Many people struggle with clicking accurately, so having large(r) buttons helps us click the right buttons without having to put excessive effort into moving our mice. That’s also one reason why GNOME removed status icons a while back (under “4. Accessible click targets”).

Many users believe that some visual designs exist just to copy from others, look unique just for the sake of uniqueness, etc. However, many design elements are strictly focused on people who struggle with certain cues, so it’s important that users are also made aware of these accommodations, and hopefully sympathize with these decisions.

Contributors Are Driven Away by Harassment

There are many efforts put into protecting and raising awareness of marginalized groups in FOSS, but, in my opinion, it’s still not enough.

A good example of an organization that protects marginalized groups is GNOME. GNOME even goes as much as making a statement about putting marginalized people’s safety in priority on its code of conduct:

The GNOME community prioritizes marginalized people’s safety over privileged people’s comfort

[…]

Outreach and diversity efforts directed at underrepresented groups are permitted under the code of conduct. For example, a social event for women would not be classified as being outside the Code of Conduct under this provision.

…but would GNOME be a good example? Since GNOME is a huge player in the FOSS space and while it protects marginalized groups, it is also one of the largest targets for malicious actors. As such, these actors may harass marginalized people outside of GNOME spaces.

During my course of interactions, I’ve chatted with a few victims who stopped involving with GNOME because of the constant harassment (outside of GNOME). One of them shared the following to me:

Message from former GNOME Foundation member: “I had to de-list from planet GNOME because of the threats, harassment, and utter garbage I was getting both in my blog comments and sent to me via other channels and I just let my foundation membership lapse and stopped being an active GNOME contributor. ☹️ I get that occasionally in other open source projects like even in Fedora, but it was really next level in terms of nastiness and volume… GNOME is just super high-visible and/or somehow is a magnet for these people”.

Obviously, the GNOME Foundation does everything it can within its resources to protect marginalized groups, but even then, the weight of attacks may unfortunately outweigh GNOME’s efforts. It really shows that, despite putting a lot of effort to increase awareness and protect marginalized groups, it may still not be enough. It doesn’t help that the FOSS ecosystem lacks contributors (code, documentation, moderation, etc.). Pushing a person away one at a time only worsens the situation as a whole.

Emotionally Draining

With the lack of awareness among tons of people, it can be very difficult to defend or explain themselves in environments that cannot relate with or understand them.

Furthermore, some even go as much as understating. Understating means to make a problem look like it’s little of importance. For example, a disabled person might say that they can’t use Linux, as it lacks tons of accessibility accommodations. Someone else could reply with “Oh come on, just take these 15 [complicated] steps and everything should work fine. You’re just being oversensitive.” Since the non-disabled person either refuses or is unable to sympathize, they might resort to understating. Even worse, many people with disabilities/disorders may pretend to be “normal”, [1] [2] when they’re in an environment that they wouldn’t fit if they expressed themselves, or they’re in an environment that doesn’t actively showcase that they’re protected.

If you’ve done customer service or technical support, you probably have gotten really annoyed to continuously repeat yourself to each person on a day-to-day basis. You’ve also probably stumbled upon the article “Don’t ask to ask, just ask”, as the article explains a really common behavior when requesting for support. Or, if you haven’t done customer service or technical support, then there’s probably going to be a few things you continuously repeat to people, perhaps even to your parents when they don’t know how to use their phone.

I’m already exhausted with explaining over 100 times to people that Flatpak isn’t some corporate overlord that wants to take over the Linux desktop. I can’t imagine how heartbreaking it may be for a marginalized person to continuously repeat themselves about their identities, disabilities, struggles or being a target for bad actors.

“Keep Politics Out of FOSS”

The most common demand we hear from opponents of these initiatives are “keep politics out of FOSS”. By “politics”, they mean “any kind of politics that is out of scope of the free software movement”, which means that FOSS should only be about promoting open and redistributable code, not alongside other activism.

The main problem with solely focusing on the “open and redistributable code” element is that we’re making the entirety of FOSS inaccessible for everyone else, including potential developers. FOSS is not only about open and redistributable code; it’s also about software development, because without software development, there isn’t software; without software, there isn’t FOSS.

Remember: software development is much more than just about maintaining code; it’s about designing, marketing, documenting, and testing as well. Then, these processes consist of designers, marketers, documenters, and quality assurance testers, respectively.

If we keep politics out of FOSS, then we’d literally be excluding all the aforementioned groups, excluding developers and maintainers; we’d be excluding tons of potential code contributors as well.

The only reason you and I are using Linux in the first place is because there were initiatives that prioritized approachability, which are politically outside of the scope of the “open and redistributable code” philosophy. If it weren’t for these initiatives and internal activism, then we wouldn’t be using FOSS at all, and FOSS would’ve been long gone. Either that, or we wouldn’t be on Linux because it would’ve been too difficult or inconvenient.

Each project has goals they want to reach, with one or multiple target audiences. So now, not only do we have the group of people within the software development space, we have our target audience(s) to take into consideration as well; we need to make it approachable for them. For example, one of GNOME’s goals is to be as inclusive as possible for contributors, especially for marginalized people and underrepresented groups. As such, GNOME does whatever it can (within reason) to make it appealing for that target audience; as a result, it protects marginalized people.

“Keep[ing] politics out of FOSS” is something that is always bound to fail, as FOSS is by nature political beyond open and redistributable code. We need groups from different backgrounds and skills to make FOSS as appealing, useful, and work as best as it can. We need marketers to make the software deliver value to their target audience(s). We need designers to make it look well for their target audience(s). We need documenters to provide self-service and make the development experience as easy as possible. We need testers to eliminate as many bugs as possible. Without politics, there is no FOSS.

One particular person I admire is a 16-year-old trans girl who is a member of the GNOME Foundation. She has made significant contributions to GNOME and implemented a feature that I have wanted for a long time. Unfortunately, I can’t go into detail because she’s not comfortable revealing her age and name together publicly, as she doesn’t want to be discriminated against and harassed outside of GNOME spaces — rightfully so. She probably wouldn’t be here if she didn’t feel protected in her environment.

I suggest reading “Politics in your FOSS project? It’s more likely than you think!” by Samsai. Their article goes into detail on this topic.

Conclusion

Unfortunately, discrimination and harassment are still common in the FOSS community. Even when large communities make a real effort to protect them, they continue to be harassed in external spaces, such as personal websites, forums, public groups, etc. It gets even worse when marginalized people are more susceptible to harassment while also being emotionally burned out by having to constantly speak out about their situation, whether because of a disability, gender/sexuality, ethnicity, or other factors.

I would like to thank Oro (trans) and Ember (physical disability) for carefully reviewing this article. Oro has been harassed by some communities because of their gender. As a friend, I think it’s really important to raise awareness about protecting marginalized groups so that we can provide a safe space for them to express their identities.


Footnotes

  1. Bad faith in this context means to exclude groups by actively and intentionally harming them. 

July 01, 2023

Niepce June 2023 updates

This is the June 2023 update for Niepce.

The importer

Back where we left, still working on the import.

It looks like there is more work than anticipated to be able to import a tree a files.

The database model

The database model was designed a while ago, when some of the concepts I had in mind weren't clear enough. So this needs to be fixed. First I need to make folders unique on the parent and the name. With the current design a folder is a physical directory on the filesystem, which is way simpler than the idea I had. Since updating the database schema also mean upgrading, I took the opportunity to make keywords unique on the keyword and parent (they are in a hierarchical organisation too).

File management

Since the inception I had the plan to have two kinds of way to "store" image imported. Managed and mot managed. Managed meant that the file would be "owned" by the library, copied into a location controlled by the app. Not managed meant it's just a reference to a file on the filesystem. This concept idea came from Apple Aperture™ that did just that, where you'd have a library occuping dozens of GB on your disk. The storage layout of the files was abstracted.

Instead I decided to simplify the approach. Importing files will by default reference, or will copy into a specific location (when importing from a camera, only the latter). The folders are actual directories in storage and this goes hand in hand with the recursive import.

So I just ripped out the Managed enum wherever it was used and ignored. This was unfortunately ported from the C++ code a while ago.

UI bug

Of course this look like an onion I peel, with each layer that make me cry. This one is a crash in the GTK code, but it was triggered by a bad algorithm causing an attempt to load an item out of the array bounds. Part of the fix is to use #[must_use] for the container associated function to prevent ignoring the result. The gorry detail is that the (custom) container push() associated functions returns the old item if it is being replaced. It happen that the code assumed the len() increased all the time. Forcing to check the return value guards (with a warning) that the result not get ignored, and in that case properly handling the container size.

The result

The main patch got merged. It adds a recursive flag to the file import UI, and update the copy_importer example that I use for manually testing. From a storage standpoint it works:

select * from folders;
1||Trash|0|1|1|0|0
32|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022|2022|0|0|0|0|0
33|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022/20220814|20220814|0|0|0|0|32
34|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022/20220815|20220815|0|0|0|0|32
35|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022/20220816|20220816|0|0|0|0|32
36|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022/20220817|20220817|0|0|0|0|32
37|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022/20220819|20220819|0|0|0|0|32
38|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022/20220820|20220820|0|0|0|0|32
39|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022/20220821|20220821|0|0|0|0|32
40|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022/20220824|20220824|0|0|0|0|32
41|/var/home/hub/Pictures/Camera import 2023-04-09 004747/2022/20221031|20221031|0|0|0|0|32

The last column is the parent_id that reference id, the first column. So the hierarchy is here.

However the UI is still not doing it:

Workspace

All the folder items should be below the 2022 folde while here they are on the same level. Note the disclosure triangles are a limitation of GtkListView.

What's next

Definitely I need to finish the UI. Some bugs are left and as well as the UI to pick the destination folders in the import dialog.

Also needing fixing is the display hierarchical keyword, I already did some work for it with the database schema update.

Thanks you for reading.

June 30, 2023

Summer Maps

 It's about time for the yearly (Northern Hemisphere) Summer Maps blog post. 😎



Since the release of 44.0 in March, aside from some fixes (like adding support for authentication HTTP headers in OpenTripPlanner plugin, as now needed by the Finnish Digitransit service) there has also been some changes on the surface in Maps.

First, Jakub Steiner has contributed with a refresh of icons used for transportation modes, and the various mode of public transit.

Here we can see the icons for transportation modes (walking, biking, car, and public transit, using a „train icon“) using the new GNOME icon style

In this capture we can see the new icons for metro (underground, subway service)


Above there is some samples from Portland, showing trams and gondolas.


Another new feature is the „Explore nearby points-of-interest“ browser mode of the main search entry, allowing the select preset categories grouped by main and sub categories. Clicking on a subcategory then proceeds with a search given the selected type of place centered around the center of the current map view sorted by, and showing the distance to the place („as the crow flies distance“):

Clicking the down-arrow button, or using the new CTRL+SHIFT+F shortcut shows the main categories

After selecting a sub category

Showing points-of-interest results with distances

Results nearby (less than 100 m, or 300 feet when using Imperial measures) are shown with a „less than“ indication as these can be somewhat imprecise, especially for things with a “two-dimensional extent“ such as larger buildings, or areas

 

This feature was something I had in mind for a while, and there has been an old issue about this floating around since the Bugzilla days (before the migration to GitLab, it was issue #3 after the migration…).

I remember envisioning the outlines of this during a walk one afternoon/evening early in winter of 2021 during the mandated „working from home“ days when we had step contest at work. But I had put the idea off before in anticipation of the GTK 4 port as I din't want introduce too much new UI needing porting before that). But now that that has been finished it was time…

 Behind the scenes this is using the Overpass API to search the OpenStreetMap database based on tags and centered around a specific coordinate.

And by the way, the set of categories is defined in JSON-like object structure in the src/potCategories.js module, so this can be altered relatively easy (and add Overpass query parameter „fragments“.


The next thing, the routing sidebar has had some issues when using a small form-factor device (e.g. phones), now there is WIP in an MR to migrate the sidebar to use an OverlaySplitView from libadwaita.


This allows swiping in from the edge to bring up the routing sidebar and „swipe away“ the sidebar to close it on touch screens.

It also „collapses“ the view so the sidebar is overlayed on the map view rather than „pushing it aside“ on narrower screens:


This is still not finally merged to main, but hopefully this should make it in for 45.0.


Another thing is that since migration to libshumate it is possible to rotate the map view using touch gestures. And as we didn't have a way to reset the rotation (beside restarting the application) to mitigate that I have added a compass button showing when the view rotated away from normal „north is up”.

 

Also the zoom buttons have been moved back to be overlay buttons as they where back in the day. They where changed to headerbar button back in 2017 as a work-around for GTK widgets not being able to be overlayed on top of Clutter surfaces on Wayland. But this is no longer an issue as we don't use Clutter nowadays.

This overlay button design might be subject for a redesign later on. Possible moving them down to the bottom end (right in LTR) corner, similar to the zoom controls of the new Loupe image viewer app. The license banner would then be displayed on startup as toast maybe, or as popover when clicking an info button. Or something like that.

And last, but not least we had a contribution from Szymon Kłos who implemented support loading tracks recorded by Garmin sport watches as shape layers. Thanks Szymon!

I will also attend GUADEC in Riga later in July, so those who go there, maybe see you there!

And I think that's about it for this time!

2023-06-30 Friday

  • Mail chew; partner call, lunch, interview, partner call. Amused to read more about the supposed panacea that is distro-less containers.
  • Also read RedHat Open Source commitment. I have a number of thoughts:
    • First what we do: Collabora Online source code is open - our releases are tags in a public git repo, which we also link from help->about - that helps us support people better, and makes everything super transparent.
    • But - I have huge sympathy with this sentiment; emphasis mine:
      "The generally accepted position that these free rebuilds are just funnels churning out RHEL experts and turning into sales just isn't reality. I wish we lived in that world, but it's not how it actually plays out".
    • One of the troubling things about being in business is having to live in the real world which can get extremely and painfully real. It is particularly annoying at such times to be told - that making it ever easier for people not to pay is the only true solution.
    • The risks of people taking your hard-work, slapping their brand on it, and not contributing significantly are ones we all have long and disappointing experience with too. Of course the FLOSS licenses allow it - but to feel entitled to have this made extra easy for you is unfortunate.
    • My take is a simple and perhaps radical one: Making it easy for everyone not to pay to support development is profoundly counter-productive. Put another way: someone needs to pay for something scarce. We can try to work out who that someone should be eg. "very large IT organizations" are a traditional favourite, and what is scarce (traditionally signed enterprise binaries), but some degree of compromise is inevitable - I have a long write up on various different compromises in the space here: Sustained Freedom from slide sixteen.
    • I have even more sympathy with the rationale, because RedHat is a pure-play FLOSS company. If you are part of the Proprietary periphary mob (also known as OpenCore) - then you have your own proprietary stuff that allows you to make it arbitrarily hard for people not to pay to support development. As such - ironically - as a community it seems we're once again focusing our criticism on those who differ least from FOSS orthodoxy, and who are doing the best job of up-stream contribution.
    • However - this rationale as presented:
      "Simply rebuilding code, without adding value or changing it in any way, represents a real threat to open source companies everywhere. This is a real threat to open source, and one that has the potential to revert open source back into a hobbyist- and hackers-only activity."
      simultaneously seems contrived. Surely that is what Linux Distros do: they dis-intermediate FLOSS projects - but it is perhaps also an opportunity, let me explain:
    • One of the significant concerns around commercially funded FLOSS projects is that of being dis-intermediated: having their latest code bundled into a Linux distro where it is simultaneously extremely hard to get leads (which drive sales) from downloaders -and- long-term support guarentees are met by that distributor; often without any feature contribution back. The same dis-intermediation threat is there around cloud provision of pure-play FLOSS products.
    • Sometimes creator companies are simply acquired to get the support team in-house; while fair - that seems far from optimal. I see a potentially lucrative opportunity for the first enterprise distro that can build a wider partner ecosystem of contributing open-source companies by - including them into their enterprise products via some transparent business and support co-development model. We have lots of excellent, standard FLOSS licenses for code, but few successful open-agreements for go-to-market FLOSS collaboration - building a more diverse and widespread Open Source business community.
    • Can you imagine the power of the possibilities of RedHat/IBM's scale and experience helping to bring their extraordinary reach into enterprises to a snowballing set of businesses built around the RedHat platform with some turbo-charged mutual partnership model? The volume of FLOSS that could be written & improved, and the niches we could fill and sustain?
    • Either way, it will be interesting to see where this goes long term. For those with very long memories I believe that Cygnus tree used to be distributed only to their customers - in the 1990s.

GSoC Update: PlayAnswer widget (wip)

After dedicating first two weeks of work to libipuz, it was time to shift our focus to the UI part of the Application.

Discussion

To create the PlayAnswer widget, we had two choices: either inherit from the PlayGrid or design a container that includes a PlayGrid as its child. Ultimately, we decided to go with inheriting from the PlayGrid.

Design doc

To commence the journey(coughs), I wrote a design document that aims to provide a hilariously clear and concise overview of the PlayAnswer widget.
You can check that out here!

Inheriting from PlayGrid

In the existing code, the PlayGrid was defined using G_DERIVE_FINAL_TYPE, which essentially marked it as a final class that couldn’t be inherited. However, our goal was to inherit the functionality of the PlayGrid in the PlayAnswer widget.

To achieve this, we had to make the PlayGrid class derivable by removing the G_DERIVE_FINAL_TYPE declaration and making it G_DECLARE_DERIVABLE_TYPE. This allowed us to extend and inherit from the PlayGrid class. Additionally, we created a private structure within the PlayGrid class to encapsulate private members and data.

Writing PlayAnswer

So next up was writing the PlayAnswer widget. To accomplish this, Initially I wrote a simple code that would dynamically generate an answer grid for each clue.
(Including all screenshots here just to showcase my progress on this :-)

As you can see, the appearance of the answer grid leaves much to be desired.

To enhance the visuals of the PlayClueRow widget(which contains the PlayAnswer), I created a separate .ui file. It represents the declaration of the widget’s layout and structure.
Also, I moved the initialization of num_label and clue_label from the play_clue_row_init() function to the .ui file itself.
I used a horizontal box to contain num_label and clue_label to align horizontally and then set the orientation of Layout Manager to vertical, aligning the horizontal box and PlayAnser widget in top-to-bottom fashion.
Perfect.

Also, we are now copying the style and the label of the cells too.

To ensure that whenever values are entered on the main grid they get immediately reflected upon the corresponding cells of the PlayAnswer grid, I wrote a play_answer_update_state function which gets called every time there is an action performed on the puzzle.
Furthermore, I introduced padding between the clue and answer grid and adjusted the zoom level of the answer grid to align it seamlessly with the main grid.

Related Merge Requests:

  • Make Private Structure for PlayGrid !108
  • Add PlayAnswer widget !110

The work on the PlayAnswer widget is still in progress. Stay tuned for the next report. See you then!

Thanks for reading!

GSoC 2023: Monthly Report

alt

Hello everyone,

It's been a month since the start of the GSoC 2023 coding period. There is a lot to talk about and show, but here are the highlights of the month and the things I worked on.

Mentors:

Sonny Piers and Andy Holmes

Project:

Make GNOME Platform demos for Workbench


Workflow:

alt

We (the team, including the mentors and contributors) use a github integrated kanban board system to track the progress of current available issues, issues in progress, backlog issues, and completed issues, which is followed by a weekly meeting to discuss blockers during work and ideas for what else we could work on in the future.


My Contributions:

For the past month, I've been working and learning more about GJS, GLib, GObject and Gio every single day. I've been able to create demos and examples on:

Apart from the above-mentioned demos and examples, I've also worked on:

Gtk.Frame

alt

Gtk.Frame is a widget that surrounds it's child with a decorative frame and optional label.

Portal.get_user_information

alt

alt

alt

Xdp.Portal.get_user_information gets information about the user.

Portal.set_wallpaper

alt

alt

alt

alt

Xdp.Portal.set_wallpaper sets a desktop background image, given by a uri.

Portal.compose_email

alt

Xdp.Portal.compose_email presents a window that lets the user compose an email, with some pre-filled information.

Portal.pick_color

alt

alt

Xdp.Portal.pick_color lets the user pick a color from the screen.

Gtk.TextView

alt

Gtk.TextView widget that displays the contents of a GtkTextBuffer.

Adw.AboutWindow

alt

alt

Adw.AboutWindow provides a window showing information about the application.

Gtk.FontDialog

alt

alt

Gtk.FontDialog object collects the arguments that are needed to present a font chooser dialog to the user, such as a title for the dialog and whether it should be modal.

Portal.take_screenshot

alt

Portal.take_screenshot takes a screenshot.

Below mentioned demos are in review/hold:

Adw.Clamp

alt

Adw.Clamp is a widget constraining its child to a given size.

Portal.location_monitor_start

alt

alt

Portal.location_monitor_start makes XdpPortal start monitoring location changes.

Currently, I'm working on Implementing a CodeFind Feature similar to GNOME text editor and implementing a demo on List Models. Below are the screenshots (Note: These are not the final results and are currently under progress )

alt

alt


My Experience:

It's been a great month full of experiences. I've learned something new every day, from random conversations to discussing about the work. It's been a wonderful experience working with my teammates and my mentors to contribute to Workbench as part of the Make GNOME Platform Demos for Workbench project. My teammates also created numerous outstanding demos and examples, some of which are listed here. I'm also very happy to reshare that Workbench currently offers more than 50 examples and demos. More information on this is available here.


Conclusion:

I'm having a great time at work and am excited to keep adding to Workbench. In addition, I'll make an effort to blog more frequently, but that's it. Thank you for taking the time to read this.

If you liked this blog and want to connect below are my socials:

#102 Contextual Back Buttons

Update on what happened across the GNOME project in the week from June 23 to June 30.

GNOME Core Apps and Libraries

Libadwaita

Building blocks for modern GNOME apps using GTK4.

Alice (she/they) says

AdwNavigationView back buttons now support a context menu allowing to pop multiple pages at once. This works with nested navigation views and even with structures like AdwNavigationSplitView containing navigation views in both content and sidebar. Additionally, back button tooltips are now more reliable - they previously didn’t work in this situation and required the app to manually sync the sidebar page’s title with its navigation view’s visible page. Now it’s automatic like in other cases

Third Party Projects

Alain reports

Hi, Planify 4.1 is out!

  • Planify has a new icon thanks to Tobias Bernard.
  • Quick Add is available, quickly add tasks from anywhere on your desktop with a simple keyboard shortcut, set it from Preferences → Quick Add
  • Preferences page redesigned.
  • The preferences to run in the background and run at startup are available.
  • The error that did not allow to see the calendar events was solved.
  • Added the preference to create tutorial project.
  • Bug fixes and performance improvement.

https://github.com/alainm23/planify/releases/tag/4.1 Available on Flathub

Felipe Kinoshita says

This week I release Wildcard 0.2.0, it brings a more streamlined layout and a nice dialog for quickly switching expression flags on/off, check it out! :D

Tagger

An easy-to-use music tag (metadata) editor.

Nick reports

Tagger is currently at V2023.7.0-beta2 !

After months of no updates, we are proud to bring Tagger back and better than ever. Tagger has received the C# treatment and has been completely rewritten from the ground up with a more stable and performant backend. To signify this great release, we even have a new icon thanks to @daudix-UFO ! Besides being written in C#, we also added some new features and fixed pre-existing bugs that you can read about below. Tagger is also now available to translate on Weblate !

We urge everyone who uses Tagger / wants to use Tagger to help us test the beta releases so we can iron out any issues before our stable release this Sunday. We hope you enjoy this release as much as we enjoyed making it :)

Here’s the full changelog:

  • Tagger has been completely rewritten in C#! With this new language comes better performance and more stable features. To signify this great change, we also updated the app icon!
  • Added a separate option in Preferences for overwriting album art with MusicBrainz metadata independently of the overwriting tag data setting
  • Added an option in Preferences for controlling how music files are sorted
  • Track values will now be padded into double digits
  • Fixed an issue where some file types were not loading album art correctly
  • Fixed an issue where applying unapplied changes would sometimes not work
  • Improved UI/UX
  • Updated translations (Thanks everyone on Weblate!)

Parabolic

Download web video and audio.

Nick announces

Parabolic V2023.6.3 is here!

If you haven’t noticed, Tube Converter has been renamed to Parabolic! We think this is a much better name for the downloader and thank all that have worked with us to come up with a new name :)

Here’s the full changelog:

  • Tube Converter has been renamed. Introducing, Parabolic!
  • Fixed a large memory leak caused by not disposing logs correctly
  • Updated translations (Thanks everyone on Weblate!)

IPlan

Your plan for improving personal life and workflow

Iman Salmani announces

IPlan 1.7.0 is now out! What’s changed:

  • Chart of Time spent in last 7 days
  • Record delete button moved to record row
  • Improve usability for the phone form factor
  • Option to disable run in background
  • Improve duration format
  • Code refactoring, Bug fixes, and UI improvements You can get it from Flathub.

GNOME Network Displays

Stream the desktop to Wi-Fi Display capable devices

Pedro Sader Azevedo says

GNOME Network Displays gained a DBus interface, that exposes the list of screencasting devices (also known as “sinks”) discovered in your network. This is part of the ongoing effort to allow GNOME Network Displays to function as a screencasting backend for other projects (e.g. GNOME Shell, GNOME Settings, xdg-portal, etc).

GNOME Foundation

Kristi Progri announces

GUADEC 2023 is coming and the team is working hard to make this experience as pleasant as possible. On July 31st we are orgaizing a day trip outside of Riga. If you are interested to join please register here: https://events.gnome.org/event/101/page/164-northern-latvia-trip. We have extended the deadline for everyone who couldn’t make it on time.

On another note, GNOME ASIA is taking place in Katmandhu, Nepal from 1st-3rd of December. We are starting to put together the plans and conference timeline, if you would be interested to join the organising team please send us an email at: [email protected]

That’s all for this week!

See you next week, and be sure to stop by #thisweek:gnome.org with updates on your own projects!

Life update

If you are looking for a Software Engineer with strong skills in C, C++, Rust and JavaScript, with a strong FLOSS background, I am available to hire.

My résumé

Give your SVGs light/dark style support

I’ve recently been contributing to Flathub and updating my own website a bit, and I found myself wanting to recolor SVGs for light and dark style support. Let’s look at some examples:

Inline SVG

When using an inline SVG (including it directly in the DOM), I still recommend using your site or page’s CSS to style your icons.

An SVG icon should look something like this, with the actual path data in the d attribute:

<svg>
  <path d="…" />
</svg>

You’ll want to make sure the SVG file doesn’t have its own styling, as that can override your page’s styles depending on specificity—it follows the “cascading” part of CSS, after all. Then just treat it like any other part of your page, remembering that paths use the fill property (instead of the color or background as you might expect).

.example svg path {
  fill: #f00;
}

Inline SVGs can inherit styles like any other element, so I always recommend specifying both color and fill anywhere you want to set the foreground color on your site, so your icons pick up the text color automatically. So for your CSS:

svg path {
  fill: inherit;
}

.example-1 {
  color: red;
  fill: red;
}

And then on your page:

<div class="example-1">
  <svg>
    <path d="…" />
  </svg>
  Some text
</div>

Result:

Some text

And as you might expect, the inline SVG will follow your page or site’s media queries for dark style support:

.example-2 {
  color: maroon;
  fill: maroon;
}

@media (prefers-color-scheme: dark) {
  .example-2 {
    color: pink;
    fill: pink;
  }
}

Result (go ahead and try toggling your OS/browser’s dark style preference!):

Some text

This is how I use SVGs if I can, as you get more flexibility over the styling, and can do things like changing the color on hover (e.g. for SVGs in a link). That all makes sense because the SVG is directly in the DOM and thus can be treated like any other element. But sometimes you aren’t including the SVG inline—what then?

<img /> or CSS background image

When you use an SVG in an <img /> tag or as a background image in CSS, it doesn’t follow the CSS rules given to DOM elements—the SVG and thus its included <path>s are not actually in the DOM. In this case, if you control the SVG file I recommend using a <style> element in the SVG itself! This is where it gets really cool: you can use media queries directly in the SVG:

<svg>
  <path d="…" />
  <style>
    path { fill: #273445 }

    @media (prefers-color-scheme: dark) {
      path { fill: #fafafa }
    }
  </style>
</svg>

Result (toggle your browser’s dark style):

Destiny icon

This does not allow you to dynamically recolor the SVG from your site or page’s CSS, so it has its limitations. But for brand icons where you want a light and a dark version, I love it.

Favicons

I discovered something thanks to my friend, GNOME designer, and Flathub brand designer Jakub Steiner: you can use a prefers-color-scheme media query directly in an SVG, and browsers will Do the Right Thing™! We are using this on the new Flathub website; go ahead and toggle your OS’s dark style preference and watch the favicon in your browser’s UI.

If you use this, make sure you provide a fallback PNG that works on both light and dark backgrounds, as not all browsers support SVG favicons; check Can I Use… for the details.

CSS filters

Another option you have when using SVGs (or really, any images) on the page is to use CSS filters in light/dark style. Depending on how you use this, it can feel a bit more hack-ish, especially if you start manipulating colors—but it works. I used this on my Star Wars: Andor page for the logo:

@media (prefers-color-scheme: light) {
  .example-4 img {
    filter: 
      brightness(0) 
      invert(14%) 
      sepia(43%) 
      saturate(4969%) 
      hue-rotate(343deg) 
      brightness(89%) 
      contrast(107%);
  }
}

As you can see, the resulting filters are really messy, and it’s impossible to tell what it’s going to look like by reading it, which I don’t love. You can use an online color to CSS filter converter to help compute the filters, as well.

Result (toggle your browser’s dark style):

Andor logo

Thanks for reading!

I hope this helps someone out there on the Internet, even if it’s just me the next time I go to manipulate SVGs. :)

June 29, 2023

2023-06-29 Thursday

  • Early morning accessibility call - good stuff, sales call. Interviews with PM candidates, partner call; mail chew. Interviews into cell group + dinner combination until late.

PDF and embedded videos

PDF supports playing back video content since version 1.5. I could do the whole shtick and shpiel routine of "surely this is working technology as the specification is over 20 years old by now". But you already know that it is not the case. Probably you came here just to see how badly broken things are. So here you go:

Time specification (or doing pointlessly verbose nested dictionaries before it was cool)

A video in PDF is a screen annotation. It is defined with a PDF object dicrionary that houses many a different subdictionaries as keys. In addition to playing back the entire video you can also specify to play back only a subsection of it. To do the latter you need an annotation dictionary with an action subdictionary of subtype rendition, which has a subdictionary of type rendition (no, that is not a typo), which has a subdictionary of a mediasource, which has three subdictionaries: a MediaClip and the start and end times.

There are three different ways of specifying the start and end times: time (in seconds), frames or bookmarks. We'll ignore the last one and look at frames first. As most people are probably not familiar with the PDF dictionary syntax, I'm going to write those out in JSON. The actual content is the same, it is just written in a different syntax. Here's what it looks like:

{
  "B": {
    "S": "F",
    "F": 1000
  }
}

Here we define the start time "B", which is a MediaOffset. It has a subtype that uses frames ("S" is "F") and finally we have the frame number of 1000. Thus you might expect that specifying the time in seconds would mean changing the value of S to T and the key–value pair "F" -> 1000 to something like "V" -> 33.4. Deep in your heart you know that not to be true, because the required format is this.

{
  "B": {
    "S": "T",
    "T": {
      "S": "S",
      "V": 33.4
    }
  }
}

Expressing the time in seconds requires an entire whole new dictionary just to hold the number. The specification says that the value of that dictionary's "S" must always be "S". Kinda makes you wonder why they chose to do this. Probably to make it possible to add other time formats in the future. But in the 20 years since this specification was written no such functionality has appeared. Even if it did, you could easily support it by adding a new value type in the outer dictionary (something like "T2" in addition to "T").

Most people probably have heard the recommendation of not doing overly general solutions to problems you don't have yet. This is one of the reasons why.

Does it work in practice?

No. Here's what happens when I tried using a plain h264 MP4 file (which is listed as a supported format on Adobe's site and which plays back in Windows and macOS using the system's native video player).

Okular


Instead of a video, Okular displays a screenshot of the user desktop with additional rendering glitches. Those are probably courtesy of bugs in either DRM code or the graphic card's device driver.

Evince

Evince shows the first frame of the video and even a player GUI, but clicking on it does nothing and the video can't be played.

Apple Preview

Displays nothing. Shows no error message.

Firefox

Displays nothing. Shows no error message.

Chromium

Displays nothing. Shows no error message.

Acrobat Reader, macOS

Plays back videos correctly but only if you play it back in its entirety. If you specify a subrange it instead prints an error message saying that the video can't be presented "in the way the author intended" and does nothing.

Acrobat Reader, Windows

Video playback works. Even the subrange file that the macOS version failed on plays and the playback is correctly limited to the time span specified.

But!

Video playback is always wonky. Either the video is replaced with a line visualisation of the audio track or the player displays a black screen until the video stream hits a key frame and plays correctly after that.

In conclusion

Out of the major PDF implementations, 100% are broken in one way or another.

June 26, 2023

(Almost) Bi-weekly GSoC Update: FlatSync GSettings Integration

This post is going to cover the latest progress on FlatSync as well as my absence from bi-weekly updates and the project.

# Absence From Blog and Project

Normally, I'd write bi-weekly blog post updates regarding GSoC and FlatSync, but both my mentor and I have been very busy with university work in this month's first week, and after that, I've been ill and bed-ridden for a little over a week. As a result, development stagnated in this period, so there was just nothing to write about. I'm somewhat back on my feet though, so you can expect regular updates again.

# Progress on FlatSync

We were thrown behind quite a bit on schedule, but luckily, the still-open, expired milestone we wanted to catch up on turned out to be less work than originally planned, and we were able to fulfill it quite easily.

# GSettings Integration

Initially, we used the OS's keyring to not only store our GitHub Gist API key but also the Gist ID as an attribute attached to the secret, as this was the most straightforward solution. However, referencing the ID later on in the project was a bit messy and involved hard-to-understand code and copy&paste (not to mention that this felt a bit like an abuse for the keyring). We planned to, later on, provide more customization to the user, like whether or not to only push changes automatically and pull them in manually, whether or not to split installed Flatpak packages into hosts or to unify them into one list, and maybe even more in the future. To accomplish our goals, we settled on implementing GSettings into our project for our configuration values. We initially planned a lot of time for this milestone as we didn't know how many configuration variables we had to provide at this stage, and how we would implement storing configuration variables. Luckily, using GSettings was very straightforward, also thanks to my mentor's previous experience regarding GSettings.

We now changed over to using GSettings to save our Gist ID, and we added the possibility to initialize FlatSync with a pre-defined Gist ID. That way, if you delete the secret and/or the Gist ID, you can re-initialize FlatSync without it creating a new Gist. This will be especially useful later on when installing FlatSync on a new or freshly installed device.

# Outro

This about wraps up the current status of the project.

Next up, we are looking into actually synchronizing the list instead of only pushing it into the Gist. This means comparing the installed packages in the Gist to the ones installed on the device, and adding and/or removing packages when needed. This will need some thought, as we want to avoid (un-)installing packages by mistake (e.g. you remove some package on device 1, boot up device 2 which posts a list of all installed packages into the Gist, including the previously removed one, and it ends up getting reinstalled on device 1 where it should have been removed from device 2 in the first place).

June 21, 2023

Evolving accessibility

Our last post on accessibility in GTK4 was a while ago, time for an update.

Thankfully, we have Lukáš Tyrychtr at Red Hat working on accessibility now.

Accessibility – How does it work?

One of the bigger changes in accessibility from GTK3 to GTK4 was that we have a new application API that is modeled on the ARIA specs from the web.

The high-level picture to keep in mind is

app → ARIA → AT-SPI → accessibility bus → AT

AT stands for accessibility technology here. In practice, that mainly means orca, the well-known screen reader (although there is a new contender, with the odilia project).

The new layer provides APIs such as

void gtk_accessible_update_state (GtkAccessible *self,
                                  GtkAccessibleState first_state,
                                  ...)

that let apps set accessible attributes of their widgets.

ARIA layer improvements

Since we’ve introduced it in GTK4, the application API layer has only seen moderate changes. That has started to change over the last 9 months or so, since Lukáš is working on it.

One thing that he has started to do is adding public interfaces so that third-party widgets (such as libadwaita) can provide full accessibility support. The first such interface is GtkAccessibleRange (new in 4.10), for range widgets like GtkScale. We are looking at adding more, notably a text interface. It will be needed to make terminals accessible.

In the 4.12 cycle, we have done some work to make our implementation match the ARIA specs more closely. This involved changing the roles of some widgets: our default role is now ‘generic’, and toplevel windows use the ‘application’ role. We’ve also redone the way accessible names and descriptions (i.e. the things you hear orca read) are computed, to match the spec.

Another improvement is that most of our widgets now have the necessary labels and relations to make orca read them. If you find that there are still things missing, please let us know!

AT-SPI translation improvements

It is no secret that we would like to see some modernization of the AT-SPI D-Bus APIs. But for now, it is what we have to work with. Our translation layer works by lazily putting just the objects on the bus that ATs have asked for, to avoid creating too much bus traffic.

One of the recent improvements in our translation is that we are now using GtkAccessibleRange, so third-party range widgets can be accessible.

We have also fixed problems with the selection implementations for GtkNotebook and GtkStackSwitcher, so ATs can now change the selected tab in notebooks and stacks.

Tools

All of this is nice to hear, but if you are an app developer, you may want to know how you can find and fix accessibility problems in your app.

It is very instructive to just turn on the Screen Reader and see what it says as you navigate through your app. But we also have some tools to help you evaluate the accessibility support of your app.

The GTK inspector has a page that shows accessibility information:

The accessibility tab in the GTK inspectorIt recently was improved to show not just the properties, states and relations that are set on each widgets, but also the name and description that GTK computes and passes to ATs 
- that is the text that orca reads.

Another tool in the inspector is brand new: the accessibility overlay shows warnings and recommendations that are based on the ARIA authoring guidelines.

It looks like this:

A window showing warnings from the Accessibility overlay, and the inspector window with the switch to enable the overlay.It is not perfect, but it should give some quick insights on where you can improve accessibility.

Summary

GTK 4.12 will have better out-of-the-box accessibility and new tools to help you make your app accessible.

Enjoy! ❤

June 17, 2023

Developer Experience

As part of my internship with the open-source project "Workbench", I've been developing demos showing off new GTK4 and Libadwaita widgets. Workbench is unique in a way, as it is an application aimed at developers. So developer experience is of utmost importance.

My fellow interns and I are not just writing code; we're crafting experiences for other developers. The demos we're creating will be thoroughly examined by everyone who uses Workbench, and our contributions are transparent and open to scrutiny by anyone who browses the Workbench Library. The code within these demos isn't just codeit's a lifeline for someone's first GTK project, a guide to help them navigate unfamiliar waters. I know this because before I was helping develop Workbench, Workbench was helping me develop my first GTK app.

While making these demos there are so many small little things that are so important to get right, we're curating a collection of tools and examples to hopefully grow the GNOME Ecosystem. Thankfully we're working on a style guide just to ensure the quality of our work.

In its current state Workbench is an amazing tool to have for anyone interested in developing for the Linux desktop, and I hope our continued focus on developer experience will bring more people to the platform.

June 16, 2023

Berlin Mobile Hackfest

Last week we had a small hackfest in Berlin, with a focus on making GNOME work better on mobile across the stack. We had about 15 people from various projects and backgrounds attending, including app developers, downstreams, hardware enablement, UX design, and more. In addition to hacking and planning sessions we also had some social events, and it was an opportunity to meet nice people from different corners of the wider community :)

The Event(s)

The postmarketOS gang had their own hackfest over the weekend before ours, and on Monday we had the Cultivation Space venue for some chill random hacking. I wasn’t there for most of this, so I’ll let others report on it.

The main hackfest days were Tuesday and Wednesday, where we had an actual unconference agenda and schedule. We met at 11:30 in the morning at Cultivation Space and then managed to get through a surprisingly large list of topics very efficiently on both days. Kudos to Sonny for coming up with the format and moderating the schedule.

Late night podcast recording on Wednesday (note the microphone in the bottle in the center)

systemd and Image-Based OSes

On Tuesday Lennart from systemd stopped by for a bit, so we had a session with him and the postmarketOS people discussing integration issues, and a larger session about various approaches for how to do image-based OSes. It’s clear that this direction is the future — literally everyone is either doing it (Endless, Silverblue, GNOME OS, SUSE, Carbon OS, Vanilla OS) or wants to do it (PureOS, postmarketOS, elementaryOS). There are lots of different approaches being tried and nobody has the perfect formula for it yet, so it’s an exciting space to watch.

Edit: Clarification from postmarketOS: “We are looking into doing a version of postmarketOS that is image-based/has an immutable file system. But it will still be possible to use postmarketOS without it.”

I personally only half followed some of this while working on other stuff, but it was definitely interesting to hear people discuss the tradeoffs of the various approaches. For example, OSTree doesn’t support offline security which is why Lennart wants A/B/C partitions instead.

OS Stack Bake-Off

After this we had a big OS stack show and tell with people from Debian, GNOME OS, and postmarketOS. From a development/app platform point of view what we need is an OS with:

  • very recent kernel/hardware enablement (because most phones are still in the process of getting mainlined)
  • large, active community around the OS, including a security team
  • image-based, with Flatpak apps
  • up to date middleware (systemd, flatpak, portals, etc.)
  • release cycle in sync with GNOME’s
  • nightly images for testing/dogfooding
  • aligned with GNOME in terms of design philosophy (upstream first, design first, suitably scared of preferences, etc.)

Currently there isn’t an obvious best option, unfortunately. Debian/Mobian has a large community and recent kernels, but its stable version doesn’t have most of the other things we need since its release cycle is way too slow and out of sync with GNOME’s (and using testing/unstable has other problems). postmarketOS is doing lots of great work on hardware enablement and has a lot of what we want, but the lack of systemd makes it impractical (especially once we implement things like homed encryption). GNOME OS nominally has most of what we want, but the community around it is still too small and not enough people are using it full-time.

We’ll see what the future holds in this regard, but I’m hopeful that one or more options will tick all the boxes in the future!

Shell History and Discussion

Another point on the agenda was longer-term planning for the shell. The Phosh/GNOME Shell question has been unaddressed for years, and we keep investing tons of time and energy into maintaining two shells that implement the exact same design.

Unfortunately we didn’t have enough people from the Phosh side at the hackfest to do any actual planning, but we did discuss the history and current state of both projects. People also discussed next steps on the GNOME Shell mobile side, primarily making it clearer that the branch is maintained even though it’s not fully upstream yet, giving it a proper README, having an issue tracker, and so on.

MPRIS Controls

Oliver was interested in MPRIS buttons, so we had a design session about it. The problem here is that different apps need different buttons in their MPRIS widget depending on the use case (e.g. podcast apps need jump buttons, pause doesn’t make much sense for a live stream, etc.). Currently we have no way for apps to tell the system what buttons it needs, so we discussed how this could be done. The results of our session are written up in this issue, and a vague consensus that this could be done as an extension of the existing MPRIS protocol.

To address the short-term issue of not having jump buttons for podcasts, Jonas statically added them to the mobile shell branch for now.

File Opening

Pablo had encountered some oddities with the file opening pattern in his work on Evince, so we took a look at the question more broadly. Turns out we have a lot of different patterns for this, and almost every app behaves slightly differently! We didn’t get as far as defining if/how this could be standardized, but at least we wrote up the status quo and current use cases in apps in an issue.

Flathub

Bart and Kolja from Flathub also joined on Monday and Tuesday, so we worked on the Flathub website a bit (primarily the app tile layout), and discussed a new initiative to make it possible to show the best apps on the home page. The current idea for this is to start off with a new toolbar on Flathub app pages when you’re logged in as a moderator, that allows a mods to set additional metadata on apps. We’d pair this with an outreach campaign to raise the bar on app metadata, and perhaps automated linting for some of the easier to check things.

Bart and Kolja also worked on finally getting the libappstream port done, so that we can have device support metadata, image captions, etc. from Flathub. They made a lot of good progress but this is still not quite done, because there’s lots of edge cases where the new system doesn’t work with some existing app metadata.

Kolja and Bart hacking on Flathub

Mobile Platform Roadmap

A few of us sat down and worked on a priority list of things we’d need in GNOME mobile to be able to realistically daily drive it, including platform, shell, and app features. We documented the results in this issue.

Various other things that happened but that I didn’t follow closely:

  • Pablo and Sonny sat down together and looked over the postmarketOS GNOME session, doing a bit of QA and finding areas where it could be closer to the upstream GNOME vision. Lots of nice improvements since we last looked at it a few months back, and lots more to come :)
  • There was a session about device wakeup APIs. I didn’t attend it, but it sounded like they made progress towards a portal for this so that e.g. alarms can wake up the device when it’s suspended or powered off.
  • Robert, Dorota, Caleb, and others had several in-depth sessions on camera hardware enablement, libcamera and the like
  • Julian got GNOME OS running on the Librem 5 with a recent kernel
GNOME OS running on the Librem 5 for the first time

Overall I found the event very productive and a lot of fun, so thanks everyone for making it a success. In particular, thanks to the GNOME Foundation for sponsoring, Elio Qoshi and Cultivation Space for hosting us, and Sonny for co-organizing. See you next time!

Cambalache 0.12.0 Released!

I am pleased to announce a new release of Cambalache a new RAD tool for Gtk 3 and 4!

Version 0.12.0 packs a year’s worth of new features and lots of improvements and bugfixes.

Workspace CSS support

Does your application use custom CSS? Now you can see CSS changes live in the workspace. All you need to do is add a new CSS file in the project and specify if you want it to be global or for one UI file in particular.

Keep in mind that GtkBuilder does not support CSS inline so your application still needs to load the CSS at runtime.

GtkBuildable Custom Tags

In an ideal world object properties would be enough to define your application UI but in reality it is not, so Gtk Widgets can define customs tags in their GtkBuildable implementation such as GtkWidget <style> or GtkComboBoxText <items>

This feature was missing in Cambalache, mostly because I do not want to add custom code to handle each case one by one, as you can imagine this would become really hard to keep up and maintain.

So I decided on a generic component that relies on minimum metadata to create the UI automatically. So let me know if your favorite Class is missing a custom tag!

Property Bindings

Another important feature missing is setting up properties binding directly in the property. While technically it was always possible to create a GBinding object, clicking on a property label and selecting the source object and property is much easier.

User Templates

Back in the day one of the main features of Glade 3 was being able to extend the widgets available to use without having to modify the source code. This was done by creating a XML widget catalog unfortunately this is a high bar for every application as it required not only writing an XML file describing all the metadata but also having a library to load the new types.

Having all of your application’s UI files in the same project allows exposing all the templates for use without any extra steps, all you need to do is have all your templates types in the same project and you will be able to use them immediately.

The only current limitation is that you wont be able to use any custom properties since they are defined in code and Cambalache does not have access to the GType. For that we still going to need to create a catalog file similar to Glade.

XML Fragments

Have you ever been in the position where you are forced to write the whole UI XML file by hand just because the shiny new (or old) feature you need is not supported?

Fear no more!! Custom XML fragments comes to the rescue!

This new feature sounds fancy but it just a way to manually define any XML fragment that you want to append to an object or UI even if it is not valid GtkBuilder.

This should be handy in lots of different situation from defining <menu> objects to setting custom properties in your customs widgets!

External object references

GtkBuilder lets you expose external objects to be reference inside the XML description, but Cambalache does not export broken references, instead it will let you declare which external object you will expose in your application using a special type “(external)” that only let you define its id.

Where to get it?

You can download the flatpak bundle from flathub

flatpak install --user flathub ar.xjuan.Cambalache

Run this first if you do not have flathub repo in your host

flatpak remote-add --user --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

You can also get the source code from gitlab (see README.md for instructions)

git clone https://gitlab.gnome.org/jpu/cambalache.git

BTW I will be making point releases as soon as translations are updated!

enjoy!

 

Juan Pablo

Carbon emissions analysis of GUADEC 2022

I’ve just finished estimating the carbon emissions from GUADEC 2022, and have a few interesting highlights from the report. The report is based on data collected from the streaming servers, registration data, and the post-conference survey.

  • Having an online component to the conference increased the audience by a factor of 10: there were around 120 in-person attendees in Mexico, but there were an average of 1300 people using Big Blue Button.
  • The carbon emissions from providing the remote infrastructure were around 2.8tCO2e, or about 2kgCO2e per remote attendee.
  • Having a remote attendance party in Berlin allowed around one tenth of the attendees to attend with a factor of 10 lower transport emissions than those who attended in-person in Mexico. The average transport emissions for those who went to Berlin were 88kgCO2e, whereas they were around 1tCO2e for those going to Mexico.
  • For context, the annual emissions per person can be at most 2.3tCO2e by 2030 in order to limit global warming to 1.5C by the end of the century. That covers all food, travel, heating, purchases, etc. So travel to Mexico was 40% of the average attendee’s annual target.
  • Half of the in-person attendees travelled from within Mexico, which will have skewed the mean transport emissions downwards. The distribution is more likely bimodal between around 50kgCO2e for locals and more like 2-3tCO2e for those coming from outside Mexico.
  • As I wrote at the time, the remote attendance party was fun to attend and felt like it worked as a way to attend the conference (there were no A/V problems, we had some nice local socials, etc.). I would do it again.
  • The post-conference survey had a low response rate of about 19% of registered attendees, which made some of this analysis hard. Please always fill in the post-conference survey! It doesn’t take long, and aside from any of this analysis, it helps the organisers plan conferences better in future.
  • Many thanks to Kristi, Caroline and Bartłomiej for collecting the data needed for this analysis.
  • If anyone spots problems in the analysis please say! This is not the kind of thing I practice doing very often and I’ve had to make a number of assumptions.

Other conferences

A post-conference survey has been done for other big GNOME events, such as GNOME.Asia and LAS. Would anyone be interested in doing a similar analysis for those events? Perhaps we can get a semi-automated pipeline in place for it.

Getting the best of tablet pads

In case they needed introduction, pads are these collections of buttons and tactile sensors (ring or strip shaped) most typically found along the side of drawing tablets. These devices will be the topic of today.

Picture of someone using a pad
The Wacom ExpressKey Remote is a rare case of disembodied pad. Picture from Wacom.

A bit of context

The concept behind pads is simple, a collection of triggers with no specific action associated, so users can configure the actions that best suit their tastes and setups. Other quality differences like the number of pressure levels aside, entry-level consumer tablets often have few buttons, more advanced tablets typically get more buttons, and “modes” that multiply the amount of available mappable actions. It you are a pro, it is likely that you want quick access to more features, and so on.

When this kind of devices came around back in many years ago, and support for them was added in X11, they sat on a rough spot. As a X11 input driver, you either tell the X server that you are adding a device with buttons and valuators that drives the pointer sprite (e.g. a mouse), or you are adding a device with keycodes and levels (e.g. keyboards) subject to keymap translations. A pad device is notably neither of those things, the way to shove the square peg on the round hole was to make them pointers, thus they would send button and scroll events directed towards the pointer position, they would just not move the pointer.

We are here also talking about a time that applications had (many still do) a hard time recognizing input from different source devices, to them a button press is just a button press, so those pads as-is would just click and scroll on things, quite far from the customizable actions promised land. The pragmatic approach taken in GNOME (Shell now, settings-daemon back in the day) was grabbing those devices at the session level, not letting them be seen by applications and converting them into something immediately useful. Given the limit of choices, these pad button and scroll events got converted to keycombos.

This at least kept two promises, it’s a) customizable and b) universal. Although it felt like falling short for long, since keycombos have some implicit problems associated:

  1. A small set of keycombos is close to universal, but many are not. So the typical choice is either universal but seldom used settings, or highly app-specific keycombos that are latently bollocks in any other application.
  2. Keycombos are also subject to developer/translator changes, etc. You might find your keycombos no longer hold if you change locale, or environment, or update your app.

But along came Wayland! Pads would not need to be subject to the limitations of X server driver API and could become their own entity with their own semantics, and indeed they became a first class citizen of the Wayland tablet protocol available in wayland-protocols. This allowed us to let pad events be sent to clients, have it be either correctly interpreted in clients/toolkits as input from a pad device or safely ignored, instead of misinterpreted as another kind of input. The Wayland protocol opened even further the possibilities like being able to provide readable strings for compositor feedback. I blogged about this… in 2016.

What since?

A drawback from the previous state of things and the massive change of approach is that no client was prepared to be in charge of pad events, so any integral support for pads had to start from the very bottom. But also, pads are most useful in very specific setups and applications, that at the time were nowhere near having Wayland support. In the GTK world, the requisites for it to work were:

  1. Using GTK >= 3.22
  2. Using GAction for application actions
  3. Using GtkPadController to hook actions

While that looks simple enough, the flagship applications were barely taking bits at the first step at the time, and after that, the second was not a small undertaking either with applications with hundreds of options.

Some weeks ago, I was fortunate to attend the Wilber Week at Amsterdam, and was very glad to meet my GIMP friends again (and Inkscape/Blender folks!) despite my week-long stint with a commuter’s life after so long.

Picture of a baked Wilber.
A moment I missed, immortalized at a splash screen.

 

Over there, I learned that not long ago, GIMP did finally port away from GtkAction into GAction, finally lifting the last barrier to get pad actions working as envisioned, in one app at least :).

Since this is something I’ve been touting for so long to GIMP maintainers, of course I had to volunteer for the task, now up in a merge request.

Of course, most of the gist there is the configuration UI, and the configuration serialization/deserialization, the conversion to actions is still done using GtkPadController created from the configuration.

Screenshot of GIMP pad configuration
While the UI could get more polish by using libwacom (e.g. nicer names, or pre-populating buttons/rings/strips in the available modes), it also needs to work with tablets not recognized by libwacom, or (maybe someday) in other platforms. The important part is that the application may define the action associated with a pad feature, and give it a friendly name. These do also show in the GNOME Shell pad OSD:

Screenshot of the pad OSD

Which in this case works as a cheat sheet of sorts, since there are pads with extreme combinations of buttons/modes (e.g. the Express Key Remote has 17 buttons by 3 modes), and per-app mapping does not make things easier to remember on itself.

Conclusion

An improved form of pad support has been stalled for too long, and it is amazing that it can already start to roll out. It looks like Inkscape is also ripe for such improvements, and during Wilber Week I was glad to hear a very positive attitude towards Wayland from Blender developers. Maybe the tides are turning to make this often neglected device shine?

A call to GNOME app developers

While these flagship applications are key for a major leap in support for pad devices, perhaps your application can do a little bit to help, if you see your app as possible part of a designer/artist/etc workflow. Using GtkPadController is rather easy with a fixed set of actions, so exposing an small set of useful actions could extend the usefulness of pad devices even further. This is for example what Nautilus does. You can also draw inspiration from the “Paint” GTK demo.

Status update, 16/06/2023

This blog is about what’s on my mind, and recently I’ve been thinking about aeroplane flights, and limiting how many I take every year.

I’m not trying to make anyone feel impressed or ashamed here, or create rules that I insist everyone needs to live by. And I am no expert in climate science, but we need to start talking about this stuff more – as you probably spotted, the climate is uncomfortably hot now, caused mainly by all the “greenhouse gases” that we keep producing in our industrious human civilization.

Commercial aviation accounts for 3.5% of global warming today, according to the 3rd result in a web search I just did. So planes aren’t the biggest issue – the biggest issue is a massive methane leak from an oil well in Turkmenistan, and I guess the best contribution I could make to avoid further heating would be to plug up that hole somehow. From here though, it seems more realistic for me to look at reducing how much money I give to airlines and oil companies.

Plane spotting

In 2020 I didn’t go anywhere due to the COVID pandemic. so lets start with 2021.

  • 2021: 3 flights, 6 hours total. (Including work stuff, family visits for Christmas).
  • 2022: 10 flights, 25 hours total (Including work stuff, family visits, Linux App Summit, OSSEU, the Audio Developer Conference, and an incredible climbing trip in Canarias)
  • 2023: So far, 6 flights, 6 hours total (the usual, plus FOSDEM and an incredible bike holiday in Wales)

My current goal is that 2023 will have no more flights than 2022. I’ve started tracking them in a spreadsheet to make me think harder about avoiding them. It’s easy to get excited about a holiday, conference, concert etc. and forget about the broader goal of not taking flights all the time. It’s also very hard to avoid them all – I’m sure I will take a flight to GUADEC in Latvia next month, which is basically as far away from me as you can get without leaving Europe. (I am hoping I can take a boat to the UK afterwards though, let’s see).

Everybody’s circumstances are different of course. I chose to live 2000km from my family, which implies extra travel. Some folk live 20,000km from their family, which implies even more. There’s no point judging people for how their life turned out. (Although it is good to judge billionaires and politicians flying everywhere in private planes – fuck those guys!!!).

My last trip from the UK supposedly avoided 4% of emissions of one passenger by using “sustainable fuel”, so that’s about 0.03% of the total emissions, and they emailed me this nice certificate.

"Emission Reduction Certificate" - Avikor certifies the supply of 0.67 litres of sustainable fuel.

The aviation industry claim that flying is actually great anyway, soon aeroplanes will be powered solely by wildflowers and insect wings, etc. etc. This is exactly what a business that sells plane tickets would say, so I am skeptical, and there is plenty of evidence of “greenwashing” such as this recent “Science Based Target Initiative“. Ultimately its hard for airline managers to turn down a fat targets-related bonus just to try and help the future of civilization … a sort of “prisoners dilemma”.

Rambling about trains

Many people assume you can take a boat between the UK and Spain, and you can do but the boat takes over 25 hours and leaves you in either Santander or Portsmouth, neither of which are very well-connected.

High speed trains are a better option, and they already exist for most of the trip: Santiago – Barcelona is less than 8 hours (not bad to travel 1000km); and Paris -> Manchester is also only 6 hrs door to door (to go 800km). The missing piece for me is Barcelona to Paris.

There’s no sleeper train option between Spain to France, since the Spanish government decided to remove it in 2012. During the day there about 2 trains, which take 6 hours each, unfortunately 8 hrs + 6 hrs + 6 hrs makes 20 hrs means you can’t do this trip in a single day. You need an expensive hotel either in Barcelona or Paris to make the trip in addition to that expensive express train to Paris. So I’ve only taken this route once so far, and it involved a few days in the French Pyrenees. (As an aside, that would be an option for every trip if it wasn’t for the restrictions of my job! In fact most of these issues would be more tractable if I could work less days a week or just take days offline sometimes.)

The French government have seen sense already and run excellent night trains from the border up to Paris, but the Spanish government have not seen any sense at all, and the connections from Barcelona are hopeless. There is some hope though, from the European Sleeper cooperative who are working on a Barcelona -> Brussels night train, due some time in 2024/2025. That could make the Santiago -> Manchester trip feasible overland in around 24 hours – which would be incredible!! They are running a share offer at the moment, if you find that sort of thing interesting.

So that’s what’s on my mind as the hot summer weather picks up across Europe.

As I said above – I’m not posting this in order to feel superior to folk who travel more by aeroplane, nor to show off about being part of the lucky 20% of people who are able to travel by aeroplane at all.

But I would like to see more people talking openly about the future and the climate crisis. When you have a problem, usually the hardest thing is to talk about it. It requires you to admit that there is a real problem which is not much fun. A lot of oil companies would prefer that we stay in denial. (They are happy making record profits!). Stay safe out in the heat.

June 15, 2023

parallel futures in mobile application development

Good morning, hackers. Today I'd like to pick up my series on mobile application development. To recap, we looked at:

  • Ionic/Capacitor, which makes mobile app development more like web app development;

  • React Native, a flavor of React that renders to platform-native UI components rather than the Web, with ahead-of-time compilation of JavaScript;

  • NativeScript, which exposes all platform capabilities directly to JavaScript and lets users layer their preferred framework on top;

  • Flutter, which bypasses the platform's native UI components to render directly using the GPU, and uses Dart instead of JavaScript/TypeScript; and

  • Ark, which is Flutter-like in its rendering, but programmed via a dialect of TypeScript, with its own multi-tier compilation and distribution pipeline.

Taking a step back, with the exception of Ark which has a special relationship to HarmonyOS and Huawei, these frameworks are all layers on top of what is provided by Android or iOS. Why would you do that? Presumably there are benefits to these interstitial layers; what are they?

Probably the most basic answer is that an app framework layer offers the promise of abstracting over the different platforms. This way you can just have one mobile application development team instead of two or more. In practice you still need to test on iOS and Android at least, but this is cheaper than having fully separate Android and iOS teams.

Given that we are abstracting over platforms, it is natural also to abandon platform-specific languages like Swift or Kotlin. This is the moment in the strategic planning process that unleashes chaos: there is a fundamental element of randomness and risk when choosing a programming language and its community. Languages exist on a hype and adoption cycle; ideally you want to catch one on its way up, and you want it to remain popular over the life of your platform (10 years or so). This is not an easy thing to do and it's quite possible to bet on the wrong horse. However the communities around popular languages also bring their own risks, in that they have fashions that change over time, and you might have to adapt your platform to the language as fashions come and go, whether or not these fashions actually make better apps.

Choosing JavaScript as your language places more emphasis on the benefits of popularity, and is in turn a promise to adapt to ongoing fads. Choosing a more niche language like Dart places more emphasis on predictability of where the language will go, and ability to shape the language's future; Flutter is a big fish in a small pond.

There are other language choices, though; if you are building your own thing, you can choose any direction you like. What if you used Rust? What if you doubled down on WebAssembly, somehow? In some ways we'll never know unless we go down one of these paths; one has to pick a direction and stick to it for long enough to ship something, and endless tergiversations on such basic questions as language are not helpful. But in the early phases of platform design, all is open, and it would be prudent to spend some time thinking about what it might look like in one of these alternate worlds. In that spirit, let us explore these futures to see how they might be.

alternate world: rust

The arc of history bends away from C and C++ and towards Rust. Given that a mobile development platform has to have some low-level code, there are arguments in favor of writing it in Rust already instead of choosing to migrate in the future.

One advantage of Rust is that programs written in it generally have fewer memory-safety bugs than their C and C++ counterparts, which is important in the context of smart phones that handle untrusted third-party data and programs, i.e., web sites.

Also, Rust makes it easy to write parallel programs. For the same implementation effort, we can expect Rust programs to make more efficient use of the hardware than C++ programs.

And relative to JavaScript et al, Rust also has the advantage of predictable performance: it requires quite a good ahead-of-time compiler, but no adaptive optimization at run-time.

These observations are just conversation-starters, though, and when it comes to imagining what a real mobile device would look like with a Rust application development framework, things get more complicated. Firstly, there is the approach to UI: how do you get pixels on the screen and events from the user? The three general solutions are to use a web browser engine, to use platform-native widgets, or to build everything in Rust using low-level graphics primitives.

The first approach is taken by the Tauri framework: an app is broken into two pieces, a Rust server and an HTML/JS/CSS front-end. Running a Tauri app creates a WebView in which to run the front-end, and establishes a bridge between the web client and the Rust server. In many ways the resulting system ends up looking a lot like Ionic/Capacitor, and many of the UI questions are left open to the user: what UI framework to use, all of the JavaScript programming, and so on.

Instead of using a platform's WebView library, a Rust app could instead ship a WebView. This would of course make the application binary size larger, but tighter coupling between the app and the WebView may allow you to run the UI logic from Rust itself instead of having a large JS component. Notably this would be an interesting opportunity to adopt the Servo web engine, which is itself written in Rust. Servo is a project that in many ways exists in potentia; with more investment it could become a viable alternative to Gecko, Blink, or WebKit, and whoever does the investment would then be in a position of influence in the web platform.

If we look towards the platform-native side, though there are quite a number of Rust libraries that provide wrappers to native widgets, practically all of these primarily target the desktop. Only cacao supports iOS widgets, and there is no equivalent binding for Android, so any NativeScript-like solution in Rust would require a significant amount of work.

In contrast, the ecosystem of Rust UI libraries that are implemented on top of OpenGL and other low-level graphics facilities is much more active and interesting. Probably the best recent overview of this landscape is by Raph Levien, (see the "quick tour of existing architectures" subsection). In summary, everything is still in motion and there is no established consensus as to how to approach the problem of UI development, but there are many interesting experiments in progress. With my engineer hat on, exploring these directions looks like fun. As Raph notes, some degree of exploration seems necessary as well: we will only know if a given approach is a good idea if we spend some time with it.

However if instead we consider the situation from the perspective of someone building a mobile application development framework, Rust seems more of a mid/long-term strategy than a concrete short-term option. Sure, build low-level libraries in Rust, to the extent possible, but there is no compelling-in-and-of-itself story yet that you can sell to potential UI developers, because everything is still so undecided.

Finally, let us consider the question of scripting: sometimes you need to add logic to a program at run-time. It could be because actually most of your app is dynamic and comes from the network; in that case your app is like a little virtual machine. If your app development framework is written in JavaScript, like Ionic/Capacitor, then you have a natural solution: just serve JavaScript. But if your app is written in Rust, what do you do? Waiting until the app store pushes a new version of the app to the user is not an option.

There would appear to be three common solutions to this problem. One is to use JavaScript -- that's what Servo does, for example. As a web engine, Servo doesn't have much of a choice, but the point stands. Currently Servo embeds a copy of SpiderMonkey, the JS engine from Firefox, and it does make sense for Servo to take advantage of an industrial, complete JS engine. Of course, SpiderMonkey is written in C++; if there were a JS engine written in Rust, probably Rust programmers would prefer it. Also it would be fun to write, or rather, fun to start writing; reaching the level of ECMA-262 conformance of SpiderMonkey is at least a hundred-million-dollar project. Anyway what I am saying is that I understand why Boa was started, and I wish them the many millions of dollars needed to see it through to completion.

You are not obliged to script your app via JavaScript, of course; there are many languages out there that have "extending a low-level core" as one of their core use cases. I think the mitigated success that this approach has had over the years—who embeds Python into an iPhone app?—should probably rule out this strategy as a core part of an application development framework. Still, I should mention one Rust-specific option, Rhai; the pitch is that by being Rust-specific, you get more expressive interoperation between Rhai and Rust than you would between Rust and any other dynamic language. Still, it is not a solution that I would bet on: Rhai internalizes so many Rust concepts (notably around borrowing and lifetimes) that I think you have to know Rust to write effective Rhai, and knowing both is quite rare. Anyone who writes Rhai would probably rather be writing Rust, and that's not a good equilibrium.

The third option for scripting Rust is WebAssembly. We'll get to that in a minute.

alternate world: the web of pixels

Let's return to Flutter for a moment, if you will. Like the more active Rust GUI development projects, Flutter is an all-in-one rendering framework based on low-level primitives; all it needs is Vulkan or Metal or (soon) WebGPU, and it handles the rest, layering on opinionated patterns for how to build user interfaces. It didn't arrive to this state in a day, though. To hear Eric Seidel tell the story, Flutter began as a kind of "reset" for the Web, a conscious attempt to determine from the pieces that compose the Web rendering stack, which ones enable smooth user interfaces and which ones get in the way. After taking away all of the parts they didn't need, Flutter wasn't left with much: just GPU texture layers, a low-level drawing toolkit, and the necessary bindings to input events. Of course what the application programmer sees is much more high-level, but underneath, these are the platform primitives that Flutter uses.

So, imagine you work at Google. You used to work on the web—maybe on WebKit and then Chrome like Eric, maybe on web standards—but you broke with this past to see what Flutter might become. Flutter works: great job everybody! The set of graphical and input primitives that you use is minimal enough that it is abstract by nature; it doesn't much matter whether you target iOS or Android, because the primitives will be there. But the web is still the web, and it is annoying, aesthetically speaking. Could we Flutter-ize the web? What would that mean?

That's exactly what former HTML specification editor and now Flutter team member Ian Hixie proposed this January in a brief manifesto, Towards a modern Web stack. The basic idea is that the web and thus the browser is, well, a bit much. Hixie proposed to start over, rebuilding the web on top of WebAssembly (for code), WebGPU (for graphics), WebHID (for input), and ARIA (for accessibility). Technically it's a very interesting proposition! After all, people that build complex web apps end up having to fight with the platform to get the results they want; if we can reorient them to focus on these primitives, perhaps web apps can compete better with native apps.

However if you game out what is being proposed, I have doubts. The existing web is largely HTML, with JavaScript and CSS as add-ons: a web of structured text. Hixie's flutterized web proposal, on the other hand, is a web of pixels. This has a number of implications. One is that each app has to ship its own text renderer and internationalization tables, which is a bit silly to say the least. And whereas we take it for granted that we can mouse over a web page and select its text, with a web of pixels it is much less obvious how that would happen. Hixie's proposal is that apps expose structure via ARIA, but as far as I understand there is no association between pixels and ARIA properties: the pixels themselves really have no built-in structure to speak of.

And of course unlike in the web of structured text, in a web of pixels it would be up each app to actually describe its structure via ARIA: it's not a built-in part of the system. But if you combine this with the rendering story (here's WebGPU, now draw the rest of the owl), Hixie's proposal leaves a void for frameworks to fill between what the app developer wants to write (e.g. Flutter/Dart) and the platform (WebGPU/ARIA/etc).

I said before that I had doubts and indeed I have doubts about my doubts. I am old enough to remember when X11 apps on Unix desktops changed from having fonts rendered on the server (i.e. by the operating system) to having them rendered on the client (i.e. the app), which was associated with a similar kind of anxiety. There were similar factors at play: slow-moving standards (X11) and not knowing at build-time what the platform would actually provide (which X server would be in use, etc). But instead of using the server, you could just ship pixels, and that's how GNOME got good text rendering, with Pango and FreeType and fontconfig, and eventually HarfBuzz, the text shaper used in Chromium and Flutter and many other places. Client-side fonts not only enabled more complex text shaping but also eliminated some round-trips for text measurement during UI layout, which is a bit of a theme in this article series. So could it be that pixels instead of text does not represent an apocalypse for the web? I don't know.

Incidentally I cannot move on from this point without pointing out another narrative thread, which is that of continued human effort over time. Raph Levien, who I mentioned above as a Rust UI toolkit developer, actually spent quite some time doing graphics for GNOME in the early 2000s; I remember working with his libart_lgpl. Behdad Esfahbod, author of HarfBuzz, built many parts of the free software text rendering stack before moving on to Chrome and many other things. I think that if you work on this low level where you are constantly translating text to textures, the accessibility and interaction benefits of using a platform-provided text library start to fade: you are the boss of text around here and you can implement the needed functionality yourself. From this perspective, pixels don't represent risk at all. In the old days of GNOME 2, client-side font rendering didn't lead to bad UI or poor accessibility. To be fair, there were other factors pushing to keep work in a commons, as the actual text rendering libraries still tended to be shipped with the operating system as shared libraries. Would similar factors prevail in a statically-linked web of pixels?

In a way it's a moot question for us, because in this series we are focussing on native app development. So, if you ship a platform, should your app development framework look like the web-of-pixels proposal, or something else? To me it is clear that as a platform, you need more. You need a common development story for how to build user-facing apps: something that looks more like Flutter and less like the primitives that Flutter uses. Though you surely will include a web-of-pixels-like low-level layer, because you need it yourself, probably you should also ship shared text rendering libraries, to reduce the install size for each individual app.

And of course, having text as part of the system has the side benefit of making it easier to get users to install OS-level security patches: it is well-known in the industry that users will make time for the update if they get a new goose emoji in exchange.

alternate world: webassembly

Hark! Have you heard the good word? Have you accepted your Lord and savior, WebAssembly, into your heart? I jest; it does sometime feel like messianic narratives surrounding WebAssembly prevent us from considering its concrete aspects. But despite the hype, WebAssembly is clearly a technology that will be a part of the future of computing. So let's dive in: what would it mean for a mobile app development platform to embrace WebAssembly?

Before answering that question, a brief summary of what WebAssembly is. WebAssembly 1.0 is portable bytecode format that is a good compilation target for C, C++, and Rust. These languages have good compiler toolchains that can produce WebAssembly. The nice thing is that when you instantiate a WebAssembly module, it is completely isolated from its host: it can't harm the host (approximately speaking). All points of interoperation with the host are via copying data into memory owned by the WebAssembly guest; the compiler toolchains abstract over these copies, allowing a Rust-compiled-to-native host to call into a Rust-compiled-to-WebAssembly module using idiomatic Rust code.

So, WebAssembly 1.0 can be used as a way to script a Rust application. The guest script can be interpreted, compiled just in time, or compiled ahead of time for peak throughput.

Of course, people that would want to script an application probably want a higher-level language than Rust. In a way, WebAssembly is in a similar situation as WebGPU in the web-of-pixels proposal: it is a low-level tool that needs higher-level toolchains and patterns to bridge the gap between developers and primitives.

Indeed, the web-of-pixels proposal specifies WebAssembly as the compute primitive. The idea is that you ship your application as a WebAssembly module, and give that module WebGPU, WebHID, and ARIA capabilities via imports. Such a WebAssembly module doesn't script an existing application: it is the app. So another way for an app development platform to use WebAssembly would be like how the web-of-pixels proposes to do it: as an interchange format and as a low-level abstraction. As in the scripting case, you can interpret or compile the module. Perhaps an infrequently-run app would just be interpreted, to save on disk space, whereas a more heavily-used app would be optimized ahead of time, or something.

We should mention another interesting benefit of WebAssembly as a distribution format, which is that it abstracts over the specific chipset on the user's device; it's the device itself that is responsible for efficiently executing the program, possibly via compilation to specialized machine code. I understand for example that RISC-V people are quite happy about this property because it lowers the barrier to entry for them relative to an ARM monoculture.

WebAssembly does have some limitations, though. One is that if the throughput of data transfer between guest and host is high, performance can be bad due to copying overhead. The nascent memory-control proposal aims to provide an mmap capability, but it is still early days. The need to copy would be a limitation for using WebGPU primitives.

More generally, as an abstraction, WebAssembly may not be able to express programs in the most efficient way for a given host platform. For example, its SIMD operations work on 128-bit vectors, whereas host platforms may have much wider vectors. Any current limitation will recede with time, as WebAssembly gains new features, but every year brings new hardware capabilities (tensor operation accelerator, anyone?), so there will be some impedance-matching to do for the foreseeable future.

The more fundamental limitation of the 1.0 version of WebAssembly is that it's only a good compilation target for some languages. This is because some of the fundamental parts of WebAssembly that enable isolation between host and guest (structured control flow, opaque stack, no instruction pointer) make it difficult to efficiently implement languages that need garbage collection, such as Java or Go. The coming WebAssembly 2.0 starts to address this need by including low-level managed arrays and records, allowing for reasonable ahead-of-time compilation of languages like Java. Getting a dynamic language like JavaScript to compile to efficient WebAssembly can still be a challenge, though, because many of the just-in-time techniques needed to efficiently implement these languages will still be missing in WebAssembly 2.0.

Before moving on to WebAssembly as part of an app development framework, one other note: currently WebAssembly modules do not compose very well with each other and with the host, requiring extensive toolchain support to enable e.g. the use of any data type that's not a scalar integer or floating-point value. The component model working group is trying to establish some abstractions and associated tooling, but (again!) it is still early days. Anyone wading into this space needs to be prepared to get their hands dirty.

To return to the question at hand, an app development framework can use WebAssembly for scripting, though the problem of how to compose a host application with a guest script requires good tooling. Or, an app development framework that exposes a web-of-pixels primitive layer can support running WebAssembly apps directly, though again, the set of imports remains to be defined. Either of these two patterns can stick with WebAssembly 1.0 or also allow for garbage collection in WebAssembly 2.0, aiming to capture mindshare among a broader community of potential developers, potentially in a wide range of languages.

As a final observation: WebAssembly is ecumenical, in the sense that it favors no specific church of how to write programs. As a platform, though, you might prefer a state religion, to avoid wasting internal and external efforts on redundant or ill-advised development. After all, if it's your platform, presumably you know best.

summary

What is to be done?

Probably there are as many answers as people, but since this is my blog, here are mine:

  1. On the shortest time-scale I think that it is entirely reasonable to base a mobile application development framework on JavaScript. I would particularly focus on TypeScript, as late error detection is more annoying in native applications.

  2. I would to build something that looks like Flutter underneath: reactive, based on low-level primitives, with a multithreaded rendering pipeline. Perhaps it makes sense to take some inspiration from WebF.

  3. In the medium-term I am sympathetic to Ark's desire to extend the language in a more ResultBuilder-like direction, though this is not without risk.

  4. Also in the medium-term I think that modifications to TypeScript to allow for sound typing could provide some of the advantages of Dart's ahead-of-time compiler to JavaScript developers.

  5. In the long term... well we can do all things with unlimited resources, right? So after solving climate change and homelessness, it makes sense to invest in frameworks that might be usable 3 or 5 years from now. WebAssembly in particular has a chance of sweeping across all platforms, and the primitives for the web-of-pixels will be present everywhere, so if you manage to produce a compelling application development story targetting those primitives, you could eat your competitors' lunch.

Well, friends, that brings this article series to an end; it has been interesting for me to dive into this space, and if you have read down to here, I can only think that you are a masochist or that you have also found it interesting. In either case, you are very welcome. Until next time, happy hacking.

June 14, 2023

Rethinking Adaptivity

TL;DR: The current adaptive widgets have significant problems and have all been replaced and deprecated. You may want to port your apps, the migration guide is here.


Over the past year, I’ve been working on replacing the old adaptive widgets in libadwaita, and as of a few days ago the last pieces have landed.

The Problems

(kgx:356880): Gtk-WARNING **: 16:38:18.039: Allocating size to AdwFlap 0x1e25c50 without calling gtk_widget_measure(). How does the code know the size to allocate?

(epiphany:91138): Gtk-WARNING **: 10:49:35.638: Allocating size to EphyWindow 0x15f8b70 without calling gtk_widget_measure(). How does the code know the size to allocate?

Ever seen an app print a warning like that on resize? If you’re unlucky, also flicker at the same time. This happens when a widget causes a resize during another resize. That’s one of the things that GTK prohibits, and yet almost every adaptive app currently does it, and most of the time it only works by accident.

Take AdwLeaflet, for example. People often use it for managing a sidebar. That’s fine, because leaflet will handle its own resize correctly. However, sidebar also implies other changes, like showing/hiding window controls and a back button. All of these changes cause resizes, and we do them when already inside a leaflet’s resize. Oops.

To an extent libadwaita widgets work around that, e.g. by emitting property notifications when they’re least likely to cause problems, but it’s never perfect. The simple truth is that this system is inherently fragile, depending on a number of things like whether the widget you’re changing is allocated before or after the leaflet (which is an implementation detail).

Flexibility

Another problem with the current approach is that while we provide premade widgets for specific cases, like a sidebar turning into stack navigation, any other kinds of adaptivity are basically unsupported.

For example, maybe your app has a header bar with a lot of widgets, and you want to move some of them into a separate bottom bar on mobile? There’s no premade widget for this, so the only way you can do it is by overriding the size_allocate() virtual function on some widget (typically the window) yourself and doing it there.

This is very finicky and easy to get wrong, and once again you’re setting yourself up for warnings and flicker — the Epiphany warning above is from doing exactly this, for example.

And it can’t even be abstracted as a widget either.

And really… a lot of apps need to do things besides what leaflet and friends provide. Even the libadwaita demo has to (ab)use GtkFlowBox for its style classes demo, while any adaptive app using AdwTabBar has to deal with size_allocate(). And the apps that don’t need tricks like that often have to compromise their design to make sure they don’t use anything unsupported. For example, at one point there was a GNOME Software design using a sidebar at desktop sizes and a view switcher at mobile sizes. That’s not possible with existing widgetry, so it never shipped.

Transition Animations

Widgets like AdwLeaflet provide an animation when switching between modes. This is cool, but also it causes the following bug when starting an app:

Notice the sidebar being briefly visible and then disappearing? This shouldn’t happen. And yet, it does and it’s not fixable.

Even when the transitions work, they are inconsistent. Sure, the leaflet itself animates, but the back button you add? It doesn’t. Even if you use a GtkRevealer to animate the button too, you’ll have to show/hide it at some point to avoid double spacing in the header bar if you have any other buttons next to it. And you can’t use a revealer for window buttons anyway.

Natural width

One thing the current approach (mostly) avoids is apps explicitly defining the size thresholds for layout changes. Instead, all of the existing adaptive widgets use natural widths of child widgets as implicit thresholds. While it avoids magic numbers, it doesn’t work very well in practice:

  • The widths can change on the fly. For example, this often happens in Settings, where you may very well have the sidebar collapse or reappear when you switch between different panels:

    Another example is Fractal, where a similar thing was happening because of long room topics. Because of that, AdwLeaflet actually defaults to using minimum widths as its threshold and not natural widths.

  • Natural widths are hard to control. Apps have no direct control over them, and often have to resort to tricks, like putting labels into GtkScrolledWindow with scrolling disabled to prevent natural width from propagating. (That’s what Fractal did to work around long room topics at first, for example)
  • If your application does more than just show/hide the sidebar — say, toggle the sidebar and also move a view switcher from the top of the window to the bottom, these changes will not happen at the same time. In fact, they may even happen in different order depending on the locale, font, etc., leading to hard to debug issues.
  • Since we’re adding and removing back buttons and window controls, the threshold actually changes between wide and narrow states. If an app does the opposite and removes widgets on narrow sizes, it can even lead to infinite loops:
    • The widgets are too wide, folding the leaflet.
    • Leaflet is now folded, removing buttons.
    • The widgets are narrow enough that they can fit side by side now, unfolding the leaflet.
    • The leaflet is unfolded, adding the buttons back.
    • The widgets are too wide, folding the leaflet.
    • … and so on.
  • Finally, there are cases where we always want narrow layout, even when the widgets fit otherwise. For example, on mobile we always want view switchers to be at the bottom, and yet there are quite a few cases where they would still fit at the top. And so, AdwViewSwitcherTitle has a hack where it monitors the window’s width and forcefully disables its view switcher when it’s below or equal to 360px, on top of artificially limiting its natural size to try and control the threshold.

While fixed thresholds present a different set of problems and are very much a tradeoff, at the end of the day you’re designing your layout to fit into a fixed resolution screen anyway.

All of that is already really bad, but the problems go even further.

API

Leaflet

Let’s look at AdwLeaflet. Most apps use this widget in one of the following two ways:

  1. The leaflet has three children: a sidebar, a separator and content. The separator is marked as non-navigatable. The leaflet has swipe to go back enabled, but not swipe to go forward. Sidebar and content both have header bars, with window buttons hidden in the middle when unfolded, and content has a back button when folded, that goes back to sidebar.
  2. The leaflet has the can-unfold property set to FALSE, and is used basically as a GtkStack with swipe gestures.

However, what is the actual purpose of this widget? Well, it acts as a GtkBox when unfolded and as a GtkStack when folded. As in, it has an unspecified number of children, shows all of them when unfolded and one of them when folded. That’s it.

For the first case we always want a separator. However, AdwLeaflet cannot automatically add it because it doesn’t know if it’s going to be used for a sidebar or not. There’s also no way to know what even is a sidebar: it’s just a leaflet child, so there’s no way to automatically style it. It can’t add a back button for you for the same reason, and so on.

Now let’s look at the second case. It works fine if you have 2 pages and just want to switch between them. You will still have to manage the back button on the second widget manually, but otherwise it’s okay — basically the same as GtkStack.

Now what if we want to have page 1 that leads to page 2 and page 3, both of which lead back to the page 1? Well, now we run into a problem.

If we simply add all three pages into the leaflet, going back from page 3 will lead to page 2 and not page 1.

There are two possible workarounds:

  1. Toggle the page visibility dynamically.
  2. Use a GtkStack with pages 2 and 3, and add that to the leaflet along with page 1.

Neither approach is particularly great.

So, AdwLeaflet is not very well suited for either case. Or more specifically, it tries to be low level and generic like GtkBox and such, while it’s primarily used for much more specific cases.

Flap

AdwFlap is not nearly as bad, as at least it has a fixed number of clearly defined children. But even then it can be used for multiple things that aren’t sidebars, like bottom sheets (even though I’m not aware of any apps actually using it as such now that Console switched to AdwTabOverview) or even for handling the window titlebar in fullscreen.

This means that it’s still too generic to be nice to use in any of these cases:

  • Just as with leaflet, you have to add a separator yourself. Many apps do it wrong and have it on the sidebar instead of a separate child.
  • The sidebar has no background because what if your use case needs it to be transparent?

And so on.

Or to summarize it: just like AdwLeaflet, flap is too generic. It tries to be a box/overlay widget and not, say, a sidebar widget.

Squeezer

AdwSqueezer is mostly used for handling view switchers in header bars. It contains multiple children and shows the largest one that can fit. So other than doing unsupported things for its most common use case, it works fine.

It still has a problem though. Since you put a view switcher and a title inside a squeezer, the header bar will center both at once. If the view switcher is too wide to fit, we show the window title, and since the whole widget is not centered at this point, the title will be off-center:

AdwHeaderBar has a centering-policy property to work around this, but it also introduces another problem:

What we actually want is this though:

But that cannot be accomplished when the view switcher and the title are in a single widget (squeezer’s natural width is that of its largest child — in this case, the view switcher — and if it doesn’t fit, the squeezer is still expanded to fit the remaining space, then the title is centered within that and not within the header bar), but we rely on that to make it adaptive. Not good.

OK, What Now?

So now that we’ve established that the existing widgetry has significant problems, what can we do differently?

Well, lets look at how other platforms handle it.

  • The Web has CSS and media queries. One can define a media query that matches a range of screen sizes (as well as many other things), and CSS rules inside its block would only apply within that range. Since CSS controls layout, people can rearrange their layout using media queries.

    Most web frameworks just use media queries or provide a similar API. This is often referred to as breakpoints.

  • Windows/UWP has AdaptiveTrigger and visual states. One can create a visual state that activates depending on the window size and toggle properties on the widgets in that window using setters.

  • Android has an elaborate resource system that allows for defining different layouts (UI files), dimensions or really any values only for specific screen size buckets or size ranges (as well as many other conditions), and when those conditions change (e.g. when the screen changes orientation), the toolkit just rebuilds the whole UI. Additionally, apps can factor out the common parts of their UI into different fragments, and then the activity (window) layouts can just arrange them in different ways, avoiding duplication.

    (This may be outdated though, I wasn’t able to find any information about how Android does adaptive layouts nowadays and my own knowledge is from Android 4.x, so about a decade old at this point)

  • UIKit uses a constraint layout, and constraints can be marked to only apply depending on screen orientation or size. While it avoids specific size values, instead preferring to target specific devices, iOS has the benefit of only running on a few select devices and can afford that. And before that system was introduced, apps could (and still can) just have different storyboards for different devices or screen sizes.

    Additionally, UIKit provides controllers like UISplitViewController that rearrange their layout automatically.

So… it’s all over the place, but let’s try to come up with something that combines the best parts of the above and hopefully avoid their pitfalls.

Breakpoints

First, we’ll need a way to do arbitrary changes to the UI based on the window size, there’s no way around that.

So let’s implement a window that can do arbitrary layout changes for its contents.

That’s hard, but not impossible. It involves temporarily hiding the contents, applying changes and showing it again on the next frame. To avoid flicker, a snapshot of the child from the last frame can be shown while it’s hidden. It also involves a special case for when mapping the child for the first time.

A hard problem here is sizing. Widgets like leaflet know what their layout will be when it’s folded and unfolded (as long as apps don’t add random back buttons or do other changes, anyway) and can expose correct minimum and natural sizes regardless of their current state. This is important because otherwise you wouldn’t be able to resize them to smaller sizes in the first place.

But we’re talking about arbitrary changes, so there’s no way to predict the size for our window unless it’s already at the smallest possible state. So, the only thing we can do is to disable the minimum size entirely and require apps to specify it, and clip the child if it doesn’t actually fit (or alternatively scale it down).

Another problem is transitions. Since we don’t know what exactly will change between different states, there’s no way to animate it beyond a global crossfade. But considering transitions were already inconsistent, I don’t think it’s a big loss.

And since libadwaita already has AdwWindow and AdwApplicationWindow, we can just reuse them and have all of this logic in there.

Now that we have this functionality we need to figure out good API for control it.

CSS in GTK doesn’t support media queries. And even if it did, they wouldn’t be useful, since CSS in GTK doesn’t control layout beyond widget margins/paddings/borders, minimum widths/heights and sometimes spacing. The overall concept is still attractive though, but we can’t use CSS for this.

However, we can still use UI files, like the UWP system does. What if our widget was defining “visual states” with a size-based condition and property setters? Following the web, let’s call them breakpoints.

GtkBuilder supports custom tags, so we can actually replicate this system in its entirety.

It looks like this:

<object class="AdwWindow">
  <property name="width-request">360</property>
  <property name="height-request">200</property>
  <property name="child">
    <!-- ... -->
  </property>
  <child>
    <object class="AdwBreakpoint">
      <condition>max-width: 500px</condition>
      <setter object="my_object" property="some-property">value</setter>
      <setter object="my_object" property="some-property">value</setter>
      <!-- ... -->
      <setter object="my_object" property="some-property">value</setter>
    </object>
  </child>
</object>

Pretty close, isn’t it? Of course, only being able to change properties is rather limiting, so each breakpoint also has apply and unapply signals.

Now we need to figure out how to deal with child overflow. In an ideal world, apps will just pick the widths where it doesn’t happen, but in practice that won’t work. Even if the widgets fit in English locale, it may not necessarily fit in German. Or with Large Text enabled.

This process isn’t really any different from just adapting an app for small screen sizes. You need it to fit into 360 pixels horizontally, at any cost. So, if you have a long label and you can’t make it shorter or rearrange your UI, you wrap it, ellipsize it or use a smaller font or letter spacing.

One addition on GTK side here is the GtkButton:can-shrink property (as well as GtkMenuButton:can-shrink). Buttons are the main source of labels outside of apps’ control, and now there’s a an easy way to deal with them.

And to deal with different text scales, we can introduce support for units other than pixels to breakpoints.

Currently three units are supported: pixels, points and scale-independent pixels. The last unit is lifted directly from Android and is basically points, but normalized to match pixels at default text scale. See the docs for the exact values.

Finally, in some cases it can be nice to be able to use breakpoints on a more localized area of the UI, for example, a specific page. And so, all of this machinery is also available as a standalone widget: AdwBreakpointBin.

Replacing Squeezer and AdwViewSwitcherTitle

Next, we need to figure out how to adapt existing patterns for this new system.

Some of them, like bottom toolbars or hiding tabs on mobile are doable as is and don’t need any additional support — you can simply toggle their visibility from your breakpoint.

And that includes view switchers: simply add an AdwViewSwitcher into your header bar, a hidden AdwViewSwitcherBar at the bottom of the window, and a breakpoint that removes the view switcher and shows the view switcher bar on narrow sizes.

One problem is that unfortunately AdwViewSwitcher defaults to the narrow policy, while for header bar you really want wide instead, so apps have to change it manually.

The new system also avoids the misaligned title problem, since in narrow state we remove the view switcher entirely and let header bar use its default title, so the view switcher’s width doesnt’t get in the way.

This was the only place libadwaita was using AdwSqueezer for, so we don’t need that widget anymore either.

Replacing Leaflet

AdwLeaflet, on the other hand, cannot be replicated with breakpoints alone and needs a replacement widget. For example, because it provides things like gestures for stack navigation, unlike GtkStack and similar widgets.

Rethinking leaflets has led to a surprising number of other improvements along the way, so let’s run through them.

Toolbar View

Long ago, before libadwaita 1.0, there was this mockup:

Unfortunately, at the time it wasn’t really possible to implement it, not least because of leaflet limitations. However, if we’re redoing it anyway, might as well implement that style, especially when newer apps like Amberol and Emblem use a similar style anyway.

But the flat headers present a problem. What about scrolling content? This has been a long-standing problem with flat header bars, and in fact the .flat style class for header bars has been deprecated along with the old widgets.

Thankfully, GtkScrolledWindow has a mechanism for displaying undershoot indicators, and libadwaita has new style classes for controlling them.

However, manually managing them can be tricky. Sure, it’s easy when you just have a header bar and add the .undershoot-top style class to scrolled window, but what about bottom bar in an app like Contacts? Manually managing .undershoot-bottom in sync with the reveal animation is tricky, and in fact not even possible if you’re using GtkActionBar without wrapping it into another GtkRevealer.

But what if we had a widget for managing it instead? And so that’s what AdwToolbarView does. It replaces GtkBox for cases like this, providing a slot for content, and allowing for adding one or multiple top and bottom bars. Then, it manages the undershoot styles on scrolled windows inside it.

However, having a widget just for that would be silly, so it also does a number of other things:

  • It allows for visually merging multiple toolbars together and remove their border and vertical spacing, making it possible to implement yet another mockup that wasn’t possible before:

  • For the raised style (like on the screenshot above) it makes it possible to have a shadow instead of a border. This is yet another thing that wasn’t possible before, because while GtkWindow’s titlebar is special cased to be drawn before content, that’s not the case in GtkBox, and the shadow would end up behind content. In AdwToolbarView we can make sure the content is drawn last and avoid this problem.
  • It automatically makes all toolbars draggable, including GtkSearchBar etc.
  • It allows extending the content behind top or bottom bars, and to reveal or hide toolbars with an animation. This means that it can be used instead of AdwFlap to manage fullscreen titlebar, and we don’t need to worry about that use case when replacing flap anymore.

Header bars are now also white in light variant instead of darker grey — mostly so that there’s more contrast against their shadow and it looks better with utilty panes using the new styling. The grey color header bars were previously using is now reserved for sidebars and utility panes instead.

Navigation View

The first of the leaflet replacements is AdwNavigationView.

One of the things I wanted to avoid was having a single widget for both sidebars and stack navigation. This is a large part of what made AdwLeaflet so cumbersome, and it was this way to avoid code duplication, as well as due to historical reasons: leaflet first came to exist back when everyone just used GtkStack for this, and provided a version that can also do sidebars, then gradually got improved to be more attractive than GtkStack even for stack navigation.

In fact, libhandy 1.x provides a variant of leaflet that never expands, called HdyDeck. While in theory it was a good idea to separate these two widgets, they had almost identical API, so it still didn’t make any sense. As such, for libadwaita I merged them back and essentially replaced deck with the can-unfold property.

So, as shown earlier, leaflet API is not well suited for this task. So, once again, let’s look at other platforms for inspiration.

Some of them don’t have any nice ways to handle this pattern — for example, in UWP it’s surprisingly involved. Others do, however, such as UIKit and its UINavigationController. In fact, let’s go ahead and use it as inspiration because it’s very well thought out.

A noticeable difference between UINavigationController and AdwLeaflet is that the former has a stack-like API. As in, stack the data structure, not GtkStack (wow, this is confusing, not helped by the fact UIKit calls a stack what GTK calls a box 😵‍💫). So, you have a navigation stack, push pages into it and pop pages off it, and it’s dynamic. For comparison, leaflet has a statically ordered list of children, and you move back and forward in that list.

However, we can’t make AdwNavigationView 100% dynamic: we still need some way to use it from UI files. It took a lot of iteration, but at the end I came up with the following scheme:

  1. Pages can be pushed directly into the navigation stack using push(). Once they are popped, they will be automatically removed.
  2. Pages can be statically added using add() or from UI files. Then, they can be pushed and popped and will stay in the widget and can be reused later. The first added page becomes the root of the navigation stack.
  3. If a page was pushed without having been added and then later add() is called for it, it won’t be autoremoved.
  4. If remove() is called for a page currently in the navigation stack, it won’t be automatically removed and is simply marked for removal, same as if add() had never been called.
  5. The whole navigation stack can be replaced using replace(), including removing any pages currently in the navigation stack if they are marked for autoremoval.

In other words, add() and remove() act a bit like pinning and unpinning the pages. The navigation stack itself is fully dynamic.

Pages also have tags and all of the functions for managing navigation stack have versions that work with tags, and pushing pages this way requires adding them first.

Tags also allow for another thing: GActions for managing the navigation stack. AdwNavigationView exposes two actions: navigation.push and navigation.pop. The first one takes a page tag as parameter, and this allows defining fairly complex structures, such as the one found in AdwAboutWindow entirely from UI files, without any code.

Unlike AdwLeafletPage, navigation view children — AdwNavigationPage — are widgets instead of auxiliary objects. Not only that, they are also derivable, so it’s possible to use templates with them. That’s why the tags are called tags and not names, otherwise there’s a name clash with GtkWidget:name.

Historically, GNOME apps have managed their back buttons themselves, no matter whether they are using GtkStack, HdyDeck or AdwLeaflet. But isn’t that silly? AdwLeaflet manages shortcuts and gestures for going back to the previous page, but not back buttons? Meanwhile UINavigationController automatically provides back buttons, so why can’t we?

Well, no reason really. While our situation is a bit different since header bars are just a widget instead of a part of the navigation controller, header bars already automatically show window controls for the window they are in. And so… AdwHeaderBar can now also automatically show a back button if it’s inside an AdwNavigationView. This is opt-out, same as for window buttons.

AdwHeaderBar has also learned to pull titles out of navigation pages same as how it does with window titles, since changing titles on header bars is a bit painful in GTK4, even with AdwWindowTitle. And since we have titles, they also double as accessible labels, as well as tooltips for back buttons. In fact, if we wanted to add labels to back buttons, like in iOS or elementary OS, we have means to do so now as well.

AdwNavigationView also supports a way to go forward, unlike UINavigationController — using the get-next-page signal. Meanwhile, AdwNavigationPage has signals for tracking its visibility, which can be used to create and destroy its contents on demand. With this apps like Nautilus can hopefully use this widget for content navigation, while with leaflet it just wasn’t feasible.

Finally, while AdwNavigationView hopefully reduces the number of situations where people have to nest them, it still works with nesting, including actions and back button integration. This will come in handy for the next widget.

Navigation Split View

And finally, now that the navigation case is well covered and flat header bars are doable in a robust way, we can reimplement sidebars with a widget called AdwNavigationSplitView.

Compared to AdwNavigationView, this widget is very simple. It has two children: sidebar and content, both of which are required to be AdwNavigationPage. By default, it shows them side by side. When the collapsed property is set to TRUE, it just reparents both into a navigation view. And that’s pretty much it… Add a breakpoint to your window, toggle the collapsed property from it and you have an adaptive sidebar.

Well OK, it does a few more things, but this is still a stark contrast with how complex AdwLeaflet was.

For example, AdwHeaderBar gains yet another feature. When used inside AdwNavigationSplitView, it automatically hides redundant window buttons, hopefully getting rid of bugs like this for good:

Actually, two features. The other one is a property for hiding the title — it’s common to have header bars without titles for split layouts, and having to put an invisible widget into the header bar just to get rid of its title is not ideal.

AdwNavigationSplitView also provides its own navigation.push and navigation.pop actions, so that they still work when not collapsed. They work exactly same as in AdwNavigationView.

Instead of inheriting the push/pop API of AdwNavigationView, split view provides a single show-content boolean property.

Since the sidebar child is now clearly defined, we can style it and implement that mockup from long ago, and if people use toolbar views, they get automatic undershoot shadows.

Moreover, there’s a special styling for triple pane layouts:

This can be achieved just by nesting two split views together, see the docs for more details.

Finally, it implements dynamic sizing for sidebars. This is probably the most complex part of this widget: instead of using the sidebar’s natural width it attempts to give the sidebar a specified fraction of the total width, which is additionally limited by a minimum and a maximum value (as well as the minimum width of the sidebar child). On top of that, the minimum and maximum widths can use the same units as breakpoints, and in fact AdwNavigationSplitView defaults to using the sp unit, so that sidebars scale with Large Text.

Replacing Flap

Unlike leaflet, AdwFlap doesn’t need a ground-up overhaul. We’ve already covered the fullscreen toolbar use case with AdwToolbarView, and flap is basically never used for bottom sheets — and even if it were, we’re going to have a specialized bottom sheet widget later anyway. So, the only case we need to care about is overlay sidebars and utility panes (which are basically the same thing as sidebars, but with a single header bar, so the widget doesn’t need to care about this distinction).

So, AdwOverlaySplitView is actually heavily based on AdwFlap, trimming it down and simplifying it API to match AdwNavigationSplitView. It also enjoys the same styling, dynamic sizing and AdwHeaderBar window button integration:

Yes, it looks exactly same as AdwNavigationSplitView when it’s not collapsed. What did you expect? Unlike that widget though, it does allow moving the sidebar to the right, as well as showing and hiding it, same as flap. (AdwNavigationSplitView doesn’t allow either because it would break the navigation model)

Unlike AdwNavigationSplitView, it doesn’t require children to be AdwNavigationPage, though it doesn’t prevent it either — it will work fine and you may still want to do that because of header bar title integration.

It also removes touchpad swipes. While it still has swipes on touchscreen, combining edge swipes on touchscreen with swipes anywhere on touchpad has never worked well and it has caused no end of problems for apps like Loupe. With touchpad swipes being partially broken in GTK4, it’s just not worth it. AdwNavigationView still supports touchpad swipes, but this widget does not.

Migrating

The old widgets have been deprecated and apps using libadwaita from main are encouraged to migrate to the new widgetry. Libadwaita also provides a migration guide for doing so.

The adaptive layouts page also uses the new widgets now.

Future

Dynamic Layouts

Even with all of that, there’s always room for improvement.

For example, while breakpoints allow rearranging the UI by setting properties, it’s still fairly limited and one needs widgets like split views for more complex layout changes. And while anything is possible with enough work, doing complex layout changes is rather painful (though less so than with the old widgets) and it has to be done programmatically.

However, we can take a bit of inspiration from Android, with its resource system and rebuilding the UI to change screen orientation etc.

As such, another thing I’ve been prototyping is a yet-to-be-named widget that does a similar thing to what AdwNavigationSplitView is doing, but generically:

  • You provide it multiple different UI files, each of them contains tagged slots.
  • You also provide it children with matching tags.
  • It creates UI from one of the UI files and inserts the children into those slots.
  • Which layout it uses is controlled by a property.

When you change its value, it unparents the tagged children, destroys the rest of the layout, creates the other layout and reparents the children there.

To better explain it, let’s take a look at this mockup of Loupe:

Here properties are presented in a sidebar at desktop sizes, and in a bottom sheet at mobile sizes. We obviously don’t have a widget that transforms between a sidebar and a bottom sheet (nor do we have a bottom sheet widget, but we will in future, so let’s assume we do), so there are two ways of implementing it:

  • Have both a sidebar and a bottom sheet, each containing a copy of the properties view, and only have one of them visible at a time. In case of Loupe, it will work fine, since the properties view is reasonably small, but it doesn’t scale in the general case.
  • Reparent them manually. This works, but is pretty fiddly and can’t be done declaratively.

With this new widget though? Define two layouts: one with a split view with two slots as its content and sidebar children, the other one a bottom sheet view with the same two slots. Add two children: an image view and a properties view, matching those slots. Then toggle the layout from your breakpoint. That’s it.

One downside of how Android does it is that apps have to thoroughly save widget state before rebuilding the UI and restore it afterwards, or it will be lost. But, since we’re reusing both widgets, there’s no need to save state here, it will just work.

We can still go a step further and do even that declaratively for state not covered by the children within the slots: have a list of object/property/tag triplets in each layout, and map each of them to the matching object/property on the other layout.

There are a few obstacles to solve here, but I have a working prototype, and maybe can even finish it this cycle.

Dialogs

Speaking of bottom sheets, one thing that’s still missing from libadwaita is support for adaptive dialogs. That’s coming too, but probably not this cycle. It needs cooperation with the compositor — especially for portals, where the dialog and its parent window are in different processes and can’t communicate. But we have a design and a tentative plan for how to get there.


I’d like to say thanks to:

  • Nahuel for pointing me towards UWP AdaptiveTrigger and answering UWP questions.
  • Chris for porting a lot of apps to try out and test the new API.
  • Tobias Bernard and Sam Hewitt for UI design and answering my countless questions about tiny and unimportant details.

As always, thanks to my employer, Purism, for letting me work on all of this.

GSoC 2023: Week 1 & 2 Report

Project

Adding Acrostic support to GNOME Crosswords

Mentor

Jonathan Blandford

Week 1

The first thing I did was fix libipuz #22. Earlier across/down clues were hardcoded in the ipuz-cell. Due to this, we would do an if-else check on across/down clues based on their direction in other parts of the code.

A clue in an acrostic puzzle has a direction IPUZ_CLUE_DIRECTION_CLUES which was not getting handled. This would cause Crosswords to crash after performing an action on an Acrostic puzzle because of clues not getting copied.
Now we are storing the clues in a GArray.

- IPuzClue *across_clue;
- IPuzClue *down_clue;
+ GArray *clues;
Now clues are getting copied onto the PuzzleStack when we perform an action. Thus, we are able to undo the action.

Week 2

I started off week 2 by adding support for ipuz files with kind http://ipuz.org/acrostic and added a load test to load the files.

Next up was implementing a getter for the Quote clue. The acrostic file may or may not contain the quote clue. So the idea was to extract it if already present otherwise generate it by iterating over each cell of the main grid.

The quote clue is stored in “Zones:Quote” object and that needs to be displayed using a separate clue list with the direction IPUZ_CLUE_DIRECTION_ZONES and label Quote. This was done by splitting the string using “:” as delimiter.

Quote clue list getting displayed while loading Acrostic

This change would also help us in switching navigation based on the focus.
A test was also added to compare quote clues for the same puzzle with and without the quote clue in the file.

Related Merge Requests:

  • Add Grray to store clues !22
  • Add support for files with kind http://ipuz.org/acrostic !23
  • Extract label !24
  • Extract quote clue if present otherwise generate !25

That’s all for this week. Thanks for reading!

June 11, 2023

scikit-survival 0.21.0 released

Today marks the release of scikit-survival 0.21.0. This release features some exciting new features and significant performance improvements:

  • Pointwise confidence intervals for the Kaplan-Meier estimator.
  • Early stopping in GradientBoostingSurvivalAnalysis.
  • Improved performance of fitting SurvivalTree and RandomSurvivalForest.
  • Reduced memory footprint of concordance_index_censored.

Pointwise Confidence Intervals for the Kaplan-Meier Estimator

kaplan_meier_estimator() can now estimate pointwise confidence intervals by specifying the conf_type parameter.

import matplotlib.pyplot as plt
from sksurv.datasets import load_veterans_lung_cancer
from sksurv.nonparametric import kaplan_meier_estimator
_, y = load_veterans_lung_cancer()
time, survival_prob, conf_int = kaplan_meier_estimator(
y["Status"], y["Survival_in_days"], conf_type="log-log"
)
plt.step(time, survival_prob, where="post")
plt.fill_between(time, conf_int[0], conf_int[1], alpha=0.25, step="post")
plt.ylim(0, 1)
plt.ylabel("est. probability of survival $\hat{S}(t)$")
plt.xlabel("time $t$")
Kaplan-Meier curve with pointwise confidence intervals.

Kaplan-Meier curve with pointwise confidence intervals.

Early Stopping in GradientBoostingSurvivalAnalysis

Early stopping enables us to determine when the model is sufficiently complex. This is usually done by continuously evaluating the model on held-out data. For GradientBoostingSurvivalAnalysis, the easiest way to achieve this is by setting n_iter_no_change and optionally validation_fraction (defaults to 0.1).

from sksurv.datasets import load_whas500
from sksurv.ensemble import GradientBoostingSurvivalAnalysis
X, y = load_whas500()
model = GradientBoostingSurvivalAnalysis(
n_estimators=1000, max_depth=2, subsample=0.8, n_iter_no_change=10, random_state=0,
)
model.fit(X, y)
print(model.n_estimators_)

In this example, model.n_estimators_ indicates that fitting stopped after 73 iterations, instead of the maximum 1000 iterations.

Alternatively, one can provide a custom callback function to the fit method. If the callback returns True, training is stopped.

model = GradientBoostingSurvivalAnalysis(
n_estimators=1000, max_depth=2, subsample=0.8, random_state=0,
)
def early_stopping_monitor(iteration, model, args):
"""Stop training if there was no improvement in the last 10 iterations"""
start = max(0, iteration - 10)
end = iteration + 1
oob_improvement = model.oob_improvement_[start:end]
return all(oob_improvement < 0)
model.fit(X, y, monitor=early_stopping_monitor)
print(model.n_estimators_)

In the example above, early stopping is determined by checking the last 10 entries of the oob_improvement_ attribute. It contains the improvement in loss on the out-of-bag samples relative to the previous iteration. This requires setting subsample to a value smaller 1, here 0.8. Using this approach, training stopped after 114 iterations.

Improved Performance of SurvivalTree and RandomSurvivalForest

Another exciting feature of scikit-survival 0.21.0 is due to a re-write of the training routine of SurvivalTree. This results in roughly 3x faster training times.

Runtime comparison of fitting SurvivalTree.

Runtime comparison of fitting SurvivalTree.

The plot above compares the time required to fit a single SurvivalTree on data with 25 features and varying number of samples. The performance difference becomes notable for data with 1000 samples and above.

Note that this improvement also speeds-up fitting RandomSurvivalForest and ExtraSurvivalTrees.

Improved concordance index

Another performance improvement is due to Christine Poerschke who significantly reduced the memory footprint of concordance_index_censored(). With scikit-survival 0.21.0, memory usage scales linear, instead of quadratic, in the number of samples, making performance evaluation on large datasets much more manageable.

For a full list of changes in scikit-survival 0.21.0, please see the release notes.

Install

Pre-built conda packages are available for Linux, macOS (Intel), and Windows, either

via pip:

pip install scikit-survival

or via conda

 conda install -c sebp scikit-survival

GSoC 2023: Rust and GTK 4 Bustle Rewrite (Week 1 & 2)

Basic Diagram Implementation and Porting to zbus

Progress Made

Before the GSoC coding period started, I started implementing the diagram used to display DBus activity with the help of the template repository made by my mentor, Maximiliano. One of the first challenges is figuring out how to load the PCAP files, which is the format Bustle uses to store the DBus messages. Without implementing that first, it would be difficult to test how the diagram will look like. 

The Rust pcap library was used to load the packets from the PCAP file, which contains the bytes of a DBus Message. It is nice to use, though it is missing an async API for loading files. That could be fixed in the future, but this week mainly focused on a basic diagram implementation. The bytes can then be parsed through GDBus into a GDbusMessage, which contains the information to implement the diagram.

Through the parsed message, the diagram can be implemented. It uses ListView to display the rows as, aside from having a nice separation of view and model, it is more efficient because it recycles widgets, especially since PCAP files could possibly contain thousands of messages. Each row contains the elapsed time, path, destination, interface, and member of the message.


Aside from the diagram, a DetailsView was also implemented. It shows the sender and the body, and also the destination, path, and the member of the message. For more information about it, you can check out the merge request.

GIO Dbus was used initially as zbus is missing the necessary APIs for parsing messages from bytes. However, the merge requests are already on the way to implement those upstream, including a nicer way to print zbus::Value that would match how GVariants are printed.

Plans for the Following Weeks

While the merge requests are in place, there is still some work to do, such as fixing up the Display implementation for zbus::Value::Str and zbus::Value::Array. Once the mentioned merge requests are merged, we could finish up the zbus port

After the application is fully ported to zbus, we could start working on the more complicated part of the Diagram implementation, like showing signals and the method call and return arrows.


That's all for this week. See you again in the next weeks. Thanks for reading!







June 09, 2023

GSoC 2023: Week 2 Report

In this post, I’ll go over everything I did in my first two weeks of GSoC.

Project

Make GNOME Platform demos for Workbench

Mentors

Sonny Piers, Andy Holmes

Project Planning

We first started out with a meeting to discuss a project plan, decide what needs to be done, and came up with a workflow that’ll work for everyone. Sonny made a Kanban board and filled it with some tickets to start us off. Our mentors briefly explained to us the functionality of some of the widgets and gave us an idea of what’s expected from the demos, so that we are not completely clueless when we start working on them. And when everyone is on the same page, we mark the ticket as “Ready” which means anyone is free to take up the ticket and start working on it. We also decided that we’ll have meetings weekly, to discuss the upcoming week’s work and also solve any issues or roadblocks that we may have come across along the way.

Week 1

The first thing I did was finish one of my previously open pull requests which was a demo for AdwHeaderBar, a simple widget but a very commonly used one. The demo shows a header bar with a primary menu, a secondary “Open” menu, and a “New Tab” button similar to Text Editor.

The next demo I did was a pretty big one, it was WebKit’s WebView. WebView is a super cool widget, it lets you render webpages, HTML, or plain text and it powers apps such as Epiphany and Tangram. A problem I had while working on this one was that the upstream documentation wasn’t up to date, but Sonny discovered that you could still access the latest documentation using DevHelp. The demo currently only shows some basic signals and properties with some controls such as a URL bar, buttons for navigation, and buttons to reload and stop loading the webpage.

In another follow-up, we plan to add some more functionality to this demo such as showing how to communicate with the webpage.

Week 2

In the next week, I worked on an “Advanced Buttons” demo which shows two different kinds of buttons. The first is AdwSplitButton which combines the functionality of a button and menu model, and the second one is not a complete widget on its own but it's pretty useful when used with other buttons which is AdwButtonContent. We already had a Button demo in the Library but we felt that this one couldn’t really fit in so we decided to separate it out into another demo.

Then I worked on the AdwTabView demo, which shows you how to create a basic tabbed layout using TabView along with other widgets such as AdwTabBar and AdwTabOverview.

With the addition of this demo, we decided to introduce a new section to the Library, the “Navigation” section which was pretty exciting.

The last one is a fun one which is AdwAnimation (WIP). The demo explains two ways to use Animation, first is using AdwAnimationTarget which simply animates a widget property and the second one is using AdwAnimationCallbackTarget which allows for more control over what can be animated. I had some difficulties trying to get a working demo for the latter one but I figured something out in the end. The demo also shows the difference between a TimedAnimation and a SpringAnimation.

The End

That’s all for this one, I hope to keep contributing to Workbench and I’ll continue giving reports every two weeks (maybe).

Help Us Test Evolution

It was not an easy task to make Evolution run nicely as a flatpak, but Milan Crha managed to do it and we’ve been fine-tuning it for the last 3 years. There are still some use cases that don’t fully work in a flatpak, but they don’t affect most users. Evolution has established itself well on Flathub, too. It has accumulated over 130k installs. There are roughly 12-15k “active” installations.

Some time ago I also started building Evolution for the beta channel on Flathub. When there are already development releases of the upcoming version (it will be 3.49.x this cycle), I build those for the beta channel. If they’re not available yet, I push stable releases there right after the upstream release is done, roughly one week before they go to the stable channel.

So the beta channel always provides the latest and greatest. Be it the latest development version, or freshly released stable version, depending on the phase of the development cycle. It really helps us find problems before they hit average users. For example, the latest release – 3.48.2 – introduced an ugly regression that made basically impossible to view some HTML messages. It was identified and patched while the version was still in the beta channel, so it never reached users on stable.

That’s why I’d like to raise awareness of the beta channel with Evolution. If you’re a bit more advanced user who would like to help with testing and you use Evolution from Flathub, consider switching to the beta channel. You wouldn’t switch to something broken. Milan doesn’t let low quality releases out. It’s for rather rare bugs that would be great to identify and fix before they hit everyone, or for early feedback when UX changes are being done.

All you need to do is to add the beta channel:

flatpak remote-add --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo

And install Evolution from it:

flatpak install flathub-beta org.gnome.Evolution

Of course, there are other ways to get the latest release of Evolution. You can build it from source yourself, or build a flatpak directly from the upstream git repo, but the beta channel on Flathub is the most convenient.

EDIT: I forgot the most important thing: where to report problems. All general issues including problems specific to Flatpak as a distribution format belong to the Evolution issue tracker on GNOME Gitlab. Only problems that are specific to Evolution on Flathub should belong to our Flathub issue tracker on Github. I understand it may be difficult for the user to distinguish between a Flatpak problem in general and a problem specific to Flathub. So if you report the former in our issue tracker on Github, it’s fine, too.

June 08, 2023

You are not actually mad at Flatpak

It’s that time of the month again, when some clueless guy tries to write a hit-piece about Flatpak and we all get dejavus.

One of my favorite past-time activities for a while now has been seeing people on the internet trying to rationalize concepts and decisions, and instead of asking why things ended up the way they did, what where the issues and goals of system A, and design B, and what were the compromise, they just pick the first idea that comes to their mind and go with it.

For example, a very common scenario is that someone picks a random Proprietary application and points out all the sandbox holes it needs to function and thus declares the sandbox as useless. At no point does one of them ever ask, “Hey why does Flatpak allow to punch such holes”, “What developments have been done to limit that”, “What tools are available to deal with that”, “Why am I a cherry-picking an evil proprietary application as my example that no distribution would be able to distribute anyway and I wouldn’t want any person to use” and “What went wrong with my life that I have to write hate posts to get attention and feel any kind of emotion”. These are just a few of the question that should have come up and given one pause, way before getting anywhere near the the publish button.

Now I can answer most of these questions, and you would be happy to know that even Chromium and Electron have been adopting more and more of the sandboxed Portal APIs as the years pass. But there isn’t any point in talking about it cause none of the delirium is about the technical decisions behind Flatpak or how it works. None.

Let me explain.

Flatpak itself is a piece of software. It provides major advantages to distributing and running applications such as: atomic updates, binary deltas, reproducible build and run environments, mandatory sandboxing for many resources, and so on. How the software is built and distributed however has nothing to do with Flatpak. If you think the distribution-model is what’s best for you, you can already use Fedora’s flatpaked applications, Canonical’s snaps or your fav distro version of this. Everything is still built from distribution packages, by your distribution vendor, vetted by the package maintainers, come with the same downstream patches you’d see in the normal rpm/deb/etc variations, and so on. And you would still get the advantages of sandboxing, atomicity, etc even though you don’t need them cause you love and trust your distro so much.

On the other hand what every single post really complains about is Flathub. You see, what Flatpak gave us was the ability to decouple the applications from the host system. Instead of taking the existing runtime from some distro, We (The platform and application developers) built our Runtimes from scratch, that we were in full control of, that we could update and mold at will, that was not bound to any existing distribution or corporation, that we could make sure our applications were fully functional with, without any downstream patches that made things orange or blue. And unlike the old distribution model, Flathub gave application developers the same autonomy. We no longer had to wait for dependencies to be packaged, or the worry about some distribution shipping an incompatible version. We didn’t have to wait until a new enough version of a library was included into an LTS release before making use of it. We could now ship our applications on our cadence, without gatekeepers, in the way we envisioned and intended.

This is what made applications truly work on any distribution. This is what was truly disruptive about Flatpak. This is what the haters are mad about.

Thanks to Flathub the social dynamic for distributing applications has changed. Now the people that create the Platforms (GNOME, KDE, Elementary, etc) and Applications are in charge of distributing them. The sysadmin-turned-distro-packager middleman trope from the 90s is gone and no developer, or user wants it back. This is why Flathub took over, this is why no application developer became a Fedora packager even when they could build Flatpaks from the rpms packaged. If we ever want “Desktop Linux” to succeed, we have to let go of the idea of Linux distributions and “Linux” as a monolith.

The old distribution model is still useful for very specific, enterprise environments where you depend on a single ISV for all your software, but unless your Surname is Mr. IBM or Mr. Canonical, you gain nothing by asking for this on your desktop.

If you want to read more on the subject I highly suggest these two blogposts, along with Richard Brown’s Fosdem 2023 talk.

 

Screen reading, part 2: looking at Coqui TTS

I did some research into open source text to speech solutions recently. As part of that I spent a while trying out Coqui TTS and, while I was trying to get it to sound nice, I learned a few things about voice synthesis.

Coqui TTS provides some speech models which are pre-trained on specific datasets. For text-to-speech, each dataset contains text snippets paired with audio recordings of humans reading the text.

Here is a summary of some of the available open datasets for English:

  • ljspeech: audio recordings taken from LibriVox audio books. One (female) speaker, reading text from 19th and 20th century non-fiction books, with 13K audio clips.
  • VCTK: 110 different speakers, ~400 sentences per speaker
  • Sam: a female voice actor (Mel Murphy) reading a script which was then post-processed to sound non-binary. There’s an interesting presentation about gender of voice assistants accompanying this dataset. ~8 hrs of audio.
  • Blizzard2013: A voice actor (Catherine Byers) reading 19th & 20th century audio books, focusing on emotion. Released as part of Blizzard 2013 challenge. ~200 hrs of audio.
  • Jenny: a single voice actor (again, female) reading various 21st century texts. ~30 hrs of audio.

There’s a clear bias towards female voices here, and a quick search for “gender of voice assistant” will turn up some interesting writing on the topc.

There are then different models that can be trained on a dataset. Most models have two stages, an encoder and a vocoder. The encoder generates a mel spectogram which the vocoder then uses to generate audio samples. Common models available in Coqui TTS are:

  • GlowTTS: a “flow-based” encoder model
  • Tacotron: a “DL-based” encoder model. Requires some trick to improve “attention”, the recommended being DDC (“double decoder consistency”).
  • VITS: an end-to-end model, combining the GlowTTS encoder and HiFiGAN vocoder.
  • Tortoise: a GPT-like end-to-end model, using the Univnet vocoder. Claimed to be very slow compared to VITS.

I tried some of the models, here are my notes. The testcase was a paragraph from LWN which is the sort of thing I want this to read aloud.

modelsizetime to rendernotes
en/sam/tacotron-DDC324MB48.5sglitchy in places, listenable
en/blizzard2013/capacitron-t2-c150_v2348MB13.6sgood quality speech, but sounds like a patchwork quilt of different speakers and environments
en/jenny/jenny1.7GB207.3very high quality, slow to render
en/ljspeech/vits–neon139MB47.7suneven but intelligible
en/multi-dataset/tortoise-v24.3GBdidn’t work – error in load_config

(Note that “Jenny” is in fact a VITS model.)

VITS seems to be the current state-of-the-art in publicly available models. This model is also what the Piper TTS engine uses.

The final installment, if I get to it, will be a quick guide on getting Piper TTS to work with Speech Dispatcher. See you then!

June 06, 2023

snegg - Python bindings for libei

After what was basically a flurry of typing, the snegg Python bindings for libei are now available. This is a Python package that provides bindings to the libei/libeis/liboeffis C libraries with a little bit of API improvement to make it not completely terrible. The main goal of these bindings (at least for now) is to provide some quick and easy way to experiment with what could possibly be done using libei - both server-side and client-side. [1] The examples directory has a minimal EI client (with portal support via liboeffis) and a minimal EIS implementation. The bindings are still quite rough and the API is nowhere near stable.

A proper way to support EI in Python would be to implement the protocol directly - there's no need for the C API quirkiness this way and you can make full use of things like async and whatnot. If you're interested in that, get in touch! Meanwhile, writing something roughly resemling xdotool is probably only a few hundred lines of python code. [2]

[1] writing these also exposed a few bugs in libei itself so I'm happy 1.0 wasn't out just yet
[2] at least the input emulation parts of xdotool

Taking back what was lost, aka the many challenges of a not so important subsystem rewrite

Being a visually impaired person poses many challenges. You not only need special software, like, for example, the Linux screen reader Orca, but you more than often find out about a redesign in a desktop application because suddenly, something breaks. And, if the redesign affects not only the application, but the library used for creating them, you can be almost sure that something slips through the cracks. And, unfortunately, the redesign of the accessibility infrastructure in GTK 4 was no exception.

So, that’s the reason why I am now working at Red Hat and trying to fix these issues.

The need for a change

But, first, why break something which was working pretty well in GTK 3?

I was not part of these discussions, so I can provide only a limited answer to that question, but one of the reasons was to remove a dependency on the ATK library, which complicated building, was an additional dependency, and it imposed restrictions on how the accessibility subsystem looks like.

Another reason was to make the APIs used by application developers much more similar to the VAI-ARIA ones, which should allow pushing accessibility much more easily than before.

The reality

Actually, many of these goals were indeet fulfilled. The build was simplified, and the APIs began looking sane. And the APIs actually had some documentation, so that was good as well.

Unfortunately, how the visually impaired people perceive an application is not only dependent on the toolkit, but on the screen reader as well. And, that started to pose a number of issues. Some of them were fixed quickly, some of them were fixed after a long time, and some of these aren’t fixed even today.

And it does not help that the community of people which would notice these issues is so small. So, what was wrong?

The missing value change notifications

Let’s start with something simple. Imagine a slider, like, for example, a volume slider in a music player. When you somehow change its value, you would expect that you actually know about such an event. In GTK 3, you did. But in GTK 4, until GTK 4.10, you did not.

The reason for that behavior is actually quite simple in this case. For a screen reader to know that a value of a control changed, it needs to receive a notification about that fact, because the screen reader is not constantly checking the entire tree of accessibility objects for a modification, that would be way too inefficient. Unfortunately, the notification was not dispatched in this particular case.

But, getting this particular issue fixed was not hard, actually, much more time took before anyone noticed it.

The cases of missing information

Another group of issues, whose fix was relatively simple, could be in general summarized as missing information. There was quite some of these, but their solution was relatively simple.

In the first group of issues, a semantic property of a GTK control, for example the indication that a control is required, or has additional details associated with it, could be set by the programmer of a GTK application (not that many did that), but the fact was not communicated to the API which a screen reader uses to get this kind of information. Fortunately, AT-SPI2, the software communicating the accessibility information under Linux, already had all the needed concepts, so it was just a simple matter of returning the correct value when translating the GTK properties.

The second group of issues was the same, so, there were semantic definitions missing, but now, they did not have GTK equivalents at all. Fortunately, it was not too complicated to add them to GTK and them bridge them to AT-SPI2. This was, for example, the case of the visited state for a link, or a link button.

The VAI-ARIA world is too ideal

As previously mentioned, one of the reasons why rewrite the entire accessibility subsystem of GTK was to make it much more similar to the VAI-ARIA model. In that model, you have elements with a role, and some properties, which can hold a value. Because GTK wanted to be similar, it adopted a similar model with roles (like a button), properties (like a detailed description of the control) and states (like the checked state for a button). And, not to deviate from VAI-ARIA too much, both states and properties could hold a value. Unfortunately, the, much older, AT-SPI2 architecture counts with the same basic building blocks, however, AT-SPI2 states are either present, or not, e. g. are just a set of binary flags. That is the reason, for example, why you need both a checked and checkable state in AT-SPI2. If you did not have them, you could not on that layer tell, for example, whether a list item is not checked, but can be (that would be aria-checked="false" in ARIA), or is not meant to be checked at all (that would be a missing aria-checked attribute).

This difference in representations of course caused issues for the translation from the GTK accessibility information to AT-SPI2. Sometimes, it was enough only to add the second state, like when representing the checked state, in other cases, like, for example, in the case of toggle buttons, it was necessary to add an additionall special role for them, so the screen reading software could report them as not pressed toggles, and not just as plain buttons, which sometimes get pressed.

The accessible tree is not always your widget tree

Maybe because there was not enough time when implementing the accessibility subsystem for the first time, it turned out rather tangled with the rest of the library. For example, even that there exists a GtkAccessible interface in GTK, which should represent an accessible object, the backend assumed that nothing else than a widget will ever be an accessible, and used the widget APIs. That worked, but at a cost. Most of the time, the widget tree was exactly the same as the tree of the accessible objects, ignoring children which were ignored in the accessibility tree, but they don’t cause any trouble, so this is not the issue. But, when implementing the GtkStack widget (a container containing a number of widgets which could be switched on demand), it was decided that the pages will be accessible objects as well.

But, because the GtkStack pages did not implement the GtkAccessible interface, the implementation of the AT-SPI2 backend had to special case them everywhere. The pages unfortunately implemented not only the Accessible interface from AT-SPI2, but a bunch of others as well.

To fix this, it was necessary to extend the G_tkAccessible interface, so you could, for example, ask for the boundaries of the object in terms of its parent coordinate system. Then, it was just a question of rewriting everything related to the core accessibility concepts in terms of the GtkAccessible interface, a few times changing its methods, so they resemble the GTK interfaces rather than the interfaces from AT-SPI2 (mainly the methods for enumerating the children of an accessible object), waiting for a review, which was not short, because the change was quite substantial, and then, in GTK 4.10, the work was merged.

The way ahead

Of course, the previously mentioned issues are not the end of the story, we’ve just started. For example, a lot of the GTK’s accessibility subsystem still assumes that accessible objects are specific widgets, for example, the implementation of the AT-SPI2 AccessibleText interface does it, as, unfortunately, not everything in GTK implements GtkEditable, so we can’t use just that.

But, having the accessibility support in the toolkit is a good beginning, however, it is still necessary to make the applications use the tools. Many of the fixes will be simple, but many will require more work, or changes in GTK itself.

Conclusion

So, now you know something about how the accessibility in GTK fares, what needed to be changed, and what are the plans. I hope you learned something interesting, and if you want to help, I’d be glad to answer any questions.

June 05, 2023

GJS plugins in GNOME Builder

As I mentioned in my last post, Builder has switched to GJS as it’s dynamic language for plugins. We already support a number of compiled languages including C, C++, Rust, and Vala.

Previously we had used PyGObject. Do to the lack of GTypeInstance support in PyGObject, that isn’t an option currently. I already ported all of Builder’s plugins written in Python to C over the course of a week last summer. That ended up making things both more stable and allow us to ship the GTK 4 port on time.

This past year I wrote a new async/futures framework for GLib called libdex which provides Fibers, Futures, Channels, await, threadpools, io_uring support, and more. That tool heavily uses the same GTypeInstance features that GTK 4 uses.

GJS has improved a lot over the years due to how it is being maintained and it’s importance in the GNOME Shell stack. I’d like to double down on that so Builder can benefit from their hard work. Therefore, if you want to write plugins in JavaScript and maintain them upstream, that’s something I’m happy to see happen.

You can see some examples for how to write a JavaScript plugin for Builder in the examples directory.