♦ Happy New Year 2022!

The past year 2021 brought us all both great and sad news, with the world still fighting the COVID pandemic. Gentoo is going strong however, and we are happy to present once more our review of the events of the last 12 months. Read on for new developers, exciting changes and improvements, and up-to-date numbers on Gentoo development.

The number of commits to the main ::gentoo repository has once more clearly grown in 2021, from 104507 to 126920, i.e., by 21%. While the number of commits by external contributors, 11775, has remained roughly constant, this number now distributes across 435 unique external authors compared to 391 last year. We may have recruited some of the top contributors. ;)

Contributions to GURU, our user-curated repository with a trusted user model, have increased enormously. We count 4702 commits, up by 73% from 2725 in 2020. The number of contributors has grown even more, to 119, up by 116% from 55 in 2020. Please join us there and help packaging the latest and greatest software!

On our bugtracker bugs.gentoo.org, the number of new bug reports decreased slightly, with 24056 bugs opened in 2021, compared to 25500 in 2020. However, more reports were closed this year, with 24076 bugs resolved in 2021, compared to 23500 in 2020. The ongoing tinderbox efforts as well as the overall high level of activity seem to be paying off!

In the past year 2021 we have gained an outstanding number of seven new Gentoo developers, much more than in recent years. In chronological order:

1. John Helmert III (ajak): ♦ John was the first one to join in February. He’s focusing on the never-ending security work, wrangling bugs and issuing GLSAs, but also on developing the internal applications and infrastructure of the security team. We will hopefully have a fresh new GLSAmaker soon!

2. Andrew Ammerlaan (andrewammerlaan): ♦ Andrew signed up in May and is well known for working on our scientific software stack (specifically physics and electronics), and also handling user contributions for both the Gentoo repository and the sci overlay. Beyond this he active in the GURU team and also in Python packaging.

3. Ionen Wolkens (ionen): ♦ Ionen started in June and by now is active in many corners of Gentoo. His specific focus area, however, is games, games, games! In addition, he has also taken over one of our somewhat “special fun” packages, nvidia-drivers, and is the author of a whole set of development tools …

4. Florian Schmaus (flow): ♦ Also having started in June, Florian is busy with Java support, co-administrating the GURU overlay, and the proxy maintenance team. In addition he contributes to Erlang packaging - one of the more exotic programming languages present in Gentoo.

5. Arthur Zamarin (arthurzam): ♦ Next, in August, came Arthur. He’s contributing a lot to our Python team, keeping the large number of Python packages maintained there up-to-date. In addition, he recently joined several architecture teams, so we can keep offering Gentoo for highly diverse hardware.

6. Jakov Smolić (jsmolic): ♦ Our second new recruit in August was Jakov. Master of odd jobs, he’s fixing bugs across the gentoo tree, solving QA problems, and also weeding out old packages. Last but not least, he has also joined our recently renewed architecture team efforts.

7. Maciej Barć (xgqt): ♦ Finally, November brought us Maciej. He’s coming from the mathematics corner, and consequently his areas of specialization are scientific and in particular mathematical packages, Scheme, but also, for example, OCamML.

♦ Very sad news reached us in February. Kent Fredric (kentnl), a driving force behind our Perl and Rust efforts, died in a drowning accident - just when he had moved to Florida to start a new phase in his life. We will all remember his enthusiasm, helpfulness and love for detail, and wish his family all the best.

Let’s look at the major changes and improvements of 2021 in Gentoo now.

• ♦ Musl: Stage 3 tarballs for the alternative libc musl are now built using the main Gentoo repository only and have been published for several more arches and configurations. Work is ongoing to import more musl-related fixes and support patches from the musl overlay, with the objective that musl-based installations eventually work out-of-the-box in Gentoo.

• libxcrypt: GNU glibc based installations have this year migrated from the deprecated internal crypt support to the external, new libxcrypt. With this we follow several other distributions; we gain modern algorithm support for one-way hashing of passwords and much easier bugfixing outside the glibc release cycle.

• ROCm: the AMD open software platform for high performance / hyperscale GPU computing is now fully packaged in Gentoo, thanks to a contribution within the Summer 2021 Open Source Promotion Plan OSPP of the Chinese Academy of Sciences and the openEuler community. Stay tuned for ROCm-enabled applications from Gentoo, such as Numba, CuPy, TensorFlow, and PyTorch.

• ♦ Python: In the meantime the default Python version in Gentoo has reached Python 3.9. Additionally we have also Python 3.10 available stable, which means we’re fully up to date with upstream, and our Python has gained support for link-time and profile-guided optimization (LTO and PGO) during compilation.

• Themes Project: The Themes Project was created to maintain X11 themes and to unify their structure.

• Stable but up-to-date: As examples of the fast pace of Gentoo, our stable set contains among other things gcc 11.2, glibc 2.33, binutils 2.37, LibreOffice 7.1.7, KDE Frameworks 5.88, Plasma 5.23.4, Gear 21.08.3, GNOME 40, and many more packages. If you want to go bleeding edge, then the very latest code releases are often available as testing packages.

• ♦ PPC64: The PowerPC profiles and downloads have seen significant updates and enhancements. Several new ppc64 little-endian profiles (desktop, Gnome, …) have been added to the Gentoo repository. Our weekly updated downloads now include little-endian stages optimized for the POWER9 CPU series, and big- and little-endian hardened musl stage files.

• ♦ RISC-V: Support for RISC-V has improved enormously over the past year. Modern desktop environments such as KDE Plasma, Gnome, but also Lxde, Xfce4, and Enlightenment are fully available, as are other packages ranging from Rust to ZFS. Many more are in preparation. Gentoo is running nicely and is actively used on many of the first physical RISC-V systems. Stage files are now published weekly for all supported ABI in both systemd and OpenRC variants. We have adapted the library directory paths to those used by other distributions for better binary compatibility.

• M68k: Gentoo on Motorola 68000 is back! We have regularly updated stages for download again, and keywording of packages is ongoing.

• LoongArch64: While this is not an official Gentoo project yet, we have already received first code contributions for Gentoo on LoongArch64, a Chinese development originally based on MIPS.

• Release Engineering: This year brought big updates of our build hardware as well as improvements in Catalyst. A new AMD Ryzen 7 3700X 8-core machine at Hetzner now handles our builds for amd64, x86, alpha, m68k, and riscv (the latter via qemu); a new ARM64 Ampere Neoverse-N1 80-core machine provided by Equinix through the Works On Arm program handles arm64 and arm; and two 16-core POWER9 machines provided by OSUOSL POWER Development Hosting handle ppc64 and ppc. This means we have had the capacity to add a large variety of builds, from openrc and systemd variants to musl-based builds whereever possible.

• HPPA: We have received a donation of a fast HP Precision Architecture (PA-RISC) machine! It will be set up during the new year and significantly help both hppa stabilization / keywording efforts and the release engineering builds.

• Internal modernization: Our infrastructure team has completed two important internal milestones: the migration from 15 years of cfengine-2 configuration management to puppet, and the update of a roughly 10 years old ganeti-2 cluster to a recent ganeti version. Both steps will help a lot with managing our servers.

• GKernelCI, the Gentoo kernel testing system (see also its dashboard page), reached its v2.0 milestone. New features includes: easier to deploy (thanks to docker), addition of new architectures under test (amd64 (tested with both gcc and clang toolchains), arm, arm64, ppc64, sparc), addition of kselftest check (kernel self test tool), and sharing results with KernelCI for supporting upstream Kernel testing and development.

• Online Gentoo workshops: A series of online workshops in German language started in 2021. The meetings take place in BBB every 2 months on the 3rd Saturday of the month. The events have been very well received, and we also want to provide workshops in English starting on 2022-02-19. All events are listed on gentoo-ev.org/.

• ♦ The move to Libera Chat: After major changes in the governance of Freenode IRC, Gentoo and many other open source projects moved their IRC presence to Libera Chat. This new IRC network, founded by former Freenode staffers, has in the meantime become the de-facto replacement of Freenode; we can certainly say that we feel very welcome and at home there and have a very strong presence with over 100 Gentoo channels.

• ♦ Matrix presence: Although we continue to use IRC as our primary means of real-time communication, we have also established presence on Matrix. In addition to Gentoo developers overseeing a native Matrix channel dedicated to our distribution #gentoo:matrix.org, we now maintain a Matrix space #gentoo-linux:matrix.org which includes both the native channel and several bridged Libera Chat IRC channels.

• Experimental binary package hosting: First steps have started to also provide binary package hosting on the Gentoo mirrors.

This year the following projects have been discontinued:

• Eudev: After several years, Gentoo maintainers decided that keeping this barely modified fork of systemd-udev alive was not worth the effort, in particular since also musl-based installations now work with the original. In the meantime, maintenance of eudev has been picked up by a cross-distribution team, which means it may be available for longer.

• µClibc: Since µClibc-ng is mostly abandoned upstream, support for the µClibc profiles was dropped, and the package itself removed end of the year. Anyone interested in an alternative libc is encouraged to move to musl.

• Desktop Miscellaneous: We decided that “miscellaneous” is not really a useful way to group packages. The packages so far maintained by this project were reviewed and reassigned to dissolve the project.

Of course, if you look in detail, there has been much more news; we can’t cover everything here. We would like to thank all Gentoo developers and all who have submitted contributions for their relentless everyday Gentoo work. As a volunteer project, Gentoo could not exist without them.

And now it’s time to break out the champagne - let’s celebrate the new year 2022, let’s hope for good days, and let’s make it even more productive!

♦

Due to an unexpected breakage on our database servers, several Gentoo websites are currently down. In particular, this includes Forums, Wiki, and Bugzilla. Please visit our Infrastructure status page for real-time monitoring and eventual outage notices.

Update, Dec 5, 2021: The outage, apparently caused by a subtle bug in the depths of MariaDB and Galera, should mostly be over. Our infrastructure team is watching the database servers closely though.

Anyone following my Twitter could have seen me complaining about things happening around Python build systems frequently. The late changes feel like people around the Python packaging ecosystem have been strongly focused on building a new infrastructure focused on Python-specific package manages such as pip and flit. Unfortunately, there seems to be very little concern on distribution packagers or backwards compatibility in this process.

In this post, I’d like to discuss how the Python packaging changes are going to affect Gentoo, and what is my suggested plan on dealing with them. In particular, I’d like to focus on three important changes:

1. Python upstream deprecating the distutils module (and build system), and planning to remove it in Python 3.12.
2. The overall rise of PEP 517-based build systems and the potential for setuptools dropping UI entirely.
3. Setuptools upstream deprecating the setup.py install command, and potentially removing it in the future.

Over the years, the distutils stdlib module has been used to build setup.py scripts for Python packages. In addition to the baseline functions providing a build system CLI for the package, it provided the ability to easily extend the build system. This led both to growth of heavily customized setup.py scripts as part of some packages, as well as third-party build systems based on distutils, most notably setuptools.

This eventually led to deprecation of distutils themselves (see: PEP 632). Python 3.10 is already warning of distutils deprecation, and the current plan is to remove it in Python 3.12. Ahead of that, the development has moved to a dedicated pypa/distutils repository, and the copy of that is bundled within setuptools.

setuptools still uses the stdlib distutils by default. However, some packages already switch to the bundled copy, and upstream plans on using it by default in the future (see: Porting from Distutils).

At this point, I don’t think there is an explicit need for Gentoo to act here. However, it seems reasonable to avoid using distutils as the build system for Gentoo projects. Since the setuptools copy of distutils is different from the one included in CPython (and PyPy) and at the moment it does not carry the full set of historical Gentoo patches, it probably makes sense to test package compatibility with it nevertheless.

The use of bundled distutils copy can be forced using the following environment variable:

SETUPTOOLS_USE_DISTUTILS=local

This can be set both in the specific ebuild or in make.conf to force globally. However, please note that you can’t change the variable in place without a version bump (revision bump is insufficient). This is because switching to the local variant involves replacing the .egg-info file with a directory that is not supported by the PMS and not handled well by Portage.

Presuming that upstream is going to change the default sooner than later (and therefore unleash the breakage upon us), I think the cleanest way forward is to:

1. Perform some initial testing (via tinderboxes).
2. Enable SETUPTOOLS_USE_DISTUTILS=local when DISTUTILS_USE_SETUPTOOLS!=no (variable name similarity is coincidental) via eclass.
3. Deprecate DISTUTILS_USE_SETUPTOOLS=no, requesting maintainers to switch when bumping packages to new versions.

The purpose of this plan is to have a good chance of testing the new default and migrating as many packages as possible before upstream forces it in place. The change of distutils provider on packages already using setuptools should be relatively safe. On the other hand, for packages using pure distutils it should happen through version bumps, in order to avoid file-directory collisions mentioned before. At the same time, the change of DISTUTILS_USE_SETUPTOOLS value will be necessary since setuptools dependency will now be necessary to provide the distutils override.

I have requested the initial tinderbox testing already. If everything goes good and we decide to follow with the plan, I will provide detailed instructions later. Please do not update the ebuilds yet.

PEP 517 (and a few more related PEPs) define a new infrastructure for installing Python packages. Long story short, they define a consistent API that can be exposed by an arbitrary build system to support using it from any package manager. Sounds great, right? Well, I’m not that enthusiastic.

Before I get to my reasons, let’s shortly summarize how building packages is supposed to work in PEP 517 world. Every project supplies at least a minimal pyproject.toml file that specifies the package providing the build system and the path to a module providing its entry points. You read that file, install the necessary packages, then call the appropriate entry point to get a wheel. Then you install the wheel. Roughly.

Firstly, TOML. This is something I’ve been repeating for quite some time already, so I’ll just quickly go over it. I like TOML, I think it’s a reasonable choice for markup. However, without a TOML parser in stdlib (and there’s no progress in providing one), this means that every single build system now depends on tomli, and involves a circular dependency. A few months back, every single build system depended on toml instead but that package became unmaintained. Does that make you feel confident?

Secondly, customization. We do pretty heavy customization of distutils/setuptools behavior at this point — build paths, install paths, the toolchain. It is understandable that PEP 517 utilizes the black box approach and doesn’t attempt to do it all. Unfortunately, the build systems built on top of PEP 517 so far seem to focus on providing an all-in-one package manager rather than a good build tool with customization support.

Thirdly, wheels. PEP 517 pretty much forces everyone into using the wheel package format, completely ignoring the fact that it’s neither the simplest solution, nor a good fit for distributions. What we lack is a trivial “put all files into a directory” entry point. What we get instead if “pack everything into a zip, and then use the next tool to unzip them”. Sure, that’s not a big deal for most packages but I just hate the idea of wasting electricity and user’s time to compress something just so it gets uncompressed back afterwards.

PEP 660 gives some hope of avoiding that by providing “editable install” support. Unfortunately, it’s so bleak it practically doesn’t specify anything. In practice, a PEP 660 editable install is usually a .dist-info + .pth file that adds source directory to sys.path — which means no files are actually installed, and it does not make it any easier for us to find the right files to install. In other words, it’s completely useless.

I have spent significant time looking for a good solution and found none so far. Back in the day, I wrote pyproject2setuppy as a stop-gap solution to install PEP 517-based packages via setuptools without having to package the new build systems (including their NIH dependencies) and figure out how to make them work sanely within our package framework. As of today, I still don’t see a better solution.

Given that setuptools seems to be aiming towards removing the CLI entirely and distutils is no longer maintained, I suspect that it is inevitable that at some point we’re going to have to bite the bullet one way or another. However, I don’t plan on making any changes for the time being — as long as setup.py install continues working, that is. When this is no longer feasible, we can research our options again.

At last, the final event that puts everything else into perspective: the setuptools upstream has deprecated the install command. While normally I would say “it’s not going to be removed anytime soon”, the indiscriminate use_2to3 removal suggests otherwise.

Just a quick recap: setuptools removed the use_2to3 support after it being deprecated for some time, summarizing it with “projects should port to a unified codebase or pin to an older version of Setuptools”. Surely, nose, a project that hasn’t seen a single commit (or accepted user patches) since 2016 is going to suddenly make a release to fix this. In the end, all the breakage is dumped on distribution packagers.

The install command removal is a bigger deal than that. It’s not just few old packages being broken, it’s whole workflows. I’ve been considering switching Gentoo to a different workflow for some time, without much effect. Even if we bite the bullet and go full PEP 517, there’s another major problem: there are projects that override the install command.

I mean, if we indiscriminately switched to installing without the install command, some packages would effectively be broken silently — they would e.g. stop installing some files. The biggest issue is that it’s non-trivial to find such packages. One I know about is called Portage.

At this point, I don’t think it’s worthwhile to put our effort into finding a replacement for setup.py install. We can cross that bridge when we get to it. Until then, it seems an unnecessary work with a fair breakage potential.

In the end, it’s still unclear what would be the best solution. It is possible we’re going to continue converting flit and poetry into setuptools to avoid having to maintain support for multiple build processes. It is possible we’re going to hack on top of existing PEP 517 tooling, or build something or own. It’s quite probable that if I find no other solution, I’m going to try monkey-patching the build system to copy files instead of zipping them, or at least disable compression.

The Python ecosystem is changing constantly, and the packaging aspect of it is no different. The original distutils build system has eventually evolved into setuptools, and is now being subsumed by it. Setuptools seems to be moving in the direction of becoming yet another PEP 517 build backend and indiscriminately removing features.

Unfortunately, this is all happening without much of a concern for backwards compatibility or feature parity. The Python developers are focused on building their own packaging infrastructure and have no interest in providing a single good workflow for distribution packagers. It is really unfortunate given that many of them rely on our work to build the environments they use to work.

At this point, our immediate goal is to get ready for distutils removal and the setuptools switch to the bundled distutils copy. This switch has real breakage potential for Gentoo users (because of the egg-info file/directory collision), and we need to handle the migration gracefully ahead of time. The other issues. notably setup.py install removal will also need to be handled in the future but right now the gain does not justify the effort.

While writing this post, I have missed an important limitation of PEP 517 builds. Distutils and setuptools both have a data_files feature that can be used to install arbitrary files into the system — either into subdirectories of sys.prefix (i.e. /usr) or via absolute paths. This was often used to install data files for the package but also to install manpages, .desktop files and so on.

The wheel specification as of today simply doesn’t support installing files outside the few Python-specific directories. Setuptools/wheel/pip seem to include them in wheels but it’s outside the specification and therefore likely to suffer from portability problems.

Unfortunately, there doesn’t seem to be an interest to actually resolve this. Unless I’m mistaken, both flit and poetry do not support installing files outside standard Python directories.

Just a small update: due to the uncontrolled multiplication of new build systems, with every single one of them aiming to be a proper XKCD#927 standard, I have decided to discontinue pyproject2setuppy. Having to replicate all the hacks and add new test cases for them was humongous amount of work. Gentoo is now switching to building its packages using PEP 517 entry points.

So what do we have, and how can you use it?

• The server builds an assortment of stable amd64 packages, with the use-flags as present in an unmodified 17.1/desktop/plasma/systemd profile (the only necessary change is USE=bindist).
• The packages can be used on all amd64 profiles that differ from desktop/plasma/systemd only by use-flag settings. This includes 17.1, 17.1/desktop/*, 17.1/no-multilib, 17.1/systemd, but not anything containing selinx, hardened, developer, musl, or a different profile version such as 17.0.
  
• Right now, the package set includes kde-plasma/plasma-meta, kde-apps/kde-apps-meta, app-office/libreoffice, media-gfx/gimp, media-gfx/inkscape, and of course all their dependencies. More will possibly be added.
• CFLAGS are chosen such that the packages will be usable on all amd64 (i.e., x86-64) machines. 
  

To use the packages, I recommend the following steps: First, create a file /etc/portage/binrepos.conf with the following content:

[binhost]
priority = 9999
sync-uri = gentoo.osuosl.org/experimental/amd64/binpkg/default/linux/17.1/x86-64/

You can pick a different mirror according to your preferences (but also see the remarks below). Then, edit /etc/portage/make.conf, and add the following EMERGE_DEFAULT_OPTS (in addition to flags that you might already have there):

EMERGE_DEFAULT_OPTS="--binpkg-respect-use=y --getbinpkg=y"

And that's it. Your next update should download the package index and use binary packages whenever the versions and use-flag settings match. Everything else is compiled as usual.

What is still missing, and what are the limitations and caveats?

• Obviously, the packages are not optimized for your processor.
• Right now, the server only carries packages for the use-flag settings in an unmodified 17.1/desktop/plasma/systemd profile. If you use other settings, you will end up compiling part of your packages (which is not really a probem, you just lose the benefit of the binary download). It is technically possible to provide binary packages for different use-flag settings at the same URL, and eventually it will be implemented if this experiment succeeds.
  
• At the moment, no cryptographic signing of the binary packages is in place yet. This is the main reason why I'm talking about an experiment. Effectively you trust our mirror admins and the https protocol. Package signing and verification is in preparation, and before the binary package hosting "moves into production", it will be enforced.

I have been taking care of periodic mass stabilization of Python packages in Gentoo for some time already. Per Guilherme Amadio‘s suggestion, I’d like to share the workflow I use for this. I think it could be helpful to others dealing with large sets of heterogeneous packages.

The workflow requires:

– app-portage/mgorny-dev-scripts, v10
– dev-util/pkgcheck

One of the features of pkgcheck is that it can report ebuilds that haven’t been changed in 30 days and therefore are due for stabilization. This isn’t perfect but in the end, it gets the job done.

I start by opening two terminals side-by-side and entering the clone of ::gentoo on both. On one of them, I run:

stablereq-eshowkw 'dev-python/*'

On the other, I do:

stablereq-find-pkg-bugs 'dev-python/*'
stablereq-make-list 'dev-python/*'

♦

This gets me three things:

1. An open Bugzilla search for all stabilization candidates.
2. A script to call file-stablereq for all stabilization candidates open in the editor.
3. eshowkw output for all stabilization candidates in the other terminal.

The three scripts pass their arguments through to pkgcheck. Instead of passing package specifications directly, you can use a simple pipeline to grab all packages with a specific maintainer:

git grep -l [email protected] '**/metadata.xml' | cut -d/ -f1-2 | xargs stablereq-eshowkw

The candidate list given by pkgcheck is pretty rough. Now it’s time to mangle it a bit.

For a start, I go through the eshowkw list to see if the packages have any newer versions that can be stabilized. Roughly speaking, I ignore all packages that have only one stabilization candidate and I check the rest.

Checking usually means looking at git log and/or pkgdiff to see if a newer version would not be a better stabilization candidate. I update the list in the editor accordingly, either changing the desired version or removing some packages altogether (e.g. because they are release candidates or to go straight for a newer version later).

I close the eshowkw results then and do the next round of filtering via Bugzilla search. I look at the Bugzilla search for bugs affecting the stabilization candidates. Once again, I update the list accordingly. Most importantly, this means removing packages that have their stablereq filed already. This is also a good opportunity to resolve obsolete bugs.

I close the search result tabs but leave the browser open (e.g. with an empty tab) for the next step.

Now I save the list into a file, and run it via shell. This generally involves a lot of file-stablereq calls that open lots of browser tabs with pre-filled stablereqs. I suppose it would be much better to use Bugzilla API to file bugs directly but I’ve never gotten around to implement that.

I use bug-assign-user-js to assign the bugs, then submit them. With some skill, you can do it pretty fast. Point the mouse at the ‘A’ box for the package, click, shift-tab, enter, ctrl-tab, repeat.

If everything went correctly, you get a lot of new bugs filed. Now it’s a good time to look into your e-mail client and mark the mails for newly filed bugs read, before NATTkA starts processing them.

The last step is to go through bug mail resulting from NATTkA operations.

If sanity check fails, it is necessary to either add dependencies on other bugs already filed, add additional packages to the package list or file additional stablereqs.

For more complex problems, app-portage/nattka 0.2.15 provides a nattka make-package-list -s CATEGORY/PACKAGE-VERSION subcommand that can prepare a package list with dependencies. However, note that it unconditionally takes newest versions available, so you will need to verify the result and replace versions whenever necessary.

Additionally, I generally look at ALLARCHES keyword being added to bugs. If a bug is missing it, I verify whether the package is suitable, and add <stabilize-allarches/> to its metadata.xml.

While PCs still usually provide a full set of analog jacks capable of outputting a 5.1 audio, other modern hardware (such as TVs) is usually limited to digital audio outputs (and sometimes analog outputs limited to stereo sound). These outputs are either S/PDIF (coaxial or optical) or HDMI. When the PC is connected to a TV, a pretty logical setup is to carry the sound via HDMI to the TV, and from there via S/PDIF or HDMI ARC to a 5.1 amplifier. However, it isn’t always as simple as it sounds.

For a start, S/PDIF is a pretty antiquated interface originally designed to carry stereo PCM audio. The modern versions of the interface have sufficient bandwidth for up to 192 kHz sampling rate and up to 24 bit audio depth. However, in order to support more than two audio channels, the transmitted sound needs to be compressed. S/PDIF hardware usually supports MPEG, AC3 and DTS formats.

HDMI is better there. HDMI 1.2 technically supports up to 8 channels of PCM audio, 2.0 up to 32 channels. However, not all hardware actually supports that. In particular, my TV seems to only support stereo PCM input, and ignores additional channels when passed 5.1 audio. Fortunately, additional audio channels work when compressed input is used. HDMI supports more audio formats, including DTS-HD MA and TrueHD.

In this post, I’d like to shortly explore our options for making a PulseAudio-enabled Linux system output compressed 5.1 over S/PDIF or HDMI (apparently both are treated the same from ALSA/PulseAudio perspective).

It’s rather unlikely that you’ll be playing uncompressed audio these days. When playing movies, you’ll often find that the audio tracks are encoded using one of the formats supported by S/PDIF or HDMI. Rather than having mpv decode them just to have ALSA compress them again (naturally with a quality loss), why not pass the encoded audio through to the output?

♦

If you’re using HDMI, the first prerequisite is to set the PulseAudio’s configuration profile to digital stereo (found on Configuration tab of pavucontrol). This could be a bit confusing but it actually enables you to transfer compressed surround sound. Of course, this implies that you’ll no longer be able to output surround PCM sound via HDMI but if you’re going to enable compressed audio output anyway, it doesn’t matter.

♦

Then, you need to enable support for additional output formats. If you’re using pavucontrol, the relevant checkboxes can be found on Output Devices tab, hidden under Advanced. Tick off all that your connected device supports (usually all).

Finally, you have to enable S/PDIF passthrough (the same option is used for HDMI) in mpv, via ~/.config/mpv/mpv.conf:

audio-spdif=ac3,dts,eac3
audio-channels=5.1

The full list of formats can be found in mpv(1) manpage.

If everything works fine, you’re going to see something like the following in mpv output:

AO: [alsa] 48000Hz stereo 2ch spdif-ac3

(ignore the stereo part, it is shown like this when passing compressed surround sound through)

Note that audio passthrough requires exclusive access to the sound card, i.e. you won’t be able to use it simultaneously with sound from other apps.

While passthrough is often good enough for watching movies, it is not a universal solution. If, say, you’d like to play a game with surround sound, you need the regular audio output to support it. Fortunately, there is a relatively easy way to use ALSA plugins to enable transparent compression and make your S/PDIF / HDMI output 5.1-friendly.

For a start, you need to install an appropriate ALSA plugin. If you’d like to use AC3 audio, the plugin is found in media-plugins/alsa-plugins[ffmpeg]. For DTS audio, the package is media-sound/dcaenc[alsa].

The next step is adding the appropriate configuration to /etc/asound.conf. The snippet for AC3 is:

pcm.a52 {
  @args [CARD]
  @args.CARD {
    type string
  }
  type rate
  slave {
    pcm {
      type a52
      bitrate 448
      channels 6
      card $CARD
    }
  rate 48000
  }
}

The version modified for DTS is:

pcm.dca {
  @args [CARD]
  @args.CARD {
    type string
  }
  type rate
  slave {
    pcm {
      type dca
      channels 6
      card $CARD
    }
  rate 48000
  }
}

Honestly, it’s some black magic how it works but somehow PulseAudio just picks it up and starts accepting 5.1 sound, and the TV happily plays it.

Finally, the Ubuntu Community wiki suggests explicitly setting sampling rate in PA to avoid compatibility issues. In /etc/pulse/daemon.conf:

default-sample-rate = 48000

• The Well-Tempered Computer: S/PDIF
• Wikipedia: HDMI
• Kodi Wiki: PulseAudio
• Reddit: HD Audio HDMI passthrough setup
• Ubuntu Community Help Wiki; DigitalAC-3Pulseaudio

♦

Following some technical reorganization and the introduction of new hardware, the Gentoo Release Engineering team is happy to offer a much-expanded set of stage files for download. Highlights are in particular the inclusion of musl-based stages and of POWER9-optimized ppc64 downloads, as well as additional systemd-based variants for many architectures.

For amd64, Hardened/SELinux stages are now available directly from the download page, as are stages based on the lightweight C standard library musl. Note that musl requires using the musl overlay, as described on the page of the Hardened musl project.

For ppc, little-endian stages optimized for the POWER9 CPU series have been added, as have been big- and little-endian Hardened musl downloads.

Additionally, for all of amd64, ppc64, x86, and arm, stages are now available in both an OpenRC and a systemd init system / service manager variant wherever that makes sense.

This all has become possible via the introduction of new build hosts. The amd64, x86 (natively), arm (via QEMU), and riscv (via QEMU) archives are built on an AMD Ryzen™ 7 3700X 8-core machine with 64GByte of RAM, located in Hetzner’s Helsinki datacentre. The ppc, ppc64, and ppc64le / power9le builds are handled by two 16-core POWER9 machines with 32GByte of RAM, provided by OSUOSL POWER Development Hosting.

Further, at the moment an arm64 (aka aarch64) machine with an 80-core Ampere Altra CPU and 256GByte of RAM, provided by Equinix through the Works On Arm program, is being prepared for improved native arm64 and arm support, so expect updates there soon!

Three years ago, I had the pleasure of announcing EAPI 7 as a major step forward in our ebuild language. It introduced preliminary support for cross-compilation, it finally provided good replacements for the last Portagisms in ebuilds and it included many small changes that made ebuilds simpler.

Only a year and a half later, I have started working on the initial EAPI 8 feature set. Similarly to EAPI 6, EAPI 8 was supposed to focus on small changes and improvements. The two killer features listed below were already proposed at the time. I have prepared a few patches to the specification, as well as the initial implementation of the respective features for Portage. Unfortunately, the work stalled at the time.

Finally, as a result of surplus of free time last month, I was able to resume the work. Along with Ulrich Müller, we have quickly prepared the EAPI 8 feature set, got it pre-approved, prepared the specification and implemented all the features in Portage and pkgcore. Last Sunday, the Council has approved EAPI 8 and it’s now ready for ~arch use.

What’s there in EAPI 8? Well, for a start we have install-time dependencies (IDEPEND) that fill a gap in our cross-compilation design. Then, selective fetch/mirror restriction make it easier to combine proprietary and free distfiles in a single package. PROPERTIES and RESTRICT are now accumulated across eclasses reducing confusion for eclass writers. There’s dosym -r to create relative symlinks conveniently from dynamic paths. Plus bunch of other improvements, updates and cleanups.

Read the full article