If you want to test the Gentoo LiveGUI, but don't feel like cutting short the precious uptime of your machine, here's a handy bash script for testing the iso files with qemu:

#!/bin/bash

qemu-system-x86_64    -m 4G \
           -enable-kvm \
           -cpu host \
           -smp 4 \
           -name "Gentoo amd64 LiveGUI" \
           --netdev user,id=vmnic,hostname=gentoovm \
           -device e1000,netdev=vmnic \
           -bios /usr/share/edk2-ovmf/OVMF_CODE.fd \
           -device intel-hda -device hda-duplex \
           -usbdevice tablet \
           -vga vmware \
           -cdrom $1

You'll need to be in the kvm group and have kvm configured in your kernel. Also you may want to adapt the memory (now 4Gbyte) and the number of CPUs (now 4) for the virtual machine. Then you can simply start the boot process with

~ $ ./qemu-livegui livegui-amd64-20220412T191925Z.iso &

Cheers!

♦

After a long break, we now have again a weekly LiveGUI ISO image for amd64 available! The 4.7 GB download, suitable for DVD burning or an USB stick, boots directly into KDE Plasma and comes with a ton of up-to-date software. This ranges from office applicactions such as LibreOffice, Inkscape, and Gimp all the way to many system administrator tools.

Now, we need your help! Let’s make this the coolest and most beautiful Linux live image ever. We’re calling for submissions of artwork, themes, actually anything from a desktop background to a boot manager animation, on the topic of Gentoo! The winning entry will be added as default setting to the official LiveGUI images, and also be available for download and installation.

Gentoo-themed artwork and branding material to make our Gentoo LiveGUI the coolest Linux live medium ever.

• Incorporates the Gentoo logo and maybe other Gentoo design elements (like Larry the Cow)
• Works for a wide range of screen resolutions etc.
• Is packaged more or less ready-to-use for our LiveGUI image
• Provides a coherent experience to the user, i.e., if it consists of different parts, these fit togehter
• Can be distributed in its entirety under the CC BY-SA 4.0 license

We could for example imagine screen backgrounds, Plasma theming, maybe even a GRUB boot menu animation or a LibreOffice splash screen… Feel free to come up with more ideas.

If you base your work on freely available source material created by others, please keep track of the sources and their licenses in an accompanying readme file.

• Do not submit anything that infringes on third-party copyrights or trademarks. While a Star Trek-themed Gentoo desktop would be cool, Paramount might object and we wouldn’t be able to distribute it. Same for My Little Pony or the Simpsons.
• Do not submit artwork falling under the not-safe-for-work (NSFW) category. We will recognize it when we see it, and we won’t be able to distribute it.
• Do not submit artwork with political or religious statements. No matter how universally acceptable you think that these are, someone will be offended by them.

The artwork should be such that kids or colleagues can walk into your office and you don’t have to quickly cover it up. :) Also, please think of your contribution in terms of the Gentoo Code of Conduct.

• Package all the relevant files into a single tar archive and upload it to a webserver of your choice, or publish the files (e.g. on github) as a single git repository.
• Add a readme file with your name and contact e-mail address, the license of the files, sources and licenses for third-party material, and detailed installation instructions
• File a bug for the release engineering team, component “LiveCD/DVD”, with the summary starting with “Artwork 2022 contest entry”, and add a link to your file.
• If you link to a git repository, please mention a tag or commit which we should use.
• By submitting your entry, you allow Gentoo to download, re-publish, and distribute your files (see also above remark about the license).

• The contest ends 31/May/2022 at 23:59 UTC.
• Please keep your files online for at least one more month after that date, so we can review and copy them.

• The jury consists of the Gentoo Council, the Release Engineering team, the Artwork team, and the Public Relations team (as of beginning of April 2022).
• The winner will be chosen by vote; depending on the amount and quality of the submissions, we may also pick a runner-up or more.
• The announcement of the winner or the winners will be made in June.

The LiveGUI image is first and foremost provided to show off Gentoo and give everyone a chance to test a full-fledged Gentoo installation. As such, we have a lot of typical “desktop applications” installed. Additionally, we tried to integrate as many system administration tools as possible, so you can also use it for everything from repartitioning your hard drives to repairing an installation.

Some of the software on the image:

• KDE Plasma as desktop environment
• Office productivity: LibreOffice, LyX, TeXstudio, XournalPP, kile
• Web browsers: Firefox, Chromium
• IRC and similar: irssi, weechat
• Editors: Emacs, vim, kate, nano, joe
• Development and source control: git, subversion, gcc, Python, Perl
• Graphics: Inkscape, Gimp, Povray, Luminance HDR, Digikam
• Video: KDEnlive
• Disk management: hddtemp, testdisk, hdparm, nvme-cli, gparted, partimage, btrfs-progs, ddrescue, dosfstools, e2fsprogs, zfs
• Network tools and daemons: nmap, tcpdump, traceroute, minicom, pptpclient, bind-tools, cifs-utils, nfs-utils, ftp, chrony, ntp, openssh, rdesktop, openfortivpn, openvpn, tor
• Backup: mt-st, fsarchiver
• Benchmarks: bonnie, bonnie++, dbench, iozone, stress, tiobench
• …

The list of targeted packages (corresponding to a world file) can be found in the catalyst specification file; we install the newest stable version in the Gentoo repository.

In addition, since - as in a normal Gentoo installation - compiler and development tools are available, you can temporarily install more software. Just run emerge --sync and then install whatever you need (though it will be kept in memory and be gone after the next reboot).

Feedback and of course bug reports are welcome! Enjoy!

The weekly builds cover at the moment for 32-bit mips2 and mips32, for 64-bit mips3 and mips64 in o32, n32, and n64 ABI - and all that for both big and little endian. Should be good as a start for just about every hardware out there.

For more information on MIPS, see the Gentoo MIPS project page (which is as of writing of this blog post still somewhat outdated), the MIPS architecture page on Wikipedia, or the MIPS corporate products page.

♦

After a long break, we now have again a weekly LiveGUI ISO image for amd64 available! The download, suitable for an USB stick or a dual-layer DVD, boots directly into KDE Plasma and comes with a ton of up-to-date software. This ranges from office applicactions such as LibreOffice, Inkscape, and Gimp all the way to many system administrator tools.

Now, we need your help! Let’s make this the coolest and most beautiful Linux live image ever. We’re calling for submissions of artwork, themes, actually anything from a desktop background to a boot manager animation, on the topic of Gentoo! The winning entry will be added as default setting to the official LiveGUI images, and also be available for download and installation.

Gentoo-themed artwork and branding material to make our Gentoo LiveGUI the coolest Linux live medium ever.

• Incorporates the Gentoo logo and maybe other Gentoo design elements (like Larry the Cow)
• Works for a wide range of screen resolutions etc.
• Is packaged more or less ready-to-use for our LiveGUI image
• Provides a coherent experience to the user, i.e., if it consists of different parts, these fit togehter
• Can be distributed in its entirety under the CC BY-SA 4.0 license

We could for example imagine screen backgrounds, Plasma theming, maybe even a GRUB boot menu animation or a LibreOffice splash screen… Feel free to come up with more ideas.

If you base your work on freely available source material created by others, please keep track of the sources and their licenses in an accompanying readme file.

• Do not submit anything that infringes on third-party copyrights or trademarks. While a Star Trek-themed Gentoo desktop would be cool, Paramount might object and we wouldn’t be able to distribute it. Same for My Little Pony or the Simpsons.
• Do not submit artwork falling under the not-safe-for-work (NSFW) category. We will recognize it when we see it, and we won’t be able to distribute it.
• Do not submit artwork with political or religious statements. No matter how universally acceptable you think that these are, someone will be offended by them.

The artwork should be such that kids or colleagues can walk into your office and you don’t have to quickly cover it up. :) Also, please think of your contribution in terms of the Gentoo Code of Conduct.

• Package all the relevant files into a single tar archive and upload it to a webserver of your choice, or publish the files (e.g. on github) as a single git repository.
• Add a readme file with your name and contact e-mail address, the license of the files, sources and licenses for third-party material, and detailed installation instructions
• File a bug for the release engineering team, component “LiveCD/DVD”, with the summary starting with “Artwork 2022 contest entry”, and add a link to your file.
• If you link to a git repository, please mention a tag or commit which we should use.
• By submitting your entry, you allow Gentoo to download, re-publish, and distribute your files (see also above remark about the license).

• The contest ends 31/May/2022 at 23:59 UTC.
• Please keep your files online for at least one more month after that date, so we can review and copy them.

• The jury consists of the Gentoo Council, the Release Engineering team, the Artwork team, and the Public Relations team (as of beginning of April 2022).
• The winner will be chosen by vote; depending on the amount and quality of the submissions, we may also pick a runner-up or more.
• The announcement of the winner or the winners will be made in June.

The LiveGUI image is first and foremost provided to show off Gentoo and give everyone a chance to test a full-fledged Gentoo installation. As such, we have a lot of typical “desktop applications” installed. Additionally, we tried to integrate as many system administration tools as possible, so you can also use it for everything from repartitioning your hard drives to repairing an installation.

Some of the software on the image:

• KDE Plasma as desktop environment
• Office productivity: LibreOffice, LyX, TeXstudio, XournalPP, kile
• Web browsers: Firefox, Chromium
• IRC and similar: irssi, weechat
• Editors: Emacs, vim, kate, nano, joe
• Development and source control: git, subversion, gcc, Python, Perl
• Graphics: Inkscape, Gimp, Povray, Luminance HDR, Digikam
• Video: KDEnlive
• Disk management: hddtemp, testdisk, hdparm, nvme-cli, gparted, partimage, btrfs-progs, ddrescue, dosfstools, e2fsprogs, zfs
• Network tools and daemons: nmap, tcpdump, traceroute, minicom, pptpclient, bind-tools, cifs-utils, nfs-utils, ftp, chrony, ntp, openssh, rdesktop, openfortivpn, openvpn, tor
• Backup: mt-st, fsarchiver
• Benchmarks: bonnie, bonnie++, dbench, iozone, stress, tiobench
• …

The list of targeted packages (corresponding to a world file) can be found in the catalyst specification file; we install the newest stable version in the Gentoo repository.

In addition, since - as in a normal Gentoo installation - compiler and development tools are available, you can temporarily install more software. Just run emerge --sync and then install whatever you need (though it will be kept in memory and be gone after the next reboot).

Feedback and of course bug reports are welcome! Enjoy!

♦

Do you want to learn more about Gentoo and contribute to your favourite free software project?! Once again, now for the 10th time, we have been accepted as a mentoring organization for this year’s Google Summer of Code!

The GSoC is an excellent opportunity for gaining real-world experience in software design and making oneself known in the broader open source community. It also looks great on a resume. Some initial project ideas can be found here, but new projects ideas are also welcome. For new projects time is of the essence: they have to be worked out, discussed with the mentors, and submitted before the April 19th deadline. It is strongly recommended that contributors refine new project ideas with a mentor before proposing the idea formally.

Potential GSoC contributors are encouraged to e-mail the GSoC admins with their name, IRC nickname, and the desired project, and discuss ideas in the #gentoo-soc IRC channel on Libera Chat. Further information can be found on the Gentoo GSoC 2022 wiki page. Those with unanswered questions should also not hesitate to contact the Summer of Code mentors via their mailing list.

♦

We have simplified the format of the downloadable file (i.e. stage 3 and iso image) signatures. Now, each of these files is accompanied by a detached GnuPG signature where the file itself is signed. The signing key remains unchanged; see our web page on release media signatures for the fingerprints.

An unsigned DIGESTS file remains available as well.

You may have noticed additional Stage 3 links marked with "desktop profile" on the Gentoo downloads page recently. So what's this about?

The only difference between a "normal" Stage 3 file and a "desktop profile" Stage 3 file is that the latter has a desktop profile selected (surprise!). As example, if a "normal" amd64 (x86-64) systemd stage uses as profile setting default/linux/amd64/17.1/systemd, then the "desktop profile" stage uses
default/linux/amd64/17.1/desktop/systemd. The packages in the @system set are exactly the same, however, in the desktop profile more use-flags are enabled, which means many additional dependencies are merged.

This leads us to the point of the "desktop profile" stages: if you do a desktop installation (for anything from KDE to Gnome or XFCE), these files give you less initial rebuilds and compiles, at the cost of a larger download. And that's all.

♦ Happy New Year 2022!

The past year 2021 brought us all both great and sad news, with the world still fighting the COVID pandemic. Gentoo is going strong however, and we are happy to present once more our review of the events of the last 12 months. Read on for new developers, exciting changes and improvements, and up-to-date numbers on Gentoo development.

The number of commits to the main ::gentoo repository has once more clearly grown in 2021, from 104507 to 126920, i.e., by 21%. While the number of commits by external contributors, 11775, has remained roughly constant, this number now distributes across 435 unique external authors compared to 391 last year. We may have recruited some of the top contributors. ;)

Contributions to GURU, our user-curated repository with a trusted user model, have increased enormously. We count 4702 commits, up by 73% from 2725 in 2020. The number of contributors has grown even more, to 119, up by 116% from 55 in 2020. Please join us there and help packaging the latest and greatest software!

On our bugtracker bugs.gentoo.org, the number of new bug reports decreased slightly, with 24056 bugs opened in 2021, compared to 25500 in 2020. However, more reports were closed this year, with 24076 bugs resolved in 2021, compared to 23500 in 2020. The ongoing tinderbox efforts as well as the overall high level of activity seem to be paying off!

In the past year 2021 we have gained an outstanding number of seven new Gentoo developers, much more than in recent years. In chronological order:

1. John Helmert III (ajak): ♦ John was the first one to join in February. He’s focusing on the never-ending security work, wrangling bugs and issuing GLSAs, but also on developing the internal applications and infrastructure of the security team. We will hopefully have a fresh new GLSAmaker soon!

2. Andrew Ammerlaan (andrewammerlaan): ♦ Andrew signed up in May and is well known for working on our scientific software stack (specifically physics and electronics), and also handling user contributions for both the Gentoo repository and the sci overlay. Beyond this he active in the GURU team and also in Python packaging.

3. Ionen Wolkens (ionen): ♦ Ionen started in June and by now is active in many corners of Gentoo. His specific focus area, however, is games, games, games! In addition, he has also taken over one of our somewhat “special fun” packages, nvidia-drivers, and is the author of a whole set of development tools …

4. Florian Schmaus (flow): ♦ Also having started in June, Florian is busy with Java support, co-administrating the GURU overlay, and the proxy maintenance team. In addition he contributes to Erlang packaging - one of the more exotic programming languages present in Gentoo.

5. Arthur Zamarin (arthurzam): ♦ Next, in August, came Arthur. He’s contributing a lot to our Python team, keeping the large number of Python packages maintained there up-to-date. In addition, he recently joined several architecture teams, so we can keep offering Gentoo for highly diverse hardware.

6. Jakov Smolić (jsmolic): ♦ Our second new recruit in August was Jakov. Master of odd jobs, he’s fixing bugs across the gentoo tree, solving QA problems, and also weeding out old packages. Last but not least, he has also joined our recently renewed architecture team efforts.

7. Maciej Barć (xgqt): ♦ Finally, November brought us Maciej. He’s coming from the mathematics corner, and consequently his areas of specialization are scientific and in particular mathematical packages, Scheme, but also, for example, OCamML.

♦ Very sad news reached us in February. Kent Fredric (kentnl), a driving force behind our Perl and Rust efforts, died in a drowning accident - just when he had moved to Florida to start a new phase in his life. We will all remember his enthusiasm, helpfulness and love for detail, and wish his family all the best.

Let’s look at the major changes and improvements of 2021 in Gentoo now.

• ♦ Musl: Stage 3 tarballs for the alternative libc musl are now built using the main Gentoo repository only and have been published for several more arches and configurations. Work is ongoing to import more musl-related fixes and support patches from the musl overlay, with the objective that musl-based installations eventually work out-of-the-box in Gentoo.

• libxcrypt: GNU glibc based installations have this year migrated from the deprecated internal crypt support to the external, new libxcrypt. With this we follow several other distributions; we gain modern algorithm support for one-way hashing of passwords and much easier bugfixing outside the glibc release cycle.

• ROCm: the AMD open software platform for high performance / hyperscale GPU computing is now fully packaged in Gentoo, thanks to a contribution within the Summer 2021 Open Source Promotion Plan OSPP of the Chinese Academy of Sciences and the openEuler community. Stay tuned for ROCm-enabled applications from Gentoo, such as Numba, CuPy, TensorFlow, and PyTorch.

• ♦ Python: In the meantime the default Python version in Gentoo has reached Python 3.9. Additionally we have also Python 3.10 available stable, which means we’re fully up to date with upstream, and our Python has gained support for link-time and profile-guided optimization (LTO and PGO) during compilation.

• Themes Project: The Themes Project was created to maintain X11 themes and to unify their structure.

• Stable but up-to-date: As examples of the fast pace of Gentoo, our stable set contains among other things gcc 11.2, glibc 2.33, binutils 2.37, LibreOffice 7.1.7, KDE Frameworks 5.88, Plasma 5.23.4, Gear 21.08.3, GNOME 40, and many more packages. If you want to go bleeding edge, then the very latest code releases are often available as testing packages.

• ♦ PPC64: The PowerPC profiles and downloads have seen significant updates and enhancements. Several new ppc64 little-endian profiles (desktop, Gnome, …) have been added to the Gentoo repository. Our weekly updated downloads now include little-endian stages optimized for the POWER9 CPU series, and big- and little-endian hardened musl stage files.

• ♦ RISC-V: Support for RISC-V has improved enormously over the past year. Modern desktop environments such as KDE Plasma, Gnome, but also Lxde, Xfce4, and Enlightenment are fully available, as are other packages ranging from Rust to ZFS. Many more are in preparation. Gentoo is running nicely and is actively used on many of the first physical RISC-V systems. Stage files are now published weekly for all supported ABI in both systemd and OpenRC variants. We have adapted the library directory paths to those used by other distributions for better binary compatibility.

• M68k: Gentoo on Motorola 68000 is back! We have regularly updated stages for download again, and keywording of packages is ongoing.

• LoongArch64: While this is not an official Gentoo project yet, we have already received first code contributions for Gentoo on LoongArch64, a Chinese development originally based on MIPS.

• Release Engineering: This year brought big updates of our build hardware as well as improvements in Catalyst. A new AMD Ryzen 7 3700X 8-core machine at Hetzner now handles our builds for amd64, x86, alpha, m68k, and riscv (the latter via qemu); a new ARM64 Ampere Neoverse-N1 80-core machine provided by Equinix through the Works On Arm program handles arm64 and arm; and two 16-core POWER9 machines provided by OSUOSL POWER Development Hosting handle ppc64 and ppc. This means we have had the capacity to add a large variety of builds, from openrc and systemd variants to musl-based builds whereever possible.

• HPPA: We have received a donation of a fast HP Precision Architecture (PA-RISC) machine! It will be set up during the new year and significantly help both hppa stabilization / keywording efforts and the release engineering builds.

• Internal modernization: Our infrastructure team has completed two important internal milestones: the migration from 15 years of cfengine-2 configuration management to puppet, and the update of a roughly 10 years old ganeti-2 cluster to a recent ganeti version. Both steps will help a lot with managing our servers.

• GKernelCI, the Gentoo kernel testing system (see also its dashboard page), reached its v2.0 milestone. New features includes: easier to deploy (thanks to docker), addition of new architectures under test (amd64 (tested with both gcc and clang toolchains), arm, arm64, ppc64, sparc), addition of kselftest check (kernel self test tool), and sharing results with KernelCI for supporting upstream Kernel testing and development.

• Online Gentoo workshops: A series of online workshops in German language started in 2021. The meetings take place in BBB every 2 months on the 3rd Saturday of the month. The events have been very well received, and we also want to provide workshops in English starting on 2022-02-19. All events are listed on gentoo-ev.org/.

• ♦ The move to Libera Chat: After major changes in the governance of Freenode IRC, Gentoo and many other open source projects moved their IRC presence to Libera Chat. This new IRC network, founded by former Freenode staffers, has in the meantime become the de-facto replacement of Freenode; we can certainly say that we feel very welcome and at home there and have a very strong presence with over 100 Gentoo channels.

• ♦ Matrix presence: Although we continue to use IRC as our primary means of real-time communication, we have also established presence on Matrix. In addition to Gentoo developers overseeing a native Matrix channel dedicated to our distribution #gentoo:matrix.org, we now maintain a Matrix space #gentoo-linux:matrix.org which includes both the native channel and several bridged Libera Chat IRC channels.

• Experimental binary package hosting: First steps have started to also provide binary package hosting on the Gentoo mirrors.

This year the following projects have been discontinued:

• Eudev: After several years, Gentoo maintainers decided that keeping this barely modified fork of systemd-udev alive was not worth the effort, in particular since also musl-based installations now work with the original. In the meantime, maintenance of eudev has been picked up by a cross-distribution team, which means it may be available for longer.

• µClibc: Since µClibc-ng is mostly abandoned upstream, support for the µClibc profiles was dropped, and the package itself removed end of the year. Anyone interested in an alternative libc is encouraged to move to musl.

• Desktop Miscellaneous: We decided that “miscellaneous” is not really a useful way to group packages. The packages so far maintained by this project were reviewed and reassigned to dissolve the project.

Of course, if you look in detail, there has been much more news; we can’t cover everything here. We would like to thank all Gentoo developers and all who have submitted contributions for their relentless everyday Gentoo work. As a volunteer project, Gentoo could not exist without them.

And now it’s time to break out the champagne - let’s celebrate the new year 2022, let’s hope for good days, and let’s make it even more productive!

♦

Due to an unexpected breakage on our database servers, several Gentoo websites are currently down. In particular, this includes Forums, Wiki, and Bugzilla. Please visit our Infrastructure status page for real-time monitoring and eventual outage notices.

Update, Dec 5, 2021: The outage, apparently caused by a subtle bug in the depths of MariaDB and Galera, should mostly be over. Our infrastructure team is watching the database servers closely though.

Anyone following my Twitter could have seen me complaining about things happening around Python build systems frequently. The late changes feel like people around the Python packaging ecosystem have been strongly focused on building a new infrastructure focused on Python-specific package manages such as pip and flit. Unfortunately, there seems to be very little concern on distribution packagers or backwards compatibility in this process.

In this post, I’d like to discuss how the Python packaging changes are going to affect Gentoo, and what is my suggested plan on dealing with them. In particular, I’d like to focus on three important changes:

1. Python upstream deprecating the distutils module (and build system), and planning to remove it in Python 3.12.
2. The overall rise of PEP 517-based build systems and the potential for setuptools dropping UI entirely.
3. Setuptools upstream deprecating the setup.py install command, and potentially removing it in the future.

Over the years, the distutils stdlib module has been used to build setup.py scripts for Python packages. In addition to the baseline functions providing a build system CLI for the package, it provided the ability to easily extend the build system. This led both to growth of heavily customized setup.py scripts as part of some packages, as well as third-party build systems based on distutils, most notably setuptools.

This eventually led to deprecation of distutils themselves (see: PEP 632). Python 3.10 is already warning of distutils deprecation, and the current plan is to remove it in Python 3.12. Ahead of that, the development has moved to a dedicated pypa/distutils repository, and the copy of that is bundled within setuptools.

setuptools still uses the stdlib distutils by default. However, some packages already switch to the bundled copy, and upstream plans on using it by default in the future (see: Porting from Distutils).

At this point, I don’t think there is an explicit need for Gentoo to act here. However, it seems reasonable to avoid using distutils as the build system for Gentoo projects. Since the setuptools copy of distutils is different from the one included in CPython (and PyPy) and at the moment it does not carry the full set of historical Gentoo patches, it probably makes sense to test package compatibility with it nevertheless.

The use of bundled distutils copy can be forced using the following environment variable:

SETUPTOOLS_USE_DISTUTILS=local

This can be set both in the specific ebuild or in make.conf to force globally. However, please note that you can’t change the variable in place without a version bump (revision bump is insufficient). This is because switching to the local variant involves replacing the .egg-info file with a directory that is not supported by the PMS and not handled well by Portage.

Presuming that upstream is going to change the default sooner than later (and therefore unleash the breakage upon us), I think the cleanest way forward is to:

1. Perform some initial testing (via tinderboxes).
2. Enable SETUPTOOLS_USE_DISTUTILS=local when DISTUTILS_USE_SETUPTOOLS!=no (variable name similarity is coincidental) via eclass.
3. Deprecate DISTUTILS_USE_SETUPTOOLS=no, requesting maintainers to switch when bumping packages to new versions.

The purpose of this plan is to have a good chance of testing the new default and migrating as many packages as possible before upstream forces it in place. The change of distutils provider on packages already using setuptools should be relatively safe. On the other hand, for packages using pure distutils it should happen through version bumps, in order to avoid file-directory collisions mentioned before. At the same time, the change of DISTUTILS_USE_SETUPTOOLS value will be necessary since setuptools dependency will now be necessary to provide the distutils override.

I have requested the initial tinderbox testing already. If everything goes good and we decide to follow with the plan, I will provide detailed instructions later. Please do not update the ebuilds yet.

PEP 517 (and a few more related PEPs) define a new infrastructure for installing Python packages. Long story short, they define a consistent API that can be exposed by an arbitrary build system to support using it from any package manager. Sounds great, right? Well, I’m not that enthusiastic.

Before I get to my reasons, let’s shortly summarize how building packages is supposed to work in PEP 517 world. Every project supplies at least a minimal pyproject.toml file that specifies the package providing the build system and the path to a module providing its entry points. You read that file, install the necessary packages, then call the appropriate entry point to get a wheel. Then you install the wheel. Roughly.

Firstly, TOML. This is something I’ve been repeating for quite some time already, so I’ll just quickly go over it. I like TOML, I think it’s a reasonable choice for markup. However, without a TOML parser in stdlib (and there’s no progress in providing one), this means that every single build system now depends on tomli, and involves a circular dependency. A few months back, every single build system depended on toml instead but that package became unmaintained. Does that make you feel confident?

Secondly, customization. We do pretty heavy customization of distutils/setuptools behavior at this point — build paths, install paths, the toolchain. It is understandable that PEP 517 utilizes the black box approach and doesn’t attempt to do it all. Unfortunately, the build systems built on top of PEP 517 so far seem to focus on providing an all-in-one package manager rather than a good build tool with customization support.

Thirdly, wheels. PEP 517 pretty much forces everyone into using the wheel package format, completely ignoring the fact that it’s neither the simplest solution, nor a good fit for distributions. What we lack is a trivial “put all files into a directory” entry point. What we get instead if “pack everything into a zip, and then use the next tool to unzip them”. Sure, that’s not a big deal for most packages but I just hate the idea of wasting electricity and user’s time to compress something just so it gets uncompressed back afterwards.

PEP 660 gives some hope of avoiding that by providing “editable install” support. Unfortunately, it’s so bleak it practically doesn’t specify anything. In practice, a PEP 660 editable install is usually a .dist-info + .pth file that adds source directory to sys.path — which means no files are actually installed, and it does not make it any easier for us to find the right files to install. In other words, it’s completely useless.

I have spent significant time looking for a good solution and found none so far. Back in the day, I wrote pyproject2setuppy as a stop-gap solution to install PEP 517-based packages via setuptools without having to package the new build systems (including their NIH dependencies) and figure out how to make them work sanely within our package framework. As of today, I still don’t see a better solution.

Given that setuptools seems to be aiming towards removing the CLI entirely and distutils is no longer maintained, I suspect that it is inevitable that at some point we’re going to have to bite the bullet one way or another. However, I don’t plan on making any changes for the time being — as long as setup.py install continues working, that is. When this is no longer feasible, we can research our options again.

At last, the final event that puts everything else into perspective: the setuptools upstream has deprecated the install command. While normally I would say “it’s not going to be removed anytime soon”, the indiscriminate use_2to3 removal suggests otherwise.

Just a quick recap: setuptools removed the use_2to3 support after it being deprecated for some time, summarizing it with “projects should port to a unified codebase or pin to an older version of Setuptools”. Surely, nose, a project that hasn’t seen a single commit (or accepted user patches) since 2016 is going to suddenly make a release to fix this. In the end, all the breakage is dumped on distribution packagers.

The install command removal is a bigger deal than that. It’s not just few old packages being broken, it’s whole workflows. I’ve been considering switching Gentoo to a different workflow for some time, without much effect. Even if we bite the bullet and go full PEP 517, there’s another major problem: there are projects that override the install command.

I mean, if we indiscriminately switched to installing without the install command, some packages would effectively be broken silently — they would e.g. stop installing some files. The biggest issue is that it’s non-trivial to find such packages. One I know about is called Portage.

At this point, I don’t think it’s worthwhile to put our effort into finding a replacement for setup.py install. We can cross that bridge when we get to it. Until then, it seems an unnecessary work with a fair breakage potential.

In the end, it’s still unclear what would be the best solution. It is possible we’re going to continue converting flit and poetry into setuptools to avoid having to maintain support for multiple build processes. It is possible we’re going to hack on top of existing PEP 517 tooling, or build something or own. It’s quite probable that if I find no other solution, I’m going to try monkey-patching the build system to copy files instead of zipping them, or at least disable compression.

The Python ecosystem is changing constantly, and the packaging aspect of it is no different. The original distutils build system has eventually evolved into setuptools, and is now being subsumed by it. Setuptools seems to be moving in the direction of becoming yet another PEP 517 build backend and indiscriminately removing features.

Unfortunately, this is all happening without much of a concern for backwards compatibility or feature parity. The Python developers are focused on building their own packaging infrastructure and have no interest in providing a single good workflow for distribution packagers. It is really unfortunate given that many of them rely on our work to build the environments they use to work.

At this point, our immediate goal is to get ready for distutils removal and the setuptools switch to the bundled distutils copy. This switch has real breakage potential for Gentoo users (because of the egg-info file/directory collision), and we need to handle the migration gracefully ahead of time. The other issues. notably setup.py install removal will also need to be handled in the future but right now the gain does not justify the effort.

While writing this post, I have missed an important limitation of PEP 517 builds. Distutils and setuptools both have a data_files feature that can be used to install arbitrary files into the system — either into subdirectories of sys.prefix (i.e. /usr) or via absolute paths. This was often used to install data files for the package but also to install manpages, .desktop files and so on.

The wheel specification as of today simply doesn’t support installing files outside the few Python-specific directories. Setuptools/wheel/pip seem to include them in wheels but it’s outside the specification and therefore likely to suffer from portability problems.

Unfortunately, there doesn’t seem to be an interest to actually resolve this. Unless I’m mistaken, both flit and poetry do not support installing files outside standard Python directories.

Just a small update: due to the uncontrolled multiplication of new build systems, with every single one of them aiming to be a proper XKCD#927 standard, I have decided to discontinue pyproject2setuppy. Having to replicate all the hacks and add new test cases for them was humongous amount of work. Gentoo is now switching to building its packages using PEP 517 entry points.

So what do we have, and how can you use it?

• The server builds an assortment of stable amd64 packages, with the use-flags as present in an unmodified 17.1/desktop/plasma/systemd profile (the only necessary change is USE=bindist).
• The packages can be used on all amd64 profiles that differ from desktop/plasma/systemd only by use-flag settings. This includes 17.1, 17.1/desktop/*, 17.1/no-multilib, 17.1/systemd, but not anything containing selinx, hardened, developer, musl, or a different profile version such as 17.0.
  
• Right now, the package set includes kde-plasma/plasma-meta, kde-apps/kde-apps-meta, app-office/libreoffice, media-gfx/gimp, media-gfx/inkscape, and of course all their dependencies. More will possibly be added.
• CFLAGS are chosen such that the packages will be usable on all amd64 (i.e., x86-64) machines. 
  

To use the packages, I recommend the following steps: First, create a file /etc/portage/binrepos.conf with the following content:

[binhost]
priority = 9999
sync-uri = gentoo.osuosl.org/experimental/amd64/binpkg/default/linux/17.1/x86-64/

You can pick a different mirror according to your preferences (but also see the remarks below). Then, edit /etc/portage/make.conf, and add the following EMERGE_DEFAULT_OPTS (in addition to flags that you might already have there):

EMERGE_DEFAULT_OPTS="--binpkg-respect-use=y --getbinpkg=y"

And that's it. Your next update should download the package index and use binary packages whenever the versions and use-flag settings match. Everything else is compiled as usual.

What is still missing, and what are the limitations and caveats?

• Obviously, the packages are not optimized for your processor.
• Right now, the server only carries packages for the use-flag settings in an unmodified 17.1/desktop/plasma/systemd profile. If you use other settings, you will end up compiling part of your packages (which is not really a probem, you just lose the benefit of the binary download). It is technically possible to provide binary packages for different use-flag settings at the same URL, and eventually it will be implemented if this experiment succeeds.
  
• At the moment, no cryptographic signing of the binary packages is in place yet. This is the main reason why I'm talking about an experiment. Effectively you trust our mirror admins and the https protocol. Package signing and verification is in preparation, and before the binary package hosting "moves into production", it will be enforced.