On 04/04/2024 15:24, Eddie Chapman wrote: > Since there appears to be some interest I'll put together a single email > to the list later today detailing everything, as I needed to do more > things overall in addition to replacing /usr/bin/xz. Below is a guide I've written to removing app-arch/xz-utils in case anyone else wants to do so. Attached is the current version of the Bash wrapper script I now use in place of /usr/bin/xz Comments, corrections on anything technical in the guide or script are welcome, apart from flames about how this is ridiculous and unnecessary :-). Best wishes, Eddie ==== Guide to removing xz utils on a Gentoo system ==== === Introduction === This guide is for people who wish to remove xz utils (app-arch/xz-utils) from a Gentoo system. I've been able to remove xz utils from two Gentoo workstations with 2412 packages and KDE 5.x as the desktop, and it has not been painful at all. I've gone on to remove it from several Gentoo server systems without any pain. These are all SElinux systems. In this guide we replace app-arch/xz-utils with app-arch/p7zip which will do all the work of uncompressing xz distfiles for Portage going forward. It works perfectly fine for that right now. I've written a bash wrapper script which is designed to be installed as /usr/bin/xz, which is referred to in the instructions below. It is attached to this email as xz.txt. It tries to takes care of decompressing .xz files transparently whenever Portage runs /usr/bin/xz, by behaving like it but using app-arch/p7zip in the background. You will need it if you want to get rid of app-arch/xz-utils. But don't blindly use it, check it yourself first of course. If you don't like it you will either need to write your own script, or hack emerge/Portage in various places to use something else to decompress xz files. You're mileage may vary with any of this, proceed at your own risk, don't blame me if you break your system or lose data. === Warnings / Caveats / Breakages === Before you do this, you should identify whether you have applications or scripts which use the Tukaani xz utils, or that link against liblzma.so.5. This could include non-Gentoo apps or scripts you run which call any of the xz utils (xz, unxz, xzgrep|xzegrep|xzfgrep, xzcat, xzcmp, xzdec, xzdiff, lzma, unlzma, lzgrep|lzegrep|lzfgrep, lzmainfo, lzmadec, lzcmp, lzdiff, lzcat). Those programs will all be gone, so you should not do this if you want or need them and cannot use alternatives. 99% of packages in Gentoo work fine without xz utils, it's just that some might optionally link against liblzma.so.5 in order to provide support for xz (de)compression along with other algorithms. We will rebuild those packages so they don't link against liblzma.so.5 anymore. xx utils is a relative newcomer to the Linux/OSS/GNU world so you will find there aren't any low level system packages that absolutely need it to do their main job. You are highly unlikely to render your system completely unbootable doing this. But removing it does carry some risk. You might discover along the way there is some application you have installed that cannot function without xz utils. You might just have to uninstall it and find an alternative, if the situation cannot be resolved by creating your own custom ebuild and tweaking configure/meson options. But worst case if you have to uninstall a package and other packages depend on it, you might have to remove them too, and I'm sure you know how that remove list can potentially turn into a long one once all deps are worked out. You will lose some things. I've had to uninstall the following two packages for now: media-gfx/gimp kde-apps/ark (and kde-apps/kdeutils-meta which depends on it) (I'll probably figure out later how to coax them into working without xz. There might even be upstream updates soon that make xz optional, who knows. I'll also need to add to my world file at some point everything that was in kde-apps/kdeutils-meta.) If you run another desktop (e.g. Gnome) I've no idea what might or might not need xz utils. The situation with your desktop environment may be worse, more painful, or impossible. You will lose lzma support in the core Python language (dev-lang/python) in 3.x versions and higher (not sure when exactly support was introduced but 2.7 does not have it, 3.11 & 3.12 do), so if you have python scripts that happen to need that, well, they will definitely throw a big error after this :-) But I was able to rebuild the 179 dev-python packages on my workstations and everything in app-portage and none of them complained. I've been able to go on and do plenty of rebuilding with Portage after this without any problem, so core Python functionality in Gentoo is fine (although see next paragraph about Gemato). There is one significant thing that breaks, which is Gemato (app-portage/gemato). Gemato requires lzma support in core python in order to do GPG signature verification. This means you will have to say goodbye (for now) to verifying upstream GPG signatures on distfiles, and verification of Portage metadata after doing an emerge --sync. These features have been added to Portage relatively recently (2022?) so are "nice to have", without them your system is just less hardened, but still with the very high level of security that Gentoo systems have has always had prior to these features, in my opinion. Personally I can live without them for now. Verifying hashes in Manifest files still works fine and that's the main thing. You may disagree in which case, well, don't do this then. I'm going to figure out an alternative way I can verify Portage metadata soon, as there are other ways if you are creative. In practise this means you have to use USE="-verify-sig" for every emerge with a package that has a corresponding sec-keys package, and you have to set: sync-rsync-verify-metamanifest = no in files in /etc/portage/repos.conf/ But after doing that all works fine. Here's some other very minor things you might lose if you are currently using them: - KDE users will lose xz compression support from KArchive (kde-frameworks/karchive). AFAICT this has NOT had any impact on my own KDE experience, I've not seen any errors and everything I use works fine in my KDE sessions. KArchive will still support GZip, BZip2 and Zstd, just not xz. I suspect nothing that uses KArchive is using xz by default, but I'm not completely sure. All I know is my KDE sessions are running fine without it, and I can do everything in KDE I did before (apart from use Ark of course, see above). I don't know anything about KArchive. Full details of compression support in KArchive are at https://api.kde.org/frameworks/karchive/html/classKCompressionDevice.html - Portage binary packages: You cannot use xz compression if you create Portage binary packages. You will need to use one of bzip2, gzip, lz4, lzip, lzop, or zstd in BINPKG_COMPRESS in make.conf instead of xz (if that is what you were using, or is it the default?). I have always used gzip so no probs for me, creating binary packages works fine, I've already updated several Gentoo systems from many binary packages I've created using gzip without xz utils installed. - Grub bootloader: If you happened to have been using the optional, not used by default, --compress argument for grub-install, and you happen to have chosen xz, well you can't anymore. You will have to use gz or lzo instead, or stop using --compress if you don't like either of those two. Grub still builds, installs, works fine without xz utils for almost everyone. But if you did happen to previously use --compress=xz with grub-install before, make sure you check out fully what you might or might not have to do before next rebooting (I have no idea, I have never used this feature, Grub has continued working fine for me after rebuilding it without xz-utils and running grub-install again on my boot drives). - Dovecot: net-mail/dovecot links liblzma.so.5 in order to support it's optional Zlib plugin ( https://doc.dovecot.org/configuration_manual/zlib_plugin/ ) for reading/writing compressed mail files. Despite the plugin being called "Zlib" it supports several different compression algorithms. At one time they supported xz, but in recent Dovecot releases they decided to deprecate it. They still support reading (not writing) xz compressed files, so when net-mail/dovecot is built, if it finds liblzma.so.5 it will use it, if it doesn't find it, it wont, and then you just have no support for xz in the Zlib plugin (again, only *if* you are using that plugin, which is not default). From what I can gather, if you use this plugin you should migrate away from using xz compressed mail files (to another supported compression). So you should do that before you do this, if that applies to you. I use Dovecot but never enabled mail file compression so this did not affect me, Dovecot has continued working fine with the mail stores I look after. - Mariadb: If you happen to make use of the optional InnoDB Page Compression feature in Mariadb ( https://mariadb.com/kb/en/innodb-page-compression/ ), and if you happen to have chosen lzma compression for that feature (not the default) rather than one of the other 5 algorithms, then that is very unlucky, you will need to change that in your MariaDB installation in order to use one of the other 5 compression algorithms instead. dev-db/mariadb during build will automatically pick up support for the compression algorithms you have installed on the system, you don't currently specify anything in the ebuild that affects that. So if you have dev-db/mariadb installed you will have to rebuild it after removing xz utils as it links against liblzma.so.5 for this feature, and on rebuilding it you will lose support for lzma in InnoDB Page Compression. If you don't know if you are using it, this sql query will tell you: SHOW GLOBAL VARIABLES LIKE 'innodb_compression_algorithm'; On my MariaDB 10.6 server it returned: +------------------------------+-------+ | Variable_name | Value | +------------------------------+-------+ | innodb_compression_algorithm | zlib | +------------------------------+-------+ So I was not using lzma and was not affected. Tested: my MariaDB 10.6 server is now using rebuilt dev-db/mariadb without liblzma.so.5 and is running with no problems. - sys-apps/fwupd might stop working properly (though it will still build fine) due to what you have to change with dev-libs/libxmlb below. I'm not sure as I haven't checked yet, I just suspect it will. So bear that in mind if you need to rely on sys-apps/fwupd at the moment. But this "might" is temporary, upstream has now decided to make lzma optional, so this will trickle down to Gentoo soon. - app-arch/rpm will probably not be able to extract some rpm archives if they are compressed with xz, but I haven't checked that yet. Though it will still build fine. This does not affect building Gentoo packages which come with .rpm distfiles (e.g. libreoffice), Portage uses rpm2tgz for that and my script takes care of the rest. === The instructions === Follow them in order. 1. Do an emerge --sync and @world update first to make sure any upgrades/updates have been applied. Makes it easier for the things you need to do after you remove xz utils. 2. Install p7zip: emerge app-arch/p7zip 3. Add -lzma to USE flags in make.conf 4. Rebuild @world. This will rebuild only a few packages which respect -lzma 5. Copy the bash wrapper script to somewhere on the machine you are doing this on (but NOT to /usr/bin yet) 6. Prepare the script to be installed. Rename it to "xz" (with no extension), set permissions to 0755, owned by root:root. 7. On an SElinux installation, set the SElinux context of the script to whatever the current /usr/bin/xz binary is set to. 8. Remove xz utils, ignoring the warning about it being part of system: emerge --unmerge app-arch/xz-utils Once it is removed Portage will tell you that it preserved liblzma.so.5. More on that below. 9. Install the bash wrapper script to /usr/bin/xz 10. Add the following line: app-arch/xz-utils-5.4.2 to /etc/portage/profile/package.provided 11. Remove kde-apps/kdeutils-meta and kde-apps/ark if you use KDE, and media-gfx/gimp if you use it: emerge --unmerge kde-apps/kdeutils-meta kde-apps/ark media-gfx/gimp 12. (optional) Add -verify-sig to USE flags in make.conf. If you do you will soon have to rebuild all packages that rely on it. If you don't, you can just add USE="-verify-sig" in front of every emerge command you have to do from now on, or add to individual packages in your package.use file. 13. Now you will need to rebuild all packages with files that rely on the preserved liblzma.so.5 library. See below for further notes about that. 14. set: sync-rsync-verify-metamanifest = no in applicable files in /etc/portage/repos.conf/ before you do your next emerge --sync 15. Eventually, you will have to rebuild all packages that have corresponding signatures in sec-keys. That's all, enjoy life without app-arch/xz-utils! But read on for more info about step 13. === Notes about Step 13 === These are the packages that I needed to rebuild on my systems before the preserved liblzma.so.5 library was finally removed by Portage: app-arch/libarchive app-arch/rpm sys-boot/grub dev-db/mariadb dev-lang/python:2.7 kde-frameworks/karchive dev-lang/python:3.11 (needs custom ebuild, see below) dev-lang/python:3.12 (needs custom ebuild, see below) net-mail/dovecot (needs custom ebuild, see below) dev-libs/libxmlb (needs custom ebuild, see last note at the bottom of this guide) There might be others on your system. In most cases just rebuilding them will be enough. Some you might be able to clone the ebuild to your local repo and tweak configure/meson options so that the package does not link against liblzma.so.5. There may be packages with issues too difficult to resolve so you might have to just uninstall them if you can't live without them :-( (or resign yourself to rolling back and having to live with xz utils) Remember you will need to specify USE="-verify-sig" for any packages that rely on that, in whichever is your preferred way. From my list I had to clone the following 3 packages to my local ebuilds directory with small modification to each in order to get them to build without linking against liblzma.so.5: net-mail/dovecot dev-lang/python:3.11 dev-lang/python:3.12 Here are 3 diffs showing what I had to change: --- /usr/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild +++ /usr/local/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild @@ -43,7 +43,6 @@ DEPEND=" app-arch/bzip2 - app-arch/xz-utils dev-libs/icu:= dev-libs/openssl:0= sys-libs/zlib:= @@ -126,7 +125,7 @@ --disable-rpath \ --with-bzlib \ --without-libbsd \ - --with-lzma \ + --without-lzma \ --with-icu \ --with-ssl \ --with-zlib \ --- /usr/portage/dev-lang/python/python-3.11.8_p1.ebuild +++ /usr/local/portage/dev-lang/python/python-3.11.8_p1.ebuild @@ -179,6 +179,7 @@ # Avoid as many dependencies as possible for the cross build. cat >> Makefile <<-EOF || die MODULE_NIS_STATE=disabled + MODULE__LZMA_STATE=disabled MODULE__DBM_STATE=disabled MODULE__GDBM_STATE=disabled MODULE__DBM_STATE=disabled @@ -328,7 +329,7 @@ fi # force-disable modules we don't want built - local disable_modules=( NIS ) + local disable_modules=( NIS _LZMA ) use gdbm || disable_modules+=( _GDBM _DBM ) use sqlite || disable_modules+=( _SQLITE3 ) use ssl || disable_modules+=( _HASHLIB _SSL ) --- /usr/portage/dev-lang/python/python-3.12.2_p1.ebuild +++ /usr/local/portage/dev-lang/python/python-3.12.2_p1.ebuild @@ -177,6 +177,7 @@ cat > Modules/Setup.local <<-EOF || die *disabled* nis + _lzma _dbm _gdbm _sqlite3 _hashlib _ssl @@ -299,6 +300,7 @@ cat > Modules/Setup.local <<-EOF || die *disabled* nis + _lzma $(usev !gdbm '_gdbm _dbm') $(usev !sqlite '_sqlite3') $(usev !ssl '_hashlib _ssl') Lastly, I needed to create a custom dev-libs/libxmlb ebuild in order to upgrade it from 0.3.14 (latest in Gentoo at time of writing) to 0.3.15. I also needed to apply a very recent patch from upstream, from this commit, which makes LZMA support optional: https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639 and I needed to make some small changes to the ebuild. So this is what you need to do at the time of writing (6th April 2024): 1. Copy the in-tree /usr/portage/dev-libs/libxmlb ebuild directory into your local ebuilds directory. 2. Rename the ebuild file from libxmlb-0.3.14.ebuild to libxmlb-0.3.15.ebuild 3. Download the raw patch, you can use this link: https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639.patch rename it to: libxmlb-0.3.15-make_lzma_optional.patch and place it in the local "files" directory. 4. Modify the new ebuild according to the diff below. Then just rebuild it. --- /usr/portage/dev-libs/libxmlb/libxmlb-0.3.14.ebuild +++ /usr/local/portage/dev-libs/libxmlb/libxmlb-0.3.15.ebuild @@ -14,15 +14,15 @@ SLOT="0/2" # libxmlb.so version KEYWORDS="amd64 ~arm arm64 ~loong ppc ppc64 ~riscv x86" -IUSE="doc introspection stemmer test +zstd" +IUSE="doc introspection -lzma stemmer test +zstd" RESTRICT="!test? ( test )" RDEPEND=" - app-arch/xz-utils dev-libs/glib:2 sys-apps/util-linux stemmer? ( dev-libs/snowball-stemmer:= ) + lzma? ( app-arch/xz-utils:= ) zstd? ( app-arch/zstd:= ) " @@ -43,6 +43,7 @@ PATCHES=( "${FILESDIR}"/${PN}-0.3.12-no_installed_tests.patch + "${FILESDIR}"/${PN}-0.3.15-make_lzma_optional.patch ) python_check_deps() { @@ -60,6 +61,7 @@ $(meson_use stemmer) $(meson_use test tests) $(meson_use zstd) + $(meson_feature lzma) ) meson_src_configure }