* [gentoo-desktop] *kit free desktop profile @ 2012-10-20 18:07 Dominique Michel 2012-10-20 19:08 ` Canek Peláez Valdés 2012-10-20 20:07 ` Alex Efros 0 siblings, 2 replies; 9+ messages in thread From: Dominique Michel @ 2012-10-20 18:07 UTC (permalink / raw To: gentoo-desktop It is at least 20 window managers / desktops in gentoo that know nothing about sys-auth/polkit, or can be installed without it, including ROX, Razor-QT, XFCE, afterstep, *box, enlightenement, fluxbox, fvwm, fvwm-crystal and wmaker. But, and this is a big but, the 3 desktop profiles in portage depend on polkit. A desktop profile that doesn't depend on polkit can be easily done from the desktop profile with the following USE changes: USE="-consolekit -policykit -udisks -upower" Can such a desktop profile be incorporated in portage? Dominique -- "We have the heroes we deserve." ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [gentoo-desktop] *kit free desktop profile 2012-10-20 18:07 [gentoo-desktop] *kit free desktop profile Dominique Michel @ 2012-10-20 19:08 ` Canek Peláez Valdés 2012-10-21 7:46 ` Dominique Michel 2012-10-20 20:07 ` Alex Efros 1 sibling, 1 reply; 9+ messages in thread From: Canek Peláez Valdés @ 2012-10-20 19:08 UTC (permalink / raw To: gentoo-desktop On Sat, Oct 20, 2012 at 1:07 PM, Dominique Michel <dominique.michel@vtxnet.ch> wrote: > It is at least 20 window managers / desktops in gentoo that know > nothing about sys-auth/polkit, or can be installed without it, > including ROX, Razor-QT, XFCE, afterstep, *box, enlightenement, fluxbox, > fvwm, fvwm-crystal and wmaker. As you yourself mention it, most of them are *window managers*, not *desktop environments*. Of the ones above that undoubtedly qualify as desktop environments (like Xfce), they can (and do) use polkit, even if it is configurable. > But, and this is a big but, the 3 desktop profiles in portage depend on > polkit. Because they are desktop environments, not only window managers. > A desktop profile that doesn't depend on polkit can be easily done from > the desktop profile with the following USE changes: > > USE="-consolekit -policykit -udisks -upower" > > Can such a desktop profile be incorporated in portage? You can roll your own profile in /etc/portage/profile. Also, you can set "-consolekit -policykit -udisks -upower" in /etc/portage/make.conf. The idea of the profiles (AFAIU), is to get a sensible set of defaults for the *majority* of users. In the case of desktop users, this means GNOME, KDE, and probably Xfce. You don't really need a profile per every single window manager in the universe. You don't get a profile for Emacs and another for vi. Hell, we systemd users don't have a systemd profile (and honestly, I don't think we need it). You don't want the kits? Mask them in /etc/portage/make.conf. Even when I use the GNOME progile, that doesn't mask everything KDE-related; I need to explicitly put "-kde -qt4" in my USE flags. I don't think a gnome-only-and-please-exorcise-kde-and-qt-from-my-system profile is necessary. Likewise, I don't think a desktop-but-please-dont-wanna-use-kits profiles is worth the inodes it would waste. My 0.02 ${CURRENCY}? Set your USE flag, like everyone else. Regards. -- Canek Peláez Valdés Posgrado en Ciencia e Ingeniería de la Computación Universidad Nacional Autónoma de México ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [gentoo-desktop] *kit free desktop profile 2012-10-20 19:08 ` Canek Peláez Valdés @ 2012-10-21 7:46 ` Dominique Michel 0 siblings, 0 replies; 9+ messages in thread From: Dominique Michel @ 2012-10-21 7:46 UTC (permalink / raw To: gentoo-desktop Le Sat, 20 Oct 2012 14:08:55 -0500, Canek Peláez Valdés <caneko@gmail.com> a écrit : > On Sat, Oct 20, 2012 at 1:07 PM, Dominique Michel > <dominique.michel@vtxnet.ch> wrote: > > It is at least 20 window managers / desktops in gentoo that know > > nothing about sys-auth/polkit, or can be installed without it, > > including ROX, Razor-QT, XFCE, afterstep, *box, enlightenement, > > fluxbox, fvwm, fvwm-crystal and wmaker. > > As you yourself mention it, most of them are *window managers*, not > *desktop environments*. Of the ones above that undoubtedly qualify as > desktop environments (like Xfce), they can (and do) use polkit, even > if it is configurable. Another one is fvwm-crystal, and it know nothing about *kit. > > > But, and this is a big but, the 3 desktop profiles in portage > > depend on polkit. > > Because they are desktop environments, not only window managers. No, because big companies need the functionalities in polkit, and also because it is no alternative at that time for those functionalities. > > > A desktop profile that doesn't depend on polkit can be easily done > > from the desktop profile with the following USE changes: > > > > USE="-consolekit -policykit -udisks -upower" > > > > Can such a desktop profile be incorporated in portage? > > You can roll your own profile in /etc/portage/profile. Also, you can > set "-consolekit -policykit -udisks -upower" in > /etc/portage/make.conf. > > The idea of the profiles (AFAIU), is to get a sensible set of defaults > for the *majority* of users. In the case of desktop users, this means > GNOME, KDE, and probably Xfce. You don't really need a profile per > every single window manager in the universe. You don't get a profile > for Emacs and another for vi. Hell, we systemd users don't have a > systemd profile (and honestly, I don't think we need it). > > You don't want the kits? Mask them in /etc/portage/make.conf. Even > when I use the GNOME progile, that doesn't mask everything > KDE-related; I need to explicitly put "-kde -qt4" in my USE flags. I > don't think a gnome-only-and-please-exorcise-kde-and-qt-from-my-system > profile is necessary. Likewise, I don't think a > desktop-but-please-dont-wanna-use-kits profiles is worth the inodes it > would waste. gnome and kde users have their profiles. Users of other desktops that need "kit can use the desktop profile. But it is no profile for users that want to use a wm or a desktop that doesn't hard depend on *kit. I am not the only one that doesn't like and doesn't want stuffs like *kit. They are a waste of resources and cause regressions (startx that doesn't work any more without running *kit, automounting that doesn't work when it was working before), without to mention the fact that polkit is totally unmanageable for one that doesn't know java script very well. In one word, *kit is completely idiotic for many (most?) users. It is alternatives for auto-mounting and power management, alternatives that just can do the job for most users, and without the need to run daemons and to incorporate a java script interpreter into the core of the system. Again, the only ones that really need the functionalities provided by *kit are big companies. All the other users do have more interesting things to do than to learn java script, and they don't will pay for managing their system either. So, definitely, I don't want polkit and I am not the only one. http://forums.gentoo.org/viewtopic-t-933724-highlight-.html > > My 0.02 ${CURRENCY}? Set your USE flag, like everyone else. It is what I have done. http://forums.gentoo.org/viewtopic-t-938680-highlight-.html Best, Dominique > > Regards. -- "We have the heroes we deserve." ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [gentoo-desktop] *kit free desktop profile 2012-10-20 18:07 [gentoo-desktop] *kit free desktop profile Dominique Michel 2012-10-20 19:08 ` Canek Peláez Valdés @ 2012-10-20 20:07 ` Alex Efros 2012-10-21 7:50 ` Dominique Michel 2012-10-21 8:02 ` [gentoo-desktop] " Duncan 1 sibling, 2 replies; 9+ messages in thread From: Alex Efros @ 2012-10-20 20:07 UTC (permalink / raw To: gentoo-desktop Hi! On Sat, Oct 20, 2012 at 08:07:31PM +0200, Dominique Michel wrote: > USE="-consolekit -policykit -udisks -upower" I'm using fluxbox, but, of course, I need ability to run gnome and kde applications (but not gnome/kde itself). Last time I tried to switch off these USE-flags it doesn't work. Best I got is: /etc/make.conf: USE="${USE} -policykit -upower" /etc/portage/package.use # required by udisks sys-fs/udev extras gudev hwdb # required by polkit, consolekit sys-auth/pambase consolekit # required by polkit, udisks, clementine sys-auth/consolekit policykit -- WBR, Alex. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [gentoo-desktop] *kit free desktop profile 2012-10-20 20:07 ` Alex Efros @ 2012-10-21 7:50 ` Dominique Michel 2012-10-21 8:02 ` [gentoo-desktop] " Duncan 1 sibling, 0 replies; 9+ messages in thread From: Dominique Michel @ 2012-10-21 7:50 UTC (permalink / raw To: gentoo-desktop Le Sat, 20 Oct 2012 23:07:40 +0300, Alex Efros <powerman@powerman.name> a écrit : > Hi! > > On Sat, Oct 20, 2012 at 08:07:31PM +0200, Dominique Michel wrote: > > USE="-consolekit -policykit -udisks -upower" > > I'm using fluxbox, but, of course, I need ability to run gnome and kde > applications (but not gnome/kde itself). Last time I tried to switch > off these USE-flags it doesn't work. Best I got is: > > /etc/make.conf: > USE="${USE} -policykit -upower" > /etc/portage/package.use > # required by udisks > sys-fs/udev extras gudev hwdb > # required by polkit, consolekit > sys-auth/pambase consolekit > # required by polkit, udisks, clementine > sys-auth/consolekit policykit > Most parts of kde just work fine without *kit. The problem is with gnome applications, but it is alternatives in most (all?) cases, and I can live without gnome. I was never fan of it. As me, you made a choice. GNU/linux is not about freedom, but also about choice, because freedom is also the freedom of choice. I will not force you to remove clementine from your system, so don't force me to install *kit. If some day in the future, it is no other choice about *kit than to install it, I just will shift to another OS. I don't want it and this is not negotiable. If GNU/linux become like windows, thanks to polkit, I will choose the tangent and find an alternative. My choice was to remove all the apps that depend on an idiotic software that cause severe regressions into my system (startx that doesn't work without to run *kit, that when my desktop, fvwm-crystal, know nothing about it! and auto mounting that doesn't worked any more). So, the only 3 things that made polkit for me was 1) to screw up a good working system, 2) to force me to run the totaly non needed daemon that screwed up my system and is a waste of system resources, and last but not least, 3) to force me to run a software that screwed up my system and that is totally non manageable. Cheers, Dominique -- "We have the heroes we deserve." ^ permalink raw reply [flat|nested] 9+ messages in thread
* [gentoo-desktop] Re: *kit free desktop profile 2012-10-20 20:07 ` Alex Efros 2012-10-21 7:50 ` Dominique Michel @ 2012-10-21 8:02 ` Duncan 2012-10-21 13:24 ` [gentoo-desktop] @system and parallel merge speedup Alex Efros 2012-10-23 17:36 ` [gentoo-desktop] Re: *kit free desktop profile Dominique Michel 1 sibling, 2 replies; 9+ messages in thread From: Duncan @ 2012-10-21 8:02 UTC (permalink / raw To: gentoo-desktop Alex Efros posted on Sat, 20 Oct 2012 23:07:40 +0300 as excerpted: > Hi! > > On Sat, Oct 20, 2012 at 08:07:31PM +0200, Dominique Michel wrote: >> USE="-consolekit -policykit -udisks -upower" > > I'm using fluxbox, but, of course, I need ability to run gnome and kde > applications (but not gnome/kde itself). Last time I tried to switch off > these USE-flags it doesn't work. Best I got is: > > /etc/make.conf: > USE="${USE} -policykit -upower" > /etc/portage/package.use > # required by udisks > sys-fs/udev extras gudev hwdb > # required by polkit, consolekit > sys-auth/pambase consolekit > # required by polkit, udisks, clementine > sys-auth/consolekit policykit FWIW, as my gentoo experience lengthens, I've gotten *MUCH* tighter with my installed-packages and USE flags policy. I run a kde desktop, but have this in my global USE file[1]: -consolekit -policykit -udisks -udisks2 -upower Also given my kde4 desktop it's worth noting: -raptor -redland -semantic-desktop -virtuoso (Of course that means no kdepim packages at all. I got fed up with akonadi problems and switched to the gtk-based claws-mail after nearly a decade on kmail! That allowed me to kill kmail and akonadi, which allowed me to kill semantic-desktop for all of kde, which allowed me to kill rasqal, redland, virtuoso, soprano... I did it mainly to get rid of akonadi and the problems it brought, but WOW, the kde4 desktop was faster without all that extra bloatware! And I had already turned off nepomuk and strigi too, and it was STILL dramatically faster when kde was built without that junk! Sort of reminds me of all those MSWormOS users and how surprised they often are to learn how badly the malware was affecting system performance once it's cleaned up. Yes, I DID just compare the semantic-desktop crap to malware!) But: USE=udev No udisks (1 or 2), upower, consolekit or policykit installed. But, I actually prefer manual mounting of removables (significantly safer that way!), etc, and the fact that automount functionality requires udisks, which (in v1) ultimately pulls in lvm2, and I'm NOT interested in having that on my system[2], so while I tolerated it for awhile, ultimately I got rid of it. Meanwhile, the below is an admitted rather long tangent, but readers may find this output from emerge --depclean ($>> is the last line of my triple-line customized bash $PS1 prompt) rather interesting indeed, and it DOES continue the "tightened up system" policy theme I presented above: $>>emerge --depclean -p [snip the standard boilerplate] !!! You have no system list. !!! Proceeding is likely to break your installation. [more boilerplate] Packages installed: 865 Packages in world: 0 Packages in system: 0 Required packages: 865 Number to remove: 0 Empty system warning, empty world, yet 865 required packages? WTF? The empty @system set warning is accurate; the empty world not quite so, tho it's anything but standard. Here's the explanation: World first, as it's easier. =^) I'm running the (still masked) portage-2.2-alpha series for sets support, which (AFAIK) hasn't yet hit the 2.1 series. My world file is INDEED empty, but that's because I took advantage of sets support to break up the long and unsorted world list into a bunch of sets, each of which is listed in the world_sets file. (That's comparable to what I did with make.conf, as noted in footnote [1] from the mention of my global USE file.) So my /var/lib/portage/world_sets file lists sets like (jed are my initials, used here to indicate that they are my custom sets, distinct from for example the sets that the kde overlay ships, tho my custom kde sets are simply the sets from the overlay, with various packages commented out, I diff them against the overlay set twice a year when I upgrade to the next kde 4.x version minor, making adjustments as necessary then): @jed.admin @jed.kde.base.kdegames @jed.portage That's just picking three from the list of about two dozen. Here's the contents of my @jed.portage set as an example: $>>cat /etc/portage/sets/jed.portage app-portage/eclass-manpages app-portage/esearch app-portage/gentoolkit app-portage/layman app-portage/mirrorselect app-portage/portage-utils app-doc/pms So while my world file is empty, my world_sets file has about two dozen sets listed, each of which contains its own list of packages I want that sort into that category. Yes, my world file is empty, but the @world / set/ is not empty. But the depclean summary hasn't yet been updated for sets. I filed a feature-request-bug[3] on that some time ago, but obviously it's down Zac's priority list quite a way, until such time as they decide to introduce proper sets support to an unmasked portage. Worst-case, they drop the still experimental sets feature instead, and I have to recombine all the packages listed in my individual sets into one big world file again. No big deal tho sets support is nice and I'd miss it. At least I've been able to use sets in the mean time. OK, that explains the "empty" world, the world _file_ is empty, but not the world_sets file and thus not @world, but the depclean summary doesn't know about world_sets yet. What about that empty @system warning? The technical information's available in the portage (5) manpage, but it comes down to this: /etc/portage/profile/* files can be used to override the normal tree profile where a gentoo users finds it necessary to do so. Basically it gets applied on top of the normal cascading profile in the tree, adding or subtracting from it just as the contents of the make.profile dir override the contents of its cascaded parents. More specifically, the packages file contains entries like this (with or without the negation, but with the *): -*sys-devel/make -*>=sys-devel/patch-2.6.1 Each (un-negated) starred package atom found in the packages file in one of the cascaded profile dirs adds that package to the @system set. Negating the entry removes a package from @system that was added by one of the cascading parents. Now here's the deal. There are two purposes that the @system set fulfills. First, the list of packages in @system is used by catalyst to create the stage tarballs used to bootstrap a gentoo system at initial installation. Basically, it's the list of packages found in a stage3. Second, once a gentoo system is installed and running, the list of packages in @system form a core set of basic dependencies that can be assumed to be installed, so they can be omitted from an ebuild's dependencies list, dramatically shortening the dependencies list and decreasing the maintenance burden both for individual package maintainers since they have a core set of packages that can be assumed to be installed, and for core system and arch maintainers, since that core list is nicely centralized, thus much easier to change that it would be to edit thousands of individual ebuild dependencies. For the gentoo user, among other things, if you mistakenly try to unmerge something in the system set, portage gives you a much stronger warning than it would otherwise, since it's very possible doing so will break the system beyond easy recoverability, forcing a boot to a rescue disk or backup in ordered to fix the problem. But there is a very practical non-zero cost to having a package in @system. Portage is more careful with these packages and tends to merge them one at a time, thus breaking emerge's parallel merge ability and the --jobs and --load-average options that control it, when merging an @system package or a dependency of one, forcing portage back to the bad old days of serialized one-package-at-a-time merging. For people with even a quad-core system and enough memory to nicely handle multiple merges who have accordingly enabled parallel merging using the --jobs and --load-average options, then, any package that's in @system or a dependency of something in @system, dramatically slows down system updates, and even more so, the emerge --empty-tree @world that many people like to do after a major gcc upgrade or change to CFLAGS/CXXFLAGS/ LDFLAGS. Keeping @system as small as possible directly affects the speed at which updates and particularly --empty-tree rebuilds complete! So the @system package list has two purposes, only one of which actually matters once you're beyond the stage-3 point of an install, while every package on that list and all its dependencies have a dramatically increased cost in terms of merge time on a modern multicore system, since those packages can't take advantage of portage's parallel emerges ability. Once a user has been on gentoo awhile and knows his way around the system well enough that he's unlikely to make the mistake of unmerging a package he well enough knows is critical, thus making that extra warning for such a mistake unnecessary, the cost of that @system list begins to look bigger and bigger in relation to the actual benefit it continues to provide. My first gentoo install was 2004.0... 8.5 years ago. If I'm sure enough about an emerge -C to do it in the first place, that extra @system warning isn't going to stop me, so I might as well not have to deal with its cost. But here, an emerge --pretend --emptytree @system wanted to remerge 300+ packages! I forgot how many were actual @system packages, but most of them were dependencies. That's a *LOT* of packages for portage to be forcing to one-at-a-time merging! So at first I tried whittling down the number of @system deps by tweaking USE flags. That did drop the total some, but not enough! And trying to sort out individual package.use exceptions was getting complex! Simple enough solution, be rid of the whole @system list! emerge --pretend @system provided a list of package in that set, without dependencies. First I did an emerge --pretend --depclean to ensure there was nothing it wants to remove with the existing system set that I needed to decide about. (There wasn't, I routinely run revdep-rebuild and emerge --depclean after I'm finished updating. Everything was in world, or in my case in the world_sets, that needed to be. No package cruft left hanging!) Then I took that emerge --pretend @system list and create /etc/portage/ profile/packages negating entries for each of them. =:^) Reran emerge --pretend @system. What was this? Two packages still in @system along with 200+ dependencies (--emptytree lists the deps too, without it, only the actual @system packages are listed)! WTF? While I tried to figure that out, I played around with USE flags some more and with just those two packages in @system, I was able to get the total @system with deps down to 120 or so. MAJOR PROGRESS, especially when I had started with 300+, and still had 200+ with only two packages in @system. BUT NOT GOOD ENOUGH! Meanwhile, while I tried to figure out what was going on there, I did another --depclean --pretend to see what packages in @system weren't still protected as dependencies of something in @world. Turns out most of them were already dependencies of something, so only a few packages came up to be depcleaned. And actually, several of those were virtuals (example, virtual/editor, default provider nano). For most of those, I already had the package that I wanted to fill that virtual in @world (in one or another of my custom @jed.* sets), so unmerging the virtual wasn't going to hurt anything because the package I wanted to fill that virtual was still installed and in @world. So I unmerged the unnecessary virtuals. For the handful of others, after figuring out which @jed.* set I wanted each one in, I added it there. After that a --depclean --pretend came up clean again and the installed package list was stable, tho @system along with its depends was still too big for comfort. I let it sit at that point for a few days, while I tried to figure out where the other two @system entries were coming from. At first I thought they must be hard-coded, which made some sense for the one, baselayout-2. But the other one was patch, and it just made no sense that /patch/, of ALL things, would be hardcoded, but things like gcc or sed and grep weren't. Turns out there was a simple explanation. Those two @system package entries weren't simply generic entries like this: *sys-apps/baselayout *sys-devel/patch They actually appeared with version specifiers like this: *>=sys-apps/baselayout-2 *>=sys-devel/patch-2.6.1 So the generic negation wasn't cutting it. I had to do this, negating the exact same entries that were added by the in-tree profile: -*>=sys-apps/baselayout-2 -*>=sys-devel/patch-2.6.1 Viola! THAT DID IT! TOTALLY ZEROED OUT @system LIST!! =:^) Repeated the --depclean --pretend but those last couple packages were deps of something already in @world so they didn't need added. So now I have an entirely empty @system, and packages parallel-merge much more efficiently. =:^) I was so happy, I took the opportunity to do a full emerge --emptytree @world, something I hadn't done since upgrading cpu/gpu/mobo/ram to a 6- core bulldozer and changing CFLAGS/CXXFLAGS accordingly, back in July, tho I had been keeping up with updates including a kde bump, so it wasn't too bad. Between the better parallel merging of the empty @system and the upgrade from a dual-dual-core opteron 290 @2.8 (so four cores of k8 tech) on the old system, to the new six-core buldozer (fx6100) slightly overclocked to 3.6, I was rather happy with the full --emptytree @world merge time improvement. =:^) In fact, I was so happy with that, that I updated my 32-bit chroot on the same machine, where I build the image I rsync to the (32-bit-only atom n270) netbook, did an upgrade there, something that I hadn't done in a year and a half so it was a bit complicated to sort out but I managed, negated its entire @system and adjusted its custom sets appropriately, and then to be sure, did an emerge --emptytree @world there as well. =:^) Bottom line, an empty @system set really does make a noticeable difference in parallel merge handling, speeding up especially --emptytree @world rebuilds but also any general update that has a significant number of otherwise @system packages and deps, dramatically. I'm happy. =:^) --- [1] Global use file: I take advantage of the make.conf source statement to source a bunch of individual files. make.conf itself simply sources a master file (dead easy to recreate make.conf that way if it gets deleted/ overwritten), which in turn sources a bunch of individual files. ($>> is the third line of my customized bash $PS1 prompt, the first is a blank line and the second is deleted for posting.) $>>cat /etc/portage/make.conf source /etc/portage/make/master *$>>cat /etc/portage/make/master source /etc/portage/make/cflags source /etc/portage/make/collision-ignore source /etc/portage/make/features source /etc/portage/make/fs source /etc/portage/make/layman source /etc/portage/make/ldflags source /etc/portage/make/log source /etc/portage/make/makeopts source /etc/portage/make/mirrors source /etc/portage/make/net source /etc/portage/make/use source /etc/portage/make/use.expand source /etc/portage/make/other So I have a file, /etc/portage/make/use, that contains only my globally applied USE flags. There's another for CFLAGS/CXXFLAGS, another for LDFLAGS, another for the filesystem (fs) settings, another for MAKEOPTS, another for FEATURES, etc. [2] LVM2: I tried lvm2 at one point, and decided the additional complexity and negative effect on my confidence in my own ability to sanely manage disaster recovery and restore access to my data, was nothing I was interested in, and it's not worth having around on the system just to handle automount, either! FWIW I DID run with an md/raid configured system for awhile altho I recently upgraded and didn't have money for another disk so am not running it now, but md/raid was nice! =:^) [3] depclean summary: world_sets line request Clear-Text: http://bugs.gentoo.org/show_bug.cgi?id=298298 Secure: https://bugs.gentoo.org/show_bug.cgi?id=298298 -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 9+ messages in thread
* [gentoo-desktop] @system and parallel merge speedup 2012-10-21 8:02 ` [gentoo-desktop] " Duncan @ 2012-10-21 13:24 ` Alex Efros 2012-10-22 6:26 ` [gentoo-desktop] " Duncan 2012-10-23 17:36 ` [gentoo-desktop] Re: *kit free desktop profile Dominique Michel 1 sibling, 1 reply; 9+ messages in thread From: Alex Efros @ 2012-10-21 13:24 UTC (permalink / raw To: gentoo-desktop Hi! On Sun, Oct 21, 2012 at 08:02:47AM +0000, Duncan wrote: > Bottom line, an empty @system set really does make a noticeable > difference in parallel merge handling, speeding up especially --emptytree > @world rebuilds but also any general update that has a significant number > of otherwise @system packages and deps, dramatically. I'm happy. =:^) I think "@system first" and "@system not merge in parallel" rules are safe to break when you just doing "--emptytree @world" on already updated OS because it's only rebuild existing packages, and all packages while compiling will see same set of other packages (including same versions). But when upgrading multiple packages (including some from original @system and some from @world) this probably may result in bugs. As for "--emptytree @world" speedup, can you provide benchmarked values? I mean, only few packages forced to use only one CPU Core while compiling. So, merging packages in parallel may save some time mostly for doing unpack/prepare/configure/install/merge. All of them except configure actually do a lot of I/O, which most likely lose a lot in speed instead of gain when done in parallel (especially keeping in mind kernel bug 12309). So, at a glance time you may win on configure you'll mostly lose on I/O, and most of time all your CPU Cores will be loaded anyway while compiling, and doing configure in parallel to compiling unlikely save some time. This is why I think without actual benchmarking we can't be sure how faster it became (if it became faster at all, which is questionable). As for me, I found very effective way to speedup emerge is upgrading from Core2Duo E6600 to i7-2600K overclocked to 4.6GHz. This speedup compilation on my system in 6 times (kernel now compiles in just 1 minute). And to speedup most other (non-compilation) portage operations I use 4GB tmpfs mount on /var/tmp/portage/. -- WBR, Alex. ^ permalink raw reply [flat|nested] 9+ messages in thread
* [gentoo-desktop] Re: @system and parallel merge speedup 2012-10-21 13:24 ` [gentoo-desktop] @system and parallel merge speedup Alex Efros @ 2012-10-22 6:26 ` Duncan 0 siblings, 0 replies; 9+ messages in thread From: Duncan @ 2012-10-22 6:26 UTC (permalink / raw To: gentoo-desktop Alex Efros posted on Sun, 21 Oct 2012 16:24:32 +0300 as excerpted: > Hi! > > On Sun, Oct 21, 2012 at 08:02:47AM +0000, Duncan wrote: >> Bottom line, an empty @system set really does make a noticeable >> difference in parallel merge handling, speeding up especially >> --emptytree @world rebuilds but also any general update that has a >> significant number of otherwise @system packages and deps, >> dramatically. I'm happy. =:^) > > I think "@system first" and "@system not merge in parallel" rules are > safe to break when you just doing "--emptytree @world" on already > updated OS because it's only rebuild existing packages, and all packages > while compiling will see same set of other packages (including same > versions). But when upgrading multiple packages (including some from > original @system and some from @world) this probably may result in bugs. In theory, you're right. In practice, I've not seen it yet, tho being cautious I'd say it needs at least six months of testing (I've only been testing it about a month, maybe six weeks) before I can say for sure. It /was/ something I was a bit concerned about, however. That was in fact one of the reasons I decided to try it on the netbook's chroot as well, which hadn't been upgraded in a year and a half. I figured if it could work reasonably well there, the chances of an undiscovered real problem were much lower. However, it /is/ worth noting that as a matter of course, I already often choose to do some system-critical upgrades (portage, gcc, glibc, openrc, udev) on their own, before doing the general upgrades, in part so I can deal with their config file changes and note any problems right away, with a relatively small changeset to deal with, as opposed to having a whole slew of updates including critical system package updates happen all at once, thus making it far more difficult to trace which update actually broke things. That's where the years of gentoo experience I originally mentioned comes in. This isn't going to be as easy for a gentoo newbie for at least two reasons. First, they're less likely to know what packages really /are/ system critical, and thus are more likely to unmerge them without the extra unmerge warning a package in the system set gets. (I mentioned that one in the first post.) Second, spotting critical updates in the initial --pretend run, knowing which packages it's a good idea to upgrade first, by themselves, dealing with config file updates, etc, for just that critical package (and any dependency updates it might pull in), before going on to the general @world upgrade, probably makes a good bit of difference in practice, and gentoo newbies are rather less likely to be able to make that differentiation. (I didn't specifically mention that one until now.) > As for "--emptytree @world" speedup, can you provide benchmarked values? > I mean, only few packages forced to use only one CPU Core while > compiling. > So, merging packages in parallel may save some time mostly for doing > unpack/prepare/configure/install/merge. All of them except configure > actually do a lot of I/O, which most likely lose a lot in speed instead > of gain when done in parallel (especially keeping in mind kernel bug > 12309). So, at a glance time you may win on configure you'll mostly lose > on I/O, and most of time all your CPU Cores will be loaded anyway while > compiling, and doing configure in parallel to compiling unlikely save > some time. This is why I think without actual benchmarking we can't be > sure how faster it became (if it became faster at all, which is > questionable). Good points, and no, I can't easily provide benchmarks, both because of the recent hardware upgrade here, and because portage itself has been gradually improving its parallel merging abilities -- a recent update changed the scheduling algorithm so it starts additional merges much sooner than it did previously. (See gentoo bug 438650 fixed in portage 2.1.11.29 and 2.2.0_alpha140, both released on Oct 17. That I know about that hints at another thing I do routinely as an experienced gentooer: I always read portage's changelog and check out any referenced bugs that look interesting, before I upgrade portage. To the extent practical without actually reading the individual git commits, I want to know about package manager changes that might affect me BEFORE I do that upgrade!) But, I believe as core-counts rise, you're underestimating the effects of portage's parallel merging abilities. In particular, a lot of packages normally in @system (or deps thereof) are relatively small packages such as grep, patch, sed... where the single-threaded configure step takes a MUCH larger share of the total package merge time than it does with larger packages. Similarly, the unpack and prepare phases, plus the package phase for folks using FEATURES=binpkg, tend to be single-threaded.[1] Thus, instead of serializing several dozen small mostly single-threaded package merges for packages like grep/sed/patch/util-linux/etc, depending on the --jobs and --load-average numbers you feed to portage, several of these end up getting done in parallel, with the portage multi-job output bumping a line every few seconds because it's doing them in parallel, instead of every minute or so, because it's doing one at a time. Meanwhile, it should be obvious, but it's worth stating anyway. The effect gets *MUCH* bigger as the number of cores increases. For a dual- core, bah, not worth the trouble, as it could cause more problems then it solves, especially if people are trying to work on other things while portage is doing its thing in the background. I suspect the break-over point is either triple-core or quad-core. One of the reasons portage is getting better lately is because someone's taken an interest that has a 32-core, with a corresponding amount of memory (64 or 128 gig IIRC). It's worth noting, as I mentioned, that I now have a 6-core, recently upgraded from a dual-dual-core (4 cores), with a corresponding memory upgrade, to 16 gigs. One of the first things I noticed doing emerges was how much more difficult it was to keep the 6-core actually peaked out to 100% CPU, than it had been the 4-core. While I suspect there would have been a difference on the quad-core (as I said I believe the break-over's probably 3-4 cores), it wasn't a big deal there. Staring at that 6-core running at 100% on 1-2 cores CPU-freq-maxed at 3.6 GHz, while the other 4-5 cores remained near idle at <20% utilization at CPU-freq-minimum 1.4 GHz... was VERY frustrating. So began my drive to empty @system and get portage properly scheduling parallel merges for former @system packages and their deps as well! For the quad-core plus hyperthreading (thus 8 threads I take it?) you mention below (4.6 GHz OC, nice! I see stock is 3.4 GHz), the boost from killing @system forced serialization should definitely make a difference (unless the hyperthreading doesn't do much for that work load, making it effectively no better than a non-hyperthreaded quad-core. For my 6-core, it made a rather big difference, and I guarantee if you had the 32-core that one of the devs working on improving portage's parallelization has, you'd be hot on the trail to improve it as well! > As for me, I found very effective way to speedup emerge is upgrading > from Core2Duo E6600 to i7-2600K overclocked to 4.6GHz. This speedup > compilation on my system in 6 times (kernel now compiles in just 1 > minute). And to speedup most other (non-compilation) portage operations > I use 4GB tmpfs mount on /var/tmp/portage/. I remember reading about the 1-minute kernel compiles on i7s. Very impressive. FWIW, there's a lot of variables to fill in the blank on, before we can be sure kernel build time comparisons are apples to apples (I had several more paragraphs written on that, but decided it was a digression too far for this post so deleted 'em), but AFAIK when I read about it (on phoronix I believe), he was doing an all-yes config, so building rather more than a typical customized-config gentooer, but was using a rather fast SSD, which probably improved his times quite a bit compared to "spinning rust". But I don't know if his timings included the actual compress (and if so with what CONFIG_KERNEL_XXX compression option) and I don't believe they included the actual install, only the build. That said, a 1-minute all-yes-config kernel build time is impressive indeed, the envy of many, including me. (OTOH, my fx6100 was on sale for $100, $109 post-tax. That's lower than pricewatch's $118 lowest quote (shipped, no tax), and only about 40% of the $273 low quote for an i7-2600k.) My build, compress (CONFIG_KERNEL_XZ) and install, runs ~2 minutes (1:58-2:07, 10+ runs, warm-cache), so yes, even if your build time doesn't include compress and install, which it might, 1-minute is still VERY impressive. Tho as I said, my CPU cost ~40% of the going price on yours, so... Meanwhile... I too use and DEFINITELY recommend a tmpfs $PORTAGE_TMPDIR. I'm running 16 gig RAM here, and didn't want to run out of room with parallel builds, so set a nice roomy 12G tmpfs size. A $PORTAGE_TMPDIR on tmpfs also reduces the I/O. At least here, the only time I've had problems, both on the old hardware and on the new, is when I go into swap. (And on the old hardware I had swap priority= striped across four disks and 4-way md/raid0, so the kernel could schedule swap- out vs read-in much better and I didn't see a problem until I hit nearly half-gig of swap loading at once; the new hardware is only single-disk ATM, and I see issues starting @ 80 meg or so of swap loading, at once.) But with 16 gig RAM on the new system, the only time I see it go into swap is when I run a kernel build with uncapped -j, thus hitting 500+ jobs and close enough to 16 gigs that whether I hit swap or not depends on what else I've been doing with the system. Basically, I/O is thus not a problem at all with portage, here, up to the --jobs=12 --load-average=12 along with MAKEOPTS="-j20 -l15" I normally run, anyway. On the old system with only six gigs of RAM, if I tried hard enough I could get portage to hit swap there, but I limited --jobs and MAKEOPTS until that wasn't an issue, and had no additional problems. Tho I should mention I also run PORTAGE_NICENESS=19 (and my kernel-build/ install script similarly renices itself to 19 before starting the kernel build), which puts it in batch-scheduling mode (idle-only scheduling, but longer timeslices). If it matters, filesystem is reiserfs, iosched is cfq, drive is sata2/ahci (amd 990fx/sb950 chipset) 2.5" seagate "spinning rust". But I definitely agree with $PORTAGE_TMPDIR on tmpfs. It makes a HUGE difference! --- [1] Compression parallelism: There are parallel-threaded alternatives to bzip2, for instance, but they have certain down-sides like decompress only being parallel where the tarball was compressed with the same parallel tool, and certain compression buffer nul-fill handling differences that make them not functionally perfect drop-in replacements. See the recent discussion on the topic on the gentoo-dev list for instance. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [gentoo-desktop] Re: *kit free desktop profile 2012-10-21 8:02 ` [gentoo-desktop] " Duncan 2012-10-21 13:24 ` [gentoo-desktop] @system and parallel merge speedup Alex Efros @ 2012-10-23 17:36 ` Dominique Michel 1 sibling, 0 replies; 9+ messages in thread From: Dominique Michel @ 2012-10-23 17:36 UTC (permalink / raw To: gentoo-desktop Le Sun, 21 Oct 2012 08:02:47 +0000 (UTC), Duncan <1i5t5.duncan@cox.net> a écrit : > Alex Efros posted on Sat, 20 Oct 2012 23:07:40 +0300 as excerpted: > > > Hi! > > > > On Sat, Oct 20, 2012 at 08:07:31PM +0200, Dominique Michel wrote: > >> USE="-consolekit -policykit -udisks -upower" > > > > I'm using fluxbox, but, of course, I need ability to run gnome and > > kde applications (but not gnome/kde itself). Last time I tried to > > switch off these USE-flags it doesn't work. Best I got is: > > > > /etc/make.conf: > > USE="${USE} -policykit -upower" > > /etc/portage/package.use > > # required by udisks > > sys-fs/udev extras gudev hwdb > > # required by polkit, consolekit > > sys-auth/pambase consolekit > > # required by polkit, udisks, clementine > > sys-auth/consolekit policykit > > FWIW, as my gentoo experience lengthens, I've gotten *MUCH* tighter > with my installed-packages and USE flags policy. > > I run a kde desktop, but have this in my global USE file[1]: > > -consolekit -policykit -udisks -udisks2 -upower > > Also given my kde4 desktop it's worth noting: > > -raptor -redland -semantic-desktop -virtuoso > > (Of course that means no kdepim packages at all. I got fed up with > akonadi problems and switched to the gtk-based claws-mail after > nearly a decade on kmail! That allowed me to kill kmail and akonadi, > which allowed me to kill semantic-desktop for all of kde, which > allowed me to kill rasqal, redland, virtuoso, soprano... I did it > mainly to get rid of akonadi and the problems it brought, but WOW, > the kde4 desktop was faster without all that extra bloatware! Wow! > And I > had already turned off nepomuk and strigi too, and it was STILL > dramatically faster when kde was built without that junk! Sort of > reminds me of all those MSWormOS users and how surprised they often > are to learn how badly the malware was affecting system performance > once it's cleaned up. Yes, I DID just compare the semantic-desktop > crap to malware!) > > > But: USE=udev > > No udisks (1 or 2), upower, consolekit or policykit installed. So, it is place for 2 *kit free profiles: desktop/nokit and desktop/kde/nokit Anyway, I will incorporate them into the proaudio overlay. They will have some added USE flags, like jack, that correspond with the goal of this overlay: audio pro. In consequence, I will use something like desktop/proaudio and desktop/proaudio-kde. -- "We have the heroes we deserve." ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2012-10-23 18:02 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-10-20 18:07 [gentoo-desktop] *kit free desktop profile Dominique Michel 2012-10-20 19:08 ` Canek Peláez Valdés 2012-10-21 7:46 ` Dominique Michel 2012-10-20 20:07 ` Alex Efros 2012-10-21 7:50 ` Dominique Michel 2012-10-21 8:02 ` [gentoo-desktop] " Duncan 2012-10-21 13:24 ` [gentoo-desktop] @system and parallel merge speedup Alex Efros 2012-10-22 6:26 ` [gentoo-desktop] " Duncan 2012-10-23 17:36 ` [gentoo-desktop] Re: *kit free desktop profile Dominique Michel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox