[gentoo-dev] [RFC] Required Features for $package_manager to Aid... Development!

[gentoo-dev] [RFC] Required Features for $package_manager to Aid... Development!

[Elfyn McBratney]

Hola,

I've been inspired by the local/global USE flag threads recently
posted by Doug (Cardoe), and it got me to thinking... I've recently
joined the pkgcore development effort, and was asked by Brian
(ferringb) what I'd like to hack on/what my niggles with portage are.
My personal one is why updates/ and binpkg mangling takes so long in
portage-$current.  But I'd like to know, what are everyone elses?

I know that this topic have been rehashed since the dawn of
time^Wgentoo-dev, but these things get lost, opinions change... and
since last year, new and viable alternate package managers have
cropped up.  So, basically I'd like to ask all on this list: - what
package manager features would make *your* life easier?

Personally, I'd like to see the following groups of changes happen one
way or another:

* Improve QA with/by:
 - Detecting conflicting USE flags;
 - Limiting the amount of times we need to call "die" in ebuilds.

Yes, that's a small list... which is why I'm sending this mail to
solicit even more ideas. :)

Comments, flames ("read the archives", "go jump in a pond", etc) and
whatever else is coming my way - hit it me with it! ;P

Best,
Elfyn

-- 
Elfyn McBratney,
Freenode and Gentoo Developer,
Nutty between the hours of 3AM and 14AM...
+----------------------------------+
|  irc.freenode.net / beu          |
|  beu [@] freenode [.] net        |
|  beu [@] gentoo [.] org          |
+-------------------------- o.0 ---+

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] Required Features for $package_manager to Aid... Development!

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1918 bytes --]

On Wed, Sep 06, 2006 at 10:29:11AM +0000, Elfyn McBratney wrote:
> Hola,
> 
> I've been inspired by the local/global USE flag threads recently
> posted by Doug (Cardoe), and it got me to thinking... I've recently
> joined the pkgcore development effort, and was asked by Brian
> (ferringb) what I'd like to hack on/what my niggles with portage are.
> My personal one is why updates/ and binpkg mangling takes so long in
> portage-$current.  But I'd like to know, what are everyone elses?
> 
> I know that this topic have been rehashed since the dawn of
> time^Wgentoo-dev, but these things get lost, opinions change... and
> since last year, new and viable alternate package managers have
> cropped up.  So, basically I'd like to ask all on this list: - what
> package manager features would make *your* life easier?
> 
> Personally, I'd like to see the following groups of changes happen one
> way or another:
> 
> * Improve QA with/by:
>  - Detecting conflicting USE flags;

That one is on the map for trying to address via eapi=1; intention was 
to resurrect the approach of using a seperate metadata key, and 
extending depset support to include xor so that representing the 
_valid_ use conditional states is possible.

Simple example, if you need mysql or postgres, but not both;
VALID_USE_STATES="^ ( mysql postgres )"

if it's not required to have them enabled, but still need one or the 
other if enabled
VALID_USE_STATES="( || ( mysql postgres ) ^ ( mysql postgres ) )"

Would need to extend depset in this case to support negation of blocks 
also (nand/nor); downside, it gets verbose.

Upshot, it uses extisting syntax folks are familar with; the verbosity 
of it would also go down if/when use groups were implemented thus 
replacing USE_EXPAND (something that needs resurrecting and is another 
bit on the map, although not guranteed to be targeted).

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: [RFC] Required Features for $package_manager to Aid... Development!

[Ryan Hill]

Elfyn McBratney wrote:

> I've been inspired by the local/global USE flag threads recently
> posted by Doug (Cardoe), and it got me to thinking... I've recently
> joined the pkgcore development effort, and was asked by Brian
> (ferringb) what I'd like to hack on/what my niggles with portage are.
> My personal one is why updates/ and binpkg mangling takes so long in
> portage-$current.  But I'd like to know, what are everyone elses?

I've been thinking about this lately too.  I think it would be a good 
idea to come up with as many different use cases as we can think of and 
figure out what we can already do, what we would like to do, and the 
best way to do it.

> I know that this topic have been rehashed since the dawn of
> time^Wgentoo-dev, but these things get lost, opinions change... and
> since last year, new and viable alternate package managers have
> cropped up.  So, basically I'd like to ask all on this list: - what
> package manager features would make *your* life easier?

- current pet peeve is some way of dealing with SRC_URI's that use 
dynamic redirects to the source files (eg. 
http://nwvault.ign.com/fms/Download.php?id=57167 -> 
http://someserver.com/randomlygeneratedhashthatchangeseveryhour/the_HeX_coda_01_v1.3.zip). 
  Since the name of the file fetched (the_HeX_coda_01_v1.3.zip) != the 
one in SRC_URI (Download.php?id=57167) portage bombs.  right now i have 
to use RESTRICT="fetch" which sucks.

--de.

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Required Features for $package_manager to Aid... Development!

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 3702 bytes --]

On Wed, 2006-09-06 at 21:19 -0600, Ryan Hill wrote:
> Elfyn McBratney wrote:
> 
> > I've been inspired by the local/global USE flag threads recently
> > posted by Doug (Cardoe), and it got me to thinking... I've recently
> > joined the pkgcore development effort, and was asked by Brian
> > (ferringb) what I'd like to hack on/what my niggles with portage are.
> > My personal one is why updates/ and binpkg mangling takes so long in
> > portage-$current.  But I'd like to know, what are everyone elses?
> 
> I've been thinking about this lately too.  I think it would be a good 
> idea to come up with as many different use cases as we can think of and 
> figure out what we can already do, what we would like to do, and the 
> best way to do it.
> 
> > I know that this topic have been rehashed since the dawn of
> > time^Wgentoo-dev, but these things get lost, opinions change... and
> > since last year, new and viable alternate package managers have
> > cropped up.  So, basically I'd like to ask all on this list: - what
> > package manager features would make *your* life easier?
> 
> - current pet peeve is some way of dealing with SRC_URI's that use 
> dynamic redirects to the source files (eg. 
> http://nwvault.ign.com/fms/Download.php?id=57167 -> 
> http://someserver.com/randomlygeneratedhashthatchangeseveryhour/the_HeX_coda_01_v1.3.zip). 
>   Since the name of the file fetched (the_HeX_coda_01_v1.3.zip) != the 
> one in SRC_URI (Download.php?id=57167) portage bombs.  right now i have 
> to use RESTRICT="fetch" which sucks.

There's also any SRC_URI that includes an "&"...

There are some things that I would like to see from a Release
Engineering standpoint.  These are things that I would like some way to
obtain, not necessarily from the normal user "front-end" to
$package_manager, but somehow.

#1. Ability to grab USE from the environment for a machine both before
and after USE_EXPAND is calculated
#2. Ability to ignore environment's USE when doing calculations, such as
the easily grabbing the contents of the "system" target with the default
USE for a profile
#3. Ability to list the stable package versions for a given profile
#4. Ability to list the testing package versions for a given profile
#5. Ability to list the used USE flags in a given set of packages
#6. Ability to list the licenses used in a given set of packages (this
is especially important as we are seeing more and more packages that we
are not allowed to redistribute being used accidentally)
#7. Ability to list the packages that use a given set of licenses
#8. Ability to list the dependency tree for packages, even if some of
the dependencies are masked by keywords, rather than throwing up the
"this package is masked by keywords" error for each one, allowing one to
see *quickly* all of the packages that need keyword changes for a
particular package to have its keywords changed... fex. "emerge
--keywords =kde-base/kde-4.0" should list all of KDE 4.0's dependencies,
and anything that is masked by keywords should show up as "~" or
something... anything masked by package.mask should show up as "M"...
this should have a way of choosing to ignore profile-level masks or not,
also... this is just an example, how we actually get the information
doesn't matter, so long as we can get it...
#9. a standardized api to connect to the soft-serve ice cream machine in
the developer lounge

I'm sure there's more, but these would be a godsend to have.  I'll be
sure to let you guys know if I can think of anything else.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: [RFC] Required Features for $package_manager to Aid... Development!

[S. Lockwood-Childs]

On Sat, 9 Sep 2006, Chris Gianelloni wrote:

> #8. Ability to list the dependency tree for packages, even if some of
> the dependencies are masked by keywords, rather than throwing up the
> "this package is masked by keywords" error for each one, allowing one to
> see *quickly* all of the packages that need keyword changes for a
> particular package to have its keywords changed... fex. "emerge
> --keywords =kde-base/kde-4.0" should list all of KDE 4.0's dependencies,
> and anything that is masked by keywords should show up as "~" or
> something... anything masked by package.mask should show up as "M"...
> this should have a way of choosing to ignore profile-level masks or not,
> also... this is just an example, how we actually get the information
> doesn't matter, so long as we can get it...

This would be *really* helpful for arch devs/ATs as well.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Required Features for $package_manager to Aid... Development!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 288 bytes --]

On Saturday 09 September 2006 09:37, Chris Gianelloni wrote:
> There's also any SRC_URI that includes an "&"...

last i checked, the only problem lies in the final filename itself ... so you 
can use encoded strings in the URL itself so long as it isnt part of the 
filename
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Re: [RFC] Required Features for $package_manager to Aid... Development!

[Alec Warner]

Chris Gianelloni wrote:
> On Wed, 2006-09-06 at 21:19 -0600, Ryan Hill wrote:
>> Elfyn McBratney wrote:
>>
>>> I've been inspired by the local/global USE flag threads recently
>>> posted by Doug (Cardoe), and it got me to thinking... I've recently
>>> joined the pkgcore development effort, and was asked by Brian
>>> (ferringb) what I'd like to hack on/what my niggles with portage are.
>>> My personal one is why updates/ and binpkg mangling takes so long in
>>> portage-$current.  But I'd like to know, what are everyone elses?
>> I've been thinking about this lately too.  I think it would be a good 
>> idea to come up with as many different use cases as we can think of and 
>> figure out what we can already do, what we would like to do, and the 
>> best way to do it.
>>
>>> I know that this topic have been rehashed since the dawn of
>>> time^Wgentoo-dev, but these things get lost, opinions change... and
>>> since last year, new and viable alternate package managers have
>>> cropped up.  So, basically I'd like to ask all on this list: - what
>>> package manager features would make *your* life easier?
>> - current pet peeve is some way of dealing with SRC_URI's that use 
>> dynamic redirects to the source files (eg. 
>> http://nwvault.ign.com/fms/Download.php?id=57167 -> 
>> http://someserver.com/randomlygeneratedhashthatchangeseveryhour/the_HeX_coda_01_v1.3.zip). 
>>   Since the name of the file fetched (the_HeX_coda_01_v1.3.zip) != the 
>> one in SRC_URI (Download.php?id=57167) portage bombs.  right now i have 
>> to use RESTRICT="fetch" which sucks.
> 
> There's also any SRC_URI that includes an "&"...
> 
> There are some things that I would like to see from a Release
> Engineering standpoint.  These are things that I would like some way to
> obtain, not necessarily from the normal user "front-end" to
> $package_manager, but somehow.
> 
> #1. Ability to grab USE from the environment for a machine both before
> and after USE_EXPAND is calculated
> #2. Ability to ignore environment's USE when doing calculations, such as
> the easily grabbing the contents of the "system" target with the default
> USE for a profile

This is possible via USE_ORDER, you can turn whatever stacking you like
on or off.  It's usage is not supported (as in you break it you fix it);
mostly this is to prevent users from futzing with USE_ORDER and then
complaining to us about it.

> #3. Ability to list the stable package versions for a given profile
> #4. Ability to list the testing package versions for a given profile
> #5. Ability to list the used USE flags in a given set of packages
> #6. Ability to list the licenses used in a given set of packages (this
> is especially important as we are seeing more and more packages that we
> are not allowed to redistribute being used accidentally)
> #7. Ability to list the packages that use a given set of licenses
> #8. Ability to list the dependency tree for packages, even if some of
> the dependencies are masked by keywords, rather than throwing up the
> "this package is masked by keywords" error for each one, allowing one to
> see *quickly* all of the packages that need keyword changes for a
> particular package to have its keywords changed... fex. "emerge
> --keywords =kde-base/kde-4.0" should list all of KDE 4.0's dependencies,
> and anything that is masked by keywords should show up as "~" or
> something... anything masked by package.mask should show up as "M"...
> this should have a way of choosing to ignore profile-level masks or not,
> also... this is just an example, how we actually get the information
> doesn't matter, so long as we can get it...

This is a tricky one (and often asked for!).  The output would be a
guess at best though.

A : Depends on B
B-1 : Depends on C
B-2 : Depends on C,D,E

All versions of B are masked; so either you get

A
~B (Warning B is masked)

or you get

A
~B (Warning B is masked)
C

or

A
~B (Warning B is masked)
C
D
E

This depends of course on which version of B we choose.  This is why
this request isn't implemented; there is no good heuristic for making
this choice and in complex dependency trees, bad choices lead to bad
results.

One of these days I will write these damn explanations down ;)
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Required Features for $package_manager to Aid... Development!

[Francesco Riosa]

Alec Warner wrote:
> Chris Gianelloni wrote:
>> On Wed, 2006-09-06 at 21:19 -0600, Ryan Hill wrote:
>>> Elfyn McBratney wrote:
>>>
>>>> I've been inspired by the local/global USE flag threads recently
>>>> posted by Doug (Cardoe), and it got me to thinking... I've recently
>>>> joined the pkgcore development effort, and was asked by Brian
>>>> (ferringb) what I'd like to hack on/what my niggles with portage are.
>>>> My personal one is why updates/ and binpkg mangling takes so long in
>>>> portage-$current.  But I'd like to know, what are everyone elses?
>>> I've been thinking about this lately too.  I think it would be a good 
>>> idea to come up with as many different use cases as we can think of and 
>>> figure out what we can already do, what we would like to do, and the 
>>> best way to do it.
>>>
>>>> I know that this topic have been rehashed since the dawn of
>>>> time^Wgentoo-dev, but these things get lost, opinions change... and
>>>> since last year, new and viable alternate package managers have
>>>> cropped up.  So, basically I'd like to ask all on this list: - what
>>>> package manager features would make *your* life easier?
>>> - current pet peeve is some way of dealing with SRC_URI's that use 
>>> dynamic redirects to the source files (eg. 
>>> http://nwvault.ign.com/fms/Download.php?id=57167 -> 
>>> http://someserver.com/randomlygeneratedhashthatchangeseveryhour/the_HeX_coda_01_v1.3.zip). 
>>>   Since the name of the file fetched (the_HeX_coda_01_v1.3.zip) != the 
>>> one in SRC_URI (Download.php?id=57167) portage bombs.  right now i have 
>>> to use RESTRICT="fetch" which sucks.
>> There's also any SRC_URI that includes an "&"...
>>
>> There are some things that I would like to see from a Release
>> Engineering standpoint.  These are things that I would like some way to
>> obtain, not necessarily from the normal user "front-end" to
>> $package_manager, but somehow.
>>
>> #1. Ability to grab USE from the environment for a machine both before
>> and after USE_EXPAND is calculated
>> #2. Ability to ignore environment's USE when doing calculations, such as
>> the easily grabbing the contents of the "system" target with the default
>> USE for a profile
> 
> This is possible via USE_ORDER, you can turn whatever stacking you like
> on or off.  It's usage is not supported (as in you break it you fix it);
> mostly this is to prevent users from futzing with USE_ORDER and then
> complaining to us about it.
> 
>> #3. Ability to list the stable package versions for a given profile
>> #4. Ability to list the testing package versions for a given profile
>> #5. Ability to list the used USE flags in a given set of packages
>> #6. Ability to list the licenses used in a given set of packages (this
>> is especially important as we are seeing more and more packages that we
>> are not allowed to redistribute being used accidentally)
>> #7. Ability to list the packages that use a given set of licenses
>> #8. Ability to list the dependency tree for packages, even if some of
>> the dependencies are masked by keywords, rather than throwing up the
>> "this package is masked by keywords" error for each one, allowing one to
>> see *quickly* all of the packages that need keyword changes for a
>> particular package to have its keywords changed... fex. "emerge
>> --keywords =kde-base/kde-4.0" should list all of KDE 4.0's dependencies,
>> and anything that is masked by keywords should show up as "~" or
>> something... anything masked by package.mask should show up as "M"...
>> this should have a way of choosing to ignore profile-level masks or not,
>> also... this is just an example, how we actually get the information
>> doesn't matter, so long as we can get it...
> 
> This is a tricky one (and often asked for!).  The output would be a
> guess at best though.
> 
> A : Depends on B
> B-1 : Depends on C
> B-2 : Depends on C,D,E
> 
> All versions of B are masked; so either you get
> 
> A
> ~B (Warning B is masked)
> 
> or you get
> 
> A
> ~B (Warning B is masked)
> C
> 
> or
> 
> A
> ~B (Warning B is masked)
> C
> D
> E
> 
> This depends of course on which version of B we choose.  This is why
> this request isn't implemented; there is no good heuristic for making
> this choice and in complex dependency trees, bad choices lead to bad
> results.

What about to decide for the less dangerous, less keywording route ?
follow pseudo-pseudo-code:

def find_all_dependants_recursive(\
  pkg, \
  acceptable_version_range='0..inf', \
  already_needed={}, \
  recursion_level=0 \
  )

  SELECT p.version, p.stability, p.depends
  INTO %p_version, %p_stable_level, %p_depends
  FROM packages AS p
  WHERE p.version BETWEEN %acceptable_version_range
  ORDER BY p.version DESC, p.stability
  LIMIT 1

  if already_needed.has_key(pkg):
    """
    already encountered this package,
    check that current needs are ok
    """

    if \ %p_version.inside(already_needed[pkg]['acceptable_version_range']):
      """
      TOCHECK:
      the range is compatible, do nothing ?
      """
      pass
    else:
      # damn, we need to try harder to look if there is a good
      # package version
      if \
already_needed[pkg]['acceptable_version_range'].inside(acceptable_version_range):
        # we can try the luck
        already_needed[pkg] = find_a_good_one(\
          pkg, \
          already_needed[pkg]['acceptable_version_range']
        )
      else:
        # there is no possibility to find a good package,
        # the tree is broken
        throw_error,_clean_and_die()
  else:
    already_needed[pkg] = {\
    'acceptable_version_range' : acceptable_version_range, \
    'version' : %p_version, \
    'stable_level' : %p_stable_level \
    }

  for dpkg in %p_depends:

    find_all_dependants_recursive(\
    %p_depends['pkg'], \
    %p_depends['acceptable_version_range'], \
    already_needed, \
    recursion_level +1 \
    )

  if recursion_level == 0:
    einfo('yay')
    return already_needed

  return 42


def find_a_good_one(pkg, acceptable_version_range)
  SELECT p.version, p.stability, p.depends
  INTO %p_version, %p_stable_level
  FROM packages AS p
  WHERE p.version BETWEEN %acceptable_version_range
  ORDER BY p.version DESC, p.stability
  LIMIT 1

  return {\
    'acceptable_version_range' : acceptable_version_range, \
    'version' : %p_version, \
    'stable_level' : %p_stable_level \
    }


if __name__ == "__main__":
  print find_all_dependants_recursive('dev-db/i_am_a_bad_puppy')

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Required Features for $package_manager to Aid... Development!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 213 bytes --]

On Wednesday 06 September 2006 23:19, Ryan Hill wrote:
> - current pet peeve is some way of dealing with SRC_URI's that use
> dynamic redirects to the source files

tell upstream people to stop being stupid
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] [RFC] Required Features for $package_manager to Aid... Development!

[Luca Barbato]

Elfyn McBratney wrote:
[snip]

1- provide a standardized api for the queries/commands (C, C++, python,
perl, ruby)
2a either provide an emerge or
2b make emerge and equery use 1 and be just a front-end

That way other tools won't have to be rewritten for a specific manager,
advanced/experimental stuff will be available from specific front-ends.

lu

-- 

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@gentoo.org mailing list