[gentoo-dev] Portage-2.0.51 STABLE TESTING

[gentoo-dev] Portage-2.0.51 STABLE TESTING

[Nicholas Jones]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]


So... We've had no bug reports on _rc9 that were beyond trivial.
We've added a few more fixes, and now we are quickly pushing 51
to stable.

As the changes are quite minor, _rc10 will only be out for the
night and 2.0.51 proper will be released stable early tomorrow
morning (Oct 20, 2004).

The only potential bugs will be experienced by LARGE SRC_URI
packages as it's hard to test the enhancement I added to portage
and repoman. Portage handles PARTIAL digesting now. It will make
assumptions that the files you do not that are in a previous
digest are valid and merge the missing and the present files to
produce the new digest.

The results of this is that portage no longer downloads all files
if you are using FEATURES=cvs, only the ones you need. You will
be REQUIRED to have all files listed in the digest, but not
required to have all files locally if you have the checksums
already. This is the area I cannot test easily. Artificial cases
are a little weak here. So this is where I might expect bugs.
I can handle devs complaining, and the users won't notice this.

I include some visual acuity enhancements and everything looks
good to me.

TEST AND REPORT BACK _ANY_ ODDITIES.

People seem to be ignoring my IRC /notice so I'll have
to come up with a way of being more noticible. :-p

--NJ




[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: Portage-2.0.51 STABLE TESTING

[Duncan]

Nicholas Jones posted <20041020014358.GA1867@twobit.net>, excerpted below,
 on Tue, 19 Oct 2004 21:43:59 -0400:

> So... We've had no bug reports on _rc9 that were beyond trivial.
> We've added a few more fixes, and now we are quickly pushing 51
> to stable.

OK, I admit this should have been bugged, but I've been lazy.  However, I
don't yet see it listed on a quick bug search on "ALL portage 51 binary"
or replacing the binary with tbz2.

This affects portage up until the last upgrade.  I haven't triggered it
yet on this version (rc9) tho I've emerged quite a bit (including the new
KDE), but that's potentially due to the issues surrounding the trigger
event.

The problem concisely is this:  Certain binary packages say they are
corrupt and refuse to install, altho the tbz2 seems to be fine as does the
ebuild tacked onto the end.  It's whatever other data is there that
portage seems to be calling corrupt.

Trigger situation setup is this:  I have hardware stability issues
apparently related to SMP on my dual Opteron (so amd64).  Thus, with
"long" ebuilds, such as gcc, glibc, xorg, and the kde ebuilds, I often
find myself rebooting in the middle of the emerge.  No problem other than
the frustration, normally, as I simply use ebuild to resume the compile or
install steps (where it usually happens, not surprisingly) and continue
from there as necessary.  Sometimes I go into the builddir and issue make
there, then when it succeeds, I go back and do the ebuild and it redoes
configure then (with most packages) zooms thru the compile since it's all
done and continues.

The actual trigger seems to be that the last ebuild session that actually
does the package, must also at least step thru portions of the compile,
and do the entire install steps, as well as the package.  If it fails in
the install or package steps, and I reinvoke ebuild package, it will
create the package, but portage will ALWAYS call the package invalid. 
The same happens if I manually complete the make and touch .compiled, so
it starts from install, altho in some cases it has been ebuild that put
the .compiled there and it /still/ fails.

Now, if I take that "invalid" package and manually untar it over the
existing file system then inject, and emerge clean to remove the old
version, all is well.

The same if I take that ready for qmerge fake install and qmerge it
directly.  That works fine.

What does NOT work is attempting to emerge the package created from the
same temporary working dataset as the qmerge that DOES work.

The qmerge works.  Manually untarring the "corrupt" tbz2 package works. 
Attempting to emerge it, the package is corrupt/invalid.

Additionally, portage continues to complain about those "invalid"
packages, every time I attempt an emerge -uD world or otherwise cause it
to scan them.

AFAIK, the only thing "invalid" about them is the checksum or other
binary data that must be included. This because the tbz2 package itself
isn't corrupt, nor does the ebuild tacked onto the end appear to be
corrupt, as it displays just fine in a text editor, at the end of the tbz2.

I don't expect this to be a stopper, as few use buildpkg or otherwise
routinely create packages, and fewer still are unstable enough to have it
occasionally crash between drop of .compiled and the successful completion
of the package step, which seems to be the critical window.  Further, it
may only happen on AMD64, and it's possible that has been fixed even tho I
can't see any bugs for it.  All that said, it'd be nice to know what's
going on and if this has come up before elsewhere than bugzilla or I
simply missed it there, and yes, I /should/ have filed a bug, but I just
never got it done. =:^(

I guess.. consider this my IRC, since I don't do IRC, but I /do/ do
newsgroups and /do/ get this list /as/ a newsgroup thru gmane.

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Portage-2.0.51 STABLE TESTING

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]

On Wednesday 20 October 2004 07:14, Duncan wrote:
> The problem concisely is this:  Certain binary packages say they are
> corrupt and refuse to install, altho the tbz2 seems to be fine as does
> the ebuild tacked onto the end.  It's whatever other data is there that
> portage seems to be calling corrupt.
>
> Trigger situation setup is this:  I have hardware stability issues
> apparently related to SMP on my dual Opteron (so amd64).  Thus, with
> "long" ebuilds, such as gcc, glibc, xorg, and the kde ebuilds, I often
> find myself rebooting in the middle of the emerge.  No problem other
> than the frustration, normally, as I simply use ebuild to resume the
> compile or install steps (where it usually happens, not surprisingly)
> and continue from there as necessary.  Sometimes I go into the builddir
> and issue make there, then when it succeeds, I go back and do the
> ebuild and it redoes configure then (with most packages) zooms thru the
> compile since it's all done and continues.

It seems that this is the actual problem

I allways use packages, also when using ebuild for example for developing 
ebuilds (also bailing out halfway, but because of errors). I've never 
seen this behaviour. The only invalid packages I encountered anytime 
where created using quickpkg instead of with ebuild package.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: Portage-2.0.51 STABLE TESTING

[Jason Stubbs]

On Wednesday 20 October 2004 14:14, Duncan wrote:
> The problem concisely is this:  Certain binary packages say they are
> corrupt and refuse to install, altho the tbz2 seems to be fine as does the
> ebuild tacked onto the end.  It's whatever other data is there that
> portage seems to be calling corrupt.

Try running the following on one of those broken packages. The command should 
be on one line and bash...tbz2 should of course be replaced.

python -c 'import xpak; tbz2=xpak.tbz2("bash-3.0-r5.tbz2"); print 
tbz2.getelements("CATEGORY")'

If it's empty, that's the problem. I had a lot of trouble following your 
previous email. Would you be able to list step-by-step how to reproduce the 
problem?

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Portage-2.0.51 STABLE TESTING

[Jason Stubbs]

On Wednesday 20 October 2004 21:10, Paul de Vrieze wrote:
> The only invalid packages I encountered anytime where created using quickpkg 
> instead of with ebuild package. 

Yep. I came across this as well. To illustrate the trigger:

jason@localhost ~ $ cd /var/db/pkg/sys-libs/
jason@localhost /var/db/pkg/sys-libs $ quickpkg glibc-2.3.4.20041006/

produced a broken package whereas:

jason@localhost ~ $ cd /var/db/pkg/
jason@localhost /var/db/pkg $ quickpkg sys-libs/glibc-2.3.4.20041006/

did not. This has been fixed and released around portage-2.0.51_rc6.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage-2.0.51 STABLE TESTING

[Alexander Futász]

Am Dienstag, den 19.10.2004, 21:43 -0400 schrieb Nicholas Jones:
> So... We've had no bug reports on _rc9 that were beyond trivial.
> We've added a few more fixes, and now we are quickly pushing 51
> to stable.
> 
> As the changes are quite minor, _rc10 will only be out for the
> night and 2.0.51 proper will be released stable early tomorrow
> morning (Oct 20, 2004).

http://bugs.gentoo.org/show_bug.cgi?id=68293

I'm having problems with /etc/portage. Am I the only one?


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage-2.0.51 STABLE TESTING

[Nicholas Jones]

[-- Attachment #1: Type: text/plain, Size: 76 bytes --]

> I'm having problems with /etc/portage. Am I the only one?

Yes. :)

--NJ


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: Re: Portage-2.0.51 STABLE TESTING

[Duncan]

Jason Stubbs posted <200410202121.42464.jstubbs@gentoo.org>, excerpted
below,  on Wed, 20 Oct 2004 21:21:42 +0900:

> Try running the following on one of those broken packages. The command should 
> be on one line and bash...tbz2 should of course be replaced.
> 
> python -c 'import xpak; tbz2=xpak.tbz2("bash-3.0-r5.tbz2"); print 
> tbz2.getelements("CATEGORY")'
> 
> If it's empty, that's the problem. I had a lot of trouble following your 
> previous email. Would you be able to list step-by-step how to reproduce the 
> problem?

That's it.  A good package ($PORTDIR=/mnt/p, $PACKAGEDIR=/mnt/p/pkg):

"/mnt/p/pkg/All/kdepim-3.3.1.tbz2"
['kde-base']

The two bad packages (followed by a ls to verify the files exist:

"/mnt/p/pkg/All/kdelibs-3.3.1.tbz2"
[]
ls "/mnt/p/pkg/All/kdelibs-3.3.1.tbz2"
/mnt/p/pkg/All/kdelibs-3.3.1.tbz2

"/mnt/p/pkg/All/xorg-x11-6.8.0-r1.tbz2"
[]
ls "/mnt/p/pkg/All/xorg-x11-6.8.0-r1.tbz2"
/mnt/p/pkg/All/xorg-x11-6.8.0-r1.tbz2

On the statement of the problem, I really can't blame you for not
following that so well.  Let's see if this helps:

The problem (variant A):

1) ebuild /path/to/package.ebuild package

2) crash/interrupt during ebuild install or package steps

3) repeat step 1 to complete package without warning or error

4) attempt to install "successfully" created package, get invalid package.

variant B) If step 2-crash occurs during ebuild compile, step 3 and 4 work
fine.

variant C) If step 2-crash occurs during compile and one attempts to do a
manual make to complete the compile, then touches .compiled, step 3 will
of course continue with ebuild install and package steps as it should.
However, package is again invalid in step 4.

All the files are in the "invalid" tbz2 package as expected. They are
browsable from mc with its compressed tar virtual file systems, and
extract correctly with only the expected error about the tacked on portage
data.  Further, an examination with a text editor demonstrates the ebuild
in uncompressed form is tacked onto the end of the tbz2 as with the normal
packages, but the data between the EOF and the ebuild doesn't look quite
right.  (tbz2 EOF verified by deleting the extra stuff and working with
the file without error, thus confirming my guess as to where the EOF was.)

I'd guess that data that doesn't look quite right contains the corrupted
info?

However, in all cases, after the package step that creates the invalid
package, if one then issues qmerge on the still existing working dir, it
will qmerge successfully without error -- from the SAME working dir that
creates the bad package.  I have experienced NO difficulty with said
qmerged ebuilds.  They function correctly and portage seems to have the
correct data about them and functions correctly calculating dependencies
and upgrades as well, with the data.

Therefore, it appears the problem is that portage does /something/ in the
ebuild compile step that isn't retained if the process is interrupted
after that.  When portage attempts to pickup with the interrupted install
step, it's missing the data from the compile step, and doesn't complete
the package successfully.  However, the data is apparently either
recalculated or collected by some other mechanism for qmerge, since that
seems to work without issue.

Thus, the problem /only/ occurs for those with buildpkg set, and /only/ if
the interruption occurred between the completion of ebuild compile, and
successful completion of ebuild package.  As I said, I don't consider this
a release stopper.

Further, I mentioned I hadn't seen it with rc9, but that could be because
I've adjusted my behavior slightly to avoid it.  Later today, I'll try to
deliberately trigger the problem, and see if I can get it to reoccur with
now rc10 (or release, if it's out by then).

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin



--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: Re: Portage-2.0.51 STABLE TESTING

[Duncan]

Duncan posted <pan.2004.10.20.21.36.50.96533@cox.net>, excerpted below, 
on Wed, 20 Oct 2004 14:36:50 -0700:

> Further, I mentioned I hadn't seen it with rc9, but that could be because
> I've adjusted my behavior slightly to avoid it.  Later today, I'll try to
> deliberately trigger the problem, and see if I can get it to reoccur with
> now rc10 (or release, if it's out by then).

OK, didn't get to it "today", but finally got to it today. <g>

Actually, between updates of portage, I /did/ manage a couple small
earlier tests but didn't get anything conclusive.  I just tried
re-emerging xorg-x11-6.8.0-r1 (I wanted to try some different cflags --
yes, that means I'm using an overlay copy of the ebuild, to comment out
strip-flags, yes I DID diff a current ebuild with my old overlay ebuild
and update as appropriate), and I'm still having the problem, now with
portage-2.0.51-r2.

To wit:

1 Emerge xorg-x11 and had it crash partway thru the compile step (my bad
hardware).

2 CD to the workdir and complete the make manually.

3 Examine the ebuild to ensure that was all that was needed to complete
the compile step, so I could...

4 Manually touch .compiled

5 ebuild /path/to/overlay/x11-base/xorg-x11-6.8.0-r1.ebuild package

6 ebuild detects a completed compile and does the install and package

7 Attempt to emerge -aK the new package.

8 Package is STILL invalid.

9 Go back to plan B and ebuild /path/to/ebuild qmerge the still-existing
portage tempdir image.

10 Since it was a re-merge of a previously merged package, emerge -c finds
no old package to clean

11 qmerged image works JUST FINE, as expected.

12 Try quickpkg xorg-x11:

 * Building package for xorg-x11-6.8.0-r1...                                                                                      [ ok ]

 * Packages now in /mnt/p/pkg:
 * xorg-x11-6.8.0-r1: 50M

 (completes without error)

13 Try emerge -aK xorg-x11 again:

These are the packages that I would merge, in order:

!!! Invalid binary package: xorg-x11-6.8.0-r1.tbz2
!!! Invalid binary package: kdelibs-3.3.1.tbz2
Calculating dependencies
!!! There are no packages available to satisfy: "xorg-x11"
!!! Either add a suitable binary package or compile from an ebuild.

The package is *STILL* invalid!

14 An ls -l of the package file confirms by timestamp that it is
indeed the quickpkg package, so /that's/ working, only it's STILL an
invalid package.

##############

What causes the package step to include invalid category and etc data?

Checking the portage log, I see this at the beginning of the install step,
so it /does/ seem to know the category there.

>>> Install xorg-x11-6.8.0-r1 into /tmp/portage/xorg-x11-6.8.0-r1/image/ category x11-base

Yet:
python -c 'import xpak; tbz2=xpak.tbz2("xorg-x11-6.8.0-r1.tbz2"); print tbz2.getelements("CATEGORY")'
[]


Perhaps it happens with the overlay only?  Don't think so, because I
hadn't touched the kdelibs-3.3.1.ebuild and it did the same thing (with an
earlier .51 portage).

FWIW, it worked in portage 2.0.50 just fine.

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin



--
gentoo-dev@gentoo.org mailing list