[gentoo-user] Can't fetch distfiles in chroot

[gentoo-user] Can't fetch distfiles in chroot

[Peter Humphrey]

I've been NFS-exporting the portage treee from a 32-bit atom box to a chroot 
on my workstation, and it's worked well for years, if slowly.

Now when I try to do the same with a 64-bit celeron machine I'm having a 
problem getting portage to work. If the required distfile is already present, 
no problem, but otherwise, trying to fetch it just hangs. No errors, no 
status, no fetch log, no progress.

Www-client/links works in the chroot as expected, so the network is set up all 
right; portage just can't use it.

I've compared /etc/exports on the two clients; also the chroot setup scripts, 
/usr/portage permissions, the USE flags of nfs-utils and everything else I can 
think of. All identical apart from obvious things like 32/64 bits and network 
names and IPs. Google hasn't helped either.

Any ideas, anyone?

-- 
Regards,
Peter.

[gentoo-user] Re: Can't fetch distfiles in chroot

[Jonathan Callen]

[-- Attachment #1.1: Type: text/plain, Size: 1161 bytes --]

On 04/22/2018 01:13 AM, Peter Humphrey wrote:
> I've been NFS-exporting the portage treee from a 32-bit atom box to a chroot 
> on my workstation, and it's worked well for years, if slowly.
> 
> Now when I try to do the same with a 64-bit celeron machine I'm having a 
> problem getting portage to work. If the required distfile is already present, 
> no problem, but otherwise, trying to fetch it just hangs. No errors, no 
> status, no fetch log, no progress.
> 
> Www-client/links works in the chroot as expected, so the network is set up all 
> right; portage just can't use it.
> 
> I've compared /etc/exports on the two clients; also the chroot setup scripts, 
> /usr/portage permissions, the USE flags of nfs-utils and everything else I can 
> think of. All identical apart from obvious things like 32/64 bits and network 
> names and IPs. Google hasn't helped either.
> 
> Any ideas, anyone?
> 

Generally, this would indicate a problem resolving DNS.  This is
normally caused by not having a correct /etc/resolv.conf inside the
chroot (it generally will need to be the same as the file outside the
chroot).

Jonathan Callen


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Peter Humphrey]

On Sunday, 22 April 2018 07:20:28 BST Jonathan Callen wrote:

> Generally, this would indicate a problem resolving DNS.  This is
> normally caused by not having a correct /etc/resolv.conf inside the
> chroot (it generally will need to be the same as the file outside the
> chroot).

Yes, I'd already thought of that and found it to be right; it's why I tried 
www-client/links inside the chroot, which works just fine.

-- 
Regards,
Peter.

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Peter Humphrey]

On Sunday, 22 April 2018 09:39:23 BST Peter Humphrey wrote:
> On Sunday, 22 April 2018 07:20:28 BST Jonathan Callen wrote:
> > Generally, this would indicate a problem resolving DNS.  This is
> > normally caused by not having a correct /etc/resolv.conf inside the
> > chroot (it generally will need to be the same as the file outside the
> > chroot).
> 
> Yes, I'd already thought of that and found it to be right; it's why I tried
> www-client/links inside the chroot, which works just fine.

I forgot to add a couple of things:

1.	The chroot host (this box) is a multilib system, but the chroot client 
(the celeron box) is no-multilib. Could that make a difference?

2.	While the fetching is hung, /bin/ps shows two emerge processes: one with 
status S and the other with D. Am I right in thinking that portage spawns 
another process to do the fetching, and waits for it to finish? I hope it's so 
anyway.

That first thought was prompted by the instruction to "mount -o bind /lib/
modules /foo/lib/modules" in this page:

https://wiki.gentoo.org/wiki/Project:X86/Chroot_Guide

-- 
Regards,
Peter.

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Peter Humphrey]

On Sunday, 22 April 2018 10:21:16 BST Peter Humphrey wrote:
> On Sunday, 22 April 2018 09:39:23 BST Peter Humphrey wrote:
> > On Sunday, 22 April 2018 07:20:28 BST Jonathan Callen wrote:
> > > Generally, this would indicate a problem resolving DNS.  This is
> > > normally caused by not having a correct /etc/resolv.conf inside the
> > > chroot (it generally will need to be the same as the file outside the
> > > chroot).
> > 
> > Yes, I'd already thought of that and found it to be right; it's why I
> > tried
> > www-client/links inside the chroot, which works just fine.
> 
> I forgot to add a couple of things:
> 
> 1.	The chroot host (this box) is a multilib system, but the chroot client
> (the celeron box) is no-multilib. Could that make a difference?

That's wrong. They're both on the plain desktop profile. What I should have 
said is that the kernel config has modules disabled - on both client and host.

> 2.	While the fetching is hung, /bin/ps shows two emerge processes: one with
> status S and the other with D. Am I right in thinking that portage spawns
> another process to do the fetching, and waits for it to finish? I hope it's
> so anyway.
> 
> That first thought was prompted by the instruction to "mount -o bind /lib/
> modules /foo/lib/modules" in this page:
> 
> https://wiki.gentoo.org/wiki/Project:X86/Chroot_Guide

-- 
Regards,
Peter.

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Mick]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]

On Sunday, 22 April 2018 10:21:16 BST Peter Humphrey wrote:
> That first thought was prompted by the instruction to "mount -o bind /lib/
> modules /foo/lib/modules" in this page:
> 
> https://wiki.gentoo.org/wiki/Project:X86/Chroot_Guide

I didn't do a detailed comparison, but you may want to take a look at the 
Handbook, the chroot mounts appear to be different.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Peter Humphrey]

On Sunday, 22 April 2018 12:13:53 BST Mick wrote:
> On Sunday, 22 April 2018 10:21:16 BST Peter Humphrey wrote:
> > That first thought was prompted by the instruction to "mount -o bind /lib/
> > modules /foo/lib/modules" in this page:
> > 
> > https://wiki.gentoo.org/wiki/Project:X86/Chroot_Guide
> 
> I didn't do a detailed comparison, but you may want to take a look at the
> Handbook, the chroot mounts appear to be different.

I think I've found the problem. It's in the -march setting, which of course 
has to be specific in the chroot, not "native." I had it set to "silvermont," 
but now I can't see why I did that. The target CPU is a celeron N3150, which 
according to an Intel site is "Products formerly Braswell" [1]. None of the 
Gentoo or GCC optimisation sites I could find even mention braswell, 
silvermont or model 76.

So I changed make.conf to braswell, and now I get "error: bad value (braswell) 
for -march= switch" from the compiler during any emerge. Now, does that mean I 
can't switch from silvermont to braswell, or does it also imply that, if I 
start installing again from scratch, I won't be able to switch from whatever's 
in the stage-3 tarball to braswell either?

Perhaps I should just try it and see.

1.	https://ark.intel.com/products/codename/66094/Braswell

-- 
Regards,
Peter.

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Walter Dnes]

> I think I've found the problem. It's in the -march setting, which of
> course has to be specific in the chroot, not "native." I had it set
> to "silvermont," but now I can't see why I did that. The target CPU
> is a celeron N3150, which according to an Intel site is "Products
> formerly Braswell" [1]. None of the Gentoo or GCC optimisation sites
> I could find even mention braswell, silvermont or model 76.
> 
> So I changed make.conf to braswell, and now I get "error: bad
> value (braswell) for -march= switch" from the compiler during any
> emerge.

  Sounds like an invalid value.  To find out what gcc thinks the machine
really is, *ON THE CELERON*, execute the command...

gcc -c -Q -march=native --help=target | grep march=

  On my ancient core2, I get...

  -march=                               core2

  Use whatever value the above query returns on the Celeron.  Silvermont
is a recent 64 bit Atom.  For a big list of gcc-7.3 supported cpus, see
https://gcc.gnu.org/onlinedocs/gcc-7.3.0/gcc/x86-Options.html#x86-Options

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1088 bytes --]

On Monday, 23 April 2018 10:52:57 BST Peter Humphrey wrote:
> On Sunday, 22 April 2018 12:13:53 BST Mick wrote:
> > On Sunday, 22 April 2018 10:21:16 BST Peter Humphrey wrote:
> > > That first thought was prompted by the instruction to "mount -o bind
> > > /lib/
> > > modules /foo/lib/modules" in this page:
> > > 
> > > https://wiki.gentoo.org/wiki/Project:X86/Chroot_Guide
> > 
> > I didn't do a detailed comparison, but you may want to take a look at the
> > Handbook, the chroot mounts appear to be different.
> 
> I think I've found the problem. It's in the -march setting, which of course
> has to be specific in the chroot, not "native." I had it set to
> "silvermont," but now I can't see why I did that. The target CPU is a
> celeron N3150, which according to an Intel site is "Products formerly
> Braswell" [1]. None of the Gentoo or GCC optimisation sites I could find
> even mention braswell, silvermont or model 76.

If you check 'man gcc' you will find "silvermont" is mentioned as a cpu_type 
you could set for march, but not the graphics core Braswell.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Peter Humphrey]

On Monday, 23 April 2018 11:57:12 BST Walter Dnes wrote:
> > I think I've found the problem. It's in the -march setting, which of
> > course has to be specific in the chroot, not "native." I had it set
> > to "silvermont," but now I can't see why I did that. The target CPU
> > is a celeron N3150, which according to an Intel site is "Products
> > formerly Braswell" [1]. None of the Gentoo or GCC optimisation sites
> > I could find even mention braswell, silvermont or model 76.
> > 
> > So I changed make.conf to braswell, and now I get "error: bad
> > value (braswell) for -march= switch" from the compiler during any
> > emerge.
> 
>   Sounds like an invalid value.  To find out what gcc thinks the machine
> really is, *ON THE CELERON*, execute the command...
> 
> gcc -c -Q -march=native --help=target | grep march=

Ah, that was it. My searching didn't turn that one up - thanks.

$ gcc -c -Q -march=native --help=target | grep march=
  -march=                               silvermont

>   On my ancient core2, I get...
> 
>   -march=                               core2
> 
>   Use whatever value the above query returns on the Celeron.  Silvermont
> is a recent 64 bit Atom.  For a big list of gcc-7.3 supported cpus, see
> https://gcc.gnu.org/onlinedocs/gcc-7.3.0/gcc/x86-Options.html#x86-Options

-- 
Regards,
Peter.

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Peter Humphrey]

On Monday, 23 April 2018 11:57:27 BST Mick wrote:
> On Monday, 23 April 2018 10:52:57 BST Peter Humphrey wrote:
> > On Sunday, 22 April 2018 12:13:53 BST Mick wrote:
> > > On Sunday, 22 April 2018 10:21:16 BST Peter Humphrey wrote:
> > > > That first thought was prompted by the instruction to "mount -o bind
> > > > /lib/
> > > > modules /foo/lib/modules" in this page:
> > > > 
> > > > https://wiki.gentoo.org/wiki/Project:X86/Chroot_Guide
> > > 
> > > I didn't do a detailed comparison, but you may want to take a look at
> > > the
> > > Handbook, the chroot mounts appear to be different.
> > 
> > I think I've found the problem. It's in the -march setting, which of
> > course
> > has to be specific in the chroot, not "native." I had it set to
> > "silvermont," but now I can't see why I did that. The target CPU is a
> > celeron N3150, which according to an Intel site is "Products formerly
> > Braswell" [1]. None of the Gentoo or GCC optimisation sites I could find
> > even mention braswell, silvermont or model 76.
> 
> If you check 'man gcc' you will find "silvermont" is mentioned as a cpu_type
> you could set for march, but not the graphics core Braswell.

Yes, so it is, but I couldn't tell which arch I needed for this CPU.

Thanks to you both.

-- 
Regards,
Peter.

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Walter Dnes]

On Mon, Apr 23, 2018 at 04:23:18PM +0100, Peter Humphrey wrote

> Ah, that was it. My searching didn't turn that one up - thanks.
> 
> $ gcc -c -Q -march=native --help=target | grep march=
>   -march=                               silvermont

  https://en.wikipedia.org/wiki/Celeron

> Celeron is a brand name given by Intel to a number of different
> low-end IA-32 and x86-64 computer microprocessor models targeted
> at budget personal computers.

  Looking at that page, "Celeron" can be a cheap, low end version any
one of a whole slew of Intel CPUs.  You have to check the cpu to get the
exact model that it's the low end version of.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Peter Humphrey]

On Monday, 23 April 2018 18:53:09 BST Walter Dnes wrote:

>   https://en.wikipedia.org/wiki/Celeron
> 
> > Celeron is a brand name given by Intel to a number of different
> > low-end IA-32 and x86-64 computer microprocessor models targeted
> > at budget personal computers.
> 
>   Looking at that page, "Celeron" can be a cheap, low end version any
> one of a whole slew of Intel CPUs.  You have to check the cpu to get the
> exact model that it's the low end version of.

..and it doesn't even mention silvermont. The page is 15 months old since its 
last update, which is about the same age as the box in question.

Anyway, this all leaves unanswered the question whether getting the -march 
setting right in a chroot matters to the operation of the chroot itself, as 
distinct from building good code for the target machine. Personally, I can't 
see how it can have any such effect.

-- 
Regards,
Peter.

Re: [gentoo-user] Can't fetch distfiles in chroot

[Peter Humphrey]

On Sunday, 22 April 2018 06:13:30 BST Peter Humphrey wrote:
> I've been NFS-exporting the portage treee from a 32-bit atom box to a chroot
> on my workstation, and it's worked well for years, if slowly.
> 
> Now when I try to do the same with a 64-bit celeron machine I'm having a
> problem getting portage to work. If the required distfile is already
> present, no problem, but otherwise, trying to fetch it just hangs. No
> errors, no status, no fetch log, no progress.
> 
> Www-client/links works in the chroot as expected, so the network is set up
> all right; portage just can't use it.
> 
> I've compared /etc/exports on the two clients; also the chroot setup
> scripts, /usr/portage permissions, the USE flags of nfs-utils and
> everything else I can think of. All identical apart from obvious things
> like 32/64 bits and network names and IPs. Google hasn't helped either.
> 
> Any ideas, anyone?

Never mind. I've rebuilt the chroot from stage 3 and it seems to be working 
fine. I only asked in the hope of not having to do that. But then, it didn't 
take too long really. Hey-ho.

-- 
Regards,
Peter.

[gentoo-user] Re: Can't fetch distfiles in chroot

[Walter Dnes]

On Tue, Apr 24, 2018 at 09:40:52AM +0100, Peter Humphrey wrote
> 
> Anyway, this all leaves unanswered the question whether getting the
> -march setting right in a chroot matters to the operation of the
> chroot itself, as distinct from building good code for the target
> machine. Personally, I can't see how it can have any such effect.

  "-march" obviously has to match the target machine that the compiled
code will run on.  You have to run the query...

gcc -c -Q -march=native --help=target | grep march=

...on the "target machine" to get the necessary value.  That's true,
regardless of whether the "target machine" is physical hardware, or a
chroot, or a VM like QEMU/VMWare/etc.  Things can get ugly in some edge
cases if the final code (-march=<target_machine>) cannot run in the
build environment (-march=<build_machine>).

1) Tests run at the end of the build, on the build machine, will obviously
fail.

2) Pale Moon (and I assume Firefox as well) use the compiled code at the
end of the build process to generate a precompiled startup cache.  I ran
into this when I was manually building Pale Moon, targetted for my Atom
netbook, using my Core2 desktop as the build machine.  The "bonnell" cpu
(ancient 32-bit-only Atom variant) has an instruction set called "MOVBE"
that the Core 2 does not.  Hence, the bonnell-targeted code would fail
on the Core2 with "illegal instruction" and other weird errors.  This
will also happen building for an AMD (with "3DNOW!" instructions) on an
intel CPU.  That's "solved" by adding...

ac_add_options --disable-precompiled-startupcache

...to the mozconfig, preventing execution of that code in the build
environment.  The side effect is that the first time a profile is opened
after an install or upgrade on an end-user machine, it takes somewhat
longer, because it has execute the compiled code on the end-user machine
to generate the cache.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Re: Can't fetch distfiles in chroot

[Peter Humphrey]

On Tuesday, 24 April 2018 20:02:37 BST Walter Dnes wrote:
> On Tue, Apr 24, 2018 at 09:40:52AM +0100, Peter Humphrey wrote
> 
> > Anyway, this all leaves unanswered the question whether getting the
> > -march setting right in a chroot matters to the operation of the
> > chroot itself, as distinct from building good code for the target
> > machine. Personally, I can't see how it can have any such effect.
> 
>   "-march" obviously has to match the target machine that the compiled
> code will run on.  You have to run the query...
> 
> gcc -c -Q -march=native --help=target | grep march=
> 
> ...on the "target machine" to get the necessary value.  That's true,
> regardless of whether the "target machine" is physical hardware, or a
> chroot, or a VM like QEMU/VMWare/etc.  Things can get ugly in some edge
> cases if the final code (-march=<target_machine>) cannot run in the
> build environment (-march=<build_machine>).

--->8

Yes, I understand all that. In my case, the chroot is only used as a build 
environment; nothing ever runs in it except as part of the build process.

Thanks for the extra detail.

-- 
Regards,
Peter.

Re: [gentoo-user] Can't fetch distfiles in chroot

[Peter Humphrey]

On Tuesday, 24 April 2018 09:43:35 BST Peter Humphrey wrote:
> On Sunday, 22 April 2018 06:13:30 BST Peter Humphrey wrote:
> > I've been NFS-exporting the portage treee from a 32-bit atom box to a
> > chroot on my workstation, and it's worked well for years, if slowly.
> > 
> > Now when I try to do the same with a 64-bit celeron machine I'm having a
> > problem getting portage to work. If the required distfile is already
> > present, no problem, but otherwise, trying to fetch it just hangs. No
> > errors, no status, no fetch log, no progress.
> > 
> > Www-client/links works in the chroot as expected, so the network is set up
> > all right; portage just can't use it.
> > 
> > I've compared /etc/exports on the two clients; also the chroot setup
> > scripts, /usr/portage permissions, the USE flags of nfs-utils and
> > everything else I can think of. All identical apart from obvious things
> > like 32/64 bits and network names and IPs. Google hasn't helped either.
> > 
> > Any ideas, anyone?
> 
> Never mind. I've rebuilt the chroot from stage 3 and it seems to be working
> fine.

The appearance was deceptive; all the distfiles needed must have been present 
already. Today, one wasn't and the fetching process hung and had to be killed.

So, again, what could possibly prevent portage from seeing the network inside 
a chroot, while other programs use it just the same as always?

-- 
Regards,
Peter.

Re: [gentoo-user] Can't fetch distfiles in chroot

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 304 bytes --]

On Thu, 26 Apr 2018 08:59:22 +0100, Peter Humphrey wrote:

> So, again, what could possibly prevent portage from seeing the network
> inside a chroot, while other programs use it just the same as always?

Are you using a proxy?


-- 
Neil Bothwick

Top Oxymorons Number 48: freewill offering

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Can't fetch distfiles in chroot

[Peter Humphrey]

On Thursday, 26 April 2018 08:59:22 BST Peter Humphrey wrote:
> On Tuesday, 24 April 2018 09:43:35 BST Peter Humphrey wrote:
> > On Sunday, 22 April 2018 06:13:30 BST Peter Humphrey wrote:
> > > I've been NFS-exporting the portage treee from a 32-bit atom box to a
> > > chroot on my workstation, and it's worked well for years, if slowly.
> > > 
> > > Now when I try to do the same with a 64-bit celeron machine I'm having a
> > > problem getting portage to work. If the required distfile is already
> > > present, no problem, but otherwise, trying to fetch it just hangs. No
> > > errors, no status, no fetch log, no progress.
> > > 
> > > Www-client/links works in the chroot as expected, so the network is set
> > > up
> > > all right; portage just can't use it.
> > > 
> > > I've compared /etc/exports on the two clients; also the chroot setup
> > > scripts, /usr/portage permissions, the USE flags of nfs-utils and
> > > everything else I can think of. All identical apart from obvious things
> > > like 32/64 bits and network names and IPs. Google hasn't helped either.
> > > 
> > > Any ideas, anyone?
> > 
> > Never mind. I've rebuilt the chroot from stage 3 and it seems to be
> > working
> > fine.
> 
> The appearance was deceptive; all the distfiles needed must have been
> present already. Today, one wasn't and the fetching process hung and had to
> be killed.
> 
> So, again, what could possibly prevent portage from seeing the network
> inside a chroot, while other programs use it just the same as always?

So, again, I went off half-cocked (sorry about the noise). The problem is that 
the NFS mount in the chroot picks different ports each time, so the client's 
firewall drops all NFS packets.

Now I just have to find out why that happens.

-- 
Regards,
Peter.

Re: [gentoo-user] Can't fetch distfiles in chroot

[Peter Humphrey]

On Thursday, 26 April 2018 09:07:51 BST Neil Bothwick wrote:
> On Thu, 26 Apr 2018 08:59:22 +0100, Peter Humphrey wrote:
> > So, again, what could possibly prevent portage from seeing the network
> > inside a chroot, while other programs use it just the same as always?
> 
> Are you using a proxy?

Nope, not since I had an unlimited data allowance.

-- 
Regards,
Peter.

Re: [gentoo-user] Can't fetch distfiles in chroot

[Tom H]

On Thu, Apr 26, 2018 at 4:12 AM, Peter Humphrey <peter@prh.myzen.co.uk> wrote:
>
> So, again, I went off half-cocked (sorry about the noise). The problem is that
> the NFS mount in the chroot picks different ports each time, so the client's
> firewall drops all NFS packets.
>
> Now I just have to find out why that happens.

Set up static ports for mountd and statd in "/etc/conf.d/nfs".

Set up static ports for lockd in "/etc/modprobe.d/" or
"/etc/sysctl.d/" (depending on how you compiled your kernel).

Non-official but more or less conventional ports (IIRC, first used in
an old Slackware howto):

mountd: "--port 32767"

statd: "--port 32765 --outgoing-port 32766"

lockd-sysctl.d:
fs.nfs.nlm_udpport=32768
fs.nfs.nlm_tcpport=32768

lockd--modprobe.d:
options lockd nlm_udpport=32768 nlm_tcpport=32768

[ If you want to be "modern," the nfs-utils tarball (v2.1.1 and above)
includes "nfs.conf" that you can copy into "/etc/" and edit ]