From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id C1D43138350 for ; Mon, 27 Jan 2020 09:44:11 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id D3999E088F; Mon, 27 Jan 2020 09:44:10 +0000 (UTC) Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 71C87E088F for ; Mon, 27 Jan 2020 09:44:10 +0000 (UTC) Received: by mail-lf1-x133.google.com with SMTP id z18so5728925lfe.2 for ; Mon, 27 Jan 2020 01:44:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=7xX+fnCy2LxGCvr50Zc09LJf5znU4bkdy9Ngh717T3k=; b=cwkJm/KyKq0Xp3pUn0YhCqE0DIig5xAS+jYjDtVh6vCs/VIzQmTLSKq2Nl3pFRNytO BVMA7s9NAxYXUh9ClNphR7SSyA3HshAB9V4By4EJPN3dfCFVoR1DKuk0FOW5vrjbwqQ9 EFuAEXW0dyuqaamh0omW1A8YJpEgO3i3rX4L62iQZVwZAYxtZESdeH+kajaX2dT+A+U9 2tIHgFwjMdvkqlRdjoFXgatKBzk+uANPQeiqlHey/3lHF6WF7DmktsVKs7vhIJ8OGDwz +igAudej65FfE7t4xXWcI6xg7mB8pMsL6es+BneYAmKDKo8JODf3Dvaev7laLebAI2QX 6bUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=7xX+fnCy2LxGCvr50Zc09LJf5znU4bkdy9Ngh717T3k=; b=EM5Yo04h4xl93qDD706jyM9mTtuD9zHFNaBj/JFmexHnEJMlDpLeBpl2wQrgWTcOjJ AZoXzGzLAxE5VmVNaTpdoP4Et8liqVNPxKvQVrXCiUvegm7lFXh7jc45AU8naFZj3bw2 bNYBMoUuNOk1R4i7jB8qUtKh0U2Y4ydIuG54L6iR3LYCs8tVIDRiWH1AX4Jyx8mkVZQ4 ESkl3uOS+9USvlVgrE6fh6qNf2IQyQ0t6G3TLAYW1Dq+MRmx3o4i6KBnWjVY6xmEADgB Fy7RRTyUlFfuB/mmCvcRu/hAt30EfRQKufFdoROXQ//leytepviyHsIWBg3aKEyqloEH Q+uQ== X-Gm-Message-State: APjAAAVeh2/j8jTxwQKgA5gkt0qtZN8fn+4AqTd+M3jxGkpcPVjLBImb HbRsEdfVTqyj/NMaP7WgnGPtVvtDZIaaUlH65GHE/2bb X-Google-Smtp-Source: APXvYqyb0PHW7LBgxiV8tPAut6vDjYWIaj2+oPu9o8686mfFWtLizJ3OaQnRCwCeQ+IHPXt53FYz9RL2dh4X5TerI1A= X-Received: by 2002:ac2:43c6:: with SMTP id u6mr7634419lfl.191.1580118248905; Mon, 27 Jan 2020 01:44:08 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-alt@lists.gentoo.org Reply-to: gentoo-alt@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <874kwwduzd.fsf@robot.e.airelinux.org> <87blqrlmf2.fsf@gentoo.org> In-Reply-To: <87blqrlmf2.fsf@gentoo.org> From: Sam Pfeiffer Date: Mon, 27 Jan 2020 20:43:57 +1100 Message-ID: Subject: Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages To: Benda Xu Cc: gentoo-alt@lists.gentoo.org Content-Type: multipart/alternative; boundary="000000000000edde49059d1bf28b" X-Archives-Salt: b8c6bb0d-4251-43ca-ad37-d6e1d6c17908 X-Archives-Hash: 9cd027a548e7d5015e9d0a107eb71540 --000000000000edde49059d1bf28b Content-Type: text/plain; charset="UTF-8" Hello Benda, On Sat, 25 Jan 2020 at 23:58, Benda Xu wrote: > Hi Sam, > > Sam Pfeiffer writes: > > > To be more specific... To easily go thru it I made a gist with the > > latest version of bootstrap-prefix.sh and I'll be pointing to specific > > lines (this also helps me recap why I did this, as I remember using > > stage{1,2,3} didn't work for me when I first came into this): > > > > When you execute bootstrap-prefix.sh without arguments (other than the > > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line: > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033 > > Calling bootstrap_interactive > > > > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX > > [noninteractive, stage{1,2,3}]) it calls this line: > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082 > > bootstrap_${TODO#non} || exit 1 > > > > Which, on the noninteractive case, goes to the same place than no > > arguments (bootstrap_interactive). But in the stage{1,2,3} calls > > directly the bootstrap_stageX function. > > > > The thing is, noninteractive does some checks, finds resources and > > sets up environment variables like: > > > > * Check for bad flags: > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247 > > * We find a gcc: > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433 > > * We get a cpu count and adjust -j for Make: > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501 > > * Check if we deal with multilib systems: > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576 > > * Check if we are boostrapping from a Gentoo system: > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608 > > * Set our EPREFIX: > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681 > > > > Which is summarised at the end by setting the environment variables > EPREFIX, CHOST, PATH, MAKEOPTS. > > > > And then finally calls bootstrap_stage1_log, which runs > > bootstrap_stage1 itself. > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726 > > > > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1... > > Calls bootstrap_stage1(): > > > https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355 > > > > And as you can see it won't do any of the extra work done in > > bootstrap_interactive(). > > Perfect! You are correct in understanding the script. > Great, that's kind of a relief. > > > I'm not that versed in this script, so I may be missing something easy > > to overcome this (or I don't know enough about what is going on), but > > there is no separate function that does everything in > > bootstrap_interactive without calling bootstrap_stage1. And > > bootstrap_stage1 does not do all that setup that bootstrap_interactive > > does. So that's how I came to my proposal. > > > > If there is some patch to enable that setup from bootstrap_interactive > > with bootstrap_stage1, that would be a good alternative, I guess. I > > still like the fact that with my approach you are just saying > > "bootstrap everything until this step" instead of needing to manually > > do bootstrap_setup (imaginary function I just named) then > > bootstrap_stage1, then bootstrap_stage2, then bootstrap_stage3, and > > then no argument for the last emerge -e system, to get to a specific > > bootstrap stage, but that's just commodity. > > Why manually do bootstrap_setup then stage 1-3, emerge -e @system a bad > thing? That gives you more control of what to do in the CI system. I > would prefer to have the explicit steps than environmental switches. > It took me a while to figure out how to do it with minimal changes... but this is the new version I came up with: https://gist.github.com/awesomebytes/c930100ca97dd4195123104c5396b645 The changes to the current bootstrap_prefix.sh are these (in .patch format): https://gist.github.com/awesomebytes/ab1f237feea2f51ae65a5ecca203842e That patch in words: I've wrapped the setup stuff in a function called *bootstrap_get_defaults* (as that's what it is doing). Then the complete bootstrap process is named *bootstrap_all_stages*. I also separated the last step calling it *bootstrap_emerge_emptytree_system*. This keeps the usage of the script the same for the interactive case, the non interactive case, but...There is a 'maybe' breaking change, as now giving the name of the function to execute first calls *bootstrap_get_defaults*, then calls the function... and does a couple of necessary workarounds. I say 'maybe' breaking, as it doesn't 'exactly' do the same it did before when executing *./bootstrap-prefix.sh EPREFIX stage2 * (but I think it was either broken or I don't know how it was used, which is the most likely option). If you don't like this new behaviour, my alternative implementation was to create extra functions like: *bootstrap_stage1_noninteractive, bootstrap_stage2_noninteractive, bootstrap_stage3_noninteractive, bootstrap_emerge_emptytree_system_noninteractive* Which would do the bootstrap_get_defaults and the workarounds. Let me know what you think. I can prepare a patch with that too. If that's the case, I'd love an explanation on how to use the current behaviour of the script. > > > Let me know if I'm wrong somewhere or you think of any better solution! > > You are nowhere wrong. I just prefer keeping the manual bootstrap > options up to date and have it drive our CI. > > Yours, > Benda > -- *Sammy Pfeiffer* PhD Candidate at The Magic Lab within UTS. --000000000000edde49059d1bf28b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Benda,



On Sat, 25 Jan 2020 at 23:58, Benda Xu <heroxbd@gentoo.org> wrote:
Hi Sam,

Sam Pfeiffer <sammypfeiffer@gmail.com> writes:

> To be more specific... To easily go thru it I made a gist with the
> latest version of bootstrap-prefix.sh and I'll be pointing to spec= ific
> lines (this also helps me recap why I did this, as I remember using > stage{1,2,3} didn't work for me when I first came into this):
>
> When you execute bootstrap-prefix.sh without arguments (other than the=
> EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#= file-bootstrap-prefix-sh-L3033
> Calling bootstrap_interactive
>
> If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> [noninteractive, stage{1,2,3}]) it calls this line:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#= file-bootstrap-prefix-sh-L3082
> bootstrap_${TODO#non} || exit 1
>
> Which, on the noninteractive case, goes to the same place than no
> arguments (bootstrap_interactive). But in the stage{1,2,3} calls
> directly the bootstrap_stageX function.
>
> The thing is, noninteractive does some checks, finds resources and
> sets up environment variables like:
>
> * Check for bad flags:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7= 998c3d#file-bootstrap-prefix-sh-L2200-L2247
> * We find a gcc:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7= 998c3d#file-bootstrap-prefix-sh-L2273-L2433
> * We get a cpu count and adjust -j for Make:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7= 998c3d#file-bootstrap-prefix-sh-L2443-L2501
> * Check if we deal with multilib systems:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7= 998c3d#file-bootstrap-prefix-sh-L2503-L2576
> * Check if we are boostrapping from a Gentoo system:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7= 998c3d#file-bootstrap-prefix-sh-L2578-L2608
> * Set our EPREFIX:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7= 998c3d#file-bootstrap-prefix-sh-L2610-L2681
>
> Which is summarised at the end by setting the environment variables EP= REFIX, CHOST, PATH, MAKEOPTS.
>
> And then finally calls bootstrap_stage1_log, which runs
> bootstrap_stage1 itself.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7= 998c3d#file-bootstrap-prefix-sh-L2724-L2726
>
> Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1... > Calls bootstrap_stage1():
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#= file-bootstrap-prefix-sh-L1355
>
> And as you can see it won't do any of the extra work done in
> bootstrap_interactive().

Perfect! You are correct in understanding the script.
=
Great, that's kind of a relief.

> I'm not that versed in this script, so I may be missing something = easy
> to overcome this (or I don't know enough about what is going on), = but
> there is no separate function that does everything in
> bootstrap_interactive without calling bootstrap_stage1. And
> bootstrap_stage1 does not do all that setup that bootstrap_interactive=
> does. So that's how I came to my proposal.
>
> If there is some patch to enable that setup from bootstrap_interactive=
> with bootstrap_stage1, that would be a good alternative, I guess. I > still like the fact that with my approach you are just saying
> "bootstrap everything until this step" instead of needing to= manually
> do bootstrap_setup (imaginary function I just named) then
> bootstrap_stage1, then bootstrap_stage2, then bootstrap_stage3, and > then no argument for the last emerge -e system, to get to a specific > bootstrap stage, but that's just commodity.

Why manually do bootstrap_setup then stage 1-3, emerge -e @system a bad
thing?=C2=A0 That gives you more control of what to do in the CI system.=C2= =A0 I
would prefer to have the explicit steps than environmental switches.

It took me a while to figure out how to do it with= minimal changes... but this is the new version I came up with:

https://gi= st.github.com/awesomebytes/c930100ca97dd4195123104c5396b645

The changes to th= e current bootstrap_prefix.sh are these (in .patch format): https://= gist.github.com/awesomebytes/ab1f237feea2f51ae65a5ecca203842e

That patch in w= ords: I've wrapped the setup stuff in a function called bootstrap_ge= t_defaults (as that's what it is doing). Then the complete bootstra= p
process is named bootstrap_all_stages<= /i>. I also separated the last step calling it bootstrap_emerge_emptytre= e_system.

This keeps the usage of the script the same for the interactive cas= e, the non interactive case, but...There is a 'maybe' breaking chan= ge,
as now giving the name of the func= tion to execute first calls bootstrap_get_defaults, then calls the f= unction...
and does a couple of necessary w= orkarounds.

I say 'maybe= ' breaking, as it doesn't 'exactly' do the same it did befo= re when executing ./bootstrap-prefix.sh EPREFIX stage2
(but I think it was either broken or I don't know how it was used, wh= ich is the most likely option).

= If you don't like this new behaviour, my alternative implementation was= to create extra functions like:
bootstrap_stage1_noninteracti= ve, bootstrap_stage2_noninteractive, bootstrap_stage3_noninteractive, boots= trap_emerge_emptytree_system_noninteractive
Which would do the bootstrap_get_defaults and the workar= ounds.

Let me know what you think. I can prepare a= patch with that too. If that's the case, I'd love an explanation o= n how to use the current behaviour of the script.
=C2=A0

> Let me know if I'm wrong somewhere or you think of any better solu= tion!

You are nowhere wrong.=C2=A0 I just prefer keeping the manual bootstrap
options up to date and have it drive our CI.

Yours,
Benda


--

Sammy Pfeiffer
PhD Candidate at The Magic Lab within UTS.
--000000000000edde49059d1bf28b--