From: Fabian Groffen <grobian@gentoo.org>
To: gentoo-alt@lists.gentoo.org
Subject: Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages
Date: Sat, 13 Jun 2020 15:53:53 +0200 [thread overview]
Message-ID: <20200613135353.GO32896@gentoo.org> (raw)
In-Reply-To: <CABVqfw84ajxkWDbqi0oag4+4rD8ZGGz_VE24yqeD_wnLAdqYHw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 16744 bytes --]
I think pretty late, but this is committed in f6bbc6f470.
Thanks,
Fabian
On 03-02-2020 14:54:26 +1100, Sam Pfeiffer wrote:
> Hey Fabian,
>
> Thanks for chipping in. I do think it's useful and user-friendly to be able to
> control the target of the interactive/noninteractive script.
> I proposed in my first email in the thread something similar with the
> $STOP_AFTER_STAGE variable, even though I was capturing it from the commandline,
> and as I understand
> you propose, it would probably be better to capture it from an environment
> variable (it would actually be more in-line with other existing variables).
>
> Benda, I'm sorry for the noise I'm causing. I just want to improve the
> experience for the next person that tries to use this amazing project by
> taking advantage of already implemented stuff. And also just having this
> discussion is good to have more information archived to google about Gentoo
> Prefix and its bootstrap process. I think.
>
> On Sun, 2 Feb 2020 at 23:08, Fabian Groffen <grobian@gentoo.org[1]> wrote:
> > What I gather from this discussion is that the bootstrap script is
> > complex, and grew more complexity over the years.
> >
> > Back in the early days, there was just documentation, one needed to
> > follow. Since this frequently needed an update, and people kept using
> > older copies, at some point the bootstrap-prefix.sh script was
> > introduced.
> >
> > This script followed the documentation to the letter. It was an exact
> > replica of the instructions in the bootstrap document. This included
> > the notions of 3 stages and a world recompile.
> >
> > Today, the purpose of the stages have blurred, and seem more like
> > batches in which to do things. Some of the initial intentions still
> > stand out, e.g. that stage1 is there to provide a working portage
> > environment with some minimal efforts. Minimal got redefined as soon as
> > we had to install python and more and more tools seemed necesary (sed,
> > path, awk, etc.).
> >
> > Now, it seems you need to cut time and sort of bootstrap up-to a stage.
> > Would it be an option to simply add a(nother) var that will control the
> > interactive target in terms of how far it will run along. Normal resume
> > strategies it employs already would simply allow a next run to run along
> > further.
> >
> > I think above can be done easily with a few lines of code.
> >
> > Let me know what you think.
> >
> > Thanks,
> > Fabian
> >
> > On 23-01-2020 13:31:05 +1100, Sam Pfeiffer wrote:
> > > Grobian, any input about this?
> > >
> > > On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <[1]sammypfeiffer@gmail.com[2]>
> > wrote:
> > >
> > > > Hello Benda,
> > >
> > > > To be more specific... To easily go thru it I made a gist with the latest
> > > > version of bootstrap-prefix.sh and I'll be pointing to specific lines (this
> > > > also helps me recap why I did this, as I remember using stage{1,2,3} didn't
> > > > work for me when I first came into this):
> > >
> > > > When you execute bootstrap-prefix.sh without arguments (other than the
> > > > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> > >
> > > > [2]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[3]
> > >
> > > > Calling bootstrap_interactive
> > >
> > > > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> > > > [noninteractive, stage{1,2,3}]) it calls this line:
> > > > [3]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[4]
> > >
> > > > bootstrap_${TODO#non} || exit 1
> > >
> > > > Which, on the noninteractive case, goes to the same place than no arguments
> > > > (bootstrap_interactive). But in the stage{1,2,3} calls directly the
> > > > bootstrap_stageX function.
> > >
> > > > The thing is, noninteractive does some checks, finds resources and sets up
> > > > environment variables like:
> > >
> > > > * Check for bad flags:
> > > > [4]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[5]
> > > > * We find a gcc:
> > > > [5]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[6]
> > > > * We get a cpu count and adjust -j for Make:
> > > > [6]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[7]
> > > > * Check if we deal with multilib systems:
> > > > [7]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[8]
> > > > * Check if we are boostrapping from a Gentoo system:
> > > > [8]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[9]
> > > > * Set our EPREFIX:
> > > > [9]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[10]
> > >
> > > > Which is summarised at the end by setting the environment variables EPREFIX,
> > > > CHOST, PATH, MAKEOPTS.
> > >
> > > > And then finally calls bootstrap_stage1_log, which runs bootstrap_stage1
> > > > itself.
> > >
> > > > [10]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[11]
> > >
> > > > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> > >
> > > > Calls bootstrap_stage1():
> > > > [11]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[12]
> > >
> > > > And as you can see it won't do any of the extra work done in
> > > > bootstrap_interactive().
> > >
> > > > I'm not that versed in this script, so I may be missing something easy to
> > > > overcome this (or I don't know enough about what is going on), but there is
> > no
> > > > separate function that does everything in bootstrap_interactive without
> > > > calling bootstrap_stage1. And bootstrap_stage1 does not do all that setup
> > that
> > > > bootstrap_interactive does. So that's how I came to my proposal.
> > >
> > > > If there is some patch to enable that setup from bootstrap_interactive with
> > > > bootstrap_stage1, that would be a good alternative, I guess. I still like
> > the
> > > > fact that with my approach you are just saying "bootstrap everything until
> > > > this step" instead of needing to manually do bootstrap_setup (imaginary
> > > > function I just named) then bootstrap_stage1, then bootstrap_stage2, then
> > > > bootstrap_stage3, and then no argument for the last emerge -e system, to get
> > > > to a specific bootstrap stage, but that's just commodity.
> > >
> > > > Let me know if I'm wrong somewhere or you think of any better solution!
> > >
> > > > On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <[12]sammypfeiffer@gmail.com[13]>
> > > > wrote:
> > >
> > > >> Hi Benda,
> > >
> > > >> Thanks for your reply.
> > >
> > > >> On Thu, 16 Jan 2020 at 13:01, Benda Xu <[13]heroxbd@gentoo.org[14]> wrote:
> > >
> > > >>> Hi Sam,
> > >
> > > >>> Sam Pfeiffer <[14]sammypfeiffer@gmail.com[15]> writes:
> > >
> > > >>> > To ease automated building and debugging of the bootstrap of Gentoo
> > > >>> > Prefix I would like to propose a patch to add the option to stop the
> > > >>> > non-interactive bootstrap on specific stages of the bootstrap.
> > > >>> >
> > > >>> > I'm currently doing this in my CI projects [1][2] and I'd love to have
> > > >>> > this added upstream.
> > > >>> >
> > > >>> > I propose a patch like this:
> > > >>> >[15]https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > [16]
> > > >>> >
> > > >>> > Which is what I'm currently using. It just adds another optional
> > > >>> >parameter to the commandline (a 3rd one) in non-interactive mode which
> > > >>> >you tell it after which stage to stop (stage1, stage2, stage3).
> > >
> > > >>> > I'm not the most versed person in bash, so maybe there is a more
> > > >>> > conceptually beautiful way of doing this. My patch is just a proposal
> > > >>> > (that has been tested and running).
> > > >>> >
> > > >>> > This patch is useful because it allows:
> > > >>> > 1) Easier debugging by stages.
> > >
> > > >>> > 2) Allows to split the bootstrap job in parts in CI environments with
> > > >>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
> > > >>> > and the bootstrap takes in between 5h40-6h40 depending on random
> > > >>> > things like time to download things, and load on the machine... or
> > > >>> > packages just taking longer to build).
> > > >>> >
> > > >>> > And this patch should not change anything to anyone else (AFAIK).
> > >
> > > >>> Are you aware that in bootstrap-prefix.sh you can specify the stage you
> > > >>> are after? e.g.
> > >
> > > >>> [16]https://wiki.gentoo.org/wiki/Project:Prefix/
> > Manual_Bootstrap#Stage_1[17]
> > >
> > > >>> The text above that could be outdated, but PATH exports and stage1~3 are
> > > >>> correct.
> > >
> > > >> I'm aware, but the behaviour (relating to environment variables and
> > choosing
> > > >> CPU numbers) differs from the
> > >
> > > >> noninteractive argument. I'd like to keep the CI environment as close as a
> > > >> user executing ./bootstrap-prefix.sh EPREFIX as possible.
> > >
> > > >> That's why I'm proposing this addition. Another approach could be modifying
> > > >> the current argument to do the same than the noninteractive one.
> > >
> > > >> But that may break the workflow of someone, and I don't want that.
> > >
> > > >>
> > >
> > > >>> > Thanks for your time and feedback!
> > > >>> >
> > > >>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
> > > >>> > interface to do a Pull Request... I don't know how to do this with
> > > >>> > [17]https://gitweb.gentoo.org/repo/proj/prefix.git/[18] If there is no
> > way,
> > > >>> > aside of the goal of this email, I'd propose to have a mirror in
> > > >>> > GitHub.
> > >
> > > >>> Yes, that's a good to have for me, but not in high priority.
> > >
> > > >>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
> > > >>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
> > > >>> > approximately.
> > >
> > > >>> Thank you so much for all your work! That's extremely helpful to keep
> > > >>> the quality of Prefix codebase.
> > >
> > > >>> > [1] [18]https://github.com/awesomebytes/gentoo_prefix_ci[19]
> > > >>> > [2] [19]https://github.com/awesomebytes/gentoo_prefix_ci_32b[20]
> > >
> > > >>> Yours,
> > > >>> Benda
> > >
> > > >> --
> > >
> > > >> Sammy Pfeiffer
> > > >> PhD Candidate at The Magic Lab within UTS.
> > >
> > > > --
> > >
> > > > Sammy Pfeiffer
> > > > PhD Candidate at The Magic Lab within UTS.
> > >
> > >
> > >
> > > References:
> > > 1. mailto:sammypfeiffer@gmail.com[21]
> > > 2. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[22]
> > > 3. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[23]
> > > 4. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[24]
> > > 5. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[25]
> > > 6. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[26]
> > > 7. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[27]
> > > 8. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[28]
> > > 9. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[29]
> > > 10. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[30]
> > > 11. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[31]
> > > 12. mailto:sammypfeiffer@gmail.com[32]
> > > 13. mailto:heroxbd@gentoo.org[33]
> > > 14. mailto:sammypfeiffer@gmail.com[34]
> > > 15. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > [35]
> > > 16. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1[36]
> > > 17. https://gitweb.gentoo.org/repo/proj/prefix.git/[37]
> > > 18. https://github.com/awesomebytes/gentoo_prefix_ci[38]
> > > 19. https://github.com/awesomebytes/gentoo_prefix_ci_32b[39]
> > >
> > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > --
> > Fabian Groffen
> > Gentoo on a different level
>
>
> --
>
> Sammy Pfeiffer
> PhD Candidate at The Magic Lab within UTS.
>
>
> References
> 1. mailto:grobian@gentoo.org
> 2. mailto:sammypfeiffer@gmail.com
> 3. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> 4. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> 5. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> 6. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> 7. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> 8. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> 9. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> 10. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> 11. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> 12. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> 13. mailto:sammypfeiffer@gmail.com
> 14. mailto:heroxbd@gentoo.org
> 15. mailto:sammypfeiffer@gmail.com
> 16. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> 17. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
> 18. https://gitweb.gentoo.org/repo/proj/prefix.git/
> 19. https://github.com/awesomebytes/gentoo_prefix_ci
> 20. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> 21. mailto:sammypfeiffer@gmail.com
> 22. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> 23. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> 24. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> 25. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> 26. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> 27. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> 28. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> 29. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> 30. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> 31. https://gist.github.com/awesomebytes/
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> 32. mailto:sammypfeiffer@gmail.com
> 33. mailto:heroxbd@gentoo.org
> 34. mailto:sammypfeiffer@gmail.com
> 35. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> 36. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
> 37. https://gitweb.gentoo.org/repo/proj/prefix.git/
> 38. https://github.com/awesomebytes/gentoo_prefix_ci
> 39. https://github.com/awesomebytes/gentoo_prefix_ci_32b
--
Fabian Groffen
Gentoo on a different level
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
next prev parent reply other threads:[~2020-06-13 13:54 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-13 9:03 [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages Sam Pfeiffer
2020-01-16 1:56 ` Benda Xu
2020-01-16 2:42 ` Sam Pfeiffer
2020-01-16 6:39 ` Sam Pfeiffer
2020-01-23 2:31 ` Sam Pfeiffer
2020-02-02 12:07 ` Fabian Groffen
2020-02-03 3:54 ` Sam Pfeiffer
2020-02-03 7:08 ` Benda Xu
2020-06-13 13:53 ` Fabian Groffen [this message]
2020-06-14 11:18 ` Sam Pfeiffer
2020-01-25 13:01 ` Benda Xu
2020-01-27 9:43 ` Sam Pfeiffer
2020-01-27 10:26 ` Benda Xu
2020-01-27 11:26 ` Sam Pfeiffer
2020-01-29 11:38 ` Benda Xu
2020-02-05 12:42 ` [gentoo-alt] " Michael Haubenwallner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200613135353.GO32896@gentoo.org \
--to=grobian@gentoo.org \
--cc=gentoo-alt@lists.gentoo.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox