[gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

[-- Attachment #1: Type: text/plain, Size: 1970 bytes --]

Hello,

To ease automated building and debugging of the bootstrap of Gentoo Prefix
I would like to propose a patch to add the option to stop the
non-interactive bootstrap on specific stages of the bootstrap.

I'm currently doing this in my CI projects [1]
<https://github.com/awesomebytes/gentoo_prefix_ci>[2]
<https://github.com/awesomebytes/gentoo_prefix_ci_32b> and I'd love to have
this added upstream.

I propose a patch like this:
https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f

Which is what I'm currently using. It just adds another optional parameter
to the commandline (a 3rd one) in non-interactive mode which you tell it
after which stage to stop (stage1, stage2, stage3).

I'm not the most versed person in bash, so maybe there is a more
conceptually beautiful way of doing this. My patch is just a proposal (that
has been tested and running).

This patch is useful because it allows:
1) Easier debugging by stages.
2) Allows to split the bootstrap job in parts in CI environments with time
limits on jobs to run (e.g. Azure Pipelines time limit is 6h*, and the
bootstrap takes in between 5h40-6h40 depending on random things like time
to download things, and load on the machine... or packages just taking
longer to build).

And this patch should not change anything to anyone else (AFAIK).

Thanks for your time and feedback!

P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
interface to do a Pull Request... I don't know how to do this with
https://gitweb.gentoo.org/repo/proj/prefix.git/
If there is no way, aside of the goal of this email, I'd propose to have a
mirror in GitHub.

*If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage 3
takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
approximately.

[1] https://github.com/awesomebytes/gentoo_prefix_ci
[2] https://github.com/awesomebytes/gentoo_prefix_ci_32b

-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.

[-- Attachment #2: Type: text/html, Size: 2827 bytes --]

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Benda Xu]

Hi Sam,

Sam Pfeiffer <sammypfeiffer@gmail.com> writes:

> To ease automated building and debugging of the bootstrap of Gentoo
> Prefix I would like to propose a patch to add the option to stop the
> non-interactive bootstrap on specific stages of the bootstrap.
>
> I'm currently doing this in my CI projects [1][2] and I'd love to have
> this added upstream.
>
> I propose a patch like this:
>https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>
> Which is what I'm currently using. It just adds another optional
>parameter to the commandline (a 3rd one) in non-interactive mode which
>you tell it after which stage to stop (stage1, stage2, stage3).

> I'm not the most versed person in bash, so maybe there is a more
> conceptually beautiful way of doing this. My patch is just a proposal
> (that has been tested and running).
>
> This patch is useful because it allows:
> 1) Easier debugging by stages.

> 2) Allows to split the bootstrap job in parts in CI environments with
> time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
> and the bootstrap takes in between 5h40-6h40 depending on random
> things like time to download things, and load on the machine... or
> packages just taking longer to build).
>
> And this patch should not change anything to anyone else (AFAIK).

Are you aware that in bootstrap-prefix.sh you can specify the stage you
are after?  e.g.

   https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1

The text above that could be outdated, but PATH exports and stage1~3 are
correct.

> Thanks for your time and feedback!
>
> P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
> interface to do a Pull Request... I don't know how to do this with
> https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is no way,
> aside of the goal of this email, I'd propose to have a mirror in
> GitHub.

Yes, that's a good to have for me, but not in high priority.

> *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
> 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
> approximately.

Thank you so much for all your work!  That's extremely helpful to keep
the quality of Prefix codebase.

> [1] https://github.com/awesomebytes/gentoo_prefix_ci
> [2] https://github.com/awesomebytes/gentoo_prefix_ci_32b

Yours,
Benda

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

[-- Attachment #1: Type: text/plain, Size: 3079 bytes --]

Hi Benda,

Thanks for your reply.

On Thu, 16 Jan 2020 at 13:01, Benda Xu <heroxbd@gentoo.org> wrote:

> Hi Sam,
>
> Sam Pfeiffer <sammypfeiffer@gmail.com> writes:
>
> > To ease automated building and debugging of the bootstrap of Gentoo
> > Prefix I would like to propose a patch to add the option to stop the
> > non-interactive bootstrap on specific stages of the bootstrap.
> >
> > I'm currently doing this in my CI projects [1][2] and I'd love to have
> > this added upstream.
> >
> > I propose a patch like this:
> >https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> >
> > Which is what I'm currently using. It just adds another optional
> >parameter to the commandline (a 3rd one) in non-interactive mode which
> >you tell it after which stage to stop (stage1, stage2, stage3).
>
> > I'm not the most versed person in bash, so maybe there is a more
> > conceptually beautiful way of doing this. My patch is just a proposal
> > (that has been tested and running).
> >
> > This patch is useful because it allows:
> > 1) Easier debugging by stages.
>
> > 2) Allows to split the bootstrap job in parts in CI environments with
> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
> > and the bootstrap takes in between 5h40-6h40 depending on random
> > things like time to download things, and load on the machine... or
> > packages just taking longer to build).
> >
> > And this patch should not change anything to anyone else (AFAIK).
>
> Are you aware that in bootstrap-prefix.sh you can specify the stage you
> are after?  e.g.
>
>    https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>
> The text above that could be outdated, but PATH exports and stage1~3 are
> correct.
>
I'm aware, but the behaviour (relating to environment variables and
choosing CPU numbers) differs from the
noninteractive argument. I'd like to keep the CI environment as close as a
user executing ./bootstrap-prefix.sh EPREFIX as possible.
That's why I'm proposing this addition. Another approach could be modifying
the current argument to do the same than the noninteractive one.
But that may break the workflow of someone, and I don't want that.


>
> > Thanks for your time and feedback!
> >
> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
> > interface to do a Pull Request... I don't know how to do this with
> > https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is no way,
> > aside of the goal of this email, I'd propose to have a mirror in
> > GitHub.
>
> Yes, that's a good to have for me, but not in high priority.
>
> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
> > approximately.
>
> Thank you so much for all your work!  That's extremely helpful to keep
> the quality of Prefix codebase.
>
> > [1] https://github.com/awesomebytes/gentoo_prefix_ci
> > [2] https://github.com/awesomebytes/gentoo_prefix_ci_32b
>
> Yours,
> Benda
>
>

-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.

[-- Attachment #2: Type: text/html, Size: 4725 bytes --]

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

[-- Attachment #1: Type: text/plain, Size: 6849 bytes --]

Hello Benda,

To be more specific... To easily go thru it I made a gist with the latest
version of *bootstrap-prefix.sh* and I'll be pointing to specific lines
(this also helps me recap why I did this, as I remember using stage{1,2,3}
didn't work for me when I first came into this):

When you execute *bootstrap-prefix.sh* without arguments (other than the
EPREFIX, *./bootstrap-prefix.sh /MY/EPREFIX*) it calls this line:
https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
Calling *bootstrap_interactive*

If you give an extra argument (*./bootstrap-prefix.sh /MY/EPREFIX
[noninteractive, stage{1,2,3}]*) it calls this line:
https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
bootstrap_${TODO#non} || exit 1

Which, on the noninteractive case, goes to the same place than no arguments
(*bootstrap_interactive*). But in the stage{1,2,3} calls directly the
*bootstrap_stageX* function.

The thing is, noninteractive does some checks, finds resources and sets up
environment variables like:

   - Check for bad flags:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
   - We find a gcc:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
   - We get a cpu count and adjust -j for Make:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
   - Check if we deal with multilib systems:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
   - Check if we are boostrapping from a Gentoo system:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
   - Set our EPREFIX:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681

Which is summarised at the end by setting the environment variables
EPREFIX, CHOST, PATH, MAKEOPTS.

And then finally calls *bootstrap_stage1_log*, which runs* bootstrap_stage1*
itself.
https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726

Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
Calls *bootstrap_stage1()*:
https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355

And as you can see it won't do any of the extra work done in
*bootstrap_interactive()*.

I'm not that versed in this script, so I may be missing something easy to
overcome this (or I don't know enough about what is going on), but there is
no separate function that does everything in* bootstrap_interactive*
without calling *bootstrap_stage1*. And *bootstrap_stage1* does not do all
that setup that *bootstrap_interactive* does. So that's how I came to my
proposal.

If there is some patch to enable that setup from *bootstrap_interactive*
with* bootstrap_stage1*, that would be a good alternative, I guess. I still
like the fact that with my approach you are just saying "bootstrap
everything until this step" instead of needing to manually do
*bootstrap_setup* (imaginary function I just named) then *bootstrap_stage1*,
then *bootstrap_stage2*, then* bootstrap_stage3*, and then no argument for
the last *emerge -e system*, to get to a specific bootstrap stage, but
that's just commodity.

Let me know if I'm wrong somewhere or you think of any better solution!

On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <sammypfeiffer@gmail.com> wrote:

> Hi Benda,
>
> Thanks for your reply.
>
> On Thu, 16 Jan 2020 at 13:01, Benda Xu <heroxbd@gentoo.org> wrote:
>
>> Hi Sam,
>>
>> Sam Pfeiffer <sammypfeiffer@gmail.com> writes:
>>
>> > To ease automated building and debugging of the bootstrap of Gentoo
>> > Prefix I would like to propose a patch to add the option to stop the
>> > non-interactive bootstrap on specific stages of the bootstrap.
>> >
>> > I'm currently doing this in my CI projects [1][2] and I'd love to have
>> > this added upstream.
>> >
>> > I propose a patch like this:
>> >https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>> >
>> > Which is what I'm currently using. It just adds another optional
>> >parameter to the commandline (a 3rd one) in non-interactive mode which
>> >you tell it after which stage to stop (stage1, stage2, stage3).
>>
>> > I'm not the most versed person in bash, so maybe there is a more
>> > conceptually beautiful way of doing this. My patch is just a proposal
>> > (that has been tested and running).
>> >
>> > This patch is useful because it allows:
>> > 1) Easier debugging by stages.
>>
>> > 2) Allows to split the bootstrap job in parts in CI environments with
>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
>> > and the bootstrap takes in between 5h40-6h40 depending on random
>> > things like time to download things, and load on the machine... or
>> > packages just taking longer to build).
>> >
>> > And this patch should not change anything to anyone else (AFAIK).
>>
>> Are you aware that in bootstrap-prefix.sh you can specify the stage you
>> are after?  e.g.
>>
>>    https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>>
>> The text above that could be outdated, but PATH exports and stage1~3 are
>> correct.
>>
> I'm aware, but the behaviour (relating to environment variables and
> choosing CPU numbers) differs from the
> noninteractive argument. I'd like to keep the CI environment as close as a
> user executing ./bootstrap-prefix.sh EPREFIX as possible.
> That's why I'm proposing this addition. Another approach could be
> modifying the current argument to do the same than the noninteractive one.
> But that may break the workflow of someone, and I don't want that.
>
>
>>
>> > Thanks for your time and feedback!
>> >
>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
>> > interface to do a Pull Request... I don't know how to do this with
>> > https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is no way,
>> > aside of the goal of this email, I'd propose to have a mirror in
>> > GitHub.
>>
>> Yes, that's a good to have for me, but not in high priority.
>>
>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
>> > approximately.
>>
>> Thank you so much for all your work!  That's extremely helpful to keep
>> the quality of Prefix codebase.
>>
>> > [1] https://github.com/awesomebytes/gentoo_prefix_ci
>> > [2] https://github.com/awesomebytes/gentoo_prefix_ci_32b
>>
>> Yours,
>> Benda
>>
>>
>
> --
>
> *Sammy Pfeiffer*
> PhD Candidate at The Magic Lab within UTS.
>


-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.

[-- Attachment #2: Type: text/html, Size: 10822 bytes --]

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

[-- Attachment #1: Type: text/plain, Size: 7186 bytes --]

Grobian, any input about this?

On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <sammypfeiffer@gmail.com> wrote:

> Hello Benda,
>
> To be more specific... To easily go thru it I made a gist with the latest
> version of *bootstrap-prefix.sh* and I'll be pointing to specific lines
> (this also helps me recap why I did this, as I remember using stage{1,2,3}
> didn't work for me when I first came into this):
>
> When you execute *bootstrap-prefix.sh* without arguments (other than the
> EPREFIX, *./bootstrap-prefix.sh /MY/EPREFIX*) it calls this line:
>
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> Calling *bootstrap_interactive*
>
> If you give an extra argument (*./bootstrap-prefix.sh /MY/EPREFIX
> [noninteractive, stage{1,2,3}]*) it calls this line:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> bootstrap_${TODO#non} || exit 1
>
> Which, on the noninteractive case, goes to the same place than no
> arguments (*bootstrap_interactive*). But in the stage{1,2,3} calls
> directly the *bootstrap_stageX* function.
>
> The thing is, noninteractive does some checks, finds resources and sets up
> environment variables like:
>
>    - Check for bad flags:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
>    - We find a gcc:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
>    - We get a cpu count and adjust -j for Make:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
>    - Check if we deal with multilib systems:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
>    - Check if we are boostrapping from a Gentoo system:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
>    - Set our EPREFIX:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
>
> Which is summarised at the end by setting the environment variables
> EPREFIX, CHOST, PATH, MAKEOPTS.
>
> And then finally calls *bootstrap_stage1_log*, which runs*
> bootstrap_stage1* itself.
>
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
>
> Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> Calls *bootstrap_stage1()*:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
>
> And as you can see it won't do any of the extra work done in
> *bootstrap_interactive()*.
>
> I'm not that versed in this script, so I may be missing something easy to
> overcome this (or I don't know enough about what is going on), but there is
> no separate function that does everything in* bootstrap_interactive*
> without calling *bootstrap_stage1*. And *bootstrap_stage1* does not do
> all that setup that *bootstrap_interactive* does. So that's how I came to
> my proposal.
>
> If there is some patch to enable that setup from *bootstrap_interactive*
> with* bootstrap_stage1*, that would be a good alternative, I guess. I
> still like the fact that with my approach you are just saying "bootstrap
> everything until this step" instead of needing to manually do
> *bootstrap_setup* (imaginary function I just named) then
> *bootstrap_stage1*, then *bootstrap_stage2*, then* bootstrap_stage3*, and
> then no argument for the last *emerge -e system*, to get to a specific
> bootstrap stage, but that's just commodity.
>
> Let me know if I'm wrong somewhere or you think of any better solution!
>
> On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <sammypfeiffer@gmail.com>
> wrote:
>
>> Hi Benda,
>>
>> Thanks for your reply.
>>
>> On Thu, 16 Jan 2020 at 13:01, Benda Xu <heroxbd@gentoo.org> wrote:
>>
>>> Hi Sam,
>>>
>>> Sam Pfeiffer <sammypfeiffer@gmail.com> writes:
>>>
>>> > To ease automated building and debugging of the bootstrap of Gentoo
>>> > Prefix I would like to propose a patch to add the option to stop the
>>> > non-interactive bootstrap on specific stages of the bootstrap.
>>> >
>>> > I'm currently doing this in my CI projects [1][2] and I'd love to have
>>> > this added upstream.
>>> >
>>> > I propose a patch like this:
>>> >https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>>> >
>>> > Which is what I'm currently using. It just adds another optional
>>> >parameter to the commandline (a 3rd one) in non-interactive mode which
>>> >you tell it after which stage to stop (stage1, stage2, stage3).
>>>
>>> > I'm not the most versed person in bash, so maybe there is a more
>>> > conceptually beautiful way of doing this. My patch is just a proposal
>>> > (that has been tested and running).
>>> >
>>> > This patch is useful because it allows:
>>> > 1) Easier debugging by stages.
>>>
>>> > 2) Allows to split the bootstrap job in parts in CI environments with
>>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
>>> > and the bootstrap takes in between 5h40-6h40 depending on random
>>> > things like time to download things, and load on the machine... or
>>> > packages just taking longer to build).
>>> >
>>> > And this patch should not change anything to anyone else (AFAIK).
>>>
>>> Are you aware that in bootstrap-prefix.sh you can specify the stage you
>>> are after?  e.g.
>>>
>>>    https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>>>
>>> The text above that could be outdated, but PATH exports and stage1~3 are
>>> correct.
>>>
>> I'm aware, but the behaviour (relating to environment variables and
>> choosing CPU numbers) differs from the
>> noninteractive argument. I'd like to keep the CI environment as close as
>> a user executing ./bootstrap-prefix.sh EPREFIX as possible.
>> That's why I'm proposing this addition. Another approach could be
>> modifying the current argument to do the same than the noninteractive one.
>> But that may break the workflow of someone, and I don't want that.
>>
>>
>>>
>>> > Thanks for your time and feedback!
>>> >
>>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
>>> > interface to do a Pull Request... I don't know how to do this with
>>> > https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is no way,
>>> > aside of the goal of this email, I'd propose to have a mirror in
>>> > GitHub.
>>>
>>> Yes, that's a good to have for me, but not in high priority.
>>>
>>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
>>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
>>> > approximately.
>>>
>>> Thank you so much for all your work!  That's extremely helpful to keep
>>> the quality of Prefix codebase.
>>>
>>> > [1] https://github.com/awesomebytes/gentoo_prefix_ci
>>> > [2] https://github.com/awesomebytes/gentoo_prefix_ci_32b
>>>
>>> Yours,
>>> Benda
>>>
>>>
>>
>> --
>>
>> *Sammy Pfeiffer*
>> PhD Candidate at The Magic Lab within UTS.
>>
>
>
> --
>
> *Sammy Pfeiffer*
> PhD Candidate at The Magic Lab within UTS.
>

[-- Attachment #2: Type: text/html, Size: 11462 bytes --]

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Benda Xu]

Hi Sam,

Sam Pfeiffer <sammypfeiffer@gmail.com> writes:

> To be more specific... To easily go thru it I made a gist with the
> latest version of bootstrap-prefix.sh and I'll be pointing to specific
> lines (this also helps me recap why I did this, as I remember using
> stage{1,2,3} didn't work for me when I first came into this):
>
> When you execute bootstrap-prefix.sh without arguments (other than the
> EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> Calling bootstrap_interactive
>
> If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> [noninteractive, stage{1,2,3}]) it calls this line:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> bootstrap_${TODO#non} || exit 1
>
> Which, on the noninteractive case, goes to the same place than no
> arguments (bootstrap_interactive). But in the stage{1,2,3} calls
> directly the bootstrap_stageX function.
>
> The thing is, noninteractive does some checks, finds resources and
> sets up environment variables like:
>
> * Check for bad flags:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> * We find a gcc:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> * We get a cpu count and adjust -j for Make:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> * Check if we deal with multilib systems:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> * Check if we are boostrapping from a Gentoo system:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> * Set our EPREFIX:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
>
> Which is summarised at the end by setting the environment variables EPREFIX, CHOST, PATH, MAKEOPTS.
>
> And then finally calls bootstrap_stage1_log, which runs
> bootstrap_stage1 itself.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
>
> Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> Calls bootstrap_stage1():
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
>
> And as you can see it won't do any of the extra work done in
> bootstrap_interactive().

Perfect! You are correct in understanding the script.

> I'm not that versed in this script, so I may be missing something easy
> to overcome this (or I don't know enough about what is going on), but
> there is no separate function that does everything in
> bootstrap_interactive without calling bootstrap_stage1. And
> bootstrap_stage1 does not do all that setup that bootstrap_interactive
> does. So that's how I came to my proposal.
>
> If there is some patch to enable that setup from bootstrap_interactive
> with bootstrap_stage1, that would be a good alternative, I guess. I
> still like the fact that with my approach you are just saying
> "bootstrap everything until this step" instead of needing to manually
> do bootstrap_setup (imaginary function I just named) then
> bootstrap_stage1, then bootstrap_stage2, then bootstrap_stage3, and
> then no argument for the last emerge -e system, to get to a specific
> bootstrap stage, but that's just commodity.

Why manually do bootstrap_setup then stage 1-3, emerge -e @system a bad
thing?  That gives you more control of what to do in the CI system.  I
would prefer to have the explicit steps than environmental switches.

> Let me know if I'm wrong somewhere or you think of any better solution!

You are nowhere wrong.  I just prefer keeping the manual bootstrap
options up to date and have it drive our CI.

Yours,
Benda

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

[-- Attachment #1: Type: text/plain, Size: 5985 bytes --]

Hello Benda,



On Sat, 25 Jan 2020 at 23:58, Benda Xu <heroxbd@gentoo.org> wrote:

> Hi Sam,
>
> Sam Pfeiffer <sammypfeiffer@gmail.com> writes:
>
> > To be more specific... To easily go thru it I made a gist with the
> > latest version of bootstrap-prefix.sh and I'll be pointing to specific
> > lines (this also helps me recap why I did this, as I remember using
> > stage{1,2,3} didn't work for me when I first came into this):
> >
> > When you execute bootstrap-prefix.sh without arguments (other than the
> > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> > Calling bootstrap_interactive
> >
> > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> > [noninteractive, stage{1,2,3}]) it calls this line:
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> > bootstrap_${TODO#non} || exit 1
> >
> > Which, on the noninteractive case, goes to the same place than no
> > arguments (bootstrap_interactive). But in the stage{1,2,3} calls
> > directly the bootstrap_stageX function.
> >
> > The thing is, noninteractive does some checks, finds resources and
> > sets up environment variables like:
> >
> > * Check for bad flags:
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> > * We find a gcc:
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> > * We get a cpu count and adjust -j for Make:
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> > * Check if we deal with multilib systems:
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> > * Check if we are boostrapping from a Gentoo system:
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> > * Set our EPREFIX:
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> >
> > Which is summarised at the end by setting the environment variables
> EPREFIX, CHOST, PATH, MAKEOPTS.
> >
> > And then finally calls bootstrap_stage1_log, which runs
> > bootstrap_stage1 itself.
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> >
> > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> > Calls bootstrap_stage1():
> >
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> >
> > And as you can see it won't do any of the extra work done in
> > bootstrap_interactive().
>
> Perfect! You are correct in understanding the script.
>

Great, that's kind of a relief.

>
> > I'm not that versed in this script, so I may be missing something easy
> > to overcome this (or I don't know enough about what is going on), but
> > there is no separate function that does everything in
> > bootstrap_interactive without calling bootstrap_stage1. And
> > bootstrap_stage1 does not do all that setup that bootstrap_interactive
> > does. So that's how I came to my proposal.
> >
> > If there is some patch to enable that setup from bootstrap_interactive
> > with bootstrap_stage1, that would be a good alternative, I guess. I
> > still like the fact that with my approach you are just saying
> > "bootstrap everything until this step" instead of needing to manually
> > do bootstrap_setup (imaginary function I just named) then
> > bootstrap_stage1, then bootstrap_stage2, then bootstrap_stage3, and
> > then no argument for the last emerge -e system, to get to a specific
> > bootstrap stage, but that's just commodity.
>
> Why manually do bootstrap_setup then stage 1-3, emerge -e @system a bad
> thing?  That gives you more control of what to do in the CI system.  I
> would prefer to have the explicit steps than environmental switches.
>

It took me a while to figure out how to do it with minimal changes... but
this is the new version I came up with:

https://gist.github.com/awesomebytes/c930100ca97dd4195123104c5396b645

The changes to the current bootstrap_prefix.sh are these (in .patch
format):
https://gist.github.com/awesomebytes/ab1f237feea2f51ae65a5ecca203842e

That patch in words: I've wrapped the setup stuff in a function called
*bootstrap_get_defaults* (as that's what it is doing). Then the complete
bootstrap
process is named *bootstrap_all_stages*. I also separated the last step
calling it *bootstrap_emerge_emptytree_system*.

This keeps the usage of the script the same for the interactive case, the
non interactive case, but...There is a 'maybe' breaking change,
as now giving the name of the function to execute first calls
*bootstrap_get_defaults*, then calls the function...
and does a couple of necessary workarounds.

I say 'maybe' breaking, as it doesn't 'exactly' do the same it did before
when executing
*./bootstrap-prefix.sh EPREFIX stage2 *
(but I think it was either broken or I don't know how it was used, which is
the most likely option).

If you don't like this new behaviour, my alternative implementation was to
create extra functions like:
*bootstrap_stage1_noninteractive, bootstrap_stage2_noninteractive,
bootstrap_stage3_noninteractive,
bootstrap_emerge_emptytree_system_noninteractive*
Which would do the bootstrap_get_defaults and the workarounds.

Let me know what you think. I can prepare a patch with that too. If that's
the case, I'd love an explanation on how to use the current behaviour of
the script.


>
> > Let me know if I'm wrong somewhere or you think of any better solution!
>
> You are nowhere wrong.  I just prefer keeping the manual bootstrap
> options up to date and have it drive our CI.
>
> Yours,
> Benda
>


-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.

[-- Attachment #2: Type: text/html, Size: 9635 bytes --]

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Benda Xu]

Hi Sam,

Sam Pfeiffer <sammypfeiffer@gmail.com> writes:

> It took me a while to figure out how to do it with minimal
> changes... but this is the new version I came up with:
>
> https://gist.github.com/awesomebytes/c930100ca97dd4195123104c5396b645
>
> The changes to the current bootstrap_prefix.sh are these (in .patch
> format):
> https://gist.github.com/awesomebytes/ab1f237feea2f51ae65a5ecca203842e
>
> That patch in words: I've wrapped the setup stuff in a function called
>bootstrap_get_defaults (as that's what it is doing). Then the complete
>bootstrap process is named bootstrap_all_stages. I also separated the
>last step calling it bootstrap_emerge_emptytree_system.
>
> This keeps the usage of the script the same for the interactive case,
> the non interactive case, but...There is a 'maybe' breaking change, as
> now giving the name of the function to execute first calls
> bootstrap_get_defaults, then calls the function...  and does a couple
> of necessary workarounds.
>
> I say 'maybe' breaking, as it doesn't 'exactly' do the same it did
> before when executing ./bootstrap-prefix.sh EPREFIX stage2 (but I
> think it was either broken or I don't know how it was used, which is
> the most likely option).

> If you don't like this new behaviour, my alternative implementation
> was to create extra functions like: bootstrap_stage1_noninteractive,
> bootstrap_stage2_noninteractive, bootstrap_stage3_noninteractive,
> bootstrap_emerge_emptytree_system_noninteractive Which would do the
> bootstrap_get_defaults and the workarounds.
>
> Let me know what you think. I can prepare a patch with that too. If
> that's the case, I'd love an explanation on how to use the current
> behaviour of the script.

I read your patches.  But I don't understand the motivation for
completely reproduce the interactive use of bootstrap-prefix.sh by
bootstrap_get_defaults() and bootstrap_emerge_emptytree_system().

The most important part of the CI is to test stages 1~3, because both
the hosts and the gentoo repository are moving targets.  In the past 10
years, there were few problems happening in bootstrap_get_defaults() and
bootstrap_emerge_emptytree_system(), or equivalents.  I would like to
propose the 3rd way:

1. set the PATH, CHOST, etc. with your CI script,
2. bootstrap-prefix.sh stage1
3. bootstrap-prefix.sh stage2
4. bootstrap-prefix.sh stage3
5. call emerge -e @system with your CI script.

What do you think?

Yours,
Benda

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

[-- Attachment #1: Type: text/plain, Size: 5110 bytes --]

Hello Benda,

On Mon, 27 Jan 2020 at 21:26, Benda Xu <heroxbd@gentoo.org> wrote:

> Hi Sam,
>
> Sam Pfeiffer <sammypfeiffer@gmail.com> writes:
>
> > It took me a while to figure out how to do it with minimal
> > changes... but this is the new version I came up with:
> >
> > https://gist.github.com/awesomebytes/c930100ca97dd4195123104c5396b645
> >
> > The changes to the current bootstrap_prefix.sh are these (in .patch
> > format):
> > https://gist.github.com/awesomebytes/ab1f237feea2f51ae65a5ecca203842e
> >
> > That patch in words: I've wrapped the setup stuff in a function called
> >bootstrap_get_defaults (as that's what it is doing). Then the complete
> >bootstrap process is named bootstrap_all_stages. I also separated the
> >last step calling it bootstrap_emerge_emptytree_system.
> >
> > This keeps the usage of the script the same for the interactive case,
> > the non interactive case, but...There is a 'maybe' breaking change, as
> > now giving the name of the function to execute first calls
> > bootstrap_get_defaults, then calls the function...  and does a couple
> > of necessary workarounds.
> >
> > I say 'maybe' breaking, as it doesn't 'exactly' do the same it did
> > before when executing ./bootstrap-prefix.sh EPREFIX stage2 (but I
> > think it was either broken or I don't know how it was used, which is
> > the most likely option).
>
> > If you don't like this new behaviour, my alternative implementation
> > was to create extra functions like: bootstrap_stage1_noninteractive,
> > bootstrap_stage2_noninteractive, bootstrap_stage3_noninteractive,
> > bootstrap_emerge_emptytree_system_noninteractive Which would do the
> > bootstrap_get_defaults and the workarounds.
> >
> > Let me know what you think. I can prepare a patch with that too. If
> > that's the case, I'd love an explanation on how to use the current
> > behaviour of the script.
>
> I read your patches.  But I don't understand the motivation for
> completely reproduce the interactive use of bootstrap-prefix.sh by
> bootstrap_get_defaults() and bootstrap_emerge_emptytree_system().
>
> The most important part of the CI is to test stages 1~3, because both
> the hosts and the gentoo repository are moving targets.  In the past 10
> years, there were few problems happening in bootstrap_get_defaults() and
> bootstrap_emerge_emptytree_system(), or equivalents.  I would like to
> propose the 3rd way:
>
> 1. set the PATH, CHOST, etc. with your CI script,
> 2. bootstrap-prefix.sh stage1
> 3. bootstrap-prefix.sh stage2
> 4. bootstrap-prefix.sh stage3
> 5. call emerge -e @system with your CI script.
>
> What do you think?
>

First, thanks for explaining your proposal in a detailed way. I sincerely
didn't understand exactly what you meant.

Let me explain further. I found "complicated" to know what environment
variables I needed to set manually in order
to make use of the bootstrap script calling it in the way you just
presented. I did attempt to do it previously and failed miserably.
When looking at the bootstrap-prefix.sh script there are a lot of variables
in a 3000+ lines script, it's kind of daunting.
And when your goal is "I just need this to bootstrap, and it looks very
complicated" you want to the most automated way
(someone smart wrote the automated way!).

Now I understood what do I need to set up.
In hindsight, it was pretty obvious, as the interactive script itself spits
out for me:
EPREFIX=/tmp/gentoo
CHOST=x86_64-pc-linux-gnu
PATH=/tmp/gentoo/usr/bin:/tmp/gentoo/bin:/tmp/gentoo/tmp/usr/bin:/tmp/gentoo/tmp/bin:/tmp/gentoo/tmp/usr/local/bin:/usr/bin:/bin
MAKEOPTS=-j5

as the variables it needs when executing the interactive/noninteractive way.
And now that I've seen that, I also see that the instructions found here:
https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap
Are not out of date as reported at the top of the document. Even though it
doesn't say you need to set CHOST and MAKEOPTS, which I guess
they are actually automatically detected. It does briefly talk about CHOST
tho.

So now I see I could write my own lines of the script that does exactly
what you said... but I still need to configure a bunch of things
by hand.

Meanwhile using the same codepath we are already using "guarantees" I don't
need to fight any of that. Neither me, you or the next
maintainer. To me, the mental model I need to create and maintain in my
head from the version I'm proposing is tinier
than from the one where I need to figure out the environment variables and
also copy-paste the MAKEOPTS guesswork.

I mean, ideally, I would just call* ./bootstrap-prefix.sh EPREFIX
noninteractive* and call it a day... but the current CI infrastructure
can't support it. So I'm trying to get the best trade-off I can making this
easier to use from my point of view.

I must say that this email thread has helped me understand better this
process. I still think that being able to call the
different stages in an automated fashion is beneficial, but I understand if
you don't want to change that.



>
> Yours,
> Benda
>
>
>

-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.

[-- Attachment #2: Type: text/html, Size: 6910 bytes --]

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Benda Xu]

Hi Sam,

Sam Pfeiffer <sammypfeiffer@gmail.com> writes:

> First, thanks for explaining your proposal in a detailed way. I
> sincerely didn't understand exactly what you meant.
>
> Let me explain further. I found "complicated" to know what environment
> variables I needed to set manually in order to make use of the
> bootstrap script calling it in the way you just presented. I did
> attempt to do it previously and failed miserably.  When looking at the
> bootstrap-prefix.sh script there are a lot of variables in a 3000+
> lines script, it's kind of daunting.  And when your goal is "I just
> need this to bootstrap, and it looks very complicated" you want to the
> most automated way (someone smart wrote the automated way!).
>
> Now I understood what do I need to set up.
> In hindsight, it was pretty obvious, as the interactive script itself spits out for me:
> EPREFIX=/tmp/gentoo
> CHOST=x86_64-pc-linux-gnu
> PATH=/tmp/gentoo/usr/bin:/tmp/gentoo/bin:/tmp/gentoo/tmp/usr/bin:/tmp/gentoo/tmp/bin:/tmp/gentoo/tmp/usr/local/bin:/usr/bin:/bin
> MAKEOPTS=-j5

You can cross out CHOST and MAKEOPTS.  Do you think setting EPREFIX and
PATH a complex task to do?

The "same codepath" you mentioned just sets these 4 variables.


I don't like this patch for extreme reproducibility.  But that's only my
personal opinion.  You can keep your version of bootstrap script for
your purpose or approach grobian for an alternative review.

> [...]

> Meanwhile using the same codepath we are already using "guarantees" I
> don't need to fight any of that. Neither me, you or the next
> maintainer. To me, the mental model I need to create and maintain in
> my head from the version I'm proposing is tinier than from the one
> where I need to figure out the environment variables and also
> copy-paste the MAKEOPTS guesswork.

I am very surprised that you need copy-paste and guesswork for the 4
variables.

> [...]

Yours,
Benda

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 10775 bytes --]

What I gather from this discussion is that the bootstrap script is
complex, and grew more complexity over the years.

Back in the early days, there was just documentation, one needed to
follow.  Since this frequently needed an update, and people kept using
older copies, at some point the bootstrap-prefix.sh script was
introduced.

This script followed the documentation to the letter.  It was an exact
replica of the instructions in the bootstrap document.  This included
the notions of 3 stages and a world recompile.

Today, the purpose of the stages have blurred, and seem more like
batches in which to do things.  Some of the initial intentions still
stand out, e.g. that stage1 is there to provide a working portage
environment with some minimal efforts.  Minimal got redefined as soon as
we had to install python and more and more tools seemed necesary (sed,
path, awk, etc.).

Now, it seems you need to cut time and sort of bootstrap up-to a stage.
Would it be an option to simply add a(nother) var that will control the
interactive target in terms of how far it will run along.  Normal resume
strategies it employs already would simply allow a next run to run along
further.

I think above can be done easily with a few lines of code.

Let me know what you think.

Thanks,
Fabian

On 23-01-2020 13:31:05 +1100, Sam Pfeiffer wrote:
> Grobian, any input about this?
> 
> On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <[1]sammypfeiffer@gmail.com> wrote:
> 
> > Hello Benda,
> 
> > To be more specific... To easily go thru it I made a gist with the latest
> > version of bootstrap-prefix.sh and I'll be pointing to specific lines (this
> > also helps me recap why I did this, as I remember using stage{1,2,3} didn't
> > work for me when I first came into this):
> 
> > When you execute bootstrap-prefix.sh without arguments (other than the
> > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> 
> > [2]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> 
> > Calling bootstrap_interactive
> 
> > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> > [noninteractive, stage{1,2,3}]) it calls this line:
> > [3]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> 
> > bootstrap_${TODO#non} || exit 1
> 
> > Which, on the noninteractive case, goes to the same place than no arguments
> > (bootstrap_interactive). But in the stage{1,2,3} calls directly the
> > bootstrap_stageX function.
> 
> > The thing is, noninteractive does some checks, finds resources and sets up
> > environment variables like:
> 
> >   * Check for bad flags:
> >   [4]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> >   * We find a gcc:
> >   [5]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> >   * We get a cpu count and adjust -j for Make:
> >   [6]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> >   * Check if we deal with multilib systems:
> >   [7]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> >   * Check if we are boostrapping from a Gentoo system:
> >   [8]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> >   * Set our EPREFIX:
> > [9]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> 
> > Which is summarised at the end by setting the environment variables EPREFIX,
> > CHOST, PATH, MAKEOPTS.
> 
> > And then finally calls bootstrap_stage1_log, which runs bootstrap_stage1
> > itself.
> 
> > [10]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> 
> > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> 
> > Calls bootstrap_stage1():
> > [11]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> 
> > And as you can see it won't do any of the extra work done in
> > bootstrap_interactive().
> 
> > I'm not that versed in this script, so I may be missing something easy to
> > overcome this (or I don't know enough about what is going on), but there is no
> > separate function that does everything in bootstrap_interactive without
> > calling bootstrap_stage1. And bootstrap_stage1 does not do all that setup that
> > bootstrap_interactive does. So that's how I came to my proposal.
> 
> > If there is some patch to enable that setup from bootstrap_interactive with
> > bootstrap_stage1, that would be a good alternative, I guess. I still like the
> > fact that with my approach you are just saying "bootstrap everything until
> > this step" instead of needing to manually do bootstrap_setup (imaginary
> > function I just named) then bootstrap_stage1, then bootstrap_stage2, then
> > bootstrap_stage3, and then no argument for the last emerge -e system, to get
> > to a specific bootstrap stage, but that's just commodity.
> 
> > Let me know if I'm wrong somewhere or you think of any better solution!
> 
> > On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <[12]sammypfeiffer@gmail.com>
> > wrote:
> 
> >> Hi Benda,
> 
> >> Thanks for your reply.
> 
> >> On Thu, 16 Jan 2020 at 13:01, Benda Xu <[13]heroxbd@gentoo.org> wrote:
> 
> >>> Hi Sam,
> 
> >>> Sam Pfeiffer <[14]sammypfeiffer@gmail.com> writes:
> 
> >>> > To ease automated building and debugging of the bootstrap of Gentoo
> >>> > Prefix I would like to propose a patch to add the option to stop the
> >>> > non-interactive bootstrap on specific stages of the bootstrap.
> >>> >
> >>> > I'm currently doing this in my CI projects [1][2] and I'd love to have
> >>> > this added upstream.
> >>> >
> >>> > I propose a patch like this:
> >>> >[15]https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> >>> >
> >>> > Which is what I'm currently using. It just adds another optional
> >>> >parameter to the commandline (a 3rd one) in non-interactive mode which
> >>> >you tell it after which stage to stop (stage1, stage2, stage3).
> 
> >>> > I'm not the most versed person in bash, so maybe there is a more
> >>> > conceptually beautiful way of doing this. My patch is just a proposal
> >>> > (that has been tested and running).
> >>> >
> >>> > This patch is useful because it allows:
> >>> > 1) Easier debugging by stages.
> 
> >>> > 2) Allows to split the bootstrap job in parts in CI environments with
> >>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
> >>> > and the bootstrap takes in between 5h40-6h40 depending on random
> >>> > things like time to download things, and load on the machine... or
> >>> > packages just taking longer to build).
> >>> >
> >>> > And this patch should not change anything to anyone else (AFAIK).
> 
> >>> Are you aware that in bootstrap-prefix.sh you can specify the stage you
> >>> are after?  e.g.
> 
> >>>    [16]https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
> 
> >>> The text above that could be outdated, but PATH exports and stage1~3 are
> >>> correct.
> 
> >> I'm aware, but the behaviour (relating to environment variables and choosing
> >> CPU numbers) differs from the
> 
> >> noninteractive argument. I'd like to keep the CI environment as close as a
> >> user executing ./bootstrap-prefix.sh EPREFIX as possible.
> 
> >> That's why I'm proposing this addition. Another approach could be modifying
> >> the current argument to do the same than the noninteractive one.
> 
> >> But that may break the workflow of someone, and I don't want that.
> 
> >>  
> 
> >>> > Thanks for your time and feedback!
> >>> >
> >>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
> >>> > interface to do a Pull Request... I don't know how to do this with
> >>> > [17]https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is no way,
> >>> > aside of the goal of this email, I'd propose to have a mirror in
> >>> > GitHub.
> 
> >>> Yes, that's a good to have for me, but not in high priority.
> 
> >>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
> >>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
> >>> > approximately.
> 
> >>> Thank you so much for all your work!  That's extremely helpful to keep
> >>> the quality of Prefix codebase.
> 
> >>> > [1] [18]https://github.com/awesomebytes/gentoo_prefix_ci
> >>> > [2] [19]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> 
> >>> Yours,
> >>> Benda
> 
> >> --
> 
> >> Sammy Pfeiffer
> >> PhD Candidate at The Magic Lab within UTS.
> 
> > --
> 
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
> 
> 
> 
>  References:
>    1. mailto:sammypfeiffer@gmail.com
>    2. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
>    3. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
>    4. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
>    5. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
>    6. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
>    7. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
>    8. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
>    9. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
>   10. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
>   11. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
>   12. mailto:sammypfeiffer@gmail.com
>   13. mailto:heroxbd@gentoo.org
>   14. mailto:sammypfeiffer@gmail.com
>   15. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>   16. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>   17. https://gitweb.gentoo.org/repo/proj/prefix.git/
>   18. https://github.com/awesomebytes/gentoo_prefix_ci
>   19. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> 
> read_char: errno==EILSEQ; invalid byte sequence for UTF-8: 
-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

[-- Attachment #1: Type: text/plain, Size: 12017 bytes --]

Hey Fabian,

Thanks for chipping in. I do think it's useful and user-friendly to be able
to control the target of the interactive/noninteractive script.
I proposed in my first email in the thread something similar with the
$STOP_AFTER_STAGE
variable, even though I was capturing it from the commandline, and as I
understand
you propose, it would probably be better to capture it from an environment
variable (it would actually be more in-line with other existing variables).

Benda, I'm sorry for the noise I'm causing. I just want to improve the
experience for the next person that tries to use this amazing project by
taking advantage of already implemented stuff. And also just having this
discussion is good to have more information archived to google about Gentoo
Prefix and its bootstrap process. I think.

On Sun, 2 Feb 2020 at 23:08, Fabian Groffen <grobian@gentoo.org> wrote:

> What I gather from this discussion is that the bootstrap script is
> complex, and grew more complexity over the years.
>
> Back in the early days, there was just documentation, one needed to
> follow.  Since this frequently needed an update, and people kept using
> older copies, at some point the bootstrap-prefix.sh script was
> introduced.
>
> This script followed the documentation to the letter.  It was an exact
> replica of the instructions in the bootstrap document.  This included
> the notions of 3 stages and a world recompile.
>
> Today, the purpose of the stages have blurred, and seem more like
> batches in which to do things.  Some of the initial intentions still
> stand out, e.g. that stage1 is there to provide a working portage
> environment with some minimal efforts.  Minimal got redefined as soon as
> we had to install python and more and more tools seemed necesary (sed,
> path, awk, etc.).
>
> Now, it seems you need to cut time and sort of bootstrap up-to a stage.
> Would it be an option to simply add a(nother) var that will control the
> interactive target in terms of how far it will run along.  Normal resume
> strategies it employs already would simply allow a next run to run along
> further.
>
> I think above can be done easily with a few lines of code.
>
> Let me know what you think.
>
> Thanks,
> Fabian
>
> On 23-01-2020 13:31:05 +1100, Sam Pfeiffer wrote:
> > Grobian, any input about this?
> >
> > On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <[1]sammypfeiffer@gmail.com>
> wrote:
> >
> > > Hello Benda,
> >
> > > To be more specific... To easily go thru it I made a gist with the
> latest
> > > version of bootstrap-prefix.sh and I'll be pointing to specific lines
> (this
> > > also helps me recap why I did this, as I remember using stage{1,2,3}
> didn't
> > > work for me when I first came into this):
> >
> > > When you execute bootstrap-prefix.sh without arguments (other than the
> > > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> >
> > > [2]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> >
> > > Calling bootstrap_interactive
> >
> > > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> > > [noninteractive, stage{1,2,3}]) it calls this line:
> > > [3]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> >
> > > bootstrap_${TODO#non} || exit 1
> >
> > > Which, on the noninteractive case, goes to the same place than no
> arguments
> > > (bootstrap_interactive). But in the stage{1,2,3} calls directly the
> > > bootstrap_stageX function.
> >
> > > The thing is, noninteractive does some checks, finds resources and
> sets up
> > > environment variables like:
> >
> > >   * Check for bad flags:
> > >   [4]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> > >   * We find a gcc:
> > >   [5]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> > >   * We get a cpu count and adjust -j for Make:
> > >   [6]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> > >   * Check if we deal with multilib systems:
> > >   [7]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> > >   * Check if we are boostrapping from a Gentoo system:
> > >   [8]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> > >   * Set our EPREFIX:
> > > [9]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> >
> > > Which is summarised at the end by setting the environment variables
> EPREFIX,
> > > CHOST, PATH, MAKEOPTS.
> >
> > > And then finally calls bootstrap_stage1_log, which runs
> bootstrap_stage1
> > > itself.
> >
> > > [10]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> >
> > > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> >
> > > Calls bootstrap_stage1():
> > > [11]
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> >
> > > And as you can see it won't do any of the extra work done in
> > > bootstrap_interactive().
> >
> > > I'm not that versed in this script, so I may be missing something easy
> to
> > > overcome this (or I don't know enough about what is going on), but
> there is no
> > > separate function that does everything in bootstrap_interactive without
> > > calling bootstrap_stage1. And bootstrap_stage1 does not do all that
> setup that
> > > bootstrap_interactive does. So that's how I came to my proposal.
> >
> > > If there is some patch to enable that setup from bootstrap_interactive
> with
> > > bootstrap_stage1, that would be a good alternative, I guess. I still
> like the
> > > fact that with my approach you are just saying "bootstrap everything
> until
> > > this step" instead of needing to manually do bootstrap_setup (imaginary
> > > function I just named) then bootstrap_stage1, then bootstrap_stage2,
> then
> > > bootstrap_stage3, and then no argument for the last emerge -e system,
> to get
> > > to a specific bootstrap stage, but that's just commodity.
> >
> > > Let me know if I'm wrong somewhere or you think of any better solution!
> >
> > > On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <[12]
> sammypfeiffer@gmail.com>
> > > wrote:
> >
> > >> Hi Benda,
> >
> > >> Thanks for your reply.
> >
> > >> On Thu, 16 Jan 2020 at 13:01, Benda Xu <[13]heroxbd@gentoo.org>
> wrote:
> >
> > >>> Hi Sam,
> >
> > >>> Sam Pfeiffer <[14]sammypfeiffer@gmail.com> writes:
> >
> > >>> > To ease automated building and debugging of the bootstrap of Gentoo
> > >>> > Prefix I would like to propose a patch to add the option to stop
> the
> > >>> > non-interactive bootstrap on specific stages of the bootstrap.
> > >>> >
> > >>> > I'm currently doing this in my CI projects [1][2] and I'd love to
> have
> > >>> > this added upstream.
> > >>> >
> > >>> > I propose a patch like this:
> > >>> >[15]
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > >>> >
> > >>> > Which is what I'm currently using. It just adds another optional
> > >>> >parameter to the commandline (a 3rd one) in non-interactive mode
> which
> > >>> >you tell it after which stage to stop (stage1, stage2, stage3).
> >
> > >>> > I'm not the most versed person in bash, so maybe there is a more
> > >>> > conceptually beautiful way of doing this. My patch is just a
> proposal
> > >>> > (that has been tested and running).
> > >>> >
> > >>> > This patch is useful because it allows:
> > >>> > 1) Easier debugging by stages.
> >
> > >>> > 2) Allows to split the bootstrap job in parts in CI environments
> with
> > >>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
> > >>> > and the bootstrap takes in between 5h40-6h40 depending on random
> > >>> > things like time to download things, and load on the machine... or
> > >>> > packages just taking longer to build).
> > >>> >
> > >>> > And this patch should not change anything to anyone else (AFAIK).
> >
> > >>> Are you aware that in bootstrap-prefix.sh you can specify the stage
> you
> > >>> are after?  e.g.
> >
> > >>>    [16]
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
> >
> > >>> The text above that could be outdated, but PATH exports and stage1~3
> are
> > >>> correct.
> >
> > >> I'm aware, but the behaviour (relating to environment variables and
> choosing
> > >> CPU numbers) differs from the
> >
> > >> noninteractive argument. I'd like to keep the CI environment as close
> as a
> > >> user executing ./bootstrap-prefix.sh EPREFIX as possible.
> >
> > >> That's why I'm proposing this addition. Another approach could be
> modifying
> > >> the current argument to do the same than the noninteractive one.
> >
> > >> But that may break the workflow of someone, and I don't want that.
> >
> > >>
> >
> > >>> > Thanks for your time and feedback!
> > >>> >
> > >>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
> > >>> > interface to do a Pull Request... I don't know how to do this with
> > >>> > [17]https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is
> no way,
> > >>> > aside of the goal of this email, I'd propose to have a mirror in
> > >>> > GitHub.
> >
> > >>> Yes, that's a good to have for me, but not in high priority.
> >
> > >>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~,
> Stage
> > >>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
> > >>> > approximately.
> >
> > >>> Thank you so much for all your work!  That's extremely helpful to
> keep
> > >>> the quality of Prefix codebase.
> >
> > >>> > [1] [18]https://github.com/awesomebytes/gentoo_prefix_ci
> > >>> > [2] [19]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >
> > >>> Yours,
> > >>> Benda
> >
> > >> --
> >
> > >> Sammy Pfeiffer
> > >> PhD Candidate at The Magic Lab within UTS.
> >
> > > --
> >
> > > Sammy Pfeiffer
> > > PhD Candidate at The Magic Lab within UTS.
> >
> >
> >
> >  References:
> >    1. mailto:sammypfeiffer@gmail.com
> >    2.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> >    3.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> >    4.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> >    5.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> >    6.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> >    7.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> >    8.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> >    9.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> >   10.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> >   11.
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> >   12. mailto:sammypfeiffer@gmail.com
> >   13. mailto:heroxbd@gentoo.org
> >   14. mailto:sammypfeiffer@gmail.com
> >   15.
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> >   16.
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
> >   17. https://gitweb.gentoo.org/repo/proj/prefix.git/
> >   18. https://github.com/awesomebytes/gentoo_prefix_ci
> >   19. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >
> > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> --
> Fabian Groffen
> Gentoo on a different level
>


-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.

[-- Attachment #2: Type: text/html, Size: 19573 bytes --]

[gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Benda Xu]

Hi Sam,

Sam Pfeiffer <sammypfeiffer@gmail.com> writes:

> Benda, I'm sorry for the noise I'm causing. I just want to improve the
> experience for the next person that tries to use this amazing project
> by taking advantage of already implemented stuff. And also just having
> this discussion is good to have more information archived to google
> about Gentoo Prefix and its bootstrap process. I think.

No problem at all for me.  Keep up with good CI work.

Benda

[gentoo-alt] Re: Allow non-interactive bootstrap to be stopped at arbitrary stages

[Michael Haubenwallner]

On 1/13/20 10:03 AM, Sam Pfeiffer wrote:
> Hello,
> 
> To ease automated building and debugging of the bootstrap of Gentoo Prefix I would like to propose a patch to add the option to stop the non-interactive bootstrap on specific stages of the bootstrap.

Given that bootstrap-prefix.sh is (=should be) able to resume from whatever
interruption happened to the bootstrap process, what do you think about this
alternative solution:

When starting up the bootstrap process, also set up some shell timer of eg.
5.5 hours (sleep 19800) to kill bootstrap-process.sh, leaving .5 hours to
upload the current docker images much like you already do even after breakage.

> I'm currently doing this in my CI projects [1] <https://github.com/awesomebytes/gentoo_prefix_ci>[2] <https://github.com/awesomebytes/gentoo_prefix_ci_32b> and I'd love to have this added upstream.
> 
> I propose a patch like this: https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> 
> Which is what I'm currently using. It just adds another optional parameter to the commandline (a 3rd one) in non-interactive mode which you tell it after which stage to stop (stage1, stage2, stage3).

FWIW, personally I do prefer commandline options over environment variables,
yet there may be some reason I fail to remember why we use env vars here.

However, I do prefer named over positional commandline arguments, especially
when they are about additional configuration to positional arguments.

> I'm not the most versed person in bash, so maybe there is a more conceptually beautiful way of doing this. My patch is just a proposal (that has been tested and running).
> 
> This patch is useful because it allows:
> 1) Easier debugging by stages.
> 2) Allows to split the bootstrap job in parts in CI environments with time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*, and the bootstrap takes in between 5h40-6h40 depending on random things like time to download things, and load on the machine... or packages just taking longer to build).

So yes, I would love to have the bootstrap fit into Azure's 6 hours limit, to
get rid of the VMs I provide using my 2 monthly EUR 45,- credit in Azure cloud.

> 
> And this patch should not change anything to anyone else (AFAIK).
> 
> Thanks for your time and feedback!
> 
> P.S.: I'm used to the GitHub/GitLab workflow of having a graphical interface to do a Pull Request... I don't know how to do this with https://gitweb.gentoo.org/repo/proj/prefix.git/
> If there is no way, aside of the goal of this email, I'd propose to have a mirror in GitHub.

Mirroring prefix-overlay to github (readonly, for PRs) may make sense indeed.

Thanks!
/haubi/

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 16744 bytes --]

I think pretty late, but this is committed in f6bbc6f470.

Thanks,
Fabian

On 03-02-2020 14:54:26 +1100, Sam Pfeiffer wrote:
> Hey Fabian,
> 
> Thanks for chipping in. I do think it's useful and user-friendly to be able to
> control the target of the interactive/noninteractive script.
> I proposed in my first email in the thread something similar with the
> $STOP_AFTER_STAGE variable, even though I was capturing it from the commandline,
> and as I understand
> you propose, it would probably be better to capture it from an environment
> variable (it would actually be more in-line with other existing variables).
> 
> Benda, I'm sorry for the noise I'm causing. I just want to improve the
> experience for the next person that tries to use this amazing project by
> taking advantage of already implemented stuff. And also just having this
> discussion is good to have more information archived to google about Gentoo
> Prefix and its bootstrap process. I think.
> 
> On Sun, 2 Feb 2020 at 23:08, Fabian Groffen <grobian@gentoo.org[1]> wrote:
> > What I gather from this discussion is that the bootstrap script is
> > complex, and grew more complexity over the years.
> >
> > Back in the early days, there was just documentation, one needed to
> > follow.  Since this frequently needed an update, and people kept using
> > older copies, at some point the bootstrap-prefix.sh script was
> > introduced.
> >
> > This script followed the documentation to the letter.  It was an exact
> > replica of the instructions in the bootstrap document.  This included
> > the notions of 3 stages and a world recompile.
> >
> > Today, the purpose of the stages have blurred, and seem more like
> > batches in which to do things.  Some of the initial intentions still
> > stand out, e.g. that stage1 is there to provide a working portage
> > environment with some minimal efforts.  Minimal got redefined as soon as
> > we had to install python and more and more tools seemed necesary (sed,
> > path, awk, etc.).
> >
> > Now, it seems you need to cut time and sort of bootstrap up-to a stage.
> > Would it be an option to simply add a(nother) var that will control the
> > interactive target in terms of how far it will run along.  Normal resume
> > strategies it employs already would simply allow a next run to run along
> > further.
> >
> > I think above can be done easily with a few lines of code.
> >
> > Let me know what you think.
> >
> > Thanks,
> > Fabian
> >
> > On 23-01-2020 13:31:05 +1100, Sam Pfeiffer wrote:
> > > Grobian, any input about this?
> > >
> > > On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <[1]sammypfeiffer@gmail.com[2]>
> > wrote:
> > >
> > > > Hello Benda,
> > >
> > > > To be more specific... To easily go thru it I made a gist with the latest
> > > > version of bootstrap-prefix.sh and I'll be pointing to specific lines (this
> > > > also helps me recap why I did this, as I remember using stage{1,2,3} didn't
> > > > work for me when I first came into this):
> > >
> > > > When you execute bootstrap-prefix.sh without arguments (other than the
> > > > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> > >
> > > > [2]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[3]
> > >
> > > > Calling bootstrap_interactive
> > >
> > > > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> > > > [noninteractive, stage{1,2,3}]) it calls this line:
> > > > [3]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[4]
> > >
> > > > bootstrap_${TODO#non} || exit 1
> > >
> > > > Which, on the noninteractive case, goes to the same place than no arguments
> > > > (bootstrap_interactive). But in the stage{1,2,3} calls directly the
> > > > bootstrap_stageX function.
> > >
> > > > The thing is, noninteractive does some checks, finds resources and sets up
> > > > environment variables like:
> > >
> > > >   * Check for bad flags:
> > > >   [4]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[5]
> > > >   * We find a gcc:
> > > >   [5]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[6]
> > > >   * We get a cpu count and adjust -j for Make:
> > > >   [6]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[7]
> > > >   * Check if we deal with multilib systems:
> > > >   [7]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[8]
> > > >   * Check if we are boostrapping from a Gentoo system:
> > > >   [8]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[9]
> > > >   * Set our EPREFIX:
> > > > [9]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[10]
> > >
> > > > Which is summarised at the end by setting the environment variables EPREFIX,
> > > > CHOST, PATH, MAKEOPTS.
> > >
> > > > And then finally calls bootstrap_stage1_log, which runs bootstrap_stage1
> > > > itself.
> > >
> > > > [10]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[11]
> > >
> > > > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> > >
> > > > Calls bootstrap_stage1():
> > > > [11]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[12]
> > >
> > > > And as you can see it won't do any of the extra work done in
> > > > bootstrap_interactive().
> > >
> > > > I'm not that versed in this script, so I may be missing something easy to
> > > > overcome this (or I don't know enough about what is going on), but there is
> > no
> > > > separate function that does everything in bootstrap_interactive without
> > > > calling bootstrap_stage1. And bootstrap_stage1 does not do all that setup
> > that
> > > > bootstrap_interactive does. So that's how I came to my proposal.
> > >
> > > > If there is some patch to enable that setup from bootstrap_interactive with
> > > > bootstrap_stage1, that would be a good alternative, I guess. I still like
> > the
> > > > fact that with my approach you are just saying "bootstrap everything until
> > > > this step" instead of needing to manually do bootstrap_setup (imaginary
> > > > function I just named) then bootstrap_stage1, then bootstrap_stage2, then
> > > > bootstrap_stage3, and then no argument for the last emerge -e system, to get
> > > > to a specific bootstrap stage, but that's just commodity.
> > >
> > > > Let me know if I'm wrong somewhere or you think of any better solution!
> > >
> > > > On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <[12]sammypfeiffer@gmail.com[13]>
> > > > wrote:
> > >
> > > >> Hi Benda,
> > >
> > > >> Thanks for your reply.
> > >
> > > >> On Thu, 16 Jan 2020 at 13:01, Benda Xu <[13]heroxbd@gentoo.org[14]> wrote:
> > >
> > > >>> Hi Sam,
> > >
> > > >>> Sam Pfeiffer <[14]sammypfeiffer@gmail.com[15]> writes:
> > >
> > > >>> > To ease automated building and debugging of the bootstrap of Gentoo
> > > >>> > Prefix I would like to propose a patch to add the option to stop the
> > > >>> > non-interactive bootstrap on specific stages of the bootstrap.
> > > >>> >
> > > >>> > I'm currently doing this in my CI projects [1][2] and I'd love to have
> > > >>> > this added upstream.
> > > >>> >
> > > >>> > I propose a patch like this:
> > > >>> >[15]https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > [16]
> > > >>> >
> > > >>> > Which is what I'm currently using. It just adds another optional
> > > >>> >parameter to the commandline (a 3rd one) in non-interactive mode which
> > > >>> >you tell it after which stage to stop (stage1, stage2, stage3).
> > >
> > > >>> > I'm not the most versed person in bash, so maybe there is a more
> > > >>> > conceptually beautiful way of doing this. My patch is just a proposal
> > > >>> > (that has been tested and running).
> > > >>> >
> > > >>> > This patch is useful because it allows:
> > > >>> > 1) Easier debugging by stages.
> > >
> > > >>> > 2) Allows to split the bootstrap job in parts in CI environments with
> > > >>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
> > > >>> > and the bootstrap takes in between 5h40-6h40 depending on random
> > > >>> > things like time to download things, and load on the machine... or
> > > >>> > packages just taking longer to build).
> > > >>> >
> > > >>> > And this patch should not change anything to anyone else (AFAIK).
> > >
> > > >>> Are you aware that in bootstrap-prefix.sh you can specify the stage you
> > > >>> are after?  e.g.
> > >
> > > >>>    [16]https://wiki.gentoo.org/wiki/Project:Prefix/
> > Manual_Bootstrap#Stage_1[17]
> > >
> > > >>> The text above that could be outdated, but PATH exports and stage1~3 are
> > > >>> correct.
> > >
> > > >> I'm aware, but the behaviour (relating to environment variables and
> > choosing
> > > >> CPU numbers) differs from the
> > >
> > > >> noninteractive argument. I'd like to keep the CI environment as close as a
> > > >> user executing ./bootstrap-prefix.sh EPREFIX as possible.
> > >
> > > >> That's why I'm proposing this addition. Another approach could be modifying
> > > >> the current argument to do the same than the noninteractive one.
> > >
> > > >> But that may break the workflow of someone, and I don't want that.
> > >
> > > >>  
> > >
> > > >>> > Thanks for your time and feedback!
> > > >>> >
> > > >>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
> > > >>> > interface to do a Pull Request... I don't know how to do this with
> > > >>> > [17]https://gitweb.gentoo.org/repo/proj/prefix.git/[18] If there is no
> > way,
> > > >>> > aside of the goal of this email, I'd propose to have a mirror in
> > > >>> > GitHub.
> > >
> > > >>> Yes, that's a good to have for me, but not in high priority.
> > >
> > > >>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
> > > >>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
> > > >>> > approximately.
> > >
> > > >>> Thank you so much for all your work!  That's extremely helpful to keep
> > > >>> the quality of Prefix codebase.
> > >
> > > >>> > [1] [18]https://github.com/awesomebytes/gentoo_prefix_ci[19]
> > > >>> > [2] [19]https://github.com/awesomebytes/gentoo_prefix_ci_32b[20]
> > >
> > > >>> Yours,
> > > >>> Benda
> > >
> > > >> --
> > >
> > > >> Sammy Pfeiffer
> > > >> PhD Candidate at The Magic Lab within UTS.
> > >
> > > > --
> > >
> > > > Sammy Pfeiffer
> > > > PhD Candidate at The Magic Lab within UTS.
> > >
> > >
> > >
> > >  References:
> > >    1. mailto:sammypfeiffer@gmail.com[21]
> > >    2. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[22]
> > >    3. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[23]
> > >    4. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[24]
> > >    5. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[25]
> > >    6. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[26]
> > >    7. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[27]
> > >    8. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[28]
> > >    9. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[29]
> > >   10. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[30]
> > >   11. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[31]
> > >   12. mailto:sammypfeiffer@gmail.com[32]
> > >   13. mailto:heroxbd@gentoo.org[33]
> > >   14. mailto:sammypfeiffer@gmail.com[34]
> > >   15. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > [35]
> > >   16. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1[36]
> > >   17. https://gitweb.gentoo.org/repo/proj/prefix.git/[37]
> > >   18. https://github.com/awesomebytes/gentoo_prefix_ci[38]
> > >   19. https://github.com/awesomebytes/gentoo_prefix_ci_32b[39]
> > >
> > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > --
> > Fabian Groffen
> > Gentoo on a different level
> 
> 
> --
> 
> Sammy Pfeiffer
> PhD Candidate at The Magic Lab within UTS.
> 
> 
> References
>    1. mailto:grobian@gentoo.org
>    2. mailto:sammypfeiffer@gmail.com
>    3. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
>    4. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
>    5. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
>    6. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
>    7. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
>    8. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
>    9. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
>   10. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
>   11. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
>   12. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
>   13. mailto:sammypfeiffer@gmail.com
>   14. mailto:heroxbd@gentoo.org
>   15. mailto:sammypfeiffer@gmail.com
>   16. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>   17. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>   18. https://gitweb.gentoo.org/repo/proj/prefix.git/
>   19. https://github.com/awesomebytes/gentoo_prefix_ci
>   20. https://github.com/awesomebytes/gentoo_prefix_ci_32b
>   21. mailto:sammypfeiffer@gmail.com
>   22. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
>   23. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
>   24. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
>   25. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
>   26. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
>   27. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
>   28. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
>   29. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
>   30. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
>   31. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
>   32. mailto:sammypfeiffer@gmail.com
>   33. mailto:heroxbd@gentoo.org
>   34. mailto:sammypfeiffer@gmail.com
>   35. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>   36. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>   37. https://gitweb.gentoo.org/repo/proj/prefix.git/
>   38. https://github.com/awesomebytes/gentoo_prefix_ci
>   39. https://github.com/awesomebytes/gentoo_prefix_ci_32b

-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

[-- Attachment #1: Type: text/plain, Size: 17466 bytes --]

Thanks a lot Fabian!

I've just adopted it and the first jobs are running with it :) This implies
I won't need to sync my bootstrap-prefix.sh with upstream by hand anymore,
that's great news!

On Sat, 13 Jun 2020 at 23:54, Fabian Groffen <grobian@gentoo.org> wrote:

> I think pretty late, but this is committed in f6bbc6f470.
>
> Thanks,
> Fabian
>
> On 03-02-2020 14:54:26 +1100, Sam Pfeiffer wrote:
> > Hey Fabian,
> >
> > Thanks for chipping in. I do think it's useful and user-friendly to be
> able to
> > control the target of the interactive/noninteractive script.
> > I proposed in my first email in the thread something similar with the
> > $STOP_AFTER_STAGE variable, even though I was capturing it from the
> commandline,
> > and as I understand
> > you propose, it would probably be better to capture it from an
> environment
> > variable (it would actually be more in-line with other existing
> variables).
> >
> > Benda, I'm sorry for the noise I'm causing. I just want to improve the
> > experience for the next person that tries to use this amazing project by
> > taking advantage of already implemented stuff. And also just having this
> > discussion is good to have more information archived to google about
> Gentoo
> > Prefix and its bootstrap process. I think.
> >
> > On Sun, 2 Feb 2020 at 23:08, Fabian Groffen <grobian@gentoo.org[1]>
> wrote:
> > > What I gather from this discussion is that the bootstrap script is
> > > complex, and grew more complexity over the years.
> > >
> > > Back in the early days, there was just documentation, one needed to
> > > follow.  Since this frequently needed an update, and people kept using
> > > older copies, at some point the bootstrap-prefix.sh script was
> > > introduced.
> > >
> > > This script followed the documentation to the letter.  It was an exact
> > > replica of the instructions in the bootstrap document.  This included
> > > the notions of 3 stages and a world recompile.
> > >
> > > Today, the purpose of the stages have blurred, and seem more like
> > > batches in which to do things.  Some of the initial intentions still
> > > stand out, e.g. that stage1 is there to provide a working portage
> > > environment with some minimal efforts.  Minimal got redefined as soon
> as
> > > we had to install python and more and more tools seemed necesary (sed,
> > > path, awk, etc.).
> > >
> > > Now, it seems you need to cut time and sort of bootstrap up-to a stage.
> > > Would it be an option to simply add a(nother) var that will control the
> > > interactive target in terms of how far it will run along.  Normal
> resume
> > > strategies it employs already would simply allow a next run to run
> along
> > > further.
> > >
> > > I think above can be done easily with a few lines of code.
> > >
> > > Let me know what you think.
> > >
> > > Thanks,
> > > Fabian
> > >
> > > On 23-01-2020 13:31:05 +1100, Sam Pfeiffer wrote:
> > > > Grobian, any input about this?
> > > >
> > > > On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <[1]sammypfeiffer@gmail.com
> [2]>
> > > wrote:
> > > >
> > > > > Hello Benda,
> > > >
> > > > > To be more specific... To easily go thru it I made a gist with the
> latest
> > > > > version of bootstrap-prefix.sh and I'll be pointing to specific
> lines (this
> > > > > also helps me recap why I did this, as I remember using
> stage{1,2,3} didn't
> > > > > work for me when I first came into this):
> > > >
> > > > > When you execute bootstrap-prefix.sh without arguments (other than
> the
> > > > > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> > > >
> > > > > [2]https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[3]
> > > >
> > > > > Calling bootstrap_interactive
> > > >
> > > > > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> > > > > [noninteractive, stage{1,2,3}]) it calls this line:
> > > > > [3]https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[4]
> > > >
> > > > > bootstrap_${TODO#non} || exit 1
> > > >
> > > > > Which, on the noninteractive case, goes to the same place than no
> arguments
> > > > > (bootstrap_interactive). But in the stage{1,2,3} calls directly the
> > > > > bootstrap_stageX function.
> > > >
> > > > > The thing is, noninteractive does some checks, finds resources and
> sets up
> > > > > environment variables like:
> > > >
> > > > >   * Check for bad flags:
> > > > >   [4]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[5]
> > > > >   * We find a gcc:
> > > > >   [5]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[6]
> > > > >   * We get a cpu count and adjust -j for Make:
> > > > >   [6]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[7]
> > > > >   * Check if we deal with multilib systems:
> > > > >   [7]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[8]
> > > > >   * Check if we are boostrapping from a Gentoo system:
> > > > >   [8]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[9]
> > > > >   * Set our EPREFIX:
> > > > > [9]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[10]
> > > >
> > > > > Which is summarised at the end by setting the environment
> variables EPREFIX,
> > > > > CHOST, PATH, MAKEOPTS.
> > > >
> > > > > And then finally calls bootstrap_stage1_log, which runs
> bootstrap_stage1
> > > > > itself.
> > > >
> > > > > [10]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[11]
> > > >
> > > > > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> > > >
> > > > > Calls bootstrap_stage1():
> > > > > [11]https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[12]
> > > >
> > > > > And as you can see it won't do any of the extra work done in
> > > > > bootstrap_interactive().
> > > >
> > > > > I'm not that versed in this script, so I may be missing something
> easy to
> > > > > overcome this (or I don't know enough about what is going on), but
> there is
> > > no
> > > > > separate function that does everything in bootstrap_interactive
> without
> > > > > calling bootstrap_stage1. And bootstrap_stage1 does not do all
> that setup
> > > that
> > > > > bootstrap_interactive does. So that's how I came to my proposal.
> > > >
> > > > > If there is some patch to enable that setup from
> bootstrap_interactive with
> > > > > bootstrap_stage1, that would be a good alternative, I guess. I
> still like
> > > the
> > > > > fact that with my approach you are just saying "bootstrap
> everything until
> > > > > this step" instead of needing to manually do bootstrap_setup
> (imaginary
> > > > > function I just named) then bootstrap_stage1, then
> bootstrap_stage2, then
> > > > > bootstrap_stage3, and then no argument for the last emerge -e
> system, to get
> > > > > to a specific bootstrap stage, but that's just commodity.
> > > >
> > > > > Let me know if I'm wrong somewhere or you think of any better
> solution!
> > > >
> > > > > On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <[12]
> sammypfeiffer@gmail.com[13]>
> > > > > wrote:
> > > >
> > > > >> Hi Benda,
> > > >
> > > > >> Thanks for your reply.
> > > >
> > > > >> On Thu, 16 Jan 2020 at 13:01, Benda Xu <[13]heroxbd@gentoo.org[14]>
> wrote:
> > > >
> > > > >>> Hi Sam,
> > > >
> > > > >>> Sam Pfeiffer <[14]sammypfeiffer@gmail.com[15]> writes:
> > > >
> > > > >>> > To ease automated building and debugging of the bootstrap of
> Gentoo
> > > > >>> > Prefix I would like to propose a patch to add the option to
> stop the
> > > > >>> > non-interactive bootstrap on specific stages of the bootstrap.
> > > > >>> >
> > > > >>> > I'm currently doing this in my CI projects [1][2] and I'd love
> to have
> > > > >>> > this added upstream.
> > > > >>> >
> > > > >>> > I propose a patch like this:
> > > > >>> >[15]
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > > [16]
> > > > >>> >
> > > > >>> > Which is what I'm currently using. It just adds another
> optional
> > > > >>> >parameter to the commandline (a 3rd one) in non-interactive
> mode which
> > > > >>> >you tell it after which stage to stop (stage1, stage2, stage3).
> > > >
> > > > >>> > I'm not the most versed person in bash, so maybe there is a
> more
> > > > >>> > conceptually beautiful way of doing this. My patch is just a
> proposal
> > > > >>> > (that has been tested and running).
> > > > >>> >
> > > > >>> > This patch is useful because it allows:
> > > > >>> > 1) Easier debugging by stages.
> > > >
> > > > >>> > 2) Allows to split the bootstrap job in parts in CI
> environments with
> > > > >>> > time limits on jobs to run (e.g. Azure Pipelines time limit is
> 6h*,
> > > > >>> > and the bootstrap takes in between 5h40-6h40 depending on
> random
> > > > >>> > things like time to download things, and load on the
> machine... or
> > > > >>> > packages just taking longer to build).
> > > > >>> >
> > > > >>> > And this patch should not change anything to anyone else
> (AFAIK).
> > > >
> > > > >>> Are you aware that in bootstrap-prefix.sh you can specify the
> stage you
> > > > >>> are after?  e.g.
> > > >
> > > > >>>    [16]https://wiki.gentoo.org/wiki/Project:Prefix/
> > > Manual_Bootstrap#Stage_1[17]
> > > >
> > > > >>> The text above that could be outdated, but PATH exports and
> stage1~3 are
> > > > >>> correct.
> > > >
> > > > >> I'm aware, but the behaviour (relating to environment variables
> and
> > > choosing
> > > > >> CPU numbers) differs from the
> > > >
> > > > >> noninteractive argument. I'd like to keep the CI environment as
> close as a
> > > > >> user executing ./bootstrap-prefix.sh EPREFIX as possible.
> > > >
> > > > >> That's why I'm proposing this addition. Another approach could be
> modifying
> > > > >> the current argument to do the same than the noninteractive one.
> > > >
> > > > >> But that may break the workflow of someone, and I don't want that.
> > > >
> > > > >>
> > > >
> > > > >>> > Thanks for your time and feedback!
> > > > >>> >
> > > > >>> > P.S.: I'm used to the GitHub/GitLab workflow of having a
> graphical
> > > > >>> > interface to do a Pull Request... I don't know how to do this
> with
> > > > >>> > [17]https://gitweb.gentoo.org/repo/proj/prefix.git/[18] If
> there is no
> > > way,
> > > > >>> > aside of the goal of this email, I'd propose to have a mirror
> in
> > > > >>> > GitHub.
> > > >
> > > > >>> Yes, that's a good to have for me, but not in high priority.
> > > >
> > > > >>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes
> 29min~, Stage
> > > > >>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure
> Pipelines,
> > > > >>> > approximately.
> > > >
> > > > >>> Thank you so much for all your work!  That's extremely helpful
> to keep
> > > > >>> the quality of Prefix codebase.
> > > >
> > > > >>> > [1] [18]https://github.com/awesomebytes/gentoo_prefix_ci[19]
> > > > >>> > [2] [19]
> https://github.com/awesomebytes/gentoo_prefix_ci_32b[20]
> > > >
> > > > >>> Yours,
> > > > >>> Benda
> > > >
> > > > >> --
> > > >
> > > > >> Sammy Pfeiffer
> > > > >> PhD Candidate at The Magic Lab within UTS.
> > > >
> > > > > --
> > > >
> > > > > Sammy Pfeiffer
> > > > > PhD Candidate at The Magic Lab within UTS.
> > > >
> > > >
> > > >
> > > >  References:
> > > >    1. mailto:sammypfeiffer@gmail.com[21]
> > > >    2. https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[22]
> > > >    3. https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[23]
> > > >    4. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[24]
> > > >    5. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[25]
> > > >    6. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[26]
> > > >    7. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[27]
> > > >    8. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[28]
> > > >    9. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[29]
> > > >   10. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[30]
> > > >   11. https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[31]
> > > >   12. mailto:sammypfeiffer@gmail.com[32]
> > > >   13. mailto:heroxbd@gentoo.org[33]
> > > >   14. mailto:sammypfeiffer@gmail.com[34]
> > > >   15.
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > > [35]
> > > >   16.
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1[36]
> > > >   17. https://gitweb.gentoo.org/repo/proj/prefix.git/[37]
> > > >   18. https://github.com/awesomebytes/gentoo_prefix_ci[38]
> > > >   19. https://github.com/awesomebytes/gentoo_prefix_ci_32b[39]
> > > >
> > > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > > --
> > > Fabian Groffen
> > > Gentoo on a different level
> >
> >
> > --
> >
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
> >
> >
> > References
> >    1. mailto:grobian@gentoo.org
> >    2. mailto:sammypfeiffer@gmail.com
> >    3. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> >    4. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> >    5. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> >    6. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> >    7. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> >    8. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> >    9. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> >   10. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> >   11. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> >   12. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> >   13. mailto:sammypfeiffer@gmail.com
> >   14. mailto:heroxbd@gentoo.org
> >   15. mailto:sammypfeiffer@gmail.com
> >   16.
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> >   17.
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
> >   18. https://gitweb.gentoo.org/repo/proj/prefix.git/
> >   19. https://github.com/awesomebytes/gentoo_prefix_ci
> >   20. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >   21. mailto:sammypfeiffer@gmail.com
> >   22. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> >   23. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> >   24. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> >   25. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> >   26. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> >   27. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> >   28. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> >   29. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> >   30. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> >   31. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> >   32. mailto:sammypfeiffer@gmail.com
> >   33. mailto:heroxbd@gentoo.org
> >   34. mailto:sammypfeiffer@gmail.com
> >   35.
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> >   36.
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
> >   37. https://gitweb.gentoo.org/repo/proj/prefix.git/
> >   38. https://github.com/awesomebytes/gentoo_prefix_ci
> >   39. https://github.com/awesomebytes/gentoo_prefix_ci_32b
>
> --
> Fabian Groffen
> Gentoo on a different level
>


-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.

[-- Attachment #2: Type: text/html, Size: 29085 bytes --]