[gentoo-catalyst] [PATCH 1/2] targets: Set LANG=C.UTF8

[gentoo-catalyst] [PATCH 1/2] targets: Set LANG=C.UTF8

[Matt Turner]

Stable glibc now always provides a UTF-8 capable locale, which many
packages require. Set this as the default LANG.

Running locale-gen in stage1 should also solve bug #536938.

Bug: https://bugs.gentoo.org/536938
Bug: https://bugs.gentoo.org/710762
Bug: https://bugs.gentoo.org/714906
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
 targets/stage1/chroot.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/targets/stage1/chroot.sh b/targets/stage1/chroot.sh
index 0caf49ee..88a36481 100755
--- a/targets/stage1/chroot.sh
+++ b/targets/stage1/chroot.sh
@@ -57,6 +57,11 @@ make_destpath /tmp/stage1root
 run_merge "--oneshot --nodeps sys-apps/baselayout"
 ${clst_sed} -i "/USE=\"${USE} -build\"/d" ${clst_make_conf}
 
+for etc in /etc /tmp/stage1root/etc; do
+	echo "LANG=C.UTF8" > ${etc}/env.d/02locale
+done
+update_env_settings
+
 # Now, we install our packages
 if [ -e ${clst_make_conf} ]; then
 	echo "CATALYST_USE=\"-* build ${BINDIST} ${clst_CATALYST_USE}\"" >> ${clst_make_conf}
@@ -71,6 +76,13 @@ fi
 
 run_merge "--oneshot ${clst_buildpkgs}"
 
+# TODO: Drop this when locale-gen in stable glibc supports ROOT.
+#
+# locale-gen does not support the ROOT variable, and as such glibc simply does
+# not run locale-gen when ROOT is set. Since we've set LANG, we need to run
+# locale-gen explicitly.
+locale-gen --destdir /tmp/stage1root/
+
 # Why are we removing these? Don't we need them for final make.conf?
 for useexpand in ${clst_HOSTUSEEXPAND}; do
 	x="clst_${useexpand}"
-- 
2.24.1

[gentoo-catalyst] [PATCH 2/2] targets: Reduce locales to C.UTF8 in stage builds

[Matt Turner]

By default, glibc generates around 500 locales with more added each
year.

With USE=-compile-locales, glibc generates the locale archive in
pkg_postinst(). Since files generated in pkg_postinst() are not recorded
in the vdb, this has the advantage of allowing users to freely change
the set of enabled locales (by editing /etc/locale.gen and running
locale-gen).

Since it is so easy for the user to generate any locales they want with
locale-gen (and they probably would have anyway to rid themselves of the
499 locales they don't want!), just disable all locales except for
C.UTF8 and save stage builders a lot of time.

The patch works by
	(1) Writing /etc/locale.gen with "C.UTF8 UTF-8"
	(2) Setting CONFIG_PROTECT so glibc doesn't overwrite
	    /etc/locale.gen
	(3) Running etc-update to reset /etc/locale.gen

In order to do this I modified scripts/bootstrap.sh in commit
0aa49828ae25 (scripts/bootstrap.sh: Allow CONFIG_PROTECT).

Reducing the set of locales cuts the user time (as reported by time(1))
of the stage2 and stage3 builds as well as the file size of the
resulting xz'd tarballs:

            stage 2           stage 3
         size     time     size      time
before    89M   22m42s     206M     45m5s
 after    77M    4m29s     195M     26m8s

An alternative solution would be to set USE=compiled-locales for glibc,
but that has the downside of being non-default and likely causing users
to unnecessarily rebuild glibc. (We'll do this for the ISOs where we
want all the locales)

Note that this patch does not change the contents of /etc/locale.gen in
the stage3 tarball.

Closes: https://bugs.gentoo.org/686862
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
In order to avoid shipping ISOs with a reduced set of locales, I'm going
to delay committing this patch until glibc-2.30 is stablized, which has
USE=compile-locales. I'll send a corresponding patch for releng.git that
enables USE=compile-locales for the ISOs.

 targets/stage1/chroot.sh            | 1 +
 targets/stage2/chroot.sh            | 6 ++++++
 targets/stage3/chroot.sh            | 7 +++++++
 targets/support/chroot-functions.sh | 6 +++++-
 4 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/targets/stage1/chroot.sh b/targets/stage1/chroot.sh
index 88a36481..639feaa4 100755
--- a/targets/stage1/chroot.sh
+++ b/targets/stage1/chroot.sh
@@ -57,6 +57,7 @@ make_destpath /tmp/stage1root
 run_merge "--oneshot --nodeps sys-apps/baselayout"
 ${clst_sed} -i "/USE=\"${USE} -build\"/d" ${clst_make_conf}
 
+echo "$locales" > /etc/locale.gen
 for etc in /etc /tmp/stage1root/etc; do
 	echo "LANG=C.UTF8" > ${etc}/env.d/02locale
 done
diff --git a/targets/stage2/chroot.sh b/targets/stage2/chroot.sh
index 5fac858f..bf98d328 100755
--- a/targets/stage2/chroot.sh
+++ b/targets/stage2/chroot.sh
@@ -4,6 +4,12 @@ source /tmp/chroot-functions.sh
 
 # Setup the environment
 export FEATURES="${clst_myfeatures} nodoc noman noinfo -news"
+export CONFIG_PROTECT="-* /etc/locale.gen"
+
+echo "$locales" > /etc/locale.gen
 
 ## START BUILD
 ${clst_repo_basedir}/${clst_repo_name}/scripts/bootstrap.sh ${bootstrap_opts} || exit 1
+
+# Replace modified /etc/locale.gen with default
+etc-update --automode -5
diff --git a/targets/stage3/chroot.sh b/targets/stage3/chroot.sh
index 4f8bb0ee..e6712015 100755
--- a/targets/stage3/chroot.sh
+++ b/targets/stage3/chroot.sh
@@ -5,4 +5,11 @@ source /tmp/chroot-functions.sh
 ## START BUILD
 setup_pkgmgr
 
+export CONFIG_PROTECT="-* /etc/locale.gen"
+
+echo "$locales" > /etc/locale.gen
+
 run_merge "-e --update --deep --with-bdeps=y @system"
+
+# Replace modified /etc/locale.gen with default
+etc-update --automode -5
diff --git a/targets/support/chroot-functions.sh b/targets/support/chroot-functions.sh
index 2207e7b4..942fc116 100755
--- a/targets/support/chroot-functions.sh
+++ b/targets/support/chroot-functions.sh
@@ -286,7 +286,7 @@ run_merge() {
 	export CLEAN_DELAY=0
 	export EBEEP_IGNORE=0
 	export EPAUSE_IGNORE=0
-	export CONFIG_PROTECT="-*"
+	[[ $CONFIG_PROTECT != "-*"* ]] && export CONFIG_PROTECT="-*"
 
 	if [[ "${clst_VERBOSE}" == "true" ]]
 	then
@@ -433,5 +433,9 @@ Comment=This is a link to the local copy of the Gentoo Linux Handbook.
 Icon=text-editor" > /usr/share/applications/gentoo-handbook.desktop
 }
 
+readonly locales="
+C.UTF8 UTF-8
+"
+
 # We do this everywhere, so why not put it in this script
 run_default_funcs
-- 
2.24.1

Re: [gentoo-catalyst] [PATCH 1/2] targets: Set LANG=C.UTF8

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1835 bytes --]

On Sun, Mar 29, 2020 at 8:26 PM Matt Turner <mattst88@gentoo.org> wrote:

> Stable glibc now always provides a UTF-8 capable locale, which many
> packages require. Set this as the default LANG.
>
> Running locale-gen in stage1 should also solve bug #536938.
>
> Bug: https://bugs.gentoo.org/536938
> Bug: https://bugs.gentoo.org/710762
> Bug: https://bugs.gentoo.org/714906
> Signed-off-by: Matt Turner <mattst88@gentoo.org>
> ---
>  targets/stage1/chroot.sh | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
>
> diff --git a/targets/stage1/chroot.sh b/targets/stage1/chroot.sh
> index 0caf49ee..88a36481 100755
> --- a/targets/stage1/chroot.sh
> +++ b/targets/stage1/chroot.sh
> @@ -57,6 +57,11 @@ make_destpath /tmp/stage1root
>  run_merge "--oneshot --nodeps sys-apps/baselayout"
>  ${clst_sed} -i "/USE=\"${USE} -build\"/d" ${clst_make_conf}
>
> +for etc in /etc /tmp/stage1root/etc; do
> +       echo "LANG=C.UTF8" > ${etc}/env.d/02locale
> +done
> +update_env_settings
> +
>

if /tmp/stage1root is "${clst_root_path}" I think it might behoove you to
use it here rather than hard coding it.
Same as below.

 # Now, we install our packages
>  if [ -e ${clst_make_conf} ]; then
>         echo "CATALYST_USE=\"-* build ${BINDIST} ${clst_CATALYST_USE}\""
> >> ${clst_make_conf}
> @@ -71,6 +76,13 @@ fi
>
>  run_merge "--oneshot ${clst_buildpkgs}"
>
> +# TODO: Drop this when locale-gen in stable glibc supports ROOT.
> +#
> +# locale-gen does not support the ROOT variable, and as such glibc simply
> does
> +# not run locale-gen when ROOT is set. Since we've set LANG, we need to
> run
> +# locale-gen explicitly.
> +locale-gen --destdir /tmp/stage1root/
> +
>  # Why are we removing these? Don't we need them for final make.conf?
>  for useexpand in ${clst_HOSTUSEEXPAND}; do
>         x="clst_${useexpand}"
> --
> 2.24.1
>
>
>

[-- Attachment #2: Type: text/html, Size: 2919 bytes --]

Re: [gentoo-catalyst] [PATCH 1/2] targets: Set LANG=C.UTF8

[Matt Turner]

On Tue, Apr 7, 2020 at 2:26 PM Alec Warner <antarus@gentoo.org> wrote:
> On Sun, Mar 29, 2020 at 8:26 PM Matt Turner <mattst88@gentoo.org> wrote:
>>
>> Stable glibc now always provides a UTF-8 capable locale, which many
>> packages require. Set this as the default LANG.
>>
>> Running locale-gen in stage1 should also solve bug #536938.
>>
>> Bug: https://bugs.gentoo.org/536938
>> Bug: https://bugs.gentoo.org/710762
>> Bug: https://bugs.gentoo.org/714906
>> Signed-off-by: Matt Turner <mattst88@gentoo.org>
>> ---
>>  targets/stage1/chroot.sh | 12 ++++++++++++
>>  1 file changed, 12 insertions(+)
>>
>> diff --git a/targets/stage1/chroot.sh b/targets/stage1/chroot.sh
>> index 0caf49ee..88a36481 100755
>> --- a/targets/stage1/chroot.sh
>> +++ b/targets/stage1/chroot.sh
>> @@ -57,6 +57,11 @@ make_destpath /tmp/stage1root
>>  run_merge "--oneshot --nodeps sys-apps/baselayout"
>>  ${clst_sed} -i "/USE=\"${USE} -build\"/d" ${clst_make_conf}
>>
>> +for etc in /etc /tmp/stage1root/etc; do
>> +       echo "LANG=C.UTF8" > ${etc}/env.d/02locale
>> +done
>> +update_env_settings
>> +
>
>
> if /tmp/stage1root is "${clst_root_path}" I think it might behoove you to use it here rather than hard coding it.
> Same as below.

Thank you. Fixed and pushed (since I realized that USE=compile-locales
is actually already unmasked on the latest stable glibc).

Re: [gentoo-catalyst] [PATCH 1/2] targets: Set LANG=C.UTF8

[Brian Dolbec]

On Tue, 7 Apr 2020 17:25:44 -0700
Matt Turner <mattst88@gentoo.org> wrote:

> On Tue, Apr 7, 2020 at 2:26 PM Alec Warner <antarus@gentoo.org> wrote:
> > On Sun, Mar 29, 2020 at 8:26 PM Matt Turner <mattst88@gentoo.org>
> > wrote:  
> >>
> >> Stable glibc now always provides a UTF-8 capable locale, which many
> >> packages require. Set this as the default LANG.
> >>
> >> Running locale-gen in stage1 should also solve bug #536938.
> >>
> >> Bug: https://bugs.gentoo.org/536938
> >> Bug: https://bugs.gentoo.org/710762
> >> Bug: https://bugs.gentoo.org/714906
> >> Signed-off-by: Matt Turner <mattst88@gentoo.org>
> >> ---
> >>  targets/stage1/chroot.sh | 12 ++++++++++++
> >>  1 file changed, 12 insertions(+)
> >>
> >> diff --git a/targets/stage1/chroot.sh b/targets/stage1/chroot.sh
> >> index 0caf49ee..88a36481 100755
> >> --- a/targets/stage1/chroot.sh
> >> +++ b/targets/stage1/chroot.sh
> >> @@ -57,6 +57,11 @@ make_destpath /tmp/stage1root
> >>  run_merge "--oneshot --nodeps sys-apps/baselayout"
> >>  ${clst_sed} -i "/USE=\"${USE} -build\"/d" ${clst_make_conf}
> >>
> >> +for etc in /etc /tmp/stage1root/etc; do
> >> +       echo "LANG=C.UTF8" > ${etc}/env.d/02locale
> >> +done
> >> +update_env_settings
> >> +  
> >
> >
> > if /tmp/stage1root is "${clst_root_path}" I think it might behoove
> > you to use it here rather than hard coding it. Same as below.  
> 
> Thank you. Fixed and pushed (since I realized that USE=compile-locales
> is actually already unmasked on the latest stable glibc).
> 

Thanks Matt

And yes, after all the work it took to fix the hard coded paths
throughout the code...   If need be add another variable to the
defaults so that it is all in one place where it is easier to change
or override as the need may arise.