* [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions
@ 2020-05-20 3:42 Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 02/21] targets: Remove check_genkernel_version() function Matt Turner
` (19 more replies)
0 siblings, 20 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Unused since the removal of the original netboot target.
I'm honestly kind of impressed with the strategy of running ldd and
recursively copying libraries to the netboot image. Unfortunately for
this strategy, dlopen() exists so this cannot work.
Fixes: 89f57e145f82 (targets: Delete the netboot target)
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
targets/support/chroot-functions.sh | 75 -----------------------------
1 file changed, 75 deletions(-)
diff --git a/targets/support/chroot-functions.sh b/targets/support/chroot-functions.sh
index d40279ae..d63e4918 100755
--- a/targets/support/chroot-functions.sh
+++ b/targets/support/chroot-functions.sh
@@ -335,81 +335,6 @@ run_default_funcs() {
fi
}
-# Functions
-# Copy libs of a executable in the chroot
-function copy_libs() {
- # Check if it's a dynamix exec
- ldd ${1} > /dev/null 2>&1 || return
-
- for lib in `ldd ${1} | awk '{ print $3 }'`
- do
- echo ${lib}
- if [ -e ${lib} ]
- then
- if [ ! -e ${clst_root_path}/${lib} ]
- then
- copy_file ${lib}
- [ -e "${clst_root_path}/${lib}" ] && \
- strip -R .comment -R .note ${clst_root_path}/${lib} \
- || echo "WARNING : Cannot strip lib ${clst_root_path}/${lib} !"
- fi
- else
- echo "WARNING : Some library was not found for ${lib} !"
- fi
- done
-}
-
-function copy_symlink() {
- STACK=${2}
- [ "${STACK}" = "" ] && STACK=16 || STACK=$((${STACK} - 1 ))
-
- if [ ${STACK} -le 0 ]
- then
- echo "WARNING : ${TARGET} : too many levels of symbolic links !"
- return
- fi
-
- [ ! -e ${clst_root_path}/`dirname ${1}` ] && \
- mkdir -p ${clst_root_path}/`dirname ${1}`
- [ ! -e ${clst_root_path}/${1} ] && \
- cp -vfdp ${1} ${clst_root_path}/${1}
-
- if [[ -n $(type -p realpath) ]]; then
- TARGET=`realpath ${1}`
- else
- TARGET=`readlink -f ${1}`
- fi
- if [ -h ${TARGET} ]
- then
- copy_symlink ${TARGET} ${STACK}
- else
- copy_file ${TARGET}
- fi
-}
-
-function copy_file() {
- f="${1}"
-
- if [ ! -e "${f}" ]
- then
- echo "WARNING : File not found : ${f}"
- continue
- fi
-
- [ ! -e ${clst_root_path}/`dirname ${f}` ] && \
- mkdir -p ${clst_root_path}/`dirname ${f}`
- [ ! -e ${clst_root_path}/${f} ] && \
- cp -vfdp ${f} ${clst_root_path}/${f}
- if [ -x ${f} -a ! -h ${f} ]
- then
- copy_libs ${f}
- strip -R .comment -R .note ${clst_root_path}/${f} > /dev/null 2>&1
- elif [ -h ${f} ]
- then
- copy_symlink ${f}
- fi
-}
-
create_handbook_icon() {
# This function creates a local icon to the Gentoo Handbook
echo "[Desktop Entry]
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 02/21] targets: Remove check_genkernel_version() function
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 03/21] catalyst: Drop --cli option Matt Turner
` (18 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Existed to ensure a minimum genkernel version, and was last updated in
2005. genkernel is emerged during the catalyst build now, so we will
always have an updated version.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
targets/support/chroot-functions.sh | 23 -----------------------
targets/support/kmerge.sh | 2 --
2 files changed, 25 deletions(-)
diff --git a/targets/support/chroot-functions.sh b/targets/support/chroot-functions.sh
index d63e4918..b6e221af 100755
--- a/targets/support/chroot-functions.sh
+++ b/targets/support/chroot-functions.sh
@@ -29,29 +29,6 @@ if [[ -z "${clst_CHOST}" ]] ; then
fi
fi
-check_genkernel_version() {
- local version parts=() major minor
-
- version=$(genkernel --version)
- if [[ -z ${version} ]] ; then
- echo "ERROR: Could not detect genkernel version!"
- exit 1
- fi
- printf 'Genkernel version '%s' found ... ' "${version}"
-
- IFS='.' read -a parts <<<"${version}"
- major=${parts[0]}
- minor=${parts[1]}
- if [[ ${major} -gt 3 || ( ${major} -eq 3 && ${minor} -ge 3 ) ]] ; then
- echo "OK"
- else
- echo "FAIL"
- echo "ERROR: Your genkernel version is too low in your seed stage."
- echo " genkernel version 3.3.0 or greater is required."
- exit 1
- fi
-}
-
get_libdir() {
ABI=$(portageq envvar ABI)
DEFAULT_ABI=$(portageq envvar DEFAULT_ABI)
diff --git a/targets/support/kmerge.sh b/targets/support/kmerge.sh
index 1a432293..6b589493 100755
--- a/targets/support/kmerge.sh
+++ b/targets/support/kmerge.sh
@@ -2,8 +2,6 @@
source /tmp/chroot-functions.sh
-check_genkernel_version
-
install -d /tmp/kerncache
PKGDIR=/tmp/kerncache/${clst_kname}/ebuilds
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 03/21] catalyst: Drop --cli option
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 02/21] targets: Remove check_genkernel_version() function Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 04/21] catalyst: Remove PythonDir setting Matt Turner
` (17 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
This confusingly named option allowed you to specify and entire spec
file on the command line. It seems that the addition of the --snapshot /
-s option in commit ac746eff5363 (new -s option for creating snapshots)
in 2004 removed all known uses, so let's remove it.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/main.py | 5 +----
doc/catalyst.1.txt | 10 ----------
2 files changed, 1 insertion(+), 14 deletions(-)
diff --git a/catalyst/main.py b/catalyst/main.py
index 4ca1aa5b..bad712fa 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -202,9 +202,6 @@ def get_parser():
help='read specfile')
group.add_argument('-s', '--snapshot', type=str,
help='Make an ebuild repo snapshot')
- group.add_argument('-C', '--cli',
- default=[], nargs=argparse.REMAINDER,
- help='catalyst commandline (MUST BE LAST OPTION)')
return parser
@@ -294,8 +291,8 @@ def _main(parser, opts):
if not myconfigs:
myconfigs = [DEFAULT_CONFIG_FILE]
myspecfile = opts.file
- mycmdline = opts.cli[:]
+ mycmdline = list()
if opts.snapshot:
mycmdline.append('target=snapshot')
mycmdline.append('snapshot_treeish=' + opts.snapshot)
diff --git a/doc/catalyst.1.txt b/doc/catalyst.1.txt
index 46e21e63..90d5a24b 100644
--- a/doc/catalyst.1.txt
+++ b/doc/catalyst.1.txt
@@ -31,11 +31,6 @@ OPTIONS
This option is to be used to clear any autoresume points that have been saved
for this target. It is used in conjunction with *-f*, *-C*, or both.
-*--cli*|*-C* 'KEY'='VALUE' ...::
-This option is to be used in place of a specfile. All options are passed
-to *catalyst* on the commandline. Please note that this option must
-be the last option passed to *catalyst* for everything to work correctly.
-
*--config*|*-c* 'FILE'::
Tell *catalyst* to use a user-defined configuration file. A sample
configuration file is installed at '/etc/catalyst/catalyst.conf'.
@@ -75,11 +70,6 @@ Print the version information and exit
EXAMPLES
--------
-Using the commandline option (*-C*, *--cli*) to build a Portage snapshot:
----------------------------------------------------
-# catalyst -C target=snapshot version_stamp=my_date
----------------------------------------------------
-
Using the specfile option (*-f*, *--file*) to build a stage target:
---------------------------------------------------
# catalyst -f stage1-specfile.spec
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 04/21] catalyst: Remove PythonDir setting
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 02/21] targets: Remove check_genkernel_version() function Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 03/21] catalyst: Drop --cli option Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 05/21] catalyst: Switch internal snapshot option parsing to SpecParser Matt Turner
` (16 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Was used to find the arch directory containing, e.g., sparc.py, but all
that code is gone now after the modules were converted to TOML.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/defaults.py | 1 -
catalyst/main.py | 3 ---
2 files changed, 4 deletions(-)
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 404f4892..f6bc1e14 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -64,7 +64,6 @@ confdefaults = {
"options": set(),
"pkgdir": "/var/cache/binpkgs",
"port_tmpdir": "/var/tmp/portage",
- "PythonDir": "./catalyst",
"repo_basedir": "/var/db/repos",
"repo_name": "gentoo",
"repos": "%(storedir)s/repos",
diff --git a/catalyst/main.py b/catalyst/main.py
index bad712fa..b01d7a6a 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -55,9 +55,6 @@ def parse_config(config_files):
else:
conf_values[x] = confdefaults[x]
- # add our python base directory to use for loading target arch's
- conf_values["PythonDir"] = os.path.dirname(os.path.realpath(__file__))
-
# print out any options messages
for opt in conf_values['options']:
if opt in option_messages:
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 05/21] catalyst: Switch internal snapshot option parsing to SpecParser
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (2 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 04/21] catalyst: Remove PythonDir setting Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 06/21] catalyst: Remove unused decompression_mode spec option Matt Turner
` (15 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
The --snapshot/-s option internally creates a .spec file but uses the
ConfigParser (nominally used for parsing catalyst.conf) rather than
SpecParser (used for parsing .spec files) and as a result has to use
'=' rather than ':' as the key/value delimiter.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/main.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/catalyst/main.py b/catalyst/main.py
index b01d7a6a..be06ccd7 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -291,8 +291,8 @@ def _main(parser, opts):
mycmdline = list()
if opts.snapshot:
- mycmdline.append('target=snapshot')
- mycmdline.append('snapshot_treeish=' + opts.snapshot)
+ mycmdline.append('target: snapshot')
+ mycmdline.append('snapshot_treeish: ' + opts.snapshot)
conf_values['DEBUG'] = opts.debug
conf_values['VERBOSE'] = opts.debug or opts.verbose
@@ -354,7 +354,7 @@ def _main(parser, opts):
if mycmdline:
try:
- cmdline = catalyst.config.ConfigParser()
+ cmdline = catalyst.config.SpecParser()
cmdline.parse_lines(mycmdline)
addlargs.update(cmdline.get_values())
except CatalystError:
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 06/21] catalyst: Remove unused decompression_mode spec option
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (3 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 05/21] catalyst: Switch internal snapshot option parsing to SpecParser Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 07/21] catalyst: Delete unused {required,valid}_build_targets Matt Turner
` (14 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Never used.
Fixes: 99e9ceabe053 (DeComp bug fixes)
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/base/stagebase.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 651bf4e4..0c311515 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -49,7 +49,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
"common_flags",
"compression_mode",
"cxxflags",
- "decompression_mode",
"distcc_hosts",
"fcflags",
"fflags",
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 07/21] catalyst: Delete unused {required,valid}_build_targets
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (4 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 06/21] catalyst: Remove unused decompression_mode spec option Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 08/21] catalyst: Disallow DEBUG and VERBOSE in spec files Matt Turner
` (13 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Fixes: 11423a21603e ([2 of 3] Update module loading for the new python structure)
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/defaults.py | 19 -------------------
1 file changed, 19 deletions(-)
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index f6bc1e14..0da717f1 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -8,25 +8,6 @@ from DeComp.definitions import COMPRESSOR_PROGRAM_OPTIONS, XATTRS_OPTIONS
from DeComp.definitions import DECOMPRESSOR_PROGRAM_OPTIONS, LIST_XATTRS_OPTIONS
-# these should never be touched
-required_build_targets = [
- "generic_stage_target",
- "targetbase",
-]
-
-# new build types should be added here
-valid_build_targets = [
- "embedded_target",
- "livecd_stage1_target",
- "livecd_stage2_target",
- "netboot_target",
- "snapshot_target",
- "stage1_target",
- "stage2_target",
- "stage3_target",
- "stage4_target",
-]
-
required_config_file_values = [
"distdir",
"portdir",
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 08/21] catalyst: Disallow DEBUG and VERBOSE in spec files
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (5 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 07/21] catalyst: Delete unused {required,valid}_build_targets Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 09/21] catalyst: Add decompressor_search_order as valid spec option Matt Turner
` (12 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Catalyst has --debug/-d and --verbose/-v options for this.
Since conf_values is assigned in a very confusing manner in main.py, I
suspect these values were added to the list due to a misunderstanding.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/defaults.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 0da717f1..ccabd88d 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -20,13 +20,11 @@ valid_config_file_values.extend([
"compression_mode",
"compressor_arch",
"compressor_options",
- "DEBUG",
"decompressor_search_order",
"digests",
"distcc",
"envscript",
"options",
- "VERBOSE",
])
confdefaults = {
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 09/21] catalyst: Add decompressor_search_order as valid spec option
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (6 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 08/21] catalyst: Disallow DEBUG and VERBOSE in spec files Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 10/21] catalyst: Remove spec file options from valid_config_file_values Matt Turner
` (11 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
The next commit is going to remove this from the list of valid config
file options, where it should not be, and that list currently is used to
populate the list of valid spec options.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/base/stagebase.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 0c311515..f39895fe 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -49,6 +49,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
"common_flags",
"compression_mode",
"cxxflags",
+ "decompressor_search_order",
"distcc_hosts",
"fcflags",
"fflags",
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 10/21] catalyst: Remove spec file options from valid_config_file_values
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (7 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 09/21] catalyst: Add decompressor_search_order as valid spec option Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 11/21] catalyst: Add a missing options to valid_config_file_values Matt Turner
` (10 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/defaults.py | 5 -----
1 file changed, 5 deletions(-)
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index ccabd88d..2f2c907f 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -17,12 +17,7 @@ required_config_file_values = [
valid_config_file_values = required_config_file_values[:]
valid_config_file_values.extend([
- "compression_mode",
- "compressor_arch",
- "compressor_options",
- "decompressor_search_order",
"digests",
- "distcc",
"envscript",
"options",
])
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 11/21] catalyst: Add a missing options to valid_config_file_values
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (8 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 10/21] catalyst: Remove spec file options from valid_config_file_values Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 12/21] catalyst: Disallow config file options in spec files Matt Turner
` (9 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
And remove required_config_file_values since we want to support running
catalyst without a config file.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/defaults.py | 20 +++++++++++---------
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 2f2c907f..412cb956 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -8,18 +8,20 @@ from DeComp.definitions import COMPRESSOR_PROGRAM_OPTIONS, XATTRS_OPTIONS
from DeComp.definitions import DECOMPRESSOR_PROGRAM_OPTIONS, LIST_XATTRS_OPTIONS
-required_config_file_values = [
- "distdir",
- "portdir",
- "sharedir",
- "storedir",
-]
-
-valid_config_file_values = required_config_file_values[:]
-valid_config_file_values.extend([
+valid_config_file_values = frozenset([
"digests",
+ "distdir",
"envscript",
"options",
+ "port_logdir",
+ "repo_basedir",
+ "repo_name",
+ "repos",
+ "sharedir",
+ "storedir",
+ "target_distdir",
+ "target_pkgdir",
+ "var_tmpfs_portage",
])
confdefaults = {
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 12/21] catalyst: Disallow config file options in spec files
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (9 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 11/21] catalyst: Add a missing options to valid_config_file_values Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 13/21] catalyst: Remove support for source_matching="loose" Matt Turner
` (8 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/support.py | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/catalyst/support.py b/catalyst/support.py
index 0925af47..c4a5c797 100644
--- a/catalyst/support.py
+++ b/catalyst/support.py
@@ -8,7 +8,6 @@ import time
from subprocess import Popen
from catalyst import log
-from catalyst.defaults import valid_config_file_values
BASH_BINARY = "/bin/bash"
@@ -211,7 +210,7 @@ def addl_arg_parse(myspec, addlargs, requiredspec, validspec):
"helper function to help targets parse additional arguments"
messages = []
for x in addlargs.keys():
- if x not in validspec and x not in valid_config_file_values and x not in requiredspec:
+ if x not in validspec and x not in requiredspec:
messages.append("Argument \""+x+"\" not recognized.")
else:
myspec[x] = addlargs[x]
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 13/21] catalyst: Remove support for source_matching="loose"
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (10 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 12/21] catalyst: Disallow config file options in spec files Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 22:47 ` Brian Dolbec
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 14/21] catalyst: Convert catalyst.conf to TOML Matt Turner
` (7 subsequent siblings)
19 siblings, 1 reply; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
This does not seem like a useful feature to me.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/base/stagebase.py | 3 ---
catalyst/defaults.py | 1 -
catalyst/support.py | 6 +++---
etc/catalyst.conf | 12 ------------
4 files changed, 3 insertions(+), 19 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index f39895fe..febaf969 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -130,8 +130,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
decomp_opt=self.settings["decomp_opt"])
self.accepted_extensions = self.decompressor.search_order_extensions(
self.settings["decompressor_search_order"])
- log.notice("Source file specification matching setting is: %s",
- self.settings["source_matching"])
log.notice("Accepted source file extensions search order: %s",
self.accepted_extensions)
# save resources, it is not always needed
@@ -409,7 +407,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
normpath(self.settings["storedir"] + "/builds/" +
self.settings["source_subpath"]),
self.accepted_extensions,
- self.settings["source_matching"] in ["strict"]
)
log.debug('Source path returned from file_check is: %s',
self.settings["source_path"])
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 412cb956..14f671fe 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -45,7 +45,6 @@ confdefaults = {
"repos": "%(storedir)s/repos",
"sharedir": "/usr/share/catalyst",
"shdir": "%(sharedir)s/targets",
- "source_matching": "strict",
"storedir": "/var/tmp/catalyst",
"target_distdir": "/var/cache/distfiles",
"target_pkgdir": "/var/cache/binpkgs",
diff --git a/catalyst/support.py b/catalyst/support.py
index c4a5c797..a6a6854a 100644
--- a/catalyst/support.py
+++ b/catalyst/support.py
@@ -59,7 +59,7 @@ def cmd(mycmd, env=None, debug=False, fail_func=None):
print_traceback=False)
-def file_check(filepath, extensions=None, strict=True):
+def file_check(filepath, extensions=None):
'''Check for the files existence and that only one exists
if others are found with various extensions
'''
@@ -73,8 +73,8 @@ def file_check(filepath, extensions=None, strict=True):
".CONTENTS") and not x.endswith(".CONTENTS.gz") and not x.endswith(".DIGESTS")]
if len(files) == 1:
return files[0]
- if len(files) > 1 and strict:
- msg = "Ambiguos Filename: %s\nPlease specify the correct extension as well" % filepath
+ if len(files) > 1:
+ msg = "Ambiguous Filename: %s\nPlease specify the correct extension as well" % filepath
raise CatalystError(msg, print_traceback=False)
target_file = None
for ext in extensions:
diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index f64fe971..d33be15f 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -50,18 +50,6 @@ envscript="/etc/catalyst/catalystrc"
# (These options can be used together)
options="autoresume bindist kerncache pkgcache seedcache"
-# source_matching specifies how catalyst will match non-specific file names
-# if the filename is not found as an exact match.
-# ie: a filename without the extension specified. "/path/to/foo"
-#
-# possible values are:
-# "strict" meaning if more than one file of that name is present with any
-# file extension, then it will raise an exception.
-# "loose" meaning it will search for an existing filename with an added
-# extension from an ordered list of extensions determined from the
-# decompressor_search_order specification in the spec file or (default)
-source_matching="strict"
-
# port_logdir is where all build logs will be kept. This dir will be automatically cleaned
# of all logs over 30 days old. If left undefined the logs will remain in the build directory
# as usual and get cleaned every time a stage build is restarted.
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 14/21] catalyst: Convert catalyst.conf to TOML
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (11 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 13/21] catalyst: Remove support for source_matching="loose" Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 15/21] catalyst: Verify config options against valid_config_file_values Matt Turner
` (6 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/base/stagebase.py | 4 +-
catalyst/main.py | 47 ++++++--------------
etc/catalyst.conf | 88 ++++++++++++++++++++++----------------
3 files changed, 67 insertions(+), 72 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index febaf969..9410f151 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -857,8 +857,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
if 'var_tmpfs_portage' not in self.settings:
continue
- mount += ['-t', 'tmpfs', '-o', 'size=' +
- self.settings['var_tmpfs_portage'] + 'G']
+ mount += ['-t', 'tmpfs', '-o',
+ f"size={self.settings['var_tmpfs_portage']}G"]
elif source == 'tmpfs':
mount += ['-t', 'tmpfs']
elif source == 'shm':
diff --git a/catalyst/main.py b/catalyst/main.py
index be06ccd7..159fe454 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -5,6 +5,8 @@ import os
import sys
import textwrap
+import toml
+
from snakeoil.process import namespaces
from DeComp.definitions import (COMPRESS_DEFINITIONS, DECOMPRESS_DEFINITIONS,
@@ -17,8 +19,7 @@ from catalyst.defaults import confdefaults, option_messages, DEFAULT_CONFIG_FILE
from catalyst.support import CatalystError
from catalyst.version import get_version
-
-conf_values = {}
+conf_values = confdefaults
def version():
@@ -30,42 +31,20 @@ def version():
def parse_config(config_files):
- # search a couple of different areas for the main config file
- myconf = {}
-
- # try and parse the config file "config_file"
for config_file in config_files:
log.notice('Loading configuration file: %s', config_file)
try:
- config = catalyst.config.ConfigParser(config_file)
- myconf.update(config.get_values())
+ conf_values.update(toml.load(config_file))
except Exception as e:
log.critical('Could not find parse configuration file: %s: %s',
config_file, e)
- # now, load up the values into conf_values so that we can use them
- for x in list(confdefaults):
- if x in myconf:
- if x == 'options':
- conf_values[x] = set(myconf[x].split())
- elif x in ["decompressor_search_order"]:
- conf_values[x] = myconf[x].split()
- else:
- conf_values[x] = myconf[x]
- else:
- conf_values[x] = confdefaults[x]
-
# print out any options messages
for opt in conf_values['options']:
if opt in option_messages:
log.info(option_messages[opt])
- for key in ["digests", "envscript", "var_tmpfs_portage", "port_logdir",
- "local_overlay", "repos"]:
- if key in myconf:
- conf_values[key] = myconf[key]
-
- if "envscript" in myconf:
+ if "envscript" in conf_values:
log.info('Envscript support enabled.')
# take care of any variable substitutions that may be left
@@ -297,17 +276,17 @@ def _main(parser, opts):
conf_values['DEBUG'] = opts.debug
conf_values['VERBOSE'] = opts.debug or opts.verbose
- options = set()
+ options = []
if opts.fetchonly:
- options.add('fetch')
+ options.append('fetch')
if opts.purge:
- options.add('purge')
+ options.append('purge')
if opts.purgeonly:
- options.add('purgeonly')
+ options.append('purgeonly')
if opts.purgetmponly:
- options.add('purgetmponly')
+ options.append('purgetmponly')
if opts.clear_autoresume:
- options.add('clear-autoresume')
+ options.append('clear-autoresume')
# Make sure we have some work before moving further.
if not myspecfile and not mycmdline:
@@ -318,7 +297,7 @@ def _main(parser, opts):
# import configuration file and import our main module using those settings
parse_config(myconfigs)
- conf_values["options"].update(options)
+ conf_values["options"].extend(options)
log.notice('conf_values[options] = %s', conf_values['options'])
# initialize our contents generator
@@ -335,7 +314,7 @@ def _main(parser, opts):
if "digests" in conf_values:
valid_digests = hashlib.algorithms_available
- digests = set(conf_values['digests'].split())
+ digests = set(conf_values['digests'])
conf_values['digests'] = digests
# First validate all the requested digests are valid keys.
diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index d33be15f..2272cb86 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -10,53 +10,69 @@
#
# $ python3 -c 'import hashlib; print(hashlib.algorithms_available)'
#
-digests="blake2b sha512"
+digests = ["blake2b", "sha512"]
# envscript allows users to set options such as http proxies, MAKEOPTS,
# GENTOO_MIRRORS, or any other environment variables needed for building.
# The envscript file sets environment variables like so:
# export FOO="bar"
-envscript="/etc/catalyst/catalystrc"
-
-# options set different build-time options for catalyst. Some examples are:
-# autoresume = Attempt to resume a failed build, clear the autoresume flags with
-# the -a option to the catalyst cmdline. -p will clear the autoresume flags
-# as well as your pkgcache and kerncache
-# ( This option is not fully tested, bug reports welcome )
-# bindist = enables the bindist USE flag, please see package specific definition,
-# however, it is suggested to enable this if redistributing builds.
-# This optional USE flag is normally cleaned from the make.conf file on
-# completion of the stage. For a non-cleaned version,
-# use sticky-config also (see below)
-# ccache = enables build time ccache support
-# distcc = enable distcc support for building. You have to set distcc_hosts in
-# your spec file.
-# icecream = enables icecream compiler cluster support for building
-# keepwork = Prevents the removal of the working chroot path and any autoresume
-# files or points.
-# kerncache = keeps a tbz2 of your built kernel and modules (useful if your
-# build stops in livecd-stage2)
-# pkgcache = keeps a tbz2 of every built package (useful if your build stops
-# prematurely)
-# seedcache = use the build output of a previous target if it exists to speed up
-# the copy
-# sticky-config = enables the code that will keep any internal 'catalyst_use' flags
-# added to the USE= for building the stage. These are usually added for legal
-# or specific needs in building the the early stage. Mostly it is the
-# 'bindist' USE flag option that is used for legal reasons, please see its
-# specific definition. It will also keep any /etc/portage/package.*
-# files or directories.
-#
-# (These options can be used together)
-options="autoresume bindist kerncache pkgcache seedcache"
+envscript = "/etc/catalyst/catalystrc"
+
+# options set different build-time options for catalyst.
+options = [
+ # Attempt to resume a failed build, clear the autoresume flags with the
+ # -a option to the catalyst cmdline. -p will clear the autoresume
+ # flags as well as your pkgcache and kerncache
+ "autoresume",
+
+ # Enables the bindist USE flag, please see package specific definition,
+ # however, it is suggested to enable this if redistributing builds.
+ # This optional USE flag is normally cleaned from the make.conf file on
+ # completion of the stage. For a non-cleaned version, use
+ # sticky-config also (see below)
+ "bindist",
+
+ # Enable FEATURES=ccache
+ # "ccache",
+
+ # Enable FEATURES=distcc. You have to set distcc_hosts in your spec
+ # file.
+ # "distcc",
+
+ # Enable FEATURES=icecream
+ # "icecream",
+
+ # Prevents the removal of the working chroot path and any autoresume
+ # files or points.
+ # "keepwork",
+
+ # keeps a tbz2 of your built kernel and modules (useful if your
+ # build stops in livecd-stage2)
+ "kerncache",
+
+ # Build and use binary packages
+ "pkgcache",
+
+ # Use the build output of a previous target if it exists rather than
+ # the tarball
+ "seedcache",
+
+ # enables the code that will keep any internal 'catalyst_use' flags
+ # added to the USE= for building the stage. These are usually added
+ # for legal or specific needs in building the the early stage. Mostly
+ # it is the 'bindist' USE flag option that is used for legal reasons,
+ # please see its specific definition. It will also keep any
+ # /etc/portage/package.* files or directories.
+ # "sticky-config",
+]
# port_logdir is where all build logs will be kept. This dir will be automatically cleaned
# of all logs over 30 days old. If left undefined the logs will remain in the build directory
# as usual and get cleaned every time a stage build is restarted.
-# port_logdir="/var/tmp/catalyst/tmp"
+# port_logdir = "/var/tmp/catalyst/tmp"
# var_tmpfs_portage will mount a tmpfs for /var/tmp/portage so building takes place in RAM
# this feature requires a pretty large tmpfs ({open,libre}office needs ~8GB to build)
# WARNING: If you use too much RAM everything will fail horribly and it is not our fault.
# set size of /var/tmp/portage tmpfs in gigabytes
-# var_tmpfs_portage=16
+# var_tmpfs_portage = 16
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 15/21] catalyst: Verify config options against valid_config_file_values
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (12 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 14/21] catalyst: Convert catalyst.conf to TOML Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 16/21] catalyst: Don't even try to make envars from dicts Matt Turner
` (5 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/main.py | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/catalyst/main.py b/catalyst/main.py
index 159fe454..543895c6 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -15,7 +15,8 @@ from DeComp.contents import ContentsMap
from catalyst import log
import catalyst.config
-from catalyst.defaults import confdefaults, option_messages, DEFAULT_CONFIG_FILE
+from catalyst.defaults import (confdefaults, option_messages,
+ DEFAULT_CONFIG_FILE, valid_config_file_values)
from catalyst.support import CatalystError
from catalyst.version import get_version
@@ -34,7 +35,12 @@ def parse_config(config_files):
for config_file in config_files:
log.notice('Loading configuration file: %s', config_file)
try:
- conf_values.update(toml.load(config_file))
+ config = toml.load(config_file)
+ for key in config:
+ if key not in valid_config_file_values:
+ log.critical("Unknown option '%s' in config file %s",
+ key, config_file)
+ conf_values.update(config)
except Exception as e:
log.critical('Could not find parse configuration file: %s: %s',
config_file, e)
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 16/21] catalyst: Don't even try to make envars from dicts
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (13 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 15/21] catalyst: Verify config options against valid_config_file_values Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 17/21] catalyst: Only emit true boolean envars Matt Turner
` (4 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
With the removal of the arch modules (presumably), the two exceptions
(compress_definitions and decompress_definitions) are the only dicts in
self.settings.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/base/stagebase.py | 22 ++--------------------
1 file changed, 2 insertions(+), 20 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 9410f151..8e2b08da 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1285,28 +1285,10 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.env[varname] = "true"
else:
self.env[varname] = "false"
- # This handles a dictionary of objects just one level deep and no deeper!
- # Its currently used only for USE_EXPAND flags which are dictionaries of
- # lists in arch/amd64.py and friends. If we wanted self.settigs[var]
- # of any depth, we should make this function recursive.
elif isinstance(self.settings[x], dict):
- if x in ["compress_definitions",
- "decompress_definitions"]:
+ if x in ['compress_definitions', 'decompress_definitions']:
continue
- self.env[varname] = ' '.join(self.settings[x].keys())
- for y in self.settings[x].keys():
- varname2 = "clst_" + y.replace("/", "_")
- varname2 = varname2.replace("-", "_")
- varname2 = varname2.replace(".", "_")
- if isinstance(self.settings[x][y], str):
- self.env[varname2] = self.settings[x][y]
- elif isinstance(self.settings[x][y], list):
- self.env[varname2] = ' '.join(self.settings[x][y])
- elif isinstance(self.settings[x][y], bool):
- if self.settings[x][y]:
- self.env[varname] = "true"
- else:
- self.env[varname] = "false"
+ log.warning("Not making envar for '%s', is a dict", x)
if "makeopts" in self.settings:
if isinstance(self.settings["makeopts"], str):
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 17/21] catalyst: Only emit true boolean envars
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (14 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 16/21] catalyst: Don't even try to make envars from dicts Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 18/21] catalyst: Support emitting int/float envars Matt Turner
` (3 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/base/stagebase.py | 2 --
targets/support/chroot-functions.sh | 6 +++---
targets/support/kmerge.sh | 4 ++--
3 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 8e2b08da..645a9f61 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1283,8 +1283,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
elif isinstance(self.settings[x], bool):
if self.settings[x]:
self.env[varname] = "true"
- else:
- self.env[varname] = "false"
elif isinstance(self.settings[x], dict):
if x in ['compress_definitions', 'decompress_definitions']:
continue
diff --git a/targets/support/chroot-functions.sh b/targets/support/chroot-functions.sh
index b6e221af..b531eb6a 100755
--- a/targets/support/chroot-functions.sh
+++ b/targets/support/chroot-functions.sh
@@ -117,7 +117,7 @@ setup_emerge_opts() {
emerge_opts=()
bootstrap_opts=()
- if [[ "${clst_VERBOSE}" == "true" ]]
+ if [ -n "${clst_VERBOSE}" ]
then
emerge_opts+=(--verbose)
bootstrap_opts+=(-v)
@@ -256,7 +256,7 @@ run_merge() {
export EPAUSE_IGNORE=0
[[ $CONFIG_PROTECT != "-*"* ]] && export CONFIG_PROTECT="-*"
- if [[ "${clst_VERBOSE}" == "true" ]]
+ if [ -n "${clst_VERBOSE}" ]
then
echo "ROOT=${ROOT} emerge ${emerge_opts[@]} -pt $@" || exit 1
emerge ${emerge_opts[@]} -pt $@ || exit 3
@@ -268,7 +268,7 @@ run_merge() {
}
show_debug() {
- if [ "${clst_DEBUG}" = "1" ]
+ if [ -n "${clst_DEBUG}" ]
then
unset PACKAGES
echo "DEBUG:"
diff --git a/targets/support/kmerge.sh b/targets/support/kmerge.sh
index 6b589493..702c5454 100755
--- a/targets/support/kmerge.sh
+++ b/targets/support/kmerge.sh
@@ -58,7 +58,7 @@ setup_gk_args() {
fi
fi
- if [[ "${clst_VERBOSE}" == "true" ]]
+ if [ -n "${clst_VERBOSE}" ]
then
GK_ARGS+=(--loglevel=2)
fi
@@ -85,7 +85,7 @@ genkernel_compile(){
esac
# Build with genkernel using the set options
# callback is put here to avoid escaping issues
- if [[ "${clst_VERBOSE}" == "true" ]]
+ if [ -n "${clst_VERBOSE}" ]
then
gk_callback_opts=(-vN)
else
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 18/21] catalyst: Support emitting int/float envars
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (15 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 17/21] catalyst: Only emit true boolean envars Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 19/21] catalyst: Set jobs/load-average via catalyst.conf Matt Turner
` (2 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/base/stagebase.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 645a9f61..5a8cd1df 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1283,6 +1283,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
elif isinstance(self.settings[x], bool):
if self.settings[x]:
self.env[varname] = "true"
+ elif isinstance(self.settings[x], (int, float)):
+ self.env[varname] = str(self.settings[x])
elif isinstance(self.settings[x], dict):
if x in ['compress_definitions', 'decompress_definitions']:
continue
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 19/21] catalyst: Set jobs/load-average via catalyst.conf
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (16 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 18/21] catalyst: Support emitting int/float envars Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
[not found] ` <94424974-cfe8-9766-8712-ae6fa8bbf825@veremit.xyz>
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 20/21] catalyst: Configure distcc_hosts in the config file Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 21/21] catalyst: Drop ConfigParser Matt Turner
19 siblings, 1 reply; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
We currently have two mechanisms of setting MAKEOPTS: in spec files and
in catalystrc.
Setting makeopts in spec files doesn't make sense. The spec should
describe the thing that's being built and not contain options that are
specific to the build system.
Setting makeopts via catalystrc is better, but it only applies to the
actual build system invocations, leaving emerge to run jobs serially or
again requiring configuration specific to the build machine to be put
into the spec file. For example:
update_seed_command: ... --jobs 5 --load-average 5
With jobs and load-average specified in catalyst.conf, catalyst has the
information required to configure both emerge and the build systems
emerge executes.
This removes the undocumented makeopts spec file option and replaces it
with jobs and load-average settings in catalyst.conf.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/base/stagebase.py | 12 +++++-------
catalyst/defaults.py | 2 ++
doc/catalyst-config.5.txt | 15 ++++++++++++---
etc/catalyst.conf | 8 ++++++++
etc/catalystrc | 3 ---
targets/support/chroot-functions.sh | 8 ++++++++
6 files changed, 35 insertions(+), 13 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 5a8cd1df..bc721ad4 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -56,7 +56,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
"hostuse",
"kerncache_path",
"ldflags",
- "makeopts",
"pkgcache_path",
"portage_confdir",
"portage_overlay",
@@ -1290,12 +1289,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
continue
log.warning("Not making envar for '%s', is a dict", x)
- if "makeopts" in self.settings:
- if isinstance(self.settings["makeopts"], str):
- self.env["MAKEOPTS"] = self.settings["makeopts"]
- else:
- # ensure makeopts is a string
- self.env["MAKEOPTS"] = ' '.join(self.settings["makeopts"])
+ makeopts = []
+ for flag, setting in {'j': 'jobs', 'l': 'load-average'}.items():
+ if setting in self.settings:
+ makeopts.append(f'-{flag}{self.settings[setting]}')
+ self.env['MAKEOPTS'] = ' '.join(makeopts)
log.debug('setup_environment(); env = %r', self.env)
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 14f671fe..b31d5b50 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -12,6 +12,8 @@ valid_config_file_values = frozenset([
"digests",
"distdir",
"envscript",
+ "jobs",
+ "load-average",
"options",
"port_logdir",
"repo_basedir",
diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
index 7ac9a2a3..cbef6092 100644
--- a/doc/catalyst-config.5.txt
+++ b/doc/catalyst-config.5.txt
@@ -48,9 +48,9 @@ $ python3 -c 'import hashlib; print(hashlib.algorithms_available)'
*envscript*::
Environment script location, which allows users to set options such as
-HTTP proxies, `MAKEOPTS`, `GENTOO_MIRRORS`, or any other environment
-variables needed for building. The envscript file sets environment
-variables using POSIX shell notation:
+HTTP proxies, `GENTOO_MIRRORS`, or any other environment variables
+needed for building. The envscript file sets environment variables
+using POSIX shell notation:
+
---------------------------------
export FOO="bar"
@@ -136,6 +136,15 @@ written to the target's make.conf if it is not the default value of
Other settings
~~~~~~~~~~~~~~
+*jobs*::
+Integral value passed to *emerge(1)* as the parameter to --jobs and is
+used to define *MAKEOPTS* during the target build.
+
+*load-average*::
+Floating-point value passed to *emerge(1)* as the parameter to
+--load-average and is used to define *MAKEOPTS* during the target
+build.
+
*sharedir*::
Catalyst runtime script location. `/usr/share/catalyst` should work for
most default installations. If you are running catalyst from a Git
diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index 2272cb86..81693c25 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -76,3 +76,11 @@ options = [
# WARNING: If you use too much RAM everything will fail horribly and it is not our fault.
# set size of /var/tmp/portage tmpfs in gigabytes
# var_tmpfs_portage = 16
+
+# Integral value passed to emerge as the parameter to --jobs and is used to
+# define MAKEOPTS during the target build.
+# jobs = 4
+
+# Floating-point value passed to emerge as the parameter to --load-average and
+# is used to define MAKEOPTS during the target build.
+# load-average = 4.0
diff --git a/etc/catalystrc b/etc/catalystrc
index bcd729af..e7904128 100755
--- a/etc/catalystrc
+++ b/etc/catalystrc
@@ -1,5 +1,2 @@
#!/bin/bash
# This is an example catalystrc. As such, it doesn't actually *do* anything.
-
-# Uncomment the following to increase the number of threads used to compile.
-# export MAKEOPTS="-j16"
diff --git a/targets/support/chroot-functions.sh b/targets/support/chroot-functions.sh
index b531eb6a..4005a1d8 100755
--- a/targets/support/chroot-functions.sh
+++ b/targets/support/chroot-functions.sh
@@ -130,6 +130,14 @@ setup_emerge_opts() {
emerge_opts+=(--fetchonly)
bootstrap_opts+=(-f)
fi
+ if [ -n "${clst_jobs}" ]
+ then
+ emerge_opts+=(--jobs "${clst_jobs}")
+ fi
+ if [ -n "${clst_load_average}" ]
+ then
+ emerge_opts+=(--load-average "${clst_load_average}")
+ fi
if [ -n "${clst_PKGCACHE}" ] && [ -z "${clst_update_seed}" -o "${clst_update_seed}" = "no" ]
then
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 20/21] catalyst: Configure distcc_hosts in the config file
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (17 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 19/21] catalyst: Set jobs/load-average via catalyst.conf Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 21/21] catalyst: Drop ConfigParser Matt Turner
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
distcc_hosts are independent of the build itself, and therefore should
be configured system-wide in catalyst.conf and not in each spec file.
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/base/stagebase.py | 1 -
catalyst/defaults.py | 1 +
doc/catalyst-config.5.txt | 7 ++++++-
doc/catalyst-spec.5.txt | 6 ------
etc/catalyst.conf | 3 +--
examples/generic_stage_template.spec | 7 -------
examples/livecd-stage1_template.spec | 7 -------
examples/livecd-stage2_template.spec | 7 -------
examples/stage4_template.spec | 7 -------
9 files changed, 8 insertions(+), 38 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index bc721ad4..00efd252 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -50,7 +50,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
"compression_mode",
"cxxflags",
"decompressor_search_order",
- "distcc_hosts",
"fcflags",
"fflags",
"hostuse",
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index b31d5b50..27c3d9fa 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -10,6 +10,7 @@ from DeComp.definitions import DECOMPRESSOR_PROGRAM_OPTIONS, LIST_XATTRS_OPTIONS
valid_config_file_values = frozenset([
"digests",
+ "distcc_hosts",
"distdir",
"envscript",
"jobs",
diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
index cbef6092..570d42c2 100644
--- a/doc/catalyst-config.5.txt
+++ b/doc/catalyst-config.5.txt
@@ -78,7 +78,7 @@ be closed invalid.
distcc::
Enable distcc support for building. You have to set distcc_hosts in
-your spec file.
+your config file.
icecream::
Enable icecream compiler cluster support for building.
@@ -136,6 +136,11 @@ written to the target's make.conf if it is not the default value of
Other settings
~~~~~~~~~~~~~~
+*distcc_hosts*::
+These are the hosts used as distcc slaves when distcc is enabled in
+your `catalyst.conf` (example: `127.0.0.1 192.168.0.1`). It follows
+the same syntax as `distcc-config --set-hosts`.
+
*jobs*::
Integral value passed to *emerge(1)* as the parameter to --jobs and is
used to define *MAKEOPTS* during the target build.
diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
index cf6b9cd7..682f4621 100644
--- a/doc/catalyst-spec.5.txt
+++ b/doc/catalyst-spec.5.txt
@@ -74,12 +74,6 @@ This specifies where the seed stage for this target comes from
`$storedir/builds`. The `rel_type` is also used as a path prefix for
the seed.
-*distcc_hosts*::
-These are the hosts used as distcc slaves when distcc is enabled in
-your `catalyst.conf` (example: `127.0.0.1 192.168.0.1`). It follows
-the same syntax as `distcc-config --set-hosts` and is entirely
-optional.
-
*portage_confdir*::
This is an optional directory containing portage configuration files
(example: `/etc/portage`). It follows the same syntax as
diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index 81693c25..b0b284fa 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -35,8 +35,7 @@ options = [
# Enable FEATURES=ccache
# "ccache",
- # Enable FEATURES=distcc. You have to set distcc_hosts in your spec
- # file.
+ # Enable FEATURES=distcc. Make sure to set distcc_hosts too.
# "distcc",
# Enable FEATURES=icecream
diff --git a/examples/generic_stage_template.spec b/examples/generic_stage_template.spec
index 01c37789..8f0375c4 100644
--- a/examples/generic_stage_template.spec
+++ b/examples/generic_stage_template.spec
@@ -82,13 +82,6 @@ compressor_arch":
#
decompressor_search_order: lbzip2 bzip2 tar pixz xz gzip squashfs
-# These are the hosts used as distcc slaves when distcc is enabled in your
-# catalyst.conf. It follows the same syntax as distcc-config --set-hosts and
-# is entirely optional.
-# example:
-# distcc_hosts: 127.0.0.1 192.168.0.1
-distcc_hosts:
-
# This is an optional directory containing portage configuration files. It
# follows the same syntax as /etc/portage and should be consistent across all
# targets to minimize problems.
diff --git a/examples/livecd-stage1_template.spec b/examples/livecd-stage1_template.spec
index c7086c91..b921372a 100644
--- a/examples/livecd-stage1_template.spec
+++ b/examples/livecd-stage1_template.spec
@@ -45,13 +45,6 @@ snapshot:
# default/stage3-x86-2006.1
source_subpath:
-# These are the hosts used as distcc slaves when distcc is enabled in your
-# catalyst.conf. It follows the same syntax as distcc-config --set-hosts and
-# is entirely optional.
-# example:
-# distcc_hosts: 127.0.0.1 192.168.0.1
-distcc_hosts:
-
# This is an optional directory containing portage configuration files. It
# follows the same syntax as /etc/portage and should be consistent across all
# targets to minimize problems.
diff --git a/examples/livecd-stage2_template.spec b/examples/livecd-stage2_template.spec
index a296cfa1..6cfd33d3 100644
--- a/examples/livecd-stage2_template.spec
+++ b/examples/livecd-stage2_template.spec
@@ -45,13 +45,6 @@ snapshot:
# default/livecd-stage1-x86-2006.1
source_subpath:
-# These are the hosts used as distcc slaves when distcc is enabled in your
-# catalyst.conf. It follows the same syntax as distcc-config --set-hosts and
-# is entirely optional.
-# example:
-# distcc_hosts: 127.0.0.1 192.168.0.1
-distcc_hosts:
-
# This is an optional directory containing portage configuration files. It
# follows the same syntax as /etc/portage and should be consistent across all
# targets to minimize problems.
diff --git a/examples/stage4_template.spec b/examples/stage4_template.spec
index 562bfaac..c901eabc 100644
--- a/examples/stage4_template.spec
+++ b/examples/stage4_template.spec
@@ -45,13 +45,6 @@ snapshot:
# default/stage3-x86-2006.1
source_subpath:
-# These are the hosts used as distcc slaves when distcc is enabled in your
-# catalyst.conf. It follows the same syntax as distcc-config --set-hosts and
-# is entirely optional.
-# example:
-# distcc_hosts: 127.0.0.1 192.168.0.1
-distcc_hosts:
-
# This is an optional directory containing portage configuration files. It
# follows the same syntax as /etc/portage and should be consistent across all
# targets to minimize problems.
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [gentoo-catalyst] [PATCH 21/21] catalyst: Drop ConfigParser
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
` (18 preceding siblings ...)
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 20/21] catalyst: Configure distcc_hosts in the config file Matt Turner
@ 2020-05-20 3:42 ` Matt Turner
19 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-20 3:42 UTC (permalink / raw
To: gentoo-catalyst; +Cc: Matt Turner
Signed-off-by: Matt Turner <mattst88@gentoo.org>
---
catalyst/config.py | 11 -----------
1 file changed, 11 deletions(-)
diff --git a/catalyst/config.py b/catalyst/config.py
index b527ada0..e1963f71 100644
--- a/catalyst/config.py
+++ b/catalyst/config.py
@@ -118,14 +118,3 @@ class SpecParser(ParserBase):
def __init__(self, filename=""):
if filename:
self.parse_file(filename)
-
-
-class ConfigParser(ParserBase):
-
- key_value_separator = '='
- multiple_values = False
- empty_values = True
-
- def __init__(self, filename=""):
- if filename:
- self.parse_file(filename)
--
2.26.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [gentoo-catalyst] [PATCH 13/21] catalyst: Remove support for source_matching="loose"
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 13/21] catalyst: Remove support for source_matching="loose" Matt Turner
@ 2020-05-20 22:47 ` Brian Dolbec
0 siblings, 0 replies; 24+ messages in thread
From: Brian Dolbec @ 2020-05-20 22:47 UTC (permalink / raw
To: gentoo-catalyst
On Tue, 19 May 2020 20:42:18 -0700
Matt Turner <mattst88@gentoo.org> wrote:
> This does not seem like a useful feature to me.
>
> Signed-off-by: Matt Turner <mattst88@gentoo.org>
> ---
> catalyst/base/stagebase.py | 3 ---
> catalyst/defaults.py | 1 -
> catalyst/support.py | 6 +++---
> etc/catalyst.conf | 12 ------------
> 4 files changed, 3 insertions(+), 19 deletions(-)
>
> diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
> index f39895fe..febaf969 100644
> --- a/catalyst/base/stagebase.py
> +++ b/catalyst/base/stagebase.py
> @@ -130,8 +130,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
> decomp_opt=self.settings["decomp_opt"])
> self.accepted_extensions =
> self.decompressor.search_order_extensions(
> self.settings["decompressor_search_order"])
> - log.notice("Source file specification matching setting is:
> %s",
> - self.settings["source_matching"])
> log.notice("Accepted source file extensions search order:
> %s", self.accepted_extensions)
> # save resources, it is not always needed
> @@ -409,7 +407,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
> normpath(self.settings["storedir"] + "/builds/" +
> self.settings["source_subpath"]),
> self.accepted_extensions,
> - self.settings["source_matching"] in ["strict"]
> )
> log.debug('Source path returned from file_check is: %s',
> self.settings["source_path"])
> diff --git a/catalyst/defaults.py b/catalyst/defaults.py
> index 412cb956..14f671fe 100644
> --- a/catalyst/defaults.py
> +++ b/catalyst/defaults.py
> @@ -45,7 +45,6 @@ confdefaults = {
> "repos": "%(storedir)s/repos",
> "sharedir": "/usr/share/catalyst",
> "shdir": "%(sharedir)s/targets",
> - "source_matching": "strict",
> "storedir": "/var/tmp/catalyst",
> "target_distdir": "/var/cache/distfiles",
> "target_pkgdir": "/var/cache/binpkgs",
> diff --git a/catalyst/support.py b/catalyst/support.py
> index c4a5c797..a6a6854a 100644
> --- a/catalyst/support.py
> +++ b/catalyst/support.py
> @@ -59,7 +59,7 @@ def cmd(mycmd, env=None, debug=False,
> fail_func=None): print_traceback=False)
>
>
> -def file_check(filepath, extensions=None, strict=True):
> +def file_check(filepath, extensions=None):
> '''Check for the files existence and that only one exists
> if others are found with various extensions
> '''
> @@ -73,8 +73,8 @@ def file_check(filepath, extensions=None,
> strict=True): ".CONTENTS") and not x.endswith(".CONTENTS.gz") and not
> x.endswith(".DIGESTS")] if len(files) == 1:
> return files[0]
> - if len(files) > 1 and strict:
> - msg = "Ambiguos Filename: %s\nPlease specify the correct
> extension as well" % filepath
> + if len(files) > 1:
> + msg = "Ambiguous Filename: %s\nPlease specify the correct
> extension as well" % filepath raise CatalystError(msg,
> print_traceback=False) target_file = None
> for ext in extensions:
I find this strict/loose option useful for testing various changes.
I have my config set to loose. That way I can change compressors at
will without changing spec file or deleting files, or editing the spec
to specify the extension.
While I find it useful. It may not be for most users. I would be ok
with this if the others agree. Otherwise, this is not complex code to
maintain. In fact is barely any code at all.
> diff --git a/etc/catalyst.conf b/etc/catalyst.conf
> index f64fe971..d33be15f 100644
> --- a/etc/catalyst.conf
> +++ b/etc/catalyst.conf
> @@ -50,18 +50,6 @@ envscript="/etc/catalyst/catalystrc"
> # (These options can be used together)
> options="autoresume bindist kerncache pkgcache seedcache"
>
> -# source_matching specifies how catalyst will match non-specific
> file names -# if the filename is not found as an exact match.
> -# ie: a filename without the extension specified. "/path/to/foo"
> -#
> -# possible values are:
> -# "strict" meaning if more than one file of that name is present
> with any -# file extension, then it will raise an
> exception. -# "loose" meaning it will search for an existing
> filename with an added -# extension from an ordered list
> of extensions determined from the -#
> decompressor_search_order specification in the spec file or (default)
> -source_matching="strict" -
> # port_logdir is where all build logs will be kept. This dir will be
> automatically cleaned # of all logs over 30 days old. If left
> undefined the logs will remain in the build directory # as usual and
> get cleaned every time a stage build is restarted.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-catalyst] [PATCH 19/21] catalyst: Set jobs/load-average via catalyst.conf
[not found] ` <94424974-cfe8-9766-8712-ae6fa8bbf825@veremit.xyz>
@ 2020-05-21 0:18 ` Matt Turner
[not found] ` <9d529c77-14b3-cb7b-f2ee-32b4bbc2f464@veremit.xyz>
0 siblings, 1 reply; 24+ messages in thread
From: Matt Turner @ 2020-05-21 0:18 UTC (permalink / raw
To: Michael 'veremitz' Everitt; +Cc: gentoo-catalyst
On Wed, May 20, 2020 at 4:00 PM Michael 'veremitz' Everitt
<gentoo@veremit.xyz> wrote:
>
> On 20/05/20 04:42, Matt Turner wrote:
> > We currently have two mechanisms of setting MAKEOPTS: in spec files and
> > in catalystrc.
> >
> > Setting makeopts in spec files doesn't make sense. The spec should
> > describe the thing that's being built and not contain options that are
> > specific to the build system.
> >
> > Setting makeopts via catalystrc is better, but it only applies to the
> > actual build system invocations, leaving emerge to run jobs serially or
> > again requiring configuration specific to the build machine to be put
> > into the spec file. For example:
> >
> > update_seed_command: ... --jobs 5 --load-average 5
> >
> > With jobs and load-average specified in catalyst.conf, catalyst has the
> > information required to configure both emerge and the build systems
> > emerge executes.
> >
> > This removes the undocumented makeopts spec file option and replaces it
> > with jobs and load-average settings in catalyst.conf.
> >
> > Signed-off-by: Matt Turner <mattst88@gentoo.org>
> > ---
> > catalyst/base/stagebase.py | 12 +++++-------
> > catalyst/defaults.py | 2 ++
> > doc/catalyst-config.5.txt | 15 ++++++++++++---
> > etc/catalyst.conf | 8 ++++++++
> > etc/catalystrc | 3 ---
> > targets/support/chroot-functions.sh | 8 ++++++++
> > 6 files changed, 35 insertions(+), 13 deletions(-)
> NACK. This and patch 20 make it impossible to customise specs for different
> arches/subarches/libc/etc using different distcc_hosts with varying numbers
> of hosts and/or cores available. In many cases, you may not be able to set
> up identical distcc 'farms' or 'clusters' with the complete set of
> toolchains/chroots applicable to a 'standard' installation, due to hardware
> limitations.
I don't follow. You mention varying distcc hosts per arches/subarches/libc.
Let's say there are 4 systems on the network. 3x amd64 systems and 1
mips system.
If I'm compiling for amd64 on system and I'm using distcc, there's a
fixed number of distcc hosts on the network, aren't there?
If I'm building for mips on the mips system with cross compilers
installed on, say two of the amd64 systems, then there's a fixed
number of distcc hosts on the network, aren't there?
So, I don't think arches (or subarches) has any bearing.
And for different libcs, distcc doesn't have any bearing on that, does
it? distcc (without pump mode, which has been removed from Gentoo)
sends preprocessed source files across the network, so what libc you
have doesn't have any bearing either.
You seem to be thinking that setting a single system-wide distcc_hosts
is somehow limiting, but I can't see how. Please explain.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-catalyst] [PATCH 19/21] catalyst: Set jobs/load-average via catalyst.conf
[not found] ` <9d529c77-14b3-cb7b-f2ee-32b4bbc2f464@veremit.xyz>
@ 2020-05-21 1:19 ` Matt Turner
0 siblings, 0 replies; 24+ messages in thread
From: Matt Turner @ 2020-05-21 1:19 UTC (permalink / raw
To: Michael 'veremitz' Everitt; +Cc: gentoo-catalyst
On Wed, May 20, 2020 at 5:37 PM Michael 'veremitz' Everitt
<gentoo@veremit.xyz> wrote:
>
> On 21/05/20 01:18, Matt Turner wrote:
> > On Wed, May 20, 2020 at 4:00 PM Michael 'veremitz' Everitt
> > <gentoo@veremit.xyz> wrote:
> >> On 20/05/20 04:42, Matt Turner wrote:
> >>> We currently have two mechanisms of setting MAKEOPTS: in spec files and
> >>> in catalystrc.
> >>>
> >>> Setting makeopts in spec files doesn't make sense. The spec should
> >>> describe the thing that's being built and not contain options that are
> >>> specific to the build system.
> >>>
> >>> Setting makeopts via catalystrc is better, but it only applies to the
> >>> actual build system invocations, leaving emerge to run jobs serially or
> >>> again requiring configuration specific to the build machine to be put
> >>> into the spec file. For example:
> >>>
> >>> update_seed_command: ... --jobs 5 --load-average 5
> >>>
> >>> With jobs and load-average specified in catalyst.conf, catalyst has the
> >>> information required to configure both emerge and the build systems
> >>> emerge executes.
> >>>
> >>> This removes the undocumented makeopts spec file option and replaces it
> >>> with jobs and load-average settings in catalyst.conf.
> >>>
> >>> Signed-off-by: Matt Turner <mattst88@gentoo.org>
> >>> ---
> >>> catalyst/base/stagebase.py | 12 +++++-------
> >>> catalyst/defaults.py | 2 ++
> >>> doc/catalyst-config.5.txt | 15 ++++++++++++---
> >>> etc/catalyst.conf | 8 ++++++++
> >>> etc/catalystrc | 3 ---
> >>> targets/support/chroot-functions.sh | 8 ++++++++
> >>> 6 files changed, 35 insertions(+), 13 deletions(-)
> >> NACK. This and patch 20 make it impossible to customise specs for different
> >> arches/subarches/libc/etc using different distcc_hosts with varying numbers
> >> of hosts and/or cores available. In many cases, you may not be able to set
> >> up identical distcc 'farms' or 'clusters' with the complete set of
> >> toolchains/chroots applicable to a 'standard' installation, due to hardware
> >> limitations.
> > I don't follow. You mention varying distcc hosts per arches/subarches/libc.
> >
> > Let's say there are 4 systems on the network. 3x amd64 systems and 1
> > mips system.
> >
> > If I'm compiling for amd64 on system and I'm using distcc, there's a
> > fixed number of distcc hosts on the network, aren't there?
> Yes, but do you feed your amd64 jobs to your mips system? how do you filter
> between them?
I wouldn't... but I don't think that matters? You just set
distcc_hosts to not send jobs to that system.
> > If I'm building for mips on the mips system with cross compilers
> > installed on, say two of the amd64 systems, then there's a fixed
> > number of distcc hosts on the network, aren't there?
> >
> > So, I don't think arches (or subarches) has any bearing.
> >
> > And for different libcs, distcc doesn't have any bearing on that, does
> > it? distcc (without pump mode, which has been removed from Gentoo)
> > sends preprocessed source files across the network, so what libc you
> > have doesn't have any bearing either.
> I have one or two systems with musl toolchains, but not all, again, how do
> I choose between them, from a common build host, for each different spec
> file variant I wish to run?
Like I said, distcc sends a preprocessed file across the network, so
there's not a dependence on the libc. I.e., a musl system running
distccd can compile files just fine for a system running glibc.
> > You seem to be thinking that setting a single system-wide distcc_hosts
> > is somehow limiting, but I can't see how. Please explain.
> >
> I fail to see how you can select from a pool of distcc slaves for a given
> job type/configuration.
>
> I'm assuming you've used distcc before? With cluster project's help I
> actually have several distcc daemons running on a given host, if I should
> require eg. gcc-8, 9 or (in the near future) 10. These run on different
> ports on each slave system.
> How would you propose selecting between these separate daemons for a given
> build in your new schema?
Yes, I use distcc to cross compile for slow mips and arm systems.
I don't understand why it takes so much more effort on the part of the
receiver to interpret your messages, but good grief.
I think the case Michael is thinking of is, e.g. a system building
both rel_type: default and rel_type: hardened stages. The set of
distcc slaves that have a non-hardened toolchain might be different
from the set of distcc slaves with a hardened toolchain.
I'm unsure how this works, so I'm asking #gentoo-toolchain.
So all of the previous email exchange was a collection of red herrings...
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2020-05-21 1:19 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-05-20 3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 02/21] targets: Remove check_genkernel_version() function Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 03/21] catalyst: Drop --cli option Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 04/21] catalyst: Remove PythonDir setting Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 05/21] catalyst: Switch internal snapshot option parsing to SpecParser Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 06/21] catalyst: Remove unused decompression_mode spec option Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 07/21] catalyst: Delete unused {required,valid}_build_targets Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 08/21] catalyst: Disallow DEBUG and VERBOSE in spec files Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 09/21] catalyst: Add decompressor_search_order as valid spec option Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 10/21] catalyst: Remove spec file options from valid_config_file_values Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 11/21] catalyst: Add a missing options to valid_config_file_values Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 12/21] catalyst: Disallow config file options in spec files Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 13/21] catalyst: Remove support for source_matching="loose" Matt Turner
2020-05-20 22:47 ` Brian Dolbec
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 14/21] catalyst: Convert catalyst.conf to TOML Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 15/21] catalyst: Verify config options against valid_config_file_values Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 16/21] catalyst: Don't even try to make envars from dicts Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 17/21] catalyst: Only emit true boolean envars Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 18/21] catalyst: Support emitting int/float envars Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 19/21] catalyst: Set jobs/load-average via catalyst.conf Matt Turner
[not found] ` <94424974-cfe8-9766-8712-ae6fa8bbf825@veremit.xyz>
2020-05-21 0:18 ` Matt Turner
[not found] ` <9d529c77-14b3-cb7b-f2ee-32b4bbc2f464@veremit.xyz>
2020-05-21 1:19 ` Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 20/21] catalyst: Configure distcc_hosts in the config file Matt Turner
2020-05-20 3:42 ` [gentoo-catalyst] [PATCH 21/21] catalyst: Drop ConfigParser Matt Turner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox