* [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE}
@ 2018-01-15 15:27 Francesco Riosa
2018-01-15 17:07 ` Mike Gilbert
2018-01-16 7:57 ` Michał Górny
0 siblings, 2 replies; 8+ messages in thread
From: Francesco Riosa @ 2018-01-15 15:27 UTC (permalink / raw
To: gentoo development
In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added
to all python eclasses, it's useful for developers that want test and
mark the package for newer versions of python.
However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not
usable if:
- the user want only python 2.7 and 3.6 (latest) installed
no python 3.5
- don't want to mess dependencies (the warnings in the eclass are scary)
So, what can be done to let the user choose it's preferred python
version(s) without having to build it's own overlay?
One solution is to change ebuilds in tree to include a user variable in
the PYTHON* arrays, example:
-PYTHON_COMPAT=( python3_{4,5} )
+PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} )
if someone want to bet that packages are ok with 3.6/latest (even before
a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to
/etc/portage/make.conf and run egencache.
Indeed biggest problem in changing $PYTHON* variables is that metadata
also change and cache is invalidated.
However if the problem is known (*), and if the change to metadata is
stable per "system"/"gentoo repo clone", then the solution to the
problem is easy; just run $(egencache --update -j$(nproc) --repo gentoo)
after each sync.
The most important thing is that this solution is scriptable and need no
human intervention.
Notice also that it's easy to have an array with duplicate values
PYTHON_COMPAT=( python3_6 python3_6 ) but at a first glance
_python_set_impls() is resilient to this.
(*) In a perfect world, where global variables that can change metadata
are known {egencache,emerge} can be made conscious and warn if the cache
is invalid, but that's out of scope at the moment.
The procedure to find out which packages are enabled for python 3.5 but
not for python 3.6 is not totally straightforward but this should be the
list of ~amd64 packages interested:
app-admin/glance [15.0.0]
app-admin/setools [4.1.1]
app-arch/bloscpack [0.11.0]
app-arch/patool [1.12]
app-arch/vimball [0.5.1]
app-backup/attic [0.16]
app-backup/untangle-https-backup [0.0.7]
app-benchmarks/bootchart2 [0.14.7-r1]
app-emulation/libguestfs [1.36.5]
app-emulation/sen [0.5.1]
app-i18n/transifex-client [0.12.4]
app-misc/asciinema [1.4.0]
app-misc/gramps [4.2.5]
app-misc/icdiff [1.9.1]
app-misc/jira-cli [2.1.5]
app-misc/lirc [0.10.0_rc2]
app-misc/solaar [0.9.2-r3]
app-misc/terminal-colors [2.2]
app-mobilephone/obexftp [0.24-r1]
app-office/libreoffice-bin [5.4.2.2-r1]
app-pda/libimobiledevice [1.2.0]
app-portage/elogviewer [2.7]
app-portage/gentoolkit-dev [0.3.1]
app-portage/gs-elpa [0.1.3-r1]
app-portage/g-sorcery [0.2.1-r1]
app-portage/gs-pypi [0.2.1-r1]
app-portage/pqlop [0.02-r1]
app-text/grip [4.2.0]
app-text/gtranslator [2.91.7-r1]
app-text/landslide [1.1.3]
app-text/pelican [3.7.1]
app-text/pytextile [2.3.3]
app-text/sigil [0.9.8]
app-text/wiki2beamer [0.9.5-r1]
app-vim/jedi [0.8_p20171015]
app-vim/pyclewn [2.1-r1]
dev-games/vamos [0.8.2]
dev-lang/hy [0.13.1]
dev-libs/gom [0.3.2]
dev-libs/Ice [3.6.3-r1]
dev-libs/libcec [4.0.2-r1]
dev-libs/libsolv [0.6.22]
dev-libs/link-grammar [5.3.11]
dev-perl/Inline-Python [0.560.0]
dev-python/abstract_rendering [0.5.1]
dev-python/algopy [0.5.3]
dev-python/amqplib [1.0.2-r1]
dev-python/ansi2html [1.3.0]
dev-python/aodhclient [0.8.0]
dev-python/argcomplete [1.9.2]
dev-python/asdf [1.2.1]
dev-python/astlib [0.8.0]
dev-python/astrodendro [0.2.0]
dev-python/astroml [0.3]
dev-python/astroml-addons [0.2.2]
dev-python/astroscrappy [1.0.5]
dev-python/awscli [1.14.16]
dev-python/backports-abc [0.5]
dev-python/beaker [1.8.1]
dev-python/behave [1.2.5-r1]
dev-python/bibtexparser [0.6.2]
dev-python/breathe [4.6.0]
dev-python/castellan [0.12.2]
dev-python/ccdproc [1.2.0]
dev-python/cgkit [2.0.0-r1]
dev-python/cgroup-utils [0.6]
dev-python/chump [1.5.2]
dev-python/cliff [2.8.0]
dev-python/cliff-tablib [1.1]
dev-python/Coffin [2.0.1]
dev-python/construct [2.8.17]
dev-python/cursive [0.1.2]
dev-python/cvxopt [1.1.9]
dev-python/d2to1 [0.2.12_p1]
dev-python/demjson [2.2.4]
dev-python/django-compressor [1.5]
dev-python/django-sortedm2m [1.3.2]
dev-python/django-tables2 [1.1.2]
dev-python/dnslib [0.9.6]
dev-python/dogpile-core [0.4.1]
dev-python/ed25519ll [0.6]
dev-python/enzyme [0.4.1-r2]
dev-python/fedmsg [0.18.3]
dev-python/feedgenerator [1.9]
dev-python/flask-admin [1.5.0]
dev-python/flask-appconfig [0.11.1-r1]
dev-python/flask-babelex [0.9.3]
dev-python/flask-bootstrap [3.3.7.1]
dev-python/flask-debug [0.4.3]
dev-python/flask-mongoengine [0.9.3]
dev-python/flask-nav [0.6]
dev-python/flask-pymongo [0.4.1]
dev-python/flask-whooshalchemy [0.8]
dev-python/flipflop [1.0]
dev-python/formencode [1.3.0-r2]
dev-python/fuzzywuzzy [0.12.0]
dev-python/gcs-oauth2-boto-plugin [1.14]
dev-python/geoalchemy2 [0.3]
dev-python/glance_store [0.22.0]
dev-python/guessit [2.1.4]
dev-python/gwcs [0.7]
dev-python/hglib [2.1]
dev-python/hiredis [0.2.0]
dev-python/httreplay [0.2.0]
dev-python/husl [4.0.3]
dev-python/hvac [0.2.17]
dev-python/id3-py [1.2-r1]
dev-python/ImageHash [3.7]
dev-python/imageio [2.1.1]
dev-python/influxdb [2.12.0]
dev-python/intelhex [2.0]
dev-python/ipy [0.83]
dev-python/irc [15.0.6]
dev-python/iso3166 [0.8]
dev-python/iso_639 [0.4.5]
dev-python/jaraco-classes [1.4]
dev-python/jaraco-collections [1.5]
dev-python/jaraco-functools [1.15.1]
dev-python/jaraco-itertools [2.0]
dev-python/jaraco-logging [1.5]
dev-python/jaraco-stream [1.1.1]
dev-python/jaraco-text [1.8]
dev-python/jellyfish [0.5.6]
dev-python/jenkinsapi [0.2.29]
dev-python/jpype [0.6.2]
dev-python/jsonpickle [0.9.3]
dev-python/jupyter [1.0.0-r1]
dev-python/jupyter_console [5.1.0]
dev-python/keystonemiddleware [4.17.0]
dev-python/kitchen [1.2.4]
dev-python/Kivy [1.10.0]
dev-python/ldappool [2.1.0]
dev-python/libcloud [1.2.1]
dev-python/libvirt-python [3.10.0]
dev-python/libzilla [1.4]
dev-python/matplotlib2tikz [0.6.14]
dev-python/maybe [0.4.0]
dev-python/meld3 [1.0.2-r1]
dev-python/meteor-ejson [1.1.0]
dev-python/microversion-parse [0.1.4]
dev-python/mmh3 [2.3.1]
dev-python/moviepy [0.2.2]
dev-python/mysql-connector-python [2.1.4]
dev-python/nagiosplugin [1.2.4]
dev-python/nbdime [0.2.0]
dev-python/netcdf4-python [1.2.2]
dev-python/neutron-lib [1.9.1]
dev-python/nose2 [0.6.5]
dev-python/Numdifftools [0.9.17]
dev-python/odfpy [1.3.2]
dev-python/openstacksdk [0.9.17]
dev-python/os-brick [1.15.5]
dev-python/osc-lib [1.7.0]
dev-python/oslo-cache [1.25.1]
dev-python/oslo-context [2.17.1]
dev-python/oslo-db [4.25.1]
dev-python/oslo-log [3.30.2]
dev-python/oslo-messaging [5.30.2]
dev-python/oslo-middleware [3.30.1]
dev-python/oslo-privsep [1.22.1]
dev-python/oslo-reports [1.22.1]
dev-python/oslo-rootwrap [5.9.2]
dev-python/oslo-service [1.25.1]
dev-python/oslo-versionedobjects [1.26.1]
dev-python/oslo-vmware [2.23.1]
dev-python/osprofiler [1.11.0]
dev-python/os-testr [0.8.0]
dev-python/os-traits [0.3.3]
dev-python/os-vif [1.7.0]
dev-python/os-win [2.2.0]
dev-python/os-xenapi [0.2.0]
dev-python/ovs [2.7.2]
dev-python/ovsdbapp [0.4.0]
dev-python/owslib [0.16.0]
dev-python/pelican-minify [0.9]
dev-python/pid [2.1.1]
dev-python/pika [0.10.0]
dev-python/pika-pool [0.1.3]
dev-python/pilkit [2.0]
dev-python/pillowfight [0.2]
dev-python/plotly [1.9.6]
dev-python/prov [1.5.1]
dev-python/pycadf [2.6.0]
dev-python/pycanberra [0_pre20130515]
dev-python/pycipher [0.2-r1]
dev-python/PyDbLite [3.0.4]
dev-python/pydot-ng [1.0.0]
dev-python/pydotplus [2.0.2]
dev-python/pyds9 [1.8.1]
dev-python/pyee [1.0.2]
dev-python/pyfeyn [1.0.0]
dev-python/pyfits [3.4-r1]
dev-python/pyflann [1.9.1]
dev-python/pygal [2.1.1]
dev-python/pygccxml [1.7.5]
dev-python/pyghmi [1.0.22]
dev-python/pygpu [0.6.7]
dev-python/pygsl [2.1.1]
dev-python/pykwalify [1.5.2]
dev-python/pylibacl [0.5.0-r1]
dev-python/pyminuit [1.2.1-r1]
dev-python/pynzb [0.1.0-r1]
dev-python/pyopencl [2017.2]
dev-python/pypowervm [1.1.6]
dev-python/pyproj [1.9.5.1]
dev-python/pyrqlite [2.0]
dev-python/pyrtf [0.45-r2]
dev-python/pysaml2 [4.5.0]
dev-python/pyscaffold [2.4.4]
dev-python/pyshark [0.3.7.2]
dev-python/pytest-qt [2.3.0]
dev-python/pytest-raisesregexp [2.1]
dev-python/python-barbicanclient [4.5.2]
dev-python/python-ceilometerclient [2.6.2]
dev-python/python-cinderclient [3.1.0]
dev-python/python-ddp [0.1.5]
dev-python/python-designateclient [2.7.0]
dev-python/python-djvulibre [0.8]
dev-python/python-gammu [2.5]
dev-python/python-gnupg [0.4.0]
dev-python/python-heatclient [1.11.1]
dev-python/python-ironicclient [1.17.0]
dev-python/python-magnumclient [2.3.1]
dev-python/python-manilaclient [1.14.0]
dev-python/python-meteor [0.1.6]
dev-python/python-mistralclient [3.1.4]
dev-python/python-monascaclient [1.7.0]
dev-python/python-mpd [0.5.5]
dev-python/python-neutronclient [6.5.0]
dev-python/python-novaclient [9.1.1]
dev-python/python-openstackclient [3.12.0]
dev-python/python_orocos_kdl [1.3.1-r1]
dev-python/python-otrs [0.3.0]
dev-python/python-rethinkdb [2.3.0]
dev-python/python-saharaclient [1.1.0]
dev-python/python-scsi [0_pre160211]
dev-python/python-senlinclient [1.2.0]
dev-python/python-swiftclient [3.4.0]
dev-python/python-troveclient [2.5.0]
dev-python/python-zaqarclient [1.2.0]
dev-python/python-zunclient [0.4.1]
dev-python/pyuv [1.2.0]
dev-python/pyzor [1.0.0]
dev-python/rackspace-auth-openstack [1.3]
dev-python/rackspace-monitoring [0.8.0]
dev-python/rackspace-novaclient [2.1]
dev-python/raven [5.31.0]
dev-python/redis-py-cluster [1.3.4]
dev-python/rencode [1.0.5]
dev-python/requests-cache [0.4.12]
dev-python/retry-decorator [1.0.0]
dev-python/rnc2rng [2.5]
dev-python/rply [0.7.4]
dev-python/rtslib-fb [2.1.62]
dev-python/ryu [4.14]
dev-python/seaborn [0.7.1]
dev-python/semantic_version [2.6.0-r1]
dev-python/sepolgen [2.6]
dev-python/slackclient [1.1.0]
dev-python/sleekxmpp [1.3.1-r1]
dev-python/spectral-cube [0.4.0]
dev-python/specutils [0.2.2]
dev-python/sqlalchemy-rqlite [1.0]
dev-python/stormpath [2.4.5]
dev-python/structlog [16.1.0]
dev-python/stsci-distutils [0.3.7]
dev-python/stsci-sphinxext [1.2.1]
dev-python/tablib [0.11.2]
dev-python/tagpy [2013.1]
dev-python/taskflow [2.14.1]
dev-python/tempest-lib [1.0.0]
dev-python/tenacity [4.4.0]
dev-python/testfixtures [4.9.1]
dev-python/tinydb [3.1.3]
dev-python/tooz [1.58.0]
dev-python/torment [3.0.3]
dev-python/transmissionrpc [0.11]
dev-python/trollius [2.1]
dev-python/tvdb_api [1.10_pre20150406-r1]
dev-python/twitter [1.17.1]
dev-python/uncompyle6 [2.10.1]
dev-python/urdf_parser_py [0.3.3]
dev-python/utmp [0.4]
dev-python/vcstools [0.1.40]
dev-python/versiontools [1.9.1-r1]
dev-python/w3lib [1.18.0]
dev-python/webassets [0.12]
dev-python/ws4py [0.3.4]
dev-python/wsgiintercept [1.3.1]
dev-python/wtf-peewee [0.2.6]
dev-python/xonsh [0.4.7]
dev-python/yapsy [1.11.223-r1]
dev-python/yaql [1.1.3]
dev-python/zake [0.2.1]
dev-util/cppcheck [1.81]
dev-util/cram [0.7]
dev-util/devhelp [3.24.0]
dev-util/distro-info [0.14]
dev-util/dogtail [0.9.10]
dev-util/fatrace [0.12]
dev-util/gcovr [3.3]
dev-util/howdoi [1.1.9]
dev-util/rosinstall [0.7.8]
dev-util/spec-cleaner [0.8.9]
dev-util/wstool [0.1.17]
dev-vcs/git-cola [2.11]
dev-vcs/gitg [3.26.0]
games-misc/ponysay [3.0.2]
games-util/nml [0.4.4-r1]
gnome-base/gnome-shell [3.24.3]
gnome-extra/chrome-gnome-shell [9]
gnome-extra/cinnamon [3.6.6-r2]
gnome-extra/gnome-builder [3.24.2]
media-gfx/fontforge [20170731-r3]
media-gfx/sigal [1.3.0]
media-libs/pyliblo [0.10.0]
media-sound/gnome-music [3.24.2]
media-sound/pithos [1.3.1]
media-tv/tvnamer [2.4]
media-video/pitivi [0.98.1]
media-video/subliminal [2.0.5-r1]
net-analyzer/carl [0.9-r2]
net-analyzer/flent [1.0.1]
net-analyzer/rrdtool [1.7.0]
net-analyzer/speedtest-cli [1.0.2]
net-irc/irker [2.18]
net-irc/limnoria [20171025]
net-libs/libsignon-glib [1.14]
net-misc/electron-cash [3.0]
net-misc/electrum [3.0.5]
net-misc/electrum-ltc [3.0_pre20171218]
net-misc/gns3-gui [2.0.3-r1]
net-misc/gns3-server [2.0.3-r1]
net-misc/openvswitch [2.8.1]
net-misc/trackma [0.7.4]
net-misc/whatportis [0.6]
net-misc/you-get [0.4.536]
net-news/liferea [1.12_rc3-r1]
net-wireless/urh [1.6.4.1]
sci-astronomy/pyephem [3.7.6.0]
sci-biology/biopython [1.68]
sci-libs/clblas [2.10]
sci-libs/grib_api [1.14.5]
sci-libs/shogun [5.0.0]
sci-mathematics/dunshire [0.1.1]
sci-mathematics/pymc [2.3.6]
sci-mathematics/relational [2.5]
sci-visualization/fityk [1.3.1]
sys-apps/policycoreutils [2.7]
sys-apps/selinux-python [2.7]
sys-auth/keystone [12.0.0]
sys-block/targetcli-fb [2.1.45]
sys-cluster/cinder [11.0.2]
sys-cluster/heat [9.0.2]
sys-cluster/nova [16.0.4]
sys-cluster/sanlock [3.4.0-r1]
sys-fs/bcache-tools [1.0.8_p20140220-r1]
sys-kernel/kergen [0.1.4]
sys-libs/libsemanage [2.7]
sys-power/acpilight [1.0-r1]
www-apps/nikola [7.8.8]
www-apps/radicale [1.1.1]
www-misc/urlwatch [2.6]
x11-misc/menumaker [0.99.10]
x11-wm/qtile [0.10.6]
dev-db/pgadmin4 [2.1]
dev-libs/protobuf [3.5.1.1]
dev-python/flask-login [0.3.2-r1]
dev-python/flask-mail [0.9.1-r1]
dev-python/flask-migrate [2.1.1]
dev-python/flask-principal [0.4.0-r1]
dev-python/flask-security [1.7.5-r1]
dev-python/flask-uploads [0.2.0-r1]
dev-python/funcsigs [1.0.2-r1]
dev-python/translate-toolkit [2.0.0]
kde-apps/lokalize [17.12.1]
media-gfx/hugin [2017.0]
media-libs/gexiv2 [0.10.6]
net-irc/hexchat [2.12.4-r2]
sys-fs/zfs [0.7.5-r1]
sys-process/glances [2.11.1]
dev-python/batinfo [0.3]
dev-python/flask-babel [0.11.2]
dev-python/flask-script [2.0.6]
dev-python/flask-testing [0.6.2]
dev-python/flask-wtf [0.14.2]
media-libs/opencv [3.3.0-r4]
dev-python/ip-associations-python-novaclient-ext [0.2]
dev-python/os-diskconfig-python-novaclient-ext [0.1.3]
dev-python/os-networksv2-python-novaclient-ext [0.26]
dev-python/os-virtual-interfacesv2-python-novaclient-ext [0.20]
dev-python/rax-default-network-flags-python-novaclient-ext [0.4.0]
dev-python/rax-scheduled-images-python-novaclient-ext [0.3.1]
net-analyzer/nagios-check_openvpn-simple [0.0.1-r1]
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} 2018-01-15 15:27 [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} Francesco Riosa @ 2018-01-15 17:07 ` Mike Gilbert 2018-01-16 0:05 ` Francesco Riosa 2018-01-16 7:57 ` Michał Górny 1 sibling, 1 reply; 8+ messages in thread From: Mike Gilbert @ 2018-01-15 17:07 UTC (permalink / raw To: Gentoo Dev On Mon, Jan 15, 2018 at 10:27 AM, Francesco Riosa <vivo75@gmail.com> wrote: > > In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added > to all python eclasses, it's useful for developers that want test and > mark the package for newer versions of python. > > However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not > usable if: > - the user want only python 2.7 and 3.6 (latest) installed > no python 3.5 > - don't want to mess dependencies (the warnings in the eclass are scary) > > So, what can be done to let the user choose it's preferred python > version(s) without having to build it's own overlay? > > One solution is to change ebuilds in tree to include a user variable in > the PYTHON* arrays, example: > > -PYTHON_COMPAT=( python3_{4,5} ) > +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} ) > > if someone want to bet that packages are ok with 3.6/latest (even before > a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to > /etc/portage/make.conf and run egencache. I like the idea to inject values into PYTHON_COMPAT instead of overriding it completely. I'm pretty sure this can/should be implemented in the eclass without touching ebuilds. I'm not sure I really like the idea of affecting the other metadata variables. I can see your point about wanting to remove python versions that would otherwise satisfy dependencies. If metadata is modified this way, it would definitely be "unsupported". As far as implementation, you would probably need to write the patches for this. Also, I expect the QA team might have some objections, so you may want to discuss it with them (especially mgorny) before spending too much time on it. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} 2018-01-15 17:07 ` Mike Gilbert @ 2018-01-16 0:05 ` Francesco Riosa 2018-01-16 0:40 ` Alec Warner 0 siblings, 1 reply; 8+ messages in thread From: Francesco Riosa @ 2018-01-16 0:05 UTC (permalink / raw To: gentoo-dev, Mike Gilbert On 15/01/2018 18:07, Mike Gilbert wrote: > On Mon, Jan 15, 2018 at 10:27 AM, Francesco Riosa <vivo75@gmail.com> wrote: >> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added >> to all python eclasses, it's useful for developers that want test and >> mark the package for newer versions of python. >> >> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not >> usable if: >> - the user want only python 2.7 and 3.6 (latest) installed >> no python 3.5 >> - don't want to mess dependencies (the warnings in the eclass are scary) >> >> So, what can be done to let the user choose it's preferred python >> version(s) without having to build it's own overlay? >> >> One solution is to change ebuilds in tree to include a user variable in >> the PYTHON* arrays, example: >> >> -PYTHON_COMPAT=( python3_{4,5} ) >> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} ) >> >> if someone want to bet that packages are ok with 3.6/latest (even before >> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to >> /etc/portage/make.conf and run egencache. > I like the idea to inject values into PYTHON_COMPAT instead of > overriding it completely. I'm pretty sure this can/should be > implemented in the eclass without touching ebuilds. In my mind that was to leave ebuilds developers the ability to opt out, but well that could also be done in the eclasses. Opt out could be beneficial for packages that only support python 2.7, or where there are known and documented problems with different python versions. > > I'm not sure I really like the idea of affecting the other metadata > variables. I can see your point about wanting to remove python > versions that would otherwise satisfy dependencies. If metadata is > modified this way, it would definitely be "unsupported". I've not tought about remove python versions, only add them (beneficial for users that like to use experimental python versions) However the supported python version are translated and build important cached variables IUSE, DEPEND, etc. so there is no way to cleanly modify those variable after the cache has been generated. The only viable option is to regenerate, update or delete it. > > As far as implementation, you would probably need to write the patches for this. If there is consensus that's not a problem, cannot guarantee to be fast however > > Also, I expect the QA team might have some objections, so you may want > to discuss it with them (especially mgorny) before spending too much > time on it. I'd like to hear mgorny opinions but that should be a more extended decision than only QA and the python eclasses developer(s). If nothing else because deciding that sometimes may be good to let the user break the cache is a global decision and documentation need to be added. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} 2018-01-16 0:05 ` Francesco Riosa @ 2018-01-16 0:40 ` Alec Warner 2018-01-16 13:16 ` Francesco Riosa 0 siblings, 1 reply; 8+ messages in thread From: Alec Warner @ 2018-01-16 0:40 UTC (permalink / raw To: Gentoo Dev; +Cc: Mike Gilbert [-- Attachment #1: Type: text/plain, Size: 3617 bytes --] On Mon, Jan 15, 2018 at 7:05 PM, Francesco Riosa <vivo75@gmail.com> wrote: > > > On 15/01/2018 18:07, Mike Gilbert wrote: > > On Mon, Jan 15, 2018 at 10:27 AM, Francesco Riosa <vivo75@gmail.com> > wrote: > >> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added > >> to all python eclasses, it's useful for developers that want test and > >> mark the package for newer versions of python. > >> > >> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not > >> usable if: > >> - the user want only python 2.7 and 3.6 (latest) installed > >> no python 3.5 > >> - don't want to mess dependencies (the warnings in the eclass are scary) > >> > >> So, what can be done to let the user choose it's preferred python > >> version(s) without having to build it's own overlay? > >> > >> One solution is to change ebuilds in tree to include a user variable in > >> the PYTHON* arrays, example: > >> > >> -PYTHON_COMPAT=( python3_{4,5} ) > >> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} ) > >> > >> if someone want to bet that packages are ok with 3.6/latest (even before > >> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to > >> /etc/portage/make.conf and run egencache. > > I like the idea to inject values into PYTHON_COMPAT instead of > > overriding it completely. I'm pretty sure this can/should be > > implemented in the eclass without touching ebuilds. > In my mind that was to leave ebuilds developers the ability to opt out, > but well that could also be done in the eclasses. > Opt out could be beneficial for packages that only support python 2.7, > or where there are known and documented problems with different python > versions. > > > > I'm not sure I really like the idea of affecting the other metadata > > variables. I can see your point about wanting to remove python > > versions that would otherwise satisfy dependencies. If metadata is > > modified this way, it would definitely be "unsupported". > I've not tought about remove python versions, only add them (beneficial > for users that like to use experimental python versions) > However the supported python version are translated and build important > cached variables IUSE, DEPEND, etc. so there is no way to cleanly modify > those variable after the cache has been generated. > The only viable option is to regenerate, update or delete it. > > > > > As far as implementation, you would probably need to write the patches > for this. > If there is consensus that's not a problem, cannot guarantee to be fast > however > > > > Also, I expect the QA team might have some objections, so you may want > > to discuss it with them (especially mgorny) before spending too much > > time on it. > I'd like to hear mgorny opinions but that should be a more extended > decision than only QA and the python eclasses developer(s). > If nothing else because deciding that sometimes may be good to let the > user break the cache is a global decision and documentation need to be > added. > I'm seeing less value in this being a 'cache-breaking' exercise and more value in it simply being a custom eclass. If you hoist the eclass into an overlay and modify it (e.g. to set the var and get the deps you want) the caching all works out fine. 1. The source of the data is the hoisted eclass. 2. The eclass mtime changed. 3. package managers can see that and update cache metadata for inheriting ebuilds. 4. Bonus, once the cache is generated we have a valid means to see if the cache remains valid. 5. Double bonus, any ebuilds not inheriting the eclass are unaffected. I'm not sure why this is so onerous. -A [-- Attachment #2: Type: text/html, Size: 4646 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} 2018-01-16 0:40 ` Alec Warner @ 2018-01-16 13:16 ` Francesco Riosa 0 siblings, 0 replies; 8+ messages in thread From: Francesco Riosa @ 2018-01-16 13:16 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 4454 bytes --] On 16/01/2018 01:40, Alec Warner wrote: > > > On Mon, Jan 15, 2018 at 7:05 PM, Francesco Riosa <vivo75@gmail.com > <mailto:vivo75@gmail.com>> wrote: > > > > On 15/01/2018 18:07, Mike Gilbert wrote: > > On Mon, Jan 15, 2018 at 10:27 AM, Francesco Riosa > <vivo75@gmail.com <mailto:vivo75@gmail.com>> wrote: > >> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized > and added > >> to all python eclasses, it's useful for developers that want > test and > >> mark the package for newer versions of python. > >> > >> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE > is not > >> usable if: > >> - the user want only python 2.7 and 3.6 (latest) installed > >> no python 3.5 > >> - don't want to mess dependencies (the warnings in the eclass > are scary) > >> > >> So, what can be done to let the user choose it's preferred python > >> version(s) without having to build it's own overlay? > >> > >> One solution is to change ebuilds in tree to include a user > variable in > >> the PYTHON* arrays, example: > >> > >> -PYTHON_COMPAT=( python3_{4,5} ) > >> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} ) > >> > >> if someone want to bet that packages are ok with 3.6/latest > (even before > >> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to > >> /etc/portage/make.conf and run egencache. > > I like the idea to inject values into PYTHON_COMPAT instead of > > overriding it completely. I'm pretty sure this can/should be > > implemented in the eclass without touching ebuilds. > In my mind that was to leave ebuilds developers the ability to opt > out, > but well that could also be done in the eclasses. > Opt out could be beneficial for packages that only support python 2.7, > or where there are known and documented problems with different python > versions. > > > > I'm not sure I really like the idea of affecting the other metadata > > variables. I can see your point about wanting to remove python > > versions that would otherwise satisfy dependencies. If metadata is > > modified this way, it would definitely be "unsupported". > I've not tought about remove python versions, only add them > (beneficial > for users that like to use experimental python versions) > However the supported python version are translated and build > important > cached variables IUSE, DEPEND, etc. so there is no way to cleanly > modify > those variable after the cache has been generated. > The only viable option is to regenerate, update or delete it. > > > > > As far as implementation, you would probably need to write the > patches for this. > If there is consensus that's not a problem, cannot guarantee to be > fast > however > > > > Also, I expect the QA team might have some objections, so you > may want > > to discuss it with them (especially mgorny) before spending too much > > time on it. > I'd like to hear mgorny opinions but that should be a more extended > decision than only QA and the python eclasses developer(s). > If nothing else because deciding that sometimes may be good to let the > user break the cache is a global decision and documentation need to be > added. > > > I'm seeing less value in this being a 'cache-breaking' exercise and > more value in it simply being a custom eclass. > > If you hoist the eclass into an overlay and modify it (e.g. to set the > var and get the deps you want) the caching > all works out fine. > > 1. The source of the data is the hoisted eclass. > 2. The eclass mtime changed. > 3. package managers can see that and update cache metadata for > inheriting ebuilds. > 4. Bonus, once the cache is generated we have a valid means to see if > the cache remains valid. > 5. Double bonus, any ebuilds not inheriting the eclass are unaffected. > > I'm not sure why this is so onerous. > > -A That's a good idea, in the past I've failed to completely understand how portage inherit eclasses in overlays, but if I can figure that out probably this could be a solution. Also depending from the magnitude of changes the four python eclasses need and the correlated maintenance burden. [-- Attachment #2: Type: text/html, Size: 7199 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} 2018-01-15 15:27 [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} Francesco Riosa 2018-01-15 17:07 ` Mike Gilbert @ 2018-01-16 7:57 ` Michał Górny 2018-01-16 13:09 ` Francesco Riosa 1 sibling, 1 reply; 8+ messages in thread From: Michał Górny @ 2018-01-16 7:57 UTC (permalink / raw To: gentoo-dev W dniu pon, 15.01.2018 o godzinie 16∶27 +0100, użytkownik Francesco Riosa napisał: > In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added > to all python eclasses, it's useful for developers that want test and > mark the package for newer versions of python. > > However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not > usable if: > - the user want only python 2.7 and 3.6 (latest) installed > no python 3.5 > - don't want to mess dependencies (the warnings in the eclass are scary) Because it is not meant to be used for that purpose, and it is not meant to be used by users at all! It's just a quick hack for developer who wants to UNLIKELY(check if package works with implementation X) before he starts the effort on modifying PYTHON_COMPAT in ebuilds. > So, what can be done to let the user choose it's preferred python > version(s) without having to build it's own overlay? > > One solution is to change ebuilds in tree to include a user variable in > the PYTHON* arrays, example: > > -PYTHON_COMPAT=( python3_{4,5} ) > +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} ) > > if someone want to bet that packages are ok with 3.6/latest (even before > a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to > /etc/portage/make.conf and run egencache. > > Indeed biggest problem in changing $PYTHON* variables is that metadata > also change and cache is invalidated. > > However if the problem is known (*), and if the change to metadata is > stable per "system"/"gentoo repo clone", then the solution to the > problem is easy; just run $(egencache --update -j$(nproc) --repo gentoo) > after each sync. This won't work. You are wrongly presuming that egencache regenerates cache unconditionally. It does so only if either ebuild or eclass content changed. So it worked for you once because you modified ebuilds and/or eclasses. It won't work when you change PYTHON_COMPAT_ADD. I haven't checked the Portage details but it may see the metadata change when installing the package. Which means it would install package with unsatisfied dependencies (because it satisfied dependencies from cache), then store the final dependencies and TADAAM, you've got broken depgraph. > The most important thing is that this solution is scriptable and need no > human intervention. So is gpy-upgrade-impl. > > Notice also that it's easy to have an array with duplicate values > PYTHON_COMPAT=( python3_6 python3_6 ) but at a first glance > _python_set_impls() is resilient to this. > > (*) In a perfect world, where global variables that can change metadata > are known {egencache,emerge} can be made conscious and warn if the cache > is invalid, but that's out of scope at the moment. This isn't perfect world. This is the exact opposite of it, it would be a mess. Also, conscious tools would probably start plotting against you if you'd give them such horrible tasks. -- Best regards, Michał Górny ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} 2018-01-16 7:57 ` Michał Górny @ 2018-01-16 13:09 ` Francesco Riosa 2018-01-16 13:20 ` Michał Górny 0 siblings, 1 reply; 8+ messages in thread From: Francesco Riosa @ 2018-01-16 13:09 UTC (permalink / raw To: gentoo-dev, Michał Górny On 16/01/2018 08:57, Michał Górny wrote: > W dniu pon, 15.01.2018 o godzinie 16∶27 +0100, użytkownik Francesco > Riosa napisał: >> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added >> to all python eclasses, it's useful for developers that want test and >> mark the package for newer versions of python. >> >> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not >> usable if: >> - the user want only python 2.7 and 3.6 (latest) installed >> no python 3.5 >> - don't want to mess dependencies (the warnings in the eclass are scary) > Because it is not meant to be used for that purpose, and it is not meant > to be used by users at all! It's just a quick hack for developer who > wants to UNLIKELY(check if package works with implementation X) before > he starts the effort on modifying PYTHON_COMPAT in ebuilds. supposed that, but at this point I fail to see the benefit versus the added complexity in the eclasses, however that's not my business. > >> So, what can be done to let the user choose it's preferred python >> version(s) without having to build it's own overlay? >> >> One solution is to change ebuilds in tree to include a user variable in >> the PYTHON* arrays, example: >> >> -PYTHON_COMPAT=( python3_{4,5} ) >> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} ) >> >> if someone want to bet that packages are ok with 3.6/latest (even before >> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to >> /etc/portage/make.conf and run egencache. >> >> Indeed biggest problem in changing $PYTHON* variables is that metadata >> also change and cache is invalidated. >> >> However if the problem is known (*), and if the change to metadata is >> stable per "system"/"gentoo repo clone", then the solution to the >> problem is easy; just run $(egencache --update -j$(nproc) --repo gentoo) >> after each sync. > This won't work. You are wrongly presuming that egencache regenerates > cache unconditionally. It does so only if either ebuild or eclass > content changed. So it worked for you once because you modified ebuilds > and/or eclasses. It won't work when you change PYTHON_COMPAT_ADD. > > I haven't checked the Portage details but it may see the metadata change > when installing the package. Which means it would install package with > unsatisfied dependencies (because it satisfied dependencies from cache), > then store the final dependencies and TADAAM, you've got broken > depgraph. ouch, that basically kill the proposal, unless portage is modified to check known cache modifying variables, which isn't something I'd like to create. > >> The most important thing is that this solution is scriptable and need no >> human intervention. > So is gpy-upgrade-impl. It seem to do the job, one piece missing is something that monitor gentoo repository to see if it has better version (still w/o wanted python), an inotify for ebuilds. Suggestions? > >> Notice also that it's easy to have an array with duplicate values >> PYTHON_COMPAT=( python3_6 python3_6 ) but at a first glance >> _python_set_impls() is resilient to this. >> >> (*) In a perfect world, where global variables that can change metadata >> are known {egencache,emerge} can be made conscious and warn if the cache >> is invalid, but that's out of scope at the moment. > This isn't perfect world. This is the exact opposite of it, it would be > a mess. Also, conscious tools would probably start plotting against you > if you'd give them such horrible tasks. > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} 2018-01-16 13:09 ` Francesco Riosa @ 2018-01-16 13:20 ` Michał Górny 0 siblings, 0 replies; 8+ messages in thread From: Michał Górny @ 2018-01-16 13:20 UTC (permalink / raw To: gentoo-dev W dniu wto, 16.01.2018 o godzinie 14∶09 +0100, użytkownik Francesco Riosa napisał: > > On 16/01/2018 08:57, Michał Górny wrote: > > W dniu pon, 15.01.2018 o godzinie 16∶27 +0100, użytkownik Francesco > > Riosa napisał: > > > In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added > > > to all python eclasses, it's useful for developers that want test and > > > mark the package for newer versions of python. > > > > > > However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not > > > usable if: > > > - the user want only python 2.7 and 3.6 (latest) installed > > > no python 3.5 > > > - don't want to mess dependencies (the warnings in the eclass are scary) > > > > Because it is not meant to be used for that purpose, and it is not meant > > to be used by users at all! It's just a quick hack for developer who > > wants to UNLIKELY(check if package works with implementation X) before > > he starts the effort on modifying PYTHON_COMPAT in ebuilds. > > supposed that, but at this point I fail to see the benefit versus the > added complexity in the eclasses, however that's not my business. > > > > > So, what can be done to let the user choose it's preferred python > > > version(s) without having to build it's own overlay? > > > > > > One solution is to change ebuilds in tree to include a user variable in > > > the PYTHON* arrays, example: > > > > > > -PYTHON_COMPAT=( python3_{4,5} ) > > > +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} ) > > > > > > if someone want to bet that packages are ok with 3.6/latest (even before > > > a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to > > > /etc/portage/make.conf and run egencache. > > > > > > Indeed biggest problem in changing $PYTHON* variables is that metadata > > > also change and cache is invalidated. > > > > > > However if the problem is known (*), and if the change to metadata is > > > stable per "system"/"gentoo repo clone", then the solution to the > > > problem is easy; just run $(egencache --update -j$(nproc) --repo gentoo) > > > after each sync. > > > > This won't work. You are wrongly presuming that egencache regenerates > > cache unconditionally. It does so only if either ebuild or eclass > > content changed. So it worked for you once because you modified ebuilds > > and/or eclasses. It won't work when you change PYTHON_COMPAT_ADD. > > > > I haven't checked the Portage details but it may see the metadata change > > when installing the package. Which means it would install package with > > unsatisfied dependencies (because it satisfied dependencies from cache), > > then store the final dependencies and TADAAM, you've got broken > > depgraph. > > ouch, that basically kill the proposal, unless portage is modified to > check known cache modifying variables, which isn't something I'd like to > create. > > > > > The most important thing is that this solution is scriptable and need no > > > human intervention. > > > > So is gpy-upgrade-impl. > > It seem to do the job, one piece missing is something that monitor > gentoo repository to see if it has better version (still w/o wanted > python), an inotify for ebuilds. Suggestions? None. I think Alec's idea would work better for you. -- Best regards, Michał Górny ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-01-16 13:21 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-01-15 15:27 [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} Francesco Riosa
2018-01-15 17:07 ` Mike Gilbert
2018-01-16 0:05 ` Francesco Riosa
2018-01-16 0:40 ` Alec Warner
2018-01-16 13:16 ` Francesco Riosa
2018-01-16 7:57 ` Michał Górny
2018-01-16 13:09 ` Francesco Riosa
2018-01-16 13:20 ` Michał Górny
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox