* [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE}
@ 2018-01-15 15:27 Francesco Riosa
2018-01-15 17:07 ` Mike Gilbert
2018-01-16 7:57 ` Michał Górny
0 siblings, 2 replies; 8+ messages in thread
From: Francesco Riosa @ 2018-01-15 15:27 UTC (permalink / raw
To: gentoo development
In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added
to all python eclasses, it's useful for developers that want test and
mark the package for newer versions of python.
However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not
usable if:
- the user want only python 2.7 and 3.6 (latest) installed
no python 3.5
- don't want to mess dependencies (the warnings in the eclass are scary)
So, what can be done to let the user choose it's preferred python
version(s) without having to build it's own overlay?
One solution is to change ebuilds in tree to include a user variable in
the PYTHON* arrays, example:
-PYTHON_COMPAT=( python3_{4,5} )
+PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} )
if someone want to bet that packages are ok with 3.6/latest (even before
a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to
/etc/portage/make.conf and run egencache.
Indeed biggest problem in changing $PYTHON* variables is that metadata
also change and cache is invalidated.
However if the problem is known (*), and if the change to metadata is
stable per "system"/"gentoo repo clone", then the solution to the
problem is easy; just run $(egencache --update -j$(nproc) --repo gentoo)
after each sync.
The most important thing is that this solution is scriptable and need no
human intervention.
Notice also that it's easy to have an array with duplicate values
PYTHON_COMPAT=( python3_6 python3_6 ) but at a first glance
_python_set_impls() is resilient to this.
(*) In a perfect world, where global variables that can change metadata
are known {egencache,emerge} can be made conscious and warn if the cache
is invalid, but that's out of scope at the moment.
The procedure to find out which packages are enabled for python 3.5 but
not for python 3.6 is not totally straightforward but this should be the
list of ~amd64 packages interested:
app-admin/glance [15.0.0]
app-admin/setools [4.1.1]
app-arch/bloscpack [0.11.0]
app-arch/patool [1.12]
app-arch/vimball [0.5.1]
app-backup/attic [0.16]
app-backup/untangle-https-backup [0.0.7]
app-benchmarks/bootchart2 [0.14.7-r1]
app-emulation/libguestfs [1.36.5]
app-emulation/sen [0.5.1]
app-i18n/transifex-client [0.12.4]
app-misc/asciinema [1.4.0]
app-misc/gramps [4.2.5]
app-misc/icdiff [1.9.1]
app-misc/jira-cli [2.1.5]
app-misc/lirc [0.10.0_rc2]
app-misc/solaar [0.9.2-r3]
app-misc/terminal-colors [2.2]
app-mobilephone/obexftp [0.24-r1]
app-office/libreoffice-bin [5.4.2.2-r1]
app-pda/libimobiledevice [1.2.0]
app-portage/elogviewer [2.7]
app-portage/gentoolkit-dev [0.3.1]
app-portage/gs-elpa [0.1.3-r1]
app-portage/g-sorcery [0.2.1-r1]
app-portage/gs-pypi [0.2.1-r1]
app-portage/pqlop [0.02-r1]
app-text/grip [4.2.0]
app-text/gtranslator [2.91.7-r1]
app-text/landslide [1.1.3]
app-text/pelican [3.7.1]
app-text/pytextile [2.3.3]
app-text/sigil [0.9.8]
app-text/wiki2beamer [0.9.5-r1]
app-vim/jedi [0.8_p20171015]
app-vim/pyclewn [2.1-r1]
dev-games/vamos [0.8.2]
dev-lang/hy [0.13.1]
dev-libs/gom [0.3.2]
dev-libs/Ice [3.6.3-r1]
dev-libs/libcec [4.0.2-r1]
dev-libs/libsolv [0.6.22]
dev-libs/link-grammar [5.3.11]
dev-perl/Inline-Python [0.560.0]
dev-python/abstract_rendering [0.5.1]
dev-python/algopy [0.5.3]
dev-python/amqplib [1.0.2-r1]
dev-python/ansi2html [1.3.0]
dev-python/aodhclient [0.8.0]
dev-python/argcomplete [1.9.2]
dev-python/asdf [1.2.1]
dev-python/astlib [0.8.0]
dev-python/astrodendro [0.2.0]
dev-python/astroml [0.3]
dev-python/astroml-addons [0.2.2]
dev-python/astroscrappy [1.0.5]
dev-python/awscli [1.14.16]
dev-python/backports-abc [0.5]
dev-python/beaker [1.8.1]
dev-python/behave [1.2.5-r1]
dev-python/bibtexparser [0.6.2]
dev-python/breathe [4.6.0]
dev-python/castellan [0.12.2]
dev-python/ccdproc [1.2.0]
dev-python/cgkit [2.0.0-r1]
dev-python/cgroup-utils [0.6]
dev-python/chump [1.5.2]
dev-python/cliff [2.8.0]
dev-python/cliff-tablib [1.1]
dev-python/Coffin [2.0.1]
dev-python/construct [2.8.17]
dev-python/cursive [0.1.2]
dev-python/cvxopt [1.1.9]
dev-python/d2to1 [0.2.12_p1]
dev-python/demjson [2.2.4]
dev-python/django-compressor [1.5]
dev-python/django-sortedm2m [1.3.2]
dev-python/django-tables2 [1.1.2]
dev-python/dnslib [0.9.6]
dev-python/dogpile-core [0.4.1]
dev-python/ed25519ll [0.6]
dev-python/enzyme [0.4.1-r2]
dev-python/fedmsg [0.18.3]
dev-python/feedgenerator [1.9]
dev-python/flask-admin [1.5.0]
dev-python/flask-appconfig [0.11.1-r1]
dev-python/flask-babelex [0.9.3]
dev-python/flask-bootstrap [3.3.7.1]
dev-python/flask-debug [0.4.3]
dev-python/flask-mongoengine [0.9.3]
dev-python/flask-nav [0.6]
dev-python/flask-pymongo [0.4.1]
dev-python/flask-whooshalchemy [0.8]
dev-python/flipflop [1.0]
dev-python/formencode [1.3.0-r2]
dev-python/fuzzywuzzy [0.12.0]
dev-python/gcs-oauth2-boto-plugin [1.14]
dev-python/geoalchemy2 [0.3]
dev-python/glance_store [0.22.0]
dev-python/guessit [2.1.4]
dev-python/gwcs [0.7]
dev-python/hglib [2.1]
dev-python/hiredis [0.2.0]
dev-python/httreplay [0.2.0]
dev-python/husl [4.0.3]
dev-python/hvac [0.2.17]
dev-python/id3-py [1.2-r1]
dev-python/ImageHash [3.7]
dev-python/imageio [2.1.1]
dev-python/influxdb [2.12.0]
dev-python/intelhex [2.0]
dev-python/ipy [0.83]
dev-python/irc [15.0.6]
dev-python/iso3166 [0.8]
dev-python/iso_639 [0.4.5]
dev-python/jaraco-classes [1.4]
dev-python/jaraco-collections [1.5]
dev-python/jaraco-functools [1.15.1]
dev-python/jaraco-itertools [2.0]
dev-python/jaraco-logging [1.5]
dev-python/jaraco-stream [1.1.1]
dev-python/jaraco-text [1.8]
dev-python/jellyfish [0.5.6]
dev-python/jenkinsapi [0.2.29]
dev-python/jpype [0.6.2]
dev-python/jsonpickle [0.9.3]
dev-python/jupyter [1.0.0-r1]
dev-python/jupyter_console [5.1.0]
dev-python/keystonemiddleware [4.17.0]
dev-python/kitchen [1.2.4]
dev-python/Kivy [1.10.0]
dev-python/ldappool [2.1.0]
dev-python/libcloud [1.2.1]
dev-python/libvirt-python [3.10.0]
dev-python/libzilla [1.4]
dev-python/matplotlib2tikz [0.6.14]
dev-python/maybe [0.4.0]
dev-python/meld3 [1.0.2-r1]
dev-python/meteor-ejson [1.1.0]
dev-python/microversion-parse [0.1.4]
dev-python/mmh3 [2.3.1]
dev-python/moviepy [0.2.2]
dev-python/mysql-connector-python [2.1.4]
dev-python/nagiosplugin [1.2.4]
dev-python/nbdime [0.2.0]
dev-python/netcdf4-python [1.2.2]
dev-python/neutron-lib [1.9.1]
dev-python/nose2 [0.6.5]
dev-python/Numdifftools [0.9.17]
dev-python/odfpy [1.3.2]
dev-python/openstacksdk [0.9.17]
dev-python/os-brick [1.15.5]
dev-python/osc-lib [1.7.0]
dev-python/oslo-cache [1.25.1]
dev-python/oslo-context [2.17.1]
dev-python/oslo-db [4.25.1]
dev-python/oslo-log [3.30.2]
dev-python/oslo-messaging [5.30.2]
dev-python/oslo-middleware [3.30.1]
dev-python/oslo-privsep [1.22.1]
dev-python/oslo-reports [1.22.1]
dev-python/oslo-rootwrap [5.9.2]
dev-python/oslo-service [1.25.1]
dev-python/oslo-versionedobjects [1.26.1]
dev-python/oslo-vmware [2.23.1]
dev-python/osprofiler [1.11.0]
dev-python/os-testr [0.8.0]
dev-python/os-traits [0.3.3]
dev-python/os-vif [1.7.0]
dev-python/os-win [2.2.0]
dev-python/os-xenapi [0.2.0]
dev-python/ovs [2.7.2]
dev-python/ovsdbapp [0.4.0]
dev-python/owslib [0.16.0]
dev-python/pelican-minify [0.9]
dev-python/pid [2.1.1]
dev-python/pika [0.10.0]
dev-python/pika-pool [0.1.3]
dev-python/pilkit [2.0]
dev-python/pillowfight [0.2]
dev-python/plotly [1.9.6]
dev-python/prov [1.5.1]
dev-python/pycadf [2.6.0]
dev-python/pycanberra [0_pre20130515]
dev-python/pycipher [0.2-r1]
dev-python/PyDbLite [3.0.4]
dev-python/pydot-ng [1.0.0]
dev-python/pydotplus [2.0.2]
dev-python/pyds9 [1.8.1]
dev-python/pyee [1.0.2]
dev-python/pyfeyn [1.0.0]
dev-python/pyfits [3.4-r1]
dev-python/pyflann [1.9.1]
dev-python/pygal [2.1.1]
dev-python/pygccxml [1.7.5]
dev-python/pyghmi [1.0.22]
dev-python/pygpu [0.6.7]
dev-python/pygsl [2.1.1]
dev-python/pykwalify [1.5.2]
dev-python/pylibacl [0.5.0-r1]
dev-python/pyminuit [1.2.1-r1]
dev-python/pynzb [0.1.0-r1]
dev-python/pyopencl [2017.2]
dev-python/pypowervm [1.1.6]
dev-python/pyproj [1.9.5.1]
dev-python/pyrqlite [2.0]
dev-python/pyrtf [0.45-r2]
dev-python/pysaml2 [4.5.0]
dev-python/pyscaffold [2.4.4]
dev-python/pyshark [0.3.7.2]
dev-python/pytest-qt [2.3.0]
dev-python/pytest-raisesregexp [2.1]
dev-python/python-barbicanclient [4.5.2]
dev-python/python-ceilometerclient [2.6.2]
dev-python/python-cinderclient [3.1.0]
dev-python/python-ddp [0.1.5]
dev-python/python-designateclient [2.7.0]
dev-python/python-djvulibre [0.8]
dev-python/python-gammu [2.5]
dev-python/python-gnupg [0.4.0]
dev-python/python-heatclient [1.11.1]
dev-python/python-ironicclient [1.17.0]
dev-python/python-magnumclient [2.3.1]
dev-python/python-manilaclient [1.14.0]
dev-python/python-meteor [0.1.6]
dev-python/python-mistralclient [3.1.4]
dev-python/python-monascaclient [1.7.0]
dev-python/python-mpd [0.5.5]
dev-python/python-neutronclient [6.5.0]
dev-python/python-novaclient [9.1.1]
dev-python/python-openstackclient [3.12.0]
dev-python/python_orocos_kdl [1.3.1-r1]
dev-python/python-otrs [0.3.0]
dev-python/python-rethinkdb [2.3.0]
dev-python/python-saharaclient [1.1.0]
dev-python/python-scsi [0_pre160211]
dev-python/python-senlinclient [1.2.0]
dev-python/python-swiftclient [3.4.0]
dev-python/python-troveclient [2.5.0]
dev-python/python-zaqarclient [1.2.0]
dev-python/python-zunclient [0.4.1]
dev-python/pyuv [1.2.0]
dev-python/pyzor [1.0.0]
dev-python/rackspace-auth-openstack [1.3]
dev-python/rackspace-monitoring [0.8.0]
dev-python/rackspace-novaclient [2.1]
dev-python/raven [5.31.0]
dev-python/redis-py-cluster [1.3.4]
dev-python/rencode [1.0.5]
dev-python/requests-cache [0.4.12]
dev-python/retry-decorator [1.0.0]
dev-python/rnc2rng [2.5]
dev-python/rply [0.7.4]
dev-python/rtslib-fb [2.1.62]
dev-python/ryu [4.14]
dev-python/seaborn [0.7.1]
dev-python/semantic_version [2.6.0-r1]
dev-python/sepolgen [2.6]
dev-python/slackclient [1.1.0]
dev-python/sleekxmpp [1.3.1-r1]
dev-python/spectral-cube [0.4.0]
dev-python/specutils [0.2.2]
dev-python/sqlalchemy-rqlite [1.0]
dev-python/stormpath [2.4.5]
dev-python/structlog [16.1.0]
dev-python/stsci-distutils [0.3.7]
dev-python/stsci-sphinxext [1.2.1]
dev-python/tablib [0.11.2]
dev-python/tagpy [2013.1]
dev-python/taskflow [2.14.1]
dev-python/tempest-lib [1.0.0]
dev-python/tenacity [4.4.0]
dev-python/testfixtures [4.9.1]
dev-python/tinydb [3.1.3]
dev-python/tooz [1.58.0]
dev-python/torment [3.0.3]
dev-python/transmissionrpc [0.11]
dev-python/trollius [2.1]
dev-python/tvdb_api [1.10_pre20150406-r1]
dev-python/twitter [1.17.1]
dev-python/uncompyle6 [2.10.1]
dev-python/urdf_parser_py [0.3.3]
dev-python/utmp [0.4]
dev-python/vcstools [0.1.40]
dev-python/versiontools [1.9.1-r1]
dev-python/w3lib [1.18.0]
dev-python/webassets [0.12]
dev-python/ws4py [0.3.4]
dev-python/wsgiintercept [1.3.1]
dev-python/wtf-peewee [0.2.6]
dev-python/xonsh [0.4.7]
dev-python/yapsy [1.11.223-r1]
dev-python/yaql [1.1.3]
dev-python/zake [0.2.1]
dev-util/cppcheck [1.81]
dev-util/cram [0.7]
dev-util/devhelp [3.24.0]
dev-util/distro-info [0.14]
dev-util/dogtail [0.9.10]
dev-util/fatrace [0.12]
dev-util/gcovr [3.3]
dev-util/howdoi [1.1.9]
dev-util/rosinstall [0.7.8]
dev-util/spec-cleaner [0.8.9]
dev-util/wstool [0.1.17]
dev-vcs/git-cola [2.11]
dev-vcs/gitg [3.26.0]
games-misc/ponysay [3.0.2]
games-util/nml [0.4.4-r1]
gnome-base/gnome-shell [3.24.3]
gnome-extra/chrome-gnome-shell [9]
gnome-extra/cinnamon [3.6.6-r2]
gnome-extra/gnome-builder [3.24.2]
media-gfx/fontforge [20170731-r3]
media-gfx/sigal [1.3.0]
media-libs/pyliblo [0.10.0]
media-sound/gnome-music [3.24.2]
media-sound/pithos [1.3.1]
media-tv/tvnamer [2.4]
media-video/pitivi [0.98.1]
media-video/subliminal [2.0.5-r1]
net-analyzer/carl [0.9-r2]
net-analyzer/flent [1.0.1]
net-analyzer/rrdtool [1.7.0]
net-analyzer/speedtest-cli [1.0.2]
net-irc/irker [2.18]
net-irc/limnoria [20171025]
net-libs/libsignon-glib [1.14]
net-misc/electron-cash [3.0]
net-misc/electrum [3.0.5]
net-misc/electrum-ltc [3.0_pre20171218]
net-misc/gns3-gui [2.0.3-r1]
net-misc/gns3-server [2.0.3-r1]
net-misc/openvswitch [2.8.1]
net-misc/trackma [0.7.4]
net-misc/whatportis [0.6]
net-misc/you-get [0.4.536]
net-news/liferea [1.12_rc3-r1]
net-wireless/urh [1.6.4.1]
sci-astronomy/pyephem [3.7.6.0]
sci-biology/biopython [1.68]
sci-libs/clblas [2.10]
sci-libs/grib_api [1.14.5]
sci-libs/shogun [5.0.0]
sci-mathematics/dunshire [0.1.1]
sci-mathematics/pymc [2.3.6]
sci-mathematics/relational [2.5]
sci-visualization/fityk [1.3.1]
sys-apps/policycoreutils [2.7]
sys-apps/selinux-python [2.7]
sys-auth/keystone [12.0.0]
sys-block/targetcli-fb [2.1.45]
sys-cluster/cinder [11.0.2]
sys-cluster/heat [9.0.2]
sys-cluster/nova [16.0.4]
sys-cluster/sanlock [3.4.0-r1]
sys-fs/bcache-tools [1.0.8_p20140220-r1]
sys-kernel/kergen [0.1.4]
sys-libs/libsemanage [2.7]
sys-power/acpilight [1.0-r1]
www-apps/nikola [7.8.8]
www-apps/radicale [1.1.1]
www-misc/urlwatch [2.6]
x11-misc/menumaker [0.99.10]
x11-wm/qtile [0.10.6]
dev-db/pgadmin4 [2.1]
dev-libs/protobuf [3.5.1.1]
dev-python/flask-login [0.3.2-r1]
dev-python/flask-mail [0.9.1-r1]
dev-python/flask-migrate [2.1.1]
dev-python/flask-principal [0.4.0-r1]
dev-python/flask-security [1.7.5-r1]
dev-python/flask-uploads [0.2.0-r1]
dev-python/funcsigs [1.0.2-r1]
dev-python/translate-toolkit [2.0.0]
kde-apps/lokalize [17.12.1]
media-gfx/hugin [2017.0]
media-libs/gexiv2 [0.10.6]
net-irc/hexchat [2.12.4-r2]
sys-fs/zfs [0.7.5-r1]
sys-process/glances [2.11.1]
dev-python/batinfo [0.3]
dev-python/flask-babel [0.11.2]
dev-python/flask-script [2.0.6]
dev-python/flask-testing [0.6.2]
dev-python/flask-wtf [0.14.2]
media-libs/opencv [3.3.0-r4]
dev-python/ip-associations-python-novaclient-ext [0.2]
dev-python/os-diskconfig-python-novaclient-ext [0.1.3]
dev-python/os-networksv2-python-novaclient-ext [0.26]
dev-python/os-virtual-interfacesv2-python-novaclient-ext [0.20]
dev-python/rax-default-network-flags-python-novaclient-ext [0.4.0]
dev-python/rax-scheduled-images-python-novaclient-ext [0.3.1]
net-analyzer/nagios-check_openvpn-simple [0.0.1-r1]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE}
2018-01-15 15:27 [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} Francesco Riosa
@ 2018-01-15 17:07 ` Mike Gilbert
2018-01-16 0:05 ` Francesco Riosa
2018-01-16 7:57 ` Michał Górny
1 sibling, 1 reply; 8+ messages in thread
From: Mike Gilbert @ 2018-01-15 17:07 UTC (permalink / raw
To: Gentoo Dev
On Mon, Jan 15, 2018 at 10:27 AM, Francesco Riosa <vivo75@gmail.com> wrote:
>
> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added
> to all python eclasses, it's useful for developers that want test and
> mark the package for newer versions of python.
>
> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not
> usable if:
> - the user want only python 2.7 and 3.6 (latest) installed
> no python 3.5
> - don't want to mess dependencies (the warnings in the eclass are scary)
>
> So, what can be done to let the user choose it's preferred python
> version(s) without having to build it's own overlay?
>
> One solution is to change ebuilds in tree to include a user variable in
> the PYTHON* arrays, example:
>
> -PYTHON_COMPAT=( python3_{4,5} )
> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} )
>
> if someone want to bet that packages are ok with 3.6/latest (even before
> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to
> /etc/portage/make.conf and run egencache.
I like the idea to inject values into PYTHON_COMPAT instead of
overriding it completely. I'm pretty sure this can/should be
implemented in the eclass without touching ebuilds.
I'm not sure I really like the idea of affecting the other metadata
variables. I can see your point about wanting to remove python
versions that would otherwise satisfy dependencies. If metadata is
modified this way, it would definitely be "unsupported".
As far as implementation, you would probably need to write the patches for this.
Also, I expect the QA team might have some objections, so you may want
to discuss it with them (especially mgorny) before spending too much
time on it.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE}
2018-01-15 17:07 ` Mike Gilbert
@ 2018-01-16 0:05 ` Francesco Riosa
2018-01-16 0:40 ` Alec Warner
0 siblings, 1 reply; 8+ messages in thread
From: Francesco Riosa @ 2018-01-16 0:05 UTC (permalink / raw
To: gentoo-dev, Mike Gilbert
On 15/01/2018 18:07, Mike Gilbert wrote:
> On Mon, Jan 15, 2018 at 10:27 AM, Francesco Riosa <vivo75@gmail.com> wrote:
>> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added
>> to all python eclasses, it's useful for developers that want test and
>> mark the package for newer versions of python.
>>
>> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not
>> usable if:
>> - the user want only python 2.7 and 3.6 (latest) installed
>> no python 3.5
>> - don't want to mess dependencies (the warnings in the eclass are scary)
>>
>> So, what can be done to let the user choose it's preferred python
>> version(s) without having to build it's own overlay?
>>
>> One solution is to change ebuilds in tree to include a user variable in
>> the PYTHON* arrays, example:
>>
>> -PYTHON_COMPAT=( python3_{4,5} )
>> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} )
>>
>> if someone want to bet that packages are ok with 3.6/latest (even before
>> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to
>> /etc/portage/make.conf and run egencache.
> I like the idea to inject values into PYTHON_COMPAT instead of
> overriding it completely. I'm pretty sure this can/should be
> implemented in the eclass without touching ebuilds.
In my mind that was to leave ebuilds developers the ability to opt out,
but well that could also be done in the eclasses.
Opt out could be beneficial for packages that only support python 2.7,
or where there are known and documented problems with different python
versions.
>
> I'm not sure I really like the idea of affecting the other metadata
> variables. I can see your point about wanting to remove python
> versions that would otherwise satisfy dependencies. If metadata is
> modified this way, it would definitely be "unsupported".
I've not tought about remove python versions, only add them (beneficial
for users that like to use experimental python versions)
However the supported python version are translated and build important
cached variables IUSE, DEPEND, etc. so there is no way to cleanly modify
those variable after the cache has been generated.
The only viable option is to regenerate, update or delete it.
>
> As far as implementation, you would probably need to write the patches for this.
If there is consensus that's not a problem, cannot guarantee to be fast
however
>
> Also, I expect the QA team might have some objections, so you may want
> to discuss it with them (especially mgorny) before spending too much
> time on it.
I'd like to hear mgorny opinions but that should be a more extended
decision than only QA and the python eclasses developer(s).
If nothing else because deciding that sometimes may be good to let the
user break the cache is a global decision and documentation need to be
added.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE}
2018-01-16 0:05 ` Francesco Riosa
@ 2018-01-16 0:40 ` Alec Warner
2018-01-16 13:16 ` Francesco Riosa
0 siblings, 1 reply; 8+ messages in thread
From: Alec Warner @ 2018-01-16 0:40 UTC (permalink / raw
To: Gentoo Dev; +Cc: Mike Gilbert
[-- Attachment #1: Type: text/plain, Size: 3617 bytes --]
On Mon, Jan 15, 2018 at 7:05 PM, Francesco Riosa <vivo75@gmail.com> wrote:
>
>
> On 15/01/2018 18:07, Mike Gilbert wrote:
> > On Mon, Jan 15, 2018 at 10:27 AM, Francesco Riosa <vivo75@gmail.com>
> wrote:
> >> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added
> >> to all python eclasses, it's useful for developers that want test and
> >> mark the package for newer versions of python.
> >>
> >> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not
> >> usable if:
> >> - the user want only python 2.7 and 3.6 (latest) installed
> >> no python 3.5
> >> - don't want to mess dependencies (the warnings in the eclass are scary)
> >>
> >> So, what can be done to let the user choose it's preferred python
> >> version(s) without having to build it's own overlay?
> >>
> >> One solution is to change ebuilds in tree to include a user variable in
> >> the PYTHON* arrays, example:
> >>
> >> -PYTHON_COMPAT=( python3_{4,5} )
> >> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} )
> >>
> >> if someone want to bet that packages are ok with 3.6/latest (even before
> >> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to
> >> /etc/portage/make.conf and run egencache.
> > I like the idea to inject values into PYTHON_COMPAT instead of
> > overriding it completely. I'm pretty sure this can/should be
> > implemented in the eclass without touching ebuilds.
> In my mind that was to leave ebuilds developers the ability to opt out,
> but well that could also be done in the eclasses.
> Opt out could be beneficial for packages that only support python 2.7,
> or where there are known and documented problems with different python
> versions.
> >
> > I'm not sure I really like the idea of affecting the other metadata
> > variables. I can see your point about wanting to remove python
> > versions that would otherwise satisfy dependencies. If metadata is
> > modified this way, it would definitely be "unsupported".
> I've not tought about remove python versions, only add them (beneficial
> for users that like to use experimental python versions)
> However the supported python version are translated and build important
> cached variables IUSE, DEPEND, etc. so there is no way to cleanly modify
> those variable after the cache has been generated.
> The only viable option is to regenerate, update or delete it.
>
> >
> > As far as implementation, you would probably need to write the patches
> for this.
> If there is consensus that's not a problem, cannot guarantee to be fast
> however
> >
> > Also, I expect the QA team might have some objections, so you may want
> > to discuss it with them (especially mgorny) before spending too much
> > time on it.
> I'd like to hear mgorny opinions but that should be a more extended
> decision than only QA and the python eclasses developer(s).
> If nothing else because deciding that sometimes may be good to let the
> user break the cache is a global decision and documentation need to be
> added.
>
I'm seeing less value in this being a 'cache-breaking' exercise and more
value in it simply being a custom eclass.
If you hoist the eclass into an overlay and modify it (e.g. to set the var
and get the deps you want) the caching
all works out fine.
1. The source of the data is the hoisted eclass.
2. The eclass mtime changed.
3. package managers can see that and update cache metadata for inheriting
ebuilds.
4. Bonus, once the cache is generated we have a valid means to see if the
cache remains valid.
5. Double bonus, any ebuilds not inheriting the eclass are unaffected.
I'm not sure why this is so onerous.
-A
[-- Attachment #2: Type: text/html, Size: 4646 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE}
2018-01-15 15:27 [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} Francesco Riosa
2018-01-15 17:07 ` Mike Gilbert
@ 2018-01-16 7:57 ` Michał Górny
2018-01-16 13:09 ` Francesco Riosa
1 sibling, 1 reply; 8+ messages in thread
From: Michał Górny @ 2018-01-16 7:57 UTC (permalink / raw
To: gentoo-dev
W dniu pon, 15.01.2018 o godzinie 16∶27 +0100, użytkownik Francesco
Riosa napisał:
> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added
> to all python eclasses, it's useful for developers that want test and
> mark the package for newer versions of python.
>
> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not
> usable if:
> - the user want only python 2.7 and 3.6 (latest) installed
> no python 3.5
> - don't want to mess dependencies (the warnings in the eclass are scary)
Because it is not meant to be used for that purpose, and it is not meant
to be used by users at all! It's just a quick hack for developer who
wants to UNLIKELY(check if package works with implementation X) before
he starts the effort on modifying PYTHON_COMPAT in ebuilds.
> So, what can be done to let the user choose it's preferred python
> version(s) without having to build it's own overlay?
>
> One solution is to change ebuilds in tree to include a user variable in
> the PYTHON* arrays, example:
>
> -PYTHON_COMPAT=( python3_{4,5} )
> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} )
>
> if someone want to bet that packages are ok with 3.6/latest (even before
> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to
> /etc/portage/make.conf and run egencache.
>
> Indeed biggest problem in changing $PYTHON* variables is that metadata
> also change and cache is invalidated.
>
> However if the problem is known (*), and if the change to metadata is
> stable per "system"/"gentoo repo clone", then the solution to the
> problem is easy; just run $(egencache --update -j$(nproc) --repo gentoo)
> after each sync.
This won't work. You are wrongly presuming that egencache regenerates
cache unconditionally. It does so only if either ebuild or eclass
content changed. So it worked for you once because you modified ebuilds
and/or eclasses. It won't work when you change PYTHON_COMPAT_ADD.
I haven't checked the Portage details but it may see the metadata change
when installing the package. Which means it would install package with
unsatisfied dependencies (because it satisfied dependencies from cache),
then store the final dependencies and TADAAM, you've got broken
depgraph.
> The most important thing is that this solution is scriptable and need no
> human intervention.
So is gpy-upgrade-impl.
>
> Notice also that it's easy to have an array with duplicate values
> PYTHON_COMPAT=( python3_6 python3_6 ) but at a first glance
> _python_set_impls() is resilient to this.
>
> (*) In a perfect world, where global variables that can change metadata
> are known {egencache,emerge} can be made conscious and warn if the cache
> is invalid, but that's out of scope at the moment.
This isn't perfect world. This is the exact opposite of it, it would be
a mess. Also, conscious tools would probably start plotting against you
if you'd give them such horrible tasks.
--
Best regards,
Michał Górny
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE}
2018-01-16 7:57 ` Michał Górny
@ 2018-01-16 13:09 ` Francesco Riosa
2018-01-16 13:20 ` Michał Górny
0 siblings, 1 reply; 8+ messages in thread
From: Francesco Riosa @ 2018-01-16 13:09 UTC (permalink / raw
To: gentoo-dev, Michał Górny
On 16/01/2018 08:57, Michał Górny wrote:
> W dniu pon, 15.01.2018 o godzinie 16∶27 +0100, użytkownik Francesco
> Riosa napisał:
>> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added
>> to all python eclasses, it's useful for developers that want test and
>> mark the package for newer versions of python.
>>
>> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not
>> usable if:
>> - the user want only python 2.7 and 3.6 (latest) installed
>> no python 3.5
>> - don't want to mess dependencies (the warnings in the eclass are scary)
> Because it is not meant to be used for that purpose, and it is not meant
> to be used by users at all! It's just a quick hack for developer who
> wants to UNLIKELY(check if package works with implementation X) before
> he starts the effort on modifying PYTHON_COMPAT in ebuilds.
supposed that, but at this point I fail to see the benefit versus the
added complexity in the eclasses, however that's not my business.
>
>> So, what can be done to let the user choose it's preferred python
>> version(s) without having to build it's own overlay?
>>
>> One solution is to change ebuilds in tree to include a user variable in
>> the PYTHON* arrays, example:
>>
>> -PYTHON_COMPAT=( python3_{4,5} )
>> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} )
>>
>> if someone want to bet that packages are ok with 3.6/latest (even before
>> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to
>> /etc/portage/make.conf and run egencache.
>>
>> Indeed biggest problem in changing $PYTHON* variables is that metadata
>> also change and cache is invalidated.
>>
>> However if the problem is known (*), and if the change to metadata is
>> stable per "system"/"gentoo repo clone", then the solution to the
>> problem is easy; just run $(egencache --update -j$(nproc) --repo gentoo)
>> after each sync.
> This won't work. You are wrongly presuming that egencache regenerates
> cache unconditionally. It does so only if either ebuild or eclass
> content changed. So it worked for you once because you modified ebuilds
> and/or eclasses. It won't work when you change PYTHON_COMPAT_ADD.
>
> I haven't checked the Portage details but it may see the metadata change
> when installing the package. Which means it would install package with
> unsatisfied dependencies (because it satisfied dependencies from cache),
> then store the final dependencies and TADAAM, you've got broken
> depgraph.
ouch, that basically kill the proposal, unless portage is modified to
check known cache modifying variables, which isn't something I'd like to
create.
>
>> The most important thing is that this solution is scriptable and need no
>> human intervention.
> So is gpy-upgrade-impl.
It seem to do the job, one piece missing is something that monitor
gentoo repository to see if it has better version (still w/o wanted
python), an inotify for ebuilds. Suggestions?
>
>> Notice also that it's easy to have an array with duplicate values
>> PYTHON_COMPAT=( python3_6 python3_6 ) but at a first glance
>> _python_set_impls() is resilient to this.
>>
>> (*) In a perfect world, where global variables that can change metadata
>> are known {egencache,emerge} can be made conscious and warn if the cache
>> is invalid, but that's out of scope at the moment.
> This isn't perfect world. This is the exact opposite of it, it would be
> a mess. Also, conscious tools would probably start plotting against you
> if you'd give them such horrible tasks.
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE}
2018-01-16 0:40 ` Alec Warner
@ 2018-01-16 13:16 ` Francesco Riosa
0 siblings, 0 replies; 8+ messages in thread
From: Francesco Riosa @ 2018-01-16 13:16 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 4454 bytes --]
On 16/01/2018 01:40, Alec Warner wrote:
>
>
> On Mon, Jan 15, 2018 at 7:05 PM, Francesco Riosa <vivo75@gmail.com
> <mailto:vivo75@gmail.com>> wrote:
>
>
>
> On 15/01/2018 18:07, Mike Gilbert wrote:
> > On Mon, Jan 15, 2018 at 10:27 AM, Francesco Riosa
> <vivo75@gmail.com <mailto:vivo75@gmail.com>> wrote:
> >> In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized
> and added
> >> to all python eclasses, it's useful for developers that want
> test and
> >> mark the package for newer versions of python.
> >>
> >> However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE
> is not
> >> usable if:
> >> - the user want only python 2.7 and 3.6 (latest) installed
> >> no python 3.5
> >> - don't want to mess dependencies (the warnings in the eclass
> are scary)
> >>
> >> So, what can be done to let the user choose it's preferred python
> >> version(s) without having to build it's own overlay?
> >>
> >> One solution is to change ebuilds in tree to include a user
> variable in
> >> the PYTHON* arrays, example:
> >>
> >> -PYTHON_COMPAT=( python3_{4,5} )
> >> +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} )
> >>
> >> if someone want to bet that packages are ok with 3.6/latest
> (even before
> >> a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to
> >> /etc/portage/make.conf and run egencache.
> > I like the idea to inject values into PYTHON_COMPAT instead of
> > overriding it completely. I'm pretty sure this can/should be
> > implemented in the eclass without touching ebuilds.
> In my mind that was to leave ebuilds developers the ability to opt
> out,
> but well that could also be done in the eclasses.
> Opt out could be beneficial for packages that only support python 2.7,
> or where there are known and documented problems with different python
> versions.
> >
> > I'm not sure I really like the idea of affecting the other metadata
> > variables. I can see your point about wanting to remove python
> > versions that would otherwise satisfy dependencies. If metadata is
> > modified this way, it would definitely be "unsupported".
> I've not tought about remove python versions, only add them
> (beneficial
> for users that like to use experimental python versions)
> However the supported python version are translated and build
> important
> cached variables IUSE, DEPEND, etc. so there is no way to cleanly
> modify
> those variable after the cache has been generated.
> The only viable option is to regenerate, update or delete it.
>
> >
> > As far as implementation, you would probably need to write the
> patches for this.
> If there is consensus that's not a problem, cannot guarantee to be
> fast
> however
> >
> > Also, I expect the QA team might have some objections, so you
> may want
> > to discuss it with them (especially mgorny) before spending too much
> > time on it.
> I'd like to hear mgorny opinions but that should be a more extended
> decision than only QA and the python eclasses developer(s).
> If nothing else because deciding that sometimes may be good to let the
> user break the cache is a global decision and documentation need to be
> added.
>
>
> I'm seeing less value in this being a 'cache-breaking' exercise and
> more value in it simply being a custom eclass.
>
> If you hoist the eclass into an overlay and modify it (e.g. to set the
> var and get the deps you want) the caching
> all works out fine.
>
> 1. The source of the data is the hoisted eclass.
> 2. The eclass mtime changed.
> 3. package managers can see that and update cache metadata for
> inheriting ebuilds.
> 4. Bonus, once the cache is generated we have a valid means to see if
> the cache remains valid.
> 5. Double bonus, any ebuilds not inheriting the eclass are unaffected.
>
> I'm not sure why this is so onerous.
>
> -A
That's a good idea, in the past I've failed to completely understand how
portage inherit eclasses in overlays, but if I can figure that out
probably this could be a solution.
Also depending from the magnitude of changes the four python eclasses
need and the correlated maintenance burden.
[-- Attachment #2: Type: text/html, Size: 7199 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE}
2018-01-16 13:09 ` Francesco Riosa
@ 2018-01-16 13:20 ` Michał Górny
0 siblings, 0 replies; 8+ messages in thread
From: Michał Górny @ 2018-01-16 13:20 UTC (permalink / raw
To: gentoo-dev
W dniu wto, 16.01.2018 o godzinie 14∶09 +0100, użytkownik Francesco
Riosa napisał:
>
> On 16/01/2018 08:57, Michał Górny wrote:
> > W dniu pon, 15.01.2018 o godzinie 16∶27 +0100, użytkownik Francesco
> > Riosa napisał:
> > > In late 2015 ${PYTHON_COMPAT_OVERRIDE} has been standardized and added
> > > to all python eclasses, it's useful for developers that want test and
> > > mark the package for newer versions of python.
> > >
> > > However (unless I'm missing something) PYTHON_COMPAT_OVERRIDE is not
> > > usable if:
> > > - the user want only python 2.7 and 3.6 (latest) installed
> > > no python 3.5
> > > - don't want to mess dependencies (the warnings in the eclass are scary)
> >
> > Because it is not meant to be used for that purpose, and it is not meant
> > to be used by users at all! It's just a quick hack for developer who
> > wants to UNLIKELY(check if package works with implementation X) before
> > he starts the effort on modifying PYTHON_COMPAT in ebuilds.
>
> supposed that, but at this point I fail to see the benefit versus the
> added complexity in the eclasses, however that's not my business.
> >
> > > So, what can be done to let the user choose it's preferred python
> > > version(s) without having to build it's own overlay?
> > >
> > > One solution is to change ebuilds in tree to include a user variable in
> > > the PYTHON* arrays, example:
> > >
> > > -PYTHON_COMPAT=( python3_{4,5} )
> > > +PYTHON_COMPAT=( python3_{4,5} ${PYTHON_COMPAT_ADD} )
> > >
> > > if someone want to bet that packages are ok with 3.6/latest (even before
> > > a developer tested it) then add PYTHON_COMPAT_ADD=python3_6 to
> > > /etc/portage/make.conf and run egencache.
> > >
> > > Indeed biggest problem in changing $PYTHON* variables is that metadata
> > > also change and cache is invalidated.
> > >
> > > However if the problem is known (*), and if the change to metadata is
> > > stable per "system"/"gentoo repo clone", then the solution to the
> > > problem is easy; just run $(egencache --update -j$(nproc) --repo gentoo)
> > > after each sync.
> >
> > This won't work. You are wrongly presuming that egencache regenerates
> > cache unconditionally. It does so only if either ebuild or eclass
> > content changed. So it worked for you once because you modified ebuilds
> > and/or eclasses. It won't work when you change PYTHON_COMPAT_ADD.
> >
> > I haven't checked the Portage details but it may see the metadata change
> > when installing the package. Which means it would install package with
> > unsatisfied dependencies (because it satisfied dependencies from cache),
> > then store the final dependencies and TADAAM, you've got broken
> > depgraph.
>
> ouch, that basically kill the proposal, unless portage is modified to
> check known cache modifying variables, which isn't something I'd like to
> create.
> >
> > > The most important thing is that this solution is scriptable and need no
> > > human intervention.
> >
> > So is gpy-upgrade-impl.
>
> It seem to do the job, one piece missing is something that monitor
> gentoo repository to see if it has better version (still w/o wanted
> python), an inotify for ebuilds. Suggestions?
None. I think Alec's idea would work better for you.
--
Best regards,
Michał Górny
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-01-16 13:21 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-01-15 15:27 [gentoo-dev] RFC: ${PYTHON_COMPAT_ADD} in addition to ${PYTHON_COMPAT_OVERRIDE} Francesco Riosa
2018-01-15 17:07 ` Mike Gilbert
2018-01-16 0:05 ` Francesco Riosa
2018-01-16 0:40 ` Alec Warner
2018-01-16 13:16 ` Francesco Riosa
2018-01-16 7:57 ` Michał Górny
2018-01-16 13:09 ` Francesco Riosa
2018-01-16 13:20 ` Michał Górny
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox