[gentoo-catalyst] Python 3 support

[gentoo-catalyst] Python 3 support

[Matthew Ruffalo]

Hi all-

First, thanks for the great tool. I've been using Gentoo for about 10 
years, and I use Catalyst to build custom Gentoo live CDs that contain 
extra software (e.g. netcat6, rsync, ddrescue, etc.).

At some point in the past the stage3 tarballs only contained Python 3, 
though it looks like the 20130516 amd64 stage3 contains both 2.7 and 
3.2. It's reasonable to assume that any current Gentoo system has a 
Python 3.2+ interpreter, so is there any objection to making Catalyst 
work on Python 3? The fewer things in the world that depend on Python 2, 
the better (in my opinion). Portage has supported Python 3 for some time 
and it makes sense for Catalyst to follow suit.

I use Python 3 for all of my own work and have a reasonable amount of 
experience porting projects to Python 3. This typically involves a lot 
of code churn even when there aren't any logic changes -- every print 
statement has to be converted to calling the print function, for 
instance. Explicit conversion between bytes and str can require a bit 
more thought but isn't difficult.

This conversion might require other miscellaneous cleanup, including 
some whitespace-only commits. Python 3 does not allow mixing tabs and 
spaces for indentation, and (e.g.) catalyst_support.py is a *mess* with 
indentation. PyCharm also notices things like unnecessary "\" line 
continuation characters, unused variables, mutable default arguments 
(e.g. catalyst_support.spawn's "env"), equality comparisons against 
None, and unreachable code (lines 183 - 220 of catalyst_lock.py).

It looks like the project has mostly standardized on using tabs for 
indentation, so that's what I'll use when changing indentation is necessary.

In any case, over the next few weeks/months I'll prepare a series of 
patches in decreasing order of priority:

 1. Changes required to run 2to3, like changing string exceptions (e.g.
    catalyst_lock.py:34) to 'raise Exception("message")'
 2. Automated 2to3 conversion
 3. Minimal indentation changes required for Python 3 to parse the code
    (as I mentioned, the diff of catalyst_support.py will still be quite
    large)
 4. Any required bytes/str fixes and other things that 2to3 isn't smart
    enough to do on its own
 5. Miscellaneous style tweaks (some of which are suggested by PEP 8), like:
     1. using os.path.join where appropriate (e.g. grp_target.py:38,
        catalyst_lock.py:80)
     2. using context managers ("with open(filename) as f:")
     3. replacing mutable default function arguments
     4. using argparse instead of getopt in the main catalyst script (maybe)
     5. catching Exception instead of using a raw 'except:' -- probably
        don't want to catch SystemExit, StopIteration and
        KeyboardInterrupt in a lot of these places
 6. Other PEP 8-inspired changes like using spaces around operators and
    function arguments

The intention is that categories 1 through 4 would be the minimum set of 
changes required to make Catalyst run on Python 3. Categories 5 and 6 
would be a mix of PEP 8 and newer Python idioms and applying these would 
be optional.

I'm not a Gentoo developer and I'll be making a lot of small commits, so 
sharing my changes will probably be easiest by publishing a clone of the 
Catalyst repository on GitHub. (I plan to make/test/release all of these 
changes whether or not they'll make it in to Catalyst in the near 
future. Even if the Gentoo releng team doesn't care about supporting 
Python 3 now, having a working port available might make it easier to 
justify that change in the future :) )

Thanks for your time and any feedback,
MMR...
-- 
Matthew Ruffalo
Department of Electrical Engineering and Computer Science
Case Western Reserve University

Re: [gentoo-catalyst] Python 3 support

[Brian Dolbec]

On Mon, 2013-05-20 at 17:06 -0400, Matthew Ruffalo wrote:
> Hi all-
> 
> First, thanks for the great tool. I've been using Gentoo for about 10 
> years, and I use Catalyst to build custom Gentoo live CDs that contain 
> extra software (e.g. netcat6, rsync, ddrescue, etc.).
> 

Yeah, py3 support is on our list of things todo.  We would love to get
help porting. BUT, things are rather slow at getting done at the moment.
I have started a rewrite branch that I am told is what they intend to
change the development branch to.  It has been restructured with
numerous other fixes done to clean up the code.  There are still a ton
more things to do.

There are a few of us waiting for the change to happen before doing a
bunch of work that may need rebasing because of other changes done to
the current master.  I had that happen once where I did a big re-write
(another project) and it took nearly 4 years to get merged.  I had to
rebase numerous times as well as port changes to the new code many
times. :(  I also had to migrate the VCS in the midst of it.

Personally I think it may be better to wait a bit on the py3
compatibility changes.  Many of the potential problem spots will need
fixing anyway, so conversion would likely be smoother after some more
re-write work is done.


I hadn't heard of pycharm, I have been using pyflakes and pylint.
I'll check it out.


you can get my rewite branch so far at
http://dev.gentoo.org/~dolsen/catalyst/

When you clone it, it will compain about not being able to check out
master.  You will need to "git checkout re-write".  But I would just add
my url as another remote to your current clone and fetch it.

I have also made changes for the code to be easily run from the
checkout.  Edit the testpath file then in the terminal you wish to run
catalyst from:

source ./tesptath


> Thanks for your time and any feedback,
> MMR...

Re: [gentoo-catalyst] Python 3 support

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On Mon, May 20, 2013 at 03:45:59PM -0700, Brian Dolbec wrote:
> BUT, things are rather slow at getting done at the moment.

jmbsvicetto was talking a while back (last month?) about cutting a new
release.  Any word on that front?

> Personally I think it may be better to wait a bit on the py3
> compatibility changes.  Many of the potential problem spots will
> need fixing anyway, so conversion would likely be smoother after
> some more re-write work is done.

This brings the total number of people chomping at the bit to refactor
catalyst code up to three‽  There must be something we can do to help
unclog the maintenance pipe?

> you can get my rewite branch so far at
> http://dev.gentoo.org/~dolsen/catalyst/
> 
> When you clone it, it will compain about not being able to check out
> master.  You will need to "git checkout re-write".  But I would just add
> my url as another remote to your current clone and fetch it.

Sounds like a job for:

  git clone --branch rewrite http://dev.gentoo.org/~dolsen/catalyst/

(your branch name doesn't actually have a hyphen ;).  FWIW, you can
also take a look at my stuff [1].

Cheers,
Trevor

[1]: http://git.tremily.us/?p=catalyst.git
     Branch summaries in, for example:
       http://thread.gmane.org/gmane.linux.gentoo.catalyst/2099/focus=2221

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Python 3 support

[Dmitry Selyutin]

Hello everyone!

Since I'm going to reimplement catalyst in GSoC, I'd like to tell my
opinion if it has some weight. :-)
Some years have passed since Python 3 was created, and nowadays I
think it is stable enough to select it. I'd rather thought about
Python 2 compatibility than Python 3, since it seems to become a
standart soon. To cut a long story short, I'd rather oriented to
Python 3 than Python 2, though I prefer to use Python 2 nowadays. Of
course compatibility will be one of main aims, but I see some benefit
if we will choose Python 3 rather than Python 2, though users may
successfully use catalyst with Python 2 in the future. It's just a
proposal, so I'd like to hear your opinions. If you don't agree, we
may rather think about Python 2.
Even more, I'd like to avoid some generators and provide this support
manually: I've always hated generators, especially code generators
(and GUI ones).

What do you think about it?

2013/5/21, Brian Dolbec <dolsen@gentoo.org>:
> On Mon, 2013-05-20 at 17:06 -0400, Matthew Ruffalo wrote:
>> Hi all-
>>
>> First, thanks for the great tool. I've been using Gentoo for about 10
>> years, and I use Catalyst to build custom Gentoo live CDs that contain
>> extra software (e.g. netcat6, rsync, ddrescue, etc.).
>>
>
> Yeah, py3 support is on our list of things todo.  We would love to get
> help porting. BUT, things are rather slow at getting done at the moment.
> I have started a rewrite branch that I am told is what they intend to
> change the development branch to.  It has been restructured with
> numerous other fixes done to clean up the code.  There are still a ton
> more things to do.
>
> There are a few of us waiting for the change to happen before doing a
> bunch of work that may need rebasing because of other changes done to
> the current master.  I had that happen once where I did a big re-write
> (another project) and it took nearly 4 years to get merged.  I had to
> rebase numerous times as well as port changes to the new code many
> times. :(  I also had to migrate the VCS in the midst of it.
>
> Personally I think it may be better to wait a bit on the py3
> compatibility changes.  Many of the potential problem spots will need
> fixing anyway, so conversion would likely be smoother after some more
> re-write work is done.
>
>
> I hadn't heard of pycharm, I have been using pyflakes and pylint.
> I'll check it out.
>
>
> you can get my rewite branch so far at
> http://dev.gentoo.org/~dolsen/catalyst/
>
> When you clone it, it will compain about not being able to check out
> master.  You will need to "git checkout re-write".  But I would just add
> my url as another remote to your current clone and fetch it.
>
> I have also made changes for the code to be easily run from the
> checkout.  Edit the testpath file then in the terminal you wish to run
> catalyst from:
>
> source ./tesptath
>
>
>> Thanks for your time and any feedback,
>> MMR...
>
>
>
>


-- 
With best regards,
Dmitry Selyutin

E-mail: ghostmansd@gmail.com
Phone: +7(985)334-07-70

Re: [gentoo-catalyst] Python 3 support

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1398 bytes --]

On Tue, May 21, 2013 at 12:33:28PM +0200, Dmitry Selyutin wrote:
> Even more, I'd like to avoid some generators and provide this
> support manually: I've always hated generators, especially code
> generators (and GUI ones).

Generators?  Are you referring to 2to3?  I think anyone converting a
Python project from Python 2 to Python 3 would be cray not to use it.
They'd also be crazy if they just assumed that it generated optimal
code ;).  It's best to run it, and then go double check to make sure
all the changes are meaningful.  Keep stuff like [1]:

  -                       raise CatalystError, "Could not open file " + filename
  +                       raise CatalystError("Could not open file " + filename)

but drop stuff like:

-               for x in self.values.keys():
+               for x in list(self.values.keys()):

Cheers,
Trevor

[1]: Although see 305cde6 (Massive pyflakes import cleanup and broken
     CatalystError calls, 2013-01-19) in dolsen's branch.  There will
     also be lots of `print ...` → `print(...)` changes, which are
     needed unless we replaced them with calls to a logging logger
     [2].
[2]: http://thread.gmane.org/gmane.linux.gentoo.catalyst/2085/focus=2103

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Python 3 support

[Brian Dolbec]

On Tue, 2013-05-21 at 12:33 +0200, Dmitry Selyutin wrote:
> Hello everyone!
> 
> Since I'm going to reimplement catalyst in GSoC, I'd like to tell my
> opinion if it has some weight. :-)
> Some years have passed since Python 3 was created, and nowadays I
> think it is stable enough to select it. I'd rather thought about
> Python 2 compatibility than Python 3, since it seems to become a
> standart soon. To cut a long story short, I'd rather oriented to
> Python 3 than Python 2, though I prefer to use Python 2 nowadays. Of
> course compatibility will be one of main aims, but I see some benefit
> if we will choose Python 3 rather than Python 2, though users may
> successfully use catalyst with Python 2 in the future. It's just a
> proposal, so I'd like to hear your opinions. If you don't agree, we
> may rather think about Python 2.

Dmitry, It's not that I don't want it to be python3 compatible as an
afterthought.  For the same reason as i told you already.  Catalyst is a
python2 application and is working code.  It is better to fix all the
poor areas of the code first before migrating to python3.  Also, that
will fix some of the compatibility issues on it's own.  That way the
changes can be properly tested.  Changing the code to py3 will bring
enough bugs into the system on it's own.  It is far better to fix the
poor code first.

> Even more, I'd like to avoid some generators and provide this support
> manually: I've always hated generators, especially code generators
> (and GUI ones).
> 
> What do you think about it?

Catalyst should not need to have 2to3 run on it's codebase in the ebuild
to be py3 compatible.  python2.6 and up are mostly py3 compatible
already.  Catalyst does not do anything wild that should need porting
between versions.  There are also a few tricks that can be done to
simplify compatibility without the need for conversion runs.



-- 
Brian Dolbec <dolsen@gentoo.org>

Re: [gentoo-catalyst] Python 3 support

[Brian Dolbec]

On Tue, 2013-05-21 at 07:20 -0400, W. Trevor King wrote:
> On Tue, May 21, 2013 at 12:33:28PM +0200, Dmitry Selyutin wrote:
> > Even more, I'd like to avoid some generators and provide this
> > support manually: I've always hated generators, especially code
> > generators (and GUI ones).
> 
> Generators?  Are you referring to 2to3?  I think anyone converting a
> Python project from Python 2 to Python 3 would be cray not to use it.
> They'd also be crazy if they just assumed that it generated optimal
> code ;).  It's best to run it, and then go double check to make sure
> all the changes are meaningful.  Keep stuff like [1]:
> 
>   -                       raise CatalystError, "Could not open file " + filename
>   +                       raise CatalystError("Could not open file " + filename)
> 
> but drop stuff like:
> 
> -               for x in self.values.keys():
> +               for x in list(self.values.keys()):
> 
> Cheers,
> Trevor
> 
> [1]: Although see 305cde6 (Massive pyflakes import cleanup and broken
>      CatalystError calls, 2013-01-19) in dolsen's branch.  There will
>      also be lots of `print ...` → `print(...)` changes, which are
>      needed unless we replaced them with calls to a logging logger
>      [2].
> [2]: http://thread.gmane.org/gmane.linux.gentoo.catalyst/2085/focus=2103
> 

I too was thinking of converting to using loggers. I want it to be
something that is easy to re-direct.  In that way it would be easier for
alternate front-ends to be created.  One such possibility could be a web
front end or other server driven monitoring application.

Re: [gentoo-catalyst] Python 3 support

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

On Tue, May 21, 2013 at 07:46:27AM -0700, Brian Dolbec wrote:
> I too was thinking of converting to using loggers.

I think the only reason we haven't done it yet is because the backlog
of pending patches has grown to large to juggle in our heads ;).
Until some of those patches land in master (or are officially
rejected?), I'm not sure there's much to do about it.  Continued
development on top of your rewrite branch will probably just increase
the time it takes to land :(.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Python 3 support

[Matthew Ruffalo]

On 05/21/2013 10:33 AM, Brian Dolbec wrote:
> On Tue, 2013-05-21 at 12:33 +0200, Dmitry Selyutin wrote:
>> Hello everyone!
>>
>> Since I'm going to reimplement catalyst in GSoC, I'd like to tell my
>> opinion if it has some weight. :-)
>> Some years have passed since Python 3 was created, and nowadays I
>> think it is stable enough to select it. I'd rather thought about
>> Python 2 compatibility than Python 3, since it seems to become a
>> standart soon. To cut a long story short, I'd rather oriented to
>> Python 3 than Python 2, though I prefer to use Python 2 nowadays. Of
>> course compatibility will be one of main aims, but I see some benefit
>> if we will choose Python 3 rather than Python 2, though users may
>> successfully use catalyst with Python 2 in the future. It's just a
>> proposal, so I'd like to hear your opinions. If you don't agree, we
>> may rather think about Python 2.
> Dmitry, It's not that I don't want it to be python3 compatible as an
> afterthought.  For the same reason as i told you already.  Catalyst is a
> python2 application and is working code.  It is better to fix all the
> poor areas of the code first before migrating to python3.  Also, that
> will fix some of the compatibility issues on it's own.  That way the
> changes can be properly tested.  Changing the code to py3 will bring
> enough bugs into the system on it's own.  It is far better to fix the
> poor code first.
>
>> Even more, I'd like to avoid some generators and provide this support
>> manually: I've always hated generators, especially code generators
>> (and GUI ones).
>>
>> What do you think about it?
> Catalyst should not need to have 2to3 run on it's codebase in the ebuild
> to be py3 compatible.  python2.6 and up are mostly py3 compatible
> already.  Catalyst does not do anything wild that should need porting
> between versions.  There are also a few tricks that can be done to
> simplify compatibility without the need for conversion runs.
Thank you all for the advice and feedback. Things like "don't base any 
real work on the master branch at 
git://git.overlays.gentoo.org/proj/catalyst.git" are exactly what I was 
hoping to hear.

Some other thoughts:

I wouldn't call what I'm proposing to do "refactoring". I've _used_ 
catalyst for quite a while but only looked at its code a few days ago. 
The extent of my changes would be "make it run on Python 3 and update 
some parts of the code to use newer/cleaner Python idioms".

Almost all of the bugs introduced in 2to3 conversion will be bytes/str 
related, and it's easy enough to decode() text that is read from a 
subprocess. Everything should be valid UTF-8, so there won't be any 
unpleasant character encoding issues.

It's likely that I'm wrong about this, but I don't think that there is 
much (any?) benefit to supporting Python 2 and Python 3 with the same 
code base. I have added 2to3 conversion to a few setup.py files, but 
these were in Python packages that still had to support Python 2 on 
multiple distributions. Catalyst is fundamentally a Gentoo 
_application_, and the implementation details don't really matter as 
long as it runs on current Gentoo systems. From my own experience as a 
user, I run the catalyst executable and I don't particularly care 
whether it's implemented in C, Java, Haskell, Python, INTERCAL, etc. I 
only care that I can 'emerge catalyst', make some tweaks to its files in 
/etc, and run 'catalyst'.

Since Python 3.2+ is available on all current Gentoo systems and will be 
from now on, I don't think there's anything to lose by doing a one-shot 
conversion and changing relevant hashbang lines to "#!/usr/bin/env python3".

(Brian: you may want to change 
http://dev.gentoo.org/~dolsen/catalyst/HEAD to contain "ref: 
refs/heads/rewrite" :) )

Thanks,
MMR...

Re: [gentoo-catalyst] Python 3 support

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

On Tue, May 21, 2013 at 11:34:06AM -0400, Matthew Ruffalo wrote:
> Since Python 3.2+ is available on all current Gentoo systems and
> will be from now on, I don't think there's anything to lose by doing
> a one-shot conversion and changing relevant hashbang lines to
> "#!/usr/bin/env python3".

I agree, but since it is still possible to run Portage and catalyst on
a Python-2-only system, I wouldn't be surprised if some folks want to
keep 2.x compatibility.  Portage only supports Python ≥2.6, and dual
2/3 compatibility in that case shouldn't be too hard to maintain.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]