From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 9C7D21387FD for ; Fri, 13 Jun 2014 22:36:50 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 7C8D7E09AB; Fri, 13 Jun 2014 22:36:44 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 80E3AE09A4 for ; Fri, 13 Jun 2014 22:36:43 +0000 (UTC) Received: from pomiot.lan (static-81-219-255-132.devs.futuro.pl [81.219.255.132]) (using SSLv3 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: mgorny) by smtp.gentoo.org (Postfix) with ESMTPSA id 6DE3633FF3A; Fri, 13 Jun 2014 22:36:41 +0000 (UTC) Date: Sat, 14 Jun 2014 00:36:02 +0200 From: =?ISO-8859-2?B?TWljaGGzIEfzcm55?= To: Cc: Subject: [gentoo-dev] python-utils-r1.eclass: fixes & improvements for python_fix_shebang Message-ID: <20140614003602.44da5f4e@pomiot.lan> Organization: Gentoo X-Mailer: Claws Mail 3.9.3 (GTK+ 2.24.23; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_//UJtiFpQMdBUYsWW+GneXFg"; protocol="application/pgp-signature" X-Archives-Salt: 5f276f93-6c95-4803-b50d-89f93c7af11d X-Archives-Hash: bd803c171bc683595b761f8592835904 --Sig_//UJtiFpQMdBUYsWW+GneXFg Content-Type: multipart/mixed; boundary="MP_/fj2dQs_dZ3PvKwzf7DR=cM+" --MP_/fj2dQs_dZ3PvKwzf7DR=cM+ Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hello, all. Attaching two patches: one that practically rewrites python_fix_shebang, and the other one that adds proper tests for it. Major changes: 1. I replaced unclear space magic that attempted to handle corner cases with plain 'for i in ${shebang}' -- trying to match various python* patterns left-to-right to shebang. This specifically fixes corner cases like: /usr/bin/python2 python (not that it's meaningful but we mangle it correctly now -- always the leftmost matching thingie is replaced!) 2. I've added --quiet and --force options, the former to silence the 'fixing shebang in ...' output, the latter to force replacing even incompatible shebangs (e.g. python3 -> python2.7). 3. Added proper tests for a lot of cases, including corner cases like: /mnt/python2/usr/bin/python3 Please review. --=20 Best regards, Micha=B3 G=F3rny --MP_/fj2dQs_dZ3PvKwzf7DR=cM+ Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=python-utils-r1.eclass.diff Index: python-utils-r1.eclass =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /var/cvsroot/gentoo-x86/eclass/python-utils-r1.eclass,v retrieving revision 1.56 diff -u -B -r1.56 python-utils-r1.eclass --- python-utils-r1.eclass 26 May 2014 16:13:35 -0000 1.56 +++ python-utils-r1.eclass 13 Jun 2014 22:28:39 -0000 @@ -670,8 +670,7 @@ =20 # don't use this at home, just call python_doscript() instead if [[ ${_PYTHON_REWRITE_SHEBANG} ]]; then - local _PYTHON_FIX_SHEBANG_QUIET=3D1 - python_fix_shebang "${ED%/}/${d}/${newfn}" + python_fix_shebang -q "${ED%/}/${d}/${newfn}" fi } =20 @@ -935,7 +934,7 @@ } =20 # @FUNCTION: python_fix_shebang -# @USAGE: ... +# @USAGE: [-f|--force] [-q|--quiet] ... # @DESCRIPTION: # Replace the shebang in Python scripts with the current Python # implementation (EPYTHON). If a directory is passed, works recursively @@ -947,13 +946,28 @@ # # Shebangs matching explicitly current Python version will be left # unmodified. Shebangs requesting another Python version will be treated -# as fatal error. +# as fatal error, unless --force is given. +# +# --force causes the function to replace even shebangs that require +# incompatible Python version. --quiet causes the function not to list +# modified files verbosely. python_fix_shebang() { debug-print-function ${FUNCNAME} "${@}" =20 - [[ ${1} ]] || die "${FUNCNAME}: no paths given" [[ ${EPYTHON} ]] || die "${FUNCNAME}: EPYTHON unset (pkg_setup not called= ?)" =20 + local force quiet + while [[ ${@} ]]; do + case "${1}" in + -f|--force) force=3D1; shift;; + -q|--quiet) quiet=3D1; shift;; + --) shift; break;; + *) break;; + esac + done + + [[ ${1} ]] || die "${FUNCNAME}: no paths given" + local path f for path; do local any_correct any_fixed is_recursive @@ -961,54 +975,88 @@ [[ -d ${path} ]] && is_recursive=3D1 =20 while IFS=3D read -r -d '' f; do - local shebang=3D$(head -n 1 "${f}") - local error + local shebang i + local error from =20 - case "${shebang} " in - '#!'*"${EPYTHON} "*) - debug-print "${FUNCNAME}: in file ${f#${D}}" - debug-print "${FUNCNAME}: shebang matches EPYTHON: ${shebang}" - - # Nothing to do, move along. - any_correct=3D1 - ;; - '#!'*python" "*|'#!'*python[23]" "*) - debug-print "${FUNCNAME}: in file ${f#${D}}" - debug-print "${FUNCNAME}: rewriting shebang: ${shebang}" - - # Note: for internal use. - if [[ ! ${_PYTHON_FIX_SHEBANG_QUIET} ]]; then - einfo "Fixing shebang in ${f#${D}}." - fi - - local from - if [[ "${shebang} " =3D=3D *'python2 '* ]]; then - from=3Dpython2 - python_is_python3 "${EPYTHON}" && error=3D1 - elif [[ "${shebang} " =3D=3D *'python3 '* ]]; then - from=3Dpython3 - python_is_python3 "${EPYTHON}" || error=3D1 - else - from=3Dpython - fi - - if [[ ! ${error} ]]; then - sed -i -e "1s:${from}:${EPYTHON}:" "${f}" || die - any_fixed=3D1 - fi - ;; - '#!'*python[23].[0123456789]" "*|'#!'*pypy" "*|'#!'*jython[23].[012345= 6789]" "*) - # Explicit mismatch. - error=3D1 - ;; - *) - # Non-Python shebang. Allowed in recursive mode, - # disallowed when specifying file explicitly. - [[ ${is_recursive} ]] || error=3D1 - ;; - esac + read shebang <"${f}" =20 - if [[ ${error} ]]; then + # First, check if it's shebang at all... + if [[ ${shebang} =3D=3D '#!'* ]]; then + # Match left-to-right in a loop, to avoid matching random + # repetitions like 'python2.7 python2'. + for i in ${shebang}; do + case "${i}" in + *"${EPYTHON}") + debug-print "${FUNCNAME}: in file ${f#${D}}" + debug-print "${FUNCNAME}: shebang matches EPYTHON: ${shebang}" + + # Nothing to do, move along. + any_correct=3D1 + from=3D${EPYTHON} + break + ;; + *python|*python[23]) + debug-print "${FUNCNAME}: in file ${f#${D}}" + debug-print "${FUNCNAME}: rewriting shebang: ${shebang}" + + if [[ ${i} =3D=3D *python2 ]]; then + from=3Dpython2 + if [[ ! ${force} ]]; then + python_is_python3 "${EPYTHON}" && error=3D1 + fi + elif [[ ${i} =3D=3D *python3 ]]; then + from=3Dpython3 + if [[ ! ${force} ]]; then + python_is_python3 "${EPYTHON}" || error=3D1 + fi + else + from=3Dpython + fi + break + ;; + *python[23].[0123456789]|*pypy|*jython[23].[0123456789]) + # Explicit mismatch. + if [[ ! ${force} ]]; then + error=3D1 + else + case "${i}" in + *python[23].[0123456789]) + from=3D"python[23].[0123456789]";; + *pypy) + from=3D"pypy";; + *jython[23].[0123456789]) + from=3D"jython[23].[0123456789]";; + *) + die "${FUNCNAME}: internal error in 2nd pattern match";; + esac + fi + break + ;; + esac + done + fi + + if [[ ! ${error} && ! ${from} ]]; then + # Non-Python shebang. Allowed in recursive mode, + # disallowed when specifying file explicitly. + [[ ${is_recursive} ]] && continue + error=3D1 + fi + + if [[ ! ${quiet} ]]; then + einfo "Fixing shebang in ${f#${D}}." + fi + + if [[ ! ${error} ]]; then + # We either want to match ${from} followed by space + # or at end-of-string. + if [[ ${shebang} =3D=3D *${from}" "* ]]; then + sed -i -e "1s:${from} :${EPYTHON} :" "${f}" || die + else + sed -i -e "1s:${from}$:${EPYTHON}:" "${f}" || die + fi + any_fixed=3D1 + else eerror "The file has incompatible shebang:" eerror " file: ${f#${D}}" eerror " current shebang: ${shebang}" --MP_/fj2dQs_dZ3PvKwzf7DR=cM+ Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=python-utils-r1.sh.diff Index: python-utils-r1.sh =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /var/cvsroot/gentoo-x86/eclass/tests/python-utils-r1.sh,v retrieving revision 1.6 diff -u -B -r1.6 python-utils-r1.sh --- python-utils-r1.sh 8 Apr 2014 16:05:30 -0000 1.6 +++ python-utils-r1.sh 13 Jun 2014 22:28:50 -0000 @@ -30,6 +30,33 @@ tend ${?} } =20 +test_fix_shebang() { + local from=3D${1} + local to=3D${2} + local expect=3D${3} + local args=3D( "${@:4}" ) + + tbegin "python_fix_shebang${args[@]+ ${args[*]}} from ${from} to ${to} (e= xp: ${expect})" + + echo "${from}" > "${tmpfile}" + output=3D$( EPYTHON=3D${to} python_fix_shebang "${args[@]}" -q "${tmpfile= }" 2>&1 ) + + if [[ ${?} !=3D 0 ]]; then + if [[ ${expect} !=3D FAIL ]]; then + echo "${output}" + tend 1 + else + tend 0 + fi + else + [[ $(<"${tmpfile}") =3D=3D ${expect} ]] \ + || eerror "${from} -> ${to}: $(<"${tmpfile}") !=3D ${expect}" + tend ${?} + fi +} + +tmpfile=3D$(mktemp) + inherit python-utils-r1 =20 test_var EPYTHON python2_7 python2.7 @@ -66,4 +93,47 @@ test_is python_is_python3 jython2.7 1 test_is python_is_python3 pypy 1 =20 +# generic shebangs +test_fix_shebang '#!/usr/bin/python' python2.7 '#!/usr/bin/python2.7' +test_fix_shebang '#!/usr/bin/python' python3.4 '#!/usr/bin/python3.4' +test_fix_shebang '#!/usr/bin/python' pypy '#!/usr/bin/pypy' +test_fix_shebang '#!/usr/bin/python' jython2.7 '#!/usr/bin/jython2.7' + +# python2/python3 matching +test_fix_shebang '#!/usr/bin/python2' python2.7 '#!/usr/bin/python2.7' +test_fix_shebang '#!/usr/bin/python3' python2.7 FAIL +test_fix_shebang '#!/usr/bin/python3' python2.7 '#!/usr/bin/python2.7' --f= orce +test_fix_shebang '#!/usr/bin/python3' python3.4 '#!/usr/bin/python3.4' +test_fix_shebang '#!/usr/bin/python2' python3.4 FAIL +test_fix_shebang '#!/usr/bin/python2' python3.4 '#!/usr/bin/python3.4' --f= orce + +# pythonX.Y matching (those mostly test the patterns) +test_fix_shebang '#!/usr/bin/python2.7' python2.7 '#!/usr/bin/python2.7' +test_fix_shebang '#!/usr/bin/python2.7' python3.2 FAIL +test_fix_shebang '#!/usr/bin/python2.7' python3.2 '#!/usr/bin/python3.2' -= -force +test_fix_shebang '#!/usr/bin/python3.2' python3.2 '#!/usr/bin/python3.2' +test_fix_shebang '#!/usr/bin/python3.2' python2.7 FAIL +test_fix_shebang '#!/usr/bin/python3.2' python2.7 '#!/usr/bin/python2.7' -= -force +test_fix_shebang '#!/usr/bin/pypy' pypy '#!/usr/bin/pypy' +test_fix_shebang '#!/usr/bin/pypy' python2.7 FAIL +test_fix_shebang '#!/usr/bin/pypy' python2.7 '#!/usr/bin/python2.7' --force +test_fix_shebang '#!/usr/bin/jython2.7' jython2.7 '#!/usr/bin/jython2.7' +test_fix_shebang '#!/usr/bin/jython2.7' jython3.2 FAIL +test_fix_shebang '#!/usr/bin/jython2.7' jython3.2 '#!/usr/bin/jython3.2' -= -force + +# fancy path handling +test_fix_shebang '#!/mnt/python2/usr/bin/python' python3.4 \ + '#!/mnt/python2/usr/bin/python3.4' +test_fix_shebang '#!/mnt/python2/usr/bin/python2' python2.7 \ + '#!/mnt/python2/usr/bin/python2.7' +test_fix_shebang '#!/mnt/python2/usr/bin/env python' python2.7 \ + '#!/mnt/python2/usr/bin/env python2.7' +test_fix_shebang '#!/mnt/python2/usr/bin/python2 python2' python2.7 \ + '#!/mnt/python2/usr/bin/python2.7 python2' +test_fix_shebang '#!/mnt/python2/usr/bin/python3 python2' python2.7 FAIL +test_fix_shebang '#!/mnt/python2/usr/bin/python3 python2' python2.7 \ + '#!/mnt/python2/usr/bin/python2.7 python2' --force + +rm "${tmpfile}" + texit --MP_/fj2dQs_dZ3PvKwzf7DR=cM+-- --Sig_//UJtiFpQMdBUYsWW+GneXFg Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJTm3zSXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2REJCMDdDQzRGMERBRDA2RUEwQUZFNDFC MDdBMUFFQUVGQjQ0NjRFAAoJELB6GurvtEZOpfsQAIO2rMrstJuj8quEEW7sRxEg 44nWLGoZe7EaJHQ/cZX0kVbV+1vg0h/lSh7bpT2MvQYMWOHdr6IlLmc6O3RlB9sV FY6Ix0rkzBIKDJDzvV8w9hfS+1m5OKsmusW/RIVBz2/UkKmkuVjv1htRy9pb1gOK yRvKYArJ2w8tuAM4xIDtm/Xli1Uy09hc7/A+VTrU8Rzc8Kr4nWdhgVAvQt9oamzv Ghg8DqDuva51XrbSfUZKoFMxU7oxKERdDD8yIuGDXTUuk77gXvEp+WKDgxxeIHkd CtZAiS4zMvyV7VBRtt5Lopa8sSYcTcQsIN89l0b2wM8vON/DiL+OnnLzonGaD4ui 5RetcIPSgmRHNvcxmXgstzifUCIK0e7IhrdQSYAoIQMGIwyzvWaEP91AdEDXC8sc XGQxfKExGdjW8KDoybcsKLBdqZ4CRSKPU71/e6eVLaGrISqB9nnzr9QivtUdJKCE 3rMyfEn+0NhIcU7whRX/W7bJulk8lKw/u7btGCd7/nrYL5z4VEyOHXzQ7+aWBS4j CSj/GWBf724cLvyBU+9H1OQtyJXPtRrhCEn6u1lA1bynkXuPfKUHhQUnMMWQ1NBd HYq9Uqk50BD1KxZj1O8zEkCoJNv8Sc/lI5XFfHpoKT10nQ/D309L+vGzVAo17CO8 vNOE+J/m8v6TX1+dq7sF =lofa -----END PGP SIGNATURE----- --Sig_//UJtiFpQMdBUYsWW+GneXFg--