From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id D571A138262 for ; Tue, 17 May 2016 10:01:48 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id F0D211419F; Tue, 17 May 2016 10:01:39 +0000 (UTC) Received: from mail-vk0-f47.google.com (mail-vk0-f47.google.com [209.85.213.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id F04F221C06B for ; Tue, 17 May 2016 10:01:38 +0000 (UTC) Received: by mail-vk0-f47.google.com with SMTP id c189so13231034vkb.1 for ; Tue, 17 May 2016 03:01:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=rkZXUWeSqG9/REpZGWFY8HILLtP1Jb66uhH8NUSH/yw=; b=ZVLqYU8RRW4Lt/4HUX7zFZaM8FsphHhitwMVOa5a9LCC4wi84GisaOm9T9aGlw02j6 BLQYTGymnHYCijGBj+iB+Kn7jF/m2e3Ku65nPR1XQzRNHCeW+rvWYJhuJy7ASbmuqFyZ c2doa4NbUm+01Y4cvynk3ceFLVpa7xhzTg/Aou0iroOkl8TWOg2cyK7d+2c2cYhlISLF CuWr57J/FbTRmrho08inNUpn4TmeQ1wAsaiGwZhOXa0suNXETBm+oA9PnUIRkUF86wrU EukH/g3qhxmxvyNNdK0hqMnNYNPKS6my2j6oMqls/wyT2jTzKaFXrhHj//H1Fw1CBbi+ UlDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=rkZXUWeSqG9/REpZGWFY8HILLtP1Jb66uhH8NUSH/yw=; b=CfJK5da3LHYikbsvLpbzRRA4VGaTiOac69NxDSMHUAh7AU57dQyOZlp+XVpsp5bqKr Fj2K9pWEjJOrInxFylYoDHt3nudX1LhLkwpiQr1ibkD/zJQ6adPMcoq04PkMNUQlHHEn Gq12CgnXV+mpJCpgpAMpgXw0X8Mgx70sWFwXJwwAEWAmUGClCJZ0EWkP/St5bugoINsn PyB5lse6Uto8CyKXiTpcJd3NRBoh1bd94RUsxLqkvYlLYDYUuWSKpzDx0cqLdHjj+diG gjMpAsRNr7jBU77j8GMG7z/nDRclCuE2WyUMJY86vrwOmR1M3585zNqM2xpPLEMBX4HT MDaA== X-Gm-Message-State: AOPr4FUGpOBSDlBHPTG24ljNYXrbZJg2N4fT5czq8fjUgw02QsyTV61KXDTHWQlFFSouHhnfOUpfhwn6O8uAqw== X-Received: by 10.176.2.87 with SMTP id 81mr168959uas.104.1463479298012; Tue, 17 May 2016 03:01:38 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Received: by 10.176.2.46 with HTTP; Tue, 17 May 2016 03:01:18 -0700 (PDT) In-Reply-To: <20160517084643.GA24972@skade.schwarzvogel.de> References: <20160516183840.4b241463@gentp.lnet> <20160517084643.GA24972@skade.schwarzvogel.de> From: Pallav Agarwal Date: Tue, 17 May 2016 15:31:18 +0530 Message-ID: Subject: Re: [gentoo-dev] Proposal for changes for the next EAPI version To: gentoo-dev@lists.gentoo.org Content-Type: multipart/alternative; boundary=001a1137634ab1bca3053306d386 X-Archives-Salt: 5f91f79c-0769-4141-9fe8-df422a25b5f4 X-Archives-Hash: f286e8dbd1db315d56babd76f53a5be1 --001a1137634ab1bca3053306d386 Content-Type: text/plain; charset=UTF-8 Hi, You are right, of course. The target is to standardize something that would encourage maintainers to actually provide non-upstream scripts to test packages. At the same time, it should be possible to use those scripts for automated testing without human interference. Even if they are provided independently, not as part of ebuilds, we need a to have them associated with a particular ebuild if need be as in case of a bug fix, or many ebuilds in case of basic functionality for all versions of a package. > And under this scheme, individual projects can define their custom > hooks and tricks in the ebuild themselves as metadata ( like we > currently to via eclass variables ), and be handled, not by portage, > or PMS, not by EAPI, but by the QA tool in conjunction with eclass > designers. > I'm not sure how to deal with eclasses but I guess I'll find out. On Tue, May 17, 2016 at 2:16 PM, Tobias Klausmann wrote: > Hi! > > On Tue, 17 May 2016, Kent Fredric wrote: > > > On 17 May 2016 at 19:37, Pallav Agarwal > wrote: > > > For normal users we wouldn't. But currently, arch-testers need to make > a > > > judgement call on what to test when a stable-req bug is filed. Tests > run in > > > src_test are provided by upstream, and does not guarantee that a > package > > > that has been merged will actually run on the system. > > > If the maintainer could add a couple small scripts to check basic > > > functionality > > > of the merged package, it would make testing for arch testers much > easier > > > and reliable. > > > Let me give an example. Let's say > > > app-misc/screenfetch-2.7.7 is the current stable package for > screenfetch in > > > the portage tree. > > > However, on running, it produces an error on the top, along with the > proper > > > output. > > > If screenfetch-3.0.0 happens to fix that error, maintainer can add a > simple > > > script > > > > > > if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then > > > eerror "Still producing error" > > > fi > > > > > > To make sure the build is properly updating the screenfetch version, > and > > > that > > > the bug has in fact been fixed. This is the only way I can see to > reliabily > > > and automatically test packages that have been merged successfully > > > > > > I don't think this needs to be an EAPI change. And if we can acheive > > the goal without one, the better. > > Agreed. > > > [... lots of interesting stuff ...] > > Or maintainers and teams could do what the Emacs team does: > provide test plans[0]. > > Naturally, I'd prefer something automated over the very hands-on > tests that are a bit of a necessity for an application like > Emacs, but they are still better than nothing. > > Either way, I think it is preferable to have non-upstream tests > of whatever shape we pick to be separate from the ebuilds and the > source files, for the simple reason that most users won't care > about them. Those who do can retrieve them in the same manner as > the ATs. > > And as for my pet peeve, tests that are known to fail, can we > also annotate that somehow so I don't waste hours running a test > suite that gives zero signal on whether I should add the stable > keyword? Even a one-line hin in the stabilization request would > be nice. As it is, I keep a list of known-to-fail packages and my > testing machinery tells me to not bother with FEATURES=test in > those case. > > Not ideal, but less time-wasty. > > Regards, > Tobias > > [0] https://wiki.gentoo.org/wiki/Project:Emacs/Test_plans > > > -- > if (user_specified) > /* Didn't work, but the user is convinced this is the > * place. */ > linux-2.4.0-test2/drivers/parport/parport_pc.c > > --001a1137634ab1bca3053306d386 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,
You are right, of course.=
The target is to standardize something that would encourage maint= ainers
to actually provide non-upstream scripts to test packages. = At the same
time, it should be possible to use those scripts = for automated testing without
human interference.

Even if they are provided independently, not as part of ebuilds, we = need a
to have them associated with a particular ebuild if ne= ed be as in case of a
bug fix, or many ebuilds in case of bas= ic functionality for all versions of a
package.
> And = under this scheme, individual projects can define their custom
> hooks and tricks in the ebuild themselves as metadata ( like we
> currently to via eclass variables ), and be handled, not by portage, > or PMS, not by EAPI, but by the QA tool in conjunction with eclass
> designers.
>
I'm not sure how to deal with ecl= asses but I guess I'll find out.

On Tue, May 17, = 2016 at 2:16 PM, Tobias Klausmann <klausman@gentoo.org> wr= ote:
Hi!

On Tue, 17 May 2016, Kent Fredric wrote:

> On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@gmail.com> wrote:
> > For normal users we wouldn't. But currently, arch-testers nee= d to make a
> > judgement call on what to test when a stable-req bug is filed. Te= sts run in
> > src_test are provided by upstream, and does not guarantee that a = package
> > that has been merged will actually run on the system.
> > If the maintainer could add a couple small scripts to check basic=
> > functionality
> > of the merged package, it would make testing for arch testers muc= h easier
> > and reliable.
> > Let me give an example. Let's say
> > app-misc/screenfetch-2.7.7 is the current stable package for scre= enfetch in
> > the portage tree.
> > However, on running, it produces an error on the top, along with = the proper
> > output.
> > If screenfetch-3.0.0 happens to fix that error, maintainer can ad= d a simple
> > script
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 if [ "$(screenfetch 2>&1 1= >/dev/null)" !=3D "" ] then
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 eerror "Stil= l producing error"
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 fi
> >
> > To make sure the build is properly updating the screenfetch versi= on, and
> > that
> > the bug has in fact been fixed. This is the only way I can see to= reliabily
> > and automatically test packages that have been merged successfull= y
>
>
> I don't think this needs to be an EAPI change. And if we can achei= ve
> the goal without one, the better.

Agreed.

> [... lots of interesting stuff ...]

Or maintainers and teams could do what the Emacs team does:
provide test plans[0].

Naturally, I'd prefer something automated over the very hands-on
tests that are a bit of a necessity for an application like
Emacs, but they are still better than nothing.

Either way, I think it is preferable to have non-upstream tests
of whatever shape we pick to be separate from the ebuilds and the
source files, for the simple reason that most users won't care
about them. Those who do can retrieve them in the same manner as
the ATs.

And as for my pet peeve, tests that are known to fail, can we
also annotate that somehow so I don't waste hours running a test
suite that gives zero signal on whether I should add the stable
keyword? Even a one-line hin in the stabilization request would
be nice. As it is, I keep a list of known-to-fail packages and my
testing machinery tells me to not bother with FEATURES=3Dtest in
those case.

Not ideal, but less time-wasty.

Regards,
Tobias

[0] https://wiki.gentoo.org/wiki/Project:Emac= s/Test_plans


--
if (user_specified)
=C2=A0 =C2=A0 /* Didn't work, but the user is convinced this is the
=C2=A0 =C2=A0 =C2=A0* place. */
=C2=A0 =C2=A0 =C2=A0 =C2=A0 linux-2.4.0-test2/drivers/parport/parport_pc.c<= br>

--001a1137634ab1bca3053306d386--