[gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1942 bytes --]

Hello, everyone.

I have the pleasure to announce that a few improvements have been
deployed by the Repository mirror & CI project today.


1. The mirror for 'gentoo' repository [1] now has a default 'stable'
branch. It is updated automatically by the gentoo-ci checker,
and therefore always contains the latest repository state that has been
confirmed 'green' by CI. While this is far from perfect, it's the first
step towards preventing major issues from being deployed on our users.

If you are already using the mirror, you will need to either switch
branch manually, or re-add it.


2. The repository QA report [2] has been extended with some repository
statistics. In particular, the timestamp of the newest commit
and the number of valid (that is, those not dying in global scope)
ebuilds are reported. Additionally, the homepage link is now included
as well.

This enables the Overlays team members to easily check which repositories
are unmaintained and/or empty, and handle the issues more efficiently.
It can also be useful to users who want to figure out whether there's
a point in using a particular repository.


3. I've tried to optimize the logic used to run QA checks on
repositories, and I think I was able to even the load better.
Additionally, I've repacked the git repositories to get rid of huge
number of loose objects.

As a result, CI now runs faster. The gentoo-ci runs are down from 10-12
minutes to 7-8 minutes, and pull requests from 16-18 minutes to 9-14
minutes. However, I don't have exact results yet as the server is still
busy removing old files ;-).


4. Finally, the mirroring code has been updated to correctly handle git
repositories for which 'master' is not the default branch.


Enjoy!


[1]:https://github.com/gentoo-mirror/gentoo
[2]:https://qa-reports.gentoo.org/output/repos/

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[rindeal]

On 5 June 2016 at 18:04, Michał Górny <mgorny@gentoo.org> wrote:
> Hello, everyone.
>
> I have the pleasure to announce that a few improvements have been
> deployed by the Repository mirror & CI project today.
>
>
> 1. The mirror for 'gentoo' repository [1] now has a default 'stable'
> branch. It is updated automatically by the gentoo-ci checker,
> and therefore always contains the latest repository state that has been
> confirmed 'green' by CI. While this is far from perfect, it's the first
> step towards preventing major issues from being deployed on our users.
>
> If you are already using the mirror, you will need to either switch
> branch manually, or re-add it.
>

Isn't no commit approach better than having broken commit + revert
commit? I'm doing so in my overlay using "GitHub protected branches"
feature + repoman running on Travis CI.

>
> 4. Finally, the mirroring code has been updated to correctly handle git
> repositories for which 'master' is not the default branch.

How is it even possible to set up with repositories.xml?

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[Kent Fredric]

On 6 June 2016 at 04:31, rindeal <dev.rindeal@gmail.com> wrote:
> Isn't no commit approach better than having broken commit + revert
> commit?


Huh?

Its doing "replicate to github on pass using a merge commit".

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[rindeal]

On 5 June 2016 at 18:40, Kent Fredric <kentfredric@gmail.com> wrote:
> On 6 June 2016 at 04:31, rindeal <dev.rindeal@gmail.com> wrote:
>> Isn't no commit approach better than having broken commit + revert
>> commit?
>
>
> Huh?
>
> Its doing "replicate to github on pass using a merge commit".

I'd like to see the master branch free of commits which do not pass
CI, instead of having broken commits and holding master back until
revert commits are introduced.

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[M. J. Everitt]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On 05/06/16 17:49, rindeal wrote:
> On 5 June 2016 at 18:40, Kent Fredric <kentfredric@gmail.com> wrote:
>> On 6 June 2016 at 04:31, rindeal <dev.rindeal@gmail.com> wrote:
>>> Isn't no commit approach better than having broken commit + revert
>>> commit?
>>
>> Huh?
>>
>> Its doing "replicate to github on pass using a merge commit".
> I'd like to see the master branch free of commits which do not pass
> CI, instead of having broken commits and holding master back until
> revert commits are introduced.
>
Which is the whole idea .... 'stable' becomes fully CI parsed good
'green light' whereas master is a 'holding bay' until the CI script can
do its stuff ..


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[Kent Fredric]

On 6 June 2016 at 04:49, rindeal <dev.rindeal@gmail.com> wrote:
> I'd like to see the master branch free of commits which do not pass
> CI, instead of having broken commits and holding master back until
> revert commits are introduced.


Just pretend "stable" is called "master" and pretend "master" is called "devel"

-> Wishes granted.

Anything else requires a dedicated merge gate-keeper of some description.

Which... we kinda have ... the gatekeeper is the QA tests, and it
merges the submission line into master when it passes QA.

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[rindeal]

On 5 June 2016 at 18:53, M. J. Everitt <m.j.everitt@iee.org> wrote:
> On 05/06/16 17:49, rindeal wrote:
>> On 5 June 2016 at 18:40, Kent Fredric <kentfredric@gmail.com> wrote:
>>> On 6 June 2016 at 04:31, rindeal <dev.rindeal@gmail.com> wrote:
>>>> Isn't no commit approach better than having broken commit + revert
>>>> commit?
>>>
>>> Huh?
>>>
>>> Its doing "replicate to github on pass using a merge commit".
>> I'd like to see the master branch free of commits which do not pass
>> CI, instead of having broken commits and holding master back until
>> revert commits are introduced.
>>
> Which is the whole idea .... 'stable' becomes fully CI parsed good
> 'green light' whereas master is a 'holding bay' until the CI script can
> do its stuff ..
>

It is not, unless CI filters the broken commits in some miraculous
way. With the current approach, both stable and master branch will
contain the pollution of broken commits + their fixes, instead of
having good commits only.

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[M. J. Everitt]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

On 05/06/16 18:09, rindeal wrote:
> On 5 June 2016 at 18:53, M. J. Everitt <m.j.everitt@iee.org> wrote:
>> On 05/06/16 17:49, rindeal wrote:
>>> On 5 June 2016 at 18:40, Kent Fredric <kentfredric@gmail.com> wrote:
>>>> On 6 June 2016 at 04:31, rindeal <dev.rindeal@gmail.com> wrote:
>>>>> Isn't no commit approach better than having broken commit + revert
>>>>> commit?
>>>> Huh?
>>>>
>>>> Its doing "replicate to github on pass using a merge commit".
>>> I'd like to see the master branch free of commits which do not pass
>>> CI, instead of having broken commits and holding master back until
>>> revert commits are introduced.
>>>
>> Which is the whole idea .... 'stable' becomes fully CI parsed good
>> 'green light' whereas master is a 'holding bay' until the CI script can
>> do its stuff ..
>>
> It is not, unless CI filters the broken commits in some miraculous
> way. With the current approach, both stable and master branch will
> contain the pollution of broken commits + their fixes, instead of
> having good commits only.
To eliminate errors completely, you also remove any form of progression.
As the old proverb goes "you can't make an omelette without breaking
eggs"....

This way, at least the bad commits /get/ fixed *before* they are pushed
out to unsuspecting users ... ok so its not perfect, but this _is_ the
real world here ......


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[Kent Fredric]

On 6 June 2016 at 05:09, rindeal <dev.rindeal@gmail.com> wrote:
> It is not, unless CI filters the broken commits in some miraculous
> way. With the current approach, both stable and master branch will
> contain the pollution of broken commits + their fixes, instead of
> having good commits only.


Doing that is of course, impossibly hard without having every
committer publish to their own branch, and having the "master" built
by *cherry* picking commit series that are "known good".

Its doable, but the complexity it entails is just way more than is
suitable for the gentoo workflow, and is likely to create more
problems than it solves.

The "no bad commits" requires there to be at last *some* branch
somewhere that is constantly not-fast-fowardwardable, and at least one
branch that serves as a synchronization point with strictly linear
history.

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[Daniel Campbell (zlg)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On June 5, 2016 9:04:26 AM PDT, "Michał Górny" <mgorny@gentoo.org> wrote:
>Hello, everyone.
>
>I have the pleasure to announce that a few improvements have been
>deployed by the Repository mirror & CI project today.
>
>
>1. The mirror for 'gentoo' repository [1] now has a default 'stable'
>branch. It is updated automatically by the gentoo-ci checker,
>and therefore always contains the latest repository state that has been
>confirmed 'green' by CI. While this is far from perfect, it's the first
>step towards preventing major issues from being deployed on our users.
>
>If you are already using the mirror, you will need to either switch
>branch manually, or re-add it.
>
>
>2. The repository QA report [2] has been extended with some repository
>statistics. In particular, the timestamp of the newest commit
>and the number of valid (that is, those not dying in global scope)
>ebuilds are reported. Additionally, the homepage link is now included
>as well.
>
>This enables the Overlays team members to easily check which
>repositories
>are unmaintained and/or empty, and handle the issues more efficiently.
>It can also be useful to users who want to figure out whether there's
>a point in using a particular repository.
>
>
>3. I've tried to optimize the logic used to run QA checks on
>repositories, and I think I was able to even the load better.
>Additionally, I've repacked the git repositories to get rid of huge
>number of loose objects.
>
>As a result, CI now runs faster. The gentoo-ci runs are down from 10-12
>minutes to 7-8 minutes, and pull requests from 16-18 minutes to 9-14
>minutes. However, I don't have exact results yet as the server is still
>busy removing old files ;-).
>
>
>4. Finally, the mirroring code has been updated to correctly handle git
>repositories for which 'master' is not the default branch.
>
>
>Enjoy!
>
>
>[1]:https://github.com/gentoo-mirror/gentoo
>[2]:https://qa-reports.gentoo.org/output/repos/

Sounds like a big improvement. I'm not too familiar with CI, so forgive me if this is obvious: should overlays now use the 'stable' branch as their primary remote, so they can ensure that any breakage introduced is caused by their own overlay instead of a flub by us? If so, then I wonder if this will help indirectly improve overlay quality.

Being able to say "as of X commit, the tree is in good shape" is a big deal imo. Thanks for working to make that happen. Is there anything other devs can do to assist your development, or do we stick to the usual 'if in doubt talk to QA'?
- --
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-----BEGIN PGP SIGNATURE-----

iQJRBAEBCgA7NBxEYW5pZWwgQ2FtcGJlbGwgKEdlbnRvbyBEZXZlbG9wZXIpIDx6
bGdAZ2VudG9vLm9yZz4FAldUXhUACgkQASQOlFA54XC2HxAAx3HV3YbjfzEJg2FC
8QJus2qwU63+Agqr84R9X9zQMMUGqkRoFglNBg6slxdSzV3s/2SASYlj3ShGWLvR
sYMgKXmDVvIZ/z5qaepIcmIz0pHc+NjITebZ+zq09IoH8uI2fFFNFibCNAwdz9X1
CWzj5q1i1D9LSaIU0Jmd9N21168V7jSViHx1HeRR/ECx8G0fvaC8mPxpiULmcdA2
ZWqzRfkXLeckNHfJsrQFQQR89WAa20IJSajTitkA9BejNZyMrnjALxFLx6reZ5W6
IK3KR2IaGgnjof5BfBoyX6q2MXT8psJZBusPeQvX1U8COfBFgpX1QfpDdEZzn1PF
BQMOa2mqwHkk7dlL5VT5ue9towFCYpN/J0PJa8+bB+jmTXFgtHYPn50POEiyebyt
il9+bieLs2jqGwIQBwhgwAwL6SXX0EcbyaW2Ja44A4z6e4kFAqgqifP2+C4g6l6Y
Mz36uedDmviL1PWttAjsvROh4X6d1P3frWKdxnYYHdlYjFXig6pEZE87Hnxc3j0C
wBGnYg5qUP8Q65MeLRzpPzBkxHJODgVEQP+sVM5f6Vs5RjAafApit/6SHgbzCqZH
LB3MzYyReJOd/taCLfUgfBp0hnlOlSyuHT3sQmqjJpqPnwisoIEerWHJuUrp7Nn3
Fz19qALKqvESRV42YCz7Zy9Z+ig=
=sgMj
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[rindeal]

On 5 June 2016 at 19:13, Kent Fredric <kentfredric@gmail.com> wrote:
> On 6 June 2016 at 05:09, rindeal <dev.rindeal@gmail.com> wrote:
>> It is not, unless CI filters the broken commits in some miraculous
>> way. With the current approach, both stable and master branch will
>> contain the pollution of broken commits + their fixes, instead of
>> having good commits only.
>
>
> Doing that is of course, impossibly hard without having every
> committer publish to their own branch, and having the "master" built
> by *cherry* picking commit series that are "known good".
>
> Its doable, but the complexity it entails is just way more than is
> suitable for the gentoo workflow, and is likely to create more
> problems than it solves.
>
> The "no bad commits" requires there to be at last *some* branch
> somewhere that is constantly not-fast-fowardwardable, and at least one
> branch that serves as a synchronization point with strictly linear
> history.

It's not hard at all. Developers have branches (personal, feature/bug
specific, ...), before merging to master they do a rebase+force push,
let the CI check it, and if it passes they're allowed to merge it. No
cherry-picking involved. Master is then always clean and happy. I'm
not sure how well it scales, but for the 100-150 commits/day the tree
sees currently, it should be doable.

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[Kent Fredric]

On 6 June 2016 at 05:34, rindeal <dev.rindeal@gmail.com> wrote:
> efore merging to master they do a rebase+force push,
> let the CI check it, and if it passes they're allowed to merge it. No
> cherry-picking involved. Master is then always clean and happy. I'm
> not sure how well it scales, but for the 100-150 commits/day the tree
> sees currently, it should be doable.


And then somebody else merges in the interval between you submitting
it, the CI check starting, and the CI check approving it (keeping in
mind the CI run is ~10 minutes long), but what merged breaks your
commit and the CI didn't see it?

Now what?

Or does everyone have to form an orderly queue where they wait for the CI ?

That' maxes out at 144 pushes / day.

This is not the kernel where we have a clean release cycle we can
target and sit on patch series for a week or more while we polish it,
and we don't have an avalanche of "this is good, it lands in this
release".

This is pretty much one of the big limitations of "Shared push"
patterns in that their workflow and "unified branching" workflows
contradict each other in lots of ways.

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 3557 bytes --]

On 06/05/2016 02:05 PM, james wrote:
> On 06/05/2016 12:15 PM, Daniel Campbell (zlg) wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>>
>> On June 5, 2016 9:04:26 AM PDT, "Michał Górny" <mgorny@gentoo.org> wrote:
>>> Hello, everyone.
>>>
>>> I have the pleasure to announce that a few improvements have been
>>> deployed by the Repository mirror & CI project today.
>>>
>>>
>>> 1. The mirror for 'gentoo' repository [1] now has a default 'stable'
>>> branch. It is updated automatically by the gentoo-ci checker,
>>> and therefore always contains the latest repository state that has been
>>> confirmed 'green' by CI. While this is far from perfect, it's the first
>>> step towards preventing major issues from being deployed on our users.
>>>
>>> If you are already using the mirror, you will need to either switch
>>> branch manually, or re-add it.
>>>
>>>
>>> 2. The repository QA report [2] has been extended with some repository
>>> statistics. In particular, the timestamp of the newest commit
>>> and the number of valid (that is, those not dying in global scope)
>>> ebuilds are reported. Additionally, the homepage link is now included
>>> as well.
>>>
>>> This enables the Overlays team members to easily check which
>>> repositories
>>> are unmaintained and/or empty, and handle the issues more efficiently.
>>> It can also be useful to users who want to figure out whether there's
>>> a point in using a particular repository.
>>>
>>>
>>> 3. I've tried to optimize the logic used to run QA checks on
>>> repositories, and I think I was able to even the load better.
>>> Additionally, I've repacked the git repositories to get rid of huge
>>> number of loose objects.
>>>
>>> As a result, CI now runs faster. The gentoo-ci runs are down from 10-12
>>> minutes to 7-8 minutes, and pull requests from 16-18 minutes to 9-14
>>> minutes. However, I don't have exact results yet as the server is still
>>> busy removing old files ;-).
>>>
>>>
>>> 4. Finally, the mirroring code has been updated to correctly handle git
>>> repositories for which 'master' is not the default branch.
>>>
>>>
>>> Enjoy!
>>>
>>>
>>> [1]:https://github.com/gentoo-mirror/gentoo
>>> [2]:https://qa-reports.gentoo.org/output/repos/
>>
>> Sounds like a big improvement. I'm not too familiar with CI, so
>> forgive me if this is obvious: should overlays now use the 'stable'
>> branch as their primary remote, so they can ensure that any breakage
>> introduced is caused by their own overlay instead of a flub by us? If
>> so, then I wonder if this will help indirectly improve overlay quality.
>>
>> Being able to say "as of X commit, the tree is in good shape" is a big
>> deal imo. Thanks for working to make that happen. Is there anything
>> other devs can do to assist your development, or do we stick to the
>> usual 'if in doubt talk to QA'?
>> - -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> 
> 
> Well, my question is very basic. In the past, I've used euscan, the site
> on gentooexperimental.org, or app-portage/euscan the software to
> determine freshness of a package, for a variety of reasons. Is euscan
> going to be tied-in or at least coordinated with CI for consistency?
> 
> 
> James
> 
> 
> 
> 
No clue. I believe patrick maintains that, so he'd be the guy to ask
about that.

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Repo mirror & CI project news: 'stable' gentoo branch, new repo stats, faster CI

[james]

On 06/05/2016 12:15 PM, Daniel Campbell (zlg) wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On June 5, 2016 9:04:26 AM PDT, "Michał Górny" <mgorny@gentoo.org> wrote:
>> Hello, everyone.
>>
>> I have the pleasure to announce that a few improvements have been
>> deployed by the Repository mirror & CI project today.
>>
>>
>> 1. The mirror for 'gentoo' repository [1] now has a default 'stable'
>> branch. It is updated automatically by the gentoo-ci checker,
>> and therefore always contains the latest repository state that has been
>> confirmed 'green' by CI. While this is far from perfect, it's the first
>> step towards preventing major issues from being deployed on our users.
>>
>> If you are already using the mirror, you will need to either switch
>> branch manually, or re-add it.
>>
>>
>> 2. The repository QA report [2] has been extended with some repository
>> statistics. In particular, the timestamp of the newest commit
>> and the number of valid (that is, those not dying in global scope)
>> ebuilds are reported. Additionally, the homepage link is now included
>> as well.
>>
>> This enables the Overlays team members to easily check which
>> repositories
>> are unmaintained and/or empty, and handle the issues more efficiently.
>> It can also be useful to users who want to figure out whether there's
>> a point in using a particular repository.
>>
>>
>> 3. I've tried to optimize the logic used to run QA checks on
>> repositories, and I think I was able to even the load better.
>> Additionally, I've repacked the git repositories to get rid of huge
>> number of loose objects.
>>
>> As a result, CI now runs faster. The gentoo-ci runs are down from 10-12
>> minutes to 7-8 minutes, and pull requests from 16-18 minutes to 9-14
>> minutes. However, I don't have exact results yet as the server is still
>> busy removing old files ;-).
>>
>>
>> 4. Finally, the mirroring code has been updated to correctly handle git
>> repositories for which 'master' is not the default branch.
>>
>>
>> Enjoy!
>>
>>
>> [1]:https://github.com/gentoo-mirror/gentoo
>> [2]:https://qa-reports.gentoo.org/output/repos/
>
> Sounds like a big improvement. I'm not too familiar with CI, so forgive me if this is obvious: should overlays now use the 'stable' branch as their primary remote, so they can ensure that any breakage introduced is caused by their own overlay instead of a flub by us? If so, then I wonder if this will help indirectly improve overlay quality.
>
> Being able to say "as of X commit, the tree is in good shape" is a big deal imo. Thanks for working to make that happen. Is there anything other devs can do to assist your development, or do we stick to the usual 'if in doubt talk to QA'?
> - --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.


Well, my question is very basic. In the past, I've used euscan, the site 
on gentooexperimental.org, or app-portage/euscan the software to 
determine freshness of a package, for a variety of reasons. Is euscan 
going to be tied-in or at least coordinated with CI for consistency?


James