[gentoo-dev] paper on oss-qm project

[gentoo-dev] paper on oss-qm project

[Enrico Weigelt]

hi folks,

just in case anybody's interested:

I've written a little paper on the OSS-QM project, which aims to
provide fixed sourcetrees to many packages+versions and so offload
much of the QM/patching work from individual distros to a common
place:

http://www.metux.de/download/oss-qm-project-2010050101.pdf

cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [gentoo-dev] paper on oss-qm project

[Sebastian Pipping]

hello enrico,


interesting concept.  i'd like to comment on a few details:

 - licensing seems not be addressed, yet.
   licensing can kill everything, it needs consideration.

 - branch and tag namespaces as currently defined have a few problems:

   - versioning:

     - the A.B.C.D scheme won't be fun to gentoo, both
       due to no-letters-in-here and because of no-pre-releases.
       while at that keeping pre-releases does not seem helpful to me.

   - vendor concept:

     - uppercase vendor names look rather odd, especially with project
       names in lowercase.

     - having the vendor first makes no sense to me.
       a "package.vendor.subbranch" keeps all zlibs together,
       instead of all gentoo stuff.  if the project is about
       packages, that makes more sense to me.

     - renaming the concept to "downstream" would make it
       fit better.  gentoo is not a vendor to me.

 - with one git repo used for many packages people
   will need to know how to clone single branches only.
   most git users probably won't, you will need to teach them.
   the PDF seems a good place to do that.

hope to see you on linuxtag berlin 2010, best,




sebastian

Re: [gentoo-dev] paper on oss-qm project

[Enrico Weigelt]

* Sebastian Pipping <sping@gentoo.org> schrieb:

Hi,

> interesting concept.  i'd like to comment on a few details:
> 
>  - licensing seems not be addressed, yet.
>    licensing can kill everything, it needs consideration.

what problems do you see w/ licensing ?

IMHO, each branch simply has to follow the upstream's license.

>  - branch and tag namespaces as currently defined have a few problems:
> 
>    - versioning:
> 
>      - the A.B.C.D scheme won't be fun to gentoo, both
>        due to no-letters-in-here and because of no-pre-releases.
>        while at that keeping pre-releases does not seem helpful to me.

simply normalize: don't use letters but numbers. and I actually don't
see any need for pre-releases: 

a) it' an real release - then it has to fit into the (linear) 
   versioning scheme
b) it's not really a release but just a development snapshot - 
   that doesnt belong into the main oss-qm repository
 
>    - vendor concept:
> 
>      - uppercase vendor names look rather odd, especially with project
>        names in lowercase.
> 
>      - having the vendor first makes no sense to me.
>        a "package.vendor.subbranch" keeps all zlibs together,
>        instead of all gentoo stuff.  if the project is about
>        packages, that makes more sense to me.

I've chosen that scheme to make the borders more clear (also for
automatic filtering, etc). In my concept, the vendor is the major
point of distinction, package comes at second, ... 

>      - renaming the concept to "downstream" would make it
>        fit better.  gentoo is not a vendor to me.

Well, the term vendor here is defined as a party which provides
packages in certain variants. "UPSTREAM" is a kind of meta vendor,
describing the upstreams. "Vendor" is IMHO more generic, since there
may be vendors who aren't actually a real distro. For example, I 
myself don't publish a complete distro, but a foundation for clean
building especially for special embedded devices or appliances.

>  - with one git repo used for many packages people
>    will need to know how to clone single branches only.
>    most git users probably won't, you will need to teach them.
>    the PDF seems a good place to do that.

Yes, that's still an open topic. I've chosen to use one big repo
for easier maintenance, but I'm aware of the problem that the
repo might become very fat some day. I see two options:

a) split it off into several ones, eg. on per-package basis
   and create a system for (semi-)automatic mass-repo maintenance
   (not completely trivial when using free git hosters as mirrors)

b) add an selective filtering system. AFIAK current stable git
   doesnt provide that yet - I've added an little patch for that:
   http://repo.or.cz/w/oss-qm-packages.git/shortlog/refs/heads/METUX.git.master
   

cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [gentoo-dev] paper on oss-qm project

[Sebastian Pipping]

On 05/08/10 22:11, Enrico Weigelt wrote:
> what problems do you see w/ licensing ?
> 
> IMHO, each branch simply has to follow the upstream's license.

i have yet to see easy cases with licensing.
i haven't thought about it in detail yet, tough, to be honest.


> simply normalize: don't use letters but numbers.

i don't believe in simple normalization before i have seen it.


> b) it's not really a release but just a development snapshot - 
>    that doesnt belong into the main oss-qm repository

why doesn't it belong in there?


> I've chosen that scheme to make the borders more clear (also for
> automatic filtering, etc). In my concept, the vendor is the major
> point of distinction, package comes at second, ... 

i guess we agree to disagree then.
i don't think the current scheme promotes cooperation well.


> Well, the term vendor here is defined as a party which provides
> packages in certain variants. "UPSTREAM" is a kind of meta vendor,
> describing the upstreams. "Vendor" is IMHO more generic, since there
> may be vendors who aren't actually a real distro. For example, I 
> myself don't publish a complete distro, but a foundation for clean
> building especially for special embedded devices or appliances.

yes, that's why i proposed "downstream" as a replacement.
you don't consider yourself downstream?


> Yes, that's still an open topic. I've chosen to use one big repo
> for easier maintenance, but I'm aware of the problem that the
> repo might become very fat some day.

my point is not about size, only about "users".


> I see two options:
> 
> a) split it off into several ones, eg. on per-package basis
>    and create a system for (semi-)automatic mass-repo maintenance
>    (not completely trivial when using free git hosters as mirrors)

are you aware that splitting it up will reduce the savings in space?
say if they all had byte-identical GPLv3 COPYING files that would be one
blob atm and N blobs in split mode.


> b) add an selective filtering system. AFIAK current stable git
>    doesnt provide that yet - I've added an little patch for that:
>    http://repo.or.cz/w/oss-qm-packages.git/shortlog/refs/heads/METUX.git.master

while i'm not sure about this in detail yet, could it be this loop
misses to filter the very first entry?

+       while (walk && (walk->next))
+       {
+               if (_filter_remote_ref(transport, walk->next))
+                       walk->next = walk->next->next;
+               else
+                       walk = walk->next;
+       }
+

best,




sebastian

Re: [gentoo-dev] paper on oss-qm project

[Enrico Weigelt]

* Sebastian Pipping <sping@gentoo.org> schrieb:

Hi,

> > what problems do you see w/ licensing ?
> > 
> > IMHO, each branch simply has to follow the upstream's license.
> 
> i have yet to see easy cases with licensing.
> i haven't thought about it in detail yet, tough, to be honest.

well, let's just see if the first realworld case happens and then
think about it ;-p
  
> > simply normalize: don't use letters but numbers.
> 
> i don't believe in simple normalization before i have seen it.

well, the right normalization scheme always depends on the upstream's
versioning scheme. most packages already have some consistent scheme,
which can be mapped directly. the few really complicated cases
(actually don't have any at the tip of my head right now) have
to be done manually. remember that we only take stable releases,
since we're in the dist maintainer role - not the upstream dev one
(so, alpha's etc dont matter here)

> > b) it's not really a release but just a development snapshot - 
> >    that doesnt belong into the main oss-qm repository
> 
> why doesn't it belong in there?

because it's not ready to use. that's the primary distinction:
snapshots are for devs (and maybe testers), not for production use.

> > I've chosen that scheme to make the borders more clear (also for
> > automatic filtering, etc). In my concept, the vendor is the major
> > point of distinction, package comes at second, ... 
> 
> i guess we agree to disagree then.
> i don't think the current scheme promotes cooperation well.

why exactly ?

BTW: if you dont like that scheme, you could add some filter for
automatically rewrites the refnames ;-p

> > Well, the term vendor here is defined as a party which provides
> > packages in certain variants. "UPSTREAM" is a kind of meta vendor,
> > describing the upstreams. "Vendor" is IMHO more generic, since there
> > may be vendors who aren't actually a real distro. For example, I 
> > myself don't publish a complete distro, but a foundation for clean
> > building especially for special embedded devices or appliances.
> 
> yes, that's why i proposed "downstream" as a replacement.
> you don't consider yourself downstream?

*I* am downstream, right. But the "UPSTREAM"+* branches are what's 
coming from the upstream. Upstream's a special kind of (meta-)vendor,
where everybody else (downstreams) forkes from.

> > Yes, that's still an open topic. I've chosen to use one big repo
> > for easier maintenance, but I'm aware of the problem that the
> > repo might become very fat some day.
> 
> my point is not about size, only about "users".

In which way does my current approach make trouble here ?
What would be the better approach ? 

> > I see two options:
> > 
> > a) split it off into several ones, eg. on per-package basis
> >    and create a system for (semi-)automatic mass-repo maintenance
> >    (not completely trivial when using free git hosters as mirrors)
> 
> are you aware that splitting it up will reduce the savings in space?
> say if they all had byte-identical GPLv3 COPYING files that would be one
> blob atm and N blobs in split mode.

Right, that's the tradeoff here. But the few COPYING files shouldnt be
the big issue ...

> > b) add an selective filtering system. AFIAK current stable git
> >    doesnt provide that yet - I've added an little patch for that:
> >    http://repo.or.cz/w/oss-qm-packages.git/shortlog/refs/heads/METUX.git.master
> 
> while i'm not sure about this in detail yet, could it be this loop
> misses to filter the very first entry?
> 
> +       while (walk && (walk->next))
> +       {
> +               if (_filter_remote_ref(transport, walk->next))
> +                       walk->next = walk->next->next;
> +               else
> +                       walk = walk->next;
> +       }
> +

you missed the previous lines ;-P

+       while ((transport->remote_refs) && (_filter_remote_ref(transport, transport->remote_refs)))
+               transport->remote_refs = transport->remote_refs->next;
+

cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------