[gentoo-dev] [Proposal] Eclass for nodejs modules

[gentoo-dev] [Proposal] Eclass for nodejs modules

[Geaaru]

Hi guys,

I create an eclass that permit to create ebuild for nodejs modules
without define every times ebuild phases and avoid install of
dependencies already present on system.
My mission is create a module like perl-gpan that permit of create
automatically all ebuilds of every dependencies (not devDependencies)
of a particular package and npmv1 eclass if first step for this.

I hope that could be accepted by gentoo team because like for perl I
think that manage nodejs modules as gentoo package is more clear that
use directly npm features.

Last version of npmv1.eclass is available on my overlay:
https://github.com/geaaru/geaaru_overlay/blob/master/eclass/npmv1.eclas
s

Thanks in advance.

geaaru

P.S. An example of ebuild that use this eclass is visible on my
overlay:
https://github.com/geaaru/geaaru_overlay/blob/master/dev-node/formidabl
e/formidable-1.0.17.ebuild

Re: [gentoo-dev] [Proposal] Eclass for nodejs modules

[Andrew Udvare]

> On 2016-02-29, at 01:38, Geaaru <geaaru@gmail.com> wrote:
> 
> Hi guys,
> 
> I create an eclass that permit to create ebuild for nodejs modules
> without define every times ebuild phases and avoid install of
> dependencies already present on system.
> My mission is create a module like perl-gpan that permit of create
> automatically all ebuilds of every dependencies (not devDependencies)
> of a particular package and npmv1 eclass if first step for this.

http://search.gmane.org/?query=nodejs&group=gmane.linux.gentoo.devel

Unfortunately the conversation has already happened and it has sort of been determined that the community (even worse than Go) is terrible at defining dependencies for a lot of projects, even major ones. Personally for ebuilds I'm only interested in the end user packages that often documentation says to install using `npm install -g` (which I never do); packages that are useful for command line generally. I have resorted to using nvm for a few reasons: ease of use, compatibility, testing, and the fact that nodejs is not slotted in the Portage tree. Even if Gentoo had slots for nodejs/npm, it would not make it easier to test against different versions the way nvm does. In particular with time some versions would go away before other distros upgrade.

Your eclass kind of does something, as far as i can tell, which is very discouraged. Line 84 shows it downloading the dependencies with npm install to create sort of a separate root of the package, and then you generate a script to launch with the correct environment variables (and yes you avoid getting stuff already on the system). It's like nvm but done with Portage, but for each package. Users are not in control of the ebuild if they cannot specify how things get downloaded (make.conf can customise the command). NPM_DEFAULT_OPTS does not document how one could add proxy options. Even if you undid the npm install by invoking the user's command to download the packages, it would still not be acceptable as packages must be able to live in distfiles. I also do not see what you are going to do when package1 is already installed depending on dep1 version 1.2 but now package2 wants to be installed wanting dep1 on version 2.2. Automagic slotting?

> 
> ...like for perl I think that manage nodejs modules as gentoo package is more clear than use directly npm features.

I kind of disagree. The current state of nodejs packaging just does not allow for easy external packaging except for basically what you are doing which is using npm install, or vendoring every package to avoid conflicts.

You must get the individual packages and set up separate ebuilds and they must be able to share dependencies in most cases. The problem is the contents of package.json is often to just get 'the latest' version of every dependency or extremely incompatible versions. The community is moving too quickly for Portage to always be up-to-date with this.

The same problem exists with Java packages that use Maven for dependency management. It is almost as bad as Go, but the importance of some Go packages (Docker) is outweighing the complexity of reverse-engineering the `go get` mentality of that community. They basically do not want users to not use `go get` if it's available. They really want users to stick to their language-specific package managers. As a result of the Go unpredictability with versions, there are plenty of projects that 'vendor' their dependencies including Docker.

https://gist.github.com/datagrok/8577287

In particular, Python and Ruby seem to have settled versioning a lot more. Most packages do not specify Git repositories @ HEAD or any strange things like that. Releases get made in a predictable manner for the popular packages, because the consumers demand stability and predictability. (Ruby is not non-guilty, take a look at Vagrant.)

Also line 236:

npm_pkg_mods=( $(ls --color=none node_modules/) )

This should be npm_pkg_mods=( node_modules/* ) (and then call basename on each when iterating).

Andrew

Re: [gentoo-dev] [Proposal] Eclass for nodejs modules

[Geaaru]

Hi Andrew,

thanks for reply. My comment hereinafter.


On Mon, 2016-02-29 at 02:12 -0800, Andrew Udvare wrote:
> > On 2016-02-29, at 01:38, Geaaru <geaaru@gmail.com> wrote:
> > 
> > Hi guys,
> > 
> > I create an eclass that permit to create ebuild for nodejs modules
> > without define every times ebuild phases and avoid install of
> > dependencies already present on system.
> > My mission is create a module like perl-gpan that permit of create
> > automatically all ebuilds of every dependencies (not
> > devDependencies)
> > of a particular package and npmv1 eclass if first step for this.
> 
> http://search.gmane.org/?query=nodejs&group=gmane.linux.gentoo.devel
> 
> Unfortunately the conversation has already happened and it has sort
> of been determined that the community (even worse than Go) is
> terrible at defining dependencies for a lot of projects, even major
> ones. Personally for ebuilds I'm only interested in the end user
> packages that often documentation says to install using `npm install
> -g` (which I never do); packages that are useful for command line
> generally. I have resorted to using nvm for a few reasons: ease of
> use, compatibility, testing, and the fact that nodejs is not slotted
> in the Portage tree. Even if Gentoo had slots for nodejs/npm, it
> would not make it easier to test against different versions the way
> nvm does. In particular with time some versions would go away before
> other distros upgrade.
> 
> Your eclass kind of does something, as far as i can tell, which is
> very discouraged. Line 84 shows it downloading the dependencies with
> npm install to create sort of a separate root of the package, and
> then you generate a script to launch with the correct environment
> variables (and yes you avoid getting stuff already on the system).
> It's like nvm but done with Portage, but for each package. Users are
> not in control of the ebuild if they cannot specify how things get
> downloaded (make.conf can customise the command). NPM_DEFAULT_OPTS
> does not document how one could add proxy options. Even if you undid
> the npm install by invoking the user's command to download the
> packages, it would still not be acceptable as packages must be able
> to live in distfiles. I also do not see what you are going to do when
> package1 is already installed depending on dep1 version 1.2 but now
> package2 wants to be installed wanting dep1 on version 2.2. Automagic
> slotting?

The idea is that as first step I create manually dependencies of every
package/ ebuild and where is available a system package I use
NPM_SYSTEM_MODULES to exclude module from install. I know, it isn't a
fine solution but as I wrote I want create a module that try to create
ebuild automatically from package.json and when I do that I can avoid
to use npm install on src_prepare phase.
For now on src_prepare I use npm install to download dependencies with
exact version match (I add -E option on NPM_DEFAULT_OPTS for do this).
About your question for dependencies conflits (and slotting) is equal
to a normal package. See you packages about Openstack project where
Juno release has a lot of dependencies restrictions like package1
>=1.2, package <= 2.2.
Yes, I know that is not easy to maintain ebuilds upgraded but like
happens now in a lot of gentoo ebuilds, if a project is not more
maintained (and so it isn't usable with more updated dependencies) is
removed from portage.


> 
> > 
> > ...like for perl I think that manage nodejs modules as gentoo
> > package is more clear than use directly npm features.
> 
> I kind of disagree. The current state of nodejs packaging just does
> not allow for easy external packaging except for basically what you
> are doing which is using npm install, or vendoring every package to
> avoid conflicts.
> 
> You must get the individual packages and set up separate ebuilds and
> they must be able to share dependencies in most cases. The problem is
> the contents of package.json is often to just get 'the latest'
> version of every dependency or extremely incompatible versions. The
> community is moving too quickly for Portage to always be up-to-date
> with this.

I do not agree with this, I think that manage nodejs module inside
Portage is possible, not easy, but possible and more easy to manage
instead of npm command.
There is same problem with perl modules, CPAN normally has more updated
modules but currently ebuild about perl modules are present.
About your comment for 'the latest', normally FWIS dependencies are
with version with (~) that it's means get the latest or same version,
but with -E option to npm command you can force use of same versions.

> 
> The same problem exists with Java packages that use Maven for
> dependency management. It is almost as bad as Go, but the importance
> of some Go packages (Docker) is outweighing the complexity of
> reverse-engineering the `go get` mentality of that community. They
> basically do not want users to not use `go get` if it's available.
> They really want users to stick to their language-specific package
> managers. As a result of the Go unpredictability with versions, there
> are plenty of projects that 'vendor' their dependencies including
> Docker.
> 
> https://gist.github.com/datagrok/8577287
> 
> In particular, Python and Ruby seem to have settled versioning a lot
> more. Most packages do not specify Git repositories @ HEAD or any
> strange things like that. Releases get made in a predictable manner
> for the popular packages, because the consumers demand stability and
> predictability. (Ruby is not non-guilty, take a look at Vagrant.)
> 
> Also line 236:
> 
> npm_pkg_mods=( $(ls --color=none node_modules/) )
> 
> This should be npm_pkg_mods=( node_modules/* ) (and then call
> basename on each when iterating).

Thank you for your suggestion, but with npm_pkg_mods=( node_modules/* )
if there aren't directories under node_modules on variable is then
present an entry with node_modules/*, instead if I use ls an empty row
is returned.

> 
> Andrew
> 
> 

In conclusion, it seems that is not accepted use of nodejs modules
ebuild inside portage. It is right?

Re: [gentoo-dev] [Proposal] Eclass for nodejs modules

[Andrew Udvare]

On 29/02/16 03:23, Geaaru wrote:
> 
> In conclusion, it seems that is not accepted use of nodejs modules
> ebuild inside portage. It is right?
> 
> 
There used to be a CoffeeScript ebuild if you search back. I do not
remember how it worked but IIRC I think it was basically just creating a
fake root for itself. The distfile itself was just that node_modules/
directory in a tarball. No direction on how those dependencies could be
shared, so it is no longer in the tree.

Andrew

Re: [gentoo-dev] [Proposal] Eclass for nodejs modules

[Michael Orlitzky]

On 02/29/2016 06:24 PM, Andrew Udvare wrote:
> On 29/02/16 03:23, Geaaru wrote:
>>
>> In conclusion, it seems that is not accepted use of nodejs modules
>> ebuild inside portage. It is right?
>>
>>
> There used to be a CoffeeScript ebuild if you search back. I do not
> remember how it worked but IIRC I think it was basically just creating a
> fake root for itself. The distfile itself was just that node_modules/
> directory in a tarball. No direction on how those dependencies could be
> shared, so it is no longer in the tree.
> 

I really needed it for one program, so I'm maintaining coffee-script in
the tree now as dev-lang/coffee-script. I have an eclass and some
ebuilds using it at https://github.com/orlitzky/npm but I didn't want to
wind up in charge of a NodeJS project and nobody else did either, so I'm
no longer working on it. The current coffee-script ebuild copy/pastes
some of the stuff that would have been handled by the eclass.

Only misery lies down this path. Every upstream project is a mess and
nobody cares, because implicitly, their roadmaps all involve realizing
that javascript is awful and abandoning the project. To get this stuff
in the tree, we would need a full-time devoted developer who just loves
javascript and is sexually aroused by having his time wasted.

Re: [gentoo-dev] [Proposal] Eclass for nodejs modules

[Jesus Rivero (Neurogeek)]

[-- Attachment #1: Type: text/plain, Size: 1806 bytes --]

On Mon, Feb 29, 2016 at 7:00 PM, Michael Orlitzky <mjo@gentoo.org> wrote:

> On 02/29/2016 06:24 PM, Andrew Udvare wrote:
> > On 29/02/16 03:23, Geaaru wrote:
> >>
> >> In conclusion, it seems that is not accepted use of nodejs modules
> >> ebuild inside portage. It is right?
> >>
> >>
> > There used to be a CoffeeScript ebuild if you search back. I do not
> > remember how it worked but IIRC I think it was basically just creating a
> > fake root for itself. The distfile itself was just that node_modules/
> > directory in a tarball. No direction on how those dependencies could be
> > shared, so it is no longer in the tree.
> >
>
> I really needed it for one program, so I'm maintaining coffee-script in
> the tree now as dev-lang/coffee-script. I have an eclass and some
> ebuilds using it at https://github.com/orlitzky/npm but I didn't want to
> wind up in charge of a NodeJS project and nobody else did either, so I'm
> no longer working on it. The current coffee-script ebuild copy/pastes
> some of the stuff that would have been handled by the eclass.
>
> Only misery lies down this path.

Couldn't have said it better. Many of us have tried:
https://github.com/neurogeek/g-npm
https://github.com/neurogeek/gentoo-overlay/tree/master/dev-nodejs

npm just doesn't care about "global" installs, so it is broken by design
for any centralized package manager:
https://github.com/gruntjs/grunt/issues/438

Just use npm.


> Every upstream project is a mess and
> nobody cares, because implicitly, their roadmaps all involve realizing
> that javascript is awful and abandoning the project. To get this stuff
> in the tree, we would need a full-time devoted developer who just loves
> javascript and is sexually aroused by having his time wasted.
>
>
>


-- 
Jesus Rivero (Neurogeek)
Gentoo Developer

[-- Attachment #2: Type: text/html, Size: 3003 bytes --]