[gentoo-user] npm: ERR! cb() never called!

[gentoo-user] npm: ERR! cb() never called!

[Alan McKinnon]

Anyone here familiar with driving nodejs and npm?

I'm trying to write an ebuild for a musicbrainz mirror server and "npm
install" keeps erroring out with one of two errors:

1. The install does finish but npm doesn't get around to exiting,
verified by stopping the emerge, running npm install manually and seeing
that it does nothing. When it stalls strace shows the last call was
(poll,<number>    ), which implies a race condition.

2. More and more often now I get the dreaded "npm: ERR! cb() never
called!" error message which Google and stackoverflow say has been an
ongoing issue for 3 years now. If I keep retrying it eventually
succeeds, implying a race condition of some sort.

I do this in src_prepare():  [1]
src_prepare() {
    npm install npm || die
    npm install || die
}
following the lead of the ony other ebuild in the tree that does npm
install - bokeh-0.7.1

The package.json file from musicbrainz.com is this, it looks OK to me:

{
  "name": "musicbrainz-server",
  "version": "0.0.0",
  "description": "package.json for keeping track of musicbrainz-server
nodejs dependencies",
  "repository": {
    "type": "git",
    "url": "https://github.com/metabrainz/musicbrainz-server.git"
  },
  "license": "GPLv2+",
  "dependencies": {
    "aclass": "0.5.1",
    "babel": "^5.8.21",
    "babel-core": "^5.8.22",
    "babelify": "^6.1.3",
    "balanced-match": "0.2.0",
    "cookie": "^0.1.2",
    "envify": "^3.4.0",
    "filesize": "2.0.4",
    "gulp": "^3.8.7",
    "gulp-less": "^3.0.0",
    "gulp-rev": "^2.0.1",
    "gulp-streamify": "^0.0.5",
    "immutable": "3.6.4",
    "jed": "1.1.0",
    "jquery": "1.11.2",
    "jquery.browser": "gabceb/jquery-browser-plugin#e4a01fd",
    "knockout": "mwiencek/knockout#a8f12df",
    "knockout-arraytransforms": "^2.0.0",
    "less-plugin-clean-css": "^1.4.0",
    "leven": "^1.0.2",
    "lodash": "^3.9.3",
    "parse-stack": "0.1.3",
    "po2json": "^0.3.2",
    "q": "^1.1.1",
    "rcss": "0.1.4",
    "react": "0.13.1",
    "shelljs": "^0.3.0",
    "tablesorter": "Mottie/tablesorter#430f8c5",
    "through2": "^0.6.1",
    "uglifyify": "^3.0.1",
    "vinyl": "^0.4.6",
    "vinyl-source-stream": "^1.0.0",
    "yarb": "^0.4.4"
  },
  "devDependencies": {
    "eslint": "^0.24.0",
    "eslint-plugin-react": "^2.6.4",
    "gulp-watch": "^4.2.1",
    "tape": "^4.0.0"
  },
  "private": true
}


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] npm: ERR! cb() never called!

[Michael Orlitzky]

On 09/17/2015 10:03 AM, Alan McKinnon wrote:
> Anyone here familiar with driving nodejs and npm?
> 
> I'm trying to write an ebuild for a musicbrainz mirror server and "npm
> install" keeps erroring out with one of two errors:
> 
> 1. The install does finish but npm doesn't get around to exiting,
> verified by stopping the emerge, running npm install manually and seeing
> that it does nothing. When it stalls strace shows the last call was
> (poll,<number>    ), which implies a race condition.
> 
> 2. More and more often now I get the dreaded "npm: ERR! cb() never
> called!" error message which Google and stackoverflow say has been an
> ongoing issue for 3 years now. If I keep retrying it eventually
> succeeds, implying a race condition of some sort.
> 

I went through this wonderful experience a few weeks ago. You're not
allowed to access the network in src_prepare, so that might be
contributing to your weirdness. I came up with two options:

1) Run `npm install` on your dev machine, and then package up the result
as a tarball. Generate the manifest from the tarball, and then in your
src_install, just copy stuff over.

  src_install(){
      local npm_module_dir="/usr/$(get_libdir)/node/${PN}"
      insinto "${npm_module_dir}"
      doins -r whatever
      ...
  }


This is the lazy way, but avoids you having to package 1,000 other
things all written by people who just "learned to code" by googling HOW
DO I HTML5.

2) The right way to do it is to use an eclass and install all of the
dependencies using separate packages. As you can imagine, this is a
nightmare if you have more than a few dependencies (looks like you do).

I started an eclass for npm packages. I left the overlay here:

  https://github.com/orlitzky/npm

but no one else seemed interested in having it in-tree, and the whole
ecosystem is kind of scary to me anyway.

So, for the large package I need, I'm doing it the lazy way: npm install
on my machine, and make an ebuild for the resulting huge tarball.

Re: [gentoo-user] npm: ERR! cb() never called!

[Alan McKinnon]

On 17/09/2015 20:50, Michael Orlitzky wrote:
> On 09/17/2015 10:03 AM, Alan McKinnon wrote:
>> Anyone here familiar with driving nodejs and npm?
>>
>> I'm trying to write an ebuild for a musicbrainz mirror server and "npm
>> install" keeps erroring out with one of two errors:
>>
>> 1. The install does finish but npm doesn't get around to exiting,
>> verified by stopping the emerge, running npm install manually and seeing
>> that it does nothing. When it stalls strace shows the last call was
>> (poll,<number>    ), which implies a race condition.
>>
>> 2. More and more often now I get the dreaded "npm: ERR! cb() never
>> called!" error message which Google and stackoverflow say has been an
>> ongoing issue for 3 years now. If I keep retrying it eventually
>> succeeds, implying a race condition of some sort.
>>
> 

Hi Michael,

I reached pretty much the same conclusions as you. And yes, it is
src_prepare() not allowing network access that caused most of the
weirdness. I must have had the packages cached causing them to install
correctly the first time. I've since cleaned the npm cache and of course
now they fail.

After I posted my mail I went through the npm bugs at bgo and the
unstated message from the gentoo devs is loud and clear - "you are not
putting that shit in the tree"

So I took the easiest possible way out: deleted src_prepare() and issued
an elog to "cd $inst_dir && npm install, and do not use -g"

> I went through this wonderful experience a few weeks ago. You're not
> allowed to access the network in src_prepare, so that might be
> contributing to your weirdness. I came up with two options:
> 
> 1) Run `npm install` on your dev machine, and then package up the result
> as a tarball. Generate the manifest from the tarball, and then in your
> src_install, just copy stuff over.
> 
>   src_install(){
>       local npm_module_dir="/usr/$(get_libdir)/node/${PN}"
>       insinto "${npm_module_dir}"
>       doins -r whatever
>       ...
>   }
> 
> 
> This is the lazy way, but avoids you having to package 1,000 other
> things all written by people who just "learned to code" by googling HOW
> DO I HTML5.

Agreed. The quality of node software is atrocious, and the package
manager is even worse. read "npm faq", it is most enlightening and gives
insight into people's heads....

> 
> 2) The right way to do it is to use an eclass and install all of the
> dependencies using separate packages. As you can imagine, this is a
> nightmare if you have more than a few dependencies (looks like you do).

Errr, no :-)

g-cpan is bad enough and those ebuilds are mostly template-able. At
least CPAN modules mostly respond correctly to make && make install.
That node stuff doesn't look like it will ever package sanely.

> 
> I started an eclass for npm packages. I left the overlay here:
> 
>   https://github.com/orlitzky/npm
> 
> but no one else seemed interested in having it in-tree, and the whole
> ecosystem is kind of scary to me anyway.

Indeed.


> 
> So, for the large package I need, I'm doing it the lazy way: npm install
> on my machine, and make an ebuild for the resulting huge tarball.

Thanks for the feedback

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] npm: ERR! cb() never called!

[Alec Ten Harmsel]

On Thu, Sep 17, 2015 at 09:24:39PM +0200, Alan McKinnon wrote:
> On 17/09/2015 20:50, Michael Orlitzky wrote:
> > On 09/17/2015 10:03 AM, Alan McKinnon wrote:
> >> Anyone here familiar with driving nodejs and npm?
> >>
> >> I'm trying to write an ebuild for a musicbrainz mirror server and "npm
> >> install" keeps erroring out with one of two errors:

Haha, npm. First time I ever ran npm, it required 3 runs before it
actually managed to fetch all the dependencies. Hopefully it's better
now.

> > 
> > 2) The right way to do it is to use an eclass and install all of the
> > dependencies using separate packages. As you can imagine, this is a
> > nightmare if you have more than a few dependencies (looks like you do).
> 
> Errr, no :-)
> 
> g-cpan is bad enough and those ebuilds are mostly template-able. At
> least CPAN modules mostly respond correctly to make && make install.
> That node stuff doesn't look like it will ever package sanely.

Unfortunately, the right way with nodejs/ruby web stuff is to use the
tooling specific to the language. If this[1] is what you're trying to
deploy, I feel sorry.

If I was serious about deploying this, I would:

1. Fork the repo and add a remote on my own server
2. Add your custom configuration
3. Write a small shell script that
  1. Runs `git pull` from your own infrastructure
  2. Installs perl/node deps locally
  3. Runs the gulp build
  4. Runs plackup
4. Add an init script that runs that start script

This sucks, but it seems to be the way a lot of web stuff is deployed
these days.

Hope this helps,

Alec

[1] https://github.com/metabrainz/musicbrainz-server

Re: [gentoo-user] npm: ERR! cb() never called!

[Alan McKinnon]

On 17/09/2015 22:53, Alec Ten Harmsel wrote:
> On Thu, Sep 17, 2015 at 09:24:39PM +0200, Alan McKinnon wrote:
>> On 17/09/2015 20:50, Michael Orlitzky wrote:
>>> On 09/17/2015 10:03 AM, Alan McKinnon wrote:
>>>> Anyone here familiar with driving nodejs and npm?
>>>>
>>>> I'm trying to write an ebuild for a musicbrainz mirror server and "npm
>>>> install" keeps erroring out with one of two errors:
> 
> Haha, npm. First time I ever ran npm, it required 3 runs before it
> actually managed to fetch all the dependencies. Hopefully it's better
> now.

Slightly OT, but the general idea of package management isn't hard.

Put the stuff you should have in a list, then compare what you should
have to what you do have. Go get and install what you don't have, then
make a record that you did it.

Everything needed to get these basics right has been known for 40 years
or more - fellows like Wirth and Dijkstra figured it all out way back when.

Sure, there's always modern stumbling blocks (like why we have subslots)
but that's extra to the essential basics.

So why oh why do the latest generation of wunderkinds (not) always get
it so completely WRONG? 3 runs to fetch all the deps? I suppose wget and
curl don't actually do what I think they do then....

> 
>>>
>>> 2) The right way to do it is to use an eclass and install all of the
>>> dependencies using separate packages. As you can imagine, this is a
>>> nightmare if you have more than a few dependencies (looks like you do).
>>
>> Errr, no :-)
>>
>> g-cpan is bad enough and those ebuilds are mostly template-able. At
>> least CPAN modules mostly respond correctly to make && make install.
>> That node stuff doesn't look like it will ever package sanely.
> 
> Unfortunately, the right way with nodejs/ruby web stuff is to use the
> tooling specific to the language. If this[1] is what you're trying to
> deploy, I feel sorry.

Yes, that's the one

> 
> If I was serious about deploying this, I would:
> 
> 1. Fork the repo and add a remote on my own server
> 2. Add your custom configuration
> 3. Write a small shell script that
>   1. Runs `git pull` from your own infrastructure
>   2. Installs perl/node deps locally
>   3. Runs the gulp build
>   4. Runs plackup
> 4. Add an init script that runs that start script

I followed that mostly except for forking the repo and writing a small
shell script - I much prefer proper ebuilds to hacky scripts. Even
though I'm a Linux sysadmin I hate ad-hoc shell scripts with a passion

> 
> This sucks, but it seems to be the way a lot of web stuff is deployed
> these days.
> 
> Hope this helps,
> 
> Alec
> 
> [1] https://github.com/metabrainz/musicbrainz-server
> 


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] npm: ERR! cb() never called!

[Michael Orlitzky]

On 09/17/2015 05:13 PM, Alan McKinnon wrote:
> 
> Slightly OT, but the general idea of package management isn't hard.
> 
> Put the stuff you should have in a list, then compare what you should
> have to what you do have. Go get and install what you don't have, then
> make a record that you did it.
> 
> Everything needed to get these basics right has been known for 40 years
> or more - fellows like Wirth and Dijkstra figured it all out way back when.
> 
> Sure, there's always modern stumbling blocks (like why we have subslots)
> but that's extra to the essential basics.
> 
> So why oh why do the latest generation of wunderkinds (not) always get
> it so completely WRONG? 3 runs to fetch all the deps? I suppose wget and
> curl don't actually do what I think they do then....
> 

Heavy on bad words and light on solutions, but it made me feel better:

http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php

Re: [gentoo-user] npm: ERR! cb() never called!

[Alan McKinnon]

On 18/09/2015 00:48, Michael Orlitzky wrote:
> On 09/17/2015 05:13 PM, Alan McKinnon wrote:
>>
>> Slightly OT, but the general idea of package management isn't hard.
>>
>> Put the stuff you should have in a list, then compare what you should
>> have to what you do have. Go get and install what you don't have, then
>> make a record that you did it.
>>
>> Everything needed to get these basics right has been known for 40 years
>> or more - fellows like Wirth and Dijkstra figured it all out way back when.
>>
>> Sure, there's always modern stumbling blocks (like why we have subslots)
>> but that's extra to the essential basics.
>>
>> So why oh why do the latest generation of wunderkinds (not) always get
>> it so completely WRONG? 3 runs to fetch all the deps? I suppose wget and
>> curl don't actually do what I think they do then....
>>
> 
> Heavy on bad words and light on solutions, but it made me feel better:
> 
> http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php

I read it, and now I also feel much better :-)


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] npm: ERR! cb() never called!

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

On Fri, 18 Sep 2015 08:57:55 +0200, Alan McKinnon wrote:

> > Heavy on bad words and light on solutions, but it made me feel better:
> > 
> > http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php  
> 
> I read it, and now I also feel much better :-)

+1

It's a shame that those it is aimed at will either never read it or read
it and think it doesn't apply to them :(


-- 
Neil Bothwick

Oxymoron: Reagan memoirs.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[gentoo-user] Re: npm: ERR! cb() never called!

[James]

Michael Orlitzky <mjo <at> gentoo.org> writes:


> Heavy on bad words and light on solutions, but it made me feel better:

> http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php


You know, when I read this I got quite a chuckle.  Then I started to 
'marinate' on this a while and stupid little things started to just
pop up in the back of my head. I've been trying to forget this post
ever since I read it; it has a bit too much truth to it, I suppose.
Then  a bombshell hit me on a very big FOSS project  (no do not ask the 
name) that I could not believe I was reading.


It seems they  have, somehow murky, all sorts of release tarballs with
proper identification, if you can find them with major.minor-revision
numbers, like we use in gentoo. But they pow_wow, pick the best one and then
release it with only major release numbers, although there are many, maybe
up to 5 or 6 tarballs floating around from the various work products
that should have the minor-rev info in the dam name. 


So just spank my ass and send me home from coding school. How the hell can
they do that, and maintain any resemblance to credibility? Deep in the
ticket system you can find the details. Seriously, it's a joke and now
there's some finger pointing amongst some big names in 'open source'
development. I guess they just want folks to download major-distro binaries
and bolt ons to the various distros. What ends up in the distro binary
downloads is even a further drift from the sources. Some serious smoke and
mirrors.

So, Micheal's profanity, although not optimal, is ceratinly warranted.
The more I hack and write code the more I'm realizing that either idiocy
or chicanery is a mainstay of some major FOSS projects and the big distros?
  Gut-wrencing experience, to say the least. I just can't believe folks this
smart that can code are such a bunch of fools............ Package manager?
These folks need a class in manners, ethics and common sense.   


Rant on, Michael, Rant on!


hth,
James

Re: [gentoo-user] npm: ERR! cb() never called!

[Alec Ten Harmsel]

On Thu, Sep 17, 2015 at 11:13:10PM +0200, Alan McKinnon wrote:
> On 17/09/2015 22:53, Alec Ten Harmsel wrote:
> > Unfortunately, the right way with nodejs/ruby web stuff is to use the
> > tooling specific to the language. If this[1] is what you're trying to
> > deploy, I feel sorry.
> 
> Yes, that's the one

Yay, perl, perl6, and JS. All at the same time.

> > 
> > If I was serious about deploying this, I would:
> > 
> > 1. Fork the repo and add a remote on my own server
> > 2. Add your custom configuration
> > 3. Write a small shell script that
> >   1. Runs `git pull` from your own infrastructure
> >   2. Installs perl/node deps locally
> >   3. Runs the gulp build
> >   4. Runs plackup
> > 4. Add an init script that runs that start script
> 
> I followed that mostly except for forking the repo and writing a small
> shell script - I much prefer proper ebuilds to hacky scripts. Even
> though I'm a Linux sysadmin I hate ad-hoc shell scripts with a passion

I am the same way; I do not like shell scripts. Anything longer than 15
lines or so is written in Ruby, other than my firewall setup script.

I'm not proud of myself for suggesting what I did; just trying to
minimize your pain. Another option could be something along the lines
of:

1. ebuild installs files to `/var/musicbrainz` or wherever
2. Add a small shell script to `/var/musicbrainz` or wherever that:
  1. Installs perl/node deps locally
  2. Runs the gulp build
  3. Runs plackup
3. Add an init script that runs that start script

I guess, since you're not developing musicbrainz thingy, that the git
repo was a bit of overkill.

> > This sucks, but it seems to be the way a lot of web stuff is deployed
> > these days.

All these dynamic languages suffer from the fun problem that developers
that don't write enough tests have essentially no guarantee that their
code actually parses. I can't count how many times I've run Ruby,
Python, or Bash scripts only to have 'variable not found' or type
errors; all of the things that compilers do really well[1].

One of the things that results from this (IMO) is that they bundle
deps/enforce strict versions on stuff because they have to just to run
stuff. For example, a while ago www-apps/jekyll was broken because a gem
it depended on happened to be a newer version and changed the API enough
that jekyll broke. To be fair to the Gentoo developers, jekyll is
~amd64, so I didn't really care.

Alec

[1] For example, below are two *valid* files; one Python, one Ruby. Both
would obviously fail to compile if it was transposed to C/C++/Java, for
good reason.

#!/usr/bin/env ruby
if false
  puts hey
else
  puts 'hey'
end


#!/usr/bin/env python
if False:
    print(hey)
else:
    print('hey')

Re: [gentoo-user] npm: ERR! cb() never called!

[Alan McKinnon]

On 18/09/2015 05:34, Alec Ten Harmsel wrote:
>>> This sucks, but it seems to be the way a lot of web stuff is deployed
>>> > > these days.
> All these dynamic languages suffer from the fun problem that developers
> that don't write enough tests have essentially no guarantee that their
> code actually parses. I can't count how many times I've run Ruby,
> Python, or Bash scripts only to have 'variable not found' or type
> errors; all of the things that compilers do really well[1].


I well remember flameeyes' bitching about packaging ruby gems. One
problem kept coming up over and over: tests would fail, usually because
the dev used some code buried deep in his own workstation, and that HAD
NEVER BEEN RELEASED. Braindeaddeaddead

-- 
Alan McKinnon
alan.mckinnon@gmail.com