public inbox for gentoo-user@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-user] post build files
@ 2014-09-09 15:20 James
  2014-09-09 16:23 ` Alec Ten Harmsel
  2014-09-09 17:41 ` [gentoo-user] " Rich Freeman
  0 siblings, 2 replies; 7+ messages in thread
From: James @ 2014-09-09 15:20 UTC (permalink / raw
  To: gentoo-user

Hello,

I'm rather new to hacking ebuilds. I have read most every doc
I can find on the subject. One thing I'm looking for is a post-build
document that shows me the path/name of everything built.

So for mesos-0.20.0 it seems to be:

/var/db/pkg/sys-cluster/mesos-0.20.0/CONTENTS

For some other packages, they seems to be quit a lot under
/var/tmp/portage/

Some of the information I'm looking for is in
/var/log/elog/

Other places to look? Mesos generates a lot of *.html file that
I'm speculating are general purpose setup interfaces for configuring
and controlling the various code components. It this typical of
Apache codes?

I feel like I'm missing the 'big picture' on where one
looks to find all these files that some packages generate.
On new software, I guess I have  to look at them all to 
figure out a runtime environment setup? I feel linke I'm on
Safari (an actual dangerous animal hunting trip) but there
has been no briefing. I feel hung_over, naked and without
compass. I feel like I need a bottle of whisky to just
start "winging it"......

Understand what I'm doing: I have stable tree ebuilds,
Overlays of various quality and then there are my
ebuilds (ugly hacks) in /usr/local/portage that I'm trying
to get my arms around with a weak comprehension of what to 
expect.

I also have read about Blueness efforts:  
"RFC: GLEP 64: Standardize contents of VDB" on the dev list
and that just leaves me scratching (more than my head).

Some discussion, suggested reading, and guidance would be welcome
as the structure(s) seem a wee bit "loose" if not inconsistent, to me......

How much does EAPI-<number> affect what a package build does? Where do I
have I go before compiling code (typically) in the unpacked sourcecode
to see/determine what is going to get build and where it will install,
as this is not strictly controlled by the ebuilds...... How much is 
controlled by the package codes and how much by setting in the Gentoo
build settings?


Most of the docs I've read, would make sense, if I knew what I was
specifically suppose to do. Learning the semantics of all of this
"echo_system" does not seem straightforward. The individual codes
are not the problem, as I can read and discern most codes. It's the
WISDOM of what goes WHERE and WHY that seems; fleeting to me.


lost & curious,
James






^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [gentoo-user] post build files
  2014-09-09 15:20 [gentoo-user] post build files James
@ 2014-09-09 16:23 ` Alec Ten Harmsel
  2014-09-11 14:02   ` [gentoo-user] " James
  2014-09-09 17:41 ` [gentoo-user] " Rich Freeman
  1 sibling, 1 reply; 7+ messages in thread
From: Alec Ten Harmsel @ 2014-09-09 16:23 UTC (permalink / raw
  To: gentoo-user

On 09/09/2014 11:20 AM, James wrote:
> Hello,
>
> I'm rather new to hacking ebuilds. I have read most every doc
> I can find on the subject. One thing I'm looking for is a post-build
> document that shows me the path/name of everything built.
>
> So for mesos-0.20.0 it seems to be:
>
> /var/db/pkg/sys-cluster/mesos-0.20.0/CONTENTS
>
> For some other packages, they seems to be quit a lot under
> /var/tmp/portage/
>
> Some of the information I'm looking for is in
> /var/log/elog/
>
> Other places to look? Mesos generates a lot of *.html file that
> I'm speculating are general purpose setup interfaces for configuring
> and controlling the various code components. It this typical of
> Apache codes?
>
> I feel like I'm missing the 'big picture' on where one
> looks to find all these files that some packages generate.
> On new software, I guess I have  to look at them all to 
> figure out a runtime environment setup? I feel linke I'm on
> Safari (an actual dangerous animal hunting trip) but there
> has been no briefing. I feel hung_over, naked and without
> compass. I feel like I need a bottle of whisky to just
> start "winging it"......
>
> Understand what I'm doing: I have stable tree ebuilds,
> Overlays of various quality and then there are my
> ebuilds (ugly hacks) in /usr/local/portage that I'm trying
> to get my arms around with a weak comprehension of what to 
> expect.
>
> I also have read about Blueness efforts:  
> "RFC: GLEP 64: Standardize contents of VDB" on the dev list
> and that just leaves me scratching (more than my head).
>
> Some discussion, suggested reading, and guidance would be welcome
> as the structure(s) seem a wee bit "loose" if not inconsistent, to me......
>
> How much does EAPI-<number> affect what a package build does? Where do I
> have I go before compiling code (typically) in the unpacked sourcecode
> to see/determine what is going to get build and where it will install,
> as this is not strictly controlled by the ebuilds...... How much is 
> controlled by the package codes and how much by setting in the Gentoo
> build settings?
>
>
> Most of the docs I've read, would make sense, if I knew what I was
> specifically suppose to do. Learning the semantics of all of this
> "echo_system" does not seem straightforward. The individual codes
> are not the problem, as I can read and discern most codes. It's the
> WISDOM of what goes WHERE and WHY that seems; fleeting to me.
>
>
> lost & curious,
> James
>
>
>
>
>
To see what all is getting installed, I'd download a separate mesos
tarball and run:

    ./configure --prefix=/home/alec/mesos-root && make && make install

As for the EAPI, I'm also new to ebuilds but from my understanding you
should just use the current level, which is 5.

As for the structure of the ebuilds, I found it to be reasonable. For
recommended reading, I read other ebuilds; reasonably complex ones like
dev-lang/R or sci-mathematics/octave showcase plenty of the features of
ebuilds.

Lastly, if you throw this in a publicly-accessible git repo I'll try and
help, even though I took a look at mesos and it seems like it'll be hard
because they bundle libraries with it and have made a couple other
(imnho) bad decisions.

Alec

P.S. I'd recommend doing ebuild testing in a chroot (if you're not
already) as well to avoid destroying your system, but that's just me


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [gentoo-user] post build files
  2014-09-09 15:20 [gentoo-user] post build files James
  2014-09-09 16:23 ` Alec Ten Harmsel
@ 2014-09-09 17:41 ` Rich Freeman
  2014-09-11 16:20   ` [gentoo-user] " James
  1 sibling, 1 reply; 7+ messages in thread
From: Rich Freeman @ 2014-09-09 17:41 UTC (permalink / raw
  To: gentoo-user

On Tue, Sep 9, 2014 at 11:20 AM, James <wireless@tampabay.rr.com> wrote:
> I'm rather new to hacking ebuilds. I have read most every doc
> I can find on the subject. One thing I'm looking for is a post-build
> document that shows me the path/name of everything built.

If you literally want a list of everything that was installed by a
Gentoo ebuild, then the simplest thing is to run qlist from
app-portage/portage-utils.

Of course, the list of what a package /could/ install can vary based
on USE flags or environment.

If you're trying to learn how ebuilds work, devmanual.gentoo.org is
the definitive resource.  If you have specific questions feel free to
ask, but just about anything you want to know is there.  By all means
try reading a few ebuilds to get a hang for things as well, but I'd
start with simple ones (avoid trying to learn by looking at packages
that use complex eclasses).  The simplest ebuilds tend to be for
simple, standalone programs.

Keep in mind that ebuilds work by extending functions defined by PMS,
so an ebuild can contain fairly little content and yet be fairly
functional (the default functions are running for most of the build
phases).  The idea is if all a package does is run configure ; make ;
make install it often needs almost nothing in the ebuild to work.

--
Rich


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [gentoo-user] Re: post build files
  2014-09-09 16:23 ` Alec Ten Harmsel
@ 2014-09-11 14:02   ` James
  0 siblings, 0 replies; 7+ messages in thread
From: James @ 2014-09-11 14:02 UTC (permalink / raw
  To: gentoo-user

Alec Ten Harmsel <alec <at> alectenharmsel.com> writes:


> Lastly, if you throw this in a publicly-accessible git repo I'll try and
> help, even though I took a look at mesos and it seems like it'll be hard
> because they bundle libraries with it and have made a couple other
> (imnho) bad decisions.

I biggest obstacle for all of the Apache clustering sofwares I want
to try, is java and other codes seem to be central to their
offerings.

For example "maven" is becoming critical for many codes [1].
In fact we are loosing jruby because nobody has the requisite
skills? [2].


Hacking and slashing through ebuilds is not difficult. I just 
have inate tendencies to that cause me to believe the path
should be 'well worn" with ample examples; but alas I do not
think that is the gentoo way, particular with how fast things
are moving on new codes and the entire linux echo_systems.


Why not champion maven and jruby so many can benefit?
Can maven be built from sources, as I see it is currently
in binary form maven-bin ?

Maven is an obstacle that I'm not certain I can go around
if I am to continue down the apache cluster pathway......

<snip-dev>
# Mask jruby 1.6.x for removal in 30 days.
# on ruby 1.8.  Many packages are no longer compatible with its
# syntax, and security support for ruby 1.8 has stopped. This
# incompatibility now has reached central packages like rubygems so we
# have decided to remove jruby 1.6 now.  jruby upstream has released
# the 1.7.x series some time ago, but unfortunately we only have a
# masked version that has too many issues to unmask. Please let us
# know if you know java and ruby and want to help out here:



hth,
James



[1] http://maven.apache.org/

[2]
http://moving-innovations.com/blog/2014/04/23/the-precarious-state-of-jruby-in-gentoo




^ permalink raw reply	[flat|nested] 7+ messages in thread

* [gentoo-user] Re: post build files
  2014-09-09 17:41 ` [gentoo-user] " Rich Freeman
@ 2014-09-11 16:20   ` James
  2014-09-11 19:08     ` Alec Ten Harmsel
  0 siblings, 1 reply; 7+ messages in thread
From: James @ 2014-09-11 16:20 UTC (permalink / raw
  To: gentoo-user

Rich Freeman <rich0 <at> gentoo.org> writes:


> If you literally want a list of everything that was installed by a
> Gentoo ebuild, then the simplest thing is to run qlist from
> app-portage/portage-utils.

wonderful idea. Despite having used qlist many times, it never dawned
on me for this purpose.....


> If you're trying to learn how ebuilds work, devmanual.gentoo.org is
> the definitive resource.  If you have specific questions feel free to
> ask, but just about anything you want to know is there.  By all means
> try reading a few ebuilds to get a hang for things as well, but I'd
> start with simple ones (avoid trying to learn by looking at packages
> that use complex eclasses).  The simplest ebuilds tend to be for
> simple, standalone programs.

Yes, I've been all over this. It's onto much of the Apache clustering
codes that are not simple to configure in the ebuild.  Besides the raw 
packege codes, like mesos, spark, scala, cassandra, etc there are
a mulitude of fast moving codes written in Java and Python that
need to be tested. Java is not difficult, but voluminous. Every problem
somebody encouters, gets solved by some java "bolt on" code, rather
than fixing/extending the main (mesos) sourcecode.  As an old C hack,
it's a tough pill to swallow, but I'm pursing this as best I can.
I sure feel empathy for the java herd, but hey, now we are doing away
with herds?


> Keep in mind that ebuilds work by extending functions defined by PMS,
> so an ebuild can contain fairly little content and yet be fairly
> functional (the default functions are running for most of the build
> phases).  The idea is if all a package does is run configure ; make ;
> make install it often needs almost nothing in the ebuild to work.

Yea, I got this (mostly....still new skills...) but it is the fancy footwork
in the java and python worlds that keeps me doing more reading and research
than coding/compiling/testing..... of the Clustering goodies.

That said, I'm a bit stressed about 'maven'. We only have maven.bin.
Much is dependant on it and java. For many reasons, java is not well
supported in Gentoo. I just hope, I do not have to leave the Gentoo
distro; because much of what I need (clustering) is java critical.

You think I can build a gentoo cluster based on these? [1] [2]
Does the Gentoo dev team operate a robust gentoo cluster for gentoo
development needs? Should they?

Clusters that perform best are built on mesos/spark. Spark is 
an "in-memory" computational enviroment for mesos clusters;
and it will change *everything* as systems become richly adormed
with ample, low cost ram  can consolidate into amazing clusters [3]  
RDD, Resilient Distributed Data is changing everying that is computationally
intensive.


I do appreciate all of the wonderful folks at gentoo (devs and users).
I have been using Gentoo since early 2004. But now, I need robust
clustering and from the open source research work I have performed,
Apache is the Only viable choice. Ubuntu has it. Debian Has it. RedHat
has it. CentOS has it, but, we cannot (willnot?) sustain Apache style
clustering at Gentoo because ? (dunno the reason; nothing seems plausible).

PS, I *HATE* oracle more than most, but for me that is not a valid
reason for piss_poor Java support. Google runs Java on top of embedded
linux (it's called Android) We even had Android on Gentoo,
it's call Gentroid [4]. If folks would just get over it (java_baggage),
we could have a robust Java platform on Gentoo; or am I missing something?

For my needs, I do not see path forward for Gentoo, without robust java
support. Hell, I'd even be willing to pay for a java support to develop
the java (sourcecode) based tools; but in the past that idea has been very
frowned up by the gentoo "think tank". But, devs with their own agendas
can spend their time doing exact what they want (thing gentoo needs); I just
cannot pay somebody to do the same (with Java)?

As hard as I can, I'm working on this, but I'm no Java maverick....
far from it.

(actually this is just to get it "off my chest". I feel better now
no needs for anyone to reply, unless you are interested in working
on Apache-mesos/spark or java/maven). Please, all flames in a new
thread.




James

[1] http://wiki.gentoo.org/wiki/Cluster

[2] http://wiki.stoney-cloud.org/wiki/Main_Page

[3] http://www.cs.berkeley.edu/~pwendell/strataconf/api/core/spark/RDD.html

[4] https://code.google.com/p/gentroid/





^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [gentoo-user] Re: post build files
  2014-09-11 16:20   ` [gentoo-user] " James
@ 2014-09-11 19:08     ` Alec Ten Harmsel
  2014-09-11 19:27       ` Alec Ten Harmsel
  0 siblings, 1 reply; 7+ messages in thread
From: Alec Ten Harmsel @ 2014-09-11 19:08 UTC (permalink / raw
  To: gentoo-user


On 09/11/2014 12:20 PM, James wrote:
> Yes, I've been all over this. It's onto much of the Apache clustering
> codes that are not simple to configure in the ebuild. Besides the raw
> packege codes, like mesos, spark, scala, cassandra, etc there are a
> mulitude of fast moving codes written in Java and Python that need to
> be tested. Java is not difficult, but voluminous. Every problem
> somebody encouters, gets solved by some java "bolt on" code, rather
> than fixing/extending the main (mesos) sourcecode. As an old C hack,
> it's a tough pill to swallow, but I'm pursing this as best I can. I
> sure feel empathy for the java herd, but hey, now we are doing away
> with herds?
Part of the problem with a lot of open source Java projects is that they
*require* Oracle's Java build, and installing Oracle's JDK in Gentoo is
not a simple 'emerge <package_name_here>'.
> That said, I'm a bit stressed about 'maven'. We only have maven.bin.
> Much is dependant on it and java. For many reasons, java is not well
> supported in Gentoo. I just hope, I do not have to leave the Gentoo
> distro; because much of what I need (clustering) is java critical. You
> think I can build a gentoo cluster based on these?
I'm not a huge fan of Maven - it's a massive, complex morass. That said,
I can look into building it.
> Does the Gentoo dev team operate a robust gentoo cluster for gentoo
> development needs? Should they?
What would it do? Other than continuous integration, I can't see much
use (although I'm not a dev, so that's not saying much).
> Clusters that perform best are built on mesos/spark.
I can't remember the last time I saw Mesos/Spark on the TOP500 list...
but also, they're both young projects.
> But now, I need robust clustering and from the open source research
> work I have performed, Apache is the Only viable choice. Ubuntu has
> it. Debian Has it. RedHat has it. CentOS has it, but, we cannot
> (willnot?) sustain Apache style clustering at Gentoo because ? (dunno
> the reason; nothing seems plausible). 
From what I can tell, the reason is that it is not worth anyone's time
to do the massive amount of development and maintainence for something
that they won't benefit from.
> Hell, I'd even be willing to pay for a java support to develop the
> java (sourcecode) based tools; but in the past that idea has been very
> frowned up by the gentoo "think tank".
If I wasn't in school I'd be all over this...

Anyways, I'll try and look at Maven. Not making any promises; I'm pretty
busy right now.

Alec


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [gentoo-user] Re: post build files
  2014-09-11 19:08     ` Alec Ten Harmsel
@ 2014-09-11 19:27       ` Alec Ten Harmsel
  0 siblings, 0 replies; 7+ messages in thread
From: Alec Ten Harmsel @ 2014-09-11 19:27 UTC (permalink / raw
  To: gentoo-user


On 09/11/2014 03:08 PM, Alec Ten Harmsel wrote:
> Anyways, I'll try and look at Maven. Not making any promises; I'm
> pretty busy right now. Alec
Also, feel free to message me directly James. I don't want to spam the
list with what will probably be a lot of small bugs and (most likely)
flaming.

While I'm not a huge fan of it, Java was the first language I taught
myself. I'm pretty familiar with it.

Alec


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2014-09-11 19:28 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-09-09 15:20 [gentoo-user] post build files James
2014-09-09 16:23 ` Alec Ten Harmsel
2014-09-11 14:02   ` [gentoo-user] " James
2014-09-09 17:41 ` [gentoo-user] " Rich Freeman
2014-09-11 16:20   ` [gentoo-user] " James
2014-09-11 19:08     ` Alec Ten Harmsel
2014-09-11 19:27       ` Alec Ten Harmsel

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox