[gentoo-user] SRC_URISRC_URI.mirror

[gentoo-user] SRC_URISRC_URI.mirror

[James]

Hello,

So I'm working on apache spark (overlay) ebuild. 
I cannot see to get the sources to download.

Here are the sources:
http://www.apache.org/dist/spark/spark-1.1.0/
or here:
http://mir2.ovh.net/ftp.apache.org/dist/spark/spark-1.1.0/

My local ebuild has these etries:

<snip>

MY_PV=${PV/_/}

SRC_URI="http://www.apache.org/dist/spark/spark/${PV}/${P}.tgz
http://mir2.ovh.net/ftp.apache.org/dist/spark/spark/${PV}/${P}.tgz"

'ebuild spark-1.1.0.ebuild manifest'

When attemping to compile (emerge spark<###> ) I get:
!!! Couldn't download 'spark-1.1.0.tgz'. Aborting.
!!! Fetch failed for spark-1.1.0.tgz, can't update Manifest

So based on 'repoman scan spark-1.1.0.ebuild' I add:

SRC_URI.mirror="mirror://apache/spark/spark/1.1.0/spark-1.1.0.tgz"
and comment out the  original "SRC_URI" line.

I can then update the  manifest, but ti fails to build:

" /usr/local/portage/sys-cluster/spark/spark-1.1.0.ebuild: line 19:
SRC_URI.mirror=mirror://apache/spark/spark/1.1.0/spark-1.1.0.tgz: No such
file or directory
"

I've tried dozens and dozens of varous SRC_URI, but it just will
not download the file.

ideas?


And here is the rest, just in case anyone wants to make suggestions.

LICENSE="Apache-2.0"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE="java python scala"

DEPEND="python? ( dev-lang/python dev-python/boto )
		java? ( virtual/jdk )
		scala? ( dev-lang/scala )
		dev-java/maven-bin
		${DEPEND}"

RDEPEND=" python? ( dev-lang/python )
		  >=virtual/jdk-1.6
		  scala? ( dev-lang/scala )
		  dev-java/maven-bin"

S="${WORKDIR}/${P}"

ECONF_SOURCE="${S}"

src_prepare() {
    mkdir "${S}/build" || die
}

src_configure() {
    cd "${S}/build"
    econf \
        $(use_enable python) \
        $(use_enable java)
}

src_compile() {
    cd "${S}/build"
    emake -j1 V=1
}

src_install() {
    cd "${S}/build"

Re: [gentoo-user] SRC_URISRC_URI.mirror

[Michael Orlitzky]

On 09/20/2014 01:07 PM, James wrote:
> Hello,
> 
> So I'm working on apache spark (overlay) ebuild. 
> I cannot see to get the sources to download.
> 
> Here are the sources:
> http://www.apache.org/dist/spark/spark-1.1.0/
> or here:
> http://mir2.ovh.net/ftp.apache.org/dist/spark/spark-1.1.0/
> 
> ...
> 
> So based on 'repoman scan spark-1.1.0.ebuild' I add:
> 
> SRC_URI.mirror="mirror://apache/spark/spark/1.1.0/spark-1.1.0.tgz"
> and comment out the  original "SRC_URI" line.

The SRC_URI.mirror warning indicates that your original ebuild has a
suspicious entry in SRC_URI, namely one of Gentoo's 3rd party mirrors.
Repoman gives a warning because you probably don't need it, and I think
it's right in this case.

The entries of the SRC_URI variable are all logical-ANDed together
rather than logical-OR. In other words, every entry is downloaded and
considered part of the source. You only need the first one (from
apache.org); Gentoo will ultimately take care of mirroring it if the
ebuild makes its way into the tree.

[gentoo-user] Re: SRC_URISRC_URI.mirror

[James]

Michael Orlitzky <mjo <at> gentoo.org> writes:



> The entries of the SRC_URI variable are all logical-ANDed together
> rather than logical-OR. In other words, every entry is downloaded and
> considered part of the source. You only need the first one (from
> apache.org); Gentoo will ultimately take care of mirroring it if the
> ebuild makes its way into the tree.

OK, now it is simplified to:

MY_PV=${PV/_/}

SRC_URI="http://www.apache.org/dist/spark/${PV}/${P}.tgz" 

But I get this error:

'ebuild spark-1.1.0.ebuild manifest'

Downloading 'http://www.apache.org/dist/spark/1.1.0/spark-1.1.0.tgz'
--2014-09-20 13:54:17--  http://www.apache.org/dist/spark/1.1.0/spark-1.1.0.tgz



ERROR 404: Not Found.

!!! Couldn't download 'spark-1.1.0.tgz'. Aborting.
!!! Fetch failed for spark-1.1.0.tgz, can't update Manifest


But th file is in:
http://www.apache.org/dist/spark/spark-1.1.0/

so PV is not forming correctly in the download, and I've tried everything
to fix it?

Downloading 'http://www.apache.org/dist/spark/1.1.0/spark-1.1.0.tgz'

should be:

Downloading 'http://www.apache.org/dist/spark/spark-1.1.0/spark-1.1.0.tgz'
or
Downloading 'http://www.apache.org/dist/spark/spark-1.1.0.tgz


????


James

Re: [gentoo-user] Re: SRC_URISRC_URI.mirror

[Michael Orlitzky]

On 09/20/2014 01:55 PM, James wrote:
> 
> OK, now it is simplified to:
> 
> MY_PV=${PV/_/}
> 
> SRC_URI="http://www.apache.org/dist/spark/${PV}/${P}.tgz" 
> 
> But I get this error:
> 
> 'ebuild spark-1.1.0.ebuild manifest'
> 
> Downloading 'http://www.apache.org/dist/spark/1.1.0/spark-1.1.0.tgz'
> --2014-09-20 13:54:17--  http://www.apache.org/dist/spark/1.1.0/spark-1.1.0.tgz
> 
> 
> 
> ERROR 404: Not Found.
> 

Because that's the wrong URL =)

[gentoo-user] Re: SRC_URISRC_URI.mirror

[James]

Michael Orlitzky <mjo <at> gentoo.org> writes:


> > ERROR 404: Not Found.

> Because that's the wrong URL =)

OK, I know this. What is the correct URL?

James

[gentoo-user] Re: SRC_URISRC_URI.mirror

[James]

Michael Orlitzky <mjo <at> gentoo.org> writes:


> > MY_PV=${PV/_/}

> > SRC_URI="http://www.apache.org/dist/spark/${PV}/${P}.tgz" 

> Because that's the wrong URL =)

SRC_URI="http://www.apache.org/dist/spark/spark-1.1.0/${P}.tgz" 

Works. Is this correct?
(sorry for being dense)


James

Re: [gentoo-user] Re: SRC_URISRC_URI.mirror

[Bryan Gardiner]

On Saturday, September 20, 2014 18:08:30 James wrote:
> Michael Orlitzky <mjo <at> gentoo.org> writes:
> > > MY_PV=${PV/_/}
> > > 
> > > SRC_URI="http://www.apache.org/dist/spark/${PV}/${P}.tgz"
> > 
> > Because that's the wrong URL =)
> 
> SRC_URI="http://www.apache.org/dist/spark/spark-1.1.0/${P}.tgz"
> 
> Works. Is this correct?
> (sorry for being dense)
> 
> 
> James

See: http://devmanual.gentoo.org/ebuild-writing/variables/index.html

${PV} is only the version number, it doesn't include the package name.

- Bryan

Re: [gentoo-user] Re: SRC_URISRC_URI.mirror

[Jc García]

2014-09-20 12:08 GMT-06:00 James <wireless@tampabay.rr.com>:
>
> Michael Orlitzky <mjo <at> gentoo.org> writes:
>
>
> > > MY_PV=${PV/_/}
>
> > > SRC_URI="http://www.apache.org/dist/spark/${PV}/${P}.tgz"
>
> > Because that's the wrong URL =)
>
> SRC_URI="http://www.apache.org/dist/spark/spark-1.1.0/${P}.tgz"
>

If you want to build the URI with the ebuild environment variables, it would be:
SRC_URI="http://apache.org/dist/${PN}/${P}/${P}.tgz"

> Works. Is this correct?
> (sorry for being dense)
>
>
> James
>
>
>
>

Re: [gentoo-user] Re: SRC_URISRC_URI.mirror

[Michael Orlitzky]

On 09/20/2014 02:08 PM, James wrote:
> Michael Orlitzky <mjo <at> gentoo.org> writes:
> 
> 
>>> MY_PV=${PV/_/}
> 
>>> SRC_URI="http://www.apache.org/dist/spark/${PV}/${P}.tgz" 
> 
>> Because that's the wrong URL =)
> 
> SRC_URI="http://www.apache.org/dist/spark/spark-1.1.0/${P}.tgz" 
> 
> Works. Is this correct?
> (sorry for being dense)
> 

Yes, and you can replace "spark-1.1.0" by ${P} in the path as well. The
link that Bryan posted has a list of all of the variables that are
available. You can go pretty crazy with some of them, but in this case
the only other thing I would replace is "spark" by ${PN}.

[gentoo-user] Re: SRC_URISRC_URI.mirror

[James]

Michael Orlitzky <mjo <at> gentoo.org> writes:


> Yes, and you can replace "spark-1.1.0" by ${P} in the path as well. The
> link that Bryan posted has a list of all of the variables that are
> available. You can go pretty crazy with some of them, but in this case
> the only other thing I would replace is "spark" by ${PN}.


OK, that behind me now......

So the build fails, so I figure I'll just build it manually, then
finish the ebuild. So I went to:


/var/tmp/portage/sys-cluster/spark-1.1.0/work/spark-1.1.0
and no configure scripts....

The README.md has this:

Spark is built on Scala 2.10. To build Spark and its example programs, run:

    ./sbt/sbt assembly

I did and it looks like the manual compile worked:

[info] Packaging
/var/tmp/portage/sys-cluster/spark-1.1.0/work/spark-1.1.0/
examples/target/scala-2.10/spark-examples-1.1.0-hadoop1.0.4.jar
...
[info] Done packaging.
[success] Total time: 786 s, completed Sep 20, 2014 3:04:22 PM

So I need to add commands to the ebuild to launch
" ./sbt/sbt assembly"

I've been all over the man 5 ebuild and the devmanual. So naturally
I've seen what to do, but missed it.

Suggested reading (which section) or syntax is most welcome.

James

Re: [gentoo-user] Re: SRC_URISRC_URI.mirror

[Michael Orlitzky]

On 09/20/2014 03:17 PM, James wrote:
> 
> OK, that behind me now......
> 
> So the build fails, so I figure I'll just build it manually, then
> finish the ebuild. So I went to:
> 
> 
> /var/tmp/portage/sys-cluster/spark-1.1.0/work/spark-1.1.0
> and no configure scripts....
> 
> The README.md has this:
> 
> Spark is built on Scala 2.10. To build Spark and its example programs, run:
> 
>     ./sbt/sbt assembly
> 
> I did and it looks like the manual compile worked:
> 
> [info] Packaging
> /var/tmp/portage/sys-cluster/spark-1.1.0/work/spark-1.1.0/
> examples/target/scala-2.10/spark-examples-1.1.0-hadoop1.0.4.jar
> ...
> [info] Done packaging.
> [success] Total time: 786 s, completed Sep 20, 2014 3:04:22 PM
> 
> So I need to add commands to the ebuild to launch
> " ./sbt/sbt assembly"
> 
> I've been all over the man 5 ebuild and the devmanual. So naturally
> I've seen what to do, but missed it.
> 

Short answer: just put the command in the ebuild (which is nothing but a
fancy bash script). It'll run. For example,

src_compile() {
	./sbt/sbt assembly || die "assembly build failed"
	[the rest of the build commands go here]
}

The long answer is that we usually have eclasses for different build
systems. The way eclass inheritance works, the eclasses override certain
phases of the ebuild. So for example, the haskell-cabal eclass knows
that it should run `runghc Setup.hs configure` or something like that
instead of ./configure (like we would run by default).

Unfortunately, I don't see any other ebuilds for scala packages in the
tree. So you're probably the first person to need such an eclass. Rather
than set yourself back implementing the eclass, you're probably better
off running all of the build commands manually and waiting for somebody
else to write the eclass.

Someone may be working on scala in an overlay; the java herd is taking
care of dev-lang/scala so perhaps you can ask in #gentoo-java.

[gentoo-user] Re: SRC_URISRC_URI.mirror

[James]

Michael Orlitzky <mjo <at> gentoo.org> writes:


> src_compile() {
> 	./sbt/sbt assembly || die "assembly build failed"
> 	[the rest of the build commands go here]
> }

> Someone may be working on scala in an overlay; the java herd is taking
> care of dev-lang/scala so perhaps you can ask in #gentoo-java.

OK, it's very close to working. I filed bug 523412 if anyone is
interested. I'll contact the java herd or wait until a dev
looks at the bug.




I also updated the bug (510912) for Apache-mesos with Mesos-0.20.0 as it
needs to be tested. In installs but it needs further testing. I also need to
figure out how to post them to sunrise. Hopefully a java dev will pick
these up and give me a hand. Also,  there may be bug or 2 in mesos-0.20.0.

Thanks for all the help!

James