Re: [gentoo-user] Clusters on Gentoo ?

[Rich Freeman <rich0@gentoo.org>]

On Tue, Aug 19, 2014 at 5:34 AM, J. Roeleveld <joost@antarean.org> wrote:
> On Monday, August 18, 2014 10:53:51 AM Alec Ten Harmsel wrote:
>> On Mon 18 Aug 2014 10:50:23 AM EDT, Rich Freeman wrote:
>> > Hadoop is a very specialized tool.  It does what it does very well,
>> > but if you want to use it for something other than map/reduce then
>> > consider carefully whether it is the right tool for the job.
>>
>> Agreed; unless you have decent hardware and can comfortably measure
>> your data in TB, it'll be quicker to use something else once you factor
>> in the administration time and learning curve.
>
> The benefit of clustering technologies is that you don't need high-end
> hardware to start with. You can use the old hardware you found collecting dust
> in the basement.
>
> The learning curve isn't as steep as it used to be. There are plenty of tools
> to make it easier to start using Hadoop.
>

As long as you're counting words and don't mind coding everything in Java.  :)

I found that if you want to avoid using Java, then the available
documentation plummets, and I'm pretty sure the version I was
attempting to use was buggy - it was losing records in the sort/reduce
phase I believe.  Or perhaps I was just using it incorrectly, but the
same exact code worked just fine when I ran it on a single host with a
smaller dataset and just piped map | sort | reduce without using
Hadoop.  The documentation was pretty sparse on how to get Hadoop to
work via stdin/out with non-Java code and it is quite possible I
wasn't quite doing things right.  In the end my problem wasn't big
enough to necessitate using Hadoop and I used GNU parallel instead.

--
Rich