Hadoop 1.0 Released 38
darthcamaro writes "There has been a tonne of hype about Big Data and specifically Hadoop in recent years. But until today, Hadoop was not a 1.0 release product. Does it matter? Not really, but it's still a big milestone. The new release includes a new web interface for the Hadoop filesystem, security, and Hbase database support. '"At this point we figured that as a community we can support this release and be compatible for the foreseeable future. That makes this release an ideal candidate to be called 1.0," Arun C. Murthy, vice president of Apache Hadoop, said.'"
What is Hadoop? (Score:5, Informative)
From Wikipedia:
Apache Hadoop is a software framework that supports data-intensive distributed applications under a free license.[1] It enables applications to work with thousands of nodes and petabytes of data. Hadoop was inspired by Google's MapReduce and Google File System (GFS) papers.
Hadoop is a top-level Apache project being built and used by a global community of contributors,[2] written in the Java programming language. Yahoo! has been the largest contributor[3] to the project, and uses Hadoop extensively across its businesses.[4]
Hadoop was created by Doug Cutting,[5] who named it after his son's toy elephant.[6] It was originally developed to support distribution for the Nutch search engine project.[7]
...in case you're as ignorant as I am. Post anonymously to avoid karma whoring.
Re: (Score:1)
Why stop there? Google is just a robot instead of a human that provides the information at your convenience. No! A thorough quest searching all the lands and oceans, involving 2 competing pirate clans, a fair lady and a magic unicorn, all spanning 7 years is the only thing that will allow you to not be ignorant.
Either that or you're confusing "ignorant" with "lazy"
Re: (Score:1)
Re: (Score:1)
this guy! the most insulting way possible to say be thankful for someone else adding the information you needed to add to the discussion.
Re: (Score:2)
Thanks for clarifying. I was thinking it was that thing that kicks people off the internet in France
That's Hadopi. Pretty bad. Nobody's been kicked out yet, they're still pondering the sanity of it all I guess. Anyways, the law is in effect.
Of course, it's dead easy to circumvent it since they only monitor the eMule network, on which you can barely find anything anymore anyways.
What the heck is Hadoop? (Score:5, Funny)
Um, what the heck is Hadoop? A filesystem? Linux distro? Database software? Something to do with web servers? Throw me a bone here, man. Why does this 'Big Data' need capitalization?
And most importantly, why did they go with the British spelling for 'tonne'? Is this a product of the UK?
What, read the article? Are you mad?
Re: (Score:2)
A tonne and a ton are different things. A tonne is a metric measure, 1000kg, a ton may be a short or long ton, and is some odd number of lbs (in the order of 2000 lbs or so).
Better late than never (Score:5, Informative)
It was actually released over a week ago, but I guess the announcement got lost over the holidays. I am actually a bit surprised they did a 1.0 version before solving the "NameNode is a single point of failure" problem with HDFS. I know for a fact that big companies (one of which was a client) are sometimes hesitant to deploy Hadoop because of this.
In theory, you can also use Hadoop with purportedly more robust distribute file systems, like KFS (Kosmos File System, I think it's called). I've never seen this in the wild though.
Re: (Score:1)
Something else I thought that got missed over the holidays was this:
HPCC Systems From LexisNexis Breaks World Record on Terasort Benchmark [msn.com]
Pretty amazing when you consider how little code it took to run the 100GiB sort, never mind that it was faster than hadoop using 1/5 of the hardware. Being able to read in from disk, perform network calls, compute the sort, and write back down to disk just over 1 gigabyte a second is BLAZING fast.
Re: (Score:2)
... except 100 gigabytes is not 1 terabyte.
Version 2.0 out in six weeks (Score:1, Funny)
Now it's released 1.0. it can increase Mozilla style.
Re: (Score:1)
To Infinity.... and beyond?
Re: (Score:2)
If your data's integrity is absolutely necessary, Hadoop (or more specifically HBase, which is the part most closely analogous to a database) is probably not for you. On the other hand, if you're working with statistics or any other application where an error affects your product trivially, you may find the speed is worth it, bearing in mind that changes are broadcast across the cluster "eventually". The strengths and weaknesses of Hadoop are different from a traditional database enough that I'd caution you
Re: (Score:1)
Haboob (n) A violent and oppressive wind blowing in summer, esp. in Sudan, bringing sand from the desert
I imagine Haboob will be the Apache foundation's non-Java version of Hadoop. Seriously, if big data is the application, better to run it on metal, not on a virtual machine.
Doesn't matter. (Score:2)
"There has been a tonne of hype about Big Data and specifically Hadoop in recent years. But until today, Hadoop was not a 1.0 release product. Does it matter? Not really
Wasn't there a slogan about "news for nerds, stuff that matters" around here somewhere?
Re: (Score:2)
Wasn't there a slogan about "news for nerds, stuff that matters" around here somewhere?
Key word being was. Or is it still around?
Hadoop is a distributed computing platform (Score:4, Informative)
Seems a fair number of you are unaware of what Hadoop is.
Hadoop is a platform that enables distributed computing. Specifically, it supports map/reduce programming in a manner similar to Google's App Engine, except that it is open source. It supports distributing data for redundancy and/or scalability (in other words, you can have multiple copies of each data item on multiple computers, or you can split a data set across multiple computers, or both, with the data set sharded across multiple machines but with copies of each shard on multiple machines).
There is a distributed filesystem built on top of hadoop called HDFS. There is a distributed key/value store (somewhat analogous to a database...actually, scratch that, it's a distributed hash map) called HBase. There are also a number of distributed computing libraries built on top of Hadoop, like Mahout (for machine learning), Hive (for ad-hoc querying of large data sets), and Pig (another distributed computing model that some consider to be easier than map/reduce).
The whole setup provides a distributed computing model similar to Google's distributed environment, supporting very large clusters, map/reduce programming, and distributed storage of very large and/or spare matrices and tables.
Re: (Score:1)
....
There is a distributed filesystem built on top of hadoop called HDFS.
...
I would say that Hadoop runs on top of HDFS. That is, the Hadoop layer expects to use a distributed filesystem which has some location awareness (allowing processing tasks to be sent to the nodes containing the data) and a level of resilience to individual datanodes failing (there is at least 2-way replication of data assumed in a typical HDFS configuration). The HDFS API expected by Hadoop can be implemented by other filesystems, such as Kosmos File System, mentioned above.
Apos for posting this comment
Apache ZooKeeper (Score:2)
Cascalog (Score:1)
Amazing that nobody mentioned Cascalog https://github.com/nathanmarz/cascalog [github.com]