Scaling Large Projects With Erlang 200
Delchanat points out a blog entry which notes,
"The two biggest computing-providers of today, Amazon and Google, are building their concurrent offerings on top of really concurrent programming languages and systems. Not only because they want to, but because they need to. If you want to build computing into a utility, you need large real-time systems running as efficiently as possible. You need your technology to be able to scale in a similar way as other, comparable utilities or large real-time systems are scaling — utilities like telephony and electricity. Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires. Amazon SimpleDB is built upon Erlang. IMDB (owned by Amazon) is switching from Perl to Erlang. Google Gears is using Erlang-style concurrency, and the list goes on."
Sufficiently? (Score:5, Interesting)
Re:Huh? (Score:5, Interesting)
Comparison of functional languages? (Score:5, Interesting)
I think the summary (and article) are somewhat poorly written, but that doesn't shadow the fact that functional languages are becoming more and more interesting these days with concurrency becoming so important.
I'd like to learn one, but there are several out there.. What I'd like to see is a good in-depth comparison of different concurrent functional languages: why would I choose Haskell, or Erlang, or OCaml, for example? Are they all interpreted? (Does one exist that compiles?) Which ones support concurrency? What language features do they boast, and what are the advantages and disadvantages of these features? Do they have a complete set of libraries?
Anyone know of an article like this? I've been searching for a while. Every article on functional languages I've found seems to concentrate on a particular one, but I can't find something helping me decide which one is most worth learning.
Re:Who wrote the summary? GWB? (Score:1, Interesting)
Oops, wrong link. I isn't learning.
http://www.youtube.com/watch?v=aAUrToY33tI
Why Erlang doesn't matter (Score:5, Interesting)
1. Invariable variables.
This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.
2. Weird syntax.
Why, exactly, are there three different kinds of (required) line endings? It seems as though the syntax is designed to be as different from C as possible, while maintaining at least as many quirks. Moreso, even -- when constructing normal, trivial programs, you're going to hit most language features head-on and at their worst. Where's my 'print "hello\n"' that works most other places?
I don't believe the important features of Erlang are mutually-exclusive with the sane syntax of, say, Ruby or Python.
3. Not Unicode-ready.
Strings are defined as ASCII -- maybe latin1. But there's no direct unicode support in the language -- if you're lucky, there are functions you can pipe it through.
There are other things I haven't mentioned, mostly implementation-specific -- things like the fact that function-reloading cannot be done when you natively-compile (with hipe) for extra speed. My plan is to take the features I actually like from Erlang and implement them elsewhere, in a language I can actually stomach for its real tasks.
Re:Comparison of functional languages? (Score:5, Interesting)
OCaml compiles down to native code, which about 10-20% slower than C. Faster than C in a few (narrow) cases.
Haskell is also compiled to native code, but difficulties with the execution model mean it's pretty slow for any practical use.
Erlang is interpreted - the execution model is similar to Perl or Python - which means its slow on single cores, but of course the whole point of Erlang is to run in highly concurrent, distributed machines. There is a project [google.com] to use OCaml for the performance-critical, single threaded parts, and Erlang for coordinating the parallelism.
Of course, this is probably missing the point. Unless you're doing intensive numerical work, you probably don't need the performance. The real advantage of these languages is how your code will be much smaller, easier to understand, safer, and faster to write.
Rich.
Wings 3D is written in Erlang... (Score:3, Interesting)
Andy
Re:Why Erlang doesn't matter (Score:3, Interesting)
Though I agree with you on 2 and 3, I'm not so sure about 1, but I might be wrong on that. As I understand it, you should look at variables in functional programming languages like Erlang more like those in a mathematical formula; such programs can be proven correct a lot easier, and since variables are effectively immutable, it facilitates forking the line of execution in a way that would not be possible without all kinds of semaphores and other concurrency stuff than if variables where not immutable. You should see this practice more like taking "pure functions" (http://en.wikipedia.org/wiki/Functional_programming#Pure_functions) to another level, namely to within functions themselves.
But I might be wrong, if so, please educate me:)
"Cloud computing" is an Xmas artifact (Score:5, Interesting)
The enthusiasm for "cloud computing" may evaporate when Xmas rolls around.
I went to a talk at Stanford by the architect of Amazon's web services. It came out in questioning that the real motivation between Amazon's low-priced web services is that their load in the Xmas shopping season is about 4x the load for the rest of the year. Their infrastructure is sized for the November-December peak, so for ten months of the year they have vast excess capacity. That's why Amazon's web services are so cheap.
Don't expect good response time during the shopping season. Although this Xmas might be OK, due to the recession.
Re:Stupid article (Score:4, Interesting)
I'm not going to disagree with most of your post, I think you're spot on. However, your suggestion of D is totally off. I like the D programming language quite a bit and version 2 is going to be really cool. However, even version 1 of D is not ready for prime-time [timburrell.net]. Version 2 of D is unstable and not recommended for production by even the author himself. All of the other languages you mentioned such as Erlang or Haskell are much more mature.
Also, "most other languages" have a foreign function interface for C, including Erlang, Haskell, Python, Java, Perl, Ruby, etc... In fact, I can't think of a well known programming language actually used by people other than the author that does not have an FFI... It is true that in most cases the FFI of other languages is more difficult to use than the one in D, but they are there.
Erland textbook: read and adapt (Score:3, Interesting)
Re:Why Erlang doesn't matter (Score:5, Interesting)
As I understand it, you should look at variables in functional programming languages like Erlang more like those in a mathematical formula; such programs can be proven correct a lot easier, and since variables are effectively immutable
All of this is based on the premise that Erlang is a functional language. It's not purely-functional, and I just don't see the point of doing it half-assedly. Erlang is effectively an imperative language dressed up like a functional language.
And they're not immutable -- they can be unbound. As I understand it, this unboundedness is detected at runtime, not compiletime. If it was detected at compiletime, you'd have a valid point.
it facilitates forking the line of execution in a way that would not be possible without all kinds of semaphores and other concurrency stuff
Except that's not how Erlang does concurrency. It does concurrency with explicit "processes" (green threads) and message-passing.
Now, it does make these very easy, and you can get it to distribute processes among a few real OS threads (one per core) -- so it's still very cool. But you're thinking of languages like Haskell, which can be automagically threaded. Erlang is manually threaded, it's just much easier to think in threads (or "processes") -- they're effectively a language feature.
Dunno if Google uses Erlang, but Facebook does (Score:2, Interesting)
They built the Facebook Chat backend using Erlang. Scaling something from 0 users one day to tens of millions of active users the next day is a challenge, and they decided Erlang was the right tool for the job. http://www.facebook.com/note.php?note_id=14218138919&id=9445547199
Re:Stupid article (Score:5, Interesting)
Yes, D is very young and has problems. But then again, what language didn't? It's easy to forget but Python was first released in 1991. It took many years before it became mainstream (and some would say it's still not there yet).
The post-mortem is an interesting document, but I disagree with the authors conclusions. The compilers are buggy, well, C++ had exactly the same problem for a long time but still was a huge success. In particular, the trend seems to be basing new compilers on LLVM, which has a pretty robust optimization core. Frontend bugs are by comparison pretty trivial and easy to fix. Another few years and I think this problem will be licked - and besides, lots of C++ code has workarounds for compiler issues. Same thing for class libraries.
You're right about C-level FFIs. However D provides a simple C++ FFI which as far as I know is unique. Such a thing would be very useful for a company like Google which has a lot of C++ code, as it'd simplify binding considerably (I don't mean to imply anything about the future direction of the codebase, by the way).
The argument about parallelism is a more interesting one. But I disagree with that too :) D provides exactly what is needed for automatic sharding of work across cores (or machines). Specifically the combination of transitive invariance, reflection and purity enforcement is a very powerful one.
Essentially, if you can write your code to consist of non-trivial trees of pure functions, then it's perfectly safe to parallelise something like this:
foreach (item; list) {
fooResults[item] = someTransform(item);
barResults[item] = anotherTransform(item);
}
If someTransform and anotherTransform are both pure, by implication their parameters are transitively invariant, and thus they can both be invoked in parallel (because the compiler knows "item" can't be changed). What's more both calls can be invoked simultaneously as well.
Once the compiler knows these things, making this code run in parallel is simply another compiler optimization. That's the whole theory behind how functional languages can be super easy to parallelize. But in fact the key concepts can be applied to imperative languages as well, with the advantage that you can still have temporary mutable state within the function scopes - you just can't modify the heap, or anything reachable through your arguments.
D has keywords that let the compiler know and enforce function purity.
Now as it happens I doubt that any D compiler today implements this optimisation - it's sophisticated and transitive invariance is newly introduced in D2. But all the pieces of the puzzle are there. This also lets the compiler do calculations on data structures available at compile time.
Re:Scala (Score:4, Interesting)
Facebook chat (Score:3, Interesting)
Re:Why Erlang doesn't matter (Score:3, Interesting)
http://www.paulgraham.com/avg.html [paulgraham.com]