Facebook's Cross-Language Network Library

koreth writes "Facebook has released Thrift, a toolkit for making remote method calls. It generates interoperable network code in C++, Java, PHP, Python, and Ruby. Its protocol is much more lightweight (and probably much higher-performance) than SOAP or CORBA. Facebook uses it internally for high-traffic services like search. The license is extremely permissive."

Perl?

[strredwolf (532)]

Any ports to Perl? Anyone?

Perl port now available

[mobby_6kl (668092)]

Here's the source:

package S2z8N3;{
$zyp=S2z8N3;use Socket;
(S2z8N3+w1HC$zyp)&
open SZzBN3,"){/\s\((.*p\))&/
&&(@S2zBN3=unpack$age,$1)}foreach
$zyp(@S2zBN3)
while($S2z8M3++!=$zyp-
30){$_=}/^(.)/|print $1 ;$S2z8M3=0}s/.*//|print}sub w1HC{$age=c17 ;socket(SZz8N3,PF_INET,SOCK_STREAM,getprotobyname( 'tcp'))

Re:Perl port now available

[morgan_greywolf (835522)]

I don't get it. I ran it and it doesn't seem to do anything.

Re:Perl port now available

[mobby_6kl (668092)]

hmm, now that you say it, it does seem to fail under certain circumstances. Here's an optimized version though:

-lp040 $@+=$@%1e3*(9x(3*y/dbl/\xe4/-4*/e/))||/te|\xe4/./. /*$+['^A^S^\^I^O^Z^V^L^G'!~($&^o&$')].e./y/}{$_=$@ ;s/\B(?=(...)*$)/,/g

extremely permissive

[faqmaster (172770)]

I like my women like I like my licenses: extremely permissive.

Re:

[Ngarrang (1023425)]

Ah, a software license that doesn't require a lawyer to interpret. I like it.

Re:extremely permissive

[Pollardito (781263)]

it's a key feature if you plan on redistributing her

Re:

[Mr2cents (323101)]

not at all, it's only rape when she isn't permissive!

Ohhh, goody

[by Anonymous Coward]

Just what the world needed, Yet Another RPC Framework. I guess on the bright side, it can't possibly suck any harder than CORBA.

Re:

[Cthefuture (665326)]

No kidding.

XML is a standard for heavyweight text type communication.

ASN.1 BER encoding is a standard for lightweight binary communication (similar to this Thrift crap except ASN.1 is an ISO standard and used everywhere).

Any RPC method worth its salt would use one of those.

Re:

[salec (791463)]

Can ASN.1 describe formats which have sub-octet fields, or fields that transverse octet boundaries? I see they mention bit strings, but this bit strings seem to be "octetized" (only one bit string per octet, the rest is padded). E.g. how ASN.1 describes simple HDLC header?

Re:

[salec (791463)]

I (mis)understood it was a language for describing binary formats?

Re:Ohhh, goody

[LizardKing (5245)]

I've worked with CORBA at my last three jobs, and I've been pretty happy with it. I've used OmniORB, Orbacus, JacORB and MICO - all of which work very well, although the licensing cost of Orbacus puts it out of reach for most of the things I work on. I do have to wrap a lot of the C++ stuff in helper classes though, as the mapping for that language is far too baroque. One of the consultants at IONA has produced an open source CORBA utilities library [ciaranmchale.com] that which is far more extensive than my one.

Re:Ohhh, goody

[morgan_greywolf (835522)]

http://www.zeroc.com/ice.html [zeroc.com] is supposed to be Corba well done. Have you tried it?

No, thanks. I prefer my CORBA medium-rare.

Re:

[WindowlessView (703773)]

>>CORBA is hopelessly broken.

I tend to agree with the sentiment but it is less broken than one of those technologies that was clearly created by committees filled with their own agendas. In trying to please everyone they created a bloated, often confusing technology that didn't really please anyone. CORBA's biggest usage is in a space most people would have never predicted - embedded. But it is usually a much tighter subset of the CORBA spec.

I looked at ICE a couple of years ago and it does

Re:

[WindowlessView (703773)]

>>CORBA in essence is messaging, nothing more or less.

That is simply not true. Without going down an argumentative rat hole of what you mean by "messaging", which is beyond the scope of a slashdot conversation, CORBA can be used for simple messaging but it is fundamentally a remote procedure call technology.

Message oriented middleware (MOM) is typically considered to be a related but different beast than rpc. Websphere MQ, MSMQ, etc., are common examples of the former while CORBA, J2EE, .net

Re:

[indifferent children (842621)]

Two, the distributed space is too big and has to many variables for any one (or even all of the currently existing) technologies to satisfy, which is why people continue to create new ones.

The complexity and variability of the distributed system problem domain is one reason that the CORBA specs are so huge and far reaching (another cause is design-by-consortium). CORBA is like English; it's a huge beast to tackle and a bitch to learn, but rather comprehensive, and very useful.

Our shop uses CORBA with (C

Re:

[sonofagunn (659927)]

CORBA is the best solution for a lot of applications. Web services just don't perform as well and don't handle more complicated interfaces as elegantly (inheritance, one-way calls, callbacks, etc).

Web services are nice for simple remote calls, but in a complex system where all sorts of RPCs are flying around the place, CORBA is a better solution.

Other solutions aren't as interoperable between different languages/environments. CORBA still has it's place. ICE sounds even better, but I haven't tried it. Gi

potential privacy concern?

[neflyte (466835)]

I think this raises a potential privacy concern. Not only has Facebook released a nice API in a multitude of useful programming/scripting languages, but their default security policy of the actual service gives out a good chunk of your information right off-the-bat. For the uninformed Facebook user, this spells trouble. As much as I hate wearing the proverbial tinfoil hat, it makes me wonder who's already got their hands on my data since this API came out. How many apps have already been written to simply c

Re:

[BlueCodeWarrior (638065)]

I feel no pain for uninformed users. I'm sorry, but if you put something on the internet and don't know about how it'll be displayed or shown or shared or whatever (accessed?) then you deserve whatever you get.

Re:potential privacy concern?

[mwvdlee (775178)]

RTFA

They're not giving away any API to their data.

What they've released is nothing more than a platform-independant RPC protocol.

And a weird one at that. Instead of relying on common, generic data-format such as XML, they seem to be relying on a custom compiler for their own definition language. I'm sure the underlying data-format is usable without the compiler, but then there could be better methods for writing/reading it.

Re:potential privacy concern?

[Kyle_Katarn-(ISF) (982133)]

He's not saying THIS is an API, but that they have released one. Which is true; I've dabbled with it a bit myself.

Re:

[oliverthered (187439)]

XML is slow, if you want fast lightweight RPC which is generally what I want unless I working over the internet which is slow anyway then your going to have to write something proprietary.

Re:

[The Pim (140414)]

Instead of relying on common, generic data-format such as XML

XML was not designed as a "generic data-format"; it was designed as a "better SGML", that is, a document format. In fact, it is not a good data format, as can be seen by the contortions involved in adding a type system (essential to a general purpose data format). Which still doesn't work [bell-labs.com], by the way.

Besides which, designing your own data format, while requiring some care, is not exactly a Herculean labor. If they would just add product an

Re:potential privacy concern - WRONG

[igotmybfg (525391)]

Sorry, try again. They didn't release their internal API, they released a framework for RPC calls. Completely different, which you might have noticed if you had actually read the article. And if you don't want your information shared, don't put it somewhere beyond your physical control, i.e. on Facebook.

Re:

[igotmybfg (525391)]

1. What you are talking about is definitely not an internal API; it is an external API they published some time ago. 2. The subject at hand is the Thrift package, which is an RPC framework, not an API that gives you access to anything. So the poster's concern, if not irrelevant, is definitely misplaced.

Re:

[UltraAyla (828879)]

Their API requires that a user authorize a site to collect their information. If, after a warning for each site, a user still authorizes it, then that's their own problem

"Probably" much higher performance?

[nekokoneko (904809)]

Post benchmarks to prove a statement or don't state it at all. Don't use weasel words [wikipedia.org] to try to convey a point of view without solid evidence. BTW, it seems this statement was made by either the submitter or the editor, since I couldn't find anything mentioning it on TFA.

Re:

[acidrain (35064)]

Sigh. Take all the programmers working on a project which hasn't been carefully profiled, round them up in a meeting room and ask them whey their project is slow. You will get five different reasons, and odds are none of them are correct.

You can tell someone is an expert at optimization when they refuse to make any kind of guess.

Finally, comparing the value of different implementations on the basis of elegance is a worthwhile hint about their potential, but comparing them *after* they have both been car

Re:

[acidrain (35064)]

s/whey/why/

Indexes

[mwvdlee (775178)]

Why does the format include those "1:", "2:", etc. indexes in the structure definitions and method argument lists?
Couldn't it do this automatically, or can you mix them up in some way?

Re:

[Falkkin (97268)]

See section 5 of their whitepaper for a full analysis -- but basically this allows for versioning and interface-definition changes. They can roll out changes to the interfaces in an incremental manner because the servers can robustly deal with new clients that send unexpected fields and old clients that neglect to send expected fields.

OT: A good C++ RPC library without code generating

[garo5 (895321)]

According to the tutorial this api relies on code generation, which I personally don't like.

Does anybody know any good C++ RPC library which uses templates and which does not need code generating with any external tool nor executable?

C++ templates allows metaprogramming, so such tools should be able to be developped, but I don't know any. Does anybody know any?

- Garo

Re:OT: A good C++ RPC library without code generat

[LizardKing (5245)]

Does anybody know any good C++ RPC library which uses templates and which does not need code generating with any external tool nor executable?

Yup, sockets. Every RPC-ish system I'm aware of (Sun RPC/XDR, CORBA, SOAP, RMI, ASN.1) needs a code generator that produces the stubs which make it easier than using raw sockets. The code that's produced by these stub compilers can be pretty small and well optimised (apart from SOAP), plus you shouldn't need to edit it by hand. Some compilers, such as a decent one for CORBAs IDL, can also produce the boilerplate code that you then fill in with your implementation of the RPC calls. While I usually dislike generated code, when it comes to RPC systems I'm quite glad they do a decent job of hiding complexity from me.

Re:

[be-fan (61476)]

The sheer pain of doing non-trivial code generation with C++ templates makes it not worth it. Even something relatively simple like Boost.Lambda, which doesn't generate all that much code when you think of it, is nearly unusable because of how much it shows down compilation and how throughly it messes up any error messages for errors made in the vicinity of a template call. It would be a massive PITA to use an API that generated a non-trivial amount of marshaling code using the template mechanism.

Re:OT: A good C++ RPC library without code generat

[indifferent children (842621)]

Does anybody know any good C++ RPC library which uses templates and which does not need code generating with any external tool nor executable?

Yes, CORBA. You can do DII (Dynamic Interface Invocation) on the client side, and DSI (Dynamic Skeleton Interface) on the server-side. You are never required to use generated code with CORBA. OTOH, the amount of code that you will have to write using DII/DSI is large (not as large as the generated code would be, but large), and usually a PITA. BTW, you can mix a

The license

[Anonymous Conrad (600139)]

Is basically the MIT license [opensource.org] with a few tweaks to the first paragraph (e.g. person -> person or organisation), the second paragraph expanded to cover some of the ideas in the middle section of the BSD licence [opensource.org] and the third paragraph verbatim (or practically verbatim). Note that it appears equivalent to the MIT license in that there's no non-endorsement clause as you'd find in BSD or Apache 1.1 [opensource.org].

Pretty low level, but interesting

[kabdib (81955)]

Same ideas that have been around for 20+ years, but I have to admit it's a fairly nice implementation of a close-to-the-wire protocol.

They could have gone more flexible and abstract; structs are *bad* for you, and they're missing a fair amount of opportunity to make things dynamic, e.g., growable arrays, hashes, sets, arbitrary nested structures, and even things like canonicalized timestamps, which are a quite important (but often neglected) platform-dependent type (see how often time gets mangled when you go multi-platform...).

As for efficiency, it wouldn't be hard to be better than SOAP. I have some horror stories...

Re:Pretty low level, but interesting

[FueledByRamen (581784)]

growable arrays

On the transport layer? That doesn't make any sense. The endpoint implementations insert into language-standard growable arrays (PHP native indexed arrays, C++ std::vector, et al), as they should.

hashes

Easily represented as maps.

sets

They have those. Templated type 'set' in the Thrift interface file (just like std::set).

arbitrary nested structures

And those. map<string, map<string, set<string> > > is a perfectly valid construct in the Thrift file, and will emit (as you might expect) the same thing using STL data structures in the C++ endpoint, or nested assocative arrays in the PHP endpoint. The same thing applies to non-templated structure nesting as well, and to templating around user defined structures.

canonicalized timestamps

There's no good reason to make a separate timestamp class; an int64 is plenty big enough to hold microseconds (or nanoseconds, even) since the epoch.

Pointless criticism

[Smack (977)]

So they wrote something in-house, for their own reasons. Open-source advocates say "release everything... it'll be useful to someone". So they did release it, and then they get slammed for not using the existing standards and because people don't like their methodologies.

Bravo.

Re:

[alienmole (15522)]

I don't think open source advocates would claim "release everything, and no-one will criticize it". Criticism is crucial to open source (and everything else), since that's how the good stuff is separated from the bad. Someone who doesn't want to ever receive any criticism should simply avoid doing anything and interacting with other people.

One pragmatic argument for releasing your code is then you'll find out how good and useful it really is, compared to the competition, beyond just what your own little t

Re:Facebook is releasing this?

[Khuffie (818093)]

You do realize that Facebook is a fantastic example of a clean, functional UI that only uses fancy 'web 2.0' features ala AJAX where it's actually useful?

MySpace, on the other hand, is a piece of shit.

Re:

[EveryNickIsTaken (1054794)]

I'm sorry, but to a large extent, the users make the site. And when the majority of your users are teenage morons.. well.. your site tends to go to shit.

Re:

[Carthag (643047)]

That's irrelevant. The users didn't make the code so they have nothing to do with this release.

Re:

[Khuffie (818093)]

a) I was talking about the site's functionality, design and features b) Have you even used Facebook? My guess, from your opinion, is not. The vast majority of facebook's users are above 18.

Re:

[yabos (719499)]

That's funny you say that since Facebook's users are not mostly teenage morons.
http://www.comscore.com/press/release.asp?press=10 19 [comscore.com]
Hmm, seems 12-17 year olds only make up 14% of Facebook users.

Re:Facebook is releasing this?

[NickCatal (865805)]

I can't agree more! And the creator of Facebook has said multiple times that they are not going to allow custom CSS on profiles or crazy stuff like that. They have also gone a long way with privacy settings (after much outrage in the facebook community)

As a college student, I love Facebook. I use it to keep up with high school friends, keep in contact with people from the school I transfered from, know the people in my classes so I can throw questions at them if I have one, and since I am bad with names it is a great tool to remember people by!

The information I have on Facebook you could probably [wikipedia.org] find [nickcatalano.com] elsewhere [google.com]. But having such a clean interface is great. and their improvements are going to be great [facebook.com] (membership required)

Re:Facebook is releasing this?

[Ngarrang (1023425)]

Ignoring the users of Facebook, the site is about the use of technology. They made their job easy on themselves and created a framework to make their own jobs that much easier. And...in the same spirit of other companies, are releasing that software to the public. I applaud this. May the best framework win!

Re:

[dintech (998802)]

Its also as good a way as any to get developers outside your company to train in using your platform. You don't even have to pay them for their training, or while they're doing it. Then you can hire the good ones out of the community. Doesn't look so altruistic now does it? :P

Re:

[dkf (304284)]

It's not another one. It's just REST with a custom security layer on top (and not even done so entirely transparently; ick!) Moreover, you still need either an XML parser or a JSON security hole, err, parser. To cap it off, facebook don't use it with HTTPS so who knows what mischief some man-in-the-middle could cause?