Coming Soon, The Google Translator 418
compuglot writes "Google gave journalists a glimpse of its next generation machine translation system at a May 19th Google Factory Tour. "Google Blogoscoped" offers an excellent overview of the presentation.
The system has been trained using the United Nations Documents as a corpus. This corpus is some 20 billion words worth of content. It uses existing source and target language translations (done by human translators at the U.N.) to find patterns it then uses to build rules for translating between those languages. Apparently it was successful where the current version had failed in translating certain phrases.
If anyone were capable of making a serious go of MT, that would have to be Google."
Needs a *bit* more work... (Score:4, Interesting)
Just to illustrate, here's the summary of this story, translated to German and back to English using Google's current version [google.com]:
Google's translator (Score:3, Interesting)
Integrate with GMAIL! (Score:5, Interesting)
Tired of getting email from Amazon.DE on my Gmail account and having to copy and paste it over to Babelfish.
That would be very useful for me.
if anyone... (Score:5, Interesting)
IBM
Look how far they ran with chess programs, because they felt like it...
If they decided to go the same distance with translation...
oh no! (Score:5, Interesting)
This will mean one less possible career for me, and fewer babelfish induced laugther moments.
As a fluently bilingual person, I often recognize expressions that were translated in Canadian government documents. "Anglicisme" is the word the french have for it.
There's subtlety to languages we may forever lose. Take for example:
"Je donne ma langue au chat" - "I give up (answering a riddle) instead of the more picturesque "I give my language to the cat". Well, that should be tongue, but hey, it's just babelfish!
"Bullshit" won't produce "merde de taureau". That is a strange expression you anglos have, don't you realize?
"Il pleut comme vache qui pisse" will give us "it's pouring cats and dogs" rather than "it's pouring like cows' a'pissin". The french also have never heard of cats and dogs falling from the sky.
While an improved Babelfish may improve our mutual comprehension, please pause for a moment to consider all the linguistic hilarity we'll forever lose.
Re:Unsupported assertions (Score:5, Interesting)
Were you talking about the PhDs at universities busy teaching classes, churning out research papers to avoid being fired (an ugly numbers game some departments play), or perhaps burning time generating volumes of grant paperwork?
Oh, maybe you were talking about the scientists employed by the private sector. I'm sure the management teams wherever they work are willing to take the time and care that Google won't.
You do know how may PhDs Google employs, right? Not to mention that they won't be fighting for resources there either. No backstabbing, liquidating MBAs trashing their corporate budget. No football-crazed alumni assassinating their funding proposals either.
Also, I would remind you that "mere scientists" often come up with the needed research (there are volumes in MT alone), but rarely can afford to put in the years that it takes into a good implementation.
Geeks love Google because it is, in many respects, where the best of business meets the best of academia.
Re:fascinating (Score:5, Interesting)
Why can't a dictionary be made of nouns, of verbs? Why can't we have it statistically analyze the grammar for ambiguous words?
Does it only recognize exact matches? Especially with verb conjugation, I'd think any words 80% similar or so should be considered matches. Not all languages are as conjugation happy as latin or spanish or even english, and you often lose some nuanced conjugations when translating from one to the other.
What will be done about idioms? Translating these word for word often makes no sense at all, and for me at least (no idea what the official stance is), I'd rather they substitute in idioms with the same general meaning, but for the culture being translated to.
Does it work on alternate character systems, is it word boundary dependent?
Does it understand punctuation rules, will this post translated to spanish have the upside down question marks where they're supposed to be?
How many of the world's existing languages have enough text for this to even be feasible?
Time to move the AI bar (Score:4, Interesting)
Second, I think we'll witness a case of having the AI ante upped once again when another traditional AI challenge is met. Wikipedia puts this best; When viewed with a moderate dose of cynicism, AI can be viewed as 'the set of computer science problems without good solutions at this point.' Once a sub-discipline results in useful work, it is carved out of artificial intelligence and given its own name.
Comment removed (Score:5, Interesting)
Re:Unsupported assertions (Score:4, Interesting)
And you're right, people have thought of this exact idea (I'm sure every other computer major and linguist has, in fact, since the birth of ENIAC--I know the idea's crossed my mind tons of times, not that I'd have the slightest clue how to do it), however actually attempting to do it with a reasonable chance of success? I'm going to say Google is the first.
Plus, I got the impression from the article that the serve is operational, just not available to the public. If you'll read the article, you'll find that the translator properly translated a fairly complicated phrase from Arabic to English. I'd guess that this service is, from a technical standpoint, at least 95% done -it's just the packaging and touching-up that needs to be done.
Wait, why? (Score:5, Interesting)
Fortunately, that's not all that google has to go on. Google has 8 billion webpages, in many different languages, most of which are written by non-speechwriters. Not only can they analyze words based on translated context, but they can analyze words based on intra-language context, to form associations between words and meanings.
The real trick is getting down two important linguistic concepts: "Sandhi Rules" (for instance, the use of "an" before a vowel and "a" before a consonant, which are totally regular but more complicated than a word-to-word matchup), and the "degree" or "quality" of words, which indicate the type of adjective most appropriate in any given context.
For instance, "erudite", "learned", "educated", "knowledgeable", "skilled", and "cunning" could all be related words, but many of them have positive or negative assocations which may only really be conveyed by understanding the meaning, irony, or sarcasm of a particular phrase.
For instance, "John has been skilled in writing beautiful code for most of his adult life" is quite different from "John has been educated in writing beautiful code for most of his adult life", or "John has been erudite...". The first one is probably right if John has had a natural inclination to doing it properly, the second if he has undergone some training (though we don't know the actual state of his ability), the third (though the word doesn't even really make sense here) if he has been arrogant about his ability, shouting RTFM! every time someone asked him a question.
words don't really have meanings (Score:5, Interesting)
Re:fascinating (Score:4, Interesting)
The often-quoted examples are: "Out of sight, out of mind" becomes "invisible idiot" and "the spirit is willing, but the flesh is weak" comes out as "The meat is rotten, but the wine's great".
How many of the world's existing languages have enough text for this to even be feasible?
Ah yes, that's the tricky part. Translating for preservation near-extinct languages that are in spoken or recorded form only. A true programming challenge.
I find the Babel-Fish translator to be nearly useless and the Systran box at www.systransoft.com very helpful when selling things on eBay to people in non-English-speaking countries. When I get a question about an auction item that has little grammar cohesion and has a offshore domain, like
"How many cost you Italia he transport?", I'll run my response through Systran's translator and add the original english afterwards. More often than not the sales and PayPal transactions are successful.
I believe that machine translation will be the 'killer application' for 64-bit home PCs.
There are five levels of machine translation:
1) word substitution.
2) phrase substitution.
3) cohesive paragraphs and idioms.
4) light literature, magazine articles, and business.
5) classical literature, law, and diplomacy.
Each level requires at least an order of magnitude more computing power than the previous one. Babel fish is on level two and systran is on three. Google is positioning themselves to be between levels four and five.
I wish them the best of luck. Without sarcasm or irony. This is important work.
"Give me a one sentence definition of 'irony'."
"Yeah, it's where the Iranians come from."
Re:fascinating (Score:5, Interesting)
I think this is precisely where statistical approaches can really shine. A purely dictionary-based conversion will translate an idiom word-for-word, which will make no sense at all. However, a statistical approach could be constructed to look for the "longest reliable match." So if the idiom "cat got your tongue" re-appears over and over, and is correlated to a different idiom in other languages (that may not use the word "cat"!), then the algorithm could tokenize "cat got your tongue" as a single entry that would map to something different in each language.
How many of the world's existing languages have enough text for this to even be feasible?
You're right... that's the killer. Translating using statistics (especially idioms) properly will require a huge database of samples. Even what's been suggested so far is not enough. If we want to translate technical documents, we need a new database. If we want to translate "free form writing" we need yet more data.
However, there's lots of data out there (already in digital format) that could be used... we just need people to see the potential and start using these datasets (or making these datasets available). For instance, for technical stuff there are thousands of abstracts for papers and for theses that are translated into various languages (for instance, many articles published in german are then also released in english... I live in Quebec, and every thesis abstract has to be translated into french also... etc.). Many legal documents (many of which are already available to the public) are also translated for various reasons. It would also be interesting if translators all around the world uploaded documents they had translated into some database (assuming it's nothing sensitive of course!). As this database grew, it would become more and more reliable. Let's face it, there's tons of human-based translation going on, forming a massive dataset... but by and large it's just scattered and not useable.
Re:fascinating (Score:2, Interesting)
The two sources used by Google are basically the only sources available for the kind of task we're talking about. Obviously the thing to do is work on creating more cross-language corpora, and I'm sure this is being done, but it takes much time to create a cross-language corpus on the scale that the UN documents or translations of the Bible have.
Re:fascinating (Score:2, Interesting)
I'd be more worried about homonyms, especially ones that are used in the similar contexts. I wonder if it will be able to handle sentences like "I turn left here, right?", which manage to confuse even humans at times.
Re:DVD's subtitle tracks (Score:2, Interesting)
I watch a lot of anime, and a lot of fansubs, subtitles are the worst way to learn a language.
Re:fascinating (Score:3, Interesting)
A novel or book is not translated like this, the best translation aren't word for word or sentence to sentece. Good translators almost rewrite the whole thing, some times with a different style.
Language has a lot of cultural meaning into it, and even the same language sometimes needs to be adpted to mean the same (and I am not saying anything about accent). Computers will hardly get to this point, I would expect from this a good 'well, at least I got the point' translation.
Re:DVD's subtitle tracks (Score:2, Interesting)
Re:except, no. (Score:3, Interesting)
This is so true. I remember being utterly amazed when my toddler was able to immediately spot a bird in real life based off a cartoonish caricature in one of his children's books. It just flabbergasts me how a mind so young can perform recognition that we can't achieve with a beowulf cluster of supercomputers.
A better solution (Score:2, Interesting)
The basic idea behind Blissymbolics is to use mostly indexical ideographs - that is to say, eg, the symbol for man looks somewhat like a stick figure man. There are some pure symbols, however, though they somewhat conventional - for instance, a heart shaped symbol represents emotion. However, it is not limited to concrete meanings, and, though I doubt it could be proved, I believe it's has the same capability for expression as any other writing system, including English writing, due to its compositionality. Couple that with the fact that it can be learned quite easily, one might begin to see that yes, this is a better solution. I am dedicated to this ideal, so if you get a chance, check out http://www.activebliss.com/ [activebliss.com] for more information about the ideal of universal communication.
Cheers,
Matt Landau
Re:fascinating (Score:3, Interesting)
As long as the translations have been created in advance, and you can obtain copies of the works in question, it should be fine, legally. I cannot see a way that a court could find the machine-state of a translation machine to be a "derived work" in the copyright sense, and it's certainly not making any literal copies.
Now, someone could distribute a text under a license agreement that forbid this type of usage, but a court decision may well find that it's a protected "fair" use. And I can't think of many texts that have license agreements that would restrict something like that.