Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

Using The Web For Linguistic Research

Posted by timothy on Sun Jan 23, 2005 03:50 AM
from the that's-rediculous dept.
The Internet Science
prostoalex writes "The Economist says linguists are gradually adopting the World Wide Web as a useful corpus for linguistic research. Google is used, among other resources, to research how the written language evolves and how some non-standard examples of usage become more or less acceptable (The Economist quotes the phrase 'He far from succeeded,' where 'far from' is used as an adverb). LanguageLog is a resource linked in the article, where linguists discuss current peculiarities of the English language."
+ -
story
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Using The Web For Linguistic Research 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.

  • They should probably avoid Slashdot (Score:5, Funny)

    by Peter Cooper (660482) on Sunday January 23 2005, @03:56AM (#11446691) Journal
    It's probably a good thing that they steer away from Slashdot as a corpus of English usage. Or, should I say, in SOVIET RUSSIA it's best Slashdot stays away from THEM! Or is it that only old people use the Internet as a corpus of the English language while pouring hot grits down a naked and petrified Natalie Portman's pants?
    • Hopefully, they'll harvest well written webpages for data and not those of 13-year old girls drooling over Orlando Bloom, AOL users, or porn sites.

      Actually, I take that back.

      It could actually be very interesting from a lexical or morphological point of view. The phenomenon of abbreviating words, such as "u" for "you" or "ur" for "you're" or "ru" for "are you." Language teachers in classrooms have been seeing it crop up in actual homework assignments. While reading such language may be like having glass
      • Because the keyboard is still the main way of inputing information into the computer, people take short cuts

        One thing that's always been at the front of my my mind, why aren't these kids learning how to type? Or at least to type with any reasonable amount of skill. The only computer I had as a child was a Commodore 64, and I was still faster than most of todays youth even with their abbreviations. I was somewhat lucky in that our schools somehow foresaw the advent of the home computer and made sure we kne

          • Because, unlike the parent's assumption, the phenomenon isn't related to computers. It's related to text messaging. It might be just as fast to type "you" instead of "u" with a keyboard, but it's noticably slower on mobile phones, especially before predictive text became popular.

            I'd still advance the argument that extensive use of computers by a larger portion of the population has contributed to the phenomenon. I remember seeing those abbreviations before cell-phone use became almost ubiquitous.

            It's al

      • It's troubling to read so many comments that worry that the linguistic researchers will find "bad language", and worse, that people have moderated such comments up. It reflects a misunderstanding of what linguists do: they want to get a description of the language as it is used, and as it changes, and historically speaking, usages that start in the gossip of teenage girls often become mainstream a couple of generations later. They need it all, and they probably need the crappy stuff most of all, because
        • I for one am not under that misconception. However I disagree with many /.ers who argue that the fluidity of language means that making it up as you go along is acceptable. "It's" is not a posessive, and you can't loose your shoe if the laces are lose.


          Right now, yes. But in a generation or two, perhaps they'll lose the distinction.

          "sociological"


          Whoops. :-)
    • What do you think Natalie Portman would do if she actually viewed Slashdot some time?

  • Indeed (Score:4, Funny)

    by Pan T. Hose (707794) on Sunday January 23 2005, @03:57AM (#11446693) Homepage Journal
    Indeed what their sayin is true. U can learn English very well, especially grammer readin /. frist psots. Teh intarweb seems to certainly kick arse for that sorta research. Very 1337 articel. Thx d00dz.

  • I rue the day... (Score:3, Funny)

    by sandstorming (850026) <johnsee.sandstorming@com> on Sunday January 23 2005, @03:57AM (#11446694)
    When we might actually say words like 'lol' out aloud. Imagine a deal going down between two mining companies and the CEO of one company with a straight face, and deadly serious demeanour saying to the cameras: "Despite many thinking we pwned them in the deal, we believe it came out leet for every1"
    • When we might actually say words like 'lol' out aloud.

      I've heard it done. I've also heard 'roffle' (an attempt at pronouncing ROTFL I guess). Bizarre, really, since those terms are attempts to turn physical real-life actions into a verbal-only form.
      • I guess I'm sorta immune to that stuff, then - maybe I'm the only one, but when I see something like "lol" or "rofl", it translates in my head to more of an idea than an actual sound - in essence, a loud "heh" rather than it's own word.
    • You may be unaware that "lol" actually is a correct word in the dutch language, meaning (having) fun.

      lol (de ~) 1 [inf.] plezier
      (taken from, www.vandale.nl, an authoritive dutch dictionary)
  • It came to me that the English language was in deep trouble when people started saying "rotfl" and "lol" in person. There seems to be kind of a backlash brewing though, with improved email composition styles dictated by employers, and such.

  • Google does it again (Score:3, Interesting)

    by vladd_rom (809133) on Sunday January 23 2005, @04:03AM (#11446708) Homepage

    This is not the first time when Google (and search engines in general) changed how we do things.

    Nowadays copyrighters use Google to search for potential violations of their intelectual property. Plagiarism is easy to detect nowadays thanks to Google as well. Instead of using rather expensive [turnitin.com] systems in order to search for duplicate work, teachers are now one search away in distinguishing original work from the rest.

  • *BSD be dyin' (Score:2, Funny)

    by Anonymous Coward
    It be now official. Netcraft gots confirmed, dig dis: *BSD be dyin'

    One mo'e cripplin' bombshell hit da damn already beleaguered *BSD community when IDC confirmed dat *BSD market share gots dropped yet again, now waaay down t'less dan some fracshun uh 1 puh'cent uh all servers. Comin' on de heels uh a recent Netcraft survey which plainly states dat *BSD gots lost mo'e market share, dis news serves t'reinfo'ce whut we've knode all along. What it is, Mama! *BSD is collapsin' in complete disarray, as fittin'

  • Be carefull thought... (Score:3, Interesting)

    by Anonymous Coward on Sunday January 23 2005, @04:23AM (#11446742)
    There are more non native speakers on the web then
    native speakers.
    In the European community the native English
    speaking persons are by far a minority. That way
    French expressions are poring into the language
    in an unstoppable way. Those expressions are then
    used by native speaking politicians and are
    broadcasted by television. That way they enter the
    mainstream of the English language.

    Regards
    • Who needs to be careful? Hopefully the Internet *will* cause languages to merge. It could be like the Tower of Babel in reverse. Wouldn't it be great if there was a unified global language?

      Now I know some people would be quite upset at the horrible "loss" of cultural diversity implied by a single global language. But we can be just as diverse in many other ways that don't cause us to be unable to communicate with each other on a basic level. And IMHO, being able to communicate is much more important

      • Wouldn't it be great if there was a unified global language?
        Now I know some people would be quite upset at the horrible "loss" of cultural diversity implied by a single global language. But we can be just as diverse in many other ways that don't cause us to be unable to communicate with each other on a basic level. And IMHO, being able to communicate is much more important than some academic's ideal of "cultural identity".

        Okay... how about the complete loss of the ability to read any of the world's litera
        • You're overdramatizing. This is a process that will take hundreds if not thousands of years, even with technology helping to accelerate it. It's not like we'll wake up 10 years from now with a unified language and forget how to read today's literature!

          By the time we have a unified language, we'll have a whole new set of literature to go along with it. Today's literature will be like ancient greek literature, and yes, it will only be readable by people with special training. It will need to be translat

    • Done: nous sommes desolés que notre president (Score:4, Insightful)

      by new500 (128819) on Sunday January 23 2005, @06:40AM (#11447059)
      . . .

      Those expressions are then
      used by native speaking politicians and are
      broadcasted by television.

      Dude, it's worse, the French have already infiltrated as far as the advertising business and are using covert channels to spread some dangerous crack i heard was called La Liberte :

      http://french.about.com/b/a/081281.htm

      Slightly more seriously :

      Apart from pointing out that your use of the word native is rather presumptive of geographic origin in this big wide internet thing, i wonder if this linguistic adoption is more one way towards English since the internet. OK the French got Le Weekend, and tons of anglicised nouns, tried to ban them all and didn't manage. But i read Friday that a British pilot training firm lost a contract to a French one. The reason cited by the Asian airline was that, whilst the training had to be in English, the French trainers spoke better, clearer, more intelligble English than did the English. I can't argue with that. Sadly.
    • Such as carefull, poring, broadcasted? ;)

  • I've used the web for corpus linguistics research (Score:2, Informative)

    by Anonymous Coward
    I've used the web for corpus linguistics research. My last big project was to look at a lot of web pages with Mexican and Chilean slang Spanish, and see if there was a difference in vocabulary usage. There was a significant difference; I could, 70% of the time, tell if a given passage was Chilean or Mexican Spanish.

    I could have gotten a higher accuracy rate, but this was just a simple undergraduate project.
  • There needs to be an anual prize for the highest compression ratio using random pages from the web as the corpus. This would probably do more for real advancement of artificial intelligence than the Turing competitions.

  • Non-official English (Score:2, Informative)

    by Anonymous Coward
    Unlike French and Italian, there is no official instution that defines 'correct' English. Essentially, the English-speaking world just 'makes it up' as it goes. Thus when I see the adverb 'really' butchered into 'real' I must try not to get annoyed. i.e. It's real hard to use your mother tongue. vs. It's really hard to use your mother tongue. Please help me here - is the misuse/non-use of 'really' something that's taught in school?
    • From Merriam-Webster Online [m-w.com]:

      real (3, adverb): VERY (he was real cool -- H. M. McLuhan)
      usage Most handbooks consider the adverb real to be informal and more suitable to speech than writing. Our evidence shows these observations to be true in the main, but real is becoming more common in writing of an informal, conversational style. It is used as an intensifier only and is not interchangeable with really except in that use.

      I'd say you're fighting a losing battle on this one. I'm not too bothered by it, e

  • Good! Natural language is a moving target. The web is an excellent communication medium and ignoring it would be quite a
    silly move. The example reminds me of "To boldly go", which was not proper, but its elegance is hard to argue against.

  • 'Language' == spoken || written? (Score:3, Insightful)

    by adam31 (817930) <adam31@ g m a il.com> on Sunday January 23 2005, @04:49AM (#11446804)
    How do you even pronounce 'pwn3d' ? Google is not a tool to study speech patterns, and there's nothing to say that speech even resembles written text.

    The article addresses this in a weird way, where it first draws attention to the distinction, but once it reaches its crux, where google is used as a tool, the distinction is ignored entirely; instead it opts to focus on stranger things.

  • I woulda thght such a thng was unpossible.

  • Popular usage != wanted usage (Score:3, Informative)

    by KiloByte (825081) on Sunday January 23 2005, @05:33AM (#11446892)
    Yes, we can record the errors made by the uneducated public (and even those done by, uhm, me). The question is: should we do that or not?

    I was pretty taken aback when a council of linguist in Poland suddenly declared some widely-chastised and not even very popular errors to be valid usage. I've been brought up in the circles of people who not only put a lot of stress to the language you use, but also cruelly point out every incorrect word or phrase you use -- and this made me quite intolerant to bad speech.

    Being but a dirty foreigner, I know that my English can sound bad in the ears of native English speakers -- that's why I sometimes ask people to correct me if they spot errors.

    In other words: some people find careless speech repulsive. Thus, we should do whatever we can to promote correct usage as opposed to legalising incorrect uses.

  • Three types of language (Score:4, Interesting)

    by Dracos (107777) on Sunday January 23 2005, @05:34AM (#11446894)

    I think that for most of the 20th century, English, and most languages in the industrialized world, was largely static, dominated by the written word which was dominated by proper grammar. Since WWII, popular culture and faster communications have increasingly exposed us to local vernaculars, mostly through radio and television. The written word lagged behind in its cultural evolution.

    Thanks to the internet (initially email, BBS's and IRC, but more widely known on the Web), we now have a hybrid of the spoken and written word: the "typed word". This form of language evolves at the same rate as the spoken word, and injects its own vernacular as a side effect of the medium: acromyn and abbreviation "words" (rofl, how r u), along with common misspellings (pwned), and mixing letters with numbers or punctuation (133t, n00b). All of these serve at least one purpose, whether as a form of super shorthand, insult, the appearance of being "cool", or are merely the result of laziness on the part of the author. Most typed-word terms don't transfer well when spoken.

    One of my hobbies is studying (European) languages and how they are related. Sometimes I worry about the damage the typed word is causing to the spoken and written word (and any proper linguist should at least be interested in the phenomenon). Luckily, most typed word expressions aren't pronounceable, and the ones that are sound absurd, because they are removed from their original context when spoken, and everyone recognizes gibberish when they hear it. How the typed word affects the written word remains to be seen. Yes both are typed now, but only the written word has a chance of going through an editorial process. I think it will take a very long time for the formal lexicon and rules of grammar to embrace, however reluctantly if ever, the typed vernacular.

    • I think that for most of the 20th century, English, and most languages in the industrialized world, was largely static, dominated by the written word which was dominated by proper grammar. Since WWII, popular culture and faster communications have increasingly exposed us to local vernaculars, mostly through radio and television. The written word lagged behind in its cultural evolution.

      You do realise that most of the 20th century happened after the second world war, don't you? A condition that became false

  • I've had the chance to use Google as a grammar or style checker in my day job as a glorified copy editor. I type two nearly identical expressions X and Y in the search box. If expression X gets 10,100 hits and expression Y only 500 hits, I use expression X.

    For example, as a non-native speaker, I found myself waffling between the expression (A) "run for mayor of" and the expression (B) "run as mayor of." Letting Google arbitrate, I found 14,900 hits for (A) and only 200 hundred hits for (B). I chose (A).

    I

  • Linguists are gradually adopting the World Wide Web as a useful corpus for linguistic research.

    I love a bit of cunning linguistics.

  • Reminds me of "Meme Tree"... (Score:4, Informative)

    by Slur (61510) on Sunday January 23 2005, @05:49AM (#11446935) Homepage Journal
    ...which was this little program I wrote around the nascence of the internet. it took any sentence as input and kept a record of which words preceded each word, and which words followed each unique word. The idea was to build up a simple map of which words could precede or follow others completely without context. From this you could follow paths that made sentences or paths that looped forever, or paths that made no sense, and some interesting paths that made unintended sense.

    Why a tree? Language and geneology seem to have a common thread. Meaning is like genetics. Language is expressive. Information is a kind of tree whose branches grow as reality elaborates and past events accumulate. New terms need to be invented for the dynamics we perceive in reality, just as new names are given to individuals as they emerge into the world. Patterns, continuity, periodicity. Such things lie at the heart of material existence and provide the hooks for consciousness itself. Information theory is the next great frontier, along with particle physics. Already they have converged and diverged and converged again. And playing with artificial trees turns out to be a lot of fun.

    As for the "Meme Tree" program ... The next iteration built up a more discreet map by scoring proximity of unique words in sentences and inclusion in sentences together. Again, the idea was to build a simple statistical map free of any context, simply to get a sense of pure lexical association.

    The theory is that the internal consistency of these various lexical maps should roughly reflect many aspects of associative meaning. You could think of the statistical map as a Godelian bubble whose "truth" - if you will - is imposed by the laws governing the statistical associations. We don't derive the laws of language and meaning from these exercises, but we create an internally-complete map that reflects something about the nature of meaning.

    There is a practical aim as well. If you can derive the strength of equivalence and the various levels and colors of associative meaning you could in theory build a "Truth Machine" capable of answering any question with a high degree of accuracy. The result of any question could be computed as any other information retrieval problem would be.

    I never got around to having my little Meme Tree programs scrape the internet for random sentences. However, this should be a very simple thing to do. Google has had programming contests in the past - programs that use the Google database in interesting ways. Statistical analysis of language is basically what they do. Research projects on their data could provide stunning insights into the nature of information itself, its relation to language and to reality, and likely into our very nature as linguistic beings.

  • BBC voices (Score:2, Informative)

    by matt me (850665)
    Link on front page of bbc.co.uk - bbc.co.uk/voices/ [bbc.co.uk] - their attempt at tracking accents and dialects across the UK.

  • Writing in Japanese (Score:4, Insightful)

    by minairia (608427) on Sunday January 23 2005, @08:04AM (#11447234)
    I am American but have to write in Japanese for work. No matter how much one learns in school, when one writes in a foreign language, you'll hit a point of wondering if what you wrote is how native speakers say something or is even understandable. Whenever I hit a point like that, I put the sentence in question (or key fragments thereof) into a Google search. If nothing comes up, I know I have to rewrite. If only a few links come up, I know what I wrote might be a little wierd, but is at least understandable. If I get pages and pages of links, I'm golden.
  • I use search engines all the time for linguistics reseach: when I'm reading or translating from one language to another, and I run into an odd usage, I just type the phrase in the magic box and *poof*, I get hundreds of contextual examples. Likewise, if I'm writing in a foreign language, and I need to know if a preposition or a construction is correct (and not simply words), again all I have to do is type it in and see what comes out.

    Measuring how the internet changes world languages is only a small part o
  • My personal interest has been in using Google to return pages related to some search query and then data mining the text on the referenced pages (my company develops a product called theConcept [mesadynamics.com] for OS X). For example, doing keyphrase analysis on the first 100 pages returned in the results from the Google search "linus torvalds" returns key pairs such as:

    • operating system
      linux kernel
      free software

    And citations linked to those pairs such as:


    • Linus torvalds as the moving force behind the operating system t
    • Something like pluralising virus to Virii?
    • "It does ? ... It does." [Partridge@Equilibrium]

      So, yes, it does seem a fracking uselessly mistyped/misknown/miswrote/misthought way to express oneself. But that is changed now, because some people with too much time on their hands think it is a new form of expression and this is the way the English language is changing. So now we are supposed to treat these insentient ideas as the new ways ? Bahh, get lost.

    • i countinously question my co-workers (social workers) in telling the youth what is propper and not.

      I'm glad they're telling the youth what is proper; you're clearly incompetent to do so.

      using words... is becoming more than just the normal, it is becoming the standard.

      Is that right? Using words is "becoming more than just the normal"? I've been using words for years now; I'm glad to hear that's becoming the standard. Your post is a perfect example of why people should learn to write in something a
        • >His meaning is perfectly intelligible, but some language snobs (very few of whom are actually linguists and know anything much about language) pretend not to be able to understand certain accent/dialects in order to feel superior.

          Incomprehension often has very little to do with that. A friend of mine moved to MA from NC at the same time as I moved from CA. She could not understand most people there, most people there could not understand her. I could, on the other hand, understand both of them. I've be
    • Oh christ, not you whackjobs again. You've infested our forum [dailyjolt.com]. Sure, --; is a neat emoticon, but when could one ever use it? On a seperate note, anyone cringe when reading "He far from succeeded."? On a completely seperate note, anyone notice how programmers write with slightly different grammar? Extra punctuation always goes outside the ", never inside, as above.
      • Adding or changing characters in a literal string seems like misquoting. Traditionally in handwritten work the comma went almost directly under the quotation mark. When people shifted to typewriters and then computers, an arbitrary choice was made to put the comma first. Most programmers I meet seem to have reversed that choice.

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2009 Geeknet, Inc.