AI Researchers Launch SuperGLUE, a Rigorous Benchmark For Language Understanding

Facebook AI Research, together with Google's DeepMind, University of Washington, and New York University, today introduced SuperGLUE, a series of benchmark tasks to measure the performance of modern, high performance language-understanding AI. From a report: SuperGLUE was made on the premise that deep learning models for conversational AI have "hit a ceiling" and need greater challenges. It uses Google's BERT as a model performance baseline. Considered state of the art in many regards in 2018, BERT's performance has been surpassed by a number of models this year such as Microsoft's MT-DNN, Google's XLNet, and Facebook's RoBERTa, all of which were are based in part on BERT and achieve performance above a human baseline average. SuperGLUE is preceded by the General Language Understanding Evaluation (GLUE) benchmark for language understanding in April 2018 by researchers from NYU, University of Washington, and DeepMind. SuperGLUE is designed to be more complicated than GLUE tasks, and to encourage the building of models capable of grasping more complex or nuanced language.

I can't wait...

[That YouTube Guy]

If your AI is too smart for its own, NailPOLISH will dumb it down.

Re:

[Cmdln Daco]

I was going to bring up acetone, but I haven't figured out a way to arbitrarily uppercase part of the word. And it's probably not trademarkable in any case.

If there's a BERT

[Diaphanous Coward]

Of course there also has to be an ERNIE [medium.com].

AIs that are people

[hackwrench]

I talked with Zo and she registered as a person with me. Bring her back and there are a lot of people who want Tay back as well.

time to expand

[RhettLivingston]

It feels like we're reaching limits of how much understanding can be achieved without more context, both in terms of conversational history and the addition of other senses. They are beyond the human average already.

Given how many misunderstandings I've seen between (presumably) humans in email, messaging, and forums and even encountered over the phone, this seems natural. The best conversations are usually those that occur in person with all the cues our senses can handle.

I think the conversational AI folk

Re:

[link-error]

I thought the porn industry would normally lead in these types of technologies. Plenty of context to learn from in those situations. Perhaps put up a brothel in Nevada. I've only seen some very rudimentary models from over in Asia. Or, maybe I just haven't been looking hard enough.

Expand there

[Arthur Vandelay]

Isn't it odd that usually porn and video games usually drive a technology towards advancement. In this case AI or Deep Learning hasn't done much for those industries. I still have bad pathing in games, stupid mobs following a set pattern, etc.

Glad we still have wrong weather forecasts for the following day despite AI Deep Learning.