Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
AI Technology

AI Has a Compute Dependency Problem, Facebook VP Says (venturebeat.com) 108

Posted by msmash from the closer-look dept.
In one of his first public speaking appearances since joining Facebook to lead its AI initiatives, VP Jerome Pesenti expressed concern about the growing amount of compute power needed to create powerful AI systems. From a report: "I can tell you this is keeping me up at night," Pesenti said. "The peak compute companies like Facebook and Google can afford for an experiment, we are reaching that already." More software innovation will be required if artificial intelligence is to grow unhindered, he said, and optimization of hardware and software -- rather than brute force compute -- may be critical to AI in years ahead. [...] "We still see gains with increase of compute, but the pressure from the problem is just going to become bigger," Pesenti said. "I think we will still continue to use more compute, you will still net, but it will go slower, because you cannot keep pace with 10 times a year. That's just not possible."
This discussion has been archived. No new comments can be posted.

AI Has a Compute Dependency Problem, Facebook VP Says More

AI Has a Compute Dependency Problem, Facebook VP Says

Comments Filter:
  • You keep using the term A[rtificial] I[ntelligence]. I do not think it means what you think it means
    • Also, Google did engineer optimized hardware, which they are calling Tensor Processing Units (TPUs)

      • Tensor Processing Units (TPUs)

        Eight, sir; seven, sir; six, sir; five, sir; four, sir; three, sir; two, sir; one!

      • yes this again, because people can't get it into their heads that any form of AI doesn't yet exist, if ever it will. We do have decades old techniques (nothing new since the 1970s) that can be done by modern fast hardware to do sometimes useful work.

        symbolic AI? genetic algorithms? Neural nets? If you're under 40 years old it's older than you are....

        • To be honest, there are *SOME* new techniques, but .. well.. in one case its a rediscovered old technique that turn out to be really good at pre-training neural networks, something that wasnt discovered until relatively recently. Specifically the use of RBM's as a pre-training step. Turns out where RBM's leave off is a very good place to start back-propagation.

          Anyways, it is correct that for the most part modern "A.I." is just the old "A.I." but on much faster hardware and with a lot more data. The faster

          • The faster hardware isnt the essential part, either

            Binarized neural networks are around 99.7% accurate compared to large-word neural networks, and you can add more nodes to make up the difference. So e.g. instead of 1,000,000 nodes with 64 SRAM bits per each weighted factor consuming 6 transistors each, you can have 1,500,000 nodes with 1 SDRAM bit per each weighted factor to represent any value from the set {-1,1}.

            The connection count goes up, but that's still 1.5M connections to 9M transistors instead of 1M connections to 64M transistors; and it's sti

        • Re: (Score:1)

          by Tablizer ( 95088 )

          yes this again, because people can't get it into their heads that any form of AI doesn't yet exist

          How do you know it doesn't exist if you can't define it? I cannot say that Foo doesn't exist if I cannot define Foo, otherwise nobody would know what to look for to verify it's existence.

          Yes, I will agree that many of the techniques discovered while doing AI research end up being called "AI", but as I mentioned in the linked thread, defining something by what goal existed in people's heads while discovering/inv

          • AI has been defined simply and easily for a long time, it's just that no computer system meets the criteria (of performing the tasks a human can do) so it's bad for marketing.

            • Re: (Score:1)

              by Tablizer ( 95088 )

              You mean define it as the essence of something we have yet to achieve? It's like a reverse definition: "X is all matter which is not a unicorn." That's one approach.

        • because people can't get it into their heads that any form of AI doesn't yet exist, if ever it will

          So you want to reserve the widely used concept of "AI" for something that may never exist, and then all agree on a new term to describe what people are working on right now ? For what reason, exactly ?

          • For the reason that computers still routinely fail at tasks humans can easily do, and we shouldn't let marketing buzzwords deceive people to believe something that isn't true.

        • Artificial Intelligence is just that: artificial intelligence. It's not General Intelligence. GI is a different problem, and one that's...let's just say the pieces are all there and nobody's put them together just yet. It's current tech, but it's undiscovered current tech.

          You're not going to need to invent e.g. room-temperature superconducting alloys for this.

        • Today, in the pages of "The Times" of London (once a reputable if unreliable newspaper of record), I saw a full-page advertisement for *toothbrushes* with "artificial intelligence".

          One sees why Tom Lehrer retired from writing satire.

    • Oh yeah!, You're right, they should totally call if F[ake] P[roblem solving] instead.

      Cause, you know, it's not like these systems are ACTUALLY solving problems that could only be solved by people before.

      The solutions they provide are fake because they only come from a simulation, even if they work and actually solve the problem - still fake!

    • Re: (Score:1)

      by Anonymous Coward

      Forget that! They're using compute as a common noun, and my reaction is uncontrollable rage!

    • I prefer the term I[diot] S[avant] and that’s being generous.

    • If everybody calls this AI, then this is what the term means.

    • Re: (Score:2)

      by gtall ( 79522 )

      Yeah, and you are a bot given the intelligence it takes to pick a headline word and repeat, "I do not think it means what you think it means".

  • This is long overdue. (Score:5, Insightful)

    by lusid1 ( 759898 ) on Monday July 15, 2019 @05:35PM (#58930706)

    In the early days compute was the expensive resource, so developers wrote efficient code that could make the best use of the available compute. These days development time is the expensive resource, so most products are a tangled mess of inefficient building blocks strung together to minimize development effort, masked by relatively abundant compute. AI forces a compute resource crunch that will require a return to efficient coding, and that is long overdue.

    • Re: (Score:1)

      by Anonymous Coward

      While this is maybe true for some types of software, in my experience it is not the case for "scientific software" like this. Sure, there may be a lot of bloat in the software packages used etc. but at the end of the day, the actual kernel consuming 99% of the compute is often highly optimized, so the waste is only in the surrounding software used only a fraction of the time. The reason it hasn't been optimized is due the very fact it uses so few resources, so it even if it was essentially optimized away to

  • Just tie everything into Skynet's vast network; problem solved!

  • CS (Score:4, Insightful)

    by JBMcB ( 73720 ) on Monday July 15, 2019 @05:39PM (#58930736)

    optimization of hardware and software -- rather than brute force compute -- may be critical to AI in years ahead.

    It's almost as if there is an entire field of academic study dedicated to solving this problem. Weird.

    • Re: (Score:2)

      by EvilSS ( 557649 )

      optimization of hardware and software -- rather than brute force compute -- may be critical to AI in years ahead.

      It's almost as if there is an entire field of academic study dedicated to solving this problem. Weird.

      Is? From what I've see it's more like "was". At least on the software side. Optimization has been pushed out of the mindset of the majority of devs. Hell forgoing optimization has even been enshrined by some as a waste of money. Traditional development has been cursed with an over-abundance of hardware resources for a while now. Don't spend time optimizing, we'll just throw more hardware at it! We need to ship!

      • In 2012 on the Kerbal Space Program forums, something along the lines of "when are you going to optimize loading speed? You appear to be parsing text files an load time, which you shouldnt be doing, and you appear to be doing it poorly as well"

        The response was of course a hundred eager feature seekers that said "optimization can wait"

        Fast forward to 2019, still parsing text files or something, and whatever its doing its still doing it poorly. There shouldnt be enormous load times on such simplistic smal

        • Re: (Score:2)

          by EvilSS ( 557649 )
          That's funny, I have a customer with a piece of software THAT THEY WROTE AND SELL that does something similar, and they can't get their devs to stop doing it. It has a configuration file (pure ASCII text file mind you, about 50 lines long), and instead of caching the config at launch by reading it and storing the settings in those new-fangled variable things, they read from it thousands of times a second, every time a parameter stored in it is used by the software. Absolutely murders disk IOPS when you run

          • Re: (Score:2)

            by Falos ( 2905315 )

            This is cancer and it hurt to read.

            Hurt. Like I was programmed by God to recall it every time my heart beats. Every word. Like running a full DRM routine every time a video game character changes position (xyz coords). On the store-shelf release.

            I saw "pacman" on my friend's iphone. 150MB. Not cosmetic remake or anything, 40-years-vanilla pacman. Not exactly a runtime remark, but still.

          • Well the thing is they did it wrong at he start, and instead of fixing it when it was easy to do so, they kept doing more and more of the wrong stuff. Now its too much to ever fix.

            "Optimization can wait" is the worst thing ever said. No, it can't.

      • The kind of optimization you make in Comp-Sci is not the same kind that you do on your average business app to make it 10x faster.

        It's the Big O notation stuff where you analyze a combinatorial problem that would take the longer than the heath-death of the universe to complete, and find a way to re-express it so that it takes a mere few hours (or weeks) of processor time. That's the kind of thinking that is needed in the "powerful AI systems" that these companies are doing in their backyard for internal con

  • Problem? What problem? (Score:4, Interesting)

    by DontBeAMoran ( 4843879 ) on Monday July 15, 2019 @05:45PM (#58930762)

    Maybe they should try using Intel's Pohoiki Beach, a neuromorphic computer capable of simulating 8 million neurons.

  • incomprehensible (Score:3)

    by epine ( 68316 ) on Monday July 15, 2019 @05:54PM (#58930804)

    Is he the VP of suspending the laws of physics/economics, and are the terms of his employment such that whether he lives or dies next week depends upon this? If so, he's got a real problem on his hands, and I can see why he's not sleeping well.

    Otherwise, as the world turns.

    • Yeah, I grew up worrying about nuclear war, he's not sleeping well because good programming is more helpful than brute force for some algorithms? wtf?

      Maybe he should discontinue some of his medications instead.

  • ... on the Void Which Binds. Duh (rolleyes)

  • An Interesting Technique for Some Problems... (Score:3)

    by raftpeople ( 844215 ) on Monday July 15, 2019 @06:03PM (#58930862)
    Reservoir computing is a relatively new technique that seems to be a promising approach from an efficiency perspective for some problems.

    The summary is that there is a reservoir (a portion of the system like a neural network) that evolves state according to the input and previous state. The reservoir does require that it has certain attributes, but other than that it can be essentially random connections/weights. Instead of training the entire network, there is a transformation of input into the reservoir to determine next state, then there is a simple final layer that is trained to map the reservoir state to the desired output.

    It's apparently a more efficient system for time based sequences compared to training a recurrent neural network.

  • ... quantum computing for the consumer is right around the corner, but I'm not saying how close that corner is.

    • ... quantum computing for the consumer is right around the corner, but I'm not saying how close that corner is.

      We can't know how far away the corner is at the same time that we know what angle the turn is, so while I agree it is just around the corner, I don't think that implies that we will get there.

  • AI does not have a compute problem... we attempting to create AI have the compute problem.

    Any serious AI is years away... likely still a century. AI development is just completely at odds with business operations. We need AI to make money right now it just can't do this. Sure we have lots of knock off AI that is not even close to AI that does great calculations but those are just algorithms.

    Until AI is able to rewire itself and recode itself we are not going to have an AI close to sentient quality. What

    • Um pretty sure he doesnt mean AI in that sense, he just means deep learning models are costly to train.

    • Until AI is able to rewire itself and recode itself we are not going to have an AI close to sentient quality.

      Who says that we want that ? If the AI is working properly for the task it was designed for, it is generally preferable that it doesn't attempt to rewire itself, and risk making new mistakes.

  • As factories scale up and patents expire, the cost of the hardware is only going to go down. It's a solution in search of a problem, and AI is too interesting of a potential problem to ignore. (Along with other stuff like protein folding and crypto breaking by three letter agencies.)

    At some point, someone (probably at some three letter agency, but maybe not, maybe a medical researcher or public AI researcher) is going to scribble some back of the napkin numbers and realize that it's feasible to build or
    • And talking about AI in particular (since that's what the article is about), it's likely that the biggest breakthroughs will involve self-modifying code. Which is something brute-forcable. If "strong AI" emerges (whatever that means and entails), it will probably have written itself. It will have bootstrapped itself in ways that are probably incomprehensible to intuitive human analysis. And--without denying the obvious importance of optimization and self-modifying code algorithms and rules used--I believe

  • How failed at the first part, convincing us we need either chickens or eggs.

  • The GPU revolution in recent years is not enough. It feels like optimization has taken a back seat with an embarrassment of luxury that modern CUDA/OpenCL hardware has provided the industry.

    On the other hand, GPU hardware keeps improving performance even better than today's anachronistic CISC CPUs from Intel and, to a lesser extent, AMD are.

  • "Compute" is a verb (Score:3)

    by iliketrash ( 624051 ) on Tuesday July 16, 2019 @04:40AM (#58932756)

    "Compute" is a verb, you f*cking nitwit.

    • Re: (Score:2)

      by necro81 ( 917438 )

      "Compute" is a verb, you f*cking nitwit.

      I came here to make the same comment, although maybe not with expletives. Something along the lines of "since when did 'compute' become a noun?" But then I realized it was a VP talking, and the article was on venturebeat.com, and I realized I could immediately disregard the whole thing.

      For context, here's IBM's Tom Watson Jr [ibm.com] back in 1970:

      A foreign language has been creeping into many of the presentations I hear and the memos I read. It adds nothing to a message

  • All this fancy computing isn't free.

    It's kind of like those who fear that all our jobs are being taking over by robots. They forget that the most sophisticated the robot has to be, the more expensive it is.

  • I guess that this kinda puts to rest the idea of a artificial intelligence which is exponentially growing more intelligent.
    Maybe it doesn't prohibit an AI which is more intelligent then it's creators but not one which is constantly growing....

    I'll put money on an general AI also having a problem with latency. As more compute units are added to it, then the latency of the entire system will decrease unless there's compute redundancy and acceptable errors that creep into the outputs of the system.

    Anyway, it's

Slashdot Top Deals

Love may laugh at locksmiths, but he has a profound respect for money bags. -- Sidney Paternoster, "The Folly of the Wise"

Close