Forgot your password?
typodupeerror
Android Google Operating Systems Technology Hardware

Android ICS Will Require 16GB RAM To Compile 357

Posted by timothy
from the all-that-dessert-makes-you-sluggish dept.
ozmanjusri writes "New smartphones may be lightweight, compact objects, but their OSs are anything but. Ice Cream Sandwich will need workstations with no less than 16 GB RAM to build the source code, twice the amount Gingerbread needed. It will take 5 hours to compile on a dual quad-core 2+GHz workstation, and need 80GB disk space for all AOSP configs. Android developers are also being warned to be cautious of undocumented APIs: 'In almost every case, there's only one reason for leaving APIs undocumented: We're not sure that what we have now is the best solution, and we think we might have to improve it, and we're not prepared to make those commitments to testing and preservation. We're not claiming that they're "Private" or "Secret" — How could they be, when anyone in the world can discover them? We're also not claiming they're forbidden: If you use them, your code will compile and probably run.'"
This discussion has been archived. No new comments can be posted.

Android ICS Will Require 16GB RAM To Compile

Comments Filter:
  • Of Course. (Score:5, Funny)

    by Frosty Piss (770223) * on Sunday October 23, 2011 @07:37PM (#37813176)

    Nobody will ever need more than 16GB...

    • by hedwards (940851)

      16GB is an awful lot of RAM, I'm really curious as to what it is that they're doing that's going to require more RAM than most of these devices have in total storage space.

      I get that optimizations take memory and that there are likely independent steps, but still 16GB of RAM?

      • Re: (Score:3, Informative)

        probably using the -pipe CFLAG.
        • by chrb (1083577)

          If I remember correctly, you could build OpenOffice on Gentoo with -pipe and various other performance tuning flags, and the hardware requirement was only a minimum of 512MB. And every other package, including big stuff like the kernel and KDE, could be built with only 64MB... though 256MB was recommended.

          I would guess the default Android build is optimized for the Google Android team, and so speed is the most important factor; they probably use a build server with multiple processors and big memory and d

          • by wvmarle (1070040)

            I would also guess that there probably aren't that many people outside of Google who build their own Android images.

            And those that would, are most likely big companies that have a very good reason for wanting to do that, and also don't mind having to buy a bit more RAM. It's not that expensive these days anyway.

            • Re:Of Course. (Score:5, Informative)

              by Calos (2281322) on Monday October 24, 2011 @05:06AM (#37815168)

              Mmm, no. Third party modders do a lot of work, and make some really awesome builds, with all kinds of customizations and new features. Cyanogenmod, for instance. Quite the opposite of working for a large company with resources, their developer are now actually being hired by big companies because of their freelance work.

      • by iluvcapra (782887)

        16GB is an awful lot of RAM, I'm really curious as to what it is that they're doing that's going to require more RAM than most of these devices have in total storage space.

        These are the hardware requirements to compile the complete AOSP Android system and platform, and not the requirements to merely develop an application on it.

        • Looking at the article; the 16GB is because they compile the code in parallell so need lots of memory. They get the 5 hours of build time down to 25 min.

      • This is a guess as to the reason.

        One of the better ways to optimize C++ code for building with GCC is to put all of the source code into one big code file. Or you can build it as a few independent modules, but the code is still quite large. Then you build it with the O3 flags. In GCC, the amount of RAM and CPU used in an O3 compile goes up by quite a lot as the code size in a single module increases. I am not sure what the exact equation is but I think it's an exponential function.

        This would easily explain

        • by Rich0 (548339)

          I run Gentoo and usually run make with -j5 on a tmpfs, and I've never managed to use even half of my 8GB RAM building anything from chromium to firefox to openoffice. And I certainly don't skimp on my CFLAGS.

          Maybe if you build this thing on a tmpfs and run -j50 or something you'd need that kind of RAM, but seriously...

          Plus, since parallel make tends to limit itself to a single module at a time in most build systems it is hard to get the parallelism to be all that high anyway.

          I'll take them at their word, b

        • by tyrione (134248)

          This is a guess as to the reason.

          One of the better ways to optimize C++ code for building with GCC is to put all of the source code into one big code file. Or you can build it as a few independent modules, but the code is still quite large. Then you build it with the O3 flags. In GCC, the amount of RAM and CPU used in an O3 compile goes up by quite a lot as the code size in a single module increases. I am not sure what the exact equation is but I think it's an exponential function.

          This would easily explain the RAM and CPU usage.

          And when LLVM/Clang gets full Concurrency in 3.1 you can bet Google will be put GCC in the rear of the bus with LLVM/Clang taking over the wheel.

        • by smash (1351)

          maybe they should switch to clang [llvm.org].

      • Re:Of Course. (Score:5, Interesting)

        by fuzzyfuzzyfungus (1223518) on Sunday October 23, 2011 @07:53PM (#37813260) Journal
        I have to wonder if the 16GB "requirement" is more of a recommendation and/or a bunch of default settings that deliberately avoid the disk as much as possible, and keep as many cores as you can throw at the job busy by compiling every little bit and piece in parallel...

        On the one hand, with 16GB of RAM in the desktop/light workstation 4x4GB only running around $100(with the more workstation-friendly 2x8GB with ECC only twice that), it seems rather pointless to burn any developer time on trying to optimize the RAM needs of building the entire OS. RAM is cheap.

        On the other hand, I have to wonder what they could possibly be doing to the process of compiling what is basically a weird-but-not-unrecognizable linux distro to make it that RAM hungry.
        • Re:Of Course. (Score:5, Interesting)

          by mjwx (966435) on Sunday October 23, 2011 @08:03PM (#37813304)

          I have to wonder if the 16GB "requirement" is more of a recommendation and/or a bunch of default settings that deliberately avoid the disk as much as possible

          I have to wonder if the 16 GB requirement is real.

          Reading the blog linked to in the summary, there is no source mentioned. The author completely fails to mention how they came across this information. Even ignoring their bad English (obviously not their first language).

          I think I'll wait for a more trustworthy source to confirm or deny this.

          • by SomePgmr (2021234)
            Indeed, and google searches only return results to that blog post with no sources, slashdot and slashdot-reposts. It's like wikipedia-style circular references.
          • Re:Of Course. (Score:4, Informative)

            by kidgenius (704962) on Sunday October 23, 2011 @08:48PM (#37813516)
            Here's the original source over at Google Groups from JBQ http://groups.google.com/group/android-building/browse_thread/thread/3757b189f4e93df0?hl=en&pli=1 [google.com]
            • Re:Of Course. (Score:5, Informative)

              by PopeRatzo (965947) * on Sunday October 23, 2011 @09:02PM (#37813576) Homepage Journal

              And if you read that original source, you'll see that they are recommendations for building future development machines:

              -6GB of download.
              -25GB disk space to do a single build.
              -80GB disk space to build all AOSP configs at the same time.
              -16GB RAM recommended, more preferred, anything less will measurably
              benefit from using an SSD.
              -5+ hours of CPU time for a single build, 25+ minutes of wall time, as
              measured on my workstation (dual-E5620 i.e. 2x quad-core 2.4GHz HT,
              with 24GB of RAM, no SSD),

              • by chrb (1083577)
                Looking at those specs, maybe it's about time to think about switching Android to a modular architecture. There is no reason why the complete build needs to be made in one go. It's like running Gentoo and compiling the entire system from source when you really just want to upgrade an application. Or like the old OpenOffice days, when they bundled everything, so compiling it required compiling every library that it depended on, plus compiling Python and everything else that was embedded. Building/distributin
                • It is modular. Different packages can be built separately. (I haven't checked ICS, but the last release from the AOSP was modular).

                  Although one caveat: figuring out the command-line to build the module you want may be beastly.
              • by PopeRatzo (965947) *

                I'm not an Android developer, or any kind of developer. Why is "wall time" so different in this case from CPU time?

                Maybe I don't understand completely what "CPU time" really means. A little help here?

          • Sounds like the dude tested it himself and just checked how much RAM was in use during compilation... or something like that?

      • by citizenr (871508)

        16GB is an awful lot of RAM, I'm really curious as to what it is that they're doing that's going to require more RAM than most of these devices have in total storage space.

        I get that optimizations take memory and that there are likely independent steps, but still 16GB of RAM?

        Im guessing they are running compiler in ARM virtual machine, compiler is of course written in JAVA.

      • I'm really curious as to what it is that they're doing that's going to require more RAM than most of these devices have in total storage space.

        Being lame. Apparently, being a certified smart person does not necessarily imply understanding how to code efficiently, including not understanding how to code a build efficiently. An excellent example of how Google hobbles its development pace by its fear of embracing community developers.

        • Just by way of illustration, there was one guy on my team at Google whose solution to a QA problem was to run the whole build under QEMU, increasing the latency of the build by two orders of magnitude. Turning it into an overnight process in fact. And this guy's approach had the full support of my manager, even being tech lead of the team I was not allowed to overrule that braindamage. No technical reason at all, just pure process of its own sake. This kind of mess is endemic at Google, including in some of

  • Next Android will need 32GB.... Then 64GB... Soon, Skynet level of resources....
  • 16GB recommended, or else you'll be waiting quite awhile. But, you could build the thing in less RAM than that, right?

    • by XaXXon (202882)

      Yes, it can always swap out. The compiler has no visibility into whether the memory space it is executing in is actually mapped to physical ram.

      But like other people said, it might take a really long time.

      • The compiler has no visibility into whether the memory space it is executing in is actually mapped to physical ram.

        If the compiler doesn't know its own resident size, then how does top know the compiler's resident size, and how does ps aux know the compiler's resident size? I imagine that if a program detects that it's being swapped out, it might be able to adjust its CPU/memory tradeoffs at runtime.

  • by Anonymous Coward on Sunday October 23, 2011 @07:57PM (#37813280)

    While Android will remain open-source, eventually it will require so much RAM/processing power/etc. to compile that only Google will have the computational resources available to compile it.

    Clever!

  • by Twinbee (767046) on Sunday October 23, 2011 @07:57PM (#37813282) Homepage

    Quick question for those with giant codebases such as this. How the heck do you test, and debug the software with those kind of lag times? Do you split everything up into smaller pieces or something? If so, then surely there are cases where you need to test something that requires EVERYTHING to be compiled. I can imagine such shot in the dark scenarios to be the stuff of pure nightmares.

    • by dwheeler (321049) on Sunday October 23, 2011 @08:09PM (#37813338) Homepage Journal
      Unless the build system is screwed up, recompiling after a change should be relatively fast. Usually source code is stored as lots of smaller files, and each file is compiled separately to produce a separate object file (e.g., .o). Then next time a rebuild is requested, the system should notice what changed, and only rebuild the needed parts. Some parts take the same time each time (e.g., a final link), but it shouldn't take anywhere near the same amount of time. There are lots of build tools, including make, cmake, and so on. If you use the venerable "make" tool, you might want to read Miller's "Recursive Make Considered Harmful": http://aegis.sourceforge.net/auug97.pdf [sourceforge.net] Cue the lovers and haters of "make", here :-).
      • In game programming, Incredibuild is a common tool for that. You run it on everyone's machine and it integrates with Visual Studio. Lets you reduce build time a ton since you have a lot of resources to use. Also tends to scale nicely as the larger the project, the more people working on it and thus the more computers available and so on. You can, of course, have dedicated servers just for compiling but many places don't bother, just having it use idle time from office systems as it is amazing how much that

      • by ben_kelley (234423) on Sunday October 23, 2011 @09:00PM (#37813568)

        Unless the build system is screwed up, recompiling after a change should be relatively fast. Usually source code is stored as lots of smaller files [...] Then next time a rebuild is requested, the system should notice what changed, and only rebuild the needed parts.

        I feel your pain brother.

    • Back in my college days we had to submit a compilation job on the mainframe, and then wait around for a couple hours for someone to put the printout containing the results (or more likely a crash dump) into the appropriate mail box slot. Then you had to wait your turn to submit a revised copy. (No, this wasn't that long ago -- 89, 90, something like that -- but the community college I went to still taught their Cobol & assembly classes on an older mainframe -- 3270 terminals though, no punch cards).

      Bu

      • by epine (68316)

        No, this wasn't that long ago -- 89, 90, something like that

        Seems pretty unlikely unless you were in a deep backwater. Interactive terminals became commonplace very early in the 1980s. It wasn't uncommon to work on a batch processing system until the mid 1980s, but not with results delivered on paper.

        The development model you're talking about properly dates to the 1960s and into 1970s, in backwaters where the future had yet to penetrate. FFS, the Xerox Alto [wikipedia.org] was introduced in 1973.

        Sixteen years later, you

      • by swalve (1980968)
        It was like that at UIC in 1993. I had to walk to a different building to pick up printouts. It was a thrilling experience.
      • by sconeu (64226)

        1982 WUSTL. It was an IBM 370. Towards the end of one semester, when everyone was submitting jobs, I had a 12 hour turnaround time for a 30 hour program.

    • by Osgeld (1900440)

      As others have said, you dont recompile the entire thing cause you changed one integer, but as others have not said you really should be testing in smaller chunks, you are not perfect enough to vomit out something that takes 5 hours of CPU time (which on the given systems is about a half hour of real time) perfectly the first time you try.

      Its much easier to write a chunk and make sure it works than to write a freeking monster blob and go hunting for a chain reaction.

    • by swalve (1980968)
      Easy: if it compiles, you release it and let the users test it.
  • not true (Score:5, Informative)

    by MrCrassic (994046) <deprecated&ema,il> on Sunday October 23, 2011 @08:06PM (#37813324) Journal
    Here's what the article *actually* says:

    16GB RAM recommended, more preferred, anything less will measurably benefit from using an SSD.

    Emphasis mine. Still pretty beast, though.

    • by Sycraft-fu (314770) on Sunday October 23, 2011 @08:24PM (#37813412)

      While it is a lot of RAM compared to what many system have, it really isn't a big deal these days. 4GB DDR3 sticks are $25 or less each, and that is for high quality RAM. Regular, consumer grade, LGA1155 boards support 4 of them. So for $100 you can have 16GB on a normal desktop system. My home system I type this on has 16GB for that reason. It was so cheap I decided "Why not?"

      They actually can support more, with 8GB chips you can have 32GB on a standard desktop, but those are still expensive.

      The enthusiast X79 LGA2011 boards coming out will have 8 sockets and thus handle 64GB. Of course beyond that there's workstation which cost a lot more, but not as much as you might first think.

      At any rate, 16GB is now a "regular desktop" amount of RAM. Standard boards the likes of which you get in cheap ($1000 or less) towers support that much, and it only costs $100 to get. It is quite a realistic thing to require, for something high end.

      • by Rich0 (548339)

        At any rate, 16GB is now a "regular desktop" amount of RAM.

        Well, it is an amount of RAM you could cram into a brand new regular desktop, but it certainly isn't something you'd find on an average desktop. I think I have two slots free in mine so I could bump it up to 16GB, but that is $50 I don't really need to spend. I rarely am using more than half of my RAM as it is, though the extra obviously helps with caching/etc.

        Android has always been RAM-intensive, and it makes sense since you have no choice but to build an entire OS at once (not like you can dynamically

        • by Lehk228 (705449)
          my ~1.5 year old desktop has 8 gigs and a SSD, but i'm crazy like that
        • Android has always been RAM-intensive, and it makes sense since you have no choice but to build an entire OS at once

          Can't you do the initial build overnight, come back the next day, do more development, and recompile only the parts that changed?

      • My nearly two year old desktop has 12GB of ram in it, and can take 24GB (i7 930 based system). My next build which should be in December or January will likely have 32GB and expandable to 96GB (Dual socketed X79).

      • by peppepz (1311345)
        Oh come on, it's twice the amount that Sandy Bridge notebook processors support at the CPU level, and it "isn't a big deal" these days?

        -- A pissed Android user who just bought a € 1.000 laptop maxing out at 8 GB

  • TFA says:

    5+ hours of CPU time for a single build, 25+ minutes of wall time, as measured on a workstation (dual-E5620 i.e. 2x quad-core 2.4GHz HT, with 24GB of RAM, no SSD)

    25 minutes of wall time is nothing for a first build. After that updates from changes in the source code will be trivial.

    25 minutes to build a complete linux distro is fantastic.

  • shitty /. summary (Score:5, Informative)

    by petermgreen (876956) <plugwash.p10link@net> on Sunday October 23, 2011 @08:18PM (#37813380) Homepage

    TFA: "5+ hours of CPU time for a single build, 25+ minutes of wall time, as measured on a workstation (dual-E5620 i.e. 2x quad-core 2.4GHz HT, with 24GB of RAM, no SSD)."
    /. Summary: "It will take 5 hours to compile on a dual quad-core 2+GHz workstation"

    • Of course that was the /. summary. Most people, including me, have never heard of "wall time" and I was trying to figure it out, until I did the calculation and figured out that "wall time" is the clock on the wall, not some fancy compiler term I've never heard before.

      Thanks /. for making it more complicated than it needs to be.

    • by wvmarle (1070040)
      You know, /. summaries are just teasers, to get you on the subject. For the meat you have to either read TFA or just read the comments.
  • by damn_registrars (1103043) <damn.registrars@gmail.com> on Sunday October 23, 2011 @08:25PM (#37813418) Homepage Journal
    I was looking for something else to do on my old 16-cpu Itanium cluster with 64gbs of shared ram. I think I just found it...
  • ...does a phone really require an OS of that complexity? Don't get me wrong, I have a current generation Android smartphone I bought 2-3 months ago, 4G enabled, it even has an HDMI out, and I completely comprehend that a modern smartphone is essentially a fully fledged computer.

    That being said, it's still a phone. And in fact, it's horrible at it. To redial the last number I have to press 3 buttons (1 physical, 2 virtual) and suffer through 3s+ of erratic lag or more. On my 4 year old boring but fun
    • by arth1 (260657)

      ...does a phone really require an OS of that complexity?

      It's not the OS complexity. It's the laziness of programmers, who want everything abstracted through seven layers until they can write single lines of code that through black box code magically expands into what they think they want done, not that they can ever know for sure.
      And the poor compiler works overtime, because it has to redo the same work every time someone writes object.method().

      There are reasons why so many of the kernel developers (including Linus Torvalds) are adamantly against using "higher

      • by JBMcB (73720)

        And the poor compiler works overtime, because it has to redo the same work every time someone writes object.method().

        A larger problem is the hacked-up Java VM Google came up with to work around licensing problems with Sun/Oracle (fat lot of good that did them) Dalvik is a fairly poor implementation of a Java VM - on the same hardware the embedded version of Java runs much faster.

        I feel the OP's pain. I have a pretty fast android device (Archos 43) and the UI feels smooth about half the time, the other half it's flaky and unresponsive. My wife's iPod touch never so much as hiccups. I still use a cheapo feature phone as a c

    • by wvmarle (1070040)

      I have the same issue with my smartphone. Considering buying a second phone, a more traditional "dumb phone", to go with it. My LG P500 has pretty poor sound quality even. For the rest it's a great device.

      That, and I don't understand why Android can't just read it's contacts from an LDAP server.

    • by muridae (966931)

      It's not just the OS, I believe. It's the whole kit that takes that long to build. Remember, there are all the debugging tools, the emulator, the virtual machine, the OS, and all the user interface goodies. In the OS you then have drivers for every different HDMI graphics chip that Android supports natively, drivers for the different radios, bluetooth, usb, drivers for the touch screens, sound chip drivers.

      How long does it take you to compile the whole linux kernel, not just the few parts you use?

      As for you

  • What is it actually doing that needs 16GB of RAM to compile?

  • I wonder how long a full compile of that takes...

  • by wisnoskij (1206448) on Sunday October 23, 2011 @09:38PM (#37813760) Homepage

    Unless the entire program is in 1 gigantic 8 billion billion billion line file why would it need that many resources or even be able to use 16 GB of RAM?
    Assuming it is like a normal program would it not just be a large collection of relatively small files that are compiled one after the other (theoretically number of CPUs + 1 threads running at a time with that many files being compiled concurrently being the optimal solution)?
    And I just do not see how you could ever use up 16 GB at any one time.

  • by JackAxe (689361) on Sunday October 23, 2011 @09:45PM (#37813788)
    This article would be shocking, but considering that 16 GB of memory -- especially the dual-channel DDR3 used for the i5 and consumer i7's -- is so cheap, less than $100, this article doesn't have any shock value. It's just informative. It's letting us know the 'recommended' memory and giving more nerds an excuse to add more RAM. That is the NERDS that don't already have 24 gigs for their virtual machines. :P
  • Note that this is the entire system, so building Ice Cream Sandwich from source is just like rebuilding an entire Linux distribution from scratch from the ground up, from the kernel down to the user tools and the windowing system, etc. If you've ever used Gentoo, this should sound familiar. I wonder if some of the tools that Gentoo users are familiar with to help speed along compilation, such as distcc [gentoo.org], could help with this.

  • Is this a case of preprocessing gone wrong? Sometimes preprocessing can be a monster because it blows up each .c file into a monster file due to (almost) every .h file being included, which lead to long compile times. This is especially the case when you have large numbers of .c files. Preprocessing was invented as a hack in the time that memory too small for the whole source to be compiled in one time. But when applied to large systems, it causes the compile time to increase, because every piece of code in

As the trials of life continue to take their toll, remember that there is always a future in Computer Maintenance. -- National Lampoon, "Deteriorata"

Working...