China Trained a 1-Trillion-Parameter LLM Using Only Domestic Chips

"China Telecom, one of the largest wireless carriers in mainland China, says that it has developed two large language models (LLMs) relying solely on domestically manufactured AI chips..." reports Tom's Hardware. "If the information is accurate, this is a crucial milestone in China's attempt at becoming independent of other countries for its semiconductor needs, especially as the U.S. is increasingly tightening and banning the supply of the latest, highest-end chips for Beijing in the U.S.-China chip war." Huawei, which has mostly been banned from the U.S. and other allied countries, is one of the leaders in China's local chip industry... If China Telecom's LLMs were indeed fully trained using Huawei chips alone, then this would be a massive success for Huawei and the Chinese government.
The project's GitHub page "contains a hint about how China Telecom may have trained the model," reports the Register, "in a mention of compatibility with the 'Ascend Atlas 800T A2 training server' — a Huawei product listed as supporting the Kunpeng 920 7265 or Kunpeng 920 5250 processors, respectively running 64 cores at 3.0GHz and 48 cores at 2.6GHz. Huawei builds those processors using the Arm 8.2 architecture and bills them as produced with a 7nm process."

The South China Morning Post says the unnamed model has 1 trillion parameters, according to China Telecom, while the TeleChat2t-115B model has over 100 billion parameters.

Thanks to long-time Slashdot reader hackingbear for sharing the news.

Re: I can do better

[sirv]

| gz

Re:

[Rei]

Exactly this. Trained it how well? I could make a model of arbitrary size and train it on a couple tokens, and ZOMG, I trained a huge model! It's not a meaningful metric.

Re:

[DrMrLordX]

SMIC doesn't really have a "5nm" process. Multipatterning can only take them so far.

And the US spends a boatload of money on education. Money alone does not produce results.

China's economy is already imploding. Enjoy the fireworks.

Re: What does not kill them, makes them stronger

[dwater]

Already imploding. Rofl.

How many times have we heard this over the past few decades?

Re: What does not kill them, makes them stronger

[memory_register]

Economic implosion does not happen all at once like in a movie. Look at the birth rates in China and extrapolate.

Re:

[angel'o'sphere]

Birth rate has nothing to do with economy.
Unless you run out of farmers ...

Re:

[ShanghaiBill]

Birth rate has nothing to do with economy.

Birth rate has everything to do with the economy, and the effect is predictable.

Initially, a falling birth rate means a lower dependency ratio as adults have fewer children to care for, leading to more two-earner families and strong economic growth.

Then later, the dependency ratio worsens as the number of pensioners soars, with a single couple burdened with four grandparents.

China reaped the demographic dividend [wikipedia.org] with remarkable growth for thirty years, but that is all going into reverse and will get much wo

Re:

[angel'o'sphere]

And all this has nothing to do with "economy". No one cares what the ratio of earners versus retirees is. You either can do it or you can't.

Economy means: is there wealth generated. Are the lower classes of the society moving upward? Is there enough food, housing, clothing, education for everyone? And all that has nothing to do with birth rate. Declining birth rates makes everything more simple, not more complicated. You are an idiot.

China reaped the demographic dividend with remarkable growth for thirty ye

Re:

[memory_register]

And all this has nothing to do with "economy". No one cares what the ratio of earners versus retirees is.

There are multiple levels of misinformation in your comments, all piled on each other. Rather than unwinding everything, I'll just say that if you're quoting Greta Thunberg in your bio, you have told me everything I need to know.

Re:

[angel'o'sphere]

Yes, unite behind the science.
Instead of inventing bullshit.

Re:

[evil_aaronm]

And the US spends a boatload of money on education. Money alone does not produce results.

Especially when the money is spent on clear violations of the First Amendment in a blatant attempt at grifting for political candidates: https://cbs2iowa.com/news/loca... [cbs2iowa.com]

Re:

[AmiMoJo]

It temporarily delays China, and then they race ahead with domestic technology. It happens every time we try to sanction them, and we never learn from our mistakes.

Re:

[gweihir]

It has also happened often enough in other contexts that its ineffectiveness is known.

Re:

[DamnOregonian]

Ahead?
I'm struggling to find a technological area they're sanctioned for in the West that they've come remotely "ahead", lol.
Do they "approach parity"- of course they do.
But "ahead"?

Please- enlighten me.

Re:

[AmiMoJo]

5G, WiFi, batteries, solar and wind generation...

Re:

[DamnOregonian]

No, no, no, no, and no.

You seem to be confusing "those who make the shit" with "those who invent the shit."

Re:

[AmiMoJo]

You didn't know that Huawei developed a lot of the technology for 5g and WiFi? That's why their hardware got the market years before everyone else.

Chinese battery tech is the best in the world. Tesla uses both Japanese/American and Chinese batteries in their cars, so it's possible to directly compare them in otherwise identical cars. The Chinese ones are significantly better, and cheaper.

Re:

[DamnOregonian]

You didn't know that Huawei developed a lot of the technology for 5g and WiFi?

Huawei has developed many 5G patents. And some day- they might even catch up with Qualcomm. But I doubt it.
WiFi 6/E? Qualcomm, again.

Huawei is China's biggest telecom tech giant, but they're still nowhere near the technology giant that Qualcomm is.
And you know what Qualcomm never did? Stole the source code from Cisco and get busted for it.

That's why their hardware got the market years before everyone else.

Their hardware didn't. Their hardware was cheap.
I'll take made up stories for $100, Alex.

The first large-scale deployment was in South Korea, using equipment from:

100 billion != 1 trillion

[Gravis Zero]

China trains 100-billion-parameter AI model on home grown infrastructure

All you had to do was copy and paste the headline from TFA. [theregister.com]

Re:100 billion != 1 trillion

[Westley]

There are multiple articles linked in the post, and explicit mentions of multiple models. Yes, the Register one only refers to a 100 billion parameter model; the Tom's Hardware and South China Morning Post articles specifically talk about a 1 trillion parameter model. So does the post itself:

> The South China Morning Post says the unnamed model has 1 trillion parameters, according to China Telecom, while the TeleChat2t-115B model has over 100 billion parameters.

So, two models: one with 100 billion parameters, and one with 1 trillion parameters - at least allegedly. While we may or may not trust the source, there's no point in claiming that the trillion parameter model isn't mentioned.

Re:

[thegarbz]

All you had to do was copy and paste the headline from TFA. [theregister.com]

Yeah but then you'd have to read the register why not paste one of the other headlines: "State-owned China Telecom has trained domestic AI LLMs using homegrown chips — one model reportedly uses 1 trillion parameters" there you go, copy and paste.

A LLM 'eats' 1 Trillion Parameters?

[NoWayNoShapeNoForm]

I bet it was hungry again in a hour /s

Re:

[HiThere]

The Post Office in question is the British Post Office. For some reason the Computer Weekly story didn't make that clear.

And...?

[dwater]

Surely they didn't think their sanctions would stop them? A delay is the best they could hope for...oh, and a big hit to US companies due to the loss of the Chinese market, and more competition around the world...not to mention an increase in the trade deficit...usa:"buy more stuff from us...no! Not that! Only things you don't want...buy more things you don't want!". So, they buy more of the things they can, but don't need, and stockpile them...and the US complains about that too. Sheesh.

Control

[TJHook3r]

It seems curious why a country of a billion people would be interested in developing AI capabilities that would potentially decimate middle-class jobs

Re:

[Baron_Yam]

Because if someone else has it and you don't, you're at an economic disadvantage. If you want the benefits of participation in the global economy, you have to deal with things like that. Both the good and the bad will equalize across borders, some things more rapidly than others.

And then there's the military angle. And surveillance (both domestic and foreign).

In areas where you can trust AI, it's faster and cheaper than humans.

Re:

[penguinoid]

The whole point of communism was that it was designed for when working became obsolete thanks to industrialization, which they thought was imminent. Turns out working wasn't obsolete yet, so the USSR collapsed and China went mostly capitalist. But obsoleting all those jobs is basically their dream -- not like in the US where it means you'll be homeless.

Re:

[iAmWaySmarterThanYou]

Yea because if you're jobless in China you are well taken care of by the state.

Re:

[Rei]

Schrödinger's Illegal Immigrant: Has an endless pot of government-provided cash, but is so broke that they're out scavenging the neighbors' pets for food and cutting their lawns with machetes.

Re: Control

[TJHook3r]

They look after you for the rest of your life!

In before...

[godrik]

... it is revealed that they mean they used NVIDIA chips which they claim is produced in China because Taiwan.

(Thought the Chinese chip are actually not too bad)

"If the information is accurate"

[RitchCraft]

Wow, that sure is a big IF isn't it?

So what?

[Rick Schumann]

1. China lies about shit all the time
2. Even it they're not, what a massive waste of hardware that could have been used for something actually useful instead of braindead excuse for AI

Re:

[Deal In One]

Maybe they should use that computing power to model more advanced weaponry?

Eventually I hope all the ML systems are found to be not very useful. That will burn alot of companies and countries which rushed into a fad.

Parameters are meaningless

[WaffleMonster]

Without at least knowing token count of training set or whether it is a sparse or dense model one can't reason about the costs / system capabilities required for pre-training.

Regardless wouldn't surprise me. China has always had some good models. One of my favorites is a 236B parameter deepseek.

Semiconductor "needs"

[50000BTU_barbecue]

Who needs a parametrized nonsense generator and custom fart sound effect generator?

Parameters? 10 trillion? 1 trillion? 100 billion?

[NobleNobbler]

https://github.com/Tele-AI/Tel... [github.com]

I can't make sense of this.

Limitations

[jon3k]

I know China doesnt have any EUV lithography equipment, but if they can manufacture 5nm, which SMIC already produces domestically, at scale with good yields they can to a large degree overcome the sanctions in scale. They will just throw unbelievable amounts of money into it and boil oceans to generate the power to operate them. They also have complete control in where and how these chips will be consolidated and operated to extract the most value.

I think the question is just long term. Unless they ca

Stolen IP

[angryman77]

Generally speaking, China doesn't care about any other nation's IP, except as something they want. My guess is that the domestically built chips they're talking about are either direct copies of or built from designs derived from an original design that some nation claims as IP.