Reddit is Crashing Because of the Growing Subreddit Blackout (theverge.com) 308
Reddit has been going through some issues for many on Monday, with the outage happening the same day as thousands of subreddits going dark to protest the site's new API pricing terms. From a report: According to Reddit, the blackout is responsible for the problems. "A significant number of subreddits shifting to private caused some expected stability issues, and we've been working on resolving the anticipated issue," spokesperson Tim Rathschmidt tells The Verge. Reddit's status page reported a "major outage" affecting Reddit's desktop and mobile sites and its native mobile apps. [...] More than 7,000 subreddits have gone private or read-only in response to the API pricing terms, which is forcing the developers of apps like Apollo for Reddit to shut down at the end of the month.
Decline is a choice (Score:5, Insightful)
Re:Decline is a choice (Score:5, Informative)
Re:Decline is a choice (Score:4, Interesting)
They just fired a bunch of people as well.
Investors like companies that don't perpetually lose money. Gotta do something about it.
Re: Decline is a choice (Score:5, Informative)
The something could have been API pricing that paid for their costs and a reasonable profit margin, not just try to drive the third party clients out of business. If itâ(TM)s costing Reddit 24Â to serve 1000 requests (even including staffing, R&D etc) theyâ(TM)re doing something *terribly* wrong. AWSâ(TM)s pricing for when youâ(TM)re serving a lot of data is 0.009Â per 1000 requests. Yes, Reddit have a bunch of engineers etc to pay to develop software, but their pricing seems to be several orders of magnitude outside reality. For another comparison point; Imgur charges $200 for the same number of requests as Reddit charges $2,000,000
Re: Decline is a choice (Score:5, Informative)
While Imgur does charge far less than what Reddit wants to charge, Christian apparently is on a grandfathered plan that gives him a much cheaper rate structure than is available now. Christian's Imgur plan is apparently $166 for 50 million requests, or $0.00332 per 1000 requests, so his comparison there isn't really fair.
But devs seem to be happy with Imgur's normal pricing. Imgur charges $500 per month for up to 7.5 million requests and $10K per month for up to 150 million requests; both plans charge $0.001 per extra request over that. That's 6.7 cents per 1000 requests on both plans, presuming you cap. Reddit wants to charge 24 cents per 1000 requests, and that's with a discount offered to Christian for the very high volume from his servers.
Re: (Score:3)
A factor of four in what might be the major cost of doing business could easily be a massive overcharge. I don't know, not being a developer, but imagine if you run a cab company and the price of gas--one of your biggest costs, if not the biggest--went up by a factor of four, and you were given very little notice about the increase. Your entire business model might fail very quickly.
What the devs want seems reasonable: set a cost comparable to other large services, and provide time to implement it. Thirty d
Re: Decline is a choice (Score:5, Insightful)
If it's costing Reddit 24c to serve 1000 requests (even including staffing, R&D etc) they're doing something *terribly* wrong.
The real reason Reddit is doing this is because they were used by OpenAI as training ground for ChatGPT, not charging much or at all for API access. They believe OpenAI, Google and others need Reddit's data for the development of the next versions of their LLMs, and since those companies are earning billions of dollars on Reddit's data, Reddit deserves a substantial chunk of that money. Hence the expensive per-call API price.
As for 3rd-party apps, they're casualties. Reddit doesn't care for their money, that's cheap change for them. They care for Microsoft's and Google's money. For that, the default API must be expensive, otherwise Reddit wouldn't have how to leverage offering lower rates for the "Enterprise level" access needed for LLM training bots.
Of course, this potential revenue path is pure delusion on Reddit's part. No AI company is going to pay that much for API access, even if discounted. Reddit's data is neither fundamental nor necessary, and even if it was, those companies already have more than enough of it from their previous download. In fact, it's even possible Reddit's CEO knows full well it won't work, but believes he can still convince potential investors it will work regardless, as a way to increase its IPO's valuation.
In short, the price Reddit is asking has nothing to do with the cost of providing API access. What it actually has to do with is the expectation of revenue from big data consumers.
Re: (Score:3)
If it's costing Reddit 24c to serve 1000 requests (even including staffing, R&D etc) they're doing something *terribly* wrong.
The real reason Reddit is doing this is because they were used by OpenAI as training ground for ChatGPT, not charging much or at all for API access. They believe OpenAI, Google and others need Reddit's data for the development of the next versions of their LLMs, and since those companies are earning billions of dollars on Reddit's data, Reddit deserves a substantial chunk of that money. Hence the expensive per-call API price.
As for 3rd-party apps, they're casualties. Reddit doesn't care for their money, that's cheap change for them. They care for Microsoft's and Google's money. For that, the default API must be expensive, otherwise Reddit wouldn't have how to leverage offering lower rates for the "Enterprise level" access needed for LLM training bots.
Possibly, but charging different prices to different customers is hardly a new idea. Just throw some language in the licensing, for instance:
"API access is $X, but you're not allowed to use the retrieved content for training of AI models. If you want to use the data to train AI models the cost is $X*1000"
Obviously you'd need some lawyers to make the language workable, but given that it's $5-10m to train an LLM it's a pretty small number of entities you need to worry about.
Re: (Score:3)
I've heard that excuse, but it doesn't really make sense to me. If the API is prohibitively expensive what's to stop the AI devs from just scraping the website for content? Sure, it's not as efficient as the API, but it's all there.
As long as there's a way for humans to read the site, bots can too.
Re: Decline is a choice (Score:3)
At a former company we got up blocked for scraping some APIs. The workaround was to spin thousands of spot instances in aws to distribute the requests from new random ip addresses. It worked well enough. There are enough cloud providers around that blocking all of them without blocking genuine requests is not practical.
My point is that IP blocking to restrict access has limited effectiveness these days.
Comment removed (Score:4, Interesting)
Re: (Score:3)
filling the gaps for - you know, little things, like accessability (maybe it's time the DoJ took a look at Reddit on that?)
Nope. They're a private entity with no government function. They are not required to meet accessibility guidelines per 508 standards. Source: am doing QA for a product that is required to meet AA standards level.
Re:Decline is a choice (Score:5, Insightful)
Firing people who make the company work is the road to long-term decline, not profitability. Yes, it may boost profits in the very short term, but it will choke off long-term profitability by ensuring the company can't keep up with change. This is the move of investors desperate to cash out while they can find greater fools to buy their stock, not a company that's set to be successful in the long term.
Re:Decline is a choice (Score:4, Insightful)
Ask Jack Welsh and GE how it went.
Re:Decline is a choice (Score:4)
Re: (Score:2)
It went great for Welsh, he got a half billion dollar payout from it.
Re: (Score:2)
They just fired a bunch of people as well.
Investors like companies that don't perpetually lose money. Gotta do something about it.
Some institutional investors look toward long-term stability, i.e., modest gains sustainable over decades. However, that's not how venture capital, hedge funds, and other short-term investors think. These investors like short-term stock pops so that they can grab their profits and run away, and they don't care about what happens to a company in ten years because that time frame is much too long for them. Unfortunately, the voices of the short-term investors tend to be very loud, aggressive, and confronta
Re: (Score:3)
Lose more money?
The crux of the issue is that the folks that are able to make reddit what it is did so while never exhibiting any business acumen.
Now, when forced to retroactively apply some business acumen, they have very poor business ideas. Like "charge for APIs and free revenue" or "force freeloading third-parties off our platform" and magically that means less expense and/or more revenue. They probably imagine marketing revenue as an exercise in API charges, but didn't come up with a solid plan for t
Re:Decline is a choice (Score:5, Insightful)
Re: (Score:3)
One of the people fired was instrumental in the "ask me anything" sub. Last time they did this nonsense they had fake guests coming on, plus the whole Woody Harrelson fiasco.
Re:Decline is a choice (Score:5, Insightful)
Re:Decline is a choice (Score:4, Insightful)
Thewe things are discussion forums in a slightly upgraded graphical wrapper. Period. That’s all they are. That’s all they do. My god, the discussion forum is literally one of the earliest uses of the internet. These things are absolutely nothing unique. If one disappears, it will be replaced by something that provides the exact same functionality.
This stuff is as basic as it gets people.
Re: (Score:2)
Re: (Score:3)
Presumably they've done a lot of work to build a resilient and secure infrastructure with all the money that VCs have been giving them. As easy as it is to roll out a discussion forum, it's another matter to build a global network of synchronized databases.
On the other hand, once it has fully departed for irrelevance, another large scale forum host will be along, just as the internet moved from Digg to Reddit over a decade ago.
Re: (Score:3)
Re: (Score:2)
Re:Decline is a choice (Score:5, Informative)
Thewe things are discussion forums in a slightly upgraded graphical wrapper. Period. That’s all they are.
It's a stretch to even call them an "upgrade" in terms of graphics or usability. The main reason for their continued success is their ability to maintain a critical mass of users, which has more to do with mob mentality than building a better mousetrap. Most of the more popular forums for discussion on the internet are actually pretty terrible to use, but they're where all the action is.
Re: (Score:3)
It's better because it's where the people are.
When you're trying to build a new platform, it's not enough to be better. You have to be so much better that staying with the existing platform doesn't make sense. Otherwise the momentum of the existing community already being where it is, of people knowing to look there for it, all of that mass is just not worth moving.
Unless the platform you're competing with screws up so handily people feel like they don't have a choice. Take a look at freenode a couple years
Re:Decline is a choice (Score:5, Insightful)
The convenience factor of having a single website with many vibrant communities is not outweighed by it being a single point of failure for admin/corporate fuckery. This has all happened before . . .
Re:Decline is a choice (Score:5, Interesting)
The problem these days in many countries is the growing regulatory burden that is being attached to anything called "social media".
The old web boards which were pretty much direct descendants of BBS discussion boards were almost always run by volunteers (or usually one volunteer), and I remember a few of the people I knew even back in the 1990s and 00s who ran these systems were being driven crazy by spammers, trolls, hosting and maintenance issues, the software maintainers wantonly doing major upgrades that broke absolutely everything, and one by one all quickly migrated to whatever the big guys like Facebook and Google were offering (the latter usually to one of Google's platforms that they would shut down a few years later).
Today to open up a board like that would likely come with the old problems, along with the risks that if someone reposts a news article, you're going to run afoul of laws in places like Canada and Australia, and someone is going to be demanding cash.
Re: (Score:3)
I don't think they care if the wheels fall off at this point. They'll still be able to cash out.
Re: Decline is a choice (Score:2)
Re:Decline is a choice (Score:4, Insightful)
Our Slashdot moderators work for free. It seems that the trick is to sucker... er... convince people that it's a special privilege.
Re: (Score:2)
Tumblr (Score:5, Interesting)
Remember when Tumblr banned adult content (and the associated cp communities which hid there) and basically ended themselves?
Hopefully this is the same kind of death knell.
I'll be curious to see where the refugees flee. Whichever service it is, will likely face a similar fate in a matter of time, after a brief surge in popularity.
Greed (Score:2)
The investors weren't happy with a successful company. The want their 10-bagger. It's simple greed, also known as "killing the goose that lays the golden eggs."
Even if they suddenly backtrack, unlikely as that us, it is likely too late. Many people are looking for alternatives, many have already left, more will follow.
Re: (Score:3)
Reddit is reportedly still operating at a loss, never having turned a profit.
That said, someone has suggested that when Steve Huffman and Alexis Ohanian sold Reddit to Conde Nast, they did so for a "mere" $10 million to $20 million. Huffman was a millionaire, but only single-digit millions. Meanwhile, all the other tech people he knew were selling their companies and getting tens or hundreds of millions oreven billions each. Huffman may still be bitter about this, and since Reddit's recent funding rounds ha
Re: (Score:3)
Imagine being bitter about your windfall only being 5-10 million dollars and not a billion.
The quest for a viable business model... (Score:2)
It's almost enough to make us feel sorry for the middlemen.
Re: (Score:3)
Maybe the real hidden driver here is just... higher interest rates shorting leash on sustainable finances.
Re: (Score:3)
Re: (Score:2)
While I wouldn't mind seeing Reddit itself going, what frustrates me is just how much information will be lost with it. Mainly crowd-sourced stuff in games, guides, all kinds of optimization routes etc., that was only ever hosted on Reddit because that's where everyone was.
Re: Tumblr (Score:2)
Good. Reddit has great search rankings but is generally the lowest quality of content in the results. Get it out of the way.
Re:Tumblr (Score:5, Interesting)
Re: (Score:2)
Yeah, Reddit's what finally replaced /. as my homepage.
I'm back for at least the next few days now though. Maybe longer...
Re: (Score:2)
If enough of you come back, I wonder if we could start having interesting technical discussions without the constant political back-biting, like we used to? Just don't bring Katz back with you. Please.
Re:Tumblr (Score:5, Funny)
Interesting technical discussions would abound! At least until someone brings up systemd.
Re: (Score:2)
I really miss my Geocities page :-)
Re: (Score:2)
Ugh, how I miss it, and I barely ever used it.
Re: (Score:3, Interesting)
same... I went from slashdot to dig to reddit, and now today I'm like "lol lemme go sign in to my slashdot account"
Re: (Score:3)
Wow that's a seriously low userid... welcome back :)
Same here sort of... Left Reddit due to the Apollo thing and decided to fire up the old RSS reader again. Still have /. as a feed so found my way back here.
Re: (Score:2)
I haven't posted on slashdot in probably 15 years, but here I am.
Ironic, because at one time, I considered a 600K Slashdot ID as either a newb or a troll account.
Re: (Score:2)
Re: (Score:2)
Before reddit, before digg, before myspace, I lived here. I haven't posted on slashdot in probably 15 years, but here I am.
So, discussion forum goes down due to user protest over 3rd party app API access, and some of the users return to older discussion forums which still operate using a pre-app paradigm? The only thing we're missing now is the old appy app apps troll.
Re:Tumblr (Score:4, Funny)
Signed,
A new Slashdot mod
Re: (Score:2)
Is this sarcasm aimed at newer social media? I've never been enough of a troll to ever warrant getting blocked/suspended on Slashdot. In fact, I'm still irritated by the existence of "anonymous coward" posting.
Reddit is more than porn: reviews + fandom (Score:5, Informative)
Remember when Tumblr banned adult content (and the associated cp communities which hid there) and basically ended themselves?
Hopefully this is the same kind of death knell.
I'll be curious to see where the refugees flee. Whichever service it is, will likely face a similar fate in a matter of time, after a brief surge in popularity.
Reddit is regarded as one of the best places to get "real" reviews. I go there for often for reviews of things that are notorious for bullshit, like nutrition, video games, or tools, etc. It's obviously not free of astroturfers and other insincere posters, but it does have less than Amazon reviews or YouTube. I often search "*** review reddit" if I am unsure about a product. There are a lot of fan communities...my wife goes there often to shit post about her favorite shows. I find a LOT of good local news there too...especially the unofficial stuff...like city issues. I subscribe to 90%+ porn subreddits, but my actual feed is about 1/4 porn because I engage a lot more with local subreddits, hobby subreddits, technology, etc.
The porn is pretty awesome, but I suspect the non-porn communities are far stronger. Tumblr was just image sharing...pretty fucking boring without nipples involved. Reddit allows for in-depth discussion and has diversified. How they can monetize this?...well, that's not easy if they want anything more than their current advertising model.
However, Reddit will be just fine without porn. Far less fun for a prolific fapper like me, but still VERY useful.
Re:Tumblr (Score:5, Insightful)
Hopefully this is the same kind of death knell.
No. Why? No one wants Reddit to shutdown or die. No one wanted that for Tumblr either. Hopefully this is a wakeup call, not a death knell.
Re: (Score:2)
No. Why? No one wants Reddit to shutdown or die
Speak for yourself.
(Though, TBF, I would prefer a radical re-working.)
Re: (Score:2)
Re: (Score:2)
No one wants a propaganda disseminator or sterile corporate environment either. Frankly, if all of Reddit turned into a version of /r/politics, I'd probably actively participate in its "death knell".
Re: (Score:2)
Re: Tumblr (Score:3)
Because Google no longer lets you blacklist sites.
Re: (Score:2)
Problem with the Fediverse is that it's not really any more decentralized as the rest of the Web. If Lemmy or Mastodon go evil, people still have to find another website and move their accounts and communities over, no different from what is happening with Reddit right now. Only advantage it has is slightly easier discovery of other servers, but even that only works as long as everybody plays nice, nothing stops one server from blocking other servers.
Re: (Score:3)
That's always a risk when using someone else's server for anything. I think the key is that if you know how you can run your own. That's what makes it resilient, more than the decentralization itself.
Expected, yeah right (Score:4, Insightful)
caused some expected stability issues, and we've been working on resolving the anticipated issue
If it was expected and anticipated then it was avoidable. The fact he tries to come across all smug sounding just shows how inept they were if they could not engineer around something that was expected. My guess is that it was NOT expected or anticipated, and thus that is why their system apparently handles denied requests to private or read-only subreddits so poorly that it brings down their entire system.
Re: Expected, yeah right (Score:5, Insightful)
Re: (Score:2)
Or it's just not worthwhile avoiding the issue. Reddit goes down every so often. It is something the site knows how to deal with and has lived through quite frequently.
Re: Expected, yeah right (Score:2)
Re: (Score:3)
If the people in charge weren't idiots (and we know they are given they got themselves into this mess in the first place) they'd stay silent and in a few days everything would be back to normal and in a month it would have been forgotten. Does anyone really remember the last time this kind of protest occurr
Re: (Score:2)
If it was expected and anticipated then it was avoidable. The fact he tries to come across all smug sounding just shows how inept they were if they could not engineer around something that was expected.
You're conflating two things there. Something can be both expected and avoidable while the operators have no intention of avoiding it or see no business value in doing so.
Why put effort into avoiding something that is known in advance to be a temporary issue?
I've been involved in countless such decisions. If we do X it will break Y. The immedate following statement is not, how do we avoid breaking Y, it is "how much will breaking Y cost, and is our effort to avoid it worthwhile."
and thus that is why their system apparently handles denied requests to private or read-only subreddits so poorly that it brings down their entire system.
And since it happens rarely
Re: (Score:2)
didnâ(TM)t code up a fix/mitigation for this expected event
Not coding for expected issues is a time honored tradition.
So a private Reddit puts more stress on the system (Score:5, Insightful)
Might be time for a code review.
Re:So a private Reddit puts more stress on the sys (Score:5, Interesting)
At a guess setting a subreddit to private doesn't just set the sub private; it changes permissions on every single post in the sub. That kind of activity is gonna be painful.
Re:So a private Reddit puts more stress on the sys (Score:5, Interesting)
So... if we do this every other day, Reddit is down for good?
Just asking for a friend...
Re: (Score:2)
Re: (Score:3)
It seems to me that a child record ought to inherit that from the parent...
It's probably caused by an early design decision in the database that was made before the designer(s) realized the need for proper parentage records. By the time it was realized that such inheritance was needed, there was probably too much technical debt to make it cost effective to modify.
I would assume that there is no centralized discussion subsystem and class hierarchy to isolate such changes, but that the queries are scattered throughout the entire code base.
Re:So a private Reddit puts more stress on the sys (Score:4, Funny)
Might be time for a code review.
Maybe. First come up with an estimate for how often such a large scale issue occurs. Then ask accounting what the cost of the outage is. We'll put it on our risk table and see if your suggestion for a code review is approved.
Re: (Score:2)
Welcome back, Redditors (Score:5, Funny)
Just kidding...
Re: (Score:3, Funny)
good to be back, what did I miss? Where is CmdrTaco?
Re:Welcome back, Redditors (Score:5, Funny)
You may want to sit down for this..
Re: (Score:2)
Is he safe? Is he alright?
Re:Welcome back, Redditors (Score:5, Informative)
He died in a freak Natalie Portman covered in hot grits accident.
Re: (Score:3)
Me too. My last post: Sept 7 2014.
Has Anyone Suggested An Alternative? (Score:5, Interesting)
Look, expecting $20M from an app dev who obviously has not grossed $20M is certainly unreasonable. No question about that, and props to the Reddit community for backing the dev. ...that being said, Reddit gets more monthly visitors than *Amazon*. Yes, they make money with Gold and things of that nature, so there's some money to be had there, but Reddit still follows the advertising-based model to a good extent. Tolerating ad blockers in a web browser is one thing, but the API doesn't show ads *and* Apollo has a subscription model of its own.
I hate ads as much as the next Slashdotter, but if ads are how Reddit balances the books, and (apparently) 7 billion API calls per month from a paid app are not showing ads...I can understand Reddit wanting to paywall its API, especially now that it's got its own first-party mobile app. Actually, upon further research, Reddit is apparently prepping for an IPO in the back half of this year...it's definitely understandable that Reddit would want to be able to show some revenue from its API prior to the IPO.
So...even if Reddit 'gets the message', and even if Reddit doesn't introduce shareholders into the mix...it seems that the subreddits are all making their stance "don't charge for the API usage" and Reddit is all "...odds are good you were either costing us money, or you'll be back after the protests"...is there no viable means of both sides getting what they want?
Re: (Score:3)
One way to do this would be to insert ads into the API, so clients like Apollo see the ads too, and mandate that third-party clients show the add.
They could remove the ads for Premium users.
I know that before all this, if ads became required in third-party clients I would understand that Reddit has to make money, and I would probably go premium to get rid of them. It's not a lot of money.
Re: (Score:2)
They should have build an app store.
You take a subscription on the app for a couple bucks a year. X% goes to reddit, 100-X% goes to the app developer, you get to use the app to make as many API requests as you want for personal use, within reason.
Re:Has Anyone Suggested An Alternative? (Score:5, Informative)
Alternatives were suggested, by Apollo's developer. Then Reddit stonewalled him. THEN Reddit went and said he was trying to blackmail them. This was proven to be a lie when Apollo's developer proceeded to post the audio of their phone calls.
Reddit is acting in bad faith. There's no point in suggesting alternatives now; they have decided on their course and they WILL NOT deviate from ramming that iceberg.
Re: (Score:2)
The alternative is for reddit's first party apps/tools to not suck. Alternatives like this exist because there is market demand. That demand is created by a deficiency.
Re: (Score:3)
Re: (Score:2)
I had figured it was an "anchoring" negotiation PR ploy, make the pricing excessive, in response to the expected backlash, reduce the pricing claiming "listening to the community" and "we support third party devs". We'll see if that's what comes or not.
The "protest" seems kinda' pointless to me, as most users won't even see the effect, as they'll still have a home feed filled with stuff, just slightly different stuff. Some users are confused and think they've been banned from their favorite subs. It's li
Re: (Score:2)
Re: (Score:2)
The alternative is use the damn site through a browser, as Apple proposed 16 years ago [youtube.com].
Re:Has Anyone Suggested An Alternative? (Score:5, Informative)
Re: (Score:2)
That's a great idea. How about Reddit actually works with developers to find a fair an equitable solution instead of acting like a bunch of out of touch, greedy hipsters.
Like, why does Reddit not have an ad server that can inject content into 3rd party apps? This is a problem that has been solved for decades.
I understand Reddit's position, but it seems like they're choosing the laziest, most outrageously dickish way of doing it. For a website that depends on their user base freely giving them content, it se
How? (Score:4, Informative)
Very simple.
I believe they have a mask for "offline/blocked/limited" posts. And like any competent backend designer, they don't cache that signal.
In other words, while you browse,
1) they bring all possible posts.
2) they remove the ones you should not see.
The second one is hammering their backends.
Whereas:
ALTER TABLE posts ADD possibly_blocked_for_most BOOL;
Could have helped (or whatever the equivalent in their system is). It would easily limit posts during retrieval to a reasonable number.
(But the question is, could they release a change in a day or two, or do they have a bureaucracy that would take many weeks).
Is history repeating itself? (Score:4, Insightful)
I feel like we've seen a nearly identical scenario before with Digg. Once a popular community, Digg is now all but forgotten after they made some sweeping changes to the functionality of their site. Users protested and said they'd leave, and they did. Reddit has the right to make money, they're a business after all, but alienating your customers is not the way to go about it.
Time for the next thing. Remember Digg? (Score:5, Insightful)
People then moved on to Reddit.
Perhaps this API change is Reddit's "Digg" moment?
Re:Time for the next thing. Remember Digg? (Score:4, Funny)