USPS Server Meltdown 238
m2pc writes "The US Postal Service is experiencing major server issues for its shipping API web services. After spending about an hour debugging my own eCommerce software for a client, I found the problem was with the USPS shipping servers being unavailable. Further research showed that message boards for OS Commerce and other e-Commerce packages are filling with posts from angry users who are experiencing crashing Web store applications and frustrated customers. Developers are scrambling to find interim solutions, from hard-coding fixed price shipping, to 'rolling their own' shipping calculation APIs based on the USPS Fixed Rate Zone Tables, to disabling the USPS option altogether. One user reported yesterday that a call to USPS yielded the response 'we expect it to be down all day.' As of 9:20 AM PST the service is still unavailable."
Bailout? (Score:5, Funny)
Re: (Score:2)
So THAT'S why the present I sent to my girlfriend resulted in an e-mail from Amazon that the package was at the Post Office!
This explains a HELL of a lot... mostly, why Amazon said it was detained at the post office, but arrived at the door just about 20 minutes ago...
Re: (Score:2)
SAme thing happened to me. WEnt to PO; no package, no slip. They knew nothing about it. 4 days later, shows up at house. Weird!
Re: (Score:3, Funny)
That's what you get.... (Score:5, Insightful)
And that's what you get for writing e-commerce packages that rely on 3rd party sites for basic functionality...
Don't say I didn't tell you so...
Re:That's what you get.... (Score:5, Insightful)
Re:That's what you get.... (Score:5, Insightful)
Re: (Score:3, Informative)
Re:That's what you get.... (Score:5, Insightful)
"While in principle I agree with that, what are they supposed to do? They are quoting you a price for a service they don't provide themselves."
Fail gracefully. For example, state that they are unable to price that shipping option at that time. Offer you to accept the option without knowing the price, select another type of shipping, or get an email when they know.
Designing software to gracefully handle external failures is a vital real-world programming skill.
Re: (Score:3, Insightful)
Wouldn't it be easier to have a background task update the USPS prices, and just use the last working snapshot?
Re:That's what you get.... (Score:4, Informative)
Really, you'd think someone in the apparently non-existent risk management departments would ask "what if the USPS Servers went down?" Hopefully the answer wasn't simply "we're screwed."
In any third party connection for an application I've built, I typically consider the implications of such web service/outside API going down, even if I label it "non vital, don't worry about it" to "have a backup plan". Oh well, no sympathies here
Re:That's what you get... for not using FedEx (Score:5, Insightful)
To be fair, what sort of "backup" calculation would you have done here, short of reverse-engineering the USPS algorithm for calculating shipping rates?
I'm not usually a rabid free-market libertarian, but this here can be seen as a result of the fact that the USPS isn't really beholden to its customers. Can you imagine FedEx or UPS being afflicted by such an issue? And, if they were, would they blow off inquiries with a glib, "We expect the servers to be down for the rest of the day?" Of course not, because, for FedEx, UPS, DHL, et. al. such an outage directly affects the health of the organization. If people can't calculate shipping rates, they can't ship, and if they don't ship, the company doesn't make money. The close linkage between revenue and working services tends to put more impetus behind keeping things working and making sure that they get fixed quickly if they do happen to go down.
Re: (Score:2)
Re:That's what you get... for not using FedEx (Score:4, Insightful)
Re:That's what you get... for not using FedEx (Score:4, Informative)
Re:That's what you get... for not using FedEx (Score:4, Informative)
The zones are clearly marked but determining which zone you are shipping to is not. They are based on distance from the originating ZIP Code so the application must use Postal Explorer to generate the zone chart ahead of time. (For now, it happens to be working.) And in the case of non-flat-rate Priority Mail, there's a huge price difference between zone 4 (zone and weight only) and zone 5 (zone and weight *OR* zone and size).
Re: (Score:2)
Re: (Score:2)
To be fair, what sort of "backup" calculation would you have done here, short of reverse-engineering the USPS algorithm for calculating shipping rates?
A store can charge whatever they want for shipping. So if they know how much it costs on average to ship their products, they can simply charge everyone that flat rate. Or if they sell something simple, they can charge one simple rate per quantity. Or they could cache the last rates they charged and reuse those. Or they could offer everyone free shipping and eat it as the cost of doing business with the USPS.
Re: (Score:2)
If TFSummary is to be believed, the least they could do is make sure the app didn't crash on bad data returned from an API. There is NO excuse, ever, for a crashing bug in an online store app like that. There's no problem with the USPS servers being funky, there's a problem when people programming dependent apps somehow didn't at least plan on it ever happening.
Re: (Score:2, Funny)
Can you imagine FedEx or UPS being afflicted by such an issue?
Having worked in IT for one of them this year... yes, without question, yes.
Re:That's what you get... for not using FedEx (Score:4, Informative)
The same has been true with the USPS since 1970. Their entire budget is financed by people buying stamps and other services. They don't get a dime of taxpayer money.
Re: (Score:2)
Except, they get to charge an exorbitant amount for their stamps and services by prohibiting competition - a luxury that Fedex and UPS do not have. (If you think their costs are reasonable, then why is there a law the prohibiting Fedex/UPS from charging below a certain amount?)
That's what you get... for not using Utilities. (Score:3, Insightful)
"Americans are pathetic sometimes -- they expect their government services to do as well as private industry, yet they don't give them the ability to charge what private industry charges. Amtrak is a similar situation, Amtrak is expected to be cash flow positive, yet they are not allowed to own their own tracks, those are owned by the freight companies, whereas their main competitors run on highways that are paid for completely by the taxpayer and gas taxes, or operate out of airports also funded by taxpaye
Re: (Score:3, Informative)
The same has been true with the USPS since 1970. Their entire budget is financed by people buying stamps and other services. They don't get a dime of taxpayer money.
How did you ever get modded +5 informative with that load of bull?
The total funding for the Postal Service in the administration's 2005 budget is just over $61.7 million. [upi.com]
Appropriation, fiscal year 2004 $65,135,000 [congress.gov]
Appropriations, 1999 $100,195,000 [congress.gov]
etc, etc
Re: (Score:2)
Re: (Score:3, Insightful)
try {
}
catch (exception ex){
}
Re: (Score:2)
Exactly! Any reliance on an external source (even if it's your own external source) should be able to handle that external source going down, that's just good coding.
All those complaints about products crashing or going down because of this should really yell at the developers of that product, they didn't do basic failure handling.
Re:That's what you get.... (Score:4, Insightful)
And that's what you get for writing e-commerce packages that rely on 3rd party sites for basic functionality...
Like a credit card processor? How many web stores have more than one?
Re:That's what you get.... (Score:5, Funny)
that's why i design all my e-commerce sites to accept cash only. (always makes sure the bills are facing up before you feed them into the floppy drive!)
Re: (Score:2)
But what happens when the cash becomes worthless? Do you have a backup system that defaults to, say, wampum shells?
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
And if the processor is offline, they deal with it. Store the info until it can be charged, whatever. Crashing or otherwise malfunctioning in the face of 3rd party inaccessibility is definitely an app problem, and never excusable.
Lots of free shipping today. (Score:3, Interesting)
Try it. Go to a USofA-based commercial site and check the shipping charges for your purchase.
Most of the sites are offering free shipping. I'm guessing that these two items are related.
Re: (Score:2)
Or, you know, it could be that it's the holidays and eating shipping charges is a major factor in converting sales for online retailers.
Re: (Score:2)
Re: (Score:2)
And how, pray tell, are you to calculate USPS' shipping rate without relying on the USPS' server?
I suppose you could construct a database of USPS rates for every possible box size and weight you might ship, but that would be a maintenance nightmare as you'd have to update it whenever they change their rates and every time you get a new item in stock.
Proof that computers corrupt all!!!! (Score:4, Funny)
"Neither snow nor rain nor heat nor gloom of night stays these couriers from the swift completion of their appointed rounds"
But the computers will!!!
Re: (Score:2)
"We'll deliver any package you bring to us...with the correct postage! BWAHAHAHAAA!"
USPS (Score:2, Interesting)
"Neither snow nor rain nor heat nor gloom of night stays these couriers from the swift completion of their appointed rounds"
But the computers will!!!
You do realize that this has never been the motto or creed of the USPS.
It is taken from the courier service of the Persian Empire. See Wikipedia.
--
This is not the sig you are looking for... Move along...
Re:USPS (Score:5, Informative)
Sure, but it would have been easier if you had provided a link at least.
The motto will also continue to be mistaken for the motto of the USPS so long as the USPS does nothing to correct that.
The motto is prominently carved over the U.S. General Post Office in NYC [wikipedia.org]
and the USPS even used it in their own television spot [wikipedia.org] (albeit in an altered form):
Note that the [2] link in the above quoted Wikipedia article takes you to this page [usps.com] on the USPS site quoting the exact text of the tvspot.
Yes, they have no Motto, but considering how much they use the meme, they might as well have one.
Re: (Score:2)
What can Brown do for you? (Score:5, Funny)
Re:What can Brown do for you? (Score:5, Informative)
Re: (Score:2)
Also, at the end of the month, they append a "fuel charge" that is completely unpredictable and not represented in the cost at the label printing time in any way, shape or form. You can't derive an estimate for it from the price of oil, from the price of fuel, from last month... no way to do it that I know of, and UPS agrees. Best thing you can do is take last month's charge and distribute it across your best estimate
Crashing Web store applications? (Score:5, Insightful)
Only a really terrible developer would hit a web services API and not code for it to fail. No one should expect a third party service to be up 100%. The apps should fail gracefully. Anyone finding their e-commerce software handling this situation poorly should find another package.
If a store offers only the USPS delivery method and the web service is down, the user could be directed to call the sales number to place their order. If the store offers other deliver methods the store front could instruct the user that USPS isn't currently available and they must choose another method.
Re:Crashing Web store applications? (Score:5, Informative)
Re: (Score:3, Informative)
Re: (Score:2)
but am I the only one who thinks that alot of popular open source projects are stinking piles of crap?
I would probably say something in agreement, and even add a bit of my own comments, but I don't feel like burning up my karma today.
Re:Crashing Web store applications? (Score:5, Insightful)
osC is insidious. You install it because it's open source and a zillion other sites are using it, so it can't be THAT bad, right? Then you have to install a few contributions to add the basic functions you need - and with each contribution, adding the next gets harder and harder.
But you get up and running. Then you discover all of the little annoying usability quirks - US state names don't use proper abbreviations, customers in Ireland can't place an order without a post code even if you set the minimum length to zero, the payment option selection looks like it has a default selected when it doesn't, users don't know that 'authorize.net' is actually the credit card payment option, and so on.
So you spend hours fixing those. Meanwhile, you've got the database populated with all of your products, and you're using it to track sales figures. Maybe you add some back-end order processing stuff for your own convenience.
Eventually this mass of haphazardly patched spaghetti code becomes an absolute nightmare to do ANYTHING with, but by this point you've spent so many hours working on it that it's easier to just keep slogging away at it than to just abandon the whole thing, install something new, and try to move your catalog and your sales records over to the new system.
And THAT is why there are so many osCommerce sites.
Re:Crashing Web store applications? (Score:5, Interesting)
Only a really terrible developer would hit a web services API and not code for it to fail. No one should expect a third party service to be up 100%. The apps should fail gracefully. Anyone finding their e-commerce software handling this situation poorly should find another package.
I have developed a fulfillment system using UPS so I have some experience with this. This was a few years ago, but at the time, UPS's system was designed on the assumption that a shopping cart system would contact UPS during the order placement process (or checkout). USPS appears to be very similar. This is incredibly moronic, because it means that if their server doesn't respond or any reason, you simply lose the sale. Problems AFTER the order has been accepted are far less serious - that just means fulfillment is interrupted - you don't lose orders. It would be even better if you could print the labels without having to reach their server, but they don't provide this (UPS's own software, however, can.)
The only reasonable way to do it (obviously?) is to have your own shopping cart software calculate the shipping rate. At the time I did this in 2001, UPS did not offer reasonable data to let you do this. Their rating system is extremely complicated, and the spreadsheets they gave me had clearly been maintained by hand and needed a lot of massage to get them into a reasonably parseable format. Not only that but it was very difficult to get these tables in the first place, requiring multiple escalations through tech reps and ultimately a "god damn it just give him the tables" from our sales rep.
I imagine things might have improved a little since then and I know there are shopping carts that can now do the rate calculations internally. But if any shipping company forces its customers to contact a server to calculate shipping costs, then they are just as retarded as the merchants who use them (for a web checkout).
Re: (Score:3, Interesting)
Most e-commerce sites, including the ones I've built, have a multi-step process to place an order. If the step for shipping (or tax, etc.) failed, then the system would re-load the page with any appropriate messages and option changes. So, for example, if the third-party payment processing service didn't respond before a timeout the user would be asked to call customer service to complete their order. The sales rep could see the order in its incomplete state and finish it over the phone.
Re: (Score:3, Informative)
Most e-commerce sites, including the ones I've built, have a multi-step process to place an order. If the step for shipping (or tax, etc.) failed, then the system would re-load the page with any appropriate messages and option changes. So, for example, if the third-party payment processing service didn't respond before a timeout the user would be asked to call customer service to complete their order. The sales rep could see the order in its incomplete state and finish it over the phone.
I would not consider
Re: (Score:2)
Well, yes. All of the e-commerce sites I've worked on had in-house or outsourced phone staff. But my point is that a site must have some sort of failure mode. Whether it's to call customer service, give free or flat shipping, or whatever, it's better that the customer can complete the sale than see an arcane error message that the system has failed.
You have to do it before. (Score:2)
Retail customers expect to check out knowing the full price they will be paying, including shipping. This means that you HAVE TO calculate shipping during order placement (checkout). You're asking for customer service headaches if you have to manually charge shipping after the customer has already (they thought) completed the order.
So your only option, if you don't want to rely on a third-party server for shipping costs, is to have your shopping cart do its own calculations.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
And what happens when they change those tables? Sounds like you're asking for a world of business hurt down the road with the costs to update the code and cover the changed price.
The tables don't change very often. When they need to do a quick rate bump it is usually in the form a simple "fuel surcharge". Even when the tables are updated, it can usually be very closely approximated by applying a simple percentage increase.
Also, most sites put a significant markup on shipping (that's the "handling" part), so
Re: (Score:2)
Re: (Score:2)
If a store offers only the USPS delivery method and the web service is down, the user could be directed to call the sales number to place their order.
You do realize that using a phone number as your fall back is a just a different disaster waiting to happen.
Very few online stores have enough idle/re-taskable manpower (much less sufficient phone lines) to handle the sales volume that normally goes through their website.
Re: (Score:2)
Then if the third-party service goes down, they should charge a flat rate for shipping or offer it free. Otherwise they deserve to lose customers for not planning for this type of issue.
Re: (Score:2, Interesting)
Very poor design (Score:2)
I don't mean the USPS, I mean the people using their service.
They never thought this might fail?
Re:Very poor design (Score:5, Funny)
This is why my company ships 2 of everything that is ordered. One via USPS and the other by UPS, just in case one gets lost. Always need redundancy.
Re:Very poor design (Score:5, Funny)
Does your company offer mail order brides?
Re: (Score:2)
Couldn't have come at a worse time (Score:2, Funny)
This is sure to piss off a lot of people being between that Thanksgiving and Christmas period. Lets hope nobody that ordered me something is having problems.
Re:Couldn't have come at a worse time (Score:4, Funny)
The crap I get, I hope that people buying me things are having problems.
Through snow and sleet and driving rain... (Score:2, Funny)
Why SAS fill eventually fail (Score:2, Interesting)
Re: (Score:3, Insightful)
There is a better way. Code your own backup solution, revert to it when the SaS you are counting on fails. In this case, that would be a static shipping 'estimate' process that substitutes data when the USPS service fails. Writing code what depends on a third party is dangerous. If your business depends on that software you are fooling yourself if you think it won't fail. It will. This is where coding disaster recovery functions into your system is important.
Many of the posts here are about how stupid this
Re: (Score:2)
If you've coded your own solution and you're ready to run it, what the hell do you need SaS for?
Re: (Score:3, Insightful)
In this case, you might want to only guarantee prices quoted from USPS and use your data as 'estimates' in the event that your own data is wrong and costs change at shipping time.
In yet other cases, the backup plan might simply be to use data from last week or yesterday if that is sufficient, but is data that is held in house on your servers.
Depending on the situation and requirements, any number of solutions are possible. By using a 'back up' solution that is not as good or perhaps as accurate, when the sy
What would USPS do? (Score:5, Funny)
Re: (Score:3, Interesting)
You laugh, but older contract offices still get their software updates, including price changes, via mail.
On 3.5" floppy.
The terminals still use green-text monitors.
UPS is flaky too (Score:4, Interesting)
Same here (Score:5, Informative)
I came in yesterday morning to find the USPS module non-functional. Worse, the only working option was DHL overnight - and in case you've missed the news, DHL is now about an order of magnitude worse than the post office for domestic delivery. Even for places they say they can do next-day delivery to, actual delivery can take more than a week.
Why? Because they hand it off to the post office rather than deliver it themselves. Why it takes the post office a week to deliver it when I can get it there in two days by sending it by priority mail myself is a mystery. In any case, DHL's out of the (US) domestic game entirely next month.
My site was up last time I checked, but if the USPS option goes down again, I think it's time for a 'free economy shipping' promotion. No messy rate tables to deal with!
Re: (Score:2)
DHL is now about an order of magnitude worse than the post office for domestic delivery. Even for places they say they can do next-day delivery to, actual delivery can take more than a week.
I'll second that. Last year, their next-day delivery took 3 weeks.
(No, that's not sarcasm, it actually took three weeks.)
Re: (Score:2)
It's the same for USPS package services (anything above First-Class Mail). You have to generate a zone chart from your originating ZIP Code before determining shipping rates. Priority is based on zone and weight or, in some cases, zone and size. Parcel Post is based on zone and weight, but some ZIP codes that are major hubs offer cheaper "Intra-BMC" rates.
Express Delivery (Score:5, Funny)
USPS
at address: 0x1234 Main St., Hometown USA, zc=0x10001
I know why (Score:2, Interesting)
posting anonymously to cover my ass... We upgraded to MySQL 5.1 last week. We had some major table corruption and no backups. Sadly, no one will be fired or even reprimanded for any of this major cluster fuck.
hosted vs downloaded/licensed e-commerce (Score:5, Interesting)
The company I work for provides a hosted e-commerce shopping cart solution, SEO-Cart, which supports the USPS Web Tools. Of course the first call coming in for the day was from a client using USPS and having incorrect shipping prices being calculated for their store.
I went ahead and called USPS and the lady who answered was quite rude and explained to me that they had a Worldwide outage which affected other applications than just their Webtools API, and also that they hire a 3rd party company to handle their Webtools API software. She couldn't provide any other information at all and I told her a company of that size should have some sort of fail over plan in place to prevent them from being down as long as they have been. I was really disappointed in the fact she didn't even ask me for my name, phone number, or company by time the conversation was over, but she was probably being bombarded with phone calls all day.
After figuring out that USPS was completely down, I looked through our fail over code and found the following equations seem to come close to the USPS pricing:
National shipping: [cart-weight]*1.6+3.00
International shipping: [cart-weight]*1.6+15.00
These also include pricing for insurance.
After tweaking the fail over pricing code to this, it seemed that everyone using USPS were happy with the results. We also had to decrease the connection timeout set for the request to the USPS Webtools API which was also slowing things down.
The Webtools API seems to be both up and down today, with some orders having shipping prices directly from USPS and others having the fallback pricing. Either way, hopefully their IT department learns from this and also provide us information as to what exactly went wrong.
On that note, this is a prime example that I use when speaking to prospects about the advantages of using a hosted shopping cart solution rather than a licensed/free download solution. Besides the obvious IT benefits that you get with a good hosted shopping cart solution, hosted shopping cart software is typically a centralized application that can provide quick updates to problems like these. Of course this is assuming that the prospect is serious about their online store and doesn't want to handle technical support themselves.
Oh, Crap! (Score:2)
This explains why my Christmas sales are in the toilet. I'm glad I took a break and decided to read Slashdot. I've had fairly good luck with my OSC connection directly into USPS. Guess not now, huh?
USPS rate API SUCKS (Score:5, Interesting)
Apart from the trouble reported in this /. article which I found occuring on one of the existing sites I wrote yesterday (simply because there were no USPS prices being returned, no error, but took about 30 seconds to work out what happened), USPS simply sucks ass.
Here's why:
Some time ago, they had an API to get rates, it was called RateV2.
Then they "updated", and now have RateV3.
RateV3 is the only specification published.
To get access to the Rate API servers, you must first test your implementation against thier testing servers successfully, when they see that they let you on the production server....
Thier testing servers only work with a limited version of RateV2.
So, in order to use the USPS API, you must:
Write to the now unpublished RateV2.
Test that RateV2 on the test servers.
Ask USPS to allow you to use Production (and get the keys etc) because you have successfully tested.
Write completely new code against RateV3.
Test that RateV3 on the production servers.
And if you try and show the USPS staff the logical problem in this process, they will reply "I can not put you on production servers, until you have done three successful tests on the test servers".
Re:USPS rate API SUCKS (Score:5, Funny)
If you want to get USPS's attention, FexEx them a package. For some reason, it really pisses them off.
You think that's bad -check the Unemployment sites (Score:2)
The California EDD (Employment Development bureau that handles unemployment) www.edd.ca.gov signup website and their phone systems are literally non-responsive.
Maybe more people are filing for unemployment than trying to get their stuff shipped.
"Crashes"? (Score:2)
Seems to me that a program that crashes rather than degrading gracefully when a remote network resource becomes unavailable needs a bit of work.
What Have We Learned From This? (Score:2)
The call from the User to the parent poster m2pc could not have been a fun one. But some interesting Ah-Ha's can be gleaned from this event. "Try-Catch-Finish" can be a troubling paradigm, but when your user employs the administrative technique of MBH, ( Management By Hysteria ), having your program tell you what its inputs were, and what the response was from the "other guy's" server in a nice log file can be; satisfying. This problem could have been reduced further by having the program email any nonst
Re:Healthcare? (Score:5, Insightful)
Re: (Score:2)
The postal service is an independent agency of the US government, like the CIA, the EPA, the FCC, the FTC, NASA, the NSF, the SSS, the SSA, the SEC.
Re:Healthcare? (Score:5, Informative)
UPS and FedEx can't, because it's illegal [aei.org].
Re: (Score:2)
Well we certainly wouldn't want private companies running our postal service, why think of all the unsolicited advertisements we'd end up with crammed into our mailboxes every day! oh, wait...
Re: (Score:2)
Sure they can. However, they aren't allowed to because it is ILLEGAL UNDER FEDERAL LAW. So much for the Post Office being "private".
Re: (Score:2)
Re: (Score:2)
First of all, Fedex and UPS are legally prevented from setting a rate below a certain number ( I don't remember what that is).
Secondly, Fedex and UPS are not allowed to deliver normal mail - only 'urgent packages'. That is why you see all the fancy mailers and boxes from them and not a simple envelope that will keep costs d
Re:Healthcare? (Score:5, Insightful)
The government can't even manage to keep a simple web service online, and people still believe that it would be wise to let them control health care.
Get real. On several occasions, I've had to manually intervene to fix idiotic billing f*ckups between my PRIVATE insurer and a PRIVATE hospital, who had entered into mutual contracts to be in the same "network". For some reason, they couldn't get their own computers to talk to each other and I had to fix their bugs by going deciphering cryptic paper printouts myself and wasting hours calling customer service. This kind of stupid private healthcare IT problem happens routinely to millions of people every year. Therefore, using your reasoning I conclude that due to a clear history of incompetence, it is unwise to let private parties handle health care, and such practice should be banned.
Re: (Score:2)
Re:Sounds like fun. (Score:5, Interesting)
USPS has IT people? Oiy veih I can imagine that job.
A few years ago, I used to be one of them...
Oy Vey, indeed...
Re:Sounds like fun. (Score:4, Funny)
Re: (Score:2, Informative)
You do realize that most of the routing of letters is done by computers, using OCR, right?