West Virgnia Auditor Finds Cisco Router Purchase Not Performed Legally

coondoggie writes "West Virginia wasted millions in federal grant money when it purchased 1,164 Cisco routers for $24 million in 2010, a state audit concluded. A report issued this month by the West Virginia Legislative Auditor found the state used a 'legally unauthorized purchasing process' when awarding the router contract, paid for with federal stimulus funds, to Cisco. The auditor also found Cisco 'showed a wanton indifference to the interests of the public' in recommending the investment in its model 3945 branch routers, the majority of which were 'oversized' for the requirements of the state agencies using them, the report (PDF) stated."

Been there, done that

[Virtucon]

I can attest that while Cisco makes great products their sales folks and technical sales consultants are very unscrupulous at times. At a company I was working for we were looking for competitive bidding on a new Wifi Infrastructure. We were currently using old Cisco equipment however management wanted to have an open process and do a competitive bid. The Cisco sales staff and their channel support did everything they could to undermine the competitors even though our bake off showed that in terms of some features, the competitors had better features and security. Ultimately when they sensed that they would lose, they used a product roadmap meeting with our CIO as an opportunity to throw my management and my entire team under the bus at our "flawed" thinking.

Hard sell techniques? Yes. Unprofessional? Definitely.

In this case, it sounds like the Cisco sales rep was looking at his bonus, which was probably very very lucrative considering the total sales contract price. Any Network Architect or Engineer worth his salt wouldn't have recommended this overblown hardware based on the requirements. Hopefully West Virginia will use this opportunity to fix the holes in their procurement process so this doesn't happen again because I don't see Cisco ever giving them a refund.

Cisco's M.O.

[pedestrian crossing]

I'm not surprised, this is Cisco's M.O.

Every quote I've ever gotten from them has been massively inflated by speccing higher end equipment than is necessary. They always give the big pitch for the bigger product - usually to upper mgmt, whether it is overkill or not. Everyone wants to believe they are "the enterprise", so Cisco talks them into enterprise-grade equipment.

Not to say that the state employees shouldn't have questioned the quote. But odds are that the only technically knowledgable people involved were Cisco's people, and they are the pros at fleecing the sheep.

Re:It's honestly slightly astonishing...

[Trepidity]

I don't see it as particularly a public/private difference, but a difference of well-run and poorly-run organizations. That might correlate, but I've seen plenty of examples on the opposite sides as well.

On the private-sector side: have you ever looked at how Enterprise procurement works? Cisco makes a ton of money doing exactly the same thing there. You find some Fortune 100 firm that has a lot of money but no clue about technology, and you recommend a ridiculously over-specced system, which they buy because nobody ever got fired for buying Cisco. Oracle makes their money doing that too.

And on the private-sector side: procurement in Scandinavia is much less of a mess than in the US and UK, which is why building the Copenhagen Metro cost less than 1/10 of the per-mile cost of most U.S. metro construction projects.

Re:Do we need to rehash old stories?

[Anonymous Coward]

It's not a rehash, it's an update. If you had bothered to read any of the links you would see that these are the state's official findings on the matter, and it puts Cisco in the position of potentially not being able to bid on state projects in the future.

Re:Actually, it is quite simple...

[Anonymous Coward]

No, the reason that the 3945s were recommended was because the state wanted routers with redundant power supplies, and the 3945 is the lowest model Cisco makes with redundant power...

Cisco and FUD

[Gim Tom]

This really doesn't surprise me. Having worked with a State government in the late 1990's I was in charge of a conversion from Token Ring to eithernet for a moderate sized network for an agency. Cisco seemed to assume that we were all dumb as dirt and insisted that no other brand of eithernet switches would work with their routers which we were already using and which we did want to stay with for the one router we needed.. A classic case of FUD. Fortunately, they were high bid on the overall project by a factor of over two! By using the vendor WE wanted (who also had the lowest total cost) for the switches, and keeping the Cisco router, the conversion went off ahead of schedule and way under budget and worked fine for as long as I was there. My experience taught me that they really didn't CARE what was best for the customer, they just wanted the sale.

Re:It's honestly slightly astonishing...

[silas_moeckel]

It looks like this was a voip build out. So they specked a router supporting a variety of interfaces (old school T1 and ethernet bits) that did local voip processing with PSTN backup and switching with POE. Getting a POE switch, a voip PBX, and the right router would have been far cheaper and probably used less rack space. To do it all in one box in cisco land it's about the correct box.

Re:Actually, it is quite simple...

[msauve]

" the state wanted routers with redundant power supplies"

Well, that's what Cisco claims, but they can't document it. The best they could do was show that redundant power was included in some spreadsheets which the state reviewed. People within the state deny making redundant power a requirement, although they did discuss it for "24/7/365 locations such as regional jails and DHHR state hospitals."

Re:Been there, done that

[Virtucon]

No, they didn't win but we did have a "called on the carpet" discussion about it. We presented our facts and also noted that the procurement organization was in charge and could verify our requirements and process. He couldn't say anything after that.
Cisco did make millions more on Nexus upgrades to the infrastructure but after that any time I see Cisco I just wince.

Re:It's honestly slightly astonishing...

[herring0]

Where I am you can add a step before (1) -
0.1) Buy it from a HUB (Historically Underutilized Business - minority or woman owned business) - even more insane prices than state list

Recently we were also told we can't use the HUBs we have been using (though costly were quite capable and providing a good service to us) and must use another HUB because now we aren't buying enough from specific minority/gender combination groups.

Only at that point are we allowed to proceed the aforementioned idiocy.

Re:Cisco Sucks BUT...

[CowTipperGore]

I likewise call WV home and I've been in IT here for nearly two decades. I've worked directly with Mark Williamson, the Cisco engineer being scapegoated in this mess, many times over the years. I'll say going in that I know I may come off as a Cisco shill. You're welcome to review my post history to see otherwise. I have purchased, implemented, and managed their products at my jobs over the years and I'm fairly agnostic about brand at this point. However, a few things need to be said about this issue and how it is being presented.

• The politicians in Charleston are responsible for this. Period. The auditor's report blames Cisco for undermining the purchasing process because the government employees didn't follow the law. It isn't Cisco's responsibility to put a purchase out to bid. The state government approached them with the grant money and a request to help them spend it. This exact routine happens regularly. This one just happens to be so egregious that local newspaper reporter (Eric Eyre with the Charleston Gazette) refused to let it pass unnoticed.
• The bloggers who are personally attacking Mark (and posting his email and phone numbers with urges to tell him what you think) are allowing the state's shuffling of blame to work. In half a dozen projects that he's helped me spec over the years, he was never the least bit underhanded or disingenuous. I know my stuff and do not take well to bullshit from vendors. I occasionally had to put the local Cisco rep in his place, but I never had to do that with Mark.
• Mark is an engineer, not a salesman. He builds quotes based upon the needs of the customer. The customer's need in this case was to spend a pile of grant money on technology somehow related to homeland security. Show me the State's RFP and agreement with Cisco that contradicts what Cisco ultimately sold and I'll concede a lot of this. Until then, I fail to see how Cisco, and Mark personally, is at fault for doing exactly what they were asked to do.
• By focusing the blame on Cisco, the State is successfully deflecting attention from the countless other scandals in this grant. The state Homeland Security chief, Jimmy Gianato was the grant administrator and led the project. Despite the position taken by the auditor, he still defends the purchase as appropriate. Mr. Gianato also defended paying his 25 year old son $73,000 of grant money across four months to help design and build a dozen microwave towers. He then defended hiring his son at $37,500 per year to inspect the same towers for the State. His son worked out of his home and was provided all expenses (rental vehicles, meals, lodging, gas, and other incidentals on his personal credit card) paid out of the grant.
• One last tidbit that everyone seems to be ignoring - the equipment was specced and provided by Cisco but it was not sold to the state by Cisco. They work through their partner network for the sales. The primary VAR for that in WV is Verizon's Network Integration group. Our state CTO, Gale Givens, was a career Verizon executive, recently in charge of the territory that includes WV. VNI made a pretty penny for little effort on this deal.

This stimulus money was treated as a windfall by Jimmy Gianato and abused like every pork barrel project in WV has been for as long as anyone remembers. Allowing the State to pin the blame on one (genuinely nice) engineer at Cisco is only continuing the abuse of the system by those really guilty here.