How $55.7 Million Doesn’t Equal $634 Million

Mark of New Jersey

Mark is a Founding Editor of The League of Ordinary Gentlemen, the predecessor of Ordinary Times.

Related Post Roulette

36 Responses

  1. J@m3z Aitch says:

    Thank you, Mark. I saw that at Popehat, too, and while cost overruns aren’t unusual, ones of that magnitude are, so it seemed a bit fishy. But who has time to look this stuff up? (Oh, you do–which client did you bill for “research”? *grin*)

    But are you sure Clark “ought to know better?” I’m enjoying his shutdown apocalypse series, but I’m not sure he’s one for rigorous fact-checking.Report

    • It really didn’t take me more than a minute or two to look things up – I know just enough about government contracts to know that the codes that look like gibberish generally are really important and that a lot of contracts are awarded under blanket agreements like this. A quick google search for that IDVPIID code pulled up the cited section of the Federal Acquisition Regulations, and as soon as I saw that link, I knew I had it figured out.Report

    • Troublesome Frog in reply to J@m3z Aitch says:

      I saw that post and immediately thought, “That sounds like absolute nonsense.” I looked at the first dozen or so responses and didn’t see anybody questioning it. Kind of disappointing.

      I immediately thought of the Fox and Friends report that the LAPD was going to spend $1 billion on 10,000 jet packs for its officers. Did nobody in the office hear that and twitch a little bit? Nobody?Report

    • Clark in reply to J@m3z Aitch says:

      I am not one for “rigorous fact checking” (I do color commentary on what I take to be valid facts, not original research), but I am one for getting the story straight and making retractions if I learn that I am in the wrong.

      I’ll update my post right now.

      Thanks for doing the work of digging into the raw data.Report

  2. BlaiseP says:

    ROFL! Groupe CGI. They’re Canadian. How the hell did a Canadian firm get this contract anyway?Report

  3. Richards says:

    @blaisep … no doubt the same way US companies somehow handle pretty much all of Canada’s health info….Report

  4. Scott Fields says:

    There was a story on NPR earlier this week that explained it this way…

    “One might look at this and go: Why can’t we get the smartest people from Facebook and Google and from Twitter to come and work out these problems?” Johnson says. “The problem is that the way that federal contracting works is so burdensome that the only people who get contracts like this are experts at lobbying and experts at regulations that require you to get these sorts of contracts. And they’re not experts at doing the job of building these websites.”The primary contractor behind the federal health exchange software is a global firm called CGI Federal, which didn’t want to comment for this story. Johnson says it’s not that CGI or other contractors behind healthcare.gov are bad. They’re probably just not the best, because the best people at these tech solutions don’t bother applying.

    Report

    • Scott Fields in reply to Scott Fields says:

      I meant that as a reply to BlaiseP’s question.Report

    • BlaiseP in reply to Scott Fields says:

      No. I know these swine. I know many of these players, I’ve done a lot of government software. CGI Federal is just a cutout division of Groupe CGI, out of Montreal.

      Here’s the score. Two separate groups worked on this site. Front end is by a bunch of granola eaters who couldn’t find their happy little asses in the dark with a map and a flashlight. Viz:

      The people that helped to build the new Healthcare.gov are unusual: Instead of some obscure sub-contractor in a nameless office park in northern Virginia, the site was iteratively created by a cross-disciplinary team of developers and editors at HHS, and contractors at Teal Design, Edward Mullen Studio, and Development Seed, a scrappy startup in a garage in the District of Columbia.

      “This is such a lean site,” said Jon Booth, head of the web and new media group at the Centers for Medicare and Medicaid Services (CMS), in an interview. “HHS had a blanket contract when we when awarded this. Aquilent got creative and brought people on with powerful skills, like Ed and Jessica, a designer at Teal Media, and Development Seed. Most of my team is working on this site; we have internal UX, information architects, designers, developers, and infrastructure people that stood up the cloud environment. Their collaboration is one of the high points of this process.”

      Isn’t that just precious?

      But the back end, where the database resides, the part that is stepping upon its closed-source Oracle appendage, was written by Groupe CGI.Report

      • Patrick in reply to BlaiseP says:

        Generally speaking, when a developer starts talking about “powerful skills”, I start sharpening knives.Report

      • Scott Fields in reply to BlaiseP says:

        Thanks, Blaise.

        I don’t know the players at all.Report

      • Harry in reply to BlaiseP says:

        This pretty much explains where all the total costs are.

        Oracle will take a huge chunk of it. They are really good at government contracting and extortionate prices. So those costs include licensing for Oracle, and some other vendors as well, probably including Symantec and, well, hard to imagine Microsoft not getting a chunk. Then there is the infrastructure, networking (redundant circuits in the data center, etc.)

        Then, as mentioned, all those high-dollar HHS employees that are involved, (i.e. “project managers”), and the allocation of their salaries to the project.

        The $55.7 was probably ONLY the cost to hire CGI to do the development. It’s not surprising at all that designing a custom system, with in-house resources and multiple vendors, that the vendor doing the custom coding earns about 10% of the total cost. 80% of the costs will be maintenance.Report

    • Why can’t we get the smartest people from Facebook and Google and from Twitter to come and work out these problems?

      One of the reasons is the same problem that so many state government software projects (which I’m more familiar with) have: the most important people at the company that wins the contract aren’t the coders, they’re the people who have the expertise to produce a meaningful requirements document for the coders to follow. When the contract went out for bids, there almost certainly wasn’t a requirements doc. The contractor was going to have to go through the law and figure out much of what had to be done. There were undoubtedly blanket statements like “Must conform to relevant HIPAA standards,” which is an enormous can of worms.

      Consider an example. The federal government requires each state to periodically get bids to replace their Medicaid Management Information System (or at least allow other companies an opportunity to bid on the maintenance of the existing system). There are only a handful of companies who ever bid on these jobs. The sheer volume of knowledge necessary to understand what the requirements are has become a barrier to entry for new vendors, particularly small new vendors.

      After I went to work for the Colorado legislature’s Joint Budget Committee, and was the only staffer who had any commercial IT experience, I observed to the staff director one day, “I notice that the departments whose budgets I am assigned have all had a major software system fiasco in the last three years.” To which he responded, “Yes, it was intentional.” I got to sit in on all sorts of post mortem analyses of those disasters. My conclusion at the end of the first year was that all of the problems had the same root cause: the State of Colorado completely lacked the expertise to meaningfully specify what the software systems they put out for bid were supposed to do.Report

      • BlaiseP in reply to Michael Cain says:

        You’ve outlined about half the problem. Now here’s the other half, from the perspective of the contractor who gets to design and implement these systems.

        Government is where IT goes to die. Accenture Government Consulting is by far the worst of these. CSC, SAIC, all of them hire the greenest, cheapest coders they can get. If they can, they outsource large portions of these to the Turtle Stack of subcontractors, of which I am one — or was — I refuse to do any more gummint work. Pays well, the work stinks, the tiers of turtles in contractor management stink worse.

        By the time it gets down to me, it’s about 90 USD / hr + accommodation + rental car and weekly airfare home. I’m sure I’m billed out at a much higher rate, found out once I was out at 300 USD per hour. But I specialise to disaster, I’m never in the first round of assholes they bring in. I’m the guy you call when the brass has already conducted the first round of executions. I usually come in with my own team, who are similarly specialised. We knew how to fix one tech stack and get it working right, an SOA bus used to connect many disparate systems.

        The RFPs and contracts I’ve seen are actually fairly straightforward. It’s the technical specs which are truly awful. You see, an outfit like Accenture Government Consulting does deals with technology vendors: Microsoft, Oracle and the like. IBM has its own government consulting division. They’ll always recommend their own wildly-inappropriate technology stack. When they do, they create these islands. To connect these islands, they rely on people like me to run a fleet of little boats between the islands, carrying data back and forth. I get to see what’s on each of these islands.

        There’s little downside to failure. Clawbacks never happen. No incentive to lower costs, bring projects in on time. These projects are pork, controlled through various appropriations committees. You don’t really think people with any experience handle the actual contract administration, did you? This is all pork. Congress critters love these things. They nibble them up like a fat man eating foie gras. Duke Cunningham got caught but there are ways of doing the same thing — perfectly legally now.Report

      • Kim in reply to Michael Cain says:

        IBM got fired around here for screwing things up really, really badly. Even in Pennsylvania, government will fire you if you suck too badly.Report

      • BlaiseP in reply to Michael Cain says:

        They all get fired at one time or another. They’re only in it for the billable hours. There’s no compunction to succeed, little (if any) payoff if they do succeed. Which do you think a consulting firm will choose, time and materials or a fixed bid with performance bonus? T&M every time.

        First wave: green jamokes — now I don’t mind working with green talent, often they’re perfectly capable of mastering the tools and do good work. But these consulting firms get in there, start a blizzard of PowerPoints and Excel spreadsheets, buffalo these clients, take up a whole office floor with a zillion more coders than anyone can possibly use, fill up the parking lot — naturally, they’ll demand root access to everything, annoy the existing programming staff, etcetera.

        The first wave are just shock troops. They don’t know shit. This is just to get a line on their resumes. Aaand… to increase billable hours, fast. See, when it’s just a 100K project, people really care about burn rate and deliverables. When it’s a four million dollar project, they need to burn off a million of that, quick, so the client is stuck with you. First wave continues until the client has an Out of Results Experience.

        Then it’s time for Second Wave. The first-wave babus, they might have done a good job but nobody’s ever given them any guidance. The second wave is rather like those poor bastards of the 40th New York reinforcing Ward’s line at Gettysburg, stepping over their own dead to get at the Confederates. Battle of the Slaughter Pen it was called. Ward’s line didn’t hold but it dealt out a terrible beating before it retreated. And that’s pretty much how I feel about the Second Wave. By the time I get in there, everyone’s ass is on fire, the politics are poisonous, the entire plan is in ruins — and something has to be done Right Now or the whole thing will degenerate into a rout.

        The big consulting firms don’t care if they succeed or not. All they have to do is get the war started.Report

      • Of course, private sector companies have their own software snafus, they just don’t usually have the results paraded about on the nightly news. The giant corporation I was working for in the early 1990s poured $150M into a new billing system project, and got precisely zero lines of working code for their money. Very large software systems that have to do tens of thousands of different things, all while retaining data integrity, seem to be something that humans just do badly.Report

  5. Jim Heffman says:

    So “$634 million” is inaccurate only in the sense that we have only agreed to spend $634 million, and have not yet actually spent it.

    Keeping in mind, of course, that the price tag on government contracts is a floor and not a ceiling.Report

  6. Kolohe says:

    The Health and Human Services is grounded.

    We installed the Obamacare exchanges so you could turn co-pay amounts on and off, not so you could throw health care raves.

    Now let’s go break open that website and pour it into Speaker Boehner’s Mountain Dew. I heard they have to pump your stomach when you drink that stuff.Report

  7. Nob Akimoto says:

    So another thing… which may or may not be relevant.

    A lot of the public affairs/public policy offices at major American IT firms are starting to fill up with former Republican staffers.

    For example, Darrell Issa’s legislative director is leaving to join Yelp. Google’s head of governmental affairs is Susan Molinari (former Republican congress woman), Microsoft’s Federal Governmental Affairs manager is Rebecca Mark, who worked for the RNC, etc.

    I wonder if there might be some cause and effect there, even if unintentional.Report

    • BlaiseP in reply to Nob Akimoto says:

      Democrats are just as bad, or worse. What you’re looking at happens in all these outfits, they just change out the Rain Makers. It’s so sordid. Leon Panetta was working a profitable little gig at Corinthian Colleges — until some enterprising reporter sussed him out.

      Turn on the lights anywhere and you’ll find Rain Makers from both parties on board. These guys have no shame. It’s a bipartisan problem.Report

      • Nob Akimoto in reply to BlaiseP says:

        Oh, I wasn’t blaming them for being there. I was just noting that it might make it harder for the major firms to be involved in something like an ACA exchange roll out, if the people who are doing the governmental affairs work didn’t have useful contacts in the contracting process.Report

      • BlaiseP in reply to BlaiseP says:

        It goes the other way. These outfits crop up like mushrooms on horse turds. F-35, the worst case I know of. I think it’s got bits and pieces being assembled in 45 states.

        Guess they were going for all 50 but someone dropped the ball…..Report

    • zic in reply to Nob Akimoto says:

      Ha. Go look at the defense industry, @nob-akimoto . You ain’t seen nothing until you look ex-hill-staffers in defense.Report

  8. greginak says:

    FWIW! regarding the good and bad experiences of people with the Ocare interface. Its not just a universal bad experience.Report

  9. Mark T says:

    Dang, it’s just so much easier to make stuff up than it is to work with “facts.” I work for the government and I hear from reliable sources that this here Obamacare not only costs a gazillion dollars a year to pay for all those parasites and takers — and that’s not a made up number, a gazillion is an actual number — but also requires subscribers to serve at least 1 month in a federal concentration camp for opponents of Obamacare for each year that they want coverage, unless they either swear an oath of loyalty to Barack Obama, are or making less than $20,000 per year (in which case the taxpayers have to send them to Tahiti for an all expense paid vacation on Uncle Sam). Now if I said that on Breitbart.com, we would undoubtedly be reading about it on 30 right wing blogs within 24 hours, and it would be the #1 hit on Google. Such is the nature of the wingnut echo chamber.Report

  10. Rufus F. says:

    Here’s an amusing anecdote that’s actually surprisingly relevant given the source (me): my mother worked for that contractor for about a year and very recently took early retirement because she said it was the worst place she’d ever worked.Report