How $55.7 Million Doesn’t Equal $634 Million
There is no denying that the rollout of the Obamacare exchange website has thus far been a complete and utter disaster. I tend to agree with the assessment that if these problems aren’t fixed – and soon – it will start to ensure that the “young invincibles” that Obamacare needs to have any chance at addressing the adverse selection problem won’t sign up.
However, there’s a story making the rounds claiming that not only is the website a complete failure thus far, but it has suffered from cost overruns on the order of 1000 percent, with an initial award of $57 million (and additional option years bringing the potential value to a little less than $100 million) but payments to date of over $634 million. This story has taken in people who should know better.
It’s also very wrong, based on a poor understanding of government contracts, and in particular a poor understanding of how data on government contracts is reported. That’s not to say I’m an expert myself- the world of government contracts law is a dark and mysterious place with its own lingo that seems to take people years to learn. Hell, it even has its own special court. I do, however, know a minimal amount about the field – enough to know that one really needs to do one’s homework before jumping to any conclusions about a given contract.
Here, the author of the original story – who appears to be an Obamacare supporter – appears to have based his conclusion on the data from this page of the government’s spending website, which filters all awards under the relevant contract number. And indeed it shows that $634 million has been awarded (not necessarily paid, since all amounts allocated by a contract are allocated to the year of the original award, not the year of payment) under the contract number that includes the exchange website. The amount of the original award for construction of the exchange website, $55 million for the first two years, with three one-year options that could bring the total value to $93 million. That contract was awarded in December 2011 – keep that date in mind.
But if you look a little bit more closely, you’ll start to see that the “contract number that includes the exchange website” is not the same thing as the “contract for the exchange website.” For instance, take a look at the “List of Transactions” for this contract number. You should quickly notice that the award dates on most of the line items are from before December 2011 – in other words, those line items, which are included in the aforementioned $634 million number, were incurred before the exchange website contract was even awarded. Except for the second line item (which happens to be for $55.7 million, the exact amount of the original award), you’ll notice that the other line items list a variety of award dates in 2012 and 2013, suggesting they’re different awards from the exchange website award (hint: they are).
Notice also the charts at the bottom of the page from which the $634 million figure was originally pulled. Those charts list three separate prime award contractors; although each is an affiliate of CGI Federal, it would be fairly odd to have three separate prime awards of the same contract; again, the implication is that the $634 million covers far more than just the exchange website contract. Next, look at the graph at the bottom of the page, which shows awards dating back to 2008, with $152 million awarded in 2011. It seems rather unlikely that between September 30, 2011 and December 31, 2011, CGI Federal was paid $152 million on a $55.7 million contract, no? Besides, as I mentioned above, contract amounts are attributed to the year in which they were awarded, not the year in which they are paid, so if this company had really overrun the award amount as much as the original story claims, the 2011 figure should be $634 million, not $152 million.
In other words, that $634 million is for multiple awards, not just the exchange website. But, you ask, how can that be if those awards are all connected by a single contract number? Well, let’s take a look at the filter used to find that contract number, which has the mysterious letters “IDVPIID.” Remember how I said government contracts have their own weird language? Well, this is a good example.
I had to do some research (again, I’m no government contracts expert), but a little looking around quickly revealed what those letters mean: “Indefinite delivery vehicle procurement instrument identifier.” The important part of that phrase for our purposes is the “indefinite delivery vehicle” part – that phrase is legally defined as an “agreement or contract against which individual orders or purchases may be placed.” In other words, the company that built the website is what might be called an “approved vendor” in other contexts, and is subject to an additional long-term “master” agreement – and has been for quite some time. That doesn’t mean the project wasn’t competitively bid – according to the contract data, four bids were received for the project – just that the project was awarded subject to the terms of the long-term agreement, and probably only similarly approved vendors were eligible to bid (though I could well be wrong about that last part).
So if the $634 million claim is inaccurate, how much has actually been spent on this website to date? The answer is that we don’t know, at least not exactly. But we do know that the data for the website award still lists the “current contract value” at the original $55.7 million. That number may increase next year and the following two years if the government exercises its renewal options, but the key point is that the obligation amount does not appear to have changed even though the initial period of the contract is almost at its end. If more than that amount has actually been spent, there’s not currently any evidence to support such a conclusion. But we can say with a good degree of certainty that the amount spent on the project is nowhere in the ballpark of $634 million, which is no more and no less than the total amount awarded to this particular company for over 110 projects since at least 2010 and possibly earlier.
We should absolutely be vigilant about government waste, and I’d hope the government would get better value for $55.7 million than it has so far, particularly as the stakes of getting this wrong are far higher than a $55.7 million contract. But $55.7 million isn’t $634 million.
UPDATE: It is worth mentioning that the above-referenced amounts are not the sole contracts for creation of the exchanges, as this GAO report indicates (via), and the total costs of the exchange creation contracts do reach into the hundreds of millions of dollars. But that is different from the claim being addressed in this post, which is that there were cost overruns in excess of 1000% on this one particular contract for the creation of the healthcare.gov website by CGI Federal.
UPDATE 2: it appears the original article at the center of all of this has been replaced by a new article that speculates about the total costs of the exchanges, which as I say above is a very different issue than the cost of this particular contract. The original claim appears to have been scrubbed and memory holed, evidenced only by a note indicating that the previous article had a “miscalculation,” but making no reference to the fact that the previous article was on a different topic altogether, nor identifying well the nature of the “miscalculation.”. I have various other problems with the replacement article, but ultimately, it’s pretty evident that much of what is in the replacement article is speculation. The proper way for the author to handle this would have been to simply place a retraction at the top of the original article along with a note that the author is investigating the related story of how much the exchange website has cost or is likely to cost.
Thank you, Mark. I saw that at Popehat, too, and while cost overruns aren’t unusual, ones of that magnitude are, so it seemed a bit fishy. But who has time to look this stuff up? (Oh, you do–which client did you bill for “research”? *grin*)
But are you sure Clark “ought to know better?” I’m enjoying his shutdown apocalypse series, but I’m not sure he’s one for rigorous fact-checking.Report
It really didn’t take me more than a minute or two to look things up – I know just enough about government contracts to know that the codes that look like gibberish generally are really important and that a lot of contracts are awarded under blanket agreements like this. A quick google search for that IDVPIID code pulled up the cited section of the Federal Acquisition Regulations, and as soon as I saw that link, I knew I had it figured out.Report
Yeah, but you’re an attorney. You get paid to look stuff up and find inconsistencies in other people’s stories.Report
I saw that post and immediately thought, “That sounds like absolute nonsense.” I looked at the first dozen or so responses and didn’t see anybody questioning it. Kind of disappointing.
I immediately thought of the Fox and Friends report that the LAPD was going to spend $1 billion on 10,000 jet packs for its officers. Did nobody in the office hear that and twitch a little bit? Nobody?Report
I am not one for “rigorous fact checking” (I do color commentary on what I take to be valid facts, not original research), but I am one for getting the story straight and making retractions if I learn that I am in the wrong.
I’ll update my post right now.
Thanks for doing the work of digging into the raw data.Report
Thanks, Clark. This is exactly why I’ve always held you folks over at Popehat in such high regard, and also why I chose your post as the one to link to rather than one of the myriad other sites that addressed this story – I knew that you guys are interested in getting facts right even when it goes against narrative.Report
I do color commentary
Which is why I like reading you (even though rumor has it that you’re ruining the Popehat brand!).Report
I’m inclined to think Clark isn’t who we think he is. He never seems to be around when….Report
ROFL! Groupe CGI. They’re Canadian. How the hell did a Canadian firm get this contract anyway?Report
@blaisep … no doubt the same way US companies somehow handle pretty much all of Canada’s health info….Report
There was a story on NPR earlier this week that explained it this way…
Report
I meant that as a reply to BlaiseP’s question.Report
No. I know these swine. I know many of these players, I’ve done a lot of government software. CGI Federal is just a cutout division of Groupe CGI, out of Montreal.
Here’s the score. Two separate groups worked on this site. Front end is by a bunch of granola eaters who couldn’t find their happy little asses in the dark with a map and a flashlight. Viz:
The people that helped to build the new Healthcare.gov are unusual: Instead of some obscure sub-contractor in a nameless office park in northern Virginia, the site was iteratively created by a cross-disciplinary team of developers and editors at HHS, and contractors at Teal Design, Edward Mullen Studio, and Development Seed, a scrappy startup in a garage in the District of Columbia.
“This is such a lean site,” said Jon Booth, head of the web and new media group at the Centers for Medicare and Medicaid Services (CMS), in an interview. “HHS had a blanket contract when we when awarded this. Aquilent got creative and brought people on with powerful skills, like Ed and Jessica, a designer at Teal Media, and Development Seed. Most of my team is working on this site; we have internal UX, information architects, designers, developers, and infrastructure people that stood up the cloud environment. Their collaboration is one of the high points of this process.”
Isn’t that just precious?
But the back end, where the database resides, the part that is stepping upon its closed-source Oracle appendage, was written by Groupe CGI.Report
Generally speaking, when a developer starts talking about “powerful skills”, I start sharpening knives.Report
Thanks, Blaise.
I don’t know the players at all.Report
This pretty much explains where all the total costs are.
Oracle will take a huge chunk of it. They are really good at government contracting and extortionate prices. So those costs include licensing for Oracle, and some other vendors as well, probably including Symantec and, well, hard to imagine Microsoft not getting a chunk. Then there is the infrastructure, networking (redundant circuits in the data center, etc.)
Then, as mentioned, all those high-dollar HHS employees that are involved, (i.e. “project managers”), and the allocation of their salaries to the project.
The $55.7 was probably ONLY the cost to hire CGI to do the development. It’s not surprising at all that designing a custom system, with in-house resources and multiple vendors, that the vendor doing the custom coding earns about 10% of the total cost. 80% of the costs will be maintenance.Report
Why can’t we get the smartest people from Facebook and Google and from Twitter to come and work out these problems?
One of the reasons is the same problem that so many state government software projects (which I’m more familiar with) have: the most important people at the company that wins the contract aren’t the coders, they’re the people who have the expertise to produce a meaningful requirements document for the coders to follow. When the contract went out for bids, there almost certainly wasn’t a requirements doc. The contractor was going to have to go through the law and figure out much of what had to be done. There were undoubtedly blanket statements like “Must conform to relevant HIPAA standards,” which is an enormous can of worms.
Consider an example. The federal government requires each state to periodically get bids to replace their Medicaid Management Information System (or at least allow other companies an opportunity to bid on the maintenance of the existing system). There are only a handful of companies who ever bid on these jobs. The sheer volume of knowledge necessary to understand what the requirements are has become a barrier to entry for new vendors, particularly small new vendors.
After I went to work for the Colorado legislature’s Joint Budget Committee, and was the only staffer who had any commercial IT experience, I observed to the staff director one day, “I notice that the departments whose budgets I am assigned have all had a major software system fiasco in the last three years.” To which he responded, “Yes, it was intentional.” I got to sit in on all sorts of post mortem analyses of those disasters. My conclusion at the end of the first year was that all of the problems had the same root cause: the State of Colorado completely lacked the expertise to meaningfully specify what the software systems they put out for bid were supposed to do.Report
You’ve outlined about half the problem. Now here’s the other half, from the perspective of the contractor who gets to design and implement these systems.
Government is where IT goes to die. Accenture Government Consulting is by far the worst of these. CSC, SAIC, all of them hire the greenest, cheapest coders they can get. If they can, they outsource large portions of these to the Turtle Stack of subcontractors, of which I am one — or was — I refuse to do any more gummint work. Pays well, the work stinks, the tiers of turtles in contractor management stink worse.
By the time it gets down to me, it’s about 90 USD / hr + accommodation + rental car and weekly airfare home. I’m sure I’m billed out at a much higher rate, found out once I was out at 300 USD per hour. But I specialise to disaster, I’m never in the first round of assholes they bring in. I’m the guy you call when the brass has already conducted the first round of executions. I usually come in with my own team, who are similarly specialised. We knew how to fix one tech stack and get it working right, an SOA bus used to connect many disparate systems.
The RFPs and contracts I’ve seen are actually fairly straightforward. It’s the technical specs which are truly awful. You see, an outfit like Accenture Government Consulting does deals with technology vendors: Microsoft, Oracle and the like. IBM has its own government consulting division. They’ll always recommend their own wildly-inappropriate technology stack. When they do, they create these islands. To connect these islands, they rely on people like me to run a fleet of little boats between the islands, carrying data back and forth. I get to see what’s on each of these islands.
There’s little downside to failure. Clawbacks never happen. No incentive to lower costs, bring projects in on time. These projects are pork, controlled through various appropriations committees. You don’t really think people with any experience handle the actual contract administration, did you? This is all pork. Congress critters love these things. They nibble them up like a fat man eating foie gras. Duke Cunningham got caught but there are ways of doing the same thing — perfectly legally now.Report
IBM got fired around here for screwing things up really, really badly. Even in Pennsylvania, government will fire you if you suck too badly.Report
They all get fired at one time or another. They’re only in it for the billable hours. There’s no compunction to succeed, little (if any) payoff if they do succeed. Which do you think a consulting firm will choose, time and materials or a fixed bid with performance bonus? T&M every time.
First wave: green jamokes — now I don’t mind working with green talent, often they’re perfectly capable of mastering the tools and do good work. But these consulting firms get in there, start a blizzard of PowerPoints and Excel spreadsheets, buffalo these clients, take up a whole office floor with a zillion more coders than anyone can possibly use, fill up the parking lot — naturally, they’ll demand root access to everything, annoy the existing programming staff, etcetera.
The first wave are just shock troops. They don’t know shit. This is just to get a line on their resumes. Aaand… to increase billable hours, fast. See, when it’s just a 100K project, people really care about burn rate and deliverables. When it’s a four million dollar project, they need to burn off a million of that, quick, so the client is stuck with you. First wave continues until the client has an Out of Results Experience.
Then it’s time for Second Wave. The first-wave babus, they might have done a good job but nobody’s ever given them any guidance. The second wave is rather like those poor bastards of the 40th New York reinforcing Ward’s line at Gettysburg, stepping over their own dead to get at the Confederates. Battle of the Slaughter Pen it was called. Ward’s line didn’t hold but it dealt out a terrible beating before it retreated. And that’s pretty much how I feel about the Second Wave. By the time I get in there, everyone’s ass is on fire, the politics are poisonous, the entire plan is in ruins — and something has to be done Right Now or the whole thing will degenerate into a rout.
The big consulting firms don’t care if they succeed or not. All they have to do is get the war started.Report
Of course, private sector companies have their own software snafus, they just don’t usually have the results paraded about on the nightly news. The giant corporation I was working for in the early 1990s poured $150M into a new billing system project, and got precisely zero lines of working code for their money. Very large software systems that have to do tens of thousands of different things, all while retaining data integrity, seem to be something that humans just do badly.Report
So “$634 million” is inaccurate only in the sense that we have only agreed to spend $634 million, and have not yet actually spent it.
Keeping in mind, of course, that the price tag on government contracts is a floor and not a ceiling.Report
I think you misread the post.Report
Uhh, no. $634 million is the amount of all contracts awarded to this company by HHS over the course of several years, only one or two of which are the healthcare.gov contracts.
Did you not read the part where I pointed out that a large chunk of the $634 million comes from contracts that were awarded before the healthcare.gov contract was even bid out?Report
The Health and Human Services is grounded.
We installed the Obamacare exchanges so you could turn co-pay amounts on and off, not so you could throw health care raves.
Now let’s go break open that website and pour it into Speaker Boehner’s Mountain Dew. I heard they have to pump your stomach when you drink that stuff.Report
So another thing… which may or may not be relevant.
A lot of the public affairs/public policy offices at major American IT firms are starting to fill up with former Republican staffers.
For example, Darrell Issa’s legislative director is leaving to join Yelp. Google’s head of governmental affairs is Susan Molinari (former Republican congress woman), Microsoft’s Federal Governmental Affairs manager is Rebecca Mark, who worked for the RNC, etc.
I wonder if there might be some cause and effect there, even if unintentional.Report
Democrats are just as bad, or worse. What you’re looking at happens in all these outfits, they just change out the Rain Makers. It’s so sordid. Leon Panetta was working a profitable little gig at Corinthian Colleges — until some enterprising reporter sussed him out.
Turn on the lights anywhere and you’ll find Rain Makers from both parties on board. These guys have no shame. It’s a bipartisan problem.Report
Oh, I wasn’t blaming them for being there. I was just noting that it might make it harder for the major firms to be involved in something like an ACA exchange roll out, if the people who are doing the governmental affairs work didn’t have useful contacts in the contracting process.Report
It goes the other way. These outfits crop up like mushrooms on horse turds. F-35, the worst case I know of. I think it’s got bits and pieces being assembled in 45 states.
Guess they were going for all 50 but someone dropped the ball…..Report
Ha. Go look at the defense industry, @nob-akimoto . You ain’t seen nothing until you look ex-hill-staffers in defense.Report
Oh, I’m more than aware of those. I mean the whole point of a governmental affairs office is to leverage connections between former hill staffers and people still working there.Report
And call girl services. They’re an essential part of Gummint Contracting.
The shit I’ve seen….Report
Blaise,
Call girls if you’re lucky.
Most of the pols like far more unsavory (and illegal) stuff than that.Report
FWIW! regarding the good and bad experiences of people with the Ocare interface. Its not just a universal bad experience.Report
Dang, it’s just so much easier to make stuff up than it is to work with “facts.” I work for the government and I hear from reliable sources that this here Obamacare not only costs a gazillion dollars a year to pay for all those parasites and takers — and that’s not a made up number, a gazillion is an actual number — but also requires subscribers to serve at least 1 month in a federal concentration camp for opponents of Obamacare for each year that they want coverage, unless they either swear an oath of loyalty to Barack Obama, are or making less than $20,000 per year (in which case the taxpayers have to send them to Tahiti for an all expense paid vacation on Uncle Sam). Now if I said that on Breitbart.com, we would undoubtedly be reading about it on 30 right wing blogs within 24 hours, and it would be the #1 hit on Google. Such is the nature of the wingnut echo chamber.Report
Here’s an amusing anecdote that’s actually surprisingly relevant given the source (me): my mother worked for that contractor for about a year and very recently took early retirement because she said it was the worst place she’d ever worked.Report