How the Government Outsourceds Intelligence to Silicon Valley

The CIA and other government agencies have invested in hundreds of high-tech startups that not only find new ways to snoop but also bring in revenue for the agencies.


For years, the outsourcing of defense and intelligence work was, with good reason, controversial in political circles. But in the last years of Bill Clinton’s administration, the president authorized the CIA’s creation of the first US government-sponsored venture capital firm, In-Q-Tel, designed to invest in cutting-edge Silicon Valley companies. The firm, named after Ian Fleming’s fictional character “Q,” who masterminds James Bond’s spy gadgets, was founded on Sept. 29, 1999, when the intelligence agencies came to realize they couldn’t produce the technology required to make sense of the vast amount of data they had acquired.

The firm’s mission is to “identify, adapt, and deliver innovative technology solutions to support the missions of the Central Intelligence Agency and broader US community.” This process provided a way of tapping the resources and creativity of Silicon Valley — which undoubtedly had gained a technological edge over government in the post-Cold War period — without the burden of trying to directly recruit the free spirits of Palo Alto into government bureaucracy.

Under the guise of In-Q-Tel, the CIA has invested in hundreds of start-ups, including a company called Keyhole, whose satellite mapping software became Google Earth. In-Q-Tel proved immensely successful in its first five years, bringing revenue into the agency and, more significantly, allowing it to discreetly co-opt technologies and companies that would exponentially enhance its spying capabilities without causing the public to ever raise an eyebrow.

The practice of tapping tech companies for government work, then, began to shed its taboo and appear increasingly attractive to other governmental entities. For example, NASA and the US Army, inspired by the success of In-Q-Tel, are currently planning to develop their own venture capital firms in its image. Thanks in part to In-Q-Tel, the already substantial for-profit investment in the intelligence area was expanded significantly under President George W. Bush, such that it constituted about 70% of the intelligence budget by 2007.

This was just the ticket for former National Security Adviser Admiral John Poindexter to ride when Congress terminated his pet program, Total Information Awareness, in 2003. The program’s aim was to “revolutionize the ability of the United States to detect, classify, and identify foreign terrorists—and decipher their plans—and thereby enable the US to take timely action to successfully preempt and defeat terrorist acts.” Poindexter was stymied in his efforts by a Congress concerned about big government’s threat to privacy.

In this burgeoning market of government/tech synergy—the core of the military-intelligence complex—for-profit corporations play a deceptively important role in disguising the extent of government spying on its own people. As is the case with the government’s seizure and exploitation of the massive data collected by Google, Facebook, and other Internet companies, there is an illusion that this information is simply an enhancement for shoppers and social networkers, rather than the means by which our government keeps tabs on all of our activities—from the political to the most personal.

For that reason, when Poindexter’s program was nixed by Congress, it was a natural fit to turn to a private, clearly for-profit Silicon Valley start-up called Palantir, in part with CIA funding. Palantir is the premier company for unraveling and interpreting dense tangles of information—data sets—for intelligence and law enforcement agencies. It offers software that, in the company’s own words, “connects data, technologies, humans and environments” and, as a bonus, a staff that pays house calls to clients’ offices to customize its programs.

But, as the ACLU notes,

“We don’t know the degree of entanglement between the company and the agencies in terms of how the software is operated. And depending on the details of how it’s used, its deployment could be anything between a good, efficient use of government resources, and a true totalitarian nightmare, monitoring the activities of innocent Americans on a mass scale, collecting the records of those activities and leaving them open for suspicionless exploration by government analysts. Unfortunately, everything we know suggests that it is likely close to the latter.”

Palantir, entrusted to mine that massive trove of personal data on behalf of both local and federal governments, was Poindexter’s Total Information Awareness program renamed and repackaged. Sure enough, it would blossom without him, a testament to his doing more than anyone to make the nightmare of a Big Brother government that knows the desires, fears and habits of each and every soul a frightening reality.

Palantir’s work for the spy agencies is highly classified, and the company has a very active public relations operation aimed at creating a benign impression of its intentions and stressing its non-spy activity, which includes data integration of investment bank knowledge bases as well as “philanthropy engineering.” According to the company, this engineering involves working on “creating slavery-free supply chains, addressing small-plot farmer food security, improving global health and fighting disease outbreaks, providing humanitarian relief in the wake of natural disasters, and more.” But it is clear that the company, from its inception, had made its primary function the designing of surveillance programs for the spy agencies. In fact, Palantir would not have managed to stay in existence were it not for a multimillion-dollar investment and substantial technical support provided by the CIA.

When the CIA and NSA first approached Palantir with funds and support in 2004, the Palo Alto–based “computer software and services company” was a fledgling start-up. Since then, as a contractor for top intelligence agencies—as well as some major private banks, like JPMorgan Chase, and multinational corporations—Palantir is valued at $9 billion and has become the most successful among the numerous CIA-backed data analyzation companies.

Peter Thiel, whose PayPal investment had left him a billionaire, founded Palantir. He was convinced that the tactics that had allowed PayPal to predict credit card fraud would work in identifying terrorists. Other investors, however, were not as convinced, and by 2005, a year after Palantir was incorporated, the company was without a single customer or investor other than Thiel.

Palantir was rescued by a referral to In-Q-Tel. It was that fortuitous contact that resulted in a $2 million CIA investment and the subsequent success of Palantir. But far more important than the CIA’s injection of financial capital was its support—access to the CIA’s secret databases, in-house technical experts, and a rolodex of prospective clients on the Hill. These would give Palantir the lift it needed over the next three years.

Alex Karp, Palantir’s CEO—a hipster exec who obtained his PhD in philosophy at the University of Frankfurt (earned, he might remind us, under the mentorship of the eminent philosopher Jürgen Habermas) and a JD at Stanford Law School—is possessed of “progressive” politics and software engineering ignorance. This eclectic persona is enormously useful to Palantir, which effectively plays the role formerly assigned to the Total Information Awareness program: a monstrous government snoop, mining our most intimate data.

This point was obviously not lost on Poindexter—seemingly the complete antithesis of Karp in terms of personal style and political outlook—as he was casting about to keep his program alive after Congress shut it down.

Palantir has a carefully honed image as a sort of countercultural spy outfit committed to privacy and individual rights in the pursuit of national security. The company’s home page and other marketing devices reek of a virtuous win-win alternative, speaking to the concern imbedded in the Fourth Amendment to the US Constitution that spying on the citizenry is an intrinsically doubtful proposition in a free society and needs to be carefully regulated. Paying homage to that notion, Palantir makes much of its built-in “audit trail,” which presumably prevents unauthorized use of private data by individuals working within a government agency. The company plays up the supposed virtues of this mechanism in the “privacy and civil liberties” section of its official website:

Palantir’s immutable and real-time audit logging technologies help ensure compliance with applicable policies designed to protect privacy and civil liberties. Palantir’s audit logs can be configured to capture the information a particular customer requires in order to identify behavior that might indicate misuse of data. Audit logs can record everything from login attempts to specific user search queries to user views of individual records. . . . Using Palantir, an investigator is able to quickly sift through large amounts of auditing data, identifying suspicious activity, and drill down to that activity to determine whether there may have been a violation of law or policy.

But the grievous problem with this formula is obvious on two counts. One is that using the audit trail to monitor a government agency’s or private corporation’s spying activity is totally self-enforcing. If a company decides to be indifferent to that trail, the Palantir system continues to mine the dat just fine. However, a more serious concern is that it may not be an aberrant individual employee who is using the software to monitor a private citizen’s activities but, in fact, the organization itself.

This is of course routinely true of the NSA, FBI, and CIA, as we have learned through the years from the revelations by whistleblowers of unauthorized spying activities. It is far more likely that the Palantir-designed audit trail will be used to identify not agency-approved practices that violate the law but, rather, that rare whistleblower within the organization who dares to tell the truth about such illegal practices.

So there you have it. So-called private companies that either are directly funded by the US government or profit from US government contracts move to destroy organizations and individuals who dare to expose the reach of government and corporate power—a classic manifestation of the government’s threat to our constitutionally protected freedoms. But because for-profit private companies are used as proxies to engage in such nefarious behavior, the government threat to freedom goes largely unnoticed. Hence the prowess and danger of the military-intelligence complex.

Excerpt from They Know Everything About You: How Data-Collecting Corporations and Snooping Government Agencies Are Destroying Democracy (Nation Books, February). Robert Scheer is editor of, where this also appeared. Email

From The Progressive Populist, April 1, 2015

Blog | Current Issue | Back Issues | Essays | Links

About the Progressive Populist | How to Subscribe | How to Contact Us

Copyright © 2015 The Progressive Populist
PO Box 819, Manchaca TX 78652