(Yankee Group) June 16, 2005

Bad Hackers Turn Pro

Once an obscure sport, vulnerability hunting has mushroomed into a bona fide economic ecosystem, with fully developed supply and demand chains. Nowhere is this phenomenon more evident than at the twin hacking conferences, DefCon and the Black Hat Briefings, held each summer in Las Vegas. Black-hat and white-hat researchers, security vendors, end-user corporations, venture capitalists and government operatives discuss new research on vulnerabilities and countermeasures. They also exchange ideas, argue, gamble and spy on each other—often simultaneously.

The roiling froth of the Las Vegas festivals disguises the increasing sophistication of the seamier elements of the ecosystem. To paraphrase Hunter Thompson, when the hacking gets weird, the weird turn pro. Anecdotal evidence from a wide variety of sources suggests that organized crime networks are increasingly packaging exploits into instruments of electronic warfare, used for compromising target systems en masse or for attacking targets of choice.

For example, a study by the German Honeynet project estimated that botnet collectives remotely control more than a million compromised Windows PCs, unbeknownst to their owners. The controllers of these “zombie grids” use the combined bandwidth and computing power as for-hire spam or denial-of-service attack engines—thus providing evidence that spam may be the ultimate grid computing application.

Flaw Finding Shifts Toward Security Vendors

Two key factors contribute to the growth of the underground vulnerability economy: historical and chronic weaknesses in the monopoly desktop operating system, Microsoft Windows; and the adolescent enthusiasm of vulnerability researchers continually trying to one-up each other.

Three years into its largely customer-imposed security push, Microsoft flaws continue to flow—but at a significantly decreased rate. Yankee Group analysis of a well-known public vulnerability data source, ICAT, suggests that flaw finders have shifted their focus toward security products.

From 2004 to May 2005 in particular, 77 disclosed vulnerabilities affected a wide array of security products. The incidents increased far faster than the rate for Microsoft (see Exhibit 1). When considering the number of affected products rather than just the number of distinct vulnerabilities, the rate of increase was as fast as that of the industry as a whole. Yankee Group believes this is because of three factors:

• Diminishing returns in operating systems: Security researchers focusing on Windows may have largely depleted the supply of the most easily exploited Microsoft flaws, especially in the wake of the watershed release of Windows XP, Service Pack 2.

  Security Vendor Vulnerabilities Increase

• Low-hanging fruit in security products: Nearly all enterprises have deployed certain classes of security products, notably antivirus and (to a lesser extent) host intrusion prevention. Third-party and press scrutiny has not yet forced security companies to acknowledge and fix potential problems with their code, as it has with operating system vendors. Therefore, flaws targeting security software stand a better chance of being successful.
• Economic self-interest of testing specialists: Although not illegal, some security assessment vendors (notably eEye) specialize in finding vulnerabilities in other vendors’ security products. These vendors then turn around and sell their own security analysis products, which conveniently include detection signatures for the other vendors’ vulnerabilities. As shown in Exhibit 2, these firms discovered one-quarter (20 of 77) of the security product vulnerabilities reported in 2004 and 2005, about the same number discovered by independent researchers. In contrast, fratricidal discoveries by competing security vendors accounted for a minority (5 of 77). One firm—ISS—accounted for four of these.

One mainstream security vendor recently speculated that minority operating systems such as Apple’s OS X and the various desktop Linux distributions would see an increased number of vulnerabilities. We agree in principle—but with installed bases many times larger than either of these, security products present more attractive greenfield opportunities for security researchers.

Some Vendors Are Slow to React

Not all security vendors are ready for the rising tide of vulnerabilities flaw-finders will inevitably discover in their products. Analysis of a cross section of data revealed that publicly disclosed vulnerabilities disproportionately affected Symantec products versus any other security vendor during 2003 and 2004; 2005 appears to be trending in the same direction. Check Point and F-Secure saw a large increase in vulnerabilities in 2004 compared to the previous year, while vendors such as McAfee saw a significant decrease.

Announced vulnerabilities don’t inevitably lead to the creation of a virus or worm. But some security researchers insist on releasing public proof-of-concept code as part of vulnerability announcements. Proof-of-concept code theoretically benefits researchers, vendors and security administrators. However, these demonstration programs are like unprocessed uranium: allegedly dual-use, malicious parties can transform them easily into munitions.

Third Parties Discover Most Security Product Flaws

To date, only one security product vulnerability led to a mass exploit. In early 2004, the Witty worm used techniques described in an eEye advisory to target a specific desktop firewall and intrusion prevention product. The worm carried a destructive payload and compromised 100% of all PCs it encountered running unpatched versions of the targeted product. Not coincidentally, the company whose product was exploited by Witty (ISS) tightened up its security processes and decreased its share of vulnerabilities last year relative to 2003. The Witty worm should have been a wake-up call to the security vendor community. It wasn’t.

Recommendations for Vendors

Three factors strongly influence the number and kind of security vulnerabilities found in software products: the underlying security design, the code quality and the degree of motivation of security researchers. In that sense, security software is just like any other type of software, with one key difference: it usually runs with elevated user privileges and can access nearly all aspects of the operating system. To prevent security software from becoming a preferred conduit for professionally designed malware, security vendors must begin their own security pushes. Vendors should:

• Review security designs early and often. Software design teams should incorporate security requirements during the planning phases for new revisions. Key issues to consider include scope and kind of runtime permissions required, file system access considerations, authentication and authorization, communications integrity, and audit and logging requirements.
• Integrate security tests into nightly builds. Nightly build processes should include unit and regression tests that exercise both expected functionality (positive testing) and out-of-bounds conditions (negative testing). Well-written negative tests can help identify potential buffer overflow and privilege elevation conditions.
• Review code base for security issues. During major revision cycles, the entire product code base should be examined for dead code, outdated functionality and dangerous functions. Combing through every line of code is expensive, but automated toolsets from application security code scanning vendors such as Ounce Labs, Secure Software and Reflective can help speed the analysis. Manual code reviews should focus on functions that take input from external sources and on authentication and authorization subsystems.
• Perform penetration testing prior to release. Simulating the tactics of an attacker is an excellent way to understand a software package’s risk of compromise. Interestingly, both of the leading antivirus companies (Symantec and McAfee) acquired some of the world’s best penetration testers during the last 18 months. Consultants from the former @stake and Foundstone organizations should be unleashed on pre-release software as a formal part of the development lifecycle; they will find the most critical flaws before the bad guys do.
• Create customer collateral describing the security program. Savvy customers already ask mainstream packaged application vendors to supply evidence of secure development processes. They will demand the same from their security vendors. Customer-facing collateral should enumerate vendors’ security assurance programs, such as design, code and implementation reviews; penetration tests; vulnerability reporting/handling; and patching processes.

Recommendations for Enterprises

Smart companies can take a proactive stance to lessen the impact of vulnerabilities in security products. They can:

• Ready the patching infrastructure. Like it or not, security flaws aren’t just a Windows operating system problem. Although patches and fixes from security vendors will not likely come as frequently as those from platform vendors such as Microsoft, enterprises need to prepare themselves. Enterprises should investigate the suitability of their preferred security products’ auto-update mechanisms. In many cases, the chore of updating the software will fall to enterprises’ existing patch management system.
• Ask pertinent questions, impertinently. The spate of recent vulnerabilities in security software products should prompt lively conversations during contract renewal season. Customers should ask their preferred vendors to enumerate their processes for developing security software in a secure manner, and for fixing problems when detected by their staff, customers or third-party researchers. Customers should interpret saber-rattling noises (“We’ll sue anyone who publishes advisories about us”) or hand-waving (explaining away the need for providing process details) as red flags.
• Diversify away risk inherent to any one package. Yankee Group’s April 2005 DecisionNote, Ubiquitous but Not Mature: Antivirus Needs to Grow Up, describes how certain enterprises increase their virus detection rates by mixing antivirus solutions from multiple vendors. Likewise, enterprises can reduce the impact of a successful exploit on their security software by ensuring that their security portfolios contain solutions from dissimilar code bases; that is, by multisourcing.