Do machines dream of electric malware?

Posted by: Oliver Fisher, Anti-Malware Team

We've explored Google's anti-malware processes several times recently, as well as our efforts to work with webmasters to help protect their users. However, there's been some confusion about the objectivity of our scanning and flagging procedures.

Google uses fully automated systems to scan the Internet for potentially dangerous sites. These systems help detect sites infected with malware and then add a warning that appears in Google search results and in many web browsers. We flag sites in this way to help protect users who might visit them. The warning is a cautionary page, and we never prevent users from viewing the affected site if they choose. It's important to note that sites are often compromised without the webmaster's knowledge, so we provide affected webmasters with further information on the issues we've identified — including showing snippets of the malicious code we find. We also offer free resources in Google Webmaster Tools to help site owners clean their sites and request a re-scan.

Site owners sometimes say that we've made a mistake and that their site does not contain malware. For example, the recent appearance of a malware warning on people.com.cn sparked discussion about how Google flags websites. Our scanners — which are automated and indifferent to a site's subject matter — first found a malicious ad on the book.people.com.cn domain at approximately 3:47 a.m. PT on October 17, 2009. Over several days, the scanners detected thousands of URLs with suspicious content in other people.com.cn domains.

Malicious content can be very difficult to detect. A previous post on this blog offered tips for finding hidden malware and cleaning up websites. There are also good tips on Google's Webmaster Central Blog. If a webmaster has indeed removed the malicious content and filed a malware review request in Webmaster Tools, the warning label will be removed shortly. If it persists, however, it's very likely that dangerous content remains. Our scanners are highly accurate, and false positives are extremely rare.

When Google's automated systems detect dangerous content on a site, an email is sent to several administrative email addresses at the site, as well as to the corresponding Webmaster Tools account if one exists. We sent a notification to people.com.cn at 11:01 a.m. PT on October 17, just as any compromised site would receive. The email includes an explanation of how the site may have become compromised and unknowingly been distributing malware. It also describes the process of removing malware from the site and getting the Google warning removed from the site. A copy of the message sent to the addresses associated with infected sites is below:

We recently discovered that some of your pages can cause users to be infected with malicious software. We have begun showing a warning page to users who visit these pages by clicking a search result on Google.com.
...
We strongly encourage you to investigate this immediately to protect your visitors. Although some sites intentionally distribute malicious software, in many cases the webmaster is unaware because:
1) the site was compromised
2) the site doesn't monitor for malicious user-contributed content
3) the site displays content from an ad network that has a malicious advertiser

If your site was compromised, it's important to not only remove the malicious (and usually hidden) content from your pages, but to also identify and fix the vulnerability. We suggest contacting your hosting provider if you are unsure of how to proceed. StopBadware also has a resource page for securing compromised sites: http://www.stopbadware.org/home/security Once you've secured your site, you can request that the warning be removed by visiting http://www.google.com/support/webmasters/bin/answer.py?answer=45432 and requesting a review. If your site is no longer harmful to users, we will remove the warning.

As the email says, the fastest way for a site to be removed from the malware list is for the webmaster to file a review request via Google Webmaster Tools. Google's automated scanners will periodically re-examine the site even if no such request is received, but the process will take longer. People.com.cn did not file a review request, but our scanners reviewed the site on October 23 and removed the malware warning after finding that the malicious ad was gone.

Malicious display ads are an increasingly common way for sites to unknowingly distribute malware. We recently wrote about the steps that Google takes to help protect our advertising networks. Also, other publishers have recently written about their experiences with deceptive display ads.