Edelman on Click Fraud: Snarky, Real, and Hard to Control

Thanks to several Internet Outsider readers (and journalists) who pointed me toward Harvard PhD candidate Ben Edelman's recent work on spyware click-fraud at Yahoo!  Edelman articulates his research and findings in extraordinary detail, and provides screen shots, video, and packet logs as proof. 

As Edelman describes (see below), the click fraud he observes is different than the usual kind (competitors clicking on links, bogus AdSense sites, etc.).  This fraud is spyware-driven: PCs are infected with spyware that serves up Yahoo! ads and, in so doing, generates clicks.  Importantly, in these cases, the PC users do not actually click the ads--the spyware produces the clicks automatically.

Edelman's videos, packet logs, and cookie logs are enough to make even the most ad-hardened online user feel sick about what is going on inside his or her machine, and advertisers who pay real dollars for such activity will, at best, be infuriated. 

In one particularly sleazy instance, Edelman clicked on a text link within a New York Times story to find that the link had been inserted by a spyware program on his PC and launched an ad.  In other words, a NYT reader who thought he or she was clicking through to additional detail within an NYT story was actually clicking on a non-NYT advertisement.  The click, meanwhile, generated revenue for Yahoo! and the spyware vendor at the advertiser's (and the NYT's) expense. 

This kind of actiivity has got to go, and if Eliot Spitzer doesn't take care of it, someone else will.  In the meantime, along with other types of click fraud, it is artificially puffing up online ad revenue and market statistics.

Unfortunately, Edelman does not--and presumably cannot--say how widespread such activity is.  He also doesn't say whether Yahoo! already offers advertisers refunds on such behavior (although he implies that it is so hard to control that they probably don't).  So, once again, we are left to conclude that click fraud is real and is a problem--and left to wonder whether how big a real problem it is.

Edelman:

Many others have alleged click fraud at Yahoo. (1, 2, 3) But others generally infer click fraud based on otherwise-inexplicable entries in their web server log files -- traffic clearly coming from competitors, from countries where advertisers do no business, or from particular users in excessive volume (i.e. many clicks from a single user). In contrast, my proof of click fraud is direct: As documented and linked above, I have captured click fraud on video and in packet logs. Yahoo may argue about advertisers' inferences in other instances, i.e. disputing that advertisers have really found click fraud. But it's far harder to deny the click fraud shown in my examples.

In the examples I show above and previously, Yahoo's problem results from bad partners within its network. Yahoo syndicates ads to numerous partners, many of whom syndicate ads to others, some of whom then syndicate ads still further. The net effect is that Yahoo does not know who it's dealing with, and therefore cannot exercise meaningful supervision over how its ads are displayed. I consider this a bad idea -- bad business, bad for quality, bad for accountability. But Yahoo need not listen to me. Instead, consider instructions from New York Attorney General staff member Ken Dreifach: "Advertisers and marketers must be wary of fraud or deceptive practices committed by their affiliates, even [affiliates] that they have no working relationships with." (Quote from MediaPost, summarizing Dreifach's remarks.)...

The many bad partners in Yahoo's network make fraud particularly hard to block: When Yahoo terminates one fraudster, that fraudster's partners find another way to continue operations...

Comments

[Jeff Kaplan]

In certain ways, the click fraud issue for Yahoo (and others) reminds me of the sweatshop issue for apparel companies. At root, the question is how Yahoo, like Gap or Nike for example, handles abuses by affiliates.

For clothing companies, this means how do they handle unacceptable business practices by downstream, overseas factories with whom they may have no direct business relationship. In many cases, they too did not know (and didn't want to know) about the textile "syndicates" with whom they were dealing, so long as the price point was right for goods delivered. Ultimately, as we know, pressures forced companies to exercise more direct supervision over their supply chains.

Perhaps pressure from regulators and others might change how online ads are syndicated and how content providers like Yahoo deal with their ad partners?

Jeff Kaplan
Open ePolicy Group
http://jakaplan.blogspot.com

[Still Inside]

Henry/all:

I think we're missing the forest for the trees here in focusing on Yahoo's affiliate network, and affiliate networks in general. (Edelman's research is extremely valuable though for the reasons I mention below).

The "Spyware" that the bad guys are currently using is trained to click on YHOO affiliate ads. However, it can also be trained to do Yahoo.com searches, Google.com searches, etc. and then click on the ads that appear on the pages. It's a a matter of a little more programming (harder, because you'd need to keep a database of the right phrases to search for, but straight-forward).

If I recall, we discussed this exact possibility ("an army of zombies") in this very blog a few months ago (we're so smart).

Further (we're not so smart): email spammers have been using this tactic for years, so it's pretty obvious to assume that click frauders are going to use it.

My opinion on click fraud? In general, its effect on the overall system is almost exactly like email spam. Companies will have to fight it at every level, and it WILL cut into the profits of the system, but it won't kill the system for the long term.

GOOG has gone on record as officially ignoring the problem. This is utterly assinie, both from the technology standpoint and (especially) from the marketing/brand standpoint.

The first company that gets in front of the curve on this problem is going to be the one that can consistently extract a premium from advertisers, and will have the best brand name in the business.

The vendor that loses here stands to lose a lot more than the N% income hit from the fraud itself, but the fact that they will force every customer to buy/build extensive fraud tracking systems and/or indicate a level of uncertainty for this marketing channel.

There is a huge opportunity for YHOO and MSFT to pick this up. GOOG might get lucky and their competitors will ignore it too (MSFT notoriously ignores bad guys, for instance). If this happens a third party will step up (which will be bad for GOOG, since such a product would make the cost-of-switching all the easier for their customers).

SI

[TallTroll]

Click fraud will always be with us. The current technological infrastructure of the WWW is such that it will always be possible to simulate real user behaviour. Also, even if the SEs somehow manage to find and discount every fraudulent click ever produced, they STILL won't kill PPC manipulation.

There are 2 types of fraud unaddressed here : budget burn and impression fraud. Different PPC models are vunerable to one or other of these.

Yahoo (Overture) pay-for-placement model : In a strict auction based system, it is possible to form a bidding syndicate that can effectively drain the budget of any unwanted competitor by co-ordinating bidding policy to max out their bid. Once their daily budget is exhausted, the cartel members simply return to garnering the cheap traffic. Due to the international nature of the web, its virtually undetectable, and totally non-actionable

Google AdWords relevance based positioning model : Where an ads' clickthrough rate (CTR) is part of the algorithm determining the display position, it's possible to identify a keyword, or keywords where your competitors are bidding, and run a bot at that page in Google. Since the bot will trigger a "display" of the ad, but won't click on it, the CTR figures for the displayed ads will drop. Under this model, that will cause effective CPC for a given position to rise, forcing your competitor to either raise their bid, or accept a lower amount of traffic

Interestingly, I hear that Yahoo are planning on moving to a more Google-like model.... and Google are planning on moving to a more Yahoo-like model. I suppose the grass IS greener over there....

[GTKdoug]

The cost of spyware to the economy is enormous and goes well beyond click fraud. Hijacked computers loaded with spyware slow to a crawl and render a computer nearly useless. I know because I've spent countless hours cleaning up infected pcs for my clients. Although it's revenue for me, it's not a productive use of my or my clients resources. This has in part fueled the migration from Windows to Macs and from IE to Firefox. Microsoft should be lobbying hard to get this problem cleaned up.

[w]

How do you measure the effectiveness of your advertising?
Through cost-benefit analysis!
Bottom line: is internet advertising effective; i.e., doesn't it generate more revenue per dollar of advertising.
If click fraud is ubiquitous across the internet, click fraud's effect would be to drive down the cost per click for $0 net effect to the advertiser.
If the clicks on your advertising aren't generating revenues you will do one of two things: pull your advertising or pay less per click. The harm there is to the search engine advertising medium, not to the advertiser.

[Neal Lachman]

Like SI says, we've got spam as an ubiquitous problem. Spyware is becoming an increasingly larger problem because of the interactivity involved.

Spam is a static problem, and although it is very annoying it can be ignored. Spyware simply (and sadly) screws up the computer.

Spam will become also a problem on our cellphones. It is a matter of when, not a matter of "if".

Clickfraud, spyware, and a great mix of both is going to be a crucial factor in the change of online marketing. I am always optimistic about technological advancements and in terms of the upsides of businesses and opportunities (ask MrFGoogle and Market Participant), but I really think this problem of click fraud, spyware, and spam will drive the current marketing systems and technologies to their respective deaths.

Where there is an end of something, there is most likely also the beginning of something. Thus, I believe that the end of the "popular" online marketing tools and means will create opportunities for new, more fool-proof technologies, which will serve the advertiser and the visitor/seeker/searcher even better.

[Market Participant]

I doubt spyware will be the death of internet advertising.

Already newer versions of Windows/IE include pop up blockers and anti-spyware tools. As legacy systems get replaced, the pool of spyware affected computers will shrink.

One thing that will reduce clickfraud is to allow domain target filtering so those advertisers will only pay for clicks from domains of relevant countries.

But ultimately the problem is that as long as clickfraud is unquantifiable, GOOG et al have no reason to work really hard on it, unless it seriously reduces advertiser ROI.

[TallTroll]

>> Already newer versions of Windows/IE include pop up blockers and anti-spyware tools. As legacy systems get replaced, the pool of spyware affected computers will shrink

Nope, most spyware gets around all but the most advanced users and tools. Pop-up blockers are no defence at all (pop-ups are completely different technology to spyware). Anti-spyware tools are by nature reactive : they can only clean up spyware already in existence, they can't provide adequate protection against future developments.

>> only pay for clicks from domains of relevant countries.

So, what do you do with clicks from IPs that don't geo-resolve? Or that come from inside corporate networks? Or come from proxies?

[Neal Lachman]

MP,

I said "I really think this problem of click fraud, spyware, and spam will drive the current marketing systems and technologies to their respective deaths".

That is not the end of Internet Marketing/Advertising. It is the end of the concepts we know today. There is going to be a shift in this industry, where online advertisers will (have to) be more empowered and re-assured.

[D]

In my humble opinion, eventually click fraud will lead to internet advertisers paying for ads based on conversion rates and not the total number of clicks.

[TallTroll]

>> lead to internet advertisers paying for ads based on conversion rates and not the total number of clicks.

So you propose that the PPC engines send traffic to other websites, and just sort of hope they don't suck too badly at converting? Sounds unlikely, frankly

PPC is used to buy traffic. Once the user has clicked, the PPC providers responsibility ends. If you can't convert the traffic that's your problem

Considering how much more trackable, and generally cost effective PPC traffic is, I don't see click fraud as an immediate threat. For many companies, online is the channel with the best ROI, and if it's not perfect yet, so what? You can spend millions on a Superbowl slot, and end up looking really dumb; are you suggesting that TV advertising is dead?

[King Troll]

Henry, cmon where's an internet update. You'd get so much more traffic with more updates.

I schedule you in every morning from 9-9:30 am. Give me something to read asshole.

[Market Participant]

Unless Google becomes more open about the entire adsense process, a better competator can swoop in and take market share away.

As is you basicly have advertisers who are concerned about paying for useless clicks, as well as many publishers wondering about how much and if they get paid. If GOOG wants to stay on top of the tiny text ad business, then they need to really refocus on keeping their core customers and suppliers happy.

[Russ Chen]

Wonder when Ben/Spitzer will out Virtumundo. They are a spyware/spam company (google virtumundo and see what you get). The tried the name change route to clean up their act, but without cleaning up their business. Oh and yes, some VC decided to dump a ton of money to help the process. Sound familiar?

Corporate name change docs can be found here:

http://www.bestsharing.com/files/ms0017270/AdKnowledge_is_Virtumundo.pdf.html

Interesting rate card data here:

http://www.walterkarl.com/content/datacard.aspx?ddcn=97198&mode=Managed