News :: story cache

Game theory in the popular press.

Market Forces May Solve Search Engines' "Tragedy of the Commons"

Robert Woodhead
February 28, 2001
text is a cache of

In the old days (1999!), submission was a numbers game. The more sites you could properly submit to, the more traffic you'd get. But as the internet matures, this is rapidly changing.

The big difference is that these days, it costs to get a listing in the major indexes (Yahoo, LookSmart and NBCi), and the search engines are also getting into the act -- Inktomi now has a paid-submission option, and the other major search engines are all considering following along.

There has been a lot of blathering on various discussion forums lately about the "death of free submission," but quite frankly I think that it's a good thing. I know that sounds weird, but bear with me for a minute.

From the standpoint of a commercial website, the problem with the net is that it is drowning in other websites. The number of webpages is growing rapidly, which makes it harder for people to find your website in the sea of other, similar websites.

When people want to find a website, they typically go to one of the major indexes or search engines. From your standpoint as a commercial website operator, would you like their search to list 10,000 websites (submitted for free), or 100 websites (each of which has paid $199 to get listed)? Obviously, the latter!

At the same time, as long as free submissions are available for non-commercial sites (and all the majors still permit this), non-commercial sites are not penalized, because they are not competing with commercial sites. So if you are not trying to make money on the web, fear not -- nothing has changed for you.

But if you are serious about doing business on the web, then paid submissions are a good thing. They cut out all the chaff. Or to put it more bluntly, "Money talks, B*llsh*t walks!"

The Economics of Paid Submission

If you make a profit of $10 when someone buys something on your site, and 1% of your visitors buy something, then each new visitor is worth 10 cents to you. If you can get visitors for 9 cents each, you make money; if they cost you 11 cents, you lose money. It's that simple.

There are three main sources for paid clicks that you should consider; paid submission to indexes (like Yahoo), paid inclusion in a search engine (currently only Inktomi provides this), and sponsored listings in a pay-per-click search engine (the biggest being All can be excellent sources of cost-effective traffic.

Paid Submission to Indexes

Three major indexes are now offering paid submission, and some of them require it in their business and shopping categories. It is important to understand that paid submission does not guarantee a listing -- if your site isn't good enough, they'll reject it and you're out the money (though you can appeal). I have an extensive tutorial on how to submit to Yahoo and the major indexes that should minimize your chances of a rejection and maximize your chances of a great listing that generates real traffic.

I paid $199 for my Yahoo listing, and last year Yahoo sent me over 25,000 visitors. $199 / 30,000 = .8 cents a visitor. That's dirt cheap, and this year I get them for nothing, since a Yahoo registration is a 1-time thing. If you're selling on the web, that $199 is the best investment you'll ever make (other than the money you send to me, of course!)

Yahoo has just started offering Sponsored Listings for between $25 to $300 a month, depending on category. 5 sponsored listings are displayed at the top of category pages (if more than 5 people buy sponsored listings, they rotate randomly). In order to get a sponsored listing, you must first get a normal listing in Yahoo, then you can apply for a sponsored listing in the category your listing is in. You can't use this to change your listing title or description, by the way; it just gets you "up top."

Is it worth it? Hard to say, at present. I've applied for one and I'm going to be watching it closely. One problem may be that as more people sign up for a sponsored listing on a particular page, each listing appears less often (for example, if 10 people sign up, on average each will get their link displayed on only half of the page views) - and it is unclear if Yahoo is going to adjust pricing to compensate for this. If you see a "more sponsored sites" link, then there are more than 5 sponsored listings for the page, click on the link to see how many.

To apply for a sponsored listing, visit the Yahoo category page that contains your listing and click on the "what is a sponsored listing?" link.

Yahoo has also introduced "Most Popular" listings underneath the sponsored listings; so far it is unclear how a site becomes a Most Popular site.

The same goes, to a lesser degree, for the paid-inclusion deal at LookSmart. My $199 LookSmart listing generated about 1600 clicks last year (about 12.5 cents a click). Not nearly as economical as Yahoo, but not bad -- and again, this year the clicks will be free.

On the other hand, I do not recommend paying for a listing at NBCi. My experience is that they don't generate much traffic at all, and most of it comes from their "LiveDirectory" listings that you can submit to for free.

There are other, 2nd-tier indexes that offer paid submissions. None of them are worth it.

Paid Inclusion on Search Engines

Currently, only Inktomi offers a paid inclusion service through a company called PositionTech. For $30 a year, PositionTech will guarantee your URL stays in the Inktomi database, and check it every couple of days for changes. The advantages of coughing up the cash are:

I have tested the Inktomi paid inclusion service and it works as advertised. I recommend that commercial sites pay $30 and list their homepage using the service. While you can spend extra and list other URLs on your site (at a discount) via paid inclusion, in most cases this isn't necessary. It appears to be the case that paid inclusion URLs are often used as starting points by the Inktomi spider, and so your other URLs tend to be quickly found and indexed with no penalty. Also, once your page has been found by the spider, resubmitting it via the free Add URL does not cause it to be penalized! So my advice is pay for your homepage, submit all your other pages via the free Add URL, and in a few weeks the spider will have found most of your pages and removed the penalty. I should note that PositionTech says that their paid inclusion service won't cause other URLs to be spidered, but my experience is otherwise. It may be that it doesn't cause your related URLs to be preferentially spidered. The jury is still out on this one.

As with every rule, there are exceptions, and if there are important pages on your site that are constantly changing, paying a little extra to have Inktomi refresh its listing more often may be a good investment.

Sponsored Listings on a Pay-Per-Click Search Engine

If the average visitor to your site earns you 10 cents in profit, and you can get one by paying 5 cents, then you're in a position to make money. That's the idea behind pay-per-click. Search engine results are ranked not by keywords, but by how much you are willing to pay for a click. You bid for each keyword you want to sponsor.

The two leaders in the field are (by far the big kahuna) and FindWhat. The other sites aren't worth much, in my opinion. But the top two can make you money if you use them correctly, and I strongly recommend that commercial sites consider using them. I've written a detailed tutorial devoted to techniques for efficiently bidding on these sites, as well as Secret Net Tools (available to contributors) that help you find appropriate keywords to bid on and manage your bids.

Future Trends

Looking into my crystal ball, I think it's pretty clear that the days of effective free submission to the major search engines are numbered. This is a classic example of "the tragedy of the commons". If a resource is freely available to everyone, then nobody has any incentive to conserve it, and everyone will try and exploit it as much as they can before the other guy ruins it. The result, in a village common, is overgrazing; on search engines, it is spam.

If on the other hand, you charge people a small amount for listings, they get a lot more selective about what they submit. The quality of listings goes up. Spam goes down. This is a good thing. Furthermore, let's be honest, it's in our interests for the search engines to make money, because a bankrupt search engine is no help to anyone.

Since at the present time, most of the search engines (except for Yahoo) are losing money hand over fist, I think that all the major ones will adopt paid inclusion plans similar to Inktomi. I think they'd be stupid not to.

Going out further on a limb, I will make a stunning prediction. I predict that Yahoo will announce that they will start charging commercial sites a yearly maintenance fee for listings, in addition to their $199 listing fee. And by the way, Yahoo guys, if you end up doing this because I suggested it, you owe me - small, unmarked, nonsequentially numbered bills, please.

© 1999-2001