Lies, Damned Lies...

Building the Perfect Display Ad Performance Dashboard, Part II – metrics

Welcome to the second installment in my Building the Perfect Display Ad Performance Dashboard series (Note to self: pick a shorter title for the next series). In the first installment, we looked at an overarching framework for thinking about ad monetization performance, comprised of a set of key measures and dimensions. In this post, we’ll drill into the first of these – the measures that you need to be looking at to understand your business.

 

How much, for how much?

As we discussed in the previous post, analysis of an online ad business needs to focus on the following:

• How much inventory was available to sell (the Supply)
• How much inventory was actually sold (the Volume Sold)
• How much the inventory was actually sold for (the Rate)

Of these, it’s the last two – the volume sold and the rate at which that volume was sold – where the buck (literally) really stops, since these two combine to deliver that magic substance, Revenue. So in this post we’ll focus on volume sold, rate and revenue as the core building-blocks of your dashboard’s metrics.

Volume, rate and revenue are inextricably linked via a fairly basic mathematical relationship:

Revenue = Rate x Volume

Another way of thinking about this is that these three measures form the vertices of a triangle:

Some business and economics textbooks call Rate and Volume “Price” and “Quantity” (or P and Q), but the terms we’re using here are more common in advertising.

Different parts of an ad business can be driven by different corners of the triangle, depending on the dynamics of how each part is transacted. Here are some examples:

• Ads sold on a time-based/”sponsorship” basis are best thought of as driving revenue performance, because deals are done on a revenue basis regardless of volume/rate (though the advertiser will have a volume & rate expectation, which they’ll want to be met).
• For premium ads sold on a CPM basis, deals revolve around Rate; the name of the game is to add value to inventory so that, impression-for-impression, it achieves more revenue.
• For remnant ads and networks, volume is king (assuming you can maintain a reasonable rate) – you’re looking to maximize the amount of inventory sold, and minimize the amount that has to be given away or sent to “house” advertising.

Because of these different dynamics, measurement of ad monetization can easily fragment into various sub-types of measure; for example, as well as cost-per-thousand (CPM) rate, some ads are purchased on a CPC or CPA basis. So a more complete version of the diagram above looks like this:

However, it’s essential to remember the key relationship and dynamic between rate, volume and revenue, which is manifested in the CPM, Impressions and Delivery Revenue measures in the diagram above. So let’s look at these measures.

 

Volume

In the online ad business, Volume is measured in Ad Impressions. I have talked about ad impressions before on this blog, in this installment of Online Advertising 101 (you may want to take a moment to read the section entitled “What’s the product?” in that post). From a measurement point of view, whenever your ad server serves an ad (or more accurately, fields a request for an ad), its measurement system should log an ad impression. How much data is logged with this impression will vary depending on the ad server you’re using, but will likely include most of the following:

• Date & time of the impression
• Advertiser
• Campaign and/or creative
• Location/placement (i.e. where the ad was served)
• Attributes of the individual who requested the ad (e.g. targeting attributes)

We’ll come back to those attributes (and how you can use them to segment your impressions for better analysis) in another post.

Capturing a true view of the ad impressions on your site can be a little more challenging if you are using multiple ad servers or networks to sell your inventory, particularly if you are using a combination of your own first-party ad server (for example, DFP) and redirecting some impressions to a third-party such as an ad network. When you have delivery systems chained together in this way, you may need to combine the impression counts (and other data) from those systems to get a true picture of impression volume, and you will need to be careful to avoid double-counting.

For reasons that will become clearer when we get on to talking about rate, it’s essential that you capture impression counts for your ad sales where you possibly can, even for parts of your site or network where the supply is not sold on an impression basis.

Other volume measures such as Clicks and Conversions become very useful when you’re looking to assess how valuable your inventory is from an advertiser perspective, since both are a proxy for true Advertiser ROI. They’re also useful for deriving effective rate, as we’ll see below.

 

Rate

At the highest level, rate is a simple function of volume and revenue – simply divide your total revenue by your total volume (and usually multiply by 1,000 to get a more usable number) and you have your overall rate – in fact, you have the most commonly used kind of rate that people talk about, known as “Effective Cost-per-Mille (Thousand)”, or eCPM (don’t as me why the e has to be small – ask e.e. cummings). Just to be clear, eCPM is calculated as:

eCPM = (Revenue) * 1000 / (Volume)

Sometimes eCPM is known as eRPM (Where the R stands for “Revenue”).

The reason we’re talking about eCPM before revenue in this post is because many advertising deals are struck on a CPM basis – i.e. the advertiser agrees to buy a certain amount of impressions at a certain pre-agreed rate. However, even for inventory is not being sold on a CPM basis, it’s essential to be able to convert the rate to eCPM. Here’s why.

The beauty about eCPM is it is the lowest common denominator – regardless of how a particular portion of your impression supply was sold (e.g. on a cost-per-click basis, or on a “share of voice” or time-based basis), if you can convert the rate back into effective CPM you can compare the performance of different subsets of your inventory on a like-for like basis. Consider the following example of delivery info for the parts of a fictional autos site:

Site area	Sold as…	Deal
Home page	Share-of-voice	$10,000 up-front
Car reviews	Reserved CPM	$2.50 CPM
Community	AdSense	$1.20 CPC

With just the information above, it’s impossible to understand whether the Home Page, Reviews or Community site areas are doing better, because they’re all sold on a different basis. But if you add impression counts (and, in the case of the Community area, click counts), it’s possible to derive an overall rate for the site, as well as to see which parts are doing best:

Site area	Sold as…	Deal	Impressions	Clicks	CPC	Revenue	eCPM
Home page	Sponsorship	$10,000 up-front	5,347,592	n/a	n/a	$10,000	$1.87
Car reviews	Reserved CPM	$2.50 CPM	3,472,183	n/a	n/a	$8,680.45	$2.50
Community	AdSense	$1.20 CPC	1,306,368	5,832	$1.20	$6,998.40	$5.36
Total			10,126,144			$25,678.85	$2.53

See? Who knew that the Community area was throwing off so much money per impression compared to the other areas?

eCPM isn’t the only rate currency you can use, though its connection to both volume and revenue puts it at a distinct advantage, and it means most to publishers because it speaks to the one thing that a publisher can exert (some) control over – the volume of impressions that are available to sell.

 

Revenue

If you sell your inventory on a fairly straightforward CPM or CPC basis, then your site’s revenue will pop neatly out of the equation:

(Revenue) = (eCPM) * (Volume) / 1000

However, if you’re running a larger site and engaging in sponsorship-type deals with advertisers, your revenue picture may look a little more complex. This is because “sponsorships” (a term which covers a multitude of sins) can contain multiple revenue components, some of which can be linked to ad delivery (and which therefore lend themselves to rate calculations), and some of which cannot.

For example, the sponsorship deal on our fictitious autos site referenced above could in fact contain the following components on the invoice sent to the advertiser or agency:

Item	Cost	Impression Target
100% Share-of-voice rotation, 300x250, Home Page (1 day)	$6,000	3,000,000
100% Share-of-voice rotation 120x600, Home Page (1 day)	$4,000	3,000,000
Sponsor branding – Home Page background (1 day)	$8,500	n/a
Sponsored article linked from Home Page (1 day)	$3,500	n/a
Sponsor watermark on Home Page featured video (1 day)	$1,500	n/a

In the above table, only the first two items are expected to be delivered through the ad server; the other three are likely to be “hard-coded” into the site’s CMS and actually deliver with the page impressions (or video stream, in the case of the last one).

There are a couple of different options for dealing with this second kind of revenue (which we’ll call “non-delivery” revenue) which can’t be directly linked to ad impressions. One is to attribute the revenue to the ad delivery anyway, kind of on the assumption that the ads “drag along” the other revenue. So in the above example, with 5,347,592 impressions delivered across the two units, the “overloaded” eCPM for the ad units would be $4.39.

The challenge with this approach is that the extra revenue is not associated with delivery of any particular ad. So in the above example, if you wanted to calculate the eCPM for just the 120x600 unit on the home page (perhaps across an entire month), would you include the non-delivery revenue? If yes, then how much of it? 50%? 40%? The lack of ability to truly associate the revenue with ad delivery makes these kinds of calls incredibly hard, and open to dispute (which is the last thing you want if you are presenting your numbers to the CEO).

The other approach is to treat the “non-delivery” revenue as a separate bucket of revenue that can’t be used in rate calculations. This keeps the data picture simpler and more consistent on the “delivery” side of the house, but you do end up with an awkward block of revenue that people are constantly poking and saying things like “I sure wish we could break that non-delivered revenue out a bit more”.

 

A complicated relationship

Once you have your arms around these three core measures, you can start to see how they interact, and there lies the magic and intricacy of the dynamics of selling display advertising. The implacable logic of the simple mathematical relationship between the three measures means that if one changes, then at least on of the others must also change. Only by looking at all three can you truly understand what is going on. We’ll dig into these relationships more in subsequent posts, but here’s a simple example of the rate achieved for ads sold on a fictional news site home page:

Someone looking at this chart may well ask “OMG! What happened to our rate in June 2009?” Well, a quick search on Wikipedia will reveal that a certain “King of Pop” died in that month, sending the traffic (and hence the ad impression volume) of most news sites sky-rocketing. In our fictional home-page example, almost all revenue is driven by “share of voice” (time-based) deals, so all that extra volume does is depress the effective rate, because the site earns the same amount per day regardless of traffic levels. So here’s volume and revenue from the same data set, to round out the picture:

We can now see that in fact, June wasn’t a bad month for Revenue; it was the huge spike in traffic that did the rate in.

The above example takes something very important for granted – namely, that we have enough segmentation (or “dimensional”) data associated with our measures to be able to break down site performance into more useful chunks (in this case, just the performance of the home page). In the next blog post, we’ll look at some of the most important of these dimensions. Stay tuned!

Building the Perfect Display Ad Performance Dashboard, Part I – creating a measurement framework

There is no shortage of pontification available about how to measure your online marketing campaigns: how to integrate social media measurement, landing page optimization, ensuring your site has the right feng shui to deliver optimal conversions, etc. But there is very little writing about the other side of the coin: if you’re the one selling the advertising, on your site, or blog, or whatever, how do you understand and then maximize the revenue that your site earns?

As I’ve covered previously in my Online Advertising 101 series, publishers have a number of tools and techniques available to manage the price that their online ad inventory is sold for. But the use of those tools is guided by data and metrics. And it’s the generation and analysis of this data that is the focus of this series of posts.

In this series, I’ll unpack the key data components that you will need to pull together to create a dashboard that will give you meaningful, actionable information about how your site is generating money – or monetizing, to use the jargon.

We’ll start by taking a high-level look at a framework for analyzing a site’s (or network’s) monetization performance. In subsequent posts, we’ll drill into the topics that we touch on briefly here.

 

Getting the measure of the business

Ultimately, for any business, revenue (or strictly speaking, income or profit) is king. If you’re not generating revenue, you can’t pay the bills (despite what trendy start-ups will tell you). But anyone running a business needs a bit more detail to make decisions that will drive increased revenue.

In the ad-supported publishing business, these decisions fall into a couple of broad buckets:

• How to create more (or more appealing) supply of sellable advertising inventory
• How to monetize the supply more effectively – either by selling more of it, or selling it for a better price, or both

Another way of thinking about these decisions is in a supply/demand framework that is common to almost all businesses: If your product is selling like hot cakes and you can’t mint enough to meet demand, you have a supply problem, and you need to focus on creating more supply. If, on the other hand, you have a lot of unsold stock sitting around in warehouses (real or virtual), you have a demand problem, and you need to think about how to make your products more compelling, or your sales force more effective, or both.

Online publishers usually suffer from both problems at the same time: Part of their inventory supply will be in high demand, and the business will be supply-constrained (it is not easy to mint new ad impressions the way a widget manufacturer can stamp out new widgets). Other parts of the inventory, on the other hand, will be hard to shift, and the business will be demand-constrained – and unlike widgets, unsold ad inventory goes poof! when the clock strikes midnight.

So analysis of an online ad business needs to be based on the following key measures:

• How much inventory was available to sell (the Supply)
• How much inventory was actually sold (the Volume Sold)
• How much the inventory was actually sold for (the Rate)

It’s ultimately these measures (and a few others that can be derived from them) that will tell you whether you’re succeeding or failing in your efforts to monetize your site. But like any reasonably complex business (and online advertising is, at the very least, unreasonably complex), it’s really how you segment the analysis that counts in terms of making decisions.

 

What did we sell, and how did we sell it?

Most businesses would be doing a pretty poor job of analysis if they couldn’t look at business performance broken out by the products they sell. A grocery chain that didn’t know if it was selling more grapes or grape-nuts would not last very long. Online advertising is no exception – in fact, quite the opposite. Because online ad inventory can be packaged so flexibly, it’s essential to answer the question “What did we sell?” in a variety of ways, such as:

• What site areas (or sub-areas) were sold
• What audience/targeting segments were sold
• What day-parts were sold
• What ad unit sizes were sold
• What rich media types were sold

The online ad sales business also has the unusual property that the same supply can (and is) sold through multiple channels at different price points. So it is very important to segment the business based on how the supply was sold, such as:

• Direct vs indirect (e.g. via a network or exchange)
• Reserved vs remnant/discretionary

Depending on the kind of site or network you’re analyzing, different aspects of these what and how dimensions will be more important. For example, if you’re running a site with lots of high-quality editorial content, analyzing sales by content area/topic will be very important; on the other hand, if the site is a community site with lots of undifferentiated content but a loyal user base, audience segments will be more relevant.

 

Bringing it together – the framework

I don’t know about you, but since I am a visual person to start with, and have spent most of the last ten years looking at spreadsheets or data tables of one sort or another, when I think of combining the components that I’ve described above, I think of a table that looks a bit like the following:

This table is really just a visual way of remembering the differences between the measures that we’re interested in (volume, rate etc) and the dimensions that we want to break things out by (the “what” and “how” detail). If you don’t spend as much of your time talking to people about data cubes as I do, these terms may be a little unfamiliar to you, which is why I’m formally introducing them here. (As an aside, I have found that if you authoritatively bandy about terms like “dimensionality” when talking about data, you come across as very wise-sounding.)

In the next posts in this series, I shall dig into these measures and dimensions (and others) in more detail, to allow us to populate the framework above with real numbers. We’ll also be looking at how you can tune the scope of your analysis to ensure that

For now, here’s an example of the kinds of questions that you would be able to answer if you looked at premium vs non-premium ad units as the “what” dimension, and direct vs indirect as the “how” dimension:

 

As this series progresses, I’d love to know what you think of it, as well as topics that you would like me to focus on. So please make use of the comments box below.