Lies, Damned Lies...

The Electrification of Marketing

At the tail end of the nineteenth century, electricity was starting to have a profound effect on the world. As dramatized in the excellent novel The Last Days of Night, and shortly in the forthcoming film The Current War, Thomas Edison battled with George Westinghouse (the latter aided by Croatian genius/madman Nikola Tesla) for control over the burgeoning market for electricity generation and supply. The popular symbol of the electrical revolution is of course Edison’s famous light bulb, but perhaps almost more important was the humble electric motor.

The electric motor was so important because it revolutionized manufacturing, enabling factories to create assembly lines and realize huge efficiency dividends. The Ball Brothers Glass Manufacturing Company, for example, replaced 36 workers with a single electric crane for moving heavy loads across the factory where they made their famous Mason jars.

But for all the benefits of electric motors, many factories were slow to embrace the new technology. As this article from the BBC World Service’s “50 Things that Made the Modern Economy” podcast explains, by 1900, almost twenty years after Thomas Edison started selling electricity from his generation plants in Manhattan and London, only 5% of factories had switched from steam to electric power. Powering a factory with a steam engine was costly, complicated, and dangerous. So why the reluctance to move to electricity?

The reason lies in the way those factories were organized to take advantage of steam power generation. A typical nineteenth century factory, for example making textiles, looked like the image above. Mechanical power was generated by a single large steam engine which ran more or less continuously, and was transferred to individual machines (such as looms or lathes) via a series of drive shafts, gears and drive belts. Because the power was being transferred mechanically, the machines were packed closely together. This, combined with the constant spinning of the drive shafts, made these factories very dangerous to work in; in 1900, over half a million people in the US (almost 1% of the population) were maimed in factory accidents.

Simply replacing the central steam engine with an electric motor did not deliver significant benefits – the drive shafts and belts to the machines still broke down, factories were still crowded, inefficient and dangerous, and the central motor (now powered by comparatively expensive electricity) still had to be kept running constantly.

To truly capitalize on electrification, factories had to reinvent themselves, replacing all their individual machines with versions that were powered by their own electric motors, with power transferred to them via unobtrusive wires rather than spinning drive shafts. In turn this meant that machines did not need to be so tightly packed together; factories could be reorganized to be more spacious and facilitate the flow of items, paving the way for the production line and improving factory conditions and safety. Ultimately, it was the qualitative transformation in the way things were made which was electrification’s biggest benefit.

Reorganizing the marketing factory

The story of electrification and how it impacted manufacturing in the first decades of the twentieth century provides an interesting parallel to the impact of data and AI on the marketing industry in the first decades of the twenty-first.

Today, many marketing organizations have adopted data in a similar way to how factories first adopted electricity: by applying it to existing business processes and ways of working. In direct marketing, the core processes of list-generation and campaign delivery have not changed fundamentally in fifty years – marketers build target audience lists, map messages to this list, deliver those messages, and then measure the response. The sophistication and complexity of all these steps has changed dramatically, but the process itself is still the same.

However, as electricity led to the development of new kinds of manufacturing machines, so data is leading the the development of new kinds of marketing machines, powered by AI. These new systems, which I have written about before, promise to transform the way that digital marketing is done. But just as before, getting there won’t be easy, and will require marketing leaders to embrace disruptive change.

The current ‘factory layout’ for many marketing organizations is based around individual teams that have responsibility for different channels, such as web, search, email, mobile and so on. These teams coordinate on key marketing calendar activities, such as holiday campaigns or new product launches, but manage their own book of work as a sequence of discrete activities. At Microsoft we’ve made progress in the last few years on bringing many of these teams together, and supporting them with a common set of customer data and common marketing automation tooling. But individual campaigns are still largely hand-crafted.

AI-driven marketing systems use a wide range of attributes at the customer level, combined with a continuous testing/learning approach, to discover which of a range of creative and messaging should be executed next, for which customers, in which channels. They break down the traditional campaign-centric model of customer communications and replace it with a customer-centric, ‘always on’ program of continuous nurture. For these systems to work well, they need a detailed picture of the customer, including their exposure and response to previous communications, and they need a wide range of actions that they can take, including the ability to choose which channel to communicate in for a given message and audience.

A fairly traditional marketing organization that is looking to evaluate the potential of AI-driven marketing will, prudently, lean towards trying the technology in a relatively limited pilot environment, likely choosing just one campaign or program in a single channel for their test. These choices make sense – few companies can easily try out new technology across multiple channels, for both technical reasons (i.e. wiring the thing up) and organizational reasons (getting multiple teams to work together).

But this approach is a bit like a 1900’s factory owner deciding to replace just a single machine in the factory with an electric version. Dedicated (and expensive) wiring would have to be laid to power the machine; it would still be crammed in with all the others, so its size and design would be limited; and it would likely need a dedicated operator. In this environment, it would be unlikely that the single machine would be so transformatively efficient that the factory owner would rush out to buy twenty more.

And so it is with AI-driven marketing. A test within a single channel, on a single campaign, will likely generate modest results, because the machine’s view of the customer will be limited to their experience with that brand in that channel; its message choices will also be limited, since it can only communicate within the single channel. These problems are exacerbated by the expense of laying dedicated data ‘lines’ to the new system, and of building many creative variants, to give the system enough message choice within a single channel.

What’s needed is for AI-based optimization to be applied as an enabling capability in all marketing campaigns, across multiple channels and products. This requires significant investment in data and channel integration; but even more importantly it requires marketers, and marketing organizations, to operate differently. Digital advertising, CRM and e-commerce teams, and their budgets, need to be brought together; instead of marketers creating many discrete campaigns, marketers need to create more evergreen programs that can be continuously optimized over time. The marketing factory needs to be organized around the customer, not the product or channel.

This kind of model represents very disruptive change for today’s marketing organizations, as it did for yesterday’s factory owners. In the end, much of the rise of electrified factories a hundred years ago was due to the efforts of newcomers to the field such as Henry Ford, who jumped straight to an electrified production line in the production of his Model T. Today’s marketing chiefs would do well to heed this lesson from history, as disruptors like Amazon, Tesla and Stitch Fix use process innovation to create streamlined, customer-centric marketing functions that are poised to exploit the transformative technology of AI.

Is Digital Marketing having its ‘Deep Blue’ moment?

Garry Kasparov will forever be remembered as perhaps the greatest chess player of all time, dominating the game for almost twenty years until his retirement in 2005. But ironically he may be best remembered for the match he failed to win twenty years ago in 1997 against IBM’s Deep Blue chess computer. That watershed moment – marking the point at which computers effectively surpassed humans in chess-playing ability – prompted much speculation and hand-wringing about the coming obsolescence of the human brain, now that a mere computer had been able to beat the best chess grandmaster in the world.

Since then, computers and chess software have only grown more powerful, to the point that a $50 commercial chess program (or even a mobile app) can beat most grandmasters easily. Faced with this, you might expect Kasparov and other top-flight players to have grown disillusioned with the game, or defensive about the encroachment of computers on their intellectual territory; but in fact the reverse is true.

Today’s chess grandmasters make extensive use of computers to practice, try out new strategies, and prepare for tournaments, in the process becoming a little more like the machines that outpaced them in 1997. Kasparov himself was instrumental in pioneering a  new type of chess game, Advanced Chess, in which humans are allowed to consult with chess software as they play. In his new book, “Deep Thinking: Where Machine Intelligence Ends and Human Intelligence Begins”, Kasparov writes about an Advanced Chess match he played in 1998 against Veselin Topalov:

“Having a computer partner also meant never having to worry about making a tactical blunder. The computer could project the consequences of each move we considered, pointing out possible outcomes and countermoves we might otherwise have missed. With that taken care of for us, we could concentrate on strategic planning instead of spending so much time on calculations. Human creativity was even more paramount under these conditions.”

What Kasparov and his successors in the competitive chess-playing world have discovered was that, when it comes to chess, the strongest player is not man or machine, but man and machine. In fact, a new kind of chess tournament has sprung up, Freestyle Chess, in which teams of humans and computers compete against one another, each bringing their respective strengths to the game: creativity, strategy and intuition from the humans, and tactical outcome prediction from the computers.

And your point is?

You may be asking what relevance this has to digital marketing. In fact, there are strong similarities between chess and marketing (particularly digital marketing):  they are both highly quantifiable pursuits with clear outcomes which have historically relied solely on human intuition and creativity for success.

As in chess, digital marketing relies upon a continuous reassessment of the ‘board’ (customer behaviors and history) in order to decide upon the next ‘move’ (a particular campaign communication aimed at a particular group of customers). Once the move has been made, the board needs to be reassessed before taking the next move.

Today’s digital marketer is much like the chess grandmaster of the early 1990s – they rely on their intuitive understanding of their audience’s makeup and preferences to decide what offers and messages they want to deliver, to which users, and in which channels. Of course, digital marketers understand that measuring campaign outcomes and audience response (using techniques like control groups and attribution analysis) is very important, but most still operate in a world where the humans make the decisions, and the computers merely provide the numbers to support the decision-making.

Luddites 2.0

When Kasparov was asked in 1990 if a computer could beat a grandmaster before the year 2000, he quipped:

“No way - and if any grandmaster has difficulties playing computers, I would be happy to provide my advice.”

Today’s digital marketers can be forgiven for exhibiting some of the same skepticism. Ask them how they came up with a new idea for an ad, or how they know that a particular product will be just right for a particular audience, and they may not be able to answer – they will just know that their intuition is sound. As a result it can seem incredible that a computer can pick the right audience for a campaign, and match the appropriate offer and creative to that audience. 

But the computers are coming. As I mentioned in my earlier post on bandit experimentation, companies like Amplero, Kahuna and Cerebri AI are pitching intelligent systems that claim to take a lot of this decision-making about creative choice, audience, channel and other campaign variables out of the hands of humans. But where does that leave the digital marketer?

We welcome our robot colleagues

The clue lies in the insights that Kasparov ultimately drew from his defeat. He realized that the strengths he brought were different and complementary to the strengths of the computer. The same holds true for digital marketing. Coming up with product value propositions, campaign messaging and creative are activities which computers are nowhere close to being good at, especially in the context of broader intangible brand attributes. On the other hand, audience selection and targeting, as well as creative optimization, are highly suited to automation, to the extent that computers can be expected to perform significantly better than their human counterparts, much as chess software outperforms human players.

Clearly humans and machines need to work together to create and execute the best performing campaigns, but exactly how this model will work is still being figured out.

Today, most digital marketers build campaign audiences by hand, identifying specific audience attributes (such as demographics or behavioral history) and applying filters to to those attributes to build segments. The more sophisticated the marketer attempts to be in selecting audience attributes for campaign segments, the more cost they incur in the setup of those campaigns, making the ROI equation harder to balance.

The emerging alternative approach is to provide an ML/AI system with a set of audience (and campaign) attributes, and let it figure out which combinations of audience and offer/creative deliver the best results by experimenting with different combinations of these attributes in outbound communications. But this raises some important questions:

• How to choose the attributes in the first place
• How to understand which attributes make a difference
• How to fit ML/AI-driven campaigns into a broader communications cadence & strategy
• How to use learnings from ML/AI-driven campaigns to develop new value propositions and creative executions

In other words, ML/AI-driven marketing systems cannot simply be ‘black boxes’ into which campaign objectives and creative are dumped, and then left to deliver clicks or conversions on the resulting campaign delivery. They need to inform and involve marketers as they do their work, so that the marketers can make their uniquely human contribution to the process of designing effective campaigns. The black box needs some knobs and dials, in other words.

The world of chess offers a further useful parallel here. Chess grandmasters make extensive use of specialized chess software like Fritz 15 or Shredder, which not only provide a comprehensive database of chess moves, but also training and analysis capabilities to help human players improve their chess and plan their games. These programs don’t simply play chess – they explain how they are making their recommendations, to enable their human counterparts to make their own decisions more effectively.

These are the kinds of systems that digital marketers need to transform their marketing with AI. In turn, marketers need to adjust the way they plan and define campaigns in the same way that chess grandmasters have dramatically changed the way they study, plan and play games of chess in the last twenty years, working alongside the computers before, during and after campaigns are run.

In 1997, it was far from clear how chess, and the people who played it , would react to the arrival of computers. Digital Marketing stands on a similar threshold today. Twenty years from now it will seem obvious how marketers’ roles would evolve, and how technology would adapt to support them. We’re in the fortunate position of getting to figure this out as it all unfolds, much as Kasparov did.

6 steps to building your Marketing Data Strategy

Your company has a Marketing Strategy, right? It’s that set of 102 slides presented by the CMO at the offsite last quarter, immediately after lunch on the second day, the session you may have nodded off in (it’s ok, nobody noticed. Probably). It was the one that talked about customer personas and brand positioning and social buzz, and had that video towards the end that made everybody laugh (and made you wake up with a start).

Your company may also have a Data Strategy. At the offsite, it was relegated to the end of the third day, after the diversity session and that presentation about patent law. Unfortunately several people had to leave early to catch their flights, so quite a few people missed it. The guy talked about using Big Data to drive product innovation through continuous improvement, and he may (at the very end, when your bladder was distracting you) have mentioned using data for marketing. But that was something of an afterthought, and was delivered with almost a sneer of disdain, as if using your company’s precious data for the slightly grubby purpose of marketing somehow cheapened it.

Which is a shame, because Marketing is one of the most noble and enlightened ways to use data, delivering a direct kick to the company’s bottom line that is hard to achieve by other means. So when it comes to data, your marketing shouldn’t just grab whatever table scraps it can and be grateful; it should actually drive the data that you produce in the first place. This is why you don’t just need a Marketing Strategy, or a Data Strategy: You need a Marketing Data Strategy.

A Marketing Data What?

What even is a Marketing Data Strategy, anyway? Is it even a thing? It certainly doesn’t get many hits on Bing, and those hits it does get tend to be about building a data-driven Marketing Strategy (i.e. a marketing strategy that focuses on data-driven activities). But that’s not what a Marketing Data Strategy is, or at least, that’s not my definition, which is:

A Marketing Data Strategy is a strategy for acquiring, managing, enriching and using data for marketing.

The four boldface words are the key here. If you want to make the best use of data for your marketing, you need to be thinking about how you can get hold of the data you need, how you can make it as useful as possible, and how you can use your marketing efforts themselves to generate even more useful data – creating a positive feedback loop and even contributing to the pool of Big Data that your Big Data guy is so excited about turning into an asset for the company.

Building your Marketing Data Strategy

So know that you know why it’s important to have a Marketing Data Strategy, how do you put one together? Everyone loves a list, so here are six steps you can take to build and then start executing on your Marketing Data Strategy.

Step 1: Be clear on your marketing goals and approach

This seems obvious, but it’s a frequently missed step. Having a clear understanding of what you’re trying to achieve with your digital marketing will help you to determine what data you need, and what you need to do with/to it to make it work for you. Ideally, you already have a marketing strategy that captures a lot of this, though the connection between the lofty goals of a marketing strategy (sorry, Marketing MBA people) and the practical data needs to execute the strategy are not always clear.

Here are a few questions you should be asking:

Get new customers, or nurture existing ones? If your primary goal is to attract new customers, you’ll need to think differently about data (for example relying on third-party sources) than if you are looking to deepen your relationship with your existing customers (about whom you presumably have some data already).

What are your goals & success criteria? If you are aiming to drive sales, are you more interested in revenue, or margin? If you’re looking to drive engagement or loyalty, are you interested in active users/customers, or engagement depth (such as frequency of usage)?

Which communications strategies & channels? The environments in which you want to engage your audience make a big difference to your data needs – for example, you may have more data at your disposal to target people using your website compared to social or mobile channels.

Who’s your target audience? What attributes identify the people you’d most like to reach with your marketing? Are they primarily demographic (e.g. gender, age, locale) or behavioral (e.g. frequent users, new users)?

What is your conversion funnel? Can you convert customers entirely online, or do you need to hand over to humans (e.g. in store) at some point? If the latter, you’ll need a way to integrate offline transaction data with your online data.

These questions will not only help you identify the data you’ll need, but also some of the data that you can expect to generate with your marketing.

Step 2: Identify the most important data for your marketing efforts

Once you’re clear on your goals and success criteria, you need to consider what data is going to be needed to help you achieve them, and to measure your success.

The best way to break this down is to consider which events (or activities) you need to capture and then which attributes (or dimensions) you need on those events. But how to pick the events and attributes you need?

Let’s start with the events. If your marketing goals include driving revenue, you will need revenue (sales) events in your data, such as actual purchase amounts. If you are looking to drive adoption, then you might need product activation events. If engagement is your goal, then you will need engagement events – this might be usage of your product, or engagement with your company website or via social channels.

Next up are the attributes. Which data points about your customers do you think would be most useful for targeted marketing? For example, does your product particularly appeal to men, or women, or people within a certain geography or demographic group?

For example, say you’re an online gambling business. You will have identified that geo/location information is very important (because online gambling is banned in some countries, such as the US). Therefore, good quality location information will be an important attribute of your data sources.

At this step in the process, try not to trip yourself up by second-guessing how easy or difficult it will be to capture a particular event or attribute. That’s what the next step (the data audit) is for.

Step 3: Audit your data sources

Now to the exciting part – a data audit! I’m sure the very term sends shivers of anticipation down your spine. But if you skip this step, you’ll be flying blind, or worse, making costly investments in acquiring data that you already have.

The principle of the data audit is relatively simple – for every dataset you have which describes your audience/customers and their interaction with you, write down whether (and at what kind of quality) they contain the data you need, as identified in the previous step:

• Events (e.g. purchases, engagement)
• Attributes (aka dimensions, e.g. geography, demographics)
• IDs (e.g. cookies, email addresses, customer IDs)

The key to keeping this process from consuming a ton of time and energy is to make sure you’re focusing on the events, attributes and IDs which are going to be useful for your marketing efforts. Documenting datasets in a structured way is notoriously challenging (some of the datasets we have here at Microsoft have hundreds or even thousands of attributes), so keep it simple, especially the first time around – you can always go back and add to your audit knowledge base later on.

The one type of data you probably do want to be fairly inclusive with is ID data. Unless you already have a good idea which ID (or IDs) you are going to use to stitch together your data, you should capture details of any ID data in your datasets. This will be important for the next step.

To get you started on this process, I’ve created a very simple data audit template which you can download here. You’re welcome.

Step 4: Decide on a common ID (or IDs)

This is a crucial step. In order for you to build a rich profile of your users/customers that will enable you to target them effectively with marketing, you need to be able to stitch the various sources of data about them together, and for this you need a common ID.

Unless you’re spectacularly lucky, you won’t be issuing (or logging) a single ID consistently across all touchpoints with your users, especially if you have things like retail stores, where IDing your customers reliably is pretty difficult (well, for the time being, at least). So you’ll need to pick an ID and use this as the basis for a strategy to stitch together data.

When deciding which ID or IDs to use, take into consideration the following attributes:

• The persistence of the ID. You might have a cookie that you set when people come visit your website, but cookie churn ensures that that ID (if it isn’t linked to a login) will change fairly regularly for many of your users, and once it’s gone, it won’t come back.
• The coverage of the ID. You might have a great ID that you capture when people make a purchase, or sign up for online support, but if it only covers a small fraction of your users, it will be of limited use as a foundation for targeted marketing unless you can extend its reach.
• Where the ID shows up. If your ID is present in the channels that you want to use for marketing (such as your own website), you’re in good shape. More likely, you’ll have an ID which has good representation in some channels, but you want to find those users in another channel, where the ID is not present.
• Privacy implications. User email address can be a good ID, but if you start transmitting large numbers of email addresses around your organization, you could end up in hot water from a privacy perspective. Likewise other sensitive data like Social Security Numbers or credit card numbers – do not use these as IDs.
• Uniqueness to your organization. If you issue your own ID (e.g. a customer number) that can have benefits in terms of separating your users from lists or extended audiences coming from other providers; though on the other hand, if you use a common ID (like a Facebook login), that can make joining data externally easier later.

Whichever ID you pick, you will need to figure out how you can extend its reach into the datasets where you don’t currently see it. There are a couple of broad strategies for achieving this:

• Look for technical strategies to extend the ID’s reach, such as cookie-matching with a third-party provider like a DMP. This can work well if you’re using multiple digital touchpoints like web and mobile (though mobile is still a challenge across multiple platforms).
• Look for strategies to increase the number of signed-in or persistently identified users across your touchpoints. This requires you to have a good reason to get people to sign up (or sign in with a third-party service like Facebook) in the first place, which is more of a business challenge than a technical one.

As you work through this, make sure you focus on the touchpoints/channels where you most want to be able to deliver targeted messaging – for example, you might decide that you really want to be able to send targeted emails and complement this with messaging on your website. In that case, finding a way to join ID data between those two specific environments should be your first priority.

Step 5: Find out what gaps you really need to fill

Your data audit and decisions around IDs will hopefully have given you some fairly good indications of where you’re weak in your data. For example, you may know that you want to target your marketing according to geography, but have very little geographic data for your users. But before you run off to put a bunch of effort into getting hold of this data, you should try to verify whether a particular event or attribute will actually help you deliver more effective marketing.

The best way to do this is to run some test marketing with a subset of your audience who has a particular attribute or behavior, and compare the results with similar messaging to a group who which does not have this attribute (but are as similar in other regards as you can make them). I could write another whole post on this topic of A/B testing, because there is a myriad of ways that you can mess up a test like this and invalidate your results, or I could just recommend you read the work of my illustrious Microsoft colleague, Ronny Kohavi.

If you are able to run a reasonably unbiased bit of test marketing, you will discover whether the datapoint(s) you were interested in actually make a difference to marketing outcomes, and are therefore worth pursuing more of. You can end up in a bit of a chicken-and-egg situation in this regard, because of course you need data in the first place to test its impact, and even if you do have some data, you need to test over a sufficiently large population to be able to draw reliable conclusions. To address this, you could try working with a third-party data provider over a limited portion of your user base, or over a population the provider provides.

Step 6: Fix what you can, patch what you can’t, keep feeding the beast

Once you’ve figured out which data you actually need and the gaps you need to fill, the last part of your Marketing Data Strategy is about tactics to actually get this data. Of course the tactics then represent an ongoing (and never-ending) process to get better and better data about your audience. Here are four approaches you can use to get the data you need:

Measure it. Adding instrumentation to your website, your product, your mobile apps, or other digital touchpoints is (in principal) a straightforward way of getting behavioral events and attributes about your users. In practice, of course, a host of challenges exist, such as actually getting the instrumentation done, getting the signals back to your datacenter, and striking a balance between well-intentioned monitoring of your users and appearing to snoop on them (we know a little bit about the challenges of striking this balance).

Gather it. If you are after explicit user attributes such as age or gender, the best way to get this data is to ask your users for it. But of course, people aren’t just going to give you this information for no reason, and an over-nosy registration or checkout form is a sure-fire way to increase drop-out from your site, which can cost you money (just ask Bryan Eisenberg). So you will need to find clever ways of gathering this data which are linked to concrete benefits for your audience.

Model it. A third way to fill in data gaps is to use data modeling to extrapolate attributes that you have on some of your audience to another part of your audience. You can use predictive or affinity modeling to model an existing attribute (e.g. gender) by using the behavioral attributes of existing users whose gender you know to predict the gender of users you don’t know; or you can use similar techniques to model more abstract attributes, such as affinity for a particular product (based on signals you already have for some of your users who have recently purchased that product). In both cases you need some data to base your models on and a large enough group to make your predictions reasonably accurate. I’ll explore these modeling techniques in another post.

Buy it. If you have money to spend, you can often (not always) buy the data you need. The simplest (and crudest) version of this is old-fashioned list-buying – you buy a standalone list of emails (possibly with some other attributes) and get spamming. The advantage of this method is that you don’t need any data of your own to go down this path; the disadvantages are that it’s a horrible way to do marketing, will deliver very poor response rates, and could even damage your brand if you’re seen as spamming people. The (much) better approach is to look for data brokers that can provide data that you can join to your existing user/customer data (e.g. they have a record for user [email protected] and so do you, so you can join the data together using the email address as a key).

Once you’ve determined which data makes the most difference for your marketing, and have hit upon a strategy (or strategies) to get more of this data, you need to keep feeding the beast. You won’t get all the data you need – whether you’re measuring it, asking for it, or modeling it – right away, so you’ll need to keep going, adjusting your approach as you go and learn about the quality of the data you’re collecting. Hopefully you can reduce your dependency on bought data as you go.

Finally, don’t forget – all this marketing you’re doing (or plan to do) is itself a very valuable source of data about your users. You should make sure you have a means to capture data about the marketing you’re exposing your users to, and how they’re responding to it, because this data is useful not just for refining your marketing as you go along, but can actually be useful other areas of your business such as product development or support. Perhaps you’ll even get your company’s Big Data people to have a bit more begrudging respect for marketing…

Got a DMP coming in? Pick up your underwear

If you’re like me, and have succumbed to the unpardonably bourgeois luxury of hiring a cleaner, then you may also have found yourself running around your house before the cleaner comes, picking up stray items of laundry and frantically doing the dishes. Much of this is motivated by “cleaner guilt”, but there is a more practical purpose – if our house is a mess when the cleaner comes, all she spends her time doing is tidying up (often in ways that turn out to be infuriating, as she piles stuff up in unlikely places) rather than actually cleaning (exhibit one: my daughter’s bedroom floor).

This analogy occurred to me as I was thinking about the experience of working with a Data Management Platform (DMP) provider. DMPs spend a lot of time coming in and “cleaning house” for their customers, tying together messy datasets and connecting them to digital marketing platforms. But if your data systems and processes are covered with the metaphorical equivalent of three layers of discarded underwear, the DMP will have to spend a lot of time picking that up (or working around it) before they can add any serious value.

So what can you do ahead of time to get the best value out of bringing in a DMP? That’s what this post is about.

What is a DMP, anyway?

That is a excellent question. DMPs have evolved and matured considerably since they emerged onto the scene a few years ago. It’s also become harder to clearly identify the boundaries of a DMP’s services because many of the leading solutions have been integrated into broader “marketing cloud” offerings (such as those from Adobe, Oracle or Salesforce). But most DMPs worth their salt provide the following three core services:

Data ingestion & integration: The starting place for DMPs, this is about bringing a marketer’s disparate audience data together in a coherent data warehouse that can then be used for analytics and audience segment building. Central to this warehouse is a master user profile  – a joined set of ID-linked data which provides the backbone of a customer’s profile, together with attributes drawn from first-party sources (such as product telemetry, historical purchase data or website usage data) and third-party sources (such as aggregated behavioral data the DMP has collected or brokered).

Analytics & segment building: DMPs typically offer their own tools for analyzing audience data and building segments, often as part of a broader campaign management workflow. These capabilities can vary in sophistication, and sometimes include lookalike modeling, where the DMP uses the attributes of an existing segment (for example, existing customers) to identify other prospects in the audience pool who have similar attributes, and conversion attribution - identifying which components of a multi-channel campaign actually influenced the desired outcomes (e.g. a sale).

Delivery system integration: The whole point of hiring a DMP to integrate data and enable segment building is to support targeted digital marketing. So DMPs now provide integration points to marketing delivery systems across email, display (via DSP and Exchange integration), in-app and other channels. This integration is typically patchy and influenced by other components of the DMP provider’s portfolio, but is steadily improving.

Making the best of your DMP relationship

The whole reason that DMPs exist in the first place is because achieving the above three things is hard – unless your organization in a position to build out and manage its own data infrastructure and put some serious investment behind data integration and development, you are unlikely to be able to replicate the services of a DMP (especially when it comes to integration with third-party data and delivery systems). But there are a number of things you can do to make sure you get the best value out of your DMP relationship.

 

1. Clean up your data

This is the area where you can make the most difference ahead of time. Bringing signals about your audience/customers together will benefit your business across the board, not just in a marketing context. You should set your sights on integrating (or at least cataloging and understanding) all data that represents customer/prospect interaction with your organization, such as:

• Website visits
• Purchases
• Product usage (if you have a product that you can track the usage of)
• Mobile app usage
• Social media interaction (e.g. tweets)
• Marketing campaign response (e.g. email clicks)
• Customer support interactions
• Survey/feedback response

You should also integrate any datasets you have that describe what you already know about your customers or users, such as previous purchases or demographic data.

The goal here is, for a given user/customer, to be able to identify all of their interactions with your organization, so that you can cross-reference that data to build interesting and useful segments that you can use to communicate with your audience. So for user XYZ123, for example, you want to know that:

• They visited your website 3 times in the past month, focusing mainly on information about your Widget3000 product
• They have downloaded your free WidgetFinder app, and run it 7 times
• They previously purchased a Widget2000, but haven’t used it for four months
• They are male, and live in Sioux Falls, South Dakota
• Last week they tweeted:
  

Unless you’re some kind of data saint (or delusional), reading the two preceding paragraphs probably filled you with exhaustion. Because all of the above kinds of data have different schemas (if they have schemas at all), and more importantly (or depressingly), they all use different (or at least independent) ways of identifying who the user/customer actually is. How are you supposed to join all this data if you don’t have a common key?

DSPs solve these problems in a couple of ways:

• They provide a unified ID system (usually via a third-party tag/cookie) for all online interaction points (such as web, display ads, some social)
• They will map/aggregate key behavioral signals onto a common schema to create a single user profile (or online user profile, at any rate), typically hosted in the DMP’s cloud

The upside of this approach is that you can achieve some degree of data integration via the (relatively) painless means of inserting another bit of JavaScript into all of your web pages and ad templates, and also that you can access other companies’ audiences who are tagged with the same cookie – so-called audience extension.

However, there are some downsides, also. Key amongst these are:

Yet another ID: If you already have multiple ways of IDing your users, adding another “master ID” to the mix may just increase complexity. And it may be difficult to link key behaviors (such as mobile app purchases) or offline data (such as purchase history) to this ID.

Your data in someone else’s cloud: Most marketing cloud/DMP solutions assume that the master audience profile dataset will be stored in the cloud. That necessarily limits the amount and detail of information you can include in the profile – for example, credit card information.

It doesn’t help your data: Just taking a post-facto approach with a DMP (i.e. fixing all your data issues downstream of the source, in the DMP’s profile store) doesn’t do anything to improve the core quality of the source data.

So what should you do? My recommendation is to catalog, clean up and join your most important datasets before you start working with a DMP, and (if possible) identify an ID that you already own that you can use as a master ID. The more you can achieve here, the less time your DMP will spend picking up your metaphorical underwear, and the more time they’ll spend providing value-added services such as audience extension and building integrations into your online marketing systems.

 

2. Think about your marketing goals and segments

You should actually think about your marketing goals before you even think about bringing in a DMP or indeed make any other investments in your digital marketing capabilities. But if your DMP is already coming in, make sure you can answer questions about what you want to achieve with your audience (for example, conversions vs engagement) and how you segment them (or would like to segment them).

Once you have an idea of the segments you want to use to target your audience, then you can see whether you have the data already in-house to build these segments. Any work you can do here up-front will save your DMP a lot of digging around to find this data themselves. It will also equip you well for conversations with the DMP about how you can go about acquiring or generating that data, and may save you from accidentally paying the DMP for third-party data that you actually don’t need.

 

3. Do your own due diligence on delivery systems and DSPs

Your DMP will come with their own set of opinions and partnerships around Demand-side Platforms (DSPs) and delivery systems (e.g. email or display ad platforms). Before you talk with the DMP on this, make sure you understand your own needs well, and ideally, do some due diligence with the solutions in the marketplace (not just the tools you’re already using) as a fit to your needs. Questions to ask here include:

• Do you need realtime (or near-realtime) targeting capabilities, and under what conditions? For example, if someone activates your product, do you want to be able to send them an email with hints and tips within a few hours?
• What kinds of customer journeys do you want to enable? If you have complex customer journeys (with several stages of consideration, multiple channels, etc) then you will need a more capable ‘journey builder’ function in your marketing workflow tools, and your DMP will need to integrate with this.
• Do you have any unusual places you want to serve digital messaging, such as in-product/in-app, via partners, or offline? Places where you can’t serve (or read) a cookie will be harder to reach with your DMP and may require custom integration.

The answers to these questions are important: on the one hand there may be a great third-party system with functionality that you really like, but which will need custom integration with your DMP; on the other hand, the solutions that the DMP can integrate with easily may get you started quickly and painlessly, but may not meet your needs over time.

 

If you can successfully perform the above housekeeping activities before your DMP arrives and starts gasping at the mountain of dishes piled up in your kitchen sink, you’ll be in pretty good shape.

Nicely executed retargeting opt-out (for a change)

Retargeting (sometimes called remessaging or remarketing) has taken off in a big way, recently – Google introduced the feature into AdWords earlier this year, and a host of other players are in the game. Consequently, the interwebs now abound with commentary on the rather spooky nature of the technology, with people being “followed around” the Internet by ads for things they were either searching for, or were looking at on e-commerce websites.

It is true that most retargeting implementations are a bit clunky, and I have been on the receiving end of plenty of them myself. Their most irritating aspect seems to be that the time window for perceived relevance of the retargeted ads seem to be ridiculously long. It’s somehow almost more irritating to be deluged by ads for that miscellaneous widget site that you once visited a few weeks ago (even though you have since satisfied your need for widgets elsewhere) than it is to be served non-targeted (or more broadly targeted) ads.

Such ads are made more bearable by a robust opt-out capability; many ad networks have adopted the IAB’s self-regulatory program, which calls for the advertiser to make it possible to opt out of these kinds of ads, which is to say, stop receiving them; stopping the data collection is a more difficult matter.

So today I want to give a little love to TellApart, not because their retargeting implementation is especially subtle or innovative, but simply because they provide a nice opt-out implementation. Last week I spent a little time looking for a desk for my daughter (who currently occupies our dining table with her homework). So since then I have been served retargeted ads on behalf of the site I visited (www.childrensdesks.com) on various sites. Here’s one from Business Insider:

The nice thing about the ad is it has a little “x” icon in the top right (which actually makes a little more sense than the IAB’s suggested “Advertising Option Icon”, which is a bit cryptic). Clicking it gives me this:

The ability to opt out right in the ad unit is nice, and makes me feel more well-disposed to the advertiser and the site that the ad is running on. Clicking through the “Learn More About These Ads” link at the bottom takes me to TellApart’s website with a little more information and the same option to opt out – though no option to opt out of certain categories of ads, or groups of advertisers.If more retargeting networks provided simpler opt-out capabilities like these, it might help to make these ads seem like less of a scary proposition.

Phorm gets the all-clear from the UK Goverment (kinda)

[Update 10/1/08: BT has announced that it will commence a new trial with Phorm to start September 30 in the UK. The trial, in accordance with the conditions below, is opt-in]

 

Beleaguered behavioral targeting outfit Phorm appears finally to have caught a bit of a lucky break - the UK Government has (belatedly) responded to the EU's queries about Phorm's business practices by saying that Phorm does not break EU data collection/retention laws. But the Department for Business, Enterprise and Regulatory Reform (BERR) - the Government department tasked with assessing Phorm's business and responding to the EU - has placed the following conditions on its approval (from an excerpt of the full letter sent to the EU which is reproduced on The Register - my highlighting added):

• The user profiling occurs with the knowledge and agreement of the customer.
• The profile is based on a unique ID allocated at random which means that there is no need to know the identity of the individual users.
• Phorm does not keep a record of the actual sites visited.
• Search terms used by the user and the advertising categories exclude certain sensitive terms and have been widely drawn so as not to reveal the identity of the user.
• Phorm does not have nor want information which would enable it to link a user ID and profile to a living individual.
• Users will be presented with an unavoidable statement about the product and asked to exercise a choice about whether to be involved.
• Users will be able to easily access information on how to change their mind at any point and are free to opt in or out of the scheme.

The two key bullets here are the last two - Phorm will be required to operate this service as an opt-in service only, with clear language and functionality enabling even opted-in users to opt out at any time. And  BERR states that it will be keeping a close eye on Phorm to ensure that it continues to comply with these conditions.

The news may do a little to shore up Phorm's deflating stock price, which has lost about 80% of its value since the heady days of March. But it's hard to imagine Phorm building much of a sustainable business on the back of an opt-in only system - it's going to be an incredibly hard sell for the ISPs that Phorm partners with (BT, TalkTalk and Virgin Media being the only ones mentioned so far). The only model I can think of is that the ISPs offer reduced rates in exchange for opting into the targeting system; but that negates the very purpose of implementing the system in the first place - to shore up sagging ISP revenues in the wake of the last few years' broadband price wars. I fear that Phorm is not out of the woods yet - especially if the recent happenings at its competitor NebuAd are anything to go by.

Google integrates DoubleClick with AdSense

In a post yesterday on the company blog, Google has announced that it's going to be introducing some DoubleClick-like features into the Google Content network (that is, the group of sites that use AdSense to serve contextual ads). The new functionality includes:

• Frequency capping and reporting
• Improved ad quality
• View-through conversions

These new capabilities are interesting because they are the kinds of functionality that brand (as opposed to direct response) advertisers are likely to be most interested in, and indicate that Google is trying to broaden the appeal of its Content Network inventory in these areas (Google already offers CPM pricing for ads placed on the Content Network).

An interesting detail of the announcement is that Google is now serving a DoubleClick cookie with AdSense ads. The touted benefit to users is that they can now opt out of DoubleClick and AdSense ad targeting with a single click, whilst integration for existing DoubleClick advertisers and publishers will be simplified. The benefit to Google, of course, is that it can start using the behavioral data from the Content Network (which is huge) to be able to sell more targeted ads to their DFA (DART for Advertisers) customers.DoubleClick previously dallied with this kind of functionality in the early part of the decade, but jettisoned the technology back in 2002 in the wake of a bunch of class-action lawsuits accusing it of infringing users' privacy.

Phorm over function

There's been plenty of buzz (more of the angry hornet variety rather than the just-inhaled-a-lungful-of-dope variety) about Phorm of late, precipitated by a press release that the company put out on Feb 14 in the UK, announcing partnerships with three major UK ISPs to provide a system "...which ensures fewer irrelevant adverts and additional protection against malicious websites". Critics of the system  (led by noted UK cage-rattler, The Register) claim that the technology is little more than spyware by another name. The negative press around Phorm's announcement has caused at least one of their ISP partners to back away from the deal, and cause their stock to plummet by more than 30%. It looks like this could be the latest in an increasingly long line of bungled targeting announcements from the industry (Beacon, anyone?). But what went wrong?

What is Phorm?

Phorm as a company is the new name for 121Media, a UK AIM-listed company who started out producing a browser toolbar which tracked your page usage to provide a social media environment, connecting you with other people who were looking at the same page. Ad-funded, the toolbar quickly picked up a reputation for being spyware (even though I agree with Phorm's protestations that it was really adware, which is better, but still tarred with the same brush), so it was dropped and the company renamed Phorm.

The new service Phorm has launched is called Webwise (not to be confused with the BBC site of the same name). Essentially it is technology that ISPs install at their data centers which analyzes the URL and textual content of web pages being served and uses this information to place users into interest categories so that they can be served behaviorally-targeted ads. The technology does this by intercepting the page request and sending a copy of it to a "Profiling" server which extracts keywords and uses this information to assign users to interest groups:

 

 

The same technology has a function to alert the user to phishing web sites; since the URL and content is being examined, phishing sites can be spotted and blocked. This functionality forms a core part of Webwise's value proposition to users.

The other part of the alleged value to users is that this profiling process does not permit the ISP to associate a user's profile with their IP address; that means that the ISP (and any government agency who subpoenaed the ISP's records) could not re-associate the Phorm data with a customer record (ISPs can tell which IP address was assigned to which customer at a particular time). The Phorm system does also not store any of the page information or extracted keywords; once the interest "channel" has been arrived at, all the rest of the data is deleted.

So Phorm claims that its system is a real step forward for user privacy on the Internet, whilst at the same time enabling advertisers to reach their audience more effectively. But the industry (and the public) haven't really seen it like this.

 

Why all the fuss?

Phorm's announcement was always bound to generate a certain amount of controversy, because it's in the sensitive area of behavioral profiling & targeting.  But there has been a particularly virulent reaction in the UK, which, whilst started by sites like the Register, has now spread to the "mainstream" media.

Some of the reasons for the fuss are (comparatively) silly things - for example, the renaming of the company from 121Media, which has just made people nervous, especially given the previous company's adware history, or the fact that the company operates out of serviced offices in the UK and doesn't really have a physical address in the US.

A more serious blunder on Phorm's part is their failure to anticipate the scrutiny that this kind of system would be placed under. In this kind of environment, given the firm's history, absolute transparency is essential, and Phorm hasn't provided this. There are still unanswered technical questions about Phorm's system, such as how it manages the opt-out (does data still get collected, or not?), and there have been inconsistencies in the claims that Phorm has made about third-party privacy audits of their software.

Phorm has also made the mistake of launching prematurely, with many of their partnerships still only half-baked. At the moment there is no benefit to users being delivered, because none of the systems that Phorm has announced are actually live within ISPs, and so all the focus is on the downside. Phorm would have done much better to wait until the service was fully baked with at least one of their partners and they had some real users onboard who could testify to the increased relevance of ads and how comfortable they were with their privacy with Phorm, before making a big splash. The press release looks like the product of an over-zealous PR agency looking to ensure their monthly coverage targets were being hit. Well, they've certainly done that.

 

What can we learn?

The main problem here is a poorly thought-out balance of benefits for 'costs' in this offer. Phorm have claimed that this system protects user privacy, but it doesn't really; it's just an ad targeting system with a better-than-average approach to protecting privacy. Users who are opted into Phorm will still receive cookies and targeted ads from other ad networks, and their behavior will still be tracked by those other networks.

Apart from the phishing protection (which is already baked into IE7 and Firefox anyway, and turned on by default), there's nothing in the Phorm system which provides users with protection of their personal data across the Internet. The only way that Phorm's entry into this market can elevate user privacy overall is if other providers of targeted ads who are storing more data decide to pack up and go home - which I doubt will happen.

The furore also highlights the challenges of partnering with ISPs for this kind of service. Because ISPs are the gatekeepers of the Internet (and because, for many people, switching ISPs is a pain in the a**), users are very sensitive to any perceived exploitation of this relationship by the ISPs. In the UK, ISPs are some of the best-known Internet brands, but also some of the least liked. Ironically the cause of this dislike (poor customer service) is a direct result of the price war that has precipitated ISPs' interest in this kind of service, as they are receiving a cut of the revenues, of course.

Ultimately the tale makes clear how careful any company has to be in launching a service like this - the balance of benefits has to be clearly stacked in favor of the user. As Chris Williams of The Register said during an interview with Phorm's CEO, Kent Ertegrul, said:

"a big difference I see between what you're doing and what Google does is that people feel that they're getting a service from Google. I don't think people feel they'll be getting a service from you"

It will be interesting to see how the Phorm saga plays out. Perhaps one day it'll find its way onto an online marketing MBA module syllabus.