There is a certain amount of randomness in the calculation of radio ratings, as any number of possible events completely out of your control can affect that month’s (or week’s or even hour’s) rankings and data. In his latest Programming To Win column, Richard Harker urges programmers to look at the bigger picture and understand this inevitable randomness.
Richard Harker
By Richard Harker
It may not be comforting, but all the hard work you and your programming staff put into creating a compelling product can (and often is) nullified by random events that you can neither see nor control. The truth is that chance has a great deal to do with ratings success.
A dearth of hits in your format can hurt you. The unexpected success of a song played by another format can hurt you. A station changing format can hurt you. Bad weather can hurt you. Your morning man leaving to take a job in a larger market can hurt you.
Any of these events can hurt the station, but none of these things can be accurately predicted. And there’s little you can do about them when they occur anyway.
Life is full of unpredictability. Shit happens. And when it happens, we are genetically wired to search for a logical reason why it happened.
So when we get a bad book, we immediately look for logical reasons why we went down. The other station had a bigger giveaway. The music was a little flat. Our big promotion fell through.
And when we get a good book we know that it was that music test, or the commercial-free hours, or the new morning show team.
All reasonable explanations, all good ways to make sense of the ratings.
Yet despite the celebrations and congratulations for good books, and all the introspection and hand-wringing for bad ones, there’s a good chance that the good month had nothing to do with your commercial-free hours, and that bad month had nothing to do with the other station’s bigger giveaway.
Nassim Nicholas Taleb writing in Fooled by Randomness calls it hindsight bias: “Past events will always look less random then they were.”
The biggest uncontrollable random component of your ratings success or failure are the numbers themselves. Randomness abounds in Nielsen Audio ratings.
There may be very reasonable, even obvious reasons to explain why the numbers went up or down, but the reality is that it is more likely that pure chance and the randomness of trends is the reason the station went up or down.
Repeated ratings analyses by Harker Research show that most month to month changes to the ratings are random, reversing direction again and again, making it highly unlikely that month to month changes have anything to do with what’s actually happening in the market.
We all know how radio ratings work: The numbers are based on the listening patterns of a small group of listeners. The listening patterns of this small group are then given different weights using complicated algorithms so that the small group’s listening can be projected across the entire population.
The process is fraught with assumptions, complexities, and computational challenges that create uncertainties that compromise the accuracy as well as consistency of rating estimates.
Take a look at the graph shown. It is the month to month performance of a radio station as measured by PPM. The station is in a medium sized market, but it has the benefit of being one of the top rated stations in the demo, so the station has a fair number of meters.
The graph shows the month to month percentage change over three years. Bars above the middle line indicate the station increased its share that month, while bars below the middle line indicate the station lost share that month. (No bar indicates the station was flat.)
Over three years the station reversed direction from one month to the next a total of 17 months. That means 44% of the time the station moved in one direction one month, and then the very next month moved in the opposite direction.
Over the three year period, the longest string of months headed in the same direction was three, and it only happened four times in 39 months.
During this time there were quite a few wide swings, some over 20%, but invariably large swings were followed by a series of small swings in the opposite direction.
In statistical terms, this is called reversion to the mean. There is some “true” market share for the station, but because of randomness (statistical noise) in the ratings, the monthlies keep missing the true share, sometimes estimating too high and other times estimating too low.
This is one station in one market, but we have found similar patterns in both PPM and diary markets, with both successful and unsuccessful stations. If you doubt it, it only takes a few minutes to calculate your own station’s performance.
Arbitron (as well as Nielsen) has long acknowledged the inevitable randomness in ratings. Their literature includes many comments on the inherent uncertainty of the numbers, but too often when the trends come out, those qualifiers and cautionary notes are forgotten.
Despite clear evidence that monthly trends reverse almost half the time, too many stations see every down month as ominous, every up-tick as evidence of emerging success.
Too many stations start changing things based on one or two months of ratings data. Songs are pulled, rotations reworked, even staff changes made, all based on a trend that is likely to reverse in a month or two.
In the example shown here, in three years the station wasn’t able to put together four months that all moved in the same direction. Not once.
Our research shows that in large markets, trends shorter than six months are likely to reverse (revert to the mean). In smaller markets it can take nine months or more for real trends to emerge from the noise, and even then reversals are more the rule than the exception.
It’s bad enough that too many stations make programming decisions based on monthly trends. We know that the time frame is too short to have reliable information. Then what about daily, hourly, or even minute by minute?
We know that ratings reliability and sample size go hand in hand. More meters equals more reliable data. Averaging months means more reliable data. And the opposite is true.
If monthly data are unreliable, that means weekly data are even less reliable, and daily even more so. If we can’t believe monthlies, we certainly can’t believe weeklies.
Yet today we have programmers making decisions based on minute by minute data!
Those who believe in the value of analyzing minute by minute data point to the apparent relationship between the number of meters “tuned” to the station and some other metric.
The problem is that programmers along with the rest of humankind suffer from an affliction called Apophenia. Apophenia is the tendency that we all have to see patterns in randomness.
If we stare at a random set of data long enough (say a few PPM meters coming or going), and believe the data can give us some insight, we will find a pattern, even if the comings and goings are in truth random.
Apophenia causes smart programmers to make bad decisions by reacting to random patterns in minute by minute ratings.
Leonard Mlodinow in his book The Drunkard’s Walk: How Randomness Rules Our Lives writes:
The human mind is built to identify for each event a definite cause and can therefore have a hard time accepting the influence of unrelated or random factors. The first step (in getting over this tendency) is to realize that success or failure sometimes arises neither from great skill nor from great incompetence.
Mlodinow points out that sports writers, team owners, players, and their fans often believe that firing a coach can turn a losing sports team around.
You probably believe it too.
That’s why good PDs get fired, and bad ones get bonuses.
Yet research shows that replacing a coach doesn’t turn a struggling team around, and the loss of a coach doesn’t hurt a winning team. Same in radio.
Good ratings depend on good radio, but there’s enough noise and randomness in the ratings (both PPM and diary) that it may not always seem to be the case.
Don’t obsess over the ratings. Don’t react to every trend like it’s real, and don’t allow a bad month (or two or three) to cause you to change direction if you are convinced you are on the right track.
And most of all, don’t believe minute by minute. We all suffer from Apophenia, but the best Program Directors manage to overcome it when making programming decisions.
Finally, try to get your boss and the sales department to read this. It can’t hurt.
Richard Harker is President of Harker Research, a company providing a wide range of research services to radio stations in North America and Europe. Twenty-years of research experience combined with Richard’s 15 years as a programmer and general manager helps Harker Research provide practical actionable solutions to ratings problems. Visit www.harkerresearch or contact Richard at (919) 954-8300
