As our publishers become increasingly dependent on revenue from digital users, Schibsted is investing in technology and data tools to help us grow our digital subscriber base. One such solution is our Subscription Purchase Prediction Model. We were really grateful to be awarded first prize for “Best Use of Data Analytics” at INMA World Congress 2017 for this project.
We have previously shared how we have used this model to optimize marketing on Facebook, with experiments showing significantly improved conversion rates and ROI. In this post we will focus on how we have used the model to target users on our own sites and how we have improved telemarketing conversion rates by 540%.
How the prediction model works
The model predicts the likelihood of an individual user purchasing a subscription, based on their behaviour on our websites and apps. To do this, we train a machine learning algorithm on a dataset of all logged-in users from a given observation period during which they do not have an active subscription, but some of them do go on to subscribe in the following target period. The algorithm learns the difference in behaviour patterns between those that do not purchase and those that do purchase during the target period.
How we use data from different time periods to create our dataset
The algorithm crunches its way through many variables. Some of the most useful variables which emerged are somewhat obvious, such as recency (how long since we have seen you on our site?), frequency (how many days did we see you during the observation period?) and volume of content consumed. There are also some less obvious signals, such as the proportion of days visited that are weekend days, and the number of devices used to visit our site.
We can test the model on historical data to ensure it performs well, and then we can use that model to make prediction scores for all of our logged-in users today based on their recent behaviour. The output scores can then be used to optimize our sales initiatives across channels, by targeting users who are most likely to purchase a subscription.
The prediction model was originally developed for, and in close collaboration with, Aftenposten but in a way that made it easily scaled to our publishers in Schibsted. We currently have the model running in production on a weekly basis for four of our publishers, with three more planned for roll-out in the coming quarter.
The machine learning pipeline is site-agnostic and scales easily to more publishers
Six times higher conversion in telemarketing
At Aftenposten, we have been carrying out telemarketing to registered users for some time. Previously we’ve done this in a mostly unsegmented fashion, with users chosen randomly to be called. Over time we have seen a stable average conversion rate of 1%. In other words, out of all the users we contact, 1% of them purchase a subscription.
Earlier this year we carried out an experiment to see if targeting users for phone calls based on their Subscription Purchase score from our model would yield a higher conversion rate. There were two groups prepared for the experiment –one group taken randomly per our usual practice, and another group selected from the top 10% of users with highest Subscription Purchase score. Both groups were contacted in the same way and over the same time period. We were pleased to see that the targeted group converted directly from the telemarketing call at a rate of 6%, six times higher than the randomly selected group who converted at the expected baseline rate of 1%.
Since the experiment, we have started targeting our calls based on Subscription Purchase scores on a weekly basis, increasing the volume of users selected and still maintaining a good average conversion rate of 5.4%. The results seen at Aftenposten encouraged us to try a similar approach for another of our newspapers, Faedrelandsvennen, where we have not tried telemarketing before. In this case we have seen a conversion rate of 8.8%.
Doubling click-through rates of in-app ads
Another use case for our Subscription Purchase Prediction Model is to tailor users’ news experience and our in-product communication based on the model. To verify this we carried out an experiment on Bergens Tidende’s (BT’s) mobile app to see if targeting an ad for a BT subscription based on the scores would result in higher engagement and conversion.
Two groups were prepared accordingly: one control group randomly selected, and another group selected because they had high Subscription Purchase scores. The experiment ran for one week, during which both groups of users would see the ad at the top of their news feed when and if they opened the BT app on their mobile.
In that time, the number of subscriptions sold via this channel was too small to draw strong conclusions from, so we also looked at two other metrics: Impression rate and Click-through rate. Impression rates were 6.3% and 28.2% for the random and targeted groups respectively, whilst the click-through rates were 0.7% and 1.3%, with statistical significance at 95% confidence.
Targeting advertisement on BT based on propensity scores gives higher impression and click-through rates
The experiment proved that the users most likely to buy a subscription according to the model were both more likely to open the app to see the ad and more likely to click the ad if they saw it, and has thus demonstrated the potential of using such scores within the product.
Next steps: Experiment, automate, monetize
Going forward we will scale the model to more of our sites, continue to experiment with new ways of using the model and automate successful use cases to improve day-to-day operations. Some ideas we are looking into are how we can use the scores in dynamic paywalls, to determine the prominence of paid vs. free articles in users’ personalized news feeds and as a signal of intent in our audience targeting ads offering.