Adikteev’s user churn prediction: tests and results

DIY or outsourced: The best solution for your business

Unfortunately, not everyone has the resources to build, tune, and run a churn prediction model in-house. If going this route, begin with having comprehensive data, tweaking the model, and then continuously working your way through the changes in the marketplace and in your product. Otherwise, should you need full-on assistance in handling a project this immense, we offer this service at Adikteev.

How Adikteev measures the accuracy of predictions

To evaluate churn prediction performance, Adikteev uses the AUC ROC metric which stands for Area under the ROC (Receiver Operating Characteristic) curve. It’s a statistical measure for evaluating machine learning model predictions using a probabilistic framework. 

AUC ROC, therefore, measures the quality of ranking. Briefly speaking, it grades the model between zero and one. A grade of one means that the model is 100% correct and 0.5 is purely random. These predictions will get better over time upon giving it more and more data to work with up until the model reaches the necessary data depth and features to achieve significant predictions.

In light of these considerations, at Adikteev, we developed and ran a churn prediction algorithm with multiple apps in different verticals. The AUC ROC score for the model can reach over 0.9 (or 90%), which is a strong result for prediction capabilities.

We have tested the viability of this 0.9 result through back testing. On production day or day D, we train our model over the time frames D-3 months to D-1 day. While we’ve already made predictions on day D, we are still not sure if they are reliable at this point because we have to wait and observe what happens next. A month later on D+1 month, we can back test our model. Here, we will compare the predictions made on day D with what has transpired between D and D+1 month. The goal is to evaluate the performance of the predictions made on day D. We have also automated this backtesting method to monitor actual predictions delivered to our clients. Through this back testing, we have favorably obtained the same results of 0.9 and above. 

Evaluating Adikteev’s AUC ROC and its accuracy

We used three models to evaluate our output of AUC ROC:

1. Random model

• Produces a random value for each user (probability of being alive) between 0 (churner) and 1 (alive).
• It is a simple model.

2. Baseline model

• A rule-based approach that predicts 0 (churner) for a user who hasn’t opened the app during the last 7 days
• It is an intuitive model.

3. Adikteev’s churn model

• A probabilistic model that is trained on specific features, such as user recency, user frequency and age of user… and generates the probability of being active for every user. It also delivers the expected number of opens during an intervention window.

Follow the link below to get the full results from our churn prediction study.

Now what?

Insights gained from user churn prediction can be applied to host different user retention strategies in product, owned media (CRM and cross-promotion), and paid media. For product, churn prediction can recognize an app’s weak points and help your product team develop a solution. For owned media such as CRM, churn prediction can be integrated into it to create user segments. As for cross-promotion, churn prediction scores can be leveraged to bring users to other apps in your ecosystem. Lasty, for paid media campaigns like retargeting, data generated from churn prediction models allows user segmentation and targeting for personalized messaging or creatives.

Adikteev’s user churn prediction technology can be applied to existing marketing strategies to decrease churn and improve retention. Instead of relying on broad-based marketing strategies, user churn prediction allows for a more granular perspective and can target actions aimed for different user behaviors. Get in touch to learn more.