ℹ️NOTE: Kaskada is now an open source project! Read the announcement blog.
Predicting customer behavior is not easy, and attempting to alter it predictably is even harder. In order to develop machine learning models that can accurately anticipate and guide customer behavior, it’s critical to understand how your model’s training contexts affect possible interventions and change the final prediction. Can we retain a subscription customer predicted to churn, prevent fraud before it happens, or get non-paying players to spend money so we don’t need to rely only on the less than 5% who usually do?
Subscription Businesses
Model Context refers to the time domain used by a model to make predictions. Therefore the relative time-slices that can be used to select training data for a model. Aligning training datasets with a model’s future use case is critical for obtaining the best performance. For example, when training a model to predict customer retention based on past customer actions, we can choose whether we want to feed the model data about the customer up to different points in time, such as 3 months or 9 months past the start of their subscription. Other activity-dependent times to consider might be 7 days after a new feature release or at the onset of a changing market condition or significant world event. The points in time at which we compute the training data and provide it to the model affect how the model understands the customer and provides insight into what actions can be taken to change their behavior, with the end goal of driving engagement and, ultimately, renewal.
Giving a model information about a customer up to 3 months past their subscription start might allow the model to perform very well at predicting customer retention 3 months in, when there is still enough time to make an impact on enterprise renewals, but new data and features are needed at 9 months or years later. More so, mismatching a model’s training context and its prediction context, i.e. using a model trained at 3 months of adoption to predict at 9 months, leads to underperforming models and often completely erroneous results. That said, smaller contexts can allow for quickly building larger datasets, finding leading indicators of success, faster model training, faster inference, and a potentially finer granularity of events upon which the model can make predictions. Thus, understanding how to choose your contexts and features iteratively with context windows of varying sizes, starting, and ending points, provides the key to training robust models that predict what is important while there is still enough time to change the outcome.
Consider an enterprise SaaS subscription business looking to allocate resources of their Customer Success representatives within a quarter. Each rep has a book of business that renews staggered over the course of the year; they are expecting to win some and to lose some, but nothing should be a surprise. In fact, the purpose of this team is to proactively manage attrition and expansion. Some accounts will start this quarter, be handed off from sales, and need a kickoff call and onboarding within the first 30 days. Others will be looking for their first quarterly review to understand if the onboarding took or if additional training or workshops are needed. Some will hit their mid-year review, 9 month review, negotiating a renewal, or come with an additional budget to upsell. Ideally, an account stays for many years and upgrades to more of your products over time.
At first glance, this example seems simple to select model context: daily, monthly and quarterly. On a daily basis, you might be able to use email drip campaigns and in product interventions to drive behavior. On a monthly basis your rep might want a list of underperforming accounts to call and ask questions. And quarterly you could prioritize bigger interventions such as onsite quarterly reviews or live training. However, a subscription may grant access to products for a certain number of users or amount of metered resources. When should usage actually begin: the start of the month, the kick off call, after cohorts attend a webinar, when each user logs in and completes an onboarding? And are the boundaries of the subscription the only context we care about? Consider the impact of fiscal quarters versus calendar quarters, new product launches, budget cycles and the impact of ensuring there is a line item in that budget to renew and a margin to upsell. What additional contexts may impact budgets: pandemics, stock valuations, hiring, attrition, layoffs, elections, protests, competitor pricing, the list goes on.
In the above example, we can see how changing the model’s context allows for identifying better leading indicators, and more accurate prediction of renewal, sooner, in order to enable the right intervention to change user behavior and increase the likelihood of renewal without wasting resources. However, for digital consumer businesses, with self-service subscription management adjusting the model’s context to real-time predictions is no longer an optional optimization. For every interaction with your product and others your user base is making a decision about their experience with your product. This is why personalized experiences directly impact subscription businesses. Still, in these scenarios, your users want you to succeed, let’s take a few more examples in different scenarios to explore further how model context impacts behavior in adversarial environments and in zero commitment situations.
Detecting Fraud
Fraud occurs in a wide range of industries, including banking, insurance, medical, government, the public sector, and law enforcement. In all of these environments, detecting and rooting out fraud is critical to ensuring a business’ continued viability. ML models are frequently used to identify potential instances of fraud and suggest the best courses of subsequent action.
Consider a scenario in which a bank is looking to implement ML-based models specialized to a variety of tasks such as the detection of money laundering, fraudulent banking claims such as a falsely labeled chargeback, payment and transaction fraud like stolen credit card purchases, take-over fraud in which a nefarious actor seizes control of another person’s account, and account fraud linked to new accounts created with a fake identity.
As compared to the previous example, in which we presume that the end user you’re making a prediction about is who they say they are and wants to use your product, here any purchase or merchant flagged as fraudulent is attempting to hide from you and is actively trying to learn what is giving their behavior away. In this adversarial environment, the model’s context impacts your fake customer and actual customer’s behavior in different ways. As an example, consider an account takeover scenario, where a hacker obtains access to your banking information using the credentials to your existing account. Once they’re logged in, they have access to your previous purchase behavior and account statements as well as the ability to request new credit cards, add authorized users on your existing cards, transfer their outstanding debt to your card, and even instantly transfer your money to other accounts.
In this instance, when should our model be making predictions that the user is in fact who they say they are? Furthermore, how does the model context impact the interventions we can design to prevent fraud, and how do those interventions impact the behavior of our real account holder and potential hackers? Certainly there are a few obvious model contexts: at the time of an attempted login, when opening a new line of credit, or at the point of any account-altering activity such as changing the account password or updating the account address. These moments allow us to re-verify identity, potentially in a way that’s different or more thorough than typical. The behavioral impact of these verifications may result in teaching the hacker what is needed next time and reassuring the real user that they are safe and even predicting the fraudulent activity that is later verified. However, remember we’re in an adversarial environment, we imagine that there exists a possibility that the hacker can get access to the information, an approved device, IP address or spoof a phone number. This leads us to consider a set of model contexts that do not reveal desirable information to fraudulent actors. Such contexts might occur during normal session activity, in real-time, or on clicks, hovers, or highlighting of text. These real-time, in-session model contexts allow for designing interventions that do not expose what was detected.
Mobile Gaming
Similarly, the free mobile gaming industry provides another compelling example of the importance of context. Player engagement, retention, monetization, and the cross-marketing of games within a subscription or label are the primary focuses of most gaming companies. For instance, a data scientist might want to predict whether a player is likely to win a level during gameplay, and based on that information trigger an offer to purchase a power-up at a suitable price. Key contexts in this setting might be after every move, when the user has lost a certain number of lives, when they have a certain amount of time remaining to complete a level, or when their in-game health is running dangerously low. By treating moments of heightened player need as opportunities for purchase enticement, changing of the gameplay difficulty, or providing hints, gaming companies may very well boost player engagement and monetization while improving the overall gameplay experience. However, learning these optimal offer points is far from simple in such dynamic environments and requires iterating over many different model contexts over time. Such large-scale testing is only possible when it is natively supported, and feature engines such as Kaskada are the only tools that fully support this level of experimentation.
The gaming example is different from the previous two for a few reasons. First, you’re changing the game as individuals play, altering the logged activity data and the outcomes that are possible to use for training in the future, complicating how new training data sets need to be generated. Second, there are no commitments; it’s possible that half of your players leave within minutes of gameplay or after months without any built-in need to know when they might be able to come back. Finally, with the changes in the privacy arena, less and less information is known about a player before they begin playing; therefore, relying on in-game or cross-game user and population level activity data becomes critical.
Conclusion
For real-time ML, choosing the appropriate model context to incorporate into your model’s training is an involved process that is made lengthy and cumbersome by traditional feature stores and data science tools. Feature engines such as Kaskada transform this previously ponderous process into one that’s quick, easy, and replicable. Via intuitive, iterative model context selection, Kaskada allows iteration over a multitude of context choices in a fraction of the time required by other systems. In doing so, it enables data scientists and engineers to make optimal decisions when choosing how to train their models, ultimately leading to improved model performance and unmatched business growth.