What Makes a “Good” Prophet Model? (1) - Avoid Black Box
If you're working in the valuation team of a life insurance company, chances are you've spent a lot of time with Prophet. It’s that powerful tool we use for things like cash flow projections and calculating reserves. But here's the thing—how often do you really stop to think about whether the Prophet model you're using is a 'GOOD' Prophet model?
I'll be honest, in my first 10 years of working with Prophet, I was just like many of you—more of a "user" than a modeler. I’d input the data, run the numbers, and trust the results without really digging into the mechanics of how the model worked. Sound familiar?
At that time, the Prophet model I was using had thousands of variables—so many that it was impossible to study every single one in detail. Plus, there were tons of customized workarounds and coding tweaks added over time. And then it happened—I came face-to-face with the risk of using a "black box" model when I discovered a simple error buried within those thousands of variables.
The issue? A simple division mistake: the denominator should have been the number of policies, but instead, it was set to 1. The result? A multi-million-dollar error that could have easily been avoided. That’s when I realized how crucial it is to really understand the models we rely on, and not just take them at face value.
In this first post of my "What Made a 'Good' Prophet Model?" series, I’ll dive into how you can identify a BLACK BOX Prophet models, as well as the risks associated to such Prophet models.
Most Importantly, Avoid Black Box
So, how can you tell if your Prophet model has turned into a black box? One way to find out is by asking yourself a couple of key questions:
- Do you really know how your Prophet model is set up?
- Do you have any documentation that explains how it’s designed?
Setting up a Prophet model isn't just about plugging variables into the libraries. In actuarial models, variables are deeply interconnected. For instance, premium income depends on the number of in-force policies, which in turn depends on mortality rates. Mortality rates are influenced by attained age and gender, while attained age is based on entry age and policy duration. It’s like a chain reaction—everything is linked.
If you don’t understand the core modeling philosophy and infrastructure (I refer as the model "backbone") behind your Prophet model, maintaining it effectively becomes almost impossible. Without this understanding, even small changes can become risky.
To make matters worse, if there’s no documentation laying out the design and philosophy behind the model, you’re left making wild guesses whenever you need to modify something. It’s like crossing your fingers and hoping for the best. You make changes, and then... well, you pray that everything will work out fine. But that’s a dangerous game when millions of dollars are potentially at stake.
So, What May Happen?
When your Prophet model becomes a black box, it introduces several significant risks that can impact the accuracy, reliability, and maintainability of your actuarial work. Let’s break down some of the biggest risks:
Risk 1: Difficulty Understanding How Calculations Are Made
One of the most obvious risks is that you may not fully understand how your calculations are being done. When the model is too complex or poorly documented, it becomes almost challenging to follow the logic behind key outputs. This lack of transparency can leave you relying on results you don’t fully grasp, which is never a good position to be in.
In this situation, we must conduct more rigorous model testing to uncover issues that may be hidden in the dark, beyond our line of sight. We also need to ensure that the libraries we use can pass basic technical checks (the validation function) with 100% accuracy — I sometimes refer to this as ensuring that at the very least, the "grammar" of all variables in the library is correct.
Risk 2: Hidden Errors Are Hard to Spot
As I mentioned earlier in the introduction, I learned this risk the hard way. When models are too convoluted, even simple errors can go unnoticed. In my case, a basic division mistake resulted in a multi-million-dollar error, buried within thousands of variables. Identifying such mistakes is much harder when you don’t have a clear understanding of the model's structure.
Risk 3: Guesswork Leads to Unintentional Errors
When we don’t have a solid understanding of how a Prophet model is set up, every change becomes a guessing game. You may need to make adjustments, but without clear documentation or insight into the model’s backbone, these changes can unintentionally create new errors. It’s a bit like tinkering with a car engine when you don’t fully understand how the parts fit together—things can go wrong fast.
Risk 4: Models Become More Complex and Eventually Unmanageable
The more changes you make without understanding the model, the more complex it becomes. Especially when you’re forced to add a lot of workaround coding, the model can quickly spiral into something that’s beyond repair. I’ve seen this happen firsthand from my Prophet modelling projects —Prophet models that have become so tangled with customizations and tweaks that fixing or upgrading them was no longer feasible.
Risk 5: Unpredictable Model Behavior
When a model becomes a black box, you lose the ability to predict how it will behave. In one of my Prophet training sessions, a participant asked me, “What will happen if I select Indicator XYZ for a product in my Prophet model?” I had to answer with no choice, “I was not the one who created the library, how am I able to know what will happens?” That’s a major red flag—if you don’t know what changes will do to your model, you’re working in dangerous territory.
It's Important to Understand Our Prophet Models
Related Posts:
- What Makes a “Good” Prophet Model? (1) - Avoid Black Box
- What Makes a "Good" Prophet Model? (2) - Reduce Redundancy
Comments
Post a Comment