Seven principles of approaching Data Science projects to foster ROI and adoption within the business
According to the NewVantage Partners 2019 summary, 96% of companies are investing in AI and Machine Learning capabilities. However, only 62% of executives are seeing measurable results from these initiatives and 77% of executives report that business adoption remains a challenge.
This prompted me to evaluate the key elements of successful data science projects I have worked on over the last five years. Of course, what constitutes a ‘successful’ project varies; in this blog I will focus on principles to improve ROI from data science initiatives and increase the adoption of insights and predictions within the business.
If you want to deliver an effective solution to the business, start with the business!
The purpose of a data science project should be to solve a business problem, and not ‘find an insight’ from a dataset. And the best place to identify a business problem or opportunity?... the functional experts and decision-makers in the business, not the PhD graduates in the data science team!
Don’t start without knowing the What, Why and Who
Be strict upfront and set yourself up for success before investing time in a technical solution.
What are you trying to solve?
It sounds obvious, but make sure you have a clearly defined problem statement. Arriving at this must be a collaborative effort. The business experts identify the challenge initially and the analytics function act in a consultative manner to boil it down to something which data science can improve.
A statement such as ‘Tell me how to increase profits’ is a data scientist’s nightmare. By comparison, ‘Knowing the likelihood of a customer upgrading and the effect of cash incentives in that decision would allow us to upsell more effectively’ is something which data science is a great use-case for and would increase profits.
Why solve it?
Define the desired outcome and quantify the benefit to solving the problem.
A helpful method here is to estimate the cost of getting it wrong versus the benefit of getting it right. Continuing the above example, this would be the profit of convincing a customer to upgrade by giving them a discount (let’s say £50) versus the loss of giving a discount to a customer that would have upgraded anyway (for example £10). This will enable you to scrutinise potential returns and set an accuracy objective for the model focusing on profit optimisation (since you know the approximate profit to loss ratio of correct and incorrect predictions).
It doesn’t need to be a data science project necessarily, but estimating benefit allows for prioritisation of opportunities, informs project scope and enables data scientists to evaluate further development time versus incremental gain. All of which increase ROI.
Setting clear goals upfront also provides an opportunity to manage the expectation of business users on the outcomes the project can deliver, which should increase user adoption in the long run and help avoid disappointment when unreasonable or lofty ‘assumed’ expectations are not met.
Who is your sponsor?
Having an executive business sponsor who understands the business problem and wants to improve it through insights is vital. Without this, it is difficult to develop a relevant solution, given the time and financial investment it deserves, thus making the project (and subsequent adoption) an uphill battle from the outset!
It’s also important to agree to a process for measuring benefit with your sponsor. This provides a clear method of evaluation within the project and gives them a mechanism of reporting success up the value chain. Having a business sponsor promote the data science use case, especially in monetary terms, has a catalytic effect on inducing adoption within the wider business.
Iterate, Collaborate, Evaluate
Be agile. You need to fail fast, succeed faster.
Hold regular playbacks with the functional experts throughout the development process. Their expertise is indispensable in giving direction to data interrogation and it keeps engagement and understanding high throughout, increasing long term adoption.
Build iteratively and continuously evaluate the model’s effectiveness, both in statistical terms as well as using historic data (train and test sets) to compare predictive accuracy against the target outcome defined upfront.
Evaluation against historic data only gets you so far. There is no substitute for test and learn through pilot implementations of the model, using things such as A/B tests, to understand real-world effectiveness and attribute ROI. This is where the measurement process agreed with your sponsor is put into practice.
In the spirit of science, there really is no such thing as a ‘failed experiment.’ Any test that yields valid data is a valid test.” - Adam Savage, creator of the Australian-American science entertainment TV programme MythBusters. The results may not always affirm expectations, but they are crucial to guiding the future direction of a solution.
It doesn’t end with model deployment!
Change management is a crucial step in embedding data science solutions within a business and driving long term adoption. But in my experience, it is seldom given the attention it requires.
Hold regular check-ins and training sessions after technical model deployment to ensure business users continue to feel supported in their interpretation and use of the model. For many people, this will be a learning curve and a lack of support will simply result in a lack of adoption. This also provides a forum for feedback and nurtures an environment for collaboration between the business and data science functions.