Thank you for this post! I am a Data Scientist but I’m also interested in the challenges around successfully integrating data science into a business, so it was really valuable to hear your perspective on this. In my experience, if the patterns can be explained in plain English, there may still be some value in using a machine learning model instead of a manual rule-based approach.

A rule-based model can overfit — When we try to manually think of and test for possible patterns in a data set, it is easy to want to make small adjustments due to observed edge cases. This results in an algorithm that does not work well on new data.

A machine learning model will automate the process of discovering patterns in your data — Even if the patterns are simple, using a machine learning algorithm will automate the process of learning them, and also automate the process of using those patterns to make predictions. If done manually, identifying and testing patterns in a dataset can be a very labor-intensive process.

I typically like to start with a rule-based model to test out the idea. That model is then used as a baseline to compare against the machine learning model in the next phase of the project.