The Art in Science Part II: How to build models for real life problems?

In my previous post I explored some art aspects of scientific work that have to do with esthetics, creativity, and self expression. Another “art” aspect of ML is what kind of model/algorithm should we build for a given real life problem. There is no specific formula/recipe for that and like many things in life getting good at modeling takes time and practice, making it more of an “art” as referred to in Neil Lawrence’s post. Nonetheless, just as in Martial Arts (yes!) there are some basic principles/guidelines we should follow. Some of those I can think about include:

1. Do not skip steps. When we want to computationally solve a real life problem there are basic steps we have to go through. These can be defined as:
   (a) Thinking how to formulate the problem. This includes the basic entities, the relations between them, the feature space, what we may be able to generalize from etc.
   (b) Deciding what kind of function we should optimize.
   (c) Deciding how to go about performing the optimization (i.e. the learning algorithm).
   (d) Thinking how we can evaluate success/accuracy and what would a “good” model give us.
   True, the above “steps” are highly connected. Still, way too often we jump to (b) and (c), the steps that are generally more technical and the focus of most course work. While we can not do without those I find that in many real life problems step (a) can be 90% of the fight. This is similar to jumping ahead to code something before you fully thought it through (a “sin” we have all committed, and likely more than once…). And just like in coding (or, for that matter, elbow escaping from a mount in BJJ) jumping ahead may work but is more likely to yield subpar results.
2. Let the data guide you. Eyeballing the data to see where the main issues arise and to get clues for model preferences is priceless. But it’s more than that. Michelangelo is known to have claimed a stone has a statue inside it and the sculptor’s role is simply to discover it. Of course, if we take a rational scientific approach to this statement it makes little sense. But I see it as an insightful comment about the process: By taking this approach we are more likely to shed our pre assumptions, biases, and egos so as to see more clearly what is in front of us and what is required. So let the data (or stone) tell you its story. It’s the data’s story, not yours.
3. Keep it simple (unless you have a good reason not to). This is basically restating Occam’s razor. As ML researchers, we commonly want to build fancy models with all the bells and whistles we just learned about or thought about. But as researchers in Computational Biology who handle real life problems we should curb that urge. Besides practicality, there is ML theory behind keeping it simple (c.f. chapter 2 in the great Kearns & Vazirani [1]), and even beauty. And, again, simplicity and minimalism are common themes related to esthetics and beauty in the (Martial) Arts.
4. Consider different lines of attack. Our first solution is not necessarily the optimal one. Especially if things do not work well try to open your mind, take a step back, and think what other approaches may work better.
5. Iterate. This is also related to the above point. That book you love may look like a beautiful result of ingenuity and creativity, but is nonetheless the end product of many iterations, small insights and little victories. So will the model you develop.

One issue with the above principles is that they are rarely articulated to students. Worse, our current educational system is not geared to teach those (more on that in a future post). Their “artistic nature” also means they are easier to grasp and master through personal instructions and closely watching someone who “has come before us”. That, btw, is the literal translation in Japanese of the word “Sensei”. So if you are a student, go find yourself a good Sensei. Oops, sorry – I meant advisor 😉

[1] An Introduction to Computational Learning Theory – Kearns & Vazirani, 1994.

Advertisement