The Art in Science Part II: How to build models for real life problems?

In my previous post I explored some art aspects of scientific work that have to do with esthetics, creativity, and self expression. Another “art” aspect of ML is what kind of model/algorithm should we build for a given real life problem. There is no specific formula/recipe for that and like many things in life getting good at modeling takes time and practice, making it more of an “art” as referred to in Neil Lawrence’s post. Nonetheless, just as in Martial Arts (yes!) there are some basic principles/guidelines we should follow. Some of those I can think about include:

  1. Do not skip steps. When we want to computationally solve a real life problem there are basic steps we have to go through. These can be defined as:
    (a) Thinking how to formulate the problem. This includes the basic entities, the relations between them, the feature space, what we may be able to generalize from etc.
    (b) Deciding what kind of function we should optimize.
    (c) Deciding how to go about performing the optimization (i.e. the learning algorithm).
    (d) Thinking how we can evaluate success/accuracy and what would a “good” model give us.
    True, the above “steps” are highly connected. Still, way too often we jump to (b) and (c), the steps that are generally more technical and the focus of most course work. While we can not do without those I find that in many real life problems step (a) can be 90% of the fight. This is similar to jumping ahead to code something before you fully thought it through (a “sin” we have all committed, and likely more than once…). And just like in coding (or, for that matter, elbow escaping from a mount in BJJ) jumping ahead may work but is more likely to yield subpar results.
  2. Let the data guide you. Eyeballing the data to see where the main issues arise and to get clues for model preferences is priceless. But it’s more than that. Michelangelo is known to have claimed a stone has a statue inside it and the sculptor’s role is simply to discover it. Of course, if we take a rational scientific approach to this statement it makes little sense. But I see it as an insightful comment about the process: By taking this approach we are more likely to shed our pre assumptions, biases, and egos so as to see more clearly what is in front of us and what is required. So let the data (or stone) tell you its story. It’s the data’s story, not yours.
  3. Keep it simple (unless you have a good reason not to). This is basically restating Occam’s razor. As ML researchers, we commonly want to build fancy models with all the bells and whistles we just learned about or thought about. But as researchers in Computational Biology who handle real life problems we should curb that urge. Besides practicality, there is ML theory behind keeping it simple (c.f. chapter 2 in the great Kearns & Vazirani [1]), and even beauty. And, again, simplicity and minimalism are common themes related to esthetics and beauty in the (Martial) Arts.
  4. Consider different lines of attack. Our first solution is not necessarily the optimal one. Especially if things do not work well try to open your mind, take a step back, and think what other approaches may work better.
  5. Iterate. This is also related to the above point. That book you love may look like a beautiful result of ingenuity and creativity, but is nonetheless the end product of many iterations, small insights and little victories. So will the model you develop.

One issue with the above principles is that they are rarely articulated to students. Worse, our current educational system is not geared to teach those (more on that in a future post). Their “artistic nature” also means they are easier to grasp and master through personal instructions and closely watching someone who “has come before us”. That, btw, is the literal translation in Japanese of the word “Sensei”. So if you are a student, go find yourself a good Sensei. Oops, sorry – I meant advisor 😉

[1] An Introduction to Computational Learning Theory – Kearns & Vazirani, 1994.


The Art in Science – Part I

I was reading recently Neil Lawrence’s excellent post on how computer science degrees should be adapted given today’s challenges. Neil nicely points out that “Teaching programming alone is like teaching someone how to write without giving them something to say”, and discusses the need to understand diverse systems – unstructured documents, speech, vision, Bioinformatics etc. Then, one implicit point in Neil’s post caught my attention. Neil states:

Sitting at the core of each of these areas is machine learning: the art of processing and assimilating a range of unstructured data sources into a single model.

I found the choice of words quite interesting: A highly accomplished scientist lays claims about artistic elements in science. Is that really so? is there Art in Science??

Art involves “the expression or application of human creative skill and imagination”. It also relates to a notion of beauty and esthetics. Indeed, after spending some time in the field of ML you start seeing the beauty and creativity in elegant formulations for a specific real life problem as well as the distinct personal signatures of those formulating the solutions. Examples I recall include learning about Shannon’s information theory for the first time, the generalization of EM by Radford & Hinton [1], and the “magic” of boosting followed by its probabilistic interpretation by Friedman Hastie and Tibshirani with subsequent discussions [2]. So, perhaps surprisingly similar to (yes!) Martial Arts, ML requires high technical skills but skills alone are not enough: you need to be creative in order to really push the boundary of what can be achieved and at a certain level you make the techniques your own, expressing your character.

This brings me to another important aspect of “the art in ML” which may have been alluded to in Neil’s post: What kind of models should you build? And more generally – what kind of questions you should be asking as a scientist? I’ll discuss this in my next post. In the meantime, anyone who has a nice personal example about where she/he found beauty and personal expression in ML papers is welcomed to leave it as a comment – it could make for an interesting reading list…

[1] A View of the EM Algorithm that Justifies Incremental, Sparse, and other Variants – Radford & Hinton 1998
[2] Additive logistic regression: A statistical view of boosting – Friedman Hastie and Tibshirani, 2000