Thanks, but no thanks

I am posting from the q-bio Summer School, where we are enjoying many discussions about modeling. Several lecturers have advised the junior modelers attending the school, who are mostly graduate students and postdocs, to find an experimental collaborator. I appreciate the advice and the benefits of having an experimental collaborator, but I am usually quite irked by the reasons stated for seeking out opportunities to collaborate with an experimentalist. One reason I've heard many times is that modelers need an experimentalist to explain the biology to them and to help them read papers critically. It certainly could be useful to have a more experienced researcher aid in formulating a model, but that person might as well be a modeler familiar with the relevant biology. I don't subscribe to the idea that modelers need a collaborator to evaluate the soundness of a paper. To suggest so seems insulting to me. Modelers do need to consult experts from time to time to understand the nuances of an unfamiliar experimental technique, for example, but so do experimentalists. I am probably more annoyed by the popular sentiment that a collaborator is essential for getting predictions tested. If I were an experimentalist, I might be insulted by this idea. It's unrealistic to think that experimentalists are lacking for ideas about which experiment to do next. If your prediction is only appealing to your experimental collaborator, then maybe it's not such an interesting prediction? Modelers should be more willing to report their predictions and let the scientific community follow up however they may, partly because it's unlikely that your collaborator is going to be the most qualified experimentalist to test each and every prediction you will ever make. I think the real reason to collaborate with an experimentalist is shared goals and interests and complementary expertise. Finding such a colleague is wonderful, but it shouldn't be forced, and the absence of a collaborator shouldn't be an impediment to progress. If you have a good prediction, you should report it, and if you want to model a system, you should pursue that. Eventually, you will know the system as well as the experimentalists studying it, if not better. After all, it's your role as a modeler to integrate data and insights, to elucidate the logical consequences of accepted understanding and plausible assumptions, and to suggest compelling experiments. Finally, I want to speak to the notion that modelers should do their own experiments. I think that's a good idea if you want to be an experimentalist. If you want to be a modeler, be a modeler.

Pathetic thinking

Modelers with shared biological interests can have varying opinions about what a useful model looks like and the purpose of modeling, or rather the opportunities that exist to perform important work in a particular field.

In a recent commentary, Jeremy Gunawardena [BMC Biol 12: 29 (2014)] argues that models in biology are “accurate descriptions of our pathetic thinking.” He also offers three points of advice for modelers: 1) “ask a question,” 2) “keep it simple,” and 3) “If the model cannot be falsified, it is not telling you anything.” I whole-heartedly agree with these points, which are truisms among modelers; however, in my experience, the advice is followed to an extreme by some researchers, who interpret “ask a question” to mean that every model should be purpose-built to address a specific, narrow question, which ignores opportunities for model reuse, and who interpret “keep it simple” to mean that models should be tractable within the framework of traditional approaches only, ignoring new approaches that ease the task of modeling and expand the scope of what’s feasible. Some extremists seem to even hold the view that the mechanistic details elucidated by biologists are too complex to consider and therefore largely irrelevant for modelers.

Gunawardena may have given these extremists encouragement with his comment, “Including all the biochemical details may reassure biologists but it is a poor way to model.” I acknowledge that simple, abstract models, which may focus on capturing certain limited influences among molecular entities and processes and/or certain limited phenomenology, have been useful, and are likely to continue to be useful for a long time. However, there are certainly many important questions that can be feasibly addressed that do depend on consideration of not “all” of the biochemical details but rather on consideration of more, or even far more, of the biochemical details than usually considered by modelers today.

The messy details would also be important for the development of “standard models,” which do not currently exist in biology. Standard models in other fields, such as the Standard Model of particle physics, drive the activities of whole communities and tend to be detailed, because they consolidate understanding and are useful in large part because they identify the outstanding gaps in understanding. Would standard models benefit biologists?

An affirmative answer is suggested by the fact that there are many complicated cellular regulatory systems that have attracted enduring interest, such as the EGFR signaling network, which has been studied for decades for diverse reasons. A comprehensive, extensively tested, and largely validated model for one of these systems, meaning a standard model, would offer the benefits of such a model (which have been proven in non-biological fields) and would aid modelers by providing a trusted reusable starting point for asking not one question but many questions.

The extremists should take note of the saying attributed to Einstein, "Everything should be as simple as possible, but not simpler."

Gunawardena J (2014). Models in biology: 'accurate descriptions of our pathetic thinking'. BMC biology, 12 (1) PMID: 24886484

Bachman, J., & Sorger, P. (2011). New approaches to modeling complex biochemistry Nature Methods, 8 (2), 130-131 DOI: 10.1038/nmeth0211-130

Chelliah V, Laibe C, & Le Novère N (2013). BioModels Database: a repository of mathematical models of biological processes. Methods in molecular biology, 1021, 189-99 PMID: 23715986

Leave the gun, take the cannoli

In the movie The Godfather, Peter Clemenza says to Rocco Lampone, "Leave the gun, take the cannoli." In this post, I want to argue that modelers should leave the sensitivity analysis and related methods, such as bootstrapping and Bayesian approaches for quantifying uncertainties of parameter estimates and model predictions [see the nice papers from Eydgahi et al. (2013) and Klinke (2009)], and take the non-obvious testable prediction. Why should we prefer a non-obvious testable prediction? First, let me say that the methodology mentioned above is valuable. I have nothing against it. I simply want to argue that these analyses are no substitute for a good, non-obvious, testable prediction. Let's consider the Bayesian methods cited above. These methods allow a modeler to generate confidence bounds on not only parameter estimates but also model predictions. That's great. However, these bounds do not guarantee the outcome of an experiment. The bounds are premised on prior knowledge, the data available, which may be incomplete and/or faulty. The same sort of limitation holds for the results of sensitivity analysis, bootstrapping, etc. I once saw a lecturer in the q-bio Summer School tell his audience that no manuscript about a modeling study should pass through peer review without inclusion of results from a sensitivity analysis. That seems like an extreme point of view to me and one that risks elevating sensitivity analysis in the minds of some to something more than it is, something that validates a model. Models can never be validated. They can only be falsified. (After many attempts to prove a model wrong, a model may however become trusted.) The way to subject a model to falsification (and to make progress in science) is to use it to make an interesting and testable prediction.

What can modeling do for you?

In the blog so far, we've talked often about computational models - how they're made, what can go wrong, and what they could be like in the future. But what exactly are they - and why should anyone (especially biologists) care?

A model is a representation, or imitation, of a system that is difficult to examine directly. Biologists already use models all the time. For example, we'd like to understand biological process in humans, but since most of us can't experiment on humans, model organisms and cell lines are used instead. "Model" also refers a working hypothesis about how a system functions, which is often presented as a cartoon diagram.

A cartoon model for how the Shc1 adaptor protein acts at different stages of signaling.

Zheng et al., Nature 2013

.
Like a cartoon model, a computational model is created based on what a person knows, or on what they hypothesize. The difference is that instead of drawing a picture, which tends to be vague and qualitative, they make concrete and quantitative statements about how molecules interact. They then use this information to create a set of equations or a computer program, which is used to simulate system behavior. In modeling of chemical kinetics, the goal of simulation is often to see how certain outputs (like protein concentration) change over time, or under different conditions. So, like a model organism, a computational model can be used to make new discoveries with potential relevance to real-world questions.

Here are some of the reasons why I think biologists can get excited about what models can offer:

1. A roadmap for pursuing experiments. Why do we do experiments? Often, it's to test a hypothesis. The more complicated the hypothesis, the greater number of experiments one could try, and the more involved each experiment might be. At the same time, even the most diligent of us want to optimize, and do minimum work for maximum information. Models can potentially help identify which tests would be most meaningful for supporting or disproving a hypothesis. 
2. A way to make sense of complicated or conflicting data. Sometimes it turns out that two seemingly contradictory ideas are actually compatible if you think about the quantitative details, which is exactly what models are good for. 
3. Consolidating and testing knowledge about a system. A typical experimental study provides information about one or a few interactions. Models can help us put together multiple pieces of information, like assembling a jigsaw puzzle, to form a more complete picture. Furthermore, by simulating such a model and comparing it to experimental data, interesting discrepancies can sometimes be identified. In other words, we can see whether we have enough puzzle pieces, or if we need to find more through additional experiments. 

At the same time, we need to remember that models won't magically provide the answers to everything. A model that simply recapitulates your expectations could be appealing; however, a model's real worth is in its ability to generate non-obvious, testable predictions. If you're going to start modeling or are thinking about starting a modeling collaboration, try to first learn about about what models can and can't do.

So what does the word "model" mean to you? And what do you think they can be used for?

Logical modeling vs. rule-based modeling

Cell signaling systems have been modeled using logical and rule-based approaches. What's the difference? A rule-based model is similar to a logical model, in that both types of models involve rules. However, the rules are usually rather different in character. In a typical logical model, rules define state transitions of biomolecules, including conditions on these transitions. They have an "if-then" flavor. The rules operate on variables representing states of whole biomolecules, and they define when and how such state variables change their values. Biomolecules in logical models are often characterized by state variables that take one of two values, e.g., 0 or 1. Such variables are introduced to represent "on" and "off" states. More than two states can be considered, but there is a limit to what's tractable, as the reachable state space tends to grow exponentially with the number of possible states. As more states are considered, there are more and more transitions between these states, each of which is usually considered explicitly when specifying a logical model. The behavior of a logical model can sometimes depend on the algorithmic protocol used for changing states in a simulation. This seems undesirable. In a rule-based model, the amount of an activated protein can be continuous or discrete, from 0 copies to all copies of the protein. This is because a rule-based model is based on the principles of chemical kinetics. The state variables implicitly defined by rules capture numbers of biomolecules in particular states and/or complexes. Rules are associated with rate laws, which govern the rates or probabilities of transitions of biomolecular site states, not the state transitions of whole molecules. With a physicochemical foundation, it is relatively easy to capture certain phenomena found in cell signaling systems, such as competition, feedback, and crosstalk. These phenomena are more difficult to capture in a logical model. At least, it seems that way to me. With model-specification languages such as BNGL (http://bionetgen.org), a set of rules can be used to perform different tasks: stochastic or deterministic simulation, via a direct or indirect method. Is it possible to modify BNGL to enable logical modeling? Although typical logical models are different from typical rule-based models, it does not seem that the rules used in the two types of models, although usually different, are necessarily fundamentally different, so my answer is a tentative "yes." What do you think? 

Dismantling the Rube-Goldberg machine

What do this puppy food commercial, an indie music video, and systems biology have in common?

They've all used Rube-Goldberg machines to great effect - devices that execute an elaborate series of steps to accomplish a goal that could have been reached through a (much) simpler process. As the ultimate elevation of means over ends, these machines have become a celebrated expression of ingenuity and humor. However, the presence of such machines in systems biology is perhaps not as obvious, intentional, or entertaining.

So what are the Rube-Goldberg machines of systems biology? In a small but a noticeable fraction of studies, complex models are used to reach conclusions that could be obtained just by looking at a diagram or by giving some thought to the question at hand - assuming that a question is at hand. It seems as though these studies primarily use models to produce plots and equations that reinforce, or embellish, intuitive explanations. However, the true usefulness of models comes into play when we leave the territory of intuition and begin to wonder about factors that can't be resolved by just thinking.

So when and why do we start thinking like Rube-Goldberg engineers, and what impact does it have on the field? A few educated guesses:

• Some models are built without a question in mind. Its creators then search for a question to address, and end up with one that the model's content isn't well-suited to. 
• We're all specialists in something, and we don't always know about all the tools and capabilities that others have developed. As a result, we sometimes try to solve a problem by reinventing the wheel, or by applying a tool that isn't a good fit for the problem, which can lead to all kinds of complications. 
• To some audiences, just the concept of doing simulations seems impressive. As a result, modelers can be drawn into just putting technical skills on display and establishing a mystique around what they do, as opposed to applying their abilities to interesting questions.  
• Obvious predictions may be easier to validate experimentally. 

I don't know if these practices have had a wholly negative impact on modeling efforts in biology - it may have even helped in some respects. But it would not be a bad idea to focus on challenging questions for which simulations are actually needed, and to try to get the most out of the models that we've taken the time and effort to build.

Do modelers have low self esteem?

When was the last time an experimental biologist experienced a manuscript rejection because the work in question didn't include modeling? As a modeler working in biology, I tend to be hesitant about trying to report work that doesn't include new data (from a collaborator), because it doesn't usually go well. Modeling without experimentation tends to be held in low regard, especially among modelers, which is a tragic irony. If physicists of the early 20th century had the same attitude as many of today's modelers and experimental biologists, "Zur Elektrodynamik bewegter Körper" would not have been published without its author first doing experiments to confirm his ideas or him finding an experimental collaborator to generate confirmatory data. I don't think that modelers should take a favorable view of every modeling study they come across but I wonder if we need to be more supportive of each other and allow more room for independence from collaborations with experimentalists. If a modeling study is based on reasonable assumptions and performed with care and it produces at least one non-obvious testable prediction, why should it not be reported immediately? It seems that some of us might be concerned that such reports will be ignored or that such reports are too untrustworthy, given all the complexities and ambiguities. It's true that models need to be tested, but it seems unlikely that someone able to build and analyze a model will also be the best person to test the model or to have a circle of friends that includes this special person. Indeed, I think the requirement to publish with data has led some modelers to produce predictions that are, let's say, "obvious," because this is the type of prediction can be confirmed easily. Let's be rigorous, but to a reasonable standard. Let's also be bold. Many experimental results turn out to be misinterpreted, or plain wrong. It's OK for models to be wrong too. Biological systems are complicated. We need models to guide our study of these systems. Most of the work being done in biology today is being performed without models. Until experimentalists start chiding each other for failing to leverage the powerful reasoning aids that models are, it makes little sense for modelers to criticize each other for work that doesn't include generation of new data.