BJS: Random Academic Musings

The utility of reading papers, attending talks, etc.
I recently read an account (I forget where) of some scientists who were asked to respond to a senator's accusation that 90% of science was useless CV-building crap, and that it was a crime for the government to fund such research. Most of the respondents contested both parts of this accusation, maintaining that most of the research was ultimately useful -- 'adding another brick to the temple of science' and stuff like that. One fellow had a more interesting answer, though: he contested only the second part of the accusation. "Look", he said (and I'm paraphrasing now), "you're right about the first part: 90% of funded science is mostly useless. The other 10%, though, results in extremely surprising and useful advances that often really help people. And here's the thing: you often can't identify that 10% of useful science in advance! Indeed, you often can't tell until the research has long been completed. Given the incredible value of these 10% of useful discoveries, though, the only solution is to fund it all." I think a lot of academic activities work this way: most of the papers you read, the talks you listen to, and the conferences you attend will not be terribly useful, and you shouldn't expect otherwise. In each case, though, there will be some precious minority of papers/talks that will be incredibly interesting and worthwhile -- either directly or via the unanticipated intuition that results from engaging other viewpoints. And such rare cases are valuable enough to make it worth slogging through the rest!

Should researchers be evaluated by their best work or by the mean quality of their work?
I used to think that most researchers did either consistently great work or consistently mediocre work. And though there are many such people, I've come to realize (largely due to reviewing papers) that there can be tremendous variance in the quality of many scientists' work. Indeed, some of the very best scientists who made the most important discoveries are also responsible for some of the silliest and wrongest new ideas. How should such people be evaluated as scientists? Should they be judged only be their best work (overlooking the rest)? By the mean quality of their work? By the median quality of their work? Does it depend on the overall variance?

When should you read the literature in a new field?
Suppose you want to move into a new research area. How much of the that area's literature should you read, and when? On the one hand, you can't ignore the literature in a new field completely, or you'll end up reinventing wheels and annoying the cognoscenti (especially if you are one of the cognoscenti in a neighboring research area). On the other hand, though, reading the literature early on can be even worse: you end up implicitly aligning your thoughts to the types of questions and answers that are currently popular, and you stifle the types of insights you may have naturally had otherwise. Syndey Brenner (a biologist who helped prove the existence of messenger RNA) recently expressed a similar thought in an interview: "My problem is that I know too much to tackle that [scientific problem]. I'm a strong believer that ignorance is important in science. If you know too much, you start seeing why things won't work. That's why it's important to change your field to collect more ignorance." So: after reading just a few papers in a new area -- perhaps a review chapter? -- to give you some sense of the area's scope and level of activity, I think you should then assiduously ignore the rest. Instead, just think about it: what seem to be the most important questions, and what are the various types of potential answers that could be given? Start speculating and perhaps even run a few quick experiments. Then, once you've puzzled things out for yourself, you can comprehensively review the literature. Most of the time this will result in disappointment -- your great ideas will have already appeared in Nature in 1964. Occasionally, though, you'll find that nobody else seems to be thinking about the problem the way you are -- and you might not have had these ideas either, if you had rushed right into the existing literature.

Breadth vs. depth: Playing the devil's advocate to Cajal's 'sword' metaphor
Santiago Ramon y Cajal, writing about the merits of breadth vs. depth in a scientific career, suggests: "The inquisitive mind is like a sword used in battle. If it has one sharp edge we have a cutting weapon; with two edges it will still cut, though less efficiently; but if three or four edges are arranged simultaneously, the effectiveness of each diminishes progressively, until the sword is reduced to a dull bludgeon." Though I'm sympathetic to this sentiment, it makes me a bit uneasy. In science, it seems to me, having a second (or a fifth) 'sharp edge' doesn't actually do much to hamper the others -- so long as they are all regularly exercised. To play the devil's advocate: The inquisitive mind is like a person going into battle. If you only have a single weapon, say a sword, you will be well-prepared to deal with certain specific challenges, but ill-prepared to deal with most of them. All the more so if the sword has only a single cutting edge: this halves your flexibility, and forces you to attack (and perceive) all challenges from a single limited position. Better to go into battle holding a longsword, but to also have a rapier at your side, and perhaps a few daggers and shuriken concealed on your person -- and to know how to use them all.

Why there's no such thing as the ideal scientist
Too often, people argue over the 'right' way to conduct a scientific career. Should you investigate a single issue in depth, or explore many different topics? Should you stay close to the data, or engage in rampant speculation? Etcetera. Such questions seem wrong-headed, because they implicitly assume that the individual scientist is an appropriate 'unit' of evaluation for such a question. It seems clear to me that the entire scientific community is the right unit of evaluation, and that the best state of affairs involves high variance. You want lots of hard-nosed methodologists, and lots of speculators. Lots of specialists, and lots of more globally-oriented researchers (and yes: lots of dilettantes). Lots of old farts who cling to orthodoxy, and lots of younger researchers who rush to explore the latest scientific fads. And everything in between. Even on issues where I take a strong position, I'm still glad to see lots of people actively exploring avenues I deem silly. In general, I think that scientific enterprises should simply have coverage -- in terms of methods, attitudes, and topics -- which will ensure that the truth is discovered even when it's hiding in surprising places. Given that, it makes no sense to ask about the ideal scientist. (Footnote: Of course, just because scientific pluralism is desirable in the community as a whole, it doesn't follow that specific departments, societies, journals, and other smaller groups shouldn't try to gather together a more cohesive group of like-minded people. You don't want a department to stagnate under a single Weltanschauung, but neither do you want there to be no coherence at all.)

A heuristic for evaluating intellectual productivity: the 'ideas per paper' quotient
There are many different kinds of productivity by which scientists and writers may be evaluated. When it comes to intellectual productivity, simply counting the number of publications an author has is obviously a poor measure. A better (though still bad, and certainly more subjective) heuristic might involve calculating the average number of (good) ideas per paper in an author's publications (weighted by the total # of publications). Most researchers, I'd hazard, have I/P quotients that hover somewhere in the neighborhood of 1. Some writers are so intellectually fecund, though, that their I/P quotients are much higher. (Robert Nozick, for instance, had one of the consistently highest I/P quotients of any academic author I know.) A single page of their writing often contains several interesting and novel ideas, each of which other writers would have promoted into a full book apiece. Other researchers seem to have I/P quotients which are well less than 1, as they write paper after interchangable paper on the same idea, tweaking only minor variables along the way. These researchers might be scientifically productive, but I would say that they aren't intellectually productive.

Seeing the intellectual forest from the trees
Some scholars may not be especially good experimentalists, and they may not even come up with new clever ideas that well or regularly -- but they may nevertheless still have a supremely impressive synthetic intellect in terms of how they encompass everything into an cohesive overarching worldview. This skill may often manifest itself in review papers, textbooks, and the like. It may also be expressed in teaching, as suggested in the following notes by Carl Seashore about George Trumbull Ladd (who did not otherwise impress Seashore): "His 'system' rang like a bell. In this respect I have not known his equal.... He had his subject thought out. His lectures were clear, convincing, and fair. He was a man of great erudition and had digested it all into his own system. He had an extraordinary power for balancing evidences and organizing logical arrangements of facts" (Seashore, 1930, p. 249). I used to think that syntheses of this sort were the result of intellectually 'menial' work -- something that you could do (algorithmically, as it were) if you weren't that good at creating and testing new ideas. Now I am less sure; it increasingly seems to me that many scientists miss the intellectual forest for the trees -- not necessarily because they've made a choice (and value judgment?) to focus on the trees, but because they have no real ability to see the forest. Accordingly, creating ones own 'system' in this sense may be a specialized intellectual skill which (1) is not necessarily correlated with 'tree'-based competence, and (2) is independently important to the intellectual enterprise, as a way of characterizing the 'forest', and thereby guiding the development of new trees.

Why I don't care about your hypotheses
Good science, we are often told, should be based on "hypotheses" -- and I have increasingly seen this stance embodied in concrete ways in research primers, instructions for student thesis proposals, etc. This way of enshrining hypotheses as such central players in science, though, also seems increasingly irrelevant or even silly to me. (Yes, hypotheses play a key role in some hybrid statistical models, but I don't think those models actually make sense upon scrutiny.) Rather than require a research program to be motivated in terms of hypotheses, it seems to me to be fine (and indeed desirable) if one simply identifies a question, motivates its importance, and then describes methods that can answer it in a compelling way. Put differently: too often we are encouraged to think in terms such as: "If my hypothesis X is the case, then I should obtain result Y." But this is the opposite of the pattern of inference that is often most useful and appropriate (in part because it implicitly short-circuits discussions of confounds, since some other hypothesis Z may also independently predict result Y for reasons that have nothing to do with X). Instead, what we really want to see in many (perhaps even most?) cases is a compelling reverse inference: "If I obtain result Y, then X must be true, but if I obtain result Q, then P must be true." (Or, in a more wishy-washy way: "If I obtain result Y, then this confers some degree of support for X -- etc.") Note that this is Certified Hypothesis-Free logic, but that this doesn't matter: a well-designed experiment can answer an important question, I just want to know what the answer is, and I see no reason to care about what the experimenter's guess was about what the answer would be!