Waste papers
To get this blog rolling, I’d like to put forward a modest idea that I’ve been chewing on for a while. Ready? Here it is:
Scientific papers are a waste of time. Therefore, we should stop writing them, and find a better way to communicate our research.
Among the likely readers of Shtetl-Optimized, I can’t imagine that this idea would cause the slightest controversy. But just in case I’m mistaken, let me stress that the idea would have seemed crackbrained to me, too, back when I was young and green.
“Are you kidding?” I would have screamed at my decrepit 24-year-old future self. ”Research papers have been humankind’s great instrument of progress for 300 years! They’re the bulwark that separates Crick from creationists, Chandrasekhar from Chopra, and Wigderson from wackballs like this! Without peer-reviewed papers, how would we verify each others’ claims? How would we establish priority? What would we fill our c.v.’s with?”
So I came to my current view slowly and reluctantly, as a result of spending the past five years struggling (often unsuccessfully) to write up results that I’d proven long before, and that I could easily explain in half an hour to anyone who asked me, and that only a tiny group of experts would ever need to see the details of, and that…
God, those years. They went by so quickly. They should’ve been the best years of my life. I should’ve been saving all of my scarce brain cycles for solving big, meaty problems. And in the meantime, I should’ve been learning how to salsa dance, and exploring San Francisco, and giving talks for elementary school kids, and having dumb affairs that I’d later regret. Instead, what was I doing? Responding to referee reports for this and this and this and this and this.
In an infamous guest post on Lance Fortnow’s blog, I advocated a rather different philosophy. I realize now that I was wrong, and I apologize to any colleagues who were hurt by what I said there. For I now understand that the true time-sucker is neither sailing nor surfing nor clubbing. These things take up only a finite number of hours in any case; once they’re done, they’re done. No, the true enemy of scientific productivity is having to write everything up in such a goddamned painstaking way.
I’ll estimate that I spend at least two months on writing for every week on research. I write, and rewrite, and rewrite. Then I compress to 10 pages for the STOC/FOCS/CCC abstract. Then I revise again for the camera-ready version. Then I decompress the paper for the journal version. Then I improve the results, and end up rewriting the entire paper to incorporate the improvements (which takes much more time than it would to just write up the improved results from scratch). Then, after several years, I get back the referee reports, which (for sound and justifiable reasons, of course) tell me to change all my notation, and redo the proofs of Theorems 6 through 12, and identify exactly which result I’m invoking from [GGLZ94], and make everything more detailed and rigorous. But by this point I’ve forgotten the results and have to re-learn them. And all this for a paper that maybe five people will ever read.
Let’s try some thought experiments. Steve Cook never bothered to write up a journal version of this STOC abstract. Suppose he did; what of it? After he wrote On Computable Numbers in 1936, Turing wrote an erratum in 1937, correcting a few bugs. Did you even know that? Do you care? Would Turing’s place in history be any different had he left the bugs unfixed?
So what’s the solution? Personally, my hope is that the Internet will eventually make not only traditional print journals obsolete (as it already has in some fields), but traditional papers as well. Instead we’ll have permanently-archived “interactive proofs”: discussions that look, more than anything else, like the emails exchanged between coauthors before they start writing up the paper.
“I think I can prove X like so.”
“But how do you handle Y and Z?”
“Well, what about W?”
Assuming the participants are serious researchers, I believe that the ”limit” of such a discussion is every bit as reliable as the paper itself. After all, if absolute rigor is the goal, then you shouldn’t believe the paper either. You should insist that everything be formalized in ZF set theory — and even then, how would you know that what was proved corresponded to the informal statement?
(To be clear, I’m not advocating some sort of woo-woo philosophy of mathematics. I don’t have a philosophy of mathematics — or if I do, then it’s naïve Platonism. All I’m advocating is that we consistently adopt the same standards of convincingness that we already adopt when arguing in front of a blackboard. I leave as an open problem how all of this applies to the “softer” sciences, like biology or string theory.)
But until the post-paper world I’m championing becomes a reality, what should you do? Here’s my advice: write the most informal, sloppy, essayistic, stream-of-consciousness, conversational papers you can possibly get away with. Write as if you were firing off an email to a skeptical but impatient friend. I promise to do my part by reviewing such papers leniently (at least in terms of the presentation), and no longer demanding pointless revisions.
Follow
Comment #1 October 5th, 2005 at 9:28 am
If your papers are going to be read by 10 people, then it’s reasonable that you should spend ten hours of work improving the writing to save each of them one hour of work deciphering the paper.
Often it’s not just the brilliant idea, but the accessible presentation that makes a paper influential. For example, we talk about Polya theory rather than Redfield theory because Redfield’s much older paper was totally unclear.
Of course, rigor isn’t the same thing as clarity. But it’s reasonable that authors should work hard to present their ideas as cleanly as possible.
Comment #2 October 5th, 2005 at 9:38 am
what do you think of the following idea: instead of having just researchers in academia, you also have espousers. what they do is try to write up a research idea in the best possible way. they *do not* try to come up with original research.
with every researcher, comes an espouser. somebody who will write up their results (the ones that take only an hour or so to “prove” at a blackboard).
you might wonder — who would ever want to become an espouser? i think plenty of people would. it’s exciting to be a part of ground-breaking research. perhaps somebody would prefer it to taking a faculty position at a lesser institution.
of course, i’m not sure how we could pay these espousers the well-earned money they deserve, what with funding being so hard to come by.
Comment #3 October 5th, 2005 at 9:46 am
I think papers are not written for your direct collegues who know your great results anyway before you publish them (and therefore don’t read the paper anyway), but rather for people that are not immediately familiar with the whole formalism and so would not understand anything from informal arguments. So maybe we need a new kind of researchers: the ones that write papers for the smartest lazy ones. Actually, this is already happening in the form of books… But the problem is that it seems that the smart guys getting the best results are also the only ones having a vision that drives the field and that cannot be conveyed by those new kind of researchers.
Comment #4 October 5th, 2005 at 9:48 am
Dr. Anonymous scooped me on espousers!
Comment #5 October 5th, 2005 at 9:54 am
As a non-regular reader (I followed a link form Lance Fortnow’s page which I found looking for sports testing info), I’d like to make a comment on this:
I’m a psych major (yeah, yeah, I know… it’s a “soft-science,” “psuedo-science” or whatever other name you can think of), and I have to agree with you, but for slightly different reasons. Having worked in both psychology and medicine (diabetes research) I’ve had to use statistics to prove, well, EVERYTHING. I would argue that it’s the hypothesis testing and inferential statisitcs that are the ruin of most good research. I mean, I’m used to writing in a rigourous way… I can do APA format in my sleep. I don’t like it, but I deal. My pet peeve is the inferences we draw from statistics. They lead us down the path to confirming our own biases/prejudices. Although I guess I should be thankful. They make my own research possible (history & theory of intelligence testing and gender/sex differences)… but I will make one point on writing: there are so many authors who have never taken an English or Philosophy class! They have no idea how to present their opinions in a clearly defined way!
Comment #6 October 5th, 2005 at 10:10 am
Anonymous and Frank: Yes, I love the idea of espousers! (I’ve heard that in some departments, they’re known as “grad students.”)
Comment #7 October 5th, 2005 at 10:13 am
Aram: If you count the time spent procrastinating, I average more like 400 hours for a paper read by 10 people.
Comment #8 October 5th, 2005 at 10:21 am
Miss HT Psych: One of the main arguments for abolishing research papers, which I forgot to make, is that doing so would dramatically decrease the amount of bad writing on this earth.
I know it’s really hard to be honest with statistics — that’s one reason why I only work on theory problems!
I don’t have any “regular readers” yet, but you’re welcome to become one.
Comment #9 October 5th, 2005 at 10:40 am
Thanks! 😀
I also work in only theoretical papers now. Unfortunately, my only publications to date are in diabetes. Oh well. Only a month into my MA, so I think that’s not so bad. But I’ve always had a passion for theory. I think it puts perspective on past research and helps drive future research… but maybe that’s my own bias showing! 😉
Comment #10 October 5th, 2005 at 11:05 am
Of course the problem, Scott, is that you are the sole author of most of your papers. Sure, having single-authored papers is great; you get all the credit, you get to use the pronoun “I” during talks instead of “we,” and you get to recount fondly the quaint little anecdotes that eventually resulted in a theorem without worrying that you’re giving away the fact that your coauthors did very little to help you.
On the flip side, you don’t have more than one capable individual around when it comes time to write. It’s an order of magnitude easier and faster to write with a partner. Especially if they feel guilty about not having proved most of the results.
Comment #11 October 5th, 2005 at 11:07 am
Here’s an alternate kind of direction: To write completely formal, machine-checkable proofs, you don’t need to write everything out in ZF set theory any more than you need to write all programs in machine language. There are plenty of tools for formalizing mathematics which are already prepared to serve as the basis for a standard repository of proofs. A repository like this can take the place of publications in mathematical research areas.
Programs like Coq provide lots of tools for automating proof generation, so you can often write proofs that are even more compact (for ditching the use of natural language) than those that you would write “informally.” There is lots of room for improvement, of course. We need as much effort put into computer proof assistants as has been put into compilers.
Comment #12 October 5th, 2005 at 1:11 pm
Good luck with your blog, or as we say in Hebrew, beshaa tova vederech tzlecha!
As for “espousers”, may I suggest the term “scribes” 🙂 As some people have noted, they already informally exist as students and coauthors, although usually much burden still remains with the originator.
Unfortunately, I believe that formal papers as well as the even more dreary and time consuming grant requests (wait until you have to write some of those) are a necessary “evil”. I can’t think of a more pleasant but still reliable medium to evaluate attained knowledge, especially considering this blog by Lance.
Comment #13 October 5th, 2005 at 1:34 pm
I think the primary “problem” with papers is that there are too many of them which are worthless to read. This unfortunate situation is caused by the “publish or perish” culture in the academia today. One solution to this problem is to give more awards to papers and ask people to list only award winning papers in their CVs/website. Such a move will encourage people to write fewer good papers.
Comment #14 October 5th, 2005 at 1:58 pm
I agree that the endless process of revising a relatively simple result for long succession of different media (tech report (or e-board post), conference submission, conference proceedings version, journal submission, revised journal version) is wasteful and needs some relief. We could limit conference proceedings “papers” to one page–just enough for a detailed abstract of the main results, and including a (mandatory!) link to a draft of a full paper.
Comment #15 October 5th, 2005 at 2:32 pm
Scott I think you are a genius –
I’d guess your main source of wasted time is not writing but refereeing,
and the best way to handle this is to start a blog and announce your rigor-free philosophy there..
Comment #16 October 5th, 2005 at 6:03 pm
I will not get into the general discussion. Istead, I would like address the issue of conference proceedings. These absolutely useless paperweights should be replaced by electronic conference proceedings that will be handed out on CD’s.
Being considerably lighter when put on a CD, papers can then be of almost any length and format, eliminating the need for a special proceedings version. A proceedings CD should also optionally include presentations, and any other information the authors might care to include (perhaps the suggested transcripts of “interactive proof” discussions). It should also contain author supplied links to sites that will eventually include updated versions of papers.
As for the “submission version”, these “short” versions of the paper where proofs and sections are cut out into an “appendix”: These versions are mostly unreadable. How often do you find yourself downloading the normal version from the author’s site instead of reviewing the submission version?
The submission version should be replaced by the most updated version of the paper avaiable in time for submission, and can be accompanied by up to three pages that describe the results, techniques, relevance, and anything else that the authors think makes the paper pulication worthy. Reviewers can then use this abstract to choose the parts of the paper they actually want to read/verify.
Comment #17 October 5th, 2005 at 11:49 pm
I agree in part. My beef with most research papers is that they display reams of algebra when a simple combinatorial argument does the job.
Part of the problem is how we are taught to do math (at least in the US). From the 7th grade until one is nearly in grad school almost all problems are taught to be solved algebraicly. Good old breath first search by hand.
Abstract Algebra should be about beautiful algorithms on sets of permutations. Not symbolic drivel.
Linear Algebra should be about the interesting properties of sets of edge weighted digraphs. Not symbolic drivel.
The problem is that coming up with the clean combinatorial argument usualy takes some work and is worthy of another paper.
Research papers should hand-wave if they do not have a clean proof. Attach the algebraic monstrosity as an appendix, but please, please, don’t include it in the middle of the paper.
-CB
Comment #18 October 6th, 2005 at 5:02 am
Talking about conference proceedings, I’m wondering why conferences don’t video-tape talks and sell them cheaply for those who cannot attend for timing or financial reasons.
I personally would love to buy some of those STOCS, FOCS videos.
Comment #19 October 6th, 2005 at 9:57 am
I agree that the “publish or perish” mentality is responsible for many crappy papers. In psychology there have even been instances of people MAKING UP their results in order to get a publication and use those fake results to substantiate their theory. I did my undergrad at the University of Western Ontario. Is anyone familiar with the name J. Phillipe Rushton? He tinkered with his data in order to try to prove an incredibly racist theory. Furthurmore, it was just bad research (meta-analysis, poorly done).
Comment #20 October 6th, 2005 at 11:13 am
Well, of-course it’s harder to tinker with formal proof than with experimental results. You can try and hand wave, however, best done when hidden by extra notation 🙂
Comment #21 October 6th, 2005 at 12:58 pm
What do you think about the approach at Living Reviews? They also provide their publishing software for anyone to use.
Comment #22 October 6th, 2005 at 3:51 pm
The immediate concern is that I know that I owe you some sections of our joint paper. Despite obvious procrastination, I am struggling to finish an unrelated project. But I hope to have some time for what I owe you “soon”.
If you think that the task of writing rigorous papers is too painstaiking, then one solution is to relax and write papers in a less painstaking way. Make your papers more like your talks. Some authors have a clear and flowing style that leaves inhuman details to the (inhuman?) reader. No salient examples in CS theory or combinatorics come to mind at the moment, but in topology there is John Milnor and in string theory there is Ed Witten. (Although their fields are so hard that an outsider might not know just how amazing they are as expositors.)
Many of my papers are “flowing” in this way in parts. The problem is that often no one else thinks that they are clear :-). Which gets to the heart of the matter. In order to win the right to skip details, you have to be in complete control of your topic. Usually I have just been sore when asked to add details to my papers, but at least twice I have been caught with my pants down, which is to say that the referee found a botched lemma. Both lemmas were correct and inevitable, but my proofs stank. The second time, in particular, I ended up with better mathematics. (The paper in question is quant-ph/0203105 and the missing result is Theorem 4.1, which is a mutual generalization of classical and quantum pigeonhole principles.)
On the other hand, I agree that traditional journal refereeing is broken in some ways and that it can lead authors to spend too much time on painstaking details. My two cents on that are in math.HO/0210144.
Comment #23 October 6th, 2005 at 7:22 pm
I think one thing which should not be overlooked is that a side-effect of spending so much time rewriting a paper often is improving the results.
In some sense this is a question of
internal vs. external motivation…of course we all like to think that we are the best governors of the course of our research. But without a referee report coming back, forcing you to dive back into distant results with a perspective wiped fresh by time,
would you ever do it on your own?
Comment #24 October 7th, 2005 at 2:59 am
Greg: Thanks for your insights! And don’t worry about the paper; I’ve been procrastinating on it too.
Comment #25 October 7th, 2005 at 11:17 am
Hi Scott,
To zeroth order, I cannot disagree with you.
Please remember to attach your posting in every paper review, if you have not gotten far enough to refuse to review papers.
I’ll forward your blog posting to my collaborators, some of them have put up with up to 3 years of delay in what I promise to write.
Comment #26 October 8th, 2005 at 1:14 am
I could not resist. Publishing in reviewed journals IS important. It is a pain, but it beats having to read incorrect results, or badly written, obscure, and convoluted proofs. Those pesky reviewers do make a huge contribution to the environment.
As for the specific examples of “futility”
1)I do not know why Steve Cook did not publish the paper in which he posed the P vs NP problem, and proved that SAT is NP-complete. (He did not use the terminology, and he actually proved that DNF is coNP-complete.) However, most people do not realize today that he also proved (essentially) that 3-SAT, and subgraph isomorphism are also NP-complete, pointed out that 2-SAT is in P, and discussed whether graph isomorphism is NP-complete–because people just do not dig out conference proceedings.
2)Turing’s correction IS important, and logicians are very aware of it. Even computer scientists should care: it shows that even geniuses can make mistakes, and referees not catch them. Turing defined “recursive reals” in the paper–this was one of his objectives–and got it slightly wrong. The technical problem has to do with the fact that 0.999999999999…. is the same real number as 1, and it is easy to see that the problem whether a given Turing machine will output an infinite string of 9s is undecidable. So all mathematicians studying recursive reals DO read the correction.
The idea of an espouser is nice–just as the idea that with each baby there should come a nanny, that changes its diapers, wakes up in the middle of the night as needed, and rocks it back to peaceful sleep, prepares organic home-made meals, and plays Mozart on the piano to improve its growing brain.
Disclaimer: I did not publish all my papers in journals. Several of them I should have.
Comment #27 October 11th, 2005 at 10:49 am
Food for thought: this from Freeman Dyson’s review of a collection of Feynman letters in the NYRB:
After Feynman’s work on the diagrams was done, a year went by before it was published. He was willing and eager to share his ideas in conversation with anyone who would listen, but he found the job of writing a formal paper distasteful and postponed it as long as he could. His seminal paper, “Space-Time Approach to Quantum Electrodynamics,”[4] might never have been written if he had not gone to Pittsburgh to stay for a few days with his friends Bert and Mulaika Corben. While he was in the Corbens’ house, they urged him to sit down and write the paper, and he made all kinds of excuses to avoid doing it. Mulaika, who was a liberated woman with a forceful personality, decided that drastic measures were needed. She was one of the few people who could stand up to Feynman in a contest of wills. She locked him in his room and refused to let him out until the paper was finished.
Comment #28 December 2nd, 2005 at 8:11 am
You’re right about writing papers, but you forgot the even worse. What about a survey (writing a paper about other people’s papers), a research statement (a paper about your papers), a grant proposal (a paper about your future papers), a teaching statement (a paper about how happy you feel about teaching since it leaves you with less time for writing papers) or a cv (a paper that lists all the difficult things you had to go through to finally be able to write papers). Yep you’re right, I’m currently on job hunt.
Comment #29 December 21st, 2006 at 8:31 pm
[…] Two weeks ago, I argued that scientific papers are basically a waste of time. Today I’d like to generalize the results of that earlier post, by explaining why scientific talks are also a waste of time. […]