Thursday, January 13, 2011

Openness in Academic Communities

by Shannon Hedke (I don't know how to invite other authors)
Sorry to mangle your post Shannon, blame html

Sam and Thomas have brought up some interesting points about what I will more generally call "openness". Is this a recent development in the biological community? What is the goal and is there inherent value in openness? I wanted to think a bit about the broader context outside of evolution/ecology/behavior, and throw out some open questions for our community.

First, Thomas discusses the idea that starting research and going to publication is long and difficult, and that we might better serve the community under a different framework. He brings up the on-line only, open access journal <a href="http://www.plosone.org/home.action">PLOS One</a>: submissions are assessed by reviewers on technical merit and not by "importance"; papers can be commented on and rated by users. The problem is, of course, that many of us would like to get jobs, or tenure, and the ranking of the journal can matter more than the quality of the research, since our research output may be evaluated by those not in our field and not able to evaluate quality. The physics community has had an intermediate framework in place for quite some time. "Pre-prints" are apparently made available regularly prior to actual publication in a journal; researchers interested in a particular topic can read and draw their own conclusions about the research, completely independent from any review process, and researchers have access to the information much earlier than they otherwise would. However, the researcher still gets the benefit of publication in whatever hot-shot journal might accept it. Would such an idea work equally well in our community? Is it possible to be an idealist about publishing and still get a job?

But Thomas additionally suggests, perhaps, that the blogosphere -- or some other web-based public forum? -- would be a great place to bring up research ideas and have others respond, perhaps catching methodological or analytical errors prior to analysis, write-up, and submission. But I know researchers who would rather stab themselves in the foot than reveal pre-publication any ideas or conclusions about their research. Are the ideas presented in evolution or ecology somehow more "steal-able" than in physics? e.g., could an idea be more broadly applicable to research in progress by others and thus show up in other's publications without citation, or easier for someone else to take the idea and replicate the experiment quickly? Or is it just hubris to think your research could be that influential? How does this differ from, say, presenting at a conference?

The second idea which came across in Sam's description of this Great Experiment (and is reflected in Thomas's posts) is "openness" in general: being comfortable expressing opinions and thoughts using your own name. I read Sam's introduction to HCJC blog (HicJic blog??) out loud to my husband, Ian, who had just been to the <a href="http://www.mla.org">MLA<
/a> (Modern Language Association) conference in Los Angeles. One of the topics discussed at that conference was increased openness or transparency, in the context of blogging and tweeting using your real identity. The bloggers at <a href="http://chronicle.com/blogs/profhacker/">ProfHacker</a>, most of whom are in English departments, discuss their experiences and thoughts on academia (hosted by the Chronicle of Higher Education), but without the shelter of anonymity. Publishing anonymously was even the subject of a recent blog post in librarian circles (<a href="http://www.inthelibrarywiththeleadpipe.org/2010/x/">In the Library with the Lead Pipe</a>). In other words, academia is trending towards openness, even for those of us who are not well-established.

Thus, I am all for using this space not just for reporting our reviews on recent published research (the J of HCJC!), but on expanding it to include more general discussions on research, publication, academia, etc.

I am so curious to see what direction this forum takes!

Wednesday, January 12, 2011

Sadly, the beautiful formatting in Lyx was lost because it cannot export HTML. Oh well, nobody's perfect.

Postmortem of a flawed research manuscript part deux

Thomas E. Keller

Abstract I describe some of the reasons why I got interested in trying to do a simulation of RNA critters, and when I got tired of working on it. I wrote this manuscript in a few hours; the story tolds winds over a couple years. The original manuscript was sent and tentatively accepted to Evolution, but I think dumping it here is a better place for now. I didn't tell Jim and Claus about this plan, but I think they'll be OK with it. I would have attached the code in all of its byzantine horror (at least most of it is python), but I'm too lazy to figure out how to attach things to blog posts. Note that I would have preferred to just publish this automatically to PLoS one instead, but I'm lazy and their submission guidelines are too detailed for me to bother. Wouldn't it be nice if the default of Lyx was ready to immediately be published, warts and all, on PLoS one?

1 I do not enjoy wet-lab work

2 The coding begins

Now, this was one of the first research projects where I used Python a lot. In retrospect, I was a little ambitious and reinvented the wheel. Many previous researchers have used simulations of RNA in an attempt to figure out how different aspects of RNA secondary structure contribute to evolution. One of Lauren Ancel Meyer's former students, Matt Cowperthwaite, had published several studies that I found very interesting.

Matt provided the source code. Unfortunately for me, it was in C and I was unwilling to tinker more with it. So I wrote a new program that did more or less the same thing. One thing I discovered is that Matt's program in fact relied on a C library called ViennaRNA to estimate how a sequence of RNA folds. How then, do I tell Python to interact with some library? Well, the option I ended up with with was Cython, which provides a relatively easy interface with C code functions.

I tried many small experiments, with input over time from Claus and Jim. Eventually, we settled on a story we felt was interesting, and might be publishable. Then the question became, where should we try to submit it? I found it hard to evaluate these questions. The manuscript at first was absolutely dreadful, because Jim and Claus forced me to try it myself; they only provided comments on clarity and sometimes suggestions for new experiments.

Submitting to a journal

We first submitted the manuscript to Genetics. In time, the reviews came back. The general consensus was that we had clearly described our intent and findings; they did not feel the results and implications sufficiently interest for publishing in Genetics.

I was pretty bummed at this point. I don't remember how long I sat on the manuscript, months at least. Eventually, Jim and Claus prodded me into submitting it to Evolution. The reviews came back; the general consensus again was that we had clearly described our intent and results; the reviews, however, had issues with how we interpreted the data. One of the reviewers signed his name. I suppose he felt concerned that he had serious issues with how we interpreted a part of the results that touched on his research interests. I don't know what the general consensus is of when to sign a peer review. I will say that the section that he picked on was something that wasn't integral to the main points, and felt tacked on. I felt much better deleting the section entirely, because I wasn't sure what the results meant and it made me uncomfortable to make strong claims about their implications.

The final manuscript attempts to address the concerns of the reviewers. In some cases, I didn't feel like doing what they suggested, but I at least acknowledged them. I will also mention that the code is capable of doing many things that are not mentioned in the manuscript, such as different types of recombination. I in fact generated many results on recombination, and showed them to some folks at the Evolution conference in the summer. I never sat down and wrote a manuscript because I dreaded having to frame an introduction and discussion that would be sufficient for publication in a journal that considered merit, or the future implications. I had developed a lot of new software that others might use in the future, but several limitations of simulations made me feel uneasy. Many theoretical studies, I feel, do not adequately acknowledge that a real worry is that some of the assumptions made in the model may differ from real life. Does this difference change our interpretation, and if so, to what extent?

I quite enjoy doing simulations, because you can rapidly generate new data that you know is based on certain assumptions. Then you can think about how different assumptions in the model are appealing for a certain biological question. One strength I enjoyed is that the simulations are based on something found in real life, RNA secondary structure. And indeed, there are organisms whose entire genome is made of RNA. However, some of our assumptions made a little less sense. We assumed that for some reason, RNA is under selection for a specific RNA secondary structure. The honest reason for why I chose this form of selection is that Matt wrote his program that way, and so that is what I did as well. Fortunately, I found out that some types of noncoding RNA have highly conserved structures, and that made me feel a little bit less uneasy about that specific assumption. For viruses whose genome is made of RNA, there are probably other factors that have a much higher selection pressure.

One final question: which type of citation (numbers or author and year) more efficiently conveys information while reading the main text in your opinion? I do not know the answer to this question, but I think it's an interesting question worth thinking and writing about.

Post-mortem of a flawed research manuscript

This blog was going to be a place to discuss that papers we read on Friday afternoons and it still will be. However, Thomas Keller had a different idea. Maybe you think it's good, maybe you think it's bad, maybe you have no idea what that idea is. Read the below and let us know what you think.  Both about Thomas' post-mortem and the direction this blog is taking. If you're worried about what's going to happen to HCJC, don't be. Friday afternoons won't change and my recent email will likely be the last sent to the listserv about this blog.
The file Thomas sent was created using lyx. It's kinda like LaTex, actually a LaTex GUI of sorts. If you don't have the program installed either you won't be able to open the file or it will be a vaguely decipherable text file. If you do have the program it will resemble the this screenshot from my computer. Here's where to get lyx:
http://www.lyx.org/
You'll probably need some kind of Tex as well:
http://www.latex-project.org/


Our Experiment in Open Discourse

I wonder if JBS Haldane could have spoken as many languages if he'd needed to speak to computers too.  I also wonder how many times we've re-invented the wheel.  But mostly I just wonder why so many aspects of this job are so dissatisfying to me.  This blog is the beginning of an experiment.  One in which we try to rectify or at least lay bare many of the frustrations we have with the current process of academic science.  In all likelihood the style and location of the Hill Country Journal Club will change.  So, if you don't like this blog format let us know.  The only thing we are sure of is that this doesn't work for anyone unless we all contribute.  It's intimidating to present something to your peers.  People you know already hold opinions about you as a person and you as a scholar.  People you know aren't going to like what you have to say.  This should be a place to overcome that anxiety.  This forum can be a powerful force for civil discourse.  Please use the comment section on each post to enter into the discussion.  If you'd like to post something send it to either Thomas or myself.  We'll publish it post hast and without edit.  Hopefully the Texas legislature doesn't put us all on the street, but if they do at least we'll have something to take with us.