How to Write a Neurips Paper [2]
So, through some wild rituals, you divined a hypothesis — a short, interesting, and unproven statement. Is it worth your time to pursue it further and whole heartedly dedicate your next few months on it? In this chapter, we’ll cover how to vet your research idea before working on it. Unlike the chaotic process of coming up with an original research idea, one can easily describe the process of vetting an idea (or other professional endeavors), even though implementing this process requires significant discipline and skill.
minimize backtracks
The worst thing that can happen to a researcher is realizing, only after extensive work, that they worked on the wrong thing and have to backtrack. Given the limited lifespan of a graduate student, frequent backtracks can be crippling, professionally and emotionally. An experienced researcher is disciplined to backtrack/pivot sooner instead of “hopefully wishing” their idea can be salvaged when the evidences are against it.
The best time to reject an idea is at the drawing board, before any physical commitments (coding, setting-up recurring meetings). This way, when you do commit to something, you’re working on an actual open research problem without an easy answer to. This way, even if you do end up backtracking, you would have learned something valuable along the way.
the three questions
Vetting your research idea involves giving an honest and disciplined attempt at answering the following questions:
- would I enjoy working on this ?
- will other people appreciate this ?
- can I actually make it happen ?
For each question, I will suggest a few action items that you can perform, which should help the vetting process. Think of it like exercising: the direction is simple, the doing is difficult, but as long as you push yourself and try your best, you will see results.
would I enjoy working on this ?
This is the most important question. In doing original research, it is expected that you take full ownership and responsibility of the project. This will be soul draining if you are stuck working on a project that you do not enjoy. Enjoyment means very different things for different people, there is no right answer: for some, it is the opportunity to contemplate long and hard on an interesting problem, for others, it is the vibe of working with their collaborators. Ask yourself this question often, and try to be truthful with your answer. You owe it to yourself.
will other people appreciate this ?
The next two questions address the supply and demand side of research. If your profession is a researcher, then your “currency” is whether other people can appreciate your work. Therefore, it is crucial that you are skilled in estimating, despite great uncertainty, how your work will be perceived by the broader research community. A great proxy for this appreciation is the peer review process, where a group of (hopefully relevant and competent, definitely grumpy) researchers answer the subjective question “do I like this work” under the guidance of somewhat objective metrics. I will share some relevant reviewing criteria that can be used here.
novelty
The biggest difference between “good work” and “good research work” is novelty. You must provide some information that makes the reviewer think “I have not thought of it”. There are many ways to be novel: is it the research question itself, the formulation, the solution, or the result? You must make this notion of novelty explicit. For instance, you could physically write it down: “This work is novel because ...”
relevancy
The foil of novelty. Your work cannot be appreciated if nobody can relate to it. Do a literature search, and model a stereotypical researcher in the field you wish to publish. Would this stereotypical researcher say, upon seeing your completed work “damn I wish I had done this!”? Or you can try to pitch your research idea to a researcher in the field, and measure this quantity: their_current_enthusiasm - their_baseline_enthusiasm.
memeability
Under the reviewer console this would be labeled as “clarity”. However, clarity is a consequence and not the cause of a well-explained work. The cause is in fact memeability: does the key message “stick” to your reader, and would your reader find it enjoyable to propagate this meme further? A good practice is to ensure your research only has one or two key ideas, and keep the rest simple: think sushi, not curry. Answer the following question: how would you explain your research in a meme image macro and a 140 characters tweet?
can I actually make it happen ?
Most times the answer is yes, given enough time, you can make it happen. A better question is “Can I make it happen faster and better than my competitor?”. You always run some risk of your work being scooped, especially in “hot” fields. I personally prefer to work in more niche fields, the risk is lack of relevancy, the gain is I’m not (typically) in a hurry to publish.
turf war
An experienced researcher would know the key players in their field, and which research is under who’s “turf”. They would know what kind of research is closer to their own turf, and which research is under effective ownership of another research group. This mental map takes years of develop, and is one of the primary asset of a student in having an advisor: it’s the professor’s responsibility to make sure their students don’t get scooped.
If you don’t have a good turf map, you should start making one. Find a research direction that you’re interested in, read as much paper in that area as you can, crawl up stream from the citation section to build a mental map of a group of papers — Which paper is the progenitor that opened a new line of research, and how did it give birth to a range of other works? How did a particular research idea get refined and explained better from one generation to the next? If possible, email one of the authors (you generally get very good response emailing grad-students and not profs) and ask if they can zoom with you. Researchers love sharing their work with people who appreciate their work, because for the most part, nobody gives a shit.
validate ownership
Now that you collected some information of the key players in the field you’d like to publish, and have a turf map, ask yourself: Could I actually own this work in a few months? I’ve had people telling me “I really want to work on architecture search and deep RL” which my response has consistently been “You think DeepMind would let you get away with that?”
If your current research is close to your own turf, then you have validated ownership. Otherwise, you need to adjust your research so that you could potentially own it. Here are some strategies:
- Get collaborators that covers your weakness (I’ll go into this in future chapter), So that some works, initially outside of your turf, become inside of your group’s collective turf.
- Straight up become a good enough expert (in detail next chapter) in the field in order to own this piece of research.
- Avoid turf war by inhabit a niche subfield yourself. The drawback is that, while you definitely own this very particular subfield, it is also a barren wasteland, and you need to work on explaining to your peers why this subfield even matters.
a short story
For the paper in question, I didn’t have to vet it too hard, it was quite easy. Working on program synthesis for many years, I know that the problem of disambiguation — given a specification for a program, f(2)=4, there can be many programs that satisfies the specification, f(x)=x+2, f(x)=4, f(x)=x² — is of great relevancy. And I know that with respect to pragmatics, if I put my head down and studied it hard, and ask the right people to explain to me how it works, I could definitely learn as much of it as I want. A quick literature review tells me that the field of cognitive science (where computational pragmatics lies) and the field of program synthesis have not put the two together systematically: there are some papers here and there that relates the two, but not as a central thesis. In a span of few weeks I probably drew the hats and glasses picture tens of times, telling all my friends I was going to cure the synthesizer of sociopathy with the hats and glasses pragmatics thingamajig. They all thought the idea hilarious , which is a great sign.
see you in the next chapter !
In this chapter, we outlined a set of fairly straight-forward thinking tools and action items one can use to vet a piece of research, before working on it. The key takeaway is that this process is straightforward to describe, but requires dedication and discipline to carry out. As long as you work on it, like going to the gym, you will soon see some sick gains.
In the next chapter, I will cover how to acquire sufficient expertise in a new field in order to carry out your research. I enjoy this process a lot, because it feels like powering up in anime. So stay tuned!
I will be updating this series once every couple weeks, depending on how much free times I have. I will be announcing the new updates on my twitter, so if you can follow it that’d be great :)
thanks for reading and high five!
— evan
chapters list :
- [0] overview : a look behind the scenes
- [1] getting a research idea : series of voodoo rituals
- [2] vetting a research idea : be honest and disciplined
- [3] acquiring knowledge : set a goal first
- [4] building a prototype : <under construction>
