So You Want to Write a Question Set?

By Max Schindler
This article was adapted from Max Schindler’s original post on the HSQB forums.

Intro

Every year, teams formulate rough ideas of their plans for next season: which tournaments they plan to play, which tournaments they plan to host, and most importantly, which sets will be used for those things. Each summer we see the announcement of a large number of “house-written” sets, meaning that a school (or small set of schools) takes on the task of writing a question set of their very own, using those questions “in-house” rather than obtaining questions from elsewhere. At first glance, a plethora of question sets appears to be a great boon to the quizbowl community — more sets mean that more tournaments can be run in each circuit, and tournament directors have more options to select what’s right for their area.

However, this general picture must be taken with a grain of salt. With an HSAPQ or NAQT set, hosts can trust the quality of what they’re getting. The first use of any house-written question set is not guaranteed to end as well, though, and runs some risk of being a worse experience. (Tournament-directing pro-tip: If the writers of a set are untested, keep regularly in contact with them and ask to see samples of their work. Don’t just trust that they’ll do a great job!) But the blame for running a bad set cannot fall at the feet of the tournament director — it lies squarely with the head editors of that set, and to a lesser degree, the writers who created the subpar questions and marketed them.

And so, those of you announcing your tournaments or seeking to produce a set of your very own, this post is made for you, and the scores who have contemplated being like you. It’s mostly an outpouring of my thoughts, gleaned from my experience head-editing high school sets over the past few years, and the wisdom imparted to me by others, who helped and encouraged me to write down those thoughts here. Ideally, it will be a brief guide to the non-writing aspects of producing a question set (there are plenty of guides on question-writing itself out there, I’m writing about the other stuff) and a cautionary note for those contemplating doing so without having committed already. I don’t pretend to know everything, but I am glad to share my thoughts and hear from people about their own experiences.


So you want to write a question set. You’ve seen the announcements of others on the forums, and now you’ve decided it’s time for you to do so yourself. So what do you do?

Some schools typically write a tournament each year (such as Hunter’s Prison Bowl, IMSA’s IMSANITY, Ladue’s LIST, or Maggie Walker’s GSAC); if you have that kind of institutional continuity available to you, ask your elders. Otherwise:

The first, most important thing to keep in mind at the very beginning is that everybody starts off without much skill at writing. You might be really good at the game of quizbowl itself, but if you’ve never written before, you will probably find writing good-quality questions very difficult. (Fun experiment: write three tossups today, save them in a Word document, then look at them months or years from now when you have a lot more writing experience and see what you think.). According to my friend, who works for a proofreading Australia company, even the best writers’ first questions require extensive polishing from experienced editors — with little-to-no experience, there’s just no way a new writer is equipped to head-edit a well-written tournament.

So how do you get good enough at writing to create a full tournament? Like everything else, expertise at writing can only be accomplished with practice. Start small: write a few of the best tossups you can (spend time on it!) and read them to experienced people on the quizbowl IRC. Submit your questions to people who know what they’re doing (like the feedback program), or even venture into packet-submission college tournaments (if your team is dedicated enough to attend ACF Fall, writing a packet will get you a discount, practice, and I’m sure feedback on the quality of your questions if you ask for it). Work on projects under experienced people if opportunities arise. The list goes on and on; the most important thing is simply to have practice writing questions before you attempt to write a full tournament. (While doing this, be sure you’ve looked at or practiced on “exemplary” and recommended question sets, and start observing them to see what makes them stand out.)

Flash-forward a few months: you’ve written some questions, gotten some feedback, and think you’re ready to tackle a tournament. Now’s when you open that Word document that I mentioned earlier, and see if your initial questions would have been up to snuff! The best thing to do now is to send out some feelers within your team. Is your team up for hosting and writing a tournament on its own? Is your coach cool with the idea? Most importantly, will your teammates support you? Much of LIST I was done by a teammate of mine and I by ourselves over spring break; had I known the rest would write so little, I may not have tried to tackle the project. Make sure your friends are on board and understand what will be demanded of them!

A much easier solution may be to collaborate with another school or two, especially if they have more writing experience than you. Perhaps, after a year of working as the second-in-command under someone more experienced on a set, you’ll be more likely to do a good job in the lead position the year after. Or, you mostly know what you’re doing, but no one on your team is any good at science or fine arts, and another school has a good science and fine arts writer. One must be very careful with collaboration though — I highly recommend in collaborative sets (and, frankly, in all other housewrites) that someone be firmly designated as the in-charge head editor, so someone is setting the schedule and keeping in contact with everyone the whole time. Many of the more problematic sets in recent memory have been collaborations (e.g. the most recent Fall Novice Tournament), and I trace many of the last Fall Novice Tournament’s problems to the presence of not two, not three, but six equal “head editors.” When none of them took responsibility for the set as a whole, many gross oversights went unchecked.

One of the first editorial decisions your writing team must make is figuring out the difficulty you’re shooting for. In your case, this is quite simple! It’s your first time writing, so you’re shooting for regular difficulty. It’s also much better, if you do err slightly, to wind up with an easier set rather than a harder one, so do your best to restrain yourself, and call out your co-writers whenever you think they’re writing overly hard questions.

So now, you’ve made it through all the preliminary preparations. You have some confidence in your writing skills, you’ve gotten advice on the basics of question construction, you’ve got a corps of mostly dedicated people who mostly know what they’re doing to back you up, and you’re ready to write the set in earnest. I assume you can figure out the distribution of your set on your own (with 20/20 packets, you have a little freedom to adjust minor categories here or there, but don’t stray too far from the typical; look at other well-received sets’ distributions as models, and don’t forget to subdistribute categories appropriately). You now need to organize your writing progress. The best way to do this is to set up collaborative spreadsheets that all the writers and editors can access, through a password-protected, shareable service such as Google Drive or Dropbox. When I work, I make an answer selection spreadsheet so the columns correspond to categories, and the rows correspond to the specific packets. Each cell of such a spreadsheet corresponds to an answer-line, and I color-code each cell by how far along the corresponding question is in the process (i.e. white background for nothing, red for written, yellow for edited, green for packeted/playtested/powermarked, or something similar). As long as you’ve established some sort of intuitive system for keeping track, you can do what works for you.

A spreadsheet with tossup answer lines.
A spreadsheet with bonus answer lines.

While the number of columns is easy to figure out (21 or 22 tossups and 21 or 22 bonuses per packet works quite nicely), the number of rows should be selected with greater caution. To put it frankly, some sets do not have enough packets to allow hosts the flexibility of setting up a good schedule for varying numbers of competitive teams. If you want to write a good set, which will allow other hosts the ability to set up a fair, good schedule, your set needs a minimum of 14 packets. The standard used to be lower – some old housewrites in the packet archive had only 10 packets, which in most cases expanded to 12 by the late 2000s, but teams are likely to play out more matches and fair tiebreaker procedures now than they were even two or three years ago. One high school tournament that I attended in Missouri I had enough demand, rooms, buzzers, and moderators to expand beyond 24 teams, but the question set they were using only had 12 packets; this led to a smaller field cap and an extraordinarily contrived schedule which messed up during the day. Had the set had 14 packets, they could have just easily done preliminary brackets of 6 then rebracketed to 8 or vice versa, and the tournament would have been a much nicer experience for all. I don’t say this to impugn the editors of that set specifically, as all the other sets available that day also had too few packets, but rather to provide an example of how this does indeed cause problems. (Hosts should keep this in mind when deciding what to mirror: if you choose to use a set which only produced 12 or fewer packets, your ability to adapt to changing field sizes is lessened.)

At the same time, you don’t want to bite off more than you can chew. This leads me to one of my two main points in this post, which I won’t claim to have come up with myself (though I can’t remember who said it now; sorry!): Writing a quizbowl set is a very easy thing to “barely finish.” It’s really, really tempting to look at twelve complete packets and have no desire to start on a thirteenth. It’s really, really easy to skim a complete set and send it out rather than spend a day proofreading it to catch as many minor errors as possible. Realize from the first time you announce your tournament that writing an acceptable set will be a lot of work. Writing a great set will take even more. If you don’t want to put in the requisite time to turn out a fantastic product — which can take months and requires about 600 individual questions — then head-editing is not the right job for you.

Let’s say, after some months of concerted effort and coordinated work, that you now have 14+ packets of questions written, and have edited them to the best of your ability. If this is the first tournament you’re head-editing, play-test all your questions as they come along. All of them. Read the questions out loud to yourself, and to a group of older people who know what they’re doing (make sure they’re not eligible to play the set), to catch wording errors. But be sure you find someone experienced and get them to look over the set for you. When I say experienced, I don’t mean a college freshman or two who haven’t head-edited anything but worked for Fall Novice once; I mean EXPERIENCED: someone who has edited multiple tournaments and can instantly spot problems in a bad question. A great place to do this is the #quizbowl IRC channel — usually many semi-busy college people are sitting around there, who are almost always happy to be read a packet and provide feedback if you ask them politely. In fact, it might even be a good idea to playtest packets of completed questions as you complete them (if time allows), so that you can nip any remaining problems in the bud. No one will write your set for you, but people will gladly spend a little time helping you find weaknesses in the set if you ask around nicely.

This leads me to my second major point: The best writer/editors are those who listen to the feedback that they’re given. If the people playtesting know what they’re doing, heed their advice! If they tell you clues are bad, replace them. If they tell you a question is not workable, replace it (or figure out why they think it’s unworkable, suggest a fix, and see what they say). It’s really easy to be lazy and not listen to others, but there’s a wealth of people out there who have been right where you are now and are glad to help – why not see what they have to say? Your set can only benefit from it. Even the most grizzled veterans of the writing/editing circuit — titans of the editing world who have been writing for a decade or more — still ask fellow college players to help with their high school questions, or retired players to look over their college questions.

So now, after getting comprehensive feedback on most or all of your questions, you’re (mostly) done. Put the questions in packets, powermark them if you’re using powers and haven’t done so yet (have one person do this near the end, so it’s uniform — I recommend doing so right before play-testing), and send the set out to the first tournaments hosting it. If this is occurring the Thursday or Friday before the first running of the set, you’re doing it too late. Shoot to have your set done and commented upon at least one week early, and don’t let yourself fall behind — question quality drops massively if you have less time to work, and you should allot the last few weeks to the proofreading/playtesting, as they take a non-negligible amount of time.

Congrats, if you did all this, you wrote a solid tournament, and you can look forward to hearing nice things in forum posts. Remind people not to discuss question content publicly until all mirrors are done, but keep listening to specific feedback in private discussion channels, so you can fix remaining problems as they arise. Sometimes issues will simply slip through the cracks, and it’s helpful to adjust them for later mirror sites.

A cautionary note

Writing a tournament is hard. If you choose to write a tournament, you are making a commitment to do your utmost to make it a fine set of questions for everyone who plays them. If you fail to do this, you are doing a disservice to yourself, the rest of your writing team, the people who host mirrors, and the quizbowl community as a whole. Your questions will leave an impression on every team which plays them; for some, it may be the first exposure they have to the notion of good quizbowl, and if you screw up, you turn them away from the game for a long time, if not forever. Even experienced teams and their coaches may be soured by a poorly produced set of questions. Do yourself and everyone else a favor and give it your all for the next few months to do a good job.

I understand that many of you are motivated to write questions because you want to improve or use the experience as a way to get better at playing quizbowl. In a vacuum, there’s nothing wrong with writing freestanding questions for this purpose — from time to time, everyone gets the itch to write quizbowl questions. This does not mean, however, that you are ready to produce a tournament for commercial audiences, unless you take in and internalize the sober reality that many sets today are subpar and fail to meet the acceptable standards detailed above. A question set is a product, and has to exist to serve the community, not just the desire for improvement in the writers. It is far, far better to take on a small part of another person’s project than start a project on your own that you won’t do properly. If all you’re in it for is improvement, then just write the questions and read them to friends.

To summarize further: Remember, ask for advice and take it to heart. Do not be lazy or bite off more than you can chew. Doing a bad job has real repurcussions, so do a good job. Don’t be ashamed to realize that a writing a good question set is in fact a lot of work, and is not for everyone!

For those who understand what they’re volunteering for and choose to undertake the large task of writing a set, I wish you good luck writing! If you have any questions/comments/concerns, I’d be glad to talk about them.