How to set up jury judging

How to set up and run a judged contest on Raflia, and what your judges experience.
For an overview of what jury selection is and when to use it, see the jury selection feature page.

Before you start

Jury selection lets a panel of people decide your winners instead of chance. It’s available on every plan.

Have these ready:

• Your entries, collected and valid. Judging runs on valid entries only, so clean up duplicates and rejected entries first.
• Your judges. Team members need an account. External judges don’t.
• Your judging method, and for rubric scoring, the criteria you’ll score against.
• How many winners you’re picking.

One thing to settle before anything else: the judging method becomes permanent once a judge submits, and changing it before that throws away any judging already done. More on that next.

Choosing a judging method

Raflia offers four ways for a panel to decide, set under Judging method in the tool’s settings. They differ in what you ask each judge to do.

Rubric scoring is the most defensible and the most work, for you and your judges. Every judge assesses every entry, so the comparison is complete.

Voting is the lightest. Judges don’t have to look at everything in the same depth, which makes it practical when the pool is large.

Ranked sits between them. A judge’s first pick counts for more than their fifth, so order carries information that a straight vote throws away. It suits a shortlist better than a big pool, because ranking fifty things properly is a lot to ask.

Manual keeps the audit trail without a panel. You pick, and the result still gets a public record like any other Raflia tool.

🔒 The method locks the moment a judge submits a ballot. After that it’s fixed for the tool, and changing it means starting a new one.

⚠️ Before that lock, switching the method discards everything already entered. Any scores, votes or rankings judges have drafted are thrown away, not converted. So if judges have begun work and you change your mind, you’re asking them to start again. Decide before you invite anyone.

Setting up a rubric

Skip this section if you’re using voting, ranked or manual.

A rubric is the set of criteria judges score against, each with a weight. You get one rubric per jury tool.

For each criterion, set:

• A name judges will see, like “Composition” or “Fits the brief”
• A weight, controlling how much it counts toward the total
• A maximum score, 10 by default

Weights don’t have to add up to anything in particular. Raflia normalizes them, so a criterion weighted 3 against one weighted 1 counts three times as much regardless of the other numbers.

Three to five criteria works best. Fewer and you’re ranking on overall impression. More and judges start scoring the same quality twice under different names, which quietly double-weights it. There’s more on writing criteria that hold up in how to judge a contest fairly.

Choosing what judges see

You control which entry fields appear on the judging screen, picked from your form’s fields.

This is how you run a blind contest. Untick the entrant’s name and the judging screen shows Entry #N instead, with only the fields you chose. Tick it back on and the name returns.

By default judges see the entrant’s name and email. For anything where bias is a risk, turn the name off before inviting judges, not after they’ve started.

⚠️ If your entries came from a CSV import rather than a form, there are no form answers to show, so judges see the entrant’s name and email only. Judging still works, but there’s nothing of the submission itself on screen.

Inviting judges

Two kinds of judge, and you can mix them.

Team members already have accounts. Add them as judges and they’ll see the contest when they sign in.

External judges don’t need an account. Raflia emails each one a personal link that signs them straight into their own judging screen. Use this for outside experts, clients or partners.

Each external judge’s link is theirs alone. If someone loses it, send a reminder rather than creating a second judge, because the reminder keeps their original link working.

While judging runs

The judging dashboard shows progress per judge, including how many have submitted. Two things are worth watching.

Draft versus submitted. A judge can edit freely while drafting. When they submit, their ballot locks and can’t be changed. Only submitted judges count toward the ranking, so a judge who scored everything but never pressed submit contributes nothing.

The submitted count before you finalize. The dashboard shows how many of your judges have submitted and warns you if you’re about to finalize while some haven’t. That warning is the one worth reading. Finalizing early silently excludes anyone still drafting.

Judges can also leave a comment on individual entries. If peer visibility is on, a judge sees nobody else’s comments until they’ve submitted their own ballot, which stops the first opinion from anchoring everyone else’s.

Finalizing and picking winners

When judging is done, Raflia ranks the entries using the submitted ballots and shows you the result.

The ranking is a recommendation, not a decision. You can reorder it, change how many winners you take, or pick an entry that ranked lower. Whatever you choose is recorded alongside what the calculation produced, so the audit trail shows both: what was recommended and what you finalized.

Ties are flagged. Raflia breaks them deterministically for display, by entry submission time, but a flagged tie is telling you the scores genuinely didn’t separate those entries. Decide it yourself rather than letting the sort order decide.

Rubric scoring has an aggregation option. The default averages each entry’s scores across submitted judges. Trimmed mean drops the single highest and lowest score per entry first, which limits how far one outlier judge can move a result. It needs at least three submitted judges; below that it falls back to the plain average. Switching between them re-ranks instantly without changing anyone’s scores.

Once you finalize, winners are created the same way as every other Raflia tool, with the same lifecycle, substitutes, public results page and cross-tool exclusion.

After the contest

The public results page shows the winners, the judging method and a timestamp, so participants can check the result without asking you.

Winner management works as it does everywhere else: notify, confirm, decline, and pull in a substitute if someone drops out.

Cross-tool exclusion sees jury winners automatically. If you run another draw in the same project with “exclude previous winners” on, the people who won here are already out.

What judges experience

A team member signs in and finds the contest waiting.

An external judge gets an email with a personal link. No account, no password. The link opens their judging screen directly.

Either way they see the entries with the fields you chose, a screen built for your method, and an optional comment box per entry. They can save progress and come back. When they’re ready they submit, which locks their ballot.

Large contests paginate, so a judge works through pages rather than one long list.

Frequently asked questions

Can I change the judging method after judges start? Once any judge has submitted a ballot, no. Before that you can still change it, but doing so discards every score, vote and ranking already entered, so judges who have started would begin again.

Do all judges have to score everything? With rubric scoring, yes, that’s what the method expects. With voting and ranked, no, a judge submits their picks or their order and that’s complete. Manual has no judges at all.

How does Borda scoring work? Each judge ranks their top K entries with 1 as best. Higher placements are worth more points, and the points are added up across every judge who submitted. An entry ranked second by several judges can beat one ranked first by a single judge.

What happens if a judge never submits? Their work doesn’t count. Only submitted ballots feed the ranking. The dashboard shows who’s outstanding, and you’ll be warned before finalizing without them.

Can I judge my own contest? Yes. Add yourself as a judge like anyone else, or use manual selection if a panel isn’t needed.

Can judges see each other’s scores? Not before submitting their own. That’s deliberate, so the first judge in doesn’t anchor everyone else.

Do external judges need an account? No. They get a personal link by email and judge from it directly.

Can I run jury selection and a random draw in the same project? Yes, and winner exclusion carries between them. Someone who won the judged contest can be kept out of the draw automatically.

How many entries can a jury contest handle? The judging screen paginates, so large pools are workable. Entry limits come from your plan rather than the tool.

Quick reference

Something not covered here? Get in touch and we’ll help.