Prompt to Build a Scoring Rubric Before You Judge Anything
Slate Hollow Film Festival received 240 short films for its November edition and has 18 slots to fill. Five jurors, eight days, three categories, and a scoring sheet that last year used a bare 1-to-5 column labelled “overall”. The jury spent ninety minutes arguing about one six-minute film because two people were scoring craft and one was scoring whether she would watch it again.
Director Nnamdi Okafor did not need better films. He needed a scale that existed before the arguments started. The prompt below turns a one-paragraph brief into weighted criteria with behavioural anchors, so five jurors can disagree about a film without disagreeing about what a 3 means.
What Went Wrong in Last Year’s Jury Room
The 2025 edition drew 198 submissions and the jury watched everything in a single shared room, scoring on paper. Four problems showed up in the post-festival review, and all four were scale problems rather than taste problems.
- No shared zero. One juror never scored below 3 because “everyone who finished a film deserves encouragement”. Another used the full range and averaged 2.4. The two scores were arithmetically comparable and practically meaningless.
- Hidden criteria. Programming notes written after the fact mentioned “distinct voice” eleven times, but no criterion on the sheet measured it.
- Category bleed. Animated shorts were judged against documentary pacing because the sheet had one column set for all three categories.
- Recency drift. The last twelve films screened on day eight received 0.6 points more on average than films screened on day one, a gap nobody noticed until the scores were plotted.
The fix is not a longer sheet. It is a sheet built from explicit priorities, with anchors specific enough that a juror can point at a sentence and say “this is a 4, not a 5”.
The Rubric Builder Prompt
Paste the block as-is and fill the bracketed fields. The prompt forces weights to add to 100 and forces every criterion to carry a negative definition, which is the part most rubric prompts skip and the part that stops two jurors from scoring different qualities under the same label.
You are an assessment designer building a scoring rubric for a film festival jury.
Work like a methodologist, not a critic. Do not evaluate any film.
Context:
- Festival: Slate Hollow Film Festival, November edition.
- Volume: 240 short-film submissions, 18 programming slots.
- Jury: five members, mixed backgrounds (one programmer, one cinematographer,
one critic, two past attendees). They score independently, then meet once.
- Categories: narrative, documentary, animation. Maximum runtime 25 minutes.
- Stated priorities, in order: (1) a distinctive directorial voice, (2) technical
control that serves the story, (3) fit with the festival's rural-venue programme,
(4) feasibility of screening from a DCP.
- Time budget: eight days of viewing, roughly 30 films per juror per day.
Produce:
1. Criteria. Between five and six criteria, no more. For each one give:
a. A short name of three words or fewer.
b. A one-sentence definition of what it measures.
c. A "not this" line naming the neighbouring quality jurors confuse it with.
d. Anchors at scores 1, 3 and 5. Each anchor must describe observable
evidence in the film, not a feeling. One sentence each.
e. The evidence a juror must be able to cite in the comment field to justify
a score of 4 or above.
2. Weights. Assign integer weights that sum to exactly 100. The weights must
reflect the stated priorities, and you must explain in one line why the
heaviest criterion is heaviest.
3. Category adjustments. Name at most two criteria whose anchors change by
category, and give the replacement anchor text for each affected category.
4. Scoring mechanics. Specify the scale (recommend 1 to 5 integers), what to do
with a film a juror cannot finish, and how to treat a score submitted after
the juror has seen 25 films in one day.
5. Tie-break. Give a deterministic tie-break for films within 0.3 points of each
other that does not reward runtime or submission order.
6. One-page juror briefing. Under 250 words, second person, telling a juror how
to score without hedging.
Constraints:
- No criterion may overlap another by more than a sentence. Say so if two
candidates collapse into one.
- Banned criterion names: "overall impression", "watchability", "production value".
- Every anchor must be checkable by a second juror watching the same film.
- Do not invent statistics about the submissions. If a number is missing, write
[NEED: description of the number].Why the negative definition earns its space: “Technical control” and “production value” collapse into each other without it. The “not this” line is what lets a juror say “the sound design is clean but the camera is doing nothing for the story” and have that land somewhere specific on the sheet.
Fields to Replace Before You Run It
Everything outside the brackets is structural. The bracketed values decide whether the rubric fits your festival or describes someone else’s.
- Volume and slots. 240 and 18 set the selectivity. If you program 60 films from 900, say so; the anchors get stricter and the tie-break matters less.
- Jury composition. The five-person mix above assumes one professional and four informed amateurs. A jury of five programmers needs a criterion about programming risk instead of audience accessibility.
- Stated priorities. This is the load-bearing input. Write them in the order you would defend to a rejected filmmaker, not the order that sounds noble.
- Category list. Add “experimental” if you accept it. Each category you add costs one anchor rewrite per affected criterion.
- Time budget. Thirty films a day is realistic for shorts under 15 minutes. At 25 minutes maximum, drop to 18 and the fatigue rule in section 4 changes.
- Banned names. Keep the three listed and add the words your jury used last year as a proxy for “I liked it”.
Reading the Rubric You Get Back
A good return for Slate Hollow looks like six criteria with weights such as 24 for directorial voice, 20 for story construction, 18 for technical control, 14 for programme fit, 14 for distinctiveness within its category, and 10 for screening feasibility. The numbers are less important than the shape: one criterion clearly dominates, two are close, and the smallest is a logistics gate rather than an artistic judgement.
- Anchors you can argue with. A 3 for voice might read “the film has a consistent visual idea that could belong to another director”. A 5 reads “at least two choices in framing or sound would be wrong in any other film”. Both are checkable.
- The “not this” lines. Expect one that separates voice from novelty, and one that separates technical control from budget. If a line reads “not the same as quality”, send it back.
- The tie-break. A usable one names a single criterion and a single comparison, for example the higher score on directorial voice, then the higher score on story. A tie-break that says “jury discussion” is not a tie-break.
- Missing numbers. The [NEED: …] markers tell you what the model could not know. For Slate Hollow the first return asked for the share of first-time directors, which turned out to be 61 percent and changed the voice anchors.
Score three films you already agree on before the jury touches the sheet. If the rubric ranks those three in the order your programming team would, it is ready. If it does not, the weights are wrong, not the films.
Three Ways to Bend the Rubric
The builder prompt generalises, but the calibration pass below is the one most juries skip and most need. It turns the rubric into a test rather than a document.
Here is the rubric we adopted: [PASTE RUBRIC].
Here are three films the programming team already ranks in a known order:
1. [FILM A] - expected top
2. [FILM B] - expected middle
3. [FILM C] - expected bottom
For each film, score every criterion using the published anchors and show the
arithmetic. Then answer:
- Does the total order match the expected order? If not, which criterion caused
the inversion?
- Which criterion produced the widest spread between the three films, and is that
spread justified by the anchors or by my description of the films?
- Rewrite the one anchor that is doing the least work, and show the before and
after text.- Weighted to a single priority. If your festival’s identity rests on one thing, push that weight to 40 and cut two criteria. A rubric with four criteria and one dominant weight is easier to defend publicly than six balanced ones.
- Two-stage scoring. Ask for a pass/fail gate first (screening feasibility, runtime, rights) and score only films that clear it. On 240 submissions this removes roughly 30 films before the jury spends attention on them.
- Per-juror normalisation. Add a line asking for a z-score conversion per juror, then compare normalised totals. This is the direct fix for the juror who never scores below 3, and it costs one line in the mechanics section.
Write the rubric before the first screening, not after the first argument. The cost is about forty minutes and one careful pass over your priorities; the return is a jury meeting that argues about films instead of about adjectives. If you want the scoring half of the loop as well, the calibration pattern in our prompt evaluation workflow runs the same logic on model outputs, and the output contract in our structured output format guide keeps the score sheet machine-readable so you can sort 240 rows without retyping them.
Sources & Notes
Slate Hollow Film Festival, Nnamdi Okafor, the 240 submissions, and all scores in this article are illustrative. The anchor structure follows standard behaviourally anchored rating scale practice, and the normalisation suggestion uses a per-rater z-score, which assumes at least 30 scored items per juror to be stable. Reviewed on 2026-11-02. Model behaviour changes between versions, so re-run the calibration pass whenever you change the underlying model or the weights.
