Designing a consensus process
Disagreement is a normal part of adjudication. Two experienced clinicians reviewing the same case will sometimes reach different conclusions, especially near the edges of an event definition. A well-designed consensus process doesn't try to prevent that. It makes sure every disagreement is resolved the same way, by the right people, with a clear record of how the final answer was reached.
Most processes are built from a few standard pieces. The design work is in choosing how to put them together.
Building block 1: independent primary review
Almost every process starts with two or more reviewers assessing each case on their own, without seeing each other's answers. That independence is what makes agreement meaningful: if two reviewers reach the same conclusion separately, the result is much stronger than if one simply confirmed the other.
Choices to make:
- How many primary reviewers? Two is common and keeps the workload manageable. Three gives a built-in majority but adds a third of the effort on every case, including the many where everyone agrees.
- Who is assigned? Random assignment spreads the work and avoids patterns. Assignment by specialty puts the right expertise on complex event types.
Building block 2: defining agreement
"Agree" needs a precise meaning. If a form has several fields, such as event confirmed, severity, and relatedness, do all of them need to match, or only the primary classification?
Requiring every field to match is strict and sends more cases to the next step. Matching only the fields that affect the analysis keeps things moving, but the charter must name those fields explicitly. Free-text rationale should never be part of the match. It is there to explain a decision, not to be compared word for word.
Building block 3: the tiebreaker
When primary reviewers disagree, a common next step is a tiebreaker: another qualified reviewer who assesses the case and decides between the existing positions or offers their own.
Choices to make:
- Does the tiebreaker see the earlier determinations? Seeing them helps focus on the point of disagreement. Not seeing them keeps the review independent. Either can be justified, but the charter should say which, and the system should enforce it.
- What if the tiebreaker disagrees with both? A three-way split usually means the case needs discussion, which leads to the next block.
Building block 4: full committee discussion
Some cases can't be settled by more individual reviews. They need the committee to talk. Common triggers are:
- no consensus after the tiebreak;
- any reviewer explicitly requesting discussion, for example because a case raises a question about how the charter should be applied;
- case types the charter says always go to committee.
At the meeting, a quorum (the minimum number of members needed for a valid decision, set in the charter) reaches a consensus determination. The committee administrator records that decision as the final outcome for the case, along with the rationale.
Choices to make:
- Quorum size. It should be large enough that decisions reflect the committee, but not so large that meetings are hard to schedule.
- How decisions are reached. Most charters aim for consensus and say what happens if the committee can't reach it, such as a vote or the chair's decision.
- Timing. Will flagged cases wait for the next scheduled meeting, or can urgent ones trigger an ad hoc session?
Putting the pieces together
A widely used design looks like this:
- Two independent primary reviews.
- Agreement on the key fields makes the determination final.
- Disagreement routes the case to a third reviewer as tiebreaker.
- A three-way split, or any request for discussion, sends the case to full committee.
- A quorum decides, and the administrator records the meeting determination.
It is efficient for the many straightforward cases, adds effort only where reviewers actually differ, and always gives difficult cases the committee's full attention.
Make the rules automatic
The weakest point in any consensus process is the hand-off. Someone has to notice that two reviews disagree, assign a tiebreaker, track three-way splits, and put the right cases on the meeting agenda. When that happens by email and spreadsheet, cases stall and steps get skipped.
In Reviewr™, routing follows the rules configured from the charter: agreement makes a determination final, disagreement routes to a tiebreaker, and unresolved cases or discussion requests are escalated to the committee, each step attributed and timestamped. If you are designing the consensus rules for a new study, we'd be glad to help.