When team members disagree on a paper's relevance, the team should establish a structured discussion process to align on evaluation criteria and reach consensus through evidence-based reasoning.
Establish Clear Relevance Criteria First
Before debating individual papers, define what "relevant" means for your specific project. Create a simple rubric with 3-4 concrete criteria — for example: addresses our research question directly, uses appropriate methodology, published within target timeframe, or provides critical background context. Having explicit standards reduces subjective disagreement and gives everyone a shared framework.
Use Evidence-Based Discussion
When ratings diverge, ask each member to cite specific passages or findings that informed their score. This shifts the conversation from opinions ("I think it's relevant") to evidence ("Table 3 shows data that directly contradicts our hypothesis"). Members often rate papers differently because they're focusing on different sections or interpreting the abstract differently than the full results.
Implement a Two-Round Rating System
Have members do an initial independent rating, then share scores and reasoning in a group meeting. After discussion, conduct a second round of ratings. Research shows this approach reduces outlier scores by 40-60% as members incorporate new perspectives they initially missed. If scores still diverge significantly after discussion, consider assigning that paper a "conditional" status for later re-evaluation.
Designate a Tiebreaker Protocol
For persistent disagreements, establish rules in advance: the team lead makes final calls, you default to inclusion (easier to exclude later than recover a missed paper), or you require majority vote with a minimum threshold. Document dissenting opinions in your team's shared notes so the reasoning is preserved.
Leverage Collaborative Reading Tools
WisPaper's Scholar QA feature can help resolve factual disputes by letting team members ask specific questions about the paper and receive answers with traceable citations to exact paragraphs — turning subjective impressions into verifiable evidence that everyone can review together. This creates a shared understanding of what the paper actually contains before rating its relevance.
