RoboCup Qualification — Reviewer Guide
The following criteria are used by the reviewers to evaluate the qualification material. Please read these points carefully. If there are any questions, please contact the Organizing Committee via Email: rc-home-oc[at]lists.robocup.org
1. Hard Criteria
All hard criteria are binary pass/fail. A single failure results in automatic rejection without further review. Reviewers do not score hard criteria — they only flag pass or fail.
1.1 Team Description Paper (TDP)
H-P1 — Submission deadline
TDP must be submitted via the official submission system before the stated deadline.
H-P2 — File format
TDP must be submitted as a single PDF file. Other formats (Word, LaTeX source, etc.) are not accepted.
H-P3 — Template compliance
TDP must use the official RoboCup LNCS/Springer template without modification to fonts, margins, column layout, or heading styles. (e.g. https://www.springer.com/gp/computer-science/lncs/conference-proceedings-guidelines, https://www.overleaf.com/latex/templates/springer-lecture-notes-in-computer-science/kzwwpvhwnvfj#.WuA4JS5uZpi )
H-P4 — Page limit (main body)
The main body of the TDP must not exceed 8 pages, including all figures, tables, and references. The annex does not count toward this limit but must be appended after the references. Submissions where the main body exceeds 8 pages are rejected.
H-P5 — Language
TDP must be written entirely in English.
H-P6 — Team identification
TDP must state the full team name, institution(s), league, and at least one contact email address.
H-P7 — No anonymization
TDP must not be anonymized. Team identity must be clearly stated.
H-P8 — Annex required
A mandatory annex must be appended after the references. The annex must contain all of the following: (1) a photo of the robot, (2) a compact list of all third-party software used (e.g. MoveIt, YOLO), (3) a compact description of all external computing devices if any are used, (4) a compact description of the robot hardware. Home-made software solutions should be marked with an asterisk (*). An annex that is missing entirely, or that omits any of the four required elements, results in rejection. The annex must not exceed 2 pages. One page is strongly encouraged.
H-P9 — Produced for this year’s submission
The TDP must be written specifically for this year’s RoboCup qualification. The RoboCup or @Home should be mentioned in at least one of the following locations: the title page, the header, the abstract, the introduction or the annex.
1.2 Demo Video
H-V1 — Submission deadline
Video link must be submitted before the stated deadline via the official submission system.
H-V2 — Accessibility
Video must be publicly accessible via a persistent URL (e.g. YouTube, Vimeo) without login, password, or regional restriction. The link must remain accessible through the full review period. Accessibility must be verified at submission and again at the start of the review window.
H-V3 — Playability
Video must be playable in full without errors at the time of review. Broken or unavailable links result in automatic rejection.
H-V4 — Maximum duration
Video must not exceed 10 minutes in total length, including any introduction or credits.
H-V5 — Physical robot required
The video must show the physical, real-world robot.
H-V6 — Robot performing competition tasks
The video must include at least one scene in which the robot attempts a task directly relevant to a RoboCup task. Footage of the robot powered on but idle, or performing unrelated tasks, does not satisfy this criterion.
H-V7 — Autonomous operation
The demonstrated task execution must be performed autonomously. Any use of teleoperation or manual guidance during task execution constitutes a failure of this criterion – unless the Task specifically calls for manual guidance as part of the challenge (please check the Rulebook).
H-V8 — Video corresponds to the team
The robot(s) shown in the video must match the robot(s) described in the TDP. Submissions where the video clearly depicts exclusively a different robot are rejected. Teams are allowed to show complex tasks with AN older robot, but should demonstrate at least “robot inspection” criteria with the new robot. We remind teams that they can sign up multiple robots per team, so if basic software has not been integrated on new robots by the end of a call, they can register both robots (and only bring one to the actual competition).
H-V9 — Minimum resolution
Video must be available in at least 720p (1280×720). If hosted on YouTube/Vimeo, the platform must offer 720p or higher as a selectable option. Videos that max out at 360p or 480p are rejected.
H-V10 — Stable footage
Video must not be so shaky, blurred, or poorly lit that the robot and its actions cannot be identified. If the robot’s movement and task execution cannot be followed at normal playback speed, the criterion fails.
H-V11 — Aspect ratio
Video must be in 16:9 or 4:3 landscape format. Portrait orientation (9:16 or similar) is not accepted. Videos recorded on a phone held vertically are rejected regardless of hosting platform or resolution.
H-V12 — Produced for this year’s submission
The video must be a single, newly produced video created specifically for this year’s RoboCup qualification. Teams must include a visible on-screen statement identifying the team name and the current competition year (e.g. ” — RoboCup Qualification Video”).
1.3 General
H-G1 — Team photo with assembled robot
Either the TDP, the video or the team website must contain at least one photograph showing the (majority of the) team members together with the fully assembled robot. The photo must be taken in the current qualification year.
Soft Criteria — Reviewer Guide
Score each criterion 0–5 (0 = not provided, 1 = poor, 3 = adequate, 5 = excellent). Complete one section fully before moving to the next.
Spread threshold: If two reviewers diverge by 9 or more total points, a third reviewer is assigned.
Section 1 — Team Description Paper (TDP)
Read the full PDF before scoring.
S-P1 — Writing clarity and structure
Is the TDP well-structured and technically precise? Vague or marketing-style writing scores low. Clear prose with logical flow scores high.
S-P2 — Hardware description
Is the robot hardware described concretely and completely, including sensors, actuators, and platform? Future plans or vague descriptions score low. A current, complete description with photos scores high.
S-P3 — Software architecture
Are key modules (perception, planning, control, HRI) described with evidence of implementation? Intentions without implementation evidence score low.
S-P4 — Technical soundness
Are the described methods technically credible and internally consistent? Unsupported claims, implausible performance figures, or contradictions between sections score low. Grounded, verifiable descriptions score high.
S-P5 — Scientific contribution and novelty
Does the TDP present an identifiable contribution — a new method, improvement, or insight? Novelty is not required but rewarded. Purely descriptive papers with no original contribution score low.
S-P6 — Awareness of related work
Does the team position their approach relative to existing methods or prior RoboCup work? No engagement with the state of the art scores low.
S-P7 — Realistic self-assessment
Does the team acknowledge limitations, failure modes, or open problems? Purely promotional writing with no caveats scores low. Honest, specific acknowledgement of what does and does not yet work scores high.
S-P8 — TDP holistic assessment
(–2 to +2) Default 0. Add up to +2 for exceptional depth or maturity beyond individual criteria. Subtract up to –2 for unresolved concerns: inconsistencies, implausible claims, or signs of a recycled submission. One-sentence written justification required for any non-zero score.
Section 1 maximum: 37 points
Section 2 — Demo Video
Watch the full video before scoring.
S-V1 — Task relevance and breadth
Does the video demonstrate abilities directly relevant to RoboCup tasks (e.g. perception, manipulation, navigation, HRI, speech)? A single isolated ability scores low; a broad range of relevant capabilities scores high.
S-V2 — Depth of task execution
Does the robot complete tasks fully or only partially? Idle demonstrations or single-step executions score low. End-to-end task completion in realistic conditions scores high.
S-V3 — Autonomy and robustness
Does execution appear genuinely autonomous and stable? Frequently interrupted or suspiciously seamless runs score low. Fluid, continuous execution with visible recovery from minor issues scores high.
S-V4 — Environment realism
Is the environment representative of real RoboCup conditions? An ideal lab bench scores lower than a realistic domestic or competition-like environment.
S-V5 — Video clarity, annotations, and structure
Scored as a single combined criterion across five qualities. A score of 5 requires all five; deduct one point for each that is clearly absent or poor:
- Robot and task execution are clearly visible throughout (lighting, framing, stability)
- Narration or commentary explains what is being shown
- Key moments are labelled with on-screen text or subtitles
- Every clip identifies the task, subtask, speed multiplier if applicable, and success/failure outcome
- Video is structured for navigation (chapters, title cards, or clear cuts between tasks)
S-V6 — Video holistic assessment
(–2 to +2) Default 0. Add up to +2 for exceptional engineering integration or impressive unrehearsed performance. Subtract up to –2 for concerns such as a staged demonstration, suspected hidden teleoperation, or significant inconsistency with the TDP. One-sentence written justification required for any non-zero score.
S-V7 — Speed-up disclosure
Any footage played at faster than real-time speed should be explicitly labelled with the speed multiplier (e.g. “4×”) on screen at all times during the sped-up segment. (0 or +1)
Section 2 maximum: 28 points
The video’s total score is multiplied by a factor of 1.25 so that the video is weighted similarly to the TDP.
Section 2 maximum (incl. factor): 35 points
Section 3 — Team Website
Check the website live. Do not rely on cached versions.
S-W1 — Team and robot information
Are team name, member roles (PI, team leader, contact), institutional affiliations, and robot clearly presented? A placeholder or incomplete site scores low.
S-W2 — Publications and research output
Are relevant publications from the last 5 years listed with accessible download links? No publications or inaccessible links score low.
S-W3 — Software openness and documentation
Are software repositories linked, active, and documented well enough for other teams to reuse? Dead, empty, or undocumented repositories score low. Well-maintained, clearly documented tools score high.
S-W4 — Website holistic assessment
(–2 to +2) Default 0. Add up to +2 for exceptional openness or community contribution. Subtract up to –2 for misleading content, inactive repositories presented as active, or a site clearly created solely for the submission. One-sentence written justification required for any non-zero score.
Section 3 maximum: 17 points
Section 4 — Team Profile
S-T1 — Team experience and preparation
Score once on a single 1–5 scale using whichever guidance applies:
- Returning team: Base the score on previous qualification, attendance, and scoring history in RoboCup or comparable competitions. Consistent participation with improving results scores high; a long history with no meaningful progress and low scores, scores low.
- First-time team: Base the score on evidence of structured preparation — testbed availability, prior robotics projects, development timeline, and relevant coursework or publications. Vague statements of intent score low; concrete, documented preparation scores high.
S-T2 — Team composition and expertise
Does the team cover the necessary skills across hardware, software, and organisation? Is the team size realistic for competition demands?
S-T3 — Cross-submission consistency
Are the TDP, video, and website coherent with each other in terms of robot, system description, and claimed capabilities? Significant discrepancies — different robots shown, inconsistent claims, or clearly outdated materials in one artifact — score low.
Section 4 maximum: 15 points
Score Summary
| Section | Criteria | Max Score |
|---|---|---|
| 1 — TDP | S-P1 to S-P8 | 37 |
| 2 — Video | S-V1 to S-V7 | 35 |
| 3 — Website | S-W1 to S-W4 | 17 |
| 4 — Team Profile | S-T1 to S-T3 | 15 |
| Total | 22 criteria | 104 |
If there are any questions, please contact the Organizing Committee via Email: rc-home-oc[at]lists.robocup.org