Overview
Visual inspection plays a central role in structural inspection, especially in the analysis of damage to critical infrastructure. However, annotation variations - deviations, uncertainties and sometimes even errors in the annotations - pose a major challenge for machine learning as they significantly affect model performance. This problem is not limited to structural inspection, but also affects many other application areas where visual data is processed.
Aim and approach
The project addresses various aspects of dealing with annotation variation (AV):
- Identification, quantification, impact analysis and correction of AV
- Developing new methods for dealing with AV during model training
- Creation of practical evaluation metrics that take into account the effects of AV
In addition, a damage detection dataset with real-world AV will be curated to advance the research and provide a foundation for future work.
18. March 2026
The code for our upcoming CVPR 2026 publication "KαLOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks" is now available.
As part of the VarInSightAI project, this toolkit provides a standardized framework to quantify annotation consistency and identify label noise in complex computer vision datasets. The paper will be presented at the IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR) in Denver in June 2026.
16. Feburary 2026
"KαLOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks" was accepted for CVPR 2026.
03. March 2025
“Label Convergence: Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations” was presented at WACV 2025.
The results and tools developed in the project will be published as freely available software. The FiftyOne plugin Multi-Annotator Toolkit already supports the identification, quantification and impact analysis of AV.
Current Publications:
- D. Tschirschwitz and V. Rodehorst: KαLOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks, IEEE/CVF Computer Vision and Pattern Recognition (CVPR), Denver, Colorado, 2026.
- D. Tschirschwitz and V. Rodehorst: Label Convergence - Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations, IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, Arizona, 2025.
Previous Publications:
- C. Benz and V. Rodehorst: CrackStructures and CrackEnsembles - The Power of Multi- View for 2.5D Crack Detection, IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, Arizona, 2025.
- D. Tschirschwitz and V. Rodehorst: Label Convergence - Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations, IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, Arizona, 2025.
- C. Benz and V. Rodehorst: ENSTRECT - A Stage-based Approach to 2.5D Structural Damage Detection, European Conference on Computer Vision (ECCV) 2. Workshop on Vision-based InduStrial InspectiON (VISION), Mailand, Italien, 2024.
- C. Benz and V. Rodehorst. „MVCrackViT: Robust Multi-View Crack Detection for Point Cloud Segmentation using View Attention“. In: Proc. IEEE Int. Conf. on Image Processing (ICIP), 2024.
- C. Benz and V. Rodehorst. „OmniCrack30k: A Benchmark for Crack Segmentation and the Reasonable Effectiveness of Transfer Learning“. In: Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024.
- J. Flotzinger, P. J. Rösch, C. Benz, M. Ahmad, M. Cankaya, H. Mayer, V. Rodehorst, N. Oswald and T. Braml. „Dacl-Challenge: Semantic Segmentation During Visual Bridge Inspections“. In: Proc. IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV) Workshops, 2024.
- D. Tschirschwitz, C. Benz, M. Florek, H. Noerderhus, B. Stein and V. Rodehorst. „Drawing the Same Bounding Box Twice? Coping Noisy Annotations in Object Detection with Repeated Labels“. In: Proc. German Conf. on Pattern Recognition (GCPR), 2023.
- C. Benz and V. Rodehorst. „Image-Based Detection of Structural Defects Using Hierarchical Multi-scale Attention“. In: Proc. German Conf. on Pattern Recognition (GCPR), 2022.
- D. Tschirschwitz, F. Klemstein, B. Stein and V. Rodehorst. „A Dataset for Analyzing Complex Document Layouts in the Digital Humanities and its Evaluation with Krippendorff ’s Alpha“. In: Proc. German Conf. on Pattern Recogntion (GCPR), 2022.
- C. Benz, P. Debus, H. K. Ha and V. Rodehorst. „Crack Segmentation on UAS-based Imagery using Transfer Learning“. In: Proc. Int. Conf. on Image and Vision Computing New Zealand (IVCNZ), 2019.
This project is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - project number 556314568.