Dr.-Ing. Simon Reiß

Dr.-Ing. Simon Reiß

Computer Vision and Machine Learning Researcher

Publications

You can also find an up to date list of my articles on my Google Scholar profile.

Conference Papers

Semantic Segmentation for Preoperative Planning in Transcatheter Aortic Valve Replacement

Published in to appear at MICCAI Workshop on Statistical Atlases and Computational Modeling of the Heart, 2025

This work supports preoperative planning for transcatheter aortic valve replacement (TAVR) by using AI-based semantic segmentation to measure relevant anatomical structures in CT scans. Fine-grained TAVR-specific pseudo-labels are derived from coarse anatomical data to train segmentation models, with a loss function adaptation improving Dice score performance by 1.27%. The pseudo-labeled dataset is publicly available at https://doi.org/10.5281/zenodo.16274176.

Download Paper | Download Bibtex

Conquering the Retina: Bringing Visual in-Context Learning to OCT

Published in to appear at MICCAI Workshop on Efficient Medical AI, 2025

The paper studies visual in-context learning (VICL) for training generalist models in retinal OCT, enabling task definition on the fly without task-specific models. It proposes an evaluation protocol for VICL in OCT, establishes a baseline using multiple datasets, and releases code to support further research.

Download Paper | Download Bibtex

Good Enough: Is it Worth Improving your Label Quality?

Published in arXiv, technical report, 2025

Improving label quality in medical image segmentation is costly, but its benefits remain unclear. We systematically evaluate its impact using multiple pseudo-labeled versions of CT datasets, generated by models like nnU-Net, TotalSegmentator, and MedSAM. Our results show that while higher-quality labels improve in-domain performance, gains remain unclear if below a small threshold. For pre-training, label quality has minimal impact, suggesting that models rather transfer general concepts than detailed annotations. These findings provide guidance on when improving label quality is worth the effort.

Download Paper | Download Bibtex

CHAOS: Chart Analysis with Outlier Samples

Published in arXiv, technical report, 2025

The paper introduces CHAOS, a robustness benchmark for testing MLLMs on charts with textual and visual perturbations at varying severity. It evaluates 13 models across ChartQA and Chart-to-Text tasks, revealing weaknesses and guiding future research in chart understanding. Data and code are publicly released.

Download Paper | Download Bibtex

Every Component Counts: Rethinking the Measure of Success for Medical Semantic Segmentation in Multi-Instance Segmentation Tasks

Published in AAAI, 2025

The paper proposes Connected-Component (CC)-Metrics, a segmentation evaluation protocol for multi-instance detection where each connected component is equally important. Designed for cases like metastases segmentation in PET/CT, it removes bias toward large tumors by evaluating metrics per component using proximity-based matching. This approach corrects issues in overlap-, distance-, and counting-based metrics, providing fairer clinical assessment.

Download Paper | Download Bibtex

Muscles in Time: Learning to Understand Human Motion In-Depth by Simulating Muscle Activations

Published in NeurIPS, 2024

The paper introduces Muscles in Time (MinT), a large-scale synthetic dataset of muscle activation generated by enriching motion capture data with OpenSim biomechanical simulations. Covering 227 subjects and 402 muscles over nine hours, MinT enables detailed study of muscle activation timing and supports training models for muscle activation estimation from pose sequences.

Download Paper | Download Bibtex

Anatomy-Guided Pathology Segmentation

Published in MICCAI, 2024

The paper presents APEx, a segmentation model that integrates anatomical and pathological features to improve pathology segmentation. Using a query-based transformer and a mixing strategy, APEx enables anatomy-informed pathology predictions, achieving state-of-the-art results on FDG-PET-CT and Chest X-ray tasks. Code and models are publicly available.

Download Paper | Download Bibtex

SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading

Published in EMNLP, 2024

This paper introduces SciEx, a multilingual, multimodal benchmark of university computer science exam questions for testing LLMs on scientific tasks. It includes freeform, image-based questions in English and German, with human expert grading revealing the best model scores only 59.4%. The authors compare LLM and student performance and propose LLM-as-a-judge grading, achieving high correlation with experts.

Download Paper | Download Bibtex

Style Transfer and Pseudo-Label Filtering Improve Transferability in Cell Organelle Segmentation Scenarios

Published in ISBI, 2024

The paper studies domain adaptation for cell organelle segmentation, where models lose over 60% accuracy on new imaging data. It evaluates unsupervised and weakly supervised methods, finding that image-level labels help. The proposed StyleFilter method leverages these labels, improving performance by 19.2% absolute Dice over naive transfer in electron microscopy adaptation.

Download Paper | Download Bibtex

Mirror U-Net: Marrying Multimodal Fission with Multi-Task Learning for Semantic Segmentation in Medical Imaging

Published in ICCV Workshops, 2023

This paper introduces Mirror U-Net, a PET/CT tumor segmentation model that replaces traditional fusion with multimodal fission, splitting features into modality-specific and shared decoders. This setup combines fission and multi-task learning to strengthen unimodal features while preserving multimodal ones. Evaluated on AutoPET and MSD BrainTumor, Mirror U-Net achieves state-of-the-art performance. Code is publicly available.

Download Paper | Download Bibtex

Delivering arbitrary-modal semantic segmentation

Published in CVPR, 2023

This paper presents DeLiVER, a benchmark for arbitrary-modal semantic segmentation across Depth, LiDAR, multiple Views, Events, and RGB, with variations for severe weather and sensor failures. It introduces CMNeXt, a flexible fusion model with a Self-Query Hub and Parallel Pooling Mixer, enabling scaling from 1 to 81 modalities with minimal extra parameters. CMNeXt achieves state-of-the-art results on six datasets, with up to +9.10% mIoU gain on DeLiVER over the RGB-only baseline.

Download Paper | Download Bibtex

Decoupled Semantic Prototypes Enable Learning From Diverse Annotation Types for Semi-Weakly Segmentation in Expert-Driven Domains

Published in CVPR, 2023

This paper addresses segmentation in expert-driven domains with scarce pixel-wise annotations, evaluating existing methods for handling diverse annotation types. Finding current approaches limited, it proposes Decoupled Semantic Prototypes (DSP), a method that learns from image-level, point, box, and pixel annotations, achieving notable accuracy gains in semi-weakly supervised organelle segmentation.

Download Paper | Download Bibtex

Accurate Fine-Grained Segmentation of Human Anatomy in Radiographs via Volumetric Pseudo-Labeling

Published in arXiv, technical report, 2023

This paper proposes using 3D CT-based pseudo-labeling to train fine-grained anatomical segmentation models for chest X-rays without manual annotations. From 10,021 thoracic CTs with 157 labels, 3D segmentations are projected to 2D, enabling high-accuracy CXR models with mIoU up to 0.93 and strong agreement with radiologists. The method supports explainable clinical measures like cardio-thoracic ratio and could aid thoracic pathology analysis.

Download Paper | Download Bibtex

Detailed Annotations of Chest X-Rays via CT Projection for Report Understanding

Published in BMVC, 2022

This paper addresses the gap in medical image processing where models lack explicit anatomical knowledge needed for understanding radiology reports. It introduces PAXRay, a dataset integrating detailed anatomical segmentations from CT scans into X-ray images to improve medical phrase grounding linking report content to image regions. Using anatomical info, models achieve up to 50% better grounding on the OpenI dataset compared to standard methods. The dataset is publicly available.

Download Paper | Download Bibtex

Graph-Constrained Contrastive Regularization for Semi-weakly Volumetric Segmentation

Published in ECCV, 2022

This paper tackles semantic volume segmentation with scarce annotations, training on sparsely labeled volumes where only some slices are annotated. It introduces Contrastive Constrained Regularization (Con2R), enabling 3D models to achieve up to 88% of fully supervised accuracy using <4% 2D labels. Experiments on retinal fluid and brain tumor segmentation show Con2R’s effectiveness in extreme low-label settings.

Download Paper | Download Bibtex

Breaking with fixed set pathology recognition through report-guided contrastive training

Published in MICCAI, 2022

This paper tackles the limitation of fixed-category supervision in radiology image classification by using direct text supervision from unstructured medical reports. It introduces a contrastive global-local dual-encoder to learn concepts without relying on predefined labels, enabling open set recognition and better use of weakly annotated data. Evaluated on large chest X-ray datasets, the approach matches performance of traditional label-supervised methods while allowing free-form classification.

Download Paper | Download Bibtex

Bending Reality: Distortion-Aware Transformers for Adapting to Panoramic Semantic Segmentation

Published in CVPR, 2022

This work tackles the challenge of training panoramic segmentation models using abundant pinhole camera annotations. The proposed Trans4PASS model uses Deformable Patch Embedding and MLP modules to handle panoramic distortions and learns shared semantics via Mutual Prototypical Adaptation for domain alignment. It matches fully supervised results indoors and sets a new state-of-the-art outdoors, reducing the need for many labeled panoramas.

Download Paper | Download Bibtex

Let’s Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games

Published in IROS, 2021

This work introduces Sims4Action, a dataset created by capturing Activities of Daily Living (ADL) from the game THE SIMS 4 to address challenges in collecting real-world annotated data. It enables flexible data generation with varied environments and views. The GamingToReal benchmark tests how well models trained on game data transfer to real videos. Results show promise but highlight challenges in mixing simulated and real data, opening new research directions. The dataset will be publicly available.

Download Paper | Download Bibtex

Capturing Omni-Range Context for Omnidirectional Segmentation

Published in CVPR, 2021

We introduce Efficient Concurrent Attention Networks (ECANets) to improve semantic segmentation on 360° omnidirectional images, addressing the large performance drop from narrow FoV models. ECANets capture long-range dependencies and leverage multi-source, omni-supervised training with both labeled and unlabeled data. Evaluated on the new WildPASS dataset and the PASS benchmark, our approach sets new state-of-the-art mIoU scores of 69.0% and 60.2%, respectively.

Download Paper | Download Bibtex

Every annotation counts: Multi-label deep supervision for medical image segmentation

Published in CVPR, 2021

We propose a semi-weakly supervised segmentation method using a student-teacher model and multi-label deep supervision. Our approach integrates various annotation types - full, bounding boxes, global labels, or none - reducing expensive labels by 94.22% while achieving segmentation close to fully supervised models. Validated on retinal fluid segmentation, it narrows the gap to the best baseline to just 5% mean IoU.

Download Paper | Download Bibtex

Deep Classification-driven Domain Adaptation for Cross-Modal Driver Behavior Recognition

Published in IV, 2020

We tackle unsupervised domain adaptation for driver activity recognition, where models trained on labeled color images adapt to unlabeled infrared images. Using enhanced Variational Auto-Encoders for image translation with classification-driven optimization, our approach learns a shared latent space, boosting cross-domain recognition by 13.75% over conventional methods.

Download Paper | Download Bibtex

CNN-based Driver Activity Understanding: Shedding Light on Deep Spatiotemporal Representations

Published in ITSC, 2020

We present a diagnostic framework to interpret deep CNNs for driver monitoring by visualizing predictions, analyzing learned features, and conducting failure analysis. Our study reveals common errors stem from object/movement biases, class similarity, and data imbalance. The Inflated 3D Net shows the best performance with more distinct feature clusters and higher recognition rates.

Download Paper | Download Bibtex

Drive&Act: A Multi-Modal Dataset for Fine-Grained Driver Behavior Recognition in Autonomous Vehicles

Published in ICCV, 2019

We present Drive&Act, a domain-specific benchmark with 12 hours and 9.6M frames of driver behaviors across 83 categories. Captured from six views using color, infrared, depth, and 3D pose data, it challenges fine-grained, multi-modal, and cross-view activity recognition. We also provide benchmarks using state-of-the-art video and pose-based methods.

Download Paper | Download Bibtex