Sitemap
A list of all the posts and pages found on the site. For you robots out there, there is an XML version available for digesting as well.
Pages
Posts
Blog not maintained currently
Published:
This post will show up by default. To disable scheduling of future posts, edit config.yml and set future: false.
portfolio
publications
Drive&Act: A Multi-Modal Dataset for Fine-Grained Driver Behavior Recognition in Autonomous Vehicles
Published in ICCV, 2019
We present Drive&Act, a domain-specific benchmark with 12 hours and 9.6M frames of driver behaviors across 83 categories. Captured from six views using color, infrared, depth, and 3D pose data, it challenges fine-grained, multi-modal, and cross-view activity recognition. We also provide benchmarks using state-of-the-art video and pose-based methods.
Activity-Aware Attributes for Zero-Shot Driver Behavior Recognition
Published in CVPR Workshops, 2020
We introduce ZS-Drive&Act, the first zero-shot benchmark for recognizing unseen fine-grained driver behaviors using activity-driven attributes. Evaluating several zero-shot methods, we improve recognition by 2.79% through a fusion of semantic attributes and word vectors in a Wasserstein GAN-based approach.
CNN-based Driver Activity Understanding: Shedding Light on Deep Spatiotemporal Representations
Published in ITSC, 2020
We present a diagnostic framework to interpret deep CNNs for driver monitoring by visualizing predictions, analyzing learned features, and conducting failure analysis. Our study reveals common errors stem from object/movement biases, class similarity, and data imbalance. The Inflated 3D Net shows the best performance with more distinct feature clusters and higher recognition rates.
Deep Classification-driven Domain Adaptation for Cross-Modal Driver Behavior Recognition
Published in IV, 2020
We tackle unsupervised domain adaptation for driver activity recognition, where models trained on labeled color images adapt to unlabeled infrared images. Using enhanced Variational Auto-Encoders for image translation with classification-driven optimization, our approach learns a shared latent space, boosting cross-domain recognition by 13.75% over conventional methods.
Every annotation counts: Multi-label deep supervision for medical image segmentation
Published in CVPR, 2021
We propose a semi-weakly supervised segmentation method using a student-teacher model and multi-label deep supervision. Our approach integrates various annotation types - full, bounding boxes, global labels, or none - reducing expensive labels by 94.22% while achieving segmentation close to fully supervised models. Validated on retinal fluid segmentation, it narrows the gap to the best baseline to just 5% mean IoU.
Capturing Omni-Range Context for Omnidirectional Segmentation
Published in CVPR, 2021
We introduce Efficient Concurrent Attention Networks (ECANets) to improve semantic segmentation on 360° omnidirectional images, addressing the large performance drop from narrow FoV models. ECANets capture long-range dependencies and leverage multi-source, omni-supervised training with both labeled and unlabeled data. Evaluated on the new WildPASS dataset and the PASS benchmark, our approach sets new state-of-the-art mIoU scores of 69.0% and 60.2%, respectively.
From Driver Talk To Future Action: Vehicle Maneuver Prediction by Learning from Driving Exam Dialogs
Published in IV, 2021
This paper proposes an automated method to predict driver intentions from YouTube mock road test videos and dialogs, avoiding costly labels. By detecting and filtering out casual smalltalk, our approach improves prediction accuracy using cleaner conversation data.
Let’s Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games
Published in IROS, 2021
This work introduces Sims4Action, a dataset created by capturing Activities of Daily Living (ADL) from the game THE SIMS 4 to address challenges in collecting real-world annotated data. It enables flexible data generation with varied environments and views. The GamingToReal benchmark tests how well models trained on game data transfer to real videos. Results show promise but highlight challenges in mixing simulated and real data, opening new research directions. The dataset will be publicly available.
Bending Reality: Distortion-Aware Transformers for Adapting to Panoramic Semantic Segmentation
Published in CVPR, 2022
This work tackles the challenge of training panoramic segmentation models using abundant pinhole camera annotations. The proposed Trans4PASS model uses Deformable Patch Embedding and MLP modules to handle panoramic distortions and learns shared semantics via Mutual Prototypical Adaptation for domain alignment. It matches fully supervised results indoors and sets a new state-of-the-art outdoors, reducing the need for many labeled panoramas.
Reference-guided Pseudo-Label Generation for Medical Semantic Segmentation
Published in AAAI, 2022
This paper introduces a semi-supervised segmentation method that matches unlabeled pixels to labeled reference pixels, avoiding confirmation bias and needing no model changes. It achieves full supervision performance with 95% fewer labels and improves retinal fluid segmentation by up to 15% IoU.
Breaking with fixed set pathology recognition through report-guided contrastive training
Published in MICCAI, 2022
This paper tackles the limitation of fixed-category supervision in radiology image classification by using direct text supervision from unstructured medical reports. It introduces a contrastive global-local dual-encoder to learn concepts without relying on predefined labels, enabling open set recognition and better use of weakly annotated data. Evaluated on large chest X-ray datasets, the approach matches performance of traditional label-supervised methods while allowing free-form classification.
Graph-Constrained Contrastive Regularization for Semi-weakly Volumetric Segmentation
Published in ECCV, 2022
This paper tackles semantic volume segmentation with scarce annotations, training on sparsely labeled volumes where only some slices are annotated. It introduces Contrastive Constrained Regularization (Con2R), enabling 3D models to achieve up to 88% of fully supervised accuracy using <4% 2D labels. Experiments on retinal fluid and brain tumor segmentation show Con2R’s effectiveness in extreme low-label settings.
Detailed Annotations of Chest X-Rays via CT Projection for Report Understanding
Published in BMVC, 2022
This paper addresses the gap in medical image processing where models lack explicit anatomical knowledge needed for understanding radiology reports. It introduces PAXRay, a dataset integrating detailed anatomical segmentations from CT scans into X-ray images to improve medical phrase grounding linking report content to image regions. Using anatomical info, models achieve up to 50% better grounding on the OpenI dataset compared to standard methods. The dataset is publicly available.
Accurate Fine-Grained Segmentation of Human Anatomy in Radiographs via Volumetric Pseudo-Labeling
Published in arXiv, technical report, 2023
This paper proposes using 3D CT-based pseudo-labeling to train fine-grained anatomical segmentation models for chest X-rays without manual annotations. From 10,021 thoracic CTs with 157 labels, 3D segmentations are projected to 2D, enabling high-accuracy CXR models with mIoU up to 0.93 and strong agreement with radiologists. The method supports explainable clinical measures like cardio-thoracic ratio and could aid thoracic pathology analysis.
Decoupled Semantic Prototypes Enable Learning From Diverse Annotation Types for Semi-Weakly Segmentation in Expert-Driven Domains
Published in CVPR, 2023
This paper addresses segmentation in expert-driven domains with scarce pixel-wise annotations, evaluating existing methods for handling diverse annotation types. Finding current approaches limited, it proposes Decoupled Semantic Prototypes (DSP), a method that learns from image-level, point, box, and pixel annotations, achieving notable accuracy gains in semi-weakly supervised organelle segmentation.
Delivering arbitrary-modal semantic segmentation
Published in CVPR, 2023
This paper presents DeLiVER, a benchmark for arbitrary-modal semantic segmentation across Depth, LiDAR, multiple Views, Events, and RGB, with variations for severe weather and sensor failures. It introduces CMNeXt, a flexible fusion model with a Self-Query Hub and Parallel Pooling Mixer, enabling scaling from 1 to 81 modalities with minimal extra parameters. CMNeXt achieves state-of-the-art results on six datasets, with up to +9.10% mIoU gain on DeLiVER over the RGB-only baseline.
Mirror U-Net: Marrying Multimodal Fission with Multi-Task Learning for Semantic Segmentation in Medical Imaging
Published in ICCV Workshops, 2023
This paper introduces Mirror U-Net, a PET/CT tumor segmentation model that replaces traditional fusion with multimodal fission, splitting features into modality-specific and shared decoders. This setup combines fission and multi-task learning to strengthen unimodal features while preserving multimodal ones. Evaluated on AutoPET and MSD BrainTumor, Mirror U-Net achieves state-of-the-art performance. Code is publicly available.
360bev: Panoramic semantic mapping for indoor bird’s-eye view
Published in WACV, 2024
This paper introduces the 360BEV task-mapping panoramic images with depth to bird’s-eye-view semantic maps for holistic indoor scene representation. It releases two datasets (360BEV-Matterport and 360BEV-Stanford) and proposes 360Mapper, a method that outperforms prior approaches by +7.60% and +9.70% mIoU on the respective datasets.
Style Transfer and Pseudo-Label Filtering Improve Transferability in Cell Organelle Segmentation Scenarios
Published in ISBI, 2024
The paper studies domain adaptation for cell organelle segmentation, where models lose over 60% accuracy on new imaging data. It evaluates unsupervised and weakly supervised methods, finding that image-level labels help. The proposed StyleFilter method leverages these labels, improving performance by 19.2% absolute Dice over naive transfer in electron microscopy adaptation.
Behind every domain there is a shift: Adapting distortion-aware vision transformers for panoramic semantic segmentation
Published in TPAMI, 2024
The paper presents Trans4PASS+, a transformer-based model for panoramic semantic segmentation, addressing image distortions and scarce annotations via Deformable Patch Embedding and Deformable MLP modules. It improves Mutual Prototypical Adaptation with pseudo-label rectification and introduces SynPASS, a 9,080-panoramic image dataset for Synthetic-to-Real adaptation. The method achieves state-of-the-art results on four domain-adaptive benchmarks.
SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading
Published in EMNLP, 2024
This paper introduces SciEx, a multilingual, multimodal benchmark of university computer science exam questions for testing LLMs on scientific tasks. It includes freeform, image-based questions in English and German, with human expert grading revealing the best model scores only 59.4%. The authors compare LLM and student performance and propose LLM-as-a-judge grading, achieving high correlation with experts.
Data Diet: Can Trimming PET/CT Datasets Enhance Lesion Segmentation?
Published in arXiv, MICCAI autoPET III Challenge, 2024
This paper details a data selection strategy for the autoPET3 datacentric track, showing that removing the easiest training samples (based on model loss) reduces false positives—especially in PSMA-PET—and improves both false negative volume and Dice score over the baseline.
Anatomy-Guided Pathology Segmentation
Published in MICCAI, 2024
The paper presents APEx, a segmentation model that integrates anatomical and pathological features to improve pathology segmentation. Using a query-based transformer and a mixing strategy, APEx enables anatomy-informed pathology predictions, achieving state-of-the-art results on FDG-PET-CT and Chest X-ray tasks. Code and models are publicly available.
Muscles in Time: Learning to Understand Human Motion In-Depth by Simulating Muscle Activations
Published in NeurIPS, 2024
The paper introduces Muscles in Time (MinT), a large-scale synthetic dataset of muscle activation generated by enriching motion capture data with OpenSim biomechanical simulations. Covering 227 subjects and 402 muscles over nine hours, MinT enables detailed study of muscle activation timing and supports training models for muscle activation estimation from pose sequences.
Foreign object segmentation in chest x-rays through anatomy-guided shape insertion
Published in arXiv, technical report, 2025
The paper addresses instance segmentation of foreign objects in chest X-rays, where data scarcity hinders training. It generates synthetic data by inserting shapes or cut-pasted labels with anatomically guided placement. This method matches fully supervised performance while using 93% fewer manual annotations.
Every Component Counts: Rethinking the Measure of Success for Medical Semantic Segmentation in Multi-Instance Segmentation Tasks
Published in AAAI, 2025
The paper proposes Connected-Component (CC)-Metrics, a segmentation evaluation protocol for multi-instance detection where each connected component is equally important. Designed for cases like metastases segmentation in PET/CT, it removes bias toward large tumors by evaluating metrics per component using proximity-based matching. This approach corrects issues in overlap-, distance-, and counting-based metrics, providing fairer clinical assessment.
CHAOS: Chart Analysis with Outlier Samples
Published in arXiv, technical report, 2025
The paper introduces CHAOS, a robustness benchmark for testing MLLMs on charts with textual and visual perturbations at varying severity. It evaluates 13 models across ChartQA and Chart-to-Text tasks, revealing weaknesses and guiding future research in chart understanding. Data and code are publicly released.
Good Enough: Is it Worth Improving your Label Quality?
Published in arXiv, technical report, 2025
Improving label quality in medical image segmentation is costly, but its benefits remain unclear. We systematically evaluate its impact using multiple pseudo-labeled versions of CT datasets, generated by models like nnU-Net, TotalSegmentator, and MedSAM. Our results show that while higher-quality labels improve in-domain performance, gains remain unclear if below a small threshold. For pre-training, label quality has minimal impact, suggesting that models rather transfer general concepts than detailed annotations. These findings provide guidance on when improving label quality is worth the effort.
Conquering the Retina: Bringing Visual in-Context Learning to OCT
Published in to appear at MICCAI Workshop on Efficient Medical AI, 2025
The paper studies visual in-context learning (VICL) for training generalist models in retinal OCT, enabling task definition on the fly without task-specific models. It proposes an evaluation protocol for VICL in OCT, establishes a baseline using multiple datasets, and releases code to support further research.
Is Visual in-Context Learning for Compositional Medical Tasks within Reach?
Published in to appear at ICCV, 2025
This paper explores visual in-context learning for adapting a single model to multi-step vision tasks without retraining. It introduces a synthetic task generation engine from segmentation data, studies codebooks and masking-based training.
Semantic Segmentation for Preoperative Planning in Transcatheter Aortic Valve Replacement
Published in to appear at MICCAI Workshop on Statistical Atlases and Computational Modeling of the Heart, 2025
This work supports preoperative planning for transcatheter aortic valve replacement (TAVR) by using AI-based semantic segmentation to measure relevant anatomical structures in CT scans. Fine-grained TAVR-specific pseudo-labels are derived from coarse anatomical data to train segmentation models, with a loss function adaptation improving Dice score performance by 1.27%. The pseudo-labeled dataset is publicly available at https://doi.org/10.5281/zenodo.16274176.
talks
Virtual presentation of ‘Activity-aware attributes for zero-shot driver behavior recognition’
Published:
 | At the CVPR 2020 workshop on Visual Learning With Limited Labels, I presented our paper on Zero-Shot Learning for Driver Behavior Recognition in a virtual poster session. |
Virtual presentation of ‘Deep Classification-driven Domain Adaptation for Cross-Modal Driver Behavior Recognition’
Published:
 | At the Intelligent Vehicles Symposium in 2020, I presented our paper on Deep Classification-driven Domain Adaptation for Cross-Modal Driver Behavior Recognition in a virtual talk in the session on Driver State Recognition. |
Virtual presentation of ‘Every Annotation Counts: Multi-label Deep Supervision for Medical Image Segmentation’
Published:
 | At the CVPR 2021, I presented our paper on Multi-label Deep Supervision for Medical Image Segmentation in a virtual poster session. |
Presentation on Scarce Resources in Segmentation at the Zeiss Innovation Hub@KIT
Published:
 | I had the pleasure to present and discuss my doctoral research on semi-weakly supervised learning to the team at the Zeiss Innovation Hub@KIT. |
Talk on Semi- and Weakly Supervised Segmentation at Web Computing GmbH
Published:
 | I was invited to give a talk on resource-effective semantic segmentation to the AI Team at Web Computing GmbH, where I presented the research areas of semi- and weakly supervised learning. |
Poster presentation of ‘Graph-Constrained Contrastive Regularization for Semi-Weakly Volumetric Segmentation’
Published:
 | In Tel Aviv at ECCV 2022 I was fortunate to presented our paper on Graph-Constrained Contrastive Regularization for Semi-Weakly Volumetric Segmentation in person, with a lot of interested researchers stopping by. |
Two poster presentations and visiting the Doctoral Consortium at CVPR
Published:
 | In Vancouver at CVPR 2023 I presented two papers, first, the paper Delivering arbitrary-modal semantic segmentation on multi-modal street-scene segmentation by lead authors Jiaming Zhang and Ruiping Liu, and later I presented the paper Decoupled semantic prototypes enable learning from diverse annotation types for semi-weakly segmentation in expert-driven domains which explores techniques for training segmentation models with diverse annotation-types to ease the annotation process. I was also fortunate to attend the CVPR Doctoral Consortium, a wonderful event where I could present my work to Prof. René Vidal and discuss computer vision in clinical settings with him. |
Talk on Artificial Intelligence at SRH Holding
Published:
 | I gave a presentation at SRH Holding, talking to the representative body for disabled employees about fundamentals of AI, current advances and challenges as well as a brief overview of some applications of AI for accessibility. |
Three poster presentations at MICCAI Workshops
Published:
 | In Daejeon, at the MICCAI Workshops 2025, together with my former student Alessio Negrini I presented our paper Conquering the Retina: Bringing Visual in-Context Learning to OCT in the poster session of the Efficient Medical AI 2025 Workshop and later I presented our poster on Semantic Segmentation for Preoperative Planning in Transcatheter Aortic Valve Replacement in the context of the STACOM workshop. I further had the pleasure to present the paper GRASPing Anatomy to Improve Pathology Segmentation for my colleague Alexander Jaus and his student Keyi Li in the the MLMI workshop’s poster session. |
Poster presentation of ‘Is Visual in-Context Learning for Compositional Medical Tasks within Reach?’
Published:
 | In Honolulu, at the ICCV 2025, apart from attending exceptional workshops on a diverse set of topics such as world models, data-efficient learning and foundation models, I presented our paper Is Visual in-Context Learning for Compositional Medical Tasks within Reach? in one of the poster sessions. Engaging with many researchers at the conference has been a joy and I gathered a lot of inspiration through the phenomenal work happening in the community. |
teaching
Practical Course: Computer Vision for Human-Computer Interaction
3 ETCS, 2SWS Practical course, Karlsruhe Institute of Technology, 2024
I contributed as a supervisor to the installments: SS2022, SS2024
Seminar: Computer Vision for Human-Computer Interaction
3 ETCS, 2SWS Seminar, Karlsruhe Institute of Technology, 2024
I contributed as a supervisor to the installments: WS2021, WS2022, WS2023, WS2024
Seminar: Multimodal Large Language Models
3 ETCS, 2SWS Seminar, Karlsruhe Institute of Technology, 2025
I contributed as lecturer and supervisor to the installments: SS2024, SS2025
Lecture: Deep Learning for Computer Vision I: Basics
3 ETCS, 2SWS Lecture, Karlsruhe Institute of Technology, 2025
Lecture shared among multiple lecturers. I contributed to the installments: SS2020, SS2021, SS2024, SS2025
Lecture: Deep Learning for Computer Vision II: Advanced Topics
3 ETCS, 2SWS Lecture, Karlsruhe Institute of Technology, 2025
Main coordinator of the lecture since 2022, shared conception and lecturing since 2021. I contributed to the installments: WS2021, WS2022, WS2023, WS2024, WS2025
Theses: Master’s and Bachelor’s Thesis Projects
Day-to-day thesis supervision, Karlsruhe Institute of Technology, 2026
Frequent supervisor in master’s and bachelor’s thesis projects