TY - CPAPER TI - Stylizing ViT : Anatomy-Preserving Instance Style Transfer for Domain Generalization AU - Doerrich, Sebastian AU - Di Salvo, Francesco AU - Alle, Jonas AU - Ledig, Christian PY - 2026 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/116855 LA - eng AB - Deep learning models in medical image analysis often struggle with generalizability across domains and demographic groups due to data heterogeneity and scarcity. Traditional augmentation improves robustness, but fails under substantial domain shifts. Recent advances in stylistic augmentation enhance domain generalization by varying image styles but fall short in terms of style diversity or by introducing artifacts into the generated images. To address these limitations, we propose Stylizing ViT, a novel Vision Transformer encoder that utilizes weight-shared attention blocks for both self- and cross-attention. This design allows the same attention block to maintain anatomical consistency through self-attention while performing style transfer via cross-attention. We assess the effectiveness of our method for domain generalization by employing it for data augmentation on three distinct image classification tasks in the context of histopathology and dermatology. Results demonstrate an improved robustness (up to +13% accuracy) over the state of the art while generating perceptually convincing images without artifacts. Additionally, we show that Stylizing ViT is effective beyond training, achieving a 17% performance improvement during inference when used for test-time augmentation. The source code is available at https://github.com/sdoerrich97/stylizing-vit . KW - style transfer KW - domain generalization KW - crossattention KW - data augmentation KW - vision transformer ER - TY - CPAPER TI - Stylizing ViT : Anatomy-Preserving Instance Style Transfer for Domain Generalization AU - Doerrich, Sebastian AU - Di Salvo, Francesco AU - Alle, Jonas AU - Ledig, Christian PY - 2026 T2 - 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI) PB - IEEE CY - Piscataway, NJ DO - 10.1109/isbi61048.2026.11515498 UR - https://fis.uni-bamberg.de/handle/uniba/117126 LA - eng AB - Deep learning models in medical image analysis often struggle with generalizability across domains and demographic groups due to data heterogeneity and scarcity. Traditional augmentation improves robustness, but fails under substantial domain shifts. Recent advances in stylistic augmentation enhance domain generalization by varying image styles but fall short in terms of style diversity or by introducing artifacts into the generated images. To address these limitations, we propose Stylizing ViT, a novel Vision Transformer encoder that utilizes weight-shared attention blocks for both self- and cross-attention. This design allows the same attention block to maintain anatomical consistency through self-attention while performing style transfer via cross-attention. We assess the effectiveness of our method for domain generalization by employing it for data augmentation on three distinct image classification tasks in the context of histopathology and dermatology. Results demonstrate an improved robustness (up to +13% accuracy) over the state of the art while generating perceptually convincing images without artifacts. Additionally, we show that Stylizing ViT is effective beyond training, achieving a 17% performance improvement during inference when used for test-time augmentation. The source code is available at https://github.com/sdoerrich97/stylizing-vit . KW - style transfer KW - domain generalization KW - crossattention KW - data augmentation KW - vision transformer ER - TY - JOUR TI - An Embedding is Worth a Thousand Noisy Labels AU - Di Salvo, Francesco AU - Doerrich, Sebastian AU - Ledig, Christian AU - Rieger, Ines PY - 2025 SP - 1-19 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/109531 LA - eng AB - The performance of deep neural networks scales with dataset size and label quality, rendering the efficient mitigation of low-quality data annotations crucial for building robust and cost-effective systems. Existing strategies to address label noise exhibit severe limitations due to computational complexity and application dependency. In this work, we propose WANN, a Weighted Adaptive Nearest Neighbor approach that builds on self-supervised feature representations obtained from foundation models. To guide the weighted voting scheme, we introduce a reliability score, which measures the likelihood of a data label being correct. WANN outperforms reference methods, including a linear layer trained with robust loss functions, on diverse datasets of varying size and under various noise types and severities. WANN also exhibits superior generalization on imbalanced data compared to both Adaptive-NNs (ANN) and fixed k-NNs. Furthermore, the proposed weighting scheme enhances supervised dimensionality reduction under noisy labels. This yields a significant boost in classification performance with 10x and 100x smaller image embeddings, minimizing latency and storage requirements. Our approach, emphasizing efficiency and explainability, emerges as a simple, robust solution to overcome inherent limitations of deep neural network training. KW - Adaptive kNN KW - Label noise KW - Robustness KW - Foundation models ER - TY - CPAPER TI - MedMNIST-C : Comprehensive benchmark and improved classifier robustness by simulating realistic image corruptions AU - Di Salvo, Francesco AU - Doerrich, Sebastian AU - Ledig, Christian PY - 2025 SP - 1-11 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/109532 LA - eng AB - The integration of neural-network-based systems into clinical practice is limited by challenges related to domain generalization and robustness. The computer vision community established benchmarks such as ImageNet-C as a fundamental prerequisite to measure progress towards those challenges. Similar datasets are largely absent in the medical imaging community which lacks a comprehensive benchmark that spans across imaging modalities and applications. To address this gap, we create and open-source MedMNIST-C, a benchmark dataset based on the MedMNIST+ collection covering 12 datasets and 9 imaging modalities. We simulate task and modality-specific image corruptions of varying severity to comprehensively evaluate the robustness of established algorithms against real-world artifacts and distribution shifts. We further provide quantitative evidence that our simple-to-use artificial corruptions allow for highly performant, lightweight data augmentation to enhance model robustness. Unlike traditional, generic augmentation strategies, our approach leverages domain knowledge, exhibiting significantly higher robustness when compared to widely adopted methods. By introducing MedMNIST-C and open-sourcing the corresponding library allowing for targeted data augmentations, we contribute to the development of increasingly robust methods tailored to the challenges of medical imaging. KW - Robustness KW - Corruptions KW - Data augmentation ER - TY - CPAPER TI - Unsupervised Feature Orthogonalization for Learning Distortion-Invariant Representations AU - Doerrich, Sebastian AU - Di Salvo, Francesco AU - Ledig, Christian PY - 2025 SP - 1-13 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/109529 LA - eng AB - This study introduces unORANIC+, a novel method that integrates unsupervised feature orthogonalization with the ability of a Vision Transformer to capture both local and global relationships for improved robustness and generalizability. The streamlined architecture of unORANIC+ effectively separates anatomical and image-specific attributes, resulting in robust and unbiased latent representations that allow the model to demonstrate excellent performance across various medical image analysis tasks and diverse datasets. Extensive experimentation demonstrates unORANIC+’s reconstruction proficiency, corruption resilience, as well as capability to revise existing image distortions. Additionally, the model exhibits notable aptitude in downstream tasks such as disease classification and corruption detection. We confirm its adaptability to diverse datasets of varying image sources and sample sizes which positions the method as a promising algorithm for advanced medical image analysis, particularly in resourceconstrained environments lacking large, tailored datasets. The source code is available at github.com/sdoerrich97/unoranic-plus. KW - feature orthogonalization KW - robustness KW - corruption revision KW - unsupervised learning KW - generalization ER - TY - CPAPER TI - Privacy-preserving datasets by capturing feature distributions with Conditional VAEs AU - Di Salvo, Francesco AU - Tafler, David E. AU - Doerrich, Sebastian AU - Ledig, Christian PY - 2025 SP - 1-14 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/109530 LA - eng AB - Large and well-annotated datasets are essential for advancing deep learning applications, however often costly or impossible to obtain by a single entity. In many areas, including the medical domain, approaches relying on data sharing have become critical to address those challenges. While effective in increasing dataset size and diversity, data sharing raises significant privacy concerns. Commonly employed anonymization methods based on the k-anonymity paradigm often fail to preserve data diversity, affecting model robustness. This work introduces a novel approach using Conditional Variational Autoencoders (CVAEs) trained on feature vectors extracted from large pre-trained vision foundation models. Foundation models effectively detect and represent complex patterns across diverse domains, allowing the CVAE to faithfully capture the embedding space of a given data distribution to generate (sample) a diverse, privacy-respecting, and potentially unbounded set of synthetic feature vectors. Our method notably outperforms traditional approaches in both medical and natural image domains, exhibiting greater dataset diversity and higher robustness against perturbations while preserving sample privacy. These results underscore the potential of generative models to significantly impact deep learning applications in data-scarce and privacy-sensitive environments. KW - Generative models KW - Privacy KW - Foundation models KW - Robustness ER - TY - JOUR TI - An Embedding is Worth a Thousand Noisy Labels AU - Di Salvo, Francesco AU - Doerrich, Sebastian AU - Rieger, Ines AU - Ledig, Christian PY - 2025 JO - Transactions on machine learning research IS - 3 SP - 1-19 PB - OpenReview.net CY - Amherst, Massachusetts SN - 2835-8856 UR - https://fis.uni-bamberg.de/handle/uniba/109535 LA - eng AB - The performance of deep neural networks scales with dataset size and label quality, rendering the efficient mitigation of low-quality data annotations crucial for building robust and cost-effective systems. Existing strategies to address label noise exhibit severe limitations due to computational complexity and application dependency. In this work, we propose WANN, a Weighted Adaptive Nearest Neighbor approach that builds on self-supervised feature representations obtained from foundation models. To guide the weighted voting scheme, we introduce a reliability score, which measures the likelihood of a data label being correct. WANN outperforms reference methods, including a linear layer trained with robust loss functions, on diverse datasets of varying size and under various noise types and severities. WANN also exhibits superior generalization on imbalanced data compared to both Adaptive-NNs (ANN) and fixed k-NNs. Furthermore, the proposed weighting scheme enhances supervised dimensionality reduction under noisy labels. This yields a significant boost in classification performance with 10x and 100x smaller image embeddings, minimizing latency and storage requirements. Our approach, emphasizing efficiency and explainability, emerges as a simple, robust solution to overcome inherent limitations of deep neural network training. KW - Adaptive kNN KW - Label noise KW - Robustness KW - Foundation models ER - TY - JOUR TI - Rethinking model prototyping through the MedMNIST+ dataset collection AU - Doerrich, Sebastian AU - Di Salvo, Francesco AU - Brockmann, Julius AU - Ledig, Christian PY - 2025 SP - 1-15 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/107030 LA - eng AB - The integration of deep learning based systems in clinical practice is often impeded by challenges rooted in limited and heterogeneous medical datasets. In addition, the field has increasingly prioritized marginal performance gains on a few, narrowly scoped benchmarks over clinical applicability, slowing down meaningful algorithmic progress. This trend often results in excessive fine-tuning of existing methods on selected datasets rather than fostering clinically relevant innovations. In response, this work introduces a comprehensive benchmark for the MedMNIST+ dataset collection, designed to diversify the evaluation landscape across several imaging modalities, anatomical regions, classification tasks and sample sizes. We systematically reassess commonly used Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) architectures across distinct medical datasets, training methodologies, and input resolutions to validate and refine existing assumptions about model effectiveness and development. Our findings suggest that computationally efficient training schemes and modern foundation models offer viable alternatives to costly end-to-end training. Additionally, we observe that higher image resolutions do not consistently improve performance beyond a certain threshold. This highlights the potential benefits of using lower resolutions, particularly in prototyping stages, to reduce computational demands without sacrificing accuracy. Notably, our analysis reaffirms the competitiveness of CNNs compared to ViTs, emphasizing the importance of comprehending the intrinsic capabilities of different architectures. Finally, by establishing a standardized evaluation framework, we aim to enhance transparency, reproducibility, and comparability within the MedMNIST+ dataset collection as well as future research. Code is available at github.com/sdoerrich97/rethinking-model-prototyping-MedMNISTPlus. KW - Benchmarking KW - Prototyping recommendations KW - Medical image classification KW - Standardized evaluation framework KW - Foundation models ER - TY - CPAPER TI - unORANIC : Unsupervised Orthogonalization of Anatomy and Image-Characteristic Features AU - Dörrich, Sebastian AU - Di Salvo, Francesco AU - Ledig, Christian PY - 2025 SP - 1-11 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/106934 LA - eng AB - We introduce unORANIC, an unsupervised approach that uses an adapted loss function to drive the orthogonalization of anatomy and image-characteristic features. The method is versatile for diverse modalities and tasks, as it does not require domain knowledge, paired data samples, or labels. During test time unORANIC is applied to potentially corrupted images, orthogonalizing their anatomy and characteristic components, to subsequently reconstruct corruption-free images, showing their domain-invariant anatomy only. This feature orthogonalization further improves generalization and robustness against corruptions. We confirm this qualitatively and quantitatively on 5 distinct datasets by assessing unORANIC’s classification accuracy, corruption detection and revision capabilities. Our approach shows promise for enhancing the generalizability and robustness of practical applications in medical image analysis. The source code is available at github.com/sdoerrich97/unORANIC. KW - feature orthogonalization KW - robustness KW - corruption revision KW - unsupervised learning KW - generalization ER - TY - JOUR TI - Rethinking model prototyping through the MedMNIST+ dataset collection AU - Doerrich, Sebastian AU - Di Salvo, Francesco AU - Brockmann, Julius AU - Ledig, Christian PY - 2025 JO - Scientific reports VL - 15 IS - 1, 7669 SP - 1-15 PB - Springer Nature CY - London SN - 2045-2322 DO - 10.1038/s41598-025-92156-9 UR - https://fis.uni-bamberg.de/handle/uniba/107047 LA - eng AB - The integration of deep learning based systems in clinical practice is often impeded by challenges rooted in limited and heterogeneous medical datasets. In addition, the field has increasingly prioritized marginal performance gains on a few, narrowly scoped benchmarks over clinical applicability, slowing down meaningful algorithmic progress. This trend often results in excessive fine-tuning of existing methods on selected datasets rather than fostering clinically relevant innovations. In response, this work introduces a comprehensive benchmark for the MedMNIST+ dataset collection, designed to diversify the evaluation landscape across several imaging modalities, anatomical regions, classification tasks and sample sizes. We systematically reassess commonly used Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) architectures across distinct medical datasets, training methodologies, and input resolutions to validate and refine existing assumptions about model effectiveness and development. Our findings suggest that computationally efficient training schemes and modern foundation models offer viable alternatives to costly end-to-end training. Additionally, we observe that higher image resolutions do not consistently improve performance beyond a certain threshold. This highlights the potential benefits of using lower resolutions, particularly in prototyping stages, to reduce computational demands without sacrificing accuracy. Notably, our analysis reaffirms the competitiveness of CNNs compared to ViTs, emphasizing the importance of comprehending the intrinsic capabilities of different architectures. Finally, by establishing a standardized evaluation framework, we aim to enhance transparency, reproducibility, and comparability within the MedMNIST+ dataset collection as well as future research. Code is available at github.com/sdoerrich97/rethinking-model-prototyping-MedMNISTPlus. KW - Benchmarking KW - Prototyping recommendations KW - Medical image classification KW - Standardized evaluation framework KW - Foundation models ER - TY - CPAPER TI - Privacy-preserving datasets by capturing feature distributions with Conditional VAEs AU - Di Salvo, Francesco AU - Tafler, David AU - Doerrich, Sebastian AU - Ledig, Christian PY - 2024 T2 - Proceedings : The 35th British Machine Vision Conference 2024 SP - 1-14 UR - https://fis.uni-bamberg.de/handle/uniba/109551 LA - eng AB - Large and well-annotated datasets are essential for advancing deep learning applications, however often costly or impossible to obtain by a single entity. In many areas, including the medical domain, approaches relying on data sharing have become critical to address those challenges. While effective in increasing dataset size and diversity, data sharing raises significant privacy concerns. Commonly employed anonymization methods based on the k-anonymity paradigm often fail to preserve data diversity, affecting model robustness. This work introduces a novel approach using Conditional Variational Autoencoders (CVAEs) trained on feature vectors extracted from large pre-trained vision foundation models. Foundation models effectively detect and represent complex patterns across diverse domains, allowing the CVAE to faithfully capture the embedding space of a given data distribution to generate (sample) a diverse, privacy-respecting, and potentially unbounded set of synthetic feature vectors. Our method notably outperforms traditional approaches in both medical and natural image domains, exhibiting greater dataset diversity and higher robustness against perturbations while preserving sample privacy. These results underscore the potential of generative models to significantly impact deep learning applications in data-scarce and privacy-sensitive environments. KW - Generative models KW - Privacy KW - Foundation models KW - Robustness ER - TY - CPAPER TI - Unsupervised Feature Orthogonalization for Learning Distortion-Invariant Representations AU - Doerrich, Sebastian AU - Di Salvo, Francesco AU - Ledig, Christian PY - 2024 T2 - Workshop Proceedings : International Workshop on Robust Recognition in the Open World at The 35th British Machine Vision Conference SP - 1-13 UR - https://fis.uni-bamberg.de/handle/uniba/109550 LA - eng AB - This study introduces unORANIC+, a novel method that integrates unsupervised feature orthogonalization with the ability of a Vision Transformer to capture both local and global relationships for improved robustness and generalizability. The streamlined architecture of unORANIC+ effectively separates anatomical and image-specific attributes, resulting in robust and unbiased latent representations that allow the model to demonstrate excellent performance across various medical image analysis tasks and diverse datasets. Extensive experimentation demonstrates unORANIC+’s reconstruction proficiency, corruption resilience, as well as capability to revise existing image distortions. Additionally, the model exhibits notable aptitude in downstream tasks such as disease classification and corruption detection. We confirm its adaptability to diverse datasets of varying image sources and sample sizes which positions the method as a promising algorithm for advanced medical image analysis, particularly in resourceconstrained environments lacking large, tailored datasets. The source code is available at github.com/sdoerrich97/unoranic-plus. KW - feature orthogonalization KW - robustness KW - corruption revision KW - unsupervised learning KW - generalization ER - TY - CPAPER TI - MedMNIST-C : Comprehensive benchmark and improved classifier robustness by simulating realistic image corruptions AU - Di Salvo, Francesco AU - Doerrich, Sebastian AU - Ledig, Christian PY - 2024 SP - 1-11 DO - 10.48550/arXiv.2406.17536 UR - https://fis.uni-bamberg.de/handle/uniba/109539 LA - eng AB - The integration of neural-network-based systems into clinical practice is limited by challenges related to domain generalization and robustness. The computer vision community established benchmarks such as ImageNet-C as a fundamental prerequisite to measure progress towards those challenges. Similar datasets are largely absent in the medical imaging community which lacks a comprehensive benchmark that spans across imaging modalities and applications. To address this gap, we create and open-source MedMNIST-C, a benchmark dataset based on the MedMNIST+ collection covering 12 datasets and 9 imaging modalities. We simulate task and modality-specific image corruptions of varying severity to comprehensively evaluate the robustness of established algorithms against real-world artifacts and distribution shifts. We further provide quantitative evidence that our simple-to-use artificial corruptions allow for highly performant, lightweight data augmentation to enhance model robustness. Unlike traditional, generic augmentation strategies, our approach leverages domain knowledge, exhibiting significantly higher robustness when compared to widely adopted methods. By introducing MedMNIST-C and open-sourcing the corresponding library allowing for targeted data augmentations, we contribute to the development of increasingly robust methods tailored to the challenges of medical imaging. KW - Robustness KW - Corruptions KW - Data augmentation ER - TY - CPAPER TI - Self-supervised Vision Transformer are Scalable Generative Models for Domain Generalization AU - Doerrich, Sebastian AU - Di Salvo, Francesco AU - Ledig, Christian PY - 2024 SP - 1-10 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/104846 LA - eng AB - Despite notable advancements, the integration of deep learning (DL) techniques into impactful clinical applications, particularly in the realm of digital histopathology, has been hindered by challenges associated with achieving robust generalization across diverse imaging domains and characteristics. Traditional mitigation strategies in this field such as data augmentation and stain color normalization have proven insufficient in addressing this limitation, necessitating the exploration of alternative methodologies. To this end, we propose a novel generative method for domain generalization in histopathology images. Our method employs a generative, self-supervised Vision Transformer to dynamically extract characteristics of image patches and seamlessly infuse them into the original images, thereby creating novel, synthetic images with diverse attributes. By enriching the dataset with such synthesized images, we aim to enhance its holistic nature, facilitating improved generalization of DL models to unseen domains. Extensive experiments conducted on two distinct histopathology datasets demonstrate the effectiveness of our proposed approach, outperforming the state of the art substantially, on the Camelyon17-wilds challenge dataset (+2%) and on a second epithelium-stroma dataset (+26%). Furthermore, we emphasize our method's ability to readily scale with increasingly available unlabeled data samples and more complex, higher parametric architectures. Source code is available at https://github.com/sdoerrich97/vits-are-generative-models. KW - domain generalization KW - self-supervised learning KW - feature orthogonalization KW - generative image synthesis ER - TY - CPAPER TI - Self-supervised Vision Transformer are Scalable Generative Models for Domain Generalization AU - Doerrich, Sebastian AU - Di Salvo, Francesco AU - Ledig, Christian PY - 2024 T2 - Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 SP - 644-654 PB - Springer Nature Switzerland CY - Cham SN - 978-3-031-72117-5 DO - 10.1007/978-3-031-72117-5_60 UR - https://fis.uni-bamberg.de/handle/uniba/105289 LA - eng AB - Despite notable advancements, the integration of deep learning (DL) techniques into impactful clinical applications, particularly in the realm of digital histopathology, has been hindered by challenges associated with achieving robust generalization across diverse imaging domains and characteristics. Traditional mitigation strategies in this field such as data augmentation and stain color normalization have proven insufficient in addressing this limitation, necessitating the exploration of alternative methodologies. To this end, we propose a novel generative method for domain generalization in histopathology images. Our method employs a generative, self-supervised Vision Transformer to dynamically extract characteristics of image patches and seamlessly infuse them into the original images, thereby creating novel, synthetic images with diverse attributes. By enriching the dataset with such synthesized images, we aim to enhance its holistic nature, facilitating improved generalization of DL models to unseen domains. Extensive experiments conducted on two distinct histopathology datasets demonstrate the effectiveness of our proposed approach, outperforming the state of the art substantially, on the Camelyon17-wilds challenge dataset (+2%) and on a second epithelium-stroma dataset (+26%). Furthermore, we emphasize our method's ability to readily scale with increasingly available unlabeled data samples and more complex, higher parametric architectures. Source code is available at https://github.com/sdoerrich97/vits-are-generative-models. KW - domain generalization KW - self-supervised learning KW - feature orthogonalization KW - generative image synthesis ER - TY - CPAPER TI - Integrating kNN with Foundation Models for Adaptable and Privacy-Aware Image Classification AU - Doerrich, Sebastian AU - Archut, Tobias AU - Di Salvo, Francesco AU - Ledig, Christian PY - 2024 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/104752 LA - eng AB - Traditional deep learning models implicitly encode knowledge limiting their transparency and ability to adapt to data changes. Yet, this adaptability is vital for addressing user data privacy concerns. We address this limitation by storing embeddings of the underlying training data independently of the model weights, enabling dynamic data modifications without retraining. Specifically, our approach integrates the k-Nearest Neighbor (k-NN) classifier with a vision-based foundation model, pre-trained self-supervised on natural images, enhancing interpretability and adaptability. We share open-source implementations of a previously unpublished baseline method as well as our performance-improving contributions. Quantitative experiments confirm improved classification across established benchmark datasets and the method’s applicability to distinct medical image classification tasks. Additionally, we assess the method’s robustness in continual learning and data removal scenarios. The approach exhibits great promise for bridging the gap between foundation models’ performance and challenges tied to data privacy. The source code is available at github.com/TobArc. KW - k-NN classifier KW - continual learning KW - transfer learning KW - few-shot classification KW - explainability ER - TY - CPAPER TI - Integrating kNN with Foundation Models for Adaptable and Privacy-Aware Image Classification AU - Doerrich, Sebastian AU - Archut, Tobias AU - Di Salvo, Francesco AU - Ledig, Christian PY - 2024 T2 - 2024 IEEE International Symposium on Biomedical Imaging (ISBI) SP - 1-5 PB - IEEE DO - 10.1109/isbi56570.2024.10635560 UR - https://fis.uni-bamberg.de/handle/uniba/104176 LA - eng AB - Traditional deep learning models implicitly encode knowledge limiting their transparency and ability to adapt to data changes. Yet, this adaptability is vital for addressing user data privacy concerns. We address this limitation by storing embeddings of the underlying training data independently of the model weights, enabling dynamic data modifications without retraining. Specifically, our approach integrates the k-Nearest Neighbor (k-NN) classifier with a vision-based foundation model, pre-trained self-supervised on natural images, enhancing interpretability and adaptability. We share open-source implementations of a previously unpublished baseline method as well as our performance-improving contributions. Quantitative experiments confirm improved classification across established benchmark datasets and the method’s applicability to distinct medical image classification tasks. Additionally, we assess the method’s robustness in continual learning and data removal scenarios. The approach exhibits great promise for bridging the gap between foundation models’ performance and challenges tied to data privacy. The source code is available at github.com/TobArc. KW - k-NN classifier KW - continual learning KW - transfer learning KW - few-shot classification KW - explainability ER - TY - CPAPER TI - unORANIC : Unsupervised Orthogonalization of Anatomy and Image-Characteristic Features AU - Doerrich, Sebastian AU - Di Salvo, Francesco AU - Ledig, Christian PY - 2023 T2 - Machine Learning in Medical Imaging : 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 8, 2023 ; Proceedings, Part I SP - 62-71 PB - Springer Nature Switzerland CY - Cham SN - 978-3-031-45672-5 DO - 10.1007/978-3-031-45673-2_7 UR - https://fis.uni-bamberg.de/handle/uniba/91842 LA - eng AB - We introduce unORANIC, an unsupervised approach that uses an adapted loss function to drive the orthogonalization of anatomy and image-characteristic features. The method is versatile for diverse modalities and tasks, as it does not require domain knowledge, paired data samples, or labels. During test time unORANIC is applied to potentially corrupted images, orthogonalizing their anatomy and characteristic components, to subsequently reconstruct corruption-free images, showing their domain-invariant anatomy only. This feature orthogonalization further improves generalization and robustness against corruptions. We confirm this qualitatively and quantitatively on 5 distinct datasets by assessing unORANIC’s classification accuracy, corruption detection and revision capabilities. Our approach shows promise for enhancing the generalizability and robustness of practical applications in medical image analysis. The source code is available at github.com/sdoerrich97/unORANIC. KW - feature orthogonalization KW - robustness KW - corruption revision KW - unsupervised learning KW - generalization ER - TY - JOUR TI - Fast 3D YOLOv3 based standard plane regression of vertebral bodies in intra-operative CBCT volumes AU - Dörrich, Sebastian AU - Kordon, Florian AU - Denzinger, Felix AU - El Barbari, Jan S. AU - Privalov, Maxim AU - Vetter, Sven Y. AU - Maier, Andreas AU - Kunze, Holger PY - 2023 SP - 1-15 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/59864 LA - eng AB - Purpose Mobile C-arm systems represent the standard imaging devices within the field of spine surgery. In addition to 2D imaging, they allow for 3D scans while preserving unrestricted patient access. For viewing, the acquired volumes are adjusted such that their anatomical standard planes align with the axes of the viewing modality. This difficult and time-consuming step is currently performed manually by the leading surgeon. This process is automatized within this work to improve the usability of C-arm systems. Thereby, the spinal region consisting of multiple vertebrae and the standard planes of all vertebrae being of interest to the surgeon need to be taken into account. Approach An object detection algorithm based on the you only look once version 3 architecture, adapted to 3D inputs, is compared with a segmentation-based approach employing a 3D U-Net. Both algorithms are trained on a dataset of 440 and tested on 218 spinal volumes. Results Although the detection-based algorithm is slightly inferior concerning the detection (91% versus 97% accuracy), localization (1.26 mm versus 0.74 mm error) and alignment accuracy (5.00 deg versus 4.73 deg error), it outperforms the segmentation-based one in terms of speed (5 s versus 38 s). Conclusions Both algorithms show similar good results. However, the speed gain of the detection-based algorithm, resulting in a run time of 5 s, makes it more suitable for usage in an intra-operative scenario. KW - 3D imaging standards KW - Object detection KW - Image segmentation KW - Cone beam computed tomography KW - Surgery KW - Detection and tracking algorithms KW - Education and training KW - Anatomy KW - Spine KW - Voxels ER - TY - JOUR TI - Fast 3D YOLOv3 based standard plane regression of vertebral bodies in intra-operative CBCT volumes AU - Dörrich, Sebastian AU - Kordon, Florian AU - Denzinger, Felix AU - El Barbari, Jan S. AU - Privalov, Maxim AU - Vetter, Sven Y. AU - Maier, Andreas AU - Kunze, Holger PY - 2023 JO - Journal of medical imaging : JMI VL - 10 IS - 3 PB - SPIE CY - Bellingham, Wash. SN - 2329-4310 DO - 10.1117/1.JMI.10.3.034503 UR - https://fis.uni-bamberg.de/handle/uniba/59826 LA - eng AB - Purpose Mobile C-arm systems represent the standard imaging devices within the field of spine surgery. In addition to 2D imaging, they allow for 3D scans while preserving unrestricted patient access. For viewing, the acquired volumes are adjusted such that their anatomical standard planes align with the axes of the viewing modality. This difficult and time-consuming step is currently performed manually by the leading surgeon. This process is automatized within this work to improve the usability of C-arm systems. Thereby, the spinal region consisting of multiple vertebrae and the standard planes of all vertebrae being of interest to the surgeon need to be taken into account. Approach An object detection algorithm based on the you only look once version 3 architecture, adapted to 3D inputs, is compared with a segmentation-based approach employing a 3D U-Net. Both algorithms are trained on a dataset of 440 and tested on 218 spinal volumes. Results Although the detection-based algorithm is slightly inferior concerning the detection (91% versus 97% accuracy), localization (1.26 mm versus 0.74 mm error) and alignment accuracy (5.00 deg versus 4.73 deg error), it outperforms the segmentation-based one in terms of speed (5 s versus 38 s). Conclusions Both algorithms show similar good results. However, the speed gain of the detection-based algorithm, resulting in a run time of 5 s, makes it more suitable for usage in an intra-operative scenario. KW - 3D imaging standards KW - Object detection KW - Image segmentation KW - Cone beam computed tomography KW - Surgery KW - Detection and tracking algorithms KW - Education and training KW - Anatomy KW - Spine KW - Voxels ER -