Clinical testing and error correction
For Signals
As part of Stage 4, work continued on improving the efficiency and validation of the proposed algorithms. An algorithm for recognizing various ECG signal disorders based on information theory, utilizing the Lempel-Ziv complexity from 1976, was developed. A decision was made to apply a hybrid CNN-LSTM model due to its ability to effectively capture spatial and temporal features from ECG signals. The Convolutional Neural Network (CNN) specializes in extracting significant features from raw ECG data, which is crucial for identifying patterns and anomalies. Simultaneously, the Long Short-Term Memory (LSTM) network handles the sequential nature of ECG signals, allowing the model to learn temporal dependencies and their changes over time. This combination uses CNNs for reliable feature extraction and LSTM methods for understanding the dynamics of time series, resulting in an efficient and accurate approach to ECG classification. Additionally, autoencoders were used for feature extraction and anomaly detection, enabling the learning of a compressed representation of signals that can then be used for classification. The Adam optimizer was used for training the neural networks, ensuring efficient convergence through adaptive learning and regularization techniques to prevent overfitting and increase the model’s ability to generalize.
Based on the PTB-XL databases (standard 12-lead ECG recordings from Physikalisch Technische Bundesanstalt) and CPSC (10,330 standard 12-lead ECG recordings from the “China Physiological Signal Challenge”), quantitative features were defined for the CineECG algorithm, focusing on describing both WaveECG and PathECG. To confirm their usefulness, CineECG-based features were used to train a logistic regression model capable of classifying ECGs into four classes: normal sinus rhythm (NSR), left bundle branch block (LBBB), right bundle branch block (RBBB), and “undefined” (UND). Some data was excluded from the training process to ensure model generalization and prevent overfitting. The results showed that the model performs well across all classes, with particularly high recall values (95.31%, 92.43%) for the LBBB and classes, indicating strong classification performance. Balanced precision and recall values for the UND class indicate reliable performance in more ambiguous cases. High specificity values demonstrate effective identification of negative cases for each class, with nearly perfect specificity for LBBB. High recall values for NSR indicate effective identification of positive cases, except for the UND class.
For Images
Based on the collected databases, segmentation of the pancreas (in all patients) and tumors (in sick patients) was performed. Additionally, the superior mesenteric vein (SMV), splenic vein (SV), portal vein (PV), abdominal aorta, and celiac trunk with its branches were isolated and transformed into 3D masks using a combination of region-based segmentation to define organs and blood vessels as volume rendering. The YOLOv7 algorithm (You Only Look Once) was then trained to detect (determine regions of interest) the pancreas and tumors. Detection was performed in three planes: axial, coronal, and sagittal. The designated regions of interest were reconstructed in 3D and used to train a neural network with a 3D U-Net architecture for automatic segmentation.
The YOLO v7 algorithm was trained using an NVIDIA T4 GPU, while the U-Net was trained using Tesla A100, applying a loss function based on a combination of the DICE index and cross-entropy. Detection using the YOLO v7 algorithm was evaluated in terms of precision and sensitivity. Segmentation (U-Net) was evaluated using the DICE index, IoU coefficient, and voxel-level ROC curve along with AUC.
The segmentation model’s performance, expressed as the DICE index, was 0.73 (95% CI: 0.68–0.77) for healthy individuals and 0.72 (95% CI: 0.7–0.74) for cancer patients in the test database, independent of the training database. IoU values were 0.58 (95% CI: 0.53–0.63) and 0.57 (95% CI: 0.54–0.6), respectively. Voxel-level AUC values were 0.9865 (95% CI: 0.98646–0.98658) and 0.9898 (95% CI: 0.98976–0.98989), respectively.
Additional segmentation of CT images, independent of previous datasets (from patient recruitment within clinical trials), was also initiated for independent verification of the detection and segmentation capabilities of the developed models. The standard approach assumed the visualization of always the same structures: bones, arteries, veins (portal and systemic), liver, gallbladder, stomach, spleen, kidneys, and pancreas, along with masses described by radiology and separately marked areas of resection and potential anastomosis. These automatically segmented 3D models were presented using mixed reality devices, such as Microsoft HoloLens 2, for procedure planning.
Clinical Validation
Segmentation techniques were clinically validated. Forty-three patients were selected based on the availability of CT scans for planned liver, colorectal, gastrointestinal, pancreatic, or stomach surgeries. The selection was made according to the order of submissions, without considering socio-economic factors. Patients gave informed consent to participate in the study (UJ bioethics committee approval, No. 1072.6120.92.2022).
Anonymized CT scans in DICOM format were used. Anatomical points were normalized in all cases. Anatomical structures were marked and segmented using YOLO v7 in dedicated 3D software. Segmented structures were transformed into colored segments and combined into 3D files in STL format (spatial data). The models were manually corrected, decompressed to reduce file size, and stored in a cloud compliant with medical data privacy requirements/server.
The models were visualized in augmented reality using Microsoft HoloLens 2. Doctors controlled the display of holograms, and patients viewed them. Patients completed two standard questionnaires assessing stress and anxiety before and after viewing the holograms.
The results were analyzed for statistical significance to determine whether holographic visualization provided subjective benefits in terms of increased understanding and reduced stress. This methodology allowed for the integration of advanced visualization techniques in patient education, potentially increasing understanding and reducing preoperative stress.
Implementation Testing
The proposed medical data visualization technology can be implemented in any field of medicine. Therefore, implementation tests were conducted in socio-economic collaboration. Tests of modern mixed reality (MR) technology were carried out at the LiderDent medical center in Kraków (https://liderdent.pl) concerning endodontic treatment. Using 3D holography based on CT data, it became possible to create precise three-dimensional models of patients’ teeth. These innovative models allowed dentists to plan and simulate procedures more accurately before performing them, contributing to increased treatment effectiveness.
Analysis based on five randomly selected CBCT scans showed that using MR technology saves about 52% of the dentist’s time compared to traditional methods (approval number of the Regional Medical Chamber: 14/KBL/OIL/2022). This technology allows the dentist to obtain a more detailed anatomical view of root canals, which is crucial for effective root canal treatment. The introduction of mixed reality technology not only increases efficiency but also significantly shortens the procedure’s duration, which benefits both patients and specialists.
