Artificial Intelligence and Computer Vision Lab (AICV)
Computer Vision for Clinical Research
Clinical-oriented research has a high requirement for accuracy, reproducibility, and time efficiency. For example, knee cartilage only experiences 2~4% change on Magnetic Resonance Imaging (MRI) during the two-year clinical trial period. The traditional algorithms cannot detect such subtle changes. During my postdoctoral training period, I have developed innovative computer algorithms to efficiently measure multiple image biomarkers (cartilage, bone marrow lesion, effusion, etc.) from 3-dimensional (3D) knee MR images [1-7]. Those measurements greatly improved efficiency, accuracy, and reproducibility and have been successfully applied to four federal-funded clinical trials/grants (R01AR054938, R01AR057802, R01AR060718, U01AR067168). The results have been published in the high impact journals (e.g., JAMA, BMC, etc.) [5-8]
Ongoing Projects
Knee Osteoarthritis (OA)
Bone Marrow Lesion (BML) segmentation from knee MRI, paper
Student: (Michelle Wang, Xinyue Sun, Zhuoran Xu)
Effusion segmentation from knee MRI
Student: (Mohammad Chowdhury)
Knee bone segmentation on MRI (femur, tibia, patella), paper
Student: (Rania Mohammedameen)
Knee Osteoarthritis (OA)
Joint Space Width (JSW) measure using regression on X-ray
Student: (Zhiheng Chang, Raj Ponnusamy)
Finger joint segmentation
Student: (Jordan blackadar, compare with YOLO and faster R-CNN)
Dip, PIP, MCP segmentation
Student: (Tino Cheung, Michelle Wang, Xinyue Sun, Hetali Tank)
Whole pipeline, paper
Student: (Zhaowei Gu, Yuan Gao)
Medical Image Segmentation Based on Deep Learning Algorithms
Image segmentation is the critical step in computer-aided diagnosis (CAD) systems. The traditional medical image segmentation algorithms need to specify the region of interest (ROI) and/or specifically adjust parameters for various image data sets to acquire reasonable accuracy. I have developed a novel segmentation framework based on deep learning architecture which learns radiologist’s segmentation experience. The advantage of this ML framework is to use less training samples (usually less than 100 samples) and it is specifically designed for medical image learning. I have successfully applied this ML framework to breast ultrasound (BUS) image segmentation and it outperformed the other state of the art methods significantly with the dice coefficient = 0.825 [9]. Experiment results showed that my method is more robust and accurate in handling various image datasets.
3D Image-to-Image Deep Learning Predictive Model
Disease progression prediction can help early detect those diseases. I expanded machine learning approaches to build a novel 3D-information-fusion mechanism for medical imaging and created a novel 3D image-to-image predictive model using the deep neural network as the core. I utilized a large image database and created a 3D model to predict the knee osteoarthritis (OA) disease progression on MRI. I achieved AUC > 0.8 in the prediction of two-year’s medial and lateral knee OA progression and received a National Science Foundation grant (NSF1723429) as the Principal Investigator to support this study [10-12]
NSF:3D Machine Learning prediction model
Machine Learning Pipeline to Automatically Generate Biomarkers
The advent of MRI offered new opportunities to study knee OA. National Institute of Health (NIH) spent over 10 years collecting a wealth of data on 4,800 participants with an eight-year span, including annual knee MRIs to create the Osteoarthritis Initiative (OAI) image database. However, there is no existing method to comprehensively analyze this image database due to the complexity of the knee structure. I am developing an automatic pipeline by using a stepwise procedure of increasing difficulty and utilize 3D machine learning segmentation algorithm I previously created to efficiently and accurately measure the whole knee structural features on MR images. I use data science FAIR (findable, accessible, interoperable, and reusable) rule to automatically generate a huge number of new features including morphological (volume, surface area, thickness, region length), intensity (mean intensity, histogram, intensity variation), texture (homogeneity, contrast, dissimilarity), and shape (smoothness, convex) features, which will be used as big data to study the pathology of the OA disease
Machine learning pipeline to automatically generate image biomarker
Multimodal Big Data Analysis
​
Knee OA impacts more than 27 million individuals in the US. Despite its importance for public health, the pathology of OA disease is still unclear. OA is viewed as a ‘whole-organ’ disorder manifesting damage to a range of soft-tissue and articular structures, such as hyaline cartilage, peri-articular bone, ligaments, and tendons. However, most of the OA studies focus only on individual image biomarker due to the technical barriers of quantitative measurements. Using individual image biomarker has been thought to be a major impediment to the rapid evaluation of promising treatments and study the pathology of knee OA disease. For example, in a recently completed knee OA clinical trial, the treatment and placebo groups were compared separately using cartilage, BML, and effusions measured from MRI which all gave non-significant results [8].
​
​I recently launched a multimodal machine learning study which provides an innovative way to combine multimodal big data to study OA disease. My study showed that the best single image biomarker to predict knee OA disease in two years is CDI (AUC = 0.773). The composition of three image biomarkers (CDI + BML + Effusion) received better prediction results (AUC = 0.791). By adding clinical data to image biomarkers, the prediction was stepwise improved (AUC = 0.823 by adding age). The best prediction was multimodal data which included all three biomarkers and two clinical data (CDI + BML + Effusion + Age + BMI, AUC = 0.843) by using Artificial Neural Network. My experiment results indicated that multimodal machine learning provides a powerful way to study OA disease and it helps future clinical trial find more responsive endpoints to facilitate the development and testing of new therapies that may affect OA progression and optimize trial efficiency. The OAI database has rich clinical data, physical function data, and biospecimen data in addition to MRI images. In my future study, I will create multimodal machine learning models to integrate image biomarkers, clinical data, and genetic data, to study OA disease. The models can help us better understand OA disease and promote its early detection and diagnosis
References
[1] Zhang, M., Driban, J.B., Price, L.L., Harper, D., Lo, G.H., Miller, E., Ward, R.J., and McAlindon, T.E.: ‘Development and Validation of A Rapid Cartilage Quantification Method’, Arthritis and rheumatism, 2013, 65, pp. S479
[2] Driban, J.B., Barbe, M.F., Amin, M., Kalariya, N.S., Zhang, M., Lo, G.H., Tassinari, A.M., Harper, D., Price, L.L., Eaton, C.B., Schneider, E., and McAlindon, T.E.: ‘Validation of quantitative magnetic resonance imaging-based apparent bone volume fraction in peri-articular tibial bone of cadaveric knees’, BMC musculoskeletal disorders, 2014, 15, pp. 143
[3] Zhang, M., Driban, J.B., Price, L.L., Harper, D., Lo, G.H., Miller, E., Ward, R.J., and McAlindon, T.E.: ‘Study of Cartilage Damage Index with Joint Space Narrowing and Kellgren-Lawrence Grade’, Osteoarthritis and cartilage, 2014, 22, pp. S297-S298
[4] Zhang, M., Driban, J.B., Price, L.L., Harper, D., Lo, G.H., Miller, E., Ward, R.J., and McAlindon, T.E.: ‘Validation of Cartilage Damage Index with Joint Space Width and Static Alignment’, Osteoarthritis and cartilage, 2014, 22, pp. S299
[5] Zhang, M., Driban, J.B., Price, L.L., Harper, D., Lo, G.H., Miller, E., Ward, R.J., and McAlindon, T.E.: ‘Development of a rapid knee cartilage damage quantification method using magnetic resonance images’, BMC musculoskeletal disorders, 2014, 15, pp. 264
[6] Zhang, M., Driban, J.B., Price, L.L., Lo, G.H., and McAlindon, T.E.: ‘Magnetic Resonance Image Sequence Influences the Relationship between Bone Marrow Lesions Volume and Pain: Data from the Osteoarthritis Initiative’, Biomed Res Int, 2015, 2015, pp. 731903
[7] Zhang, M., Driban, J.B., Price, L.L., Lo, G.H., Miller, E., and McAlindon, T.E.: ‘Development of a Rapid Cartilage Damage Quantification Method for the Lateral Tibiofemoral Compartment Using Magnetic Resonance Images: Data from the Osteoarthritis Initiative’, Biomed Res Int, 2015, 2015, pp. 634275
[8] McAlindon, T.E., LaValley, M.P., Harvey, W.F., Price, L.L., Driban, J.B., Zhang, M., and Ward, R.J.: ‘Effect of Intra-articular Triamcinolone vs Saline on Knee Cartilage Volume and Pain in Patients with Knee Osteoarthritis: A Randomized Clinical Trial’, JAMA, 2017, 317, (19), pp. 1967-1975
[9] R, A., J, S., YD, D., and M, Z.: ‘Development of a Deep Learning Based Method for Breast Ultrasound Image Segmentation’. Proc. IEEE International Conference on Machine Learning and Applications (ICMLA) 2018
[10] Zhang, M., and Shan, J.: ‘A Novel 3D image predictive model for osteoarthritis disease’, NSF, 2017-2019, NSF1723429
[11] YD, D., J, S., and M, Z.: ‘Knee Osteoarthritis Prediction on MR Images Using Cartilage Damage Index on MR Images and Machine Learning Methods’. Proc. IEEE International Conference on Bioinformatics, and Biomedicine (BIBM) 2017
[12] Du, Y., Almajalid, R., Shan, J., and Zhang, M.: ‘A Novel Method to Predict Knee Osteoarthritis Progression on MRI Using Machine Learning Methods’, IEEE Trans Nanobioscience, 2018, 17, (3), pp. 228-236