While risk assessment scores for stratifying pulmonary arterial hypertension (PAH) patients have been around for many decades, clinicians frequently rely on gestalt for assessing risk. The goal of this work was to benefit from the deep clinical knowledge of clinical experts to derive a predictive model for pulmonary arterial hypertension (PAH.) We then compared the performance of this hybrid model to one derived exclusively by machine learning.
Eleven clinicians were presented with a diagram containing 14 predictor variables and invited to draw arcs (lines) indicating causal connections among them, and with the outcome of survival at 12 months. These variables were the same as found in the REVEAL 2.0 calculator, although the clinicians were not required to use every variable. After creating networks that represented their understanding of causality for mortality risk, each network was programmed as a Bayesian network, using a modeling user interface (GeNIe). A combined expert model was also developed, using arcs that were common to at least two individual experts. Independently, the parameters were learned using the REVEAL registry data from patients at 12 months post enrollment. Each model was evaluated using 10-fold cross validation and the areas under the receiver-operator characteristic curve (AUCs) were compared.
Seventeen expert models were created (some clinicians made multiple versions) and their performance ranged from an AUC 0.66 to 0.73. The combined expert model performed with an AUC of 0.72. The model learned from data using a tree-augmented naïve Bayesian network performed with an AUC of 0.86.
Experts do not agree on what parameters are most related to survival outcomes. Networks built by the experts varied in the quantity of variables and the relationships among them. A combination of expert models gave a reasonably accurate model, but its performance was inferior to the machine-learned model. There is a need for inclusion of mathematically derived risk assessments to complement the clinicians’ assessment of patients.