Accuracy of predicting treatment failure in online cognitive-behavioral therapy for insomnia

In a 4-week period of 9-week treatment using a relatively simple classification algorithm, treatment failure in Internet-based cognitive behavioral therapy for insomnia (ICBT-i) could be predicted with a balanced accuracy of 67%, according to study results published in Internet interventions.1

The key question when using an adaptive treatment strategy is “how accurate is the prediction before one can act on it”. In adaptive treatment strategies, each patient’s outcome is predicted early in the course of treatment, with treatment adapting to those at risk of failure. Investigators in the current clinical trial ( Identifier: NCT01663844) sought to establish a minimal, experimentally supported accuracy of the classifier to be clinically useful in an adaptive treatment strategy. They also explored and compared prediction models of varying complexity and feasibility.

In order to achieve these goals, the following three goals were cited by the researchers: (1) to determine the accuracy of a randomized trial (RCT) classifier used to predict treatment failure in a previous proof-of-concept study, and thus establish a standard for minimal, empirically supported accuracy in future developments of treatment strategies adaptive; (ii) To check the relative value of each of the predictors of the classifier; and (3) to explore the added value of different logical sets of predictors.

Continue reading

Data for the current study were derived from a previously published randomized controlled trial,2 Patients who underwent ICBT-i were classified as ‘red’ (risk of treatment failure) or ‘green’ (non-risk) during treatment week 4 out of 9 weeks. In the current study, ‘green’ is used to refer to those patients whose good outcome (treatment success) was predicted based on the algorithm, and ‘red’ is used to refer to those patients whose trial was predicted. Bad result (treatment failure) according to the algorithm.

Classification was made at Week 4 for two reasons: (1) Week 4 was the point at which patients who were working at the set pace would go through psychoeducation and rationale, and would initiate sleep restriction; and (2) researchers still want as much time remaining in treatment as possible to adapt treatment and help individuals who thought they would not benefit adequately from their current course.

Overall, half of the ‘red’ patients were randomly assigned to receive appropriate treatment. The current study evaluated data from all ‘green’ patients (n = 149) and from only those ‘red’ patients who did Not Receiving adaptive therapy (n = 51), allowing assessment of classifier accuracy without conditioning treatment as an influencer. The study was conducted in an online psychiatry clinic – a care clinic specializing in psychiatry within public health care in Sweden.

Study results revealed that the final rating from the proof-of-concept study, which was shown to be of clinical value, had a balanced accuracy of 67% and a lower confidence interval that was significantly above 50% (chance). Overall, 11 out of 21 predictors were significantly associated with failure. The model that used all predictors was able to explain 56% of the outcome variance, with simpler models explaining between 16% and 47% of the outcome variance.

Key indicators included patient-rated stress, change in depression, treatment reliability, and symptoms of insomnia at week 3, as well as clinician-rated attitudes toward homework and sleep medication.

A major limitation of the current study is the fact that the sample size was chosen to detect clinically relevant group differences in RCTs, rather than to assess classification performance. In addition, the researchers did not examine all the available data, instead, they only examined the data used by the classification algorithm.

The researchers concluded that “simpler predictive models have shown some hope and should be developed further, possibly using machine learning methods.”

Disclosure: None of the study authors announced their affiliations with biotechnology, pharmaceutical, and/or device companies.


1. Forsell E, Jernelöv S, Blom K, Kaldo V. Clinically adequate classification accuracy and key predictors of treatment failure in a randomized controlled trial of cognitive behavioral therapy for insomnia online. Internet Interv. Published online June 25, 2022. doi: 10.1016 / j.invent.2022.100554

2. Forsell E, Gernilov S, Bloom K, et al. Proof of concept for an adaptive treatment strategy to prevent failures in online cognitive-behavioral therapy: a randomized controlled clinical trial with insomnia patients. I’m J pssyuter. Published online January 30, 2019. doi: 10.1176 / appi.ajp.2018.18060699

Leave a Comment

Your email address will not be published.

beautiful home decor Williams vs Radokano match postponed Brian Kelly, LSU player Myles Brennan, retires from football Black Adam and Stripe are seemingly heading to MultiVersus Bryce Dallas Howard claims she received payment.
beautiful home decor Williams vs Radokano match postponed Brian Kelly, LSU player Myles Brennan, retires from football Black Adam and Stripe are seemingly heading to MultiVersus Bryce Dallas Howard claims she received payment.