Risk of Bias Rating Assistant

In almost all psychotherapy trials, this will be ‘No/PN’. Exceptions can be specifically studied for the trials comparing two active psychological interventions (e.g., both active interventions are presented to participants as equally effective, the trial is designed as a non-inferiority trial, etc.).

In almost all psychotherapy trials, this will be ‘No/PN’. Exceptions can be specifically studied for the trials comparing two active psychological interventions (e.g., both active interventions are presented to participants as equally effective, the trial is designed as a non-inferiority trial, etc.).

When participants and carers are not blinded, there could be deviations from the intended interventions associated with this lack of blinding . An example of such a type of deviation is when participants who are told that they were on the waitlist seek the intervention of the experimental arm outside the trial or other interventions. Another example is that, due to the trial context, participants in the psychotherapy group have easier access to the prescription of antidepressant medication than participants in the control group.

Deviations from the intended interventions that would arise even if the intervention took place outside the trial do not constitute a risk of bias with regard to the intention-to-treat analysis. An example could be drop-outs due to side effects of antidepressants: such drop-outs are deviations from the intended intervention but would occur even if the patients were taking antidepressants outside the trial context.

To pose a risk, these deviations should be influential for the outcome. For example, participants taking benzodiazepines in a depression psychotherapy trial might not be influential, but receiving the psychotherapy of interest or antidepressant medication is influential.

If there are deviations related to the outcome but they are balanced between the groups, bias will be less likely. However, if these influential deviations are unbalanced between the groups, it is more likely that the intervention effect estimate is biased. For example, when a substantially larger proportion of participants allocated to the psychotherapy group are taking antidepressants compared to the control group. Reviewers should make agreements on how an imbalance between groups is defined. In the previous example, an imbalance of 20% could be taken as a rough indication.

In these cases, the trial can be judged as high risk, by giving a “No” answer to this item.

This item evaluates whether trialists report to have adhered to the intention-to-treat principles, or to the so-called modified ITT (where the authors drop the randomized participants because their outcomes are missing).

Dropping the randomized participants because they did not receive the intervention they were allocated to, or counting the participants who received the intervention that they were not allocated to as those receiving this other intervention is inappropriate. In some cases, it might be sensible to assume that trialists adhered to ITT principles even if it is not explicitly stated (e.g., when authors used statistical analyses that are implicitly associated with the ITT principle).

If no appropriate analysis was conducted (or there is not enough information), reviewers should examine whether there was likely an impact of not using an appropriate analysis to estimate the effect of assignment to intervention. In cases in which excluded participants (e.g., due to not receiving a minimum number of sessions) or participants analyzed in the wrong group (e.g., analyzed in the waitlist group because in the end they did not receive the intervention) are less than 5%, part 2 of domain 2 will be rated as “Some concerns”. High risk will be given for thresholds above 5% or for “No information”. Missing outcome data (proportion of participants that did not answer the questionnaires) is not assessed here, but rather in domain 3 (item 10).

For meta-analyses focused on continuous outcomes: if the proportion of available data is above >95%, then this domain is directly rated as low risk and it is not needed to answer the next questions. Note that this refers to the proportion of participants who have endpoint data (i.e., who completed the questionnaires), and not the LOCF or imputed data.

For meta-analyses focused on dichotomous outcomes: For dichotomous outcomes, the proportion required is directly linked to the risk of the event. If the observed number of events is much greater than the number of participants with missing outcome data, the bias would necessarily be small.

For example, the following approaches are considered appropriate for handling the impact of missing data:

• MMRMs (mixed models for repeated measures, also known as mixed models, growth curve analyses) based on two or more measurements after baseline (e.g., mid-treatment, post-test, follow-ups, etc.).
• Multiple imputation, if the following criteria are met:
• A) “Rubin's rules” or other methods to account for imputation uncertainty were applied.
• B.1 and B.2 or C.1) applies:
• B.1) Important auxiliary variables were used in the imputation model, such as intermediate outcome assessments.
• B.2) Imputations were generated separately by RCT groups (“bygroup” imputation).
• C.1) Controlled/reference-based imputation (e.g., “jump-to-reference”) is used, using appropriate external information.
• Sensitivity analyses corresponding with a range of plausible reasons for missingness to confirm the primary analyses.

When data are missing not at random, most of the aforementioned models will be biased. “Last observation carried forward” is not considered an appropriate approach for handling missing data.

A trial is considered to be of moderate to large size when it includes at least 40 participants per arm.

This is fulfilled when at least 70% of the randomized participants in each of the arms completed the assessments at post-treatment.

This item examines the following sources of indirect evidence that might signal risk of bias:

• a) whether the overall reported reasons for missing data indicate that missingness might depend on its true value (e.g., the general reason for missing data is termination of funding would not be indicative of risk of bias),
• b) large imbalances in the proportion of missing data between the groups: As a rule of thumb, the difference in study drop-out between the groups should be less than 20% (e.g., the intervention arm has a study drop-out of 10% and the wait-list control has a drop-out of 20%).
• c) differing reasons for missing data between the groups (e.g., more participants from the intervention group abandon the trial due to being more critically symptomatic could be an indicator of missigness being related to the outcome).

If any of the three is present, reviewers should answer with a “No”, given that missing data might be related to the outcome. No information for evaluating at least one of these three sources also leads to a high risk of bias.

The instrument used to quantify the outcome should be identical in both groups, and should reliably measure the outcome of interest.

This will be the case for most psychometrically validated outcome rating scales.

In the context of psychological interventions, blinding is often not feasible or even impossible to achieve.

Empirical evidence shows that self-reports are not associated with overestimated treatment effects despite the lack of participant blinding (Cuijpers et al., 2010).

Therefore, if there is no evidence that participants under-reported or there is a safeguard to prevent this (e.g., therapists did not have access to the self-reports), reviewers may consider self-reports as low risk of bias.

If it is likely that patients under-reported their severity (for example, to please the therapists or the researchers when self-reports are filled out in front of the therapist at the last session), reviewers should consider setting this item to ‘No/PN’, which will lead to a high risk of bias for this domain.

Assessor-rated instruments are filled out by the therapist, independent evaluators, or research personnel. These are usually administered as a structured interview with the participant or are completed by the evaluator through observation of the participant's behavior.

This item evaluates whether the assessors evaluating the outcome were masked to treatment allocation. This item should be rated as 'Yes' when trialists clearly report that the assessors were masked. If this is not reported, it might be sensible to assume that no masking procedures were implemented ('No/PN').

Meta-analysts might be interested in including all available results for one outcome domain within a trial. For example, in a depression trial, there might be post-test data from the Hamilton Depression Rating Scale (assessor-rated) and the Beck Depression Inventory (self-report). There could be two strategies for these cases, depending on the meta-analysis protocol and analysis approach:

• Separate risk of bias scores could be performed for each outcome and numeric result, or
• Two or more ratings could be combined into an aggregated score, conservatively rating the aggregated score as high risk when one of the measurements is at high risk.

For the Metapsy depression database we use the second strategy, combining ratings into an aggregated score.

If there is no access to documents describing the trial's protocol, pre-defined outcome measures, or analysis plan, this domain should be directly evaluated as “Some concerns”.

If reviewers have access to such documents, the next questions have to be answered.

This item evaluates whether the outcome measures in a trial were pre-specified before the start of data collection (i.e., start of participant enrolment). For example, when the trial was registered (and specified the outcomes) before the start of data collection, which is known as prospective registration. Outcome domain, scale, time point, metric, and analysis should ideally be specified, although reviewers can answer this item as “Yes” if at least scale and time point were specified.

Retrospective registrations (registering the trial after start of data collection) or lack of information should lead to a “No” answer in this item, which results in “Some concerns” for this domain.

This will partly depend on the meta-analysis protocol. If meta-analysts aim to include all available instruments for a given outcome (e.g., depression severity), the article should report all instruments in full (with enough data to be entered in the meta-analysis).

If meta-analysts have a pre-specified hierarchy for instrument inclusion (e.g., select HAM-D over BDI), then selective reporting will be judged regarding the availability of outcomes based on the pre-specified hierarchy. It will also be rated as high risk when a non-registered outcome is added in the publication of the paper. Reviewers should consider whether data from the outcome of interest can be made available through other means (e.g., contact through authors).

"Pre-specified" is interpreted as "before unblinded data is available". Dates of publication of the analysis plan should precede participant enrollment.

Any deviation from the original plan should be reported in the final publication and should be justified. If properly justified, meta-analysts can rate this signalling question as 'Yes/PY', after careful assessment of how this deviation can affect the studied meta-analytic result.

As part of the Metapsy initiative, we do not consider this question in the algorithm unless there is evidence of unjustified deviations from a pre-specified analysis plan, in which case the entire domain is downgraded to "high risk".