4.1. Targeted engagement exercise with external experts:

The targeted engagement exercise focussed on draft recommendations for referral (or non-referral) of signs and symptoms in category 1. Once the committee had formulated draft consensus recommendations for the identified questions, the targeted engagement exercise provided an opportunity for expert advisors to ratify and suggest changes to the recommendations for inclusion in the final draft of the guideline before the main consultation.

The targeted engagement exercise is summarised below:

1. The targeted engagement exercise took place towards the end of the development phase. An online survey using modified RAND rating for agreement was run for 4 weeks, from 23 January to 17 February 2017.
2. Recruitment of external experts: recruitment took place in late autumn 2016. A formal open recruitment process was followed. An advertisement was posted on the NICE website for 1 month, which explained the reasons for recruitment and highlighted the time commitment and input required. Key stakeholder organisations including the Association of British Neurologists, British Paediatric Neurology Association, Primary Care Neurology Society, Royal College of GPs and the Royal College of Emergency Medicine, and other specialist societies were notified of the advertisement. External experts were invited to apply with their CV and to declare any conflicting interests. The Chair and Guideline Lead checked external experts based on their level of expertise and the relevance of their role. External experts were asked to sign consent and confidentiality forms. The list of expert advisors is listed in appendix P.
3. Constituency of external experts: The aim was to recruit approximately 30 external experts to account for withdrawal and non-responders and to include both adult and paediatric neurology specialists and non-specialists including primary care physicians.
4. Online survey using modified RAND rating for agreement: An online survey was developed asking external experts whether they agreed with the draft recommendations. If they did not agree, external experts were asked to provide comments on their reasoning (and if possible, alternative suggestions or wording) for the committee’s consideration.
5. Results and analysis:

   1. There were 43 external experts in total consisting of the following expertise: 9 adult neurologists (20.9%), 7 paediatric neurologists (16.3%), 9 general practitioners (20.9%) and 18 other professionals including paediatricians, psychiatrists and physiotherapists (41.9%).
   2. The external experts were given the option of commenting on either the adult or the children’s recommendations or both. Fifteen external experts (34.9%) provided feedback on adult recommendations, 21 external experts (48.8%) provided feedback on the children’s recommendations and 7 external experts (16.3%) provided feedback on recommendations for both age groups.
   3. The committee agreed that the threshold for agreement was 75%, which is a threshold that is widely used for other consensus methods such as Delphi and RAND and was determined to be a good threshold for agreement in a recent systematic review by Diamond et al (2014)⁴. However, the results showed that although most recommendations reached the 75% agreement threshold, many external experts had provided very helpful comments that would improve the clarity and content of the recommendations. Therefore, the committee revisited the majority of the recommendations in light of the external experts’ feedback. A summary of the feedback from the targeted engagement exercise and the actions the committee took for each recommendation is included in the ‘recommendations and link to evidence’ sections under ‘other considerations’.
6. Following the targeted engagement exercise, the revised draft recommendations were submitted for the main consultation with all stakeholders along with the other recommendations in categories 2 and 3.

All results from the targeted engagement exercise are available in appendix S.

4.2. Evidence reviews

Sections 4.2, 4.3, 4.4 and 4.5 describe the process used to identify and review clinical evidence (summarised in Figure 1), section 4.6 describes the process used to identify and review the health economic evidence, and section 4.7 describes the process used to develop recommendations.

Figure 1

Step-by-step process of review of evidence in the guideline.

4.3. Developing the review questions and outcomes

Review questions were developed using a framework of population, index tests, reference standard and target condition for reviews of diagnostic test accuracy and using population, presence or absence of factors under investigation and outcomes for clinical prediction reviews. This use of a framework guided the literature searching process, critical appraisal and synthesis of evidence, and facilitated the committee’s development of the recommendations. The NGC technical team drafted the review questions, and the committee refined and validated them. The questions were based on the key clinical areas identified in the scope (appendix A).

A total of 10 review questions were identified.

Full literature searches, critical appraisals and evidence reviews were completed for all the specified review questions.

For questions on category 2 symptoms, the committee wanted to consider any studies that determine whether a certain sign or symptom accompanying a main presenting symptom (for example, hearing loss in the presence of dizziness) is indicative of a neurological condition that requires onward referral for a specialist assessment. Therefore, measures of diagnostic accuracy including sensitivity, specificity, positive predictive value, negative predictive value, ROC and AUC were considered as the main outcomes of interest. However, the committee was aware that there was limited evidence available in this area; therefore, the committee considered clinical prediction studies where a multivariate analysis was conducted and adjusted odds ratios for outcomes of interest were presented. Hence, some of the clinical questions were reviewed using a mix of diagnostic and clinical prediction strategies.

Table 1

Review questions on category 2 symptoms.

4.4. Searching for clinical evidence

4.5. Identifying and analysing clinical evidence

Research fellows conducted the tasks listed below, which are described in further detail in the rest of this section:

• Identified potentially relevant studies for each review question from the relevant search results by reviewing titles and abstracts. Full papers were then obtained.
• Reviewed full papers against pre-specified inclusion and exclusion criteria to identify studies that addressed the review question in the appropriate population and reported on outcomes of interest (review protocols are included in appendix C).
• Critically appraised relevant studies using the appropriate study design checklist as specified in the NICE guidelines manual.¹¹ Clinical prediction studies were critically appraised using NGC checklists. Generated summaries of the evidence by outcome. Outcome data were combined, analysed and reported according to study design:

  • Clinical prediction data were meta-analysed where appropriate and reported in GRADE-like profile tables.
  • Diagnostic data studies were meta-analysed where appropriate or presented as a range of values in adapted GRADE profile tables
• A sample of a minimum of 10% of the abstract lists of the first 3 sifts by new reviewers, and a senior research fellow double sifted those for complex review questions (for example, clinical prediction reviews). Any discrepancies were rectified. All of the evidence reviews were quality assured by a senior research fellow. This included checking:

  • papers were included or excluded appropriately
  • a sample of the data extractions
  • correct methods were used to synthesise data
  • a sample of the risk of bias assessments.

4.5.4. Appraising the quality of evidence by outcomes

4.5.4.1. Diagnostic studies

Risk of bias and indirectness of evidence for diagnostic data were evaluated by study using the Quality Assessment of Diagnostic Accuracy Studies version 2 (QUADAS-2) checklists (see appendix H in the NICE guidelines manual 2014¹¹). Risk of bias and applicability in primary diagnostic accuracy studies in QUADAS-2 consists of 4 domains (see Table 2):

• patient selection
• index test
• reference standard
• flow and timing.

Table 2

Summary of QUADAS-2 with list of signalling, risk of bias and applicability questions.

4.5.4.1.1. Inconsistency

Inconsistency refers to an unexplained heterogeneity of results for an outcome across different studies. Inconsistency was assessed by inspection of the sensitivity or specificity (based on the most important outcome for each particular review question) using the point estimates and 95% CIs of the individual studies on the forest plots. Particular attention was placed on values above or below 50% (diagnosis based on chance alone) and the threshold the committee set (the threshold above which it would be acceptable to recommend a test). For example, the committee might have set a threshold of 90% sensitivity as an acceptable level to recommend a test. The evidence was downgraded by 1 increment if the individual studies varied across 2 areas (for example, 50–90% and 90–100%) and by 2 increments if the individual studies varied across 3 areas (for example, 0–50%, 50–90% and 90–100%).

4.5.4.1.2. Imprecision

The judgement of precision was based on visual inspection of the confidence region around the summary sensitivity and specificity point from the diagnostic meta-analysis, if a diagnostic meta-analysis was conducted. Where a diagnostic meta-analysis was not conducted, imprecision was assessed according to the range of point estimates or, if only 1 study contributed to the evidence, the 95% CI around the single study. As a general rule (after discussion with the committee), a variation of 0–20% was considered precise, 20–40% serious imprecision, and >40% very serious imprecision. Imprecision was assessed on the primary outcome measure for decision-making.

4.5.4.1.3. Overall grading

Once an outcome had been appraised for the main quality elements, as above, an overall quality grade was calculated for that outcome. The scores (0, −1 or −2) from each of the main quality elements were summed to give a score that could be anything from 0 (the best possible) to −8 (the worst possible). However, scores were capped at −3. This final score was then applied to the starting grade that had originally been applied to the outcome by default, based on study design. Quality rating started at High for prospective and retrospective cross-sectional studies, and each major limitation (risk of bias, indirectness, inconsistency and imprecision) brought the rating down by 1 increment. Each of these studies started at High and the overall quality became Moderate, Low or Very Low if the overall score was −1, −2 or −3 points respectively. The significance of these overall ratings is explained in Table 3. The reasons for downgrading in each case were specified in the footnotes of the GRADE tables.

Table 3

Overall quality of outcome evidence in modified GRADE.

4.5.4.2. Clinical prediction reviews

A modified GRADE methodology was used for clinical prediction studies, considering risk of bias, indirectness, inconsistency and imprecision.

The quality of evidence for risk prediction studies was evaluated according to the criteria given in Table 4. This table was adapted from the Quality In Prognosis Studies tool (QUIPS)⁵. If data were meta-analysed, the quality for pooled studies was presented. If the data were not pooled, then a quality rating was presented for each study.

Table 4

Description of quality elements for prospective studies (adapted from the QUIPS tool).

The risk of bias rating was assigned per study for each combination of risk factor or outcome. When studies were pooled, the overall risk of bias for all studies covering a specific risk factor or outcome was determined by a weighted mean of the ratings across the studies (no risk = 0; serious risk = −1 and very serious risk = −2). The weighting depended on the weighting used in the meta-analysis, as in intervention reviews. Where a meta-analysis had not been conducted, a simple average was used.

4.5.4.2.1. Indirectness

Indirectness refers to the extent to which the populations, risk factors and outcome measures are dissimilar to those defined in the inclusion criteria for the reviews, as explained for intervention reviews. As for risk of bias, each outcome had its indirectness assessed within each study first. For each paper, if there were no sources of indirectness, indirectness was given a rating of 0. If there was indirectness in just 1 source (for example, in terms of population), indirectness was given a ‘serious’ rating of −1, but if there was indirectness in 2 or more sources (for example, in terms of population and risk factor), the indirectness was given a ‘very serious’ rating of −2. A weighted average score was then calculated across all studies contributing to the outcome, by taking into account the weights in the meta-analysis.

4.5.4.2.2. Inconsistency

Inconsistency refers to an unexplained heterogeneity of results for an outcome across different studies. When heterogeneity existed within an outcome (chi-squared p<0.1, or I²>50%), but no plausible explanation could be found, the quality of evidence for that outcome was downgraded. Inconsistency for that outcome was given a ‘serious’ score of −1 if the I² was 50–74% and a ‘very serious’ score of −2 if the I² was 75% or more.

If inconsistency could be explained based on pre-specified subgroup analysis (that is, each subgroup had an I²<50%), the committee considered this and considered whether to make separate recommendations on new outcomes based on the subgroups defined by the assumed explanatory factors. In such a situation, the quality of evidence was not downgraded for those emergent outcomes.

4.5.4.2.3. Imprecision

The criteria applied for imprecision were based on the confidence intervals around the estimate of association between the risk factor/predictor and the outcome (condition of interest). The decision to downgrade was discussed with the committee and was based on the interpretations of the width of the confidence intervals and how certain the committee was in drawing a conclusion (that is, how certain that there is no association, or a positive association, or a negative association (protective) between the risk factor/predictor and the outcome (condition of interest).

4.5.4.2.4. Overall grading

Quality rating started at High for prospective studies, and each major limitation brought the rating down by 1 increment to a minimum grade of Very Low, as explained for diagnostic reviews above. For clinical prediction reviews, prospective cohort studies with a multivariate analysis are regarded as the gold standard because RCTs are usually inappropriate for these types of review for ethical or pragmatic reasons. Furthermore, if the study looked at more than 1 risk factor of interest then randomisation would be inappropriate, as it can only be applied to 1 of the risk factors.

4.6. Identifying and analysing evidence of cost effectiveness

The committee is required to make decisions based on the best available evidence of both clinical effectiveness and cost effectiveness. Guideline recommendations should be based on the expected costs of the different options in relation to their expected health benefits (that is, their ‘cost effectiveness’) rather than the total implementation cost. However, the committee will also need to be increasingly confident in the cost effectiveness of a recommendation as the cost of implementation increases. Therefore, the committee may require more robust evidence on the effectiveness and cost effectiveness of any recommendations that are expected to have a substantial impact on resources; any uncertainties must be offset by a compelling argument in favour of the recommendation. The cost impact or savings potential of a recommendation should not be the sole reason for the committee’s decision.¹¹

Health economic evidence was sought relating to the key clinical issues being addressed in the guideline. Health economists:

• Undertook a systematic review of the published economic literature.
• Considered options for new cost-effectiveness analysis in priority areas.

4.7. Developing recommendations

Over the course of the guideline development process, the committee was presented with:

• Evidence tables of the clinical evidence reviewed from the literature. All evidence tables are in appendix H.
• Summaries of clinical and health economic evidence and quality (as presented in chapters 5-7).
• Forest plots and summary ROC curves (appendix K).
• A description of the methods and results of the cost-impact analysis undertaken for the guideline (appendix N).

Recommendations were drafted based on the committee’s interpretation of the available evidence, taking into account the balance of benefits, harms and costs between different courses of action. This was completed either formally in an economic model or informally. Firstly, the net clinical benefit over harm (clinical effectiveness) was considered, focusing on the critical outcomes. When this was done informally, the committee took into account the clinical benefits and harms when one intervention was compared with another. The assessment of net clinical benefit was moderated by the importance placed on the outcomes (the committee’s values and preferences) and the confidence the committee had in the evidence (evidence quality). Secondly, the committee assessed whether the net clinical benefit justified any differences in costs between the alternative interventions.

When clinical and health economic evidence was of poor quality, conflicting or absent, the committee drafted recommendations based on its expert opinion. The considerations for making consensus-based recommendations include the balance between potential harms and benefits, the economic costs compared to the economic benefits, current practices, recommendations made in other relevant guidelines, patient preferences and equality issues. The consensus recommendations were agreed through discussions in the committee. The committee also considered whether the uncertainty was sufficient to justify a delay in making a recommendation to await further research, taking into account the potential harm of failing to make a clear recommendation (see section 4.7.1 below).

The committee considered the appropriate ‘strength’ of each recommendation. This takes into account the quality of the evidence but is conceptually different. Some recommendations are ‘strong’ in that the committee believes that the vast majority of healthcare and other professionals and patients would choose a particular intervention (referral in this guideline) if they considered the evidence in the same way that the committee has. This is generally the case if the benefits clearly outweigh the harms for most people and the intervention is likely to be cost effective. However, there is often a closer balance between benefits and harms, and some patients would not choose an intervention whereas others would. This may happen, for example, if some patients are particularly averse to some side effects and others are not. In these circumstances, the recommendation is generally weaker, although it may be possible to make stronger recommendations about specific groups of patients.

The committee focused on the following factors in agreeing the wording of the recommendations:

• The actions health professionals need to take.
• The information readers need to know.
• The strength of the recommendation (for example, the word ‘offer’ was used for strong recommendations and ‘consider’ for weaker recommendations).
• The involvement of patients (and their carers if needed) in decisions on treatment and care.
• Consistency with NICE’s standard advice on recommendations about drugs, waiting times and ineffective interventions (see section 9.2 in the 2014 NICE guidelines manual¹¹).

The main considerations specific to each recommendation are outlined in the ‘Recommendations and link to evidence’ sections within each chapter.