Clinical validation of a genetic model to estimate the risk of developing choroidal neovascular age-related macular degeneration

Abstract

Predictive tests for estimating the risk of developing late-stage neovascular age-related macular degeneration (AMD) are subject to unique challenges. AMD prevalence increases with age, clinical phenotypes are heterogeneous and control collections are prone to high false-negative rates, as many control subjects are likely to develop disease with advancing age. Risk prediction tests have been presented previously, using up to ten genetic markers and a range of self-reported non-genetic variables such as body mass index (BMI) and smoking history. In order to maximise the accuracy of prediction for mainstream genetic testing, we sought to derive a test comparable in performance to earlier testing models but based purely on genetic markers, which are static through life and not subject to misreporting. We report a multicentre assessment of a larger panel of single nucleotide polymorphisms (SNPs) than previously analysed, to improve further the classification performance of a predictive test to estimate the risk of developing choroidal neovascular (CNV) disease. We developed a predictive model based solely on genetic markers and avoided inclusion of self-reported variables (eg smoking history) or non-static factors (BMI, education status) that might otherwise introduce inaccuracies in calculating individual risk estimates. We describe the performance of a test panel comprising 13 SNPs genotyped across a consolidated collection of four patient cohorts obtained from academic centres deemed appropriate for pooling. We report on predictive effect sizes and their classification performance. By incorporating multiple cohorts of homogeneous ethnic origin, we obtained >80 per cent power to detect differences in genetic variants observed between cases and controls. We focused our study on CNV, a subtype of advanced AMD associated with a severe and potentially treatable form of the disease. Lastly, we followed a two-stage strategy involving both test model development and test model validation to present estimates of classification performance anticipated in the larger clinical setting. The model contained nine SNPs tagging variants in the regulators of complement activation (RCA) locus spanning the complement factor H (CFH), complement factor H-related 4 (CFHR4), complement factor H-related 5 (CFHR5) and coagulation factor XIII B subunit (F13B) genes; the four remaining SNPs targeted polymorphisms in the complement component 2 (C2), complement factor B (CFB), complement component 3 (C3) and age-related maculopathy susceptibility protein 2 (ARMS2) genes. The pooled sample size (1,132 CNV cases, 822 controls) allowed for both model development and model validation to confirm the accuracy of risk prediction. At the validation stage, our test model yielded 82 per cent sensitivity and 63 per cent specificity, comparable with metrics reported with earlier testing models that included environmental risk factors. Our test had an area under the curve of 0.80, reflecting a modest improvement compared with tests reported with fewer SNPs.

Introduction

Many diseases of ageing characterised by complex inheritance patterns are progressive; the individual may be asymptomatic in the early stages. One of these diseases, age-related macular degeneration (AMD), is the most common cause of visual impairment and the leading cause of blindness in the elderly population in the developed world. The prevalence of AMD increases with advancing age in all populations studied. Thus, in developed nations such as the USA, UK, Canada and Australia, with increasingly aged populations, the condition affects a progressively larger segment of the population and has become a major public health issue. Early- or late-stage AMD is present in 15 per cent of individuals over the age of 60 years [1]. It is estimated that there are currently 9.1 million patients in the USA with AMD, of which 1.7 million suffer with the vision-threatening late-stage complications of choroidal neovascularisation (CNV) or geographic atrophy [1]. Moreover, it is predicted that the number of cases of early AMD will increase to 17.8 million by 2050 and, if untreated, cases of late-stage blinding AMD will increase to 3.8 million [1]. It has been determined that vision loss from AMD decreases quality of life by 60 per cent, similar to the experience of dealing with a stroke that requires intensive nursing care [2].

The clinical presentation and natural course of AMD are highly variable. The disease may present as early as the fifth decade of life or as late as the ninth decade. The clinical symptoms of AMD range from no visual disturbances in early disease to profound loss of central vision in the advanced late stages of the disease. Some patients never progress beyond early AMD; however, in 10-15 per cent of Caucasian patients with early-stage disease, the condition progresses to an exudative neovascular (or 'wet' form) or geographic atrophic (or 'dry' form) AMD, which threatens vision. The phenotype is characterised by development of subretinal choroidal neovascular complexes, haemorrhage and fibrosis and is typically associated with severe central vision loss [3, 4].

AMD has been one of the success stories of the genome revolution and is probably one of the best characterised of the complex trait diseases in terms of genetic predisposition (for reviews, see Allikmets and Dean [5] and Swaroop et al. [6]). Besides age, genetic background is the most significant non-modifiable risk factor for all stages of AMD, while smoking is the most significant modifiable risk factor [7, 8]. Initial groundbreaking studies established that loci on chromosomes (Chr) 1 and 10 -- in particular the complement factor H (CFH) and the age-related maculopathy susceptibility protein 2 (ARMS2)/high temperature requirement factor A1 (HTRA1) genes, respectively -- are significantly associated with AMD risk and protection in populations of various ethnicities [9–19]. Although the specific role(s) of the Chr 10 genes in AMD pathobiology has not yet been elucidated, the role of the alternative complement pathway, where CFH functions as a major fluid-phase regulator, is well established (see Anderson et al. [20, 21] Gehrs et al. [22, 23] Hageman et al. [24, 25] and Mullins et al. [26] for overviews). Early pathobiological investigations showed dysregulation of the complement cascade to be a critical early predisposing step in the development of AMD. This spurred the discovery of the association of CFH variants with AMD risk. Subsequent genetic investigations revealed additional associations between AMD and risk/protective variants in various complement pathway-associated genes, including complement component 2 (C2), complement factor B (CFB), complement component 3 (C3), complement factor H-related 1 and 3 (CFHR1 and CFHR3) and complement factor I (CFI) [21, 27–38]. Using a genome-wide association approach, a handful of additional AMD-associated loci have been reported recently; these appear to be modestly associated with AMD risk and will probably require replication in additional cohorts to establish their role in AMD pathogenesis [39, 40] (see also Gehrs et al. [23] for a review).

A prerequisite for a new era in genetic testing and diagnosis for AMD is a robust test that accurately captures the impact of consistently replicated AMD risk variants in predicting the risk of developing CNV. Patients with CNV represent an important segment of the AMD population that would benefit from early diagnosis, given the current availability of an effective therapeutic intervention. Jakobsdottir and coworkers [41] recently concluded that the diagnostic value of three variants in the CFH, ARMS2/HTRA1 and C2 genes was not sufficient to discriminate between individuals with and without AMD because of the relatively low sensitivity and specificity of the combined test panel, in combination with the relatively low prevalence of late-stage disease in the general population. They applied a three single nucleotide polymorphism (SNP) test to their cohort of 640 late-stage AMD cases and 142 controls to demonstrate a clinical sensitivity of 74 per cent and a specificity of 69 per cent, with a reported area under the curve (AUC) -- a measure of how well a test or classifier can distinguish between cases and controls -- of 0.79. Perfect test discrimination would yield an AUC of 1.0. Jakobsdottir and colleagues also reported that the positive predictive value (PPV) of the same test is affected by different values of disease prevalence reflective of age. Seddon and colleagues [42] evaluated six AMD risk-associated variants in CFH, ARMS2/HTRA1, C2, CFB and C3 with the goal of developing a predictive risk test for late-stage AMD. After controlling for smoking, body mass index (BMI) and vitamin intake, they demonstrated a strong association between these six risk variants and the prevalence of late-stage AMD, as well as progression to late-stage disease in early AMD patients. The progression test described by Seddon et al.,[42] which included genetic, environmental and treatment variables, achieved a performance of 83 per cent sensitivity and 68 per cent specificity, with a reported AUC of 0.82. McKay and co-workers [43] extended this test further, proposing a ten-SNP panel plus smoking history to predict the risk of late-stage AMD. Their inclusion of six CFH SNPs was designed to capture the haplotype structure of the locus, to improve classification performance. Zanke and colleagues [44] have presented risk scores by selecting marker-specific odds ratios from disparate sources and multiplying them together. As the latter approach does not benefit from a joint assessment of the markers (as they perform in combination), it may overestimate an individual's risk of disease.

In this study, we assessed the accuracy of a panel of 13 SNPs without consideration of environmental risk factors such as smoking or BMI, to predict the risk of developing CNV in Caucasian individuals 60 years of age and older. Test model development and validation were designed to evaluate these variants in eight AMD-associated genes (CFH, complement factor H-related 4 (CFHR4), complement factor H-related 5 (CFHR5) and coagulation factor XIII B subunit (F13B) located within the regulators of complement activation (RCA) region on Chr 1, C2 and CFB on Chr 6, C3 on Chr 19 and ARMS2 on Chr 10. The panel of 13 SNPs was tested in well-established case-control and sibling pair cohorts from five academic centres (University of Iowa, University of Utah, Columbia University, Harvard University and Melbourne University) to validate the accuracy of the predictive test and to estimate an individual's genetic risk for developing late-stage CNV. Most of the disease-associated genetic variants in CFH, ARMS2, C2, CFB and C3 were selected based on prior replication in multiple studies and performance in resolving the most frequent CFH haplotype combinations. Additional SNPs detecting variants in CFHR4 (rs1409153), CFHR5 (rs10922153 and rs1750311) and F13B (rs698859 and rs2990510) tagged novel extended haplotypes spanning the CFH-to-F13B region and were included to maximise the resolution of clinically relevant subtypes suspected to have high association with disease [45]. The additional SNPs were selected to distinguish the novel haplotypes from the more prevalent haplotypes reported previously (H1, H2, H3, H4) [13]. The performance metrics obtained during the clinical validation of the 13-SNP panel were used as a benchmark to compare with other published AMD-predictive tests directed at estimating an individual's risk of developing late-stage disease. Since the inclusion of several established non-genetic factors (eg smoking) was highly variable across the published tests, the focus of this investigation was to isolate the contribution conferred by genetic variation alone, in order to determine whether the more comprehensive collection of SNPs could further improve prediction accuracy. The methodology used in the clinical validation of the 13-SNP test panel was subsequently applied to two panels of markers [32, 42] that had been assessed previously and contained variants that overlapped with the markers contained within our 13 SNP panel. Both test panels were evaluated in the large collective cohort by using a validation step absent in prior publications. Testing the two panels in a large collection of subjects from different centres assembled from several independent collections was designed to minimise the introduction of selection bias inherent in a single cohort study. Additionally, the use of an independent validation sample was intended to aggressively challenge the 13-SNP panel, to anticipate performance metrics in a broader clinical setting more accurately. Running the three test panels (three SNPs, six SNPs and 13 SNPs) on the same samples allowed for the comparison of performance metrics based exclusively on genetic variants.

Materials and methods

Subjects

Four well-characterised cohorts (Iowa,[13, 30] Boston,[38] Columbia,[13, 30] and Melbourne [46, 47]) and one recently acquired, but as yet unreported, cohort (Utah), together comprised 1,709 patients diagnosed with CNV and 1,473 disease-free controls (for which genotyping data were already available), were assessed (Table 1). All individuals were of white European ancestry, 60 years of age and older and matched for age. All patients had given their consent and were enrolled under Institutional Review Board-approved protocols. The methods used in this study conformed to the tenets of the Declaration of Helsinki (2000) of the World Medical Association. Study subjects were examined and photographed by trained ophthalmologists; fundus photographs were graded according to published standardised classification systems. The worst affected eye of each case was used for classification purposes. All cohorts were case-controlled, with the exception of the Boston sib-pair cohort. Index patients in the Boston cohort aged 60 years or older were included in the analyses and had CNV, (as defined by subretinal haemorrhage, fibrosis or fluorescein angiographic presence of neovascularisation documented at the time of, or prior to, enrolment in the study) in at least one eye. The unaffected siblings had normal maculae at an age older than that at which the index patient was first diagnosed with CNV, as previously described [38]. The Utah case-control cohort was recently ascertained at the John A. Moran Eye Center, University of Utah, in Salt Lake City, Utah, USA, in a fashion identical to that of the Iowa cohort.

Table 1 Number of cases (CNV disease) and controls in individual cohorts

Markers

Thirteen SNPs, spanning four physically separate genomic loci, were genotyped in all five cohorts (Table 2). One locus spans the CFH, CFHR4, CFHR5 and F13B genes and comprises nine SNPs; the second consists of two SNPs, one each in C2 and CFB; the third consists of a single SNP in C3; and the fourth consists of a single SNP in ARMS2. One of the CFH SNPs (rs12144939) included in the panel tags the CFHR3/1 deletion. The 13 SNPs were selected on the basis of the following characteristics: prior published replication, magnitude of estimated effect size and power to resolve clinically relevant haplotypes (CFH) [5–19].

Table 2 Single nucleotide polymorphisms employed in first stage

Statistical methods

Previous analyses of each cohort involved standard quality checks and exclusions. Prior to analysis, the consistency of the assignment of the DNA strand used to detect the SNPs was assessed for all available datasets and any inconsistencies resolved. The percentage of missing data and the genotype frequencies were calculated and tabulated for each SNP, both by study (data not shown) and overall (Table 3). No SNPs showed significant deviation from Hardy-Weinberg equilibrium in the control population (P > 0.05).

Table 3 Homogeneity of variance

In order to determine the appropriateness of pooling the available cohorts, a chi-squared test of homogeneity of allele frequency was applied to compare frequencies across cohorts. Cohorts or subcohorts found to be a source of a departure from homogeneity of allele frequency (chi square P < 0.001) were excluded from the main analysis.

Individuals with CNV were compared with the control group of subjects with no recorded disease. Genotypic multivariate and univariate unconditional logistic regression analyses were performed to evaluate the relationships between risk of CNV and the additively coded genotypes (Supplementary Analysis 1). Odds ratios (ORs) and 95 per cent confidence intervals (CIs) were calculated. The full 13-SNP panel was evaluated both with and without demographic factors of age and sex. Backward elimination was performed on the training set using a threshold of P < 0.05.

Two published test models containing, respectively, three and six SNPs, and a nine-SNP model generated from backward elimination, were compared with the 13-SNP panel in terms of AUC in training and independent validation. In the event that an SNP was not present in the 13-SNP panel, a SNP with demonstrated linkage disequilibrium was used as a surrogate.

Training of classifiers was performed using 500 cases and 500 controls balanced by age and sex and randomly selected from the whole cohort. The remaining 322 controls and 632 cases were used for validation. In both analyses, ten-fold cross-validation was applied [50]. The predicted probability of affliction for each subject was calculated by applying the inverse-logit function; sensitivity, specificity and AUC were derived to assess classification performance.

A risk score for CNV was calculated as follows: $Sj=\mathsf{\text{intercept}}+{\sum }_{i=1}^{13}\beta i*Xi$ where Sj is the risk score for subject j and βi is the adjusted log-odds ratio for Xi, the additively coded genotype at marker i. The probability of risk for subject j was calculated as pj = exp(Sj)/[1 +exp(Sj)].

The optimal classification threshold was determined on the basis of accuracy, defined as the proportion of correct predictions observed in cases and controls. Different levels of prevalence, reflecting age-specific differences, were considered. The accuracy in the validation set was determined, and positive and negative predictive values were calculated. Calibration was assessed graphically as histograms showing disease incidence at different levels of predicted risk for controls and cases.

The area under the receiver operating characteristic (ROC) curve and CIs were estimated using SAS Macro %ROC [51]. In addition, c-statistics and CIs were calculated for the training, tenfold cross-validation and validation datasets [52, 53].

All analyses were conducted using SAS 9.1 [52].

Results

The average ages (± standard deviation [SD]) of cases and controls among all cohorts were 76.4 (± 7.3) and 76.5 (± 7.1) years, respectively, and the differences were not significant (p = 0.86). Age matching was applied during cohort ascertainment. The chi-square test was used to assess homogeneity of allele frequency across cohorts. Frequencies of markers rs10490924, rs403846, rs1409153, rs698859, rs403846 and rs10922153 were significantly different (P < 0.001) across cohorts. The frequencies of four markers -- rs10490924, (ARMS2) rs403846, (CFH) rs1409153 (CFHR4) and rs10922153 (CFHR5) -- in the control population and two markers -- rs698859 (F13B) and rs403846 (CFH) -- in the CNV population were unbalanced (Table 3). Removal of the Columbia University cohort eliminated four of the five deviations, leaving only one SNP (rs10490924) outstanding in the Boston control population. The Boston controls and Columbia cases and controls were excluded from the main analyses based on these observations. The remaining study population contained 1,132 CNV cases and 822 controls. For the purposes of the current analysis, investigations into the differences were not pursued but could be evaluated in the future by performing structure analysis to identify potential causes for the observed differences.

Table 4 shows unadjusted association test results between the demographic and genetic factors and the risk of CNV. All factors except age were associated with risk of CNV. The c-statistic column shows the ability of a genetic factor to predict CNV risk. SNPs rs10490924, rs1061170, rs403846 and rs2274700 had c-statistics ≥0.65.

Table 4 Univariate association between demographic, genetic factors and risk of choroidal neovascular (CNV) disease

Table 5 displays multivariate adjusted ORs that were significantly associated with the risk of CNV, using the additive genotype model applied to the 13-SNP panel. The ARMS2 variant rs10490924 was positively associated with risk of CNV (OR 4.279, 95 per cent CI 3.346-5.472, p < 0.0001).

Table 5 Calculation of choroidal neovascular disease risk score: $S=\mathsf{\text{intercept}}+{\sum }_{i=1}^{13}\beta i*Xi$, where β and X are as follows

The performance of the 13-SNP panel to predict CNV relative to the control population was evaluated using tenfold cross-validation and an independent dataset. Independent datasets were scored using model parameters displayed in Table 5. Table 6 shows the AUC evaluated for training (AUC 0.82 [0.79-0.85]), tenfold cross-validation (AUC 0.81 [0.79-0.84]) and validation (AUC 0.79 [0.77-0.83]). The c-statistics results were identical to AUC. These data show that the difference in performance of the training and validation sets was not significant (P < 0.05). There were no significant differences between the AUC curves for the training and validation datasets with demographic factors (age, sex) added into the test model (Table 7), presumably due to the balanced study design.

Table 6 Area under the curve for training, tenfold cross-validation and independent validation on 13-SNP model

Table 7 Comparison of 13-SNP model with and without demographic factors

The sensitivity and specificity of predictions were calculated in an independent dataset using the test panels in Table 5. The ROC curve is shown in Figure 1. The probability of the risk of CNV was plotted as histograms for controls and cases in the independent dataset in Figure 2. It shows good separation between the two groups, with cases having a substantially higher probability of CNV, although some overlap is present.

Figure 1

ROC curve for validation. ROC, receiver operating characteristic.

Figure 2

Probability of choroidal neovascular (CNV) disease, calculated for validation dataset using model described in Table 2. Red bars represent controls and blue bars represent patients with CNV disease.

Accuracy, specificity, sensitivity, PPV and negative predicted values (NPV) are shown in Table 8 as a function of probability cut-off and three prevalence values. A cut-off of 0.4 corresponds to the highest accuracy (0.73), with a sensitivity of 0.82 and a specificity of 0.63. The PPV for 5.5 per cent, 10 per cent and 15 per cent prevalence values were 0.11, 0.20 and 0.28, respectively. The NPVs were all above 0.95.

Table 8 Classification table

We compared several published predictive models with our current 13-SNP panel (Table 9). The differences in test performance were evaluated at training and validation stages. The performance of the 13-SNP panel was slightly better than that of the next best test [41, 42]. Results from the nine-SNP panel generated from the backwards elimination procedure realised gains in genotyping efficiency, with four fewer variants in the panel, while demonstrating only slightly lower performance in terms of AUC.

Table 9 Comparison of models containing different numbers of single nucleotide polymorphisms (SNPs)

Discussion

Although the incorporation of non-static and self-reported variables is important in elucidating the modifiable risk factors that contribute to disease, their inclusion can degrade test performance in mainstream genetic testing. Ideally, a robust test panel, subject to rigorous validation, which captures the maximal genetic component should improve classification performance and accuracy of reporting. In line with these criteria, which are much stricter than in a discovery cohort, the Boston cohort controls and the Columbia cohort cases and controls were not considered for the calculation of the model. Possible explanations for the allele frequency deviations in these cohorts include admixture, cryptic population stratification, subtle differences in grading criteria, cohort age range, concomitant illnesses or medications, and should be explored further.

In order to compare performance across tests, a ROC curve was generated for each prediction panel to evaluate the AUC. By evaluating each test across the large collective cohort using the same validation procedure, we compared the power of the genetic variants to evaluate classification performance. The performance of the three-SNP panel described by Jakobsdottir and colleagues [41] revealed an AUC value of 0.77, compared with a value of 0.79 observed in the original study of 642 late-stage AMD cases and 142 controls. The differences in AUC values obtained between the original and the current study are likely to reflect the impact of testing across a large collection of independently collected cohorts compared with a single study that is potentially more sensitive to subject selection bias. The performance of the six-SNP test panel reported by Seddon and colleagues [42] as part of a joint gene-environment model exhibited a drop in AUC from 0.81 to 0.79 from training to validation in our data (significant at P < 0.05), similar to most of the tests evaluated. This decrease in AUC reveals the value of the inclusion of an independent validation set to challenge test performance and estimate metrics achievable in the broader clinical setting more accurately. We have emphasised the importance of both study design features to report performance more accurately and to anticipate utility in the more diverse clinical testing market more closely. Finally, modest gains in our 13-SNP panel were demonstrated with the highest AUC value obtained among all models evaluated (0.80). The additional variants that contributed to the performance of the predictive test located in CFHR5 and F13B highlight the complexity of the genetic structure of the RCA region and influence AMD disease biology.

In summary, the 13-SNP panel had a clinical sensitivity of 82 per cent and a specificity of 63 per cent, achieving clinical performance metrics comparable with models with fewer SNPs that include self-reported and/or non-static risk factors. The PPV of the panel was evaluated at different levels of prevalence, reflecting ranges covering estimates of late-stage disease in individuals > 40, > 65 and > 80 years of age in the general population. More favourable estimates of PPV were observed as the prevalence of disease increases with age. The values obtained revealed 11 per cent PPV at 5.5 per cent prevalence, 20 per cent PPV at 10 per cent prevalence and 28 per cent PPV at 15 per cent prevalence in the general population [41]. The prevalence figures reflect conservative estimates of late-stage disease in the general population and would be further enhanced and more clinically applicable in a setting of diseased patients, as in the study conducted by Seddon and colleagues [42]. The longitudinal study design of the Age-Related Eye Disease Study (AREDS) cohort used in Seddon's study was ideal for evaluating incident AMD by distinguishing between 'progressors' and 'non-progressors' but, more importantly, it established that the same set of variants were effective at distinguishing non-disease controls from patients with late-stage disease. Not surprisingly, the same core panel of SNPs covering the major genes associated with disease used in Seddon and co-workers' test panel was also utilised in the study conducted by Jakobsdottir and colleagues,[41] as well as in our current study.

The present confirmatory findings reflect the utility of these variants to predict the development of CNV in non-diseased subjects in our study, as well as the progression to late-stage disease in patients diagnosed with early forms of AMD [42]. PPVs improve significantly when applied to the population of patients diagnosed with early stages of disease. The utility of AMD genetic testing will advance if the result of a predictive test translates into actionable information for the physician. This study highlights the need to continue to explore the biology of CNV, to improve our understanding of the genetics associated with disease and extend these findings in future studies to evaluate clinical performance metrics in the more acute clinical population diagnosed with early-stage disease. A genetic test identifying individuals at high risk of developing CNV holds the promise for earlier detection through risk-based surveillance protocols and improved outcomes arising from more timely intervention.

Supplementary Analysis 1

Logistic regression results

Table 10

Backward elimination procedure

Table 11 Step 0. The following effects were entered: Intercept rs10490924 rs1061170 rs10922153 rs12144939 rs1409153 rs1750311 rs2230199 rs2274700 rs2990510 rs403846 rs641153 rs698859 rs9332739

Table 12 Step 1. Effect rs698859 is removed:

Table 13 Step 2. Effect rs1409153 is removed:

Table 14 Step 3. Effect rs1750311 is removed:

Table 15 Step 4. Effect rs12144939 is removed: