Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Biased efficacy estimates in phase-III dengue vaccine trials due to heterogeneous exposure and differential detectability of primary infections across trial arms

  • Guido España ,

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft

    guido.espana@nd.edu (GE); taperkins@nd.edu (TAP)

    Affiliation Department of Biological Sciences and Eck Institute for Global Health, University of Notre Dame, Notre Dame, IN, United States of America

  • Cosmina Hogea,

    Roles Conceptualization, Investigation, Writing – review & editing

    Affiliation GlaxoSmithKline, Rockville, MD, United States of America

  • Adrienne Guignard,

    Roles Conceptualization, Writing – review & editing

    Affiliation GlaxoSmithKline, Rockville, MD, United States of America

  • Quirine A. ten Bosch,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Department of Biological Sciences and Eck Institute for Global Health, University of Notre Dame, Notre Dame, IN, United States of America

  • Amy C. Morrison,

    Roles Writing – review & editing

    Affiliations United States Naval Medical Research Unit No. 6, Lima, Peru, Department of Entomology and Nematology, University of California, Davis, CA, United States of America

  • David L. Smith,

    Roles Writing – review & editing

    Affiliation Institute for Health Metrics and Evaluation, University of Washington, Seattle, WA, United States of America

  • Thomas W. Scott,

    Roles Writing – review & editing

    Affiliation Department of Entomology and Nematology, University of California, Davis, CA, United States of America

  • Alexander Schmidt,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation GlaxoSmithKline, Rockville, MD, United States of America

  • T. Alex Perkins

    Roles Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Resources, Supervision, Visualization, Writing – original draft

    guido.espana@nd.edu (GE); taperkins@nd.edu (TAP)

    Affiliation Department of Biological Sciences and Eck Institute for Global Health, University of Notre Dame, Notre Dame, IN, United States of America

Abstract

Vaccine efficacy (VE) estimates are crucial for assessing the suitability of dengue vaccine candidates for public health implementation, but efficacy trials are subject to a known bias to estimate VE toward the null if heterogeneous exposure is not accounted for in the analysis of trial data. In light of many well-characterized sources of heterogeneity in dengue virus (DENV) transmission, our goal was to estimate the potential magnitude of this bias in VE estimates for a hypothetical dengue vaccine. To ensure that we realistically modeled heterogeneous exposure, we simulated city-wide DENV transmission and vaccine trial protocols using an agent-based model calibrated with entomological and epidemiological data from long-term field studies in Iquitos, Peru. By simulating a vaccine with a true VE of 0.8 in 1,000 replicate trials each designed to attain 90% power, we found that conventional methods underestimated VE by as much as 21% due to heterogeneous exposure. Accounting for the number of exposures in the vaccine and placebo arms eliminated this bias completely, and the more realistic option of including a frailty term to model exposure as a random effect reduced this bias partially. We also discovered a distinct bias in VE estimates away from the null due to lower detectability of primary DENV infections among seronegative individuals in the vaccinated group. This difference in detectability resulted from our assumption that primary infections in vaccinees who are seronegative at baseline resemble secondary infections, which experience a shorter window of detectable viremia due to a quicker immune response. This resulted in an artefactual finding that VE estimates for the seronegative group were approximately 1% greater than for the seropositive group. Simulation models of vaccine trials that account for these factors can be used to anticipate the extent of bias in field trials and to aid in their interpretation.

Introduction

An estimated 390 million people worldwide experience dengue virus (DENV) infections each year, with approximately 96 million of those showing clinically apparent symptoms [1]. Only one dengue vaccine (Dengvaxia, Sanofi Pasteur) has been licensed to date, although safety concerns have limited its rollout [2]. Beyond Dengvaxia, 22 investigational dengue vaccines are at various stages of development, with six of them in clinical trials [3]. Two tetravalent vaccines have advanced to phase-III clinical trials to evaluate vaccine efficacy against symptomatic, virologically-confirmed dengue disease [4,5]. In addition, 14 investigational Zika vaccines are under development, some of which may also advance to late-phase trials [6]. Late-phase vaccine trials for dengue, Zika, and chikungunya—three viruses with a common mosquito vector in Aedes aegypti—share a number of common challenges.

A major challenge for vaccine trials for these diseases derives from the observation that their transmission is characterized by extensive spatial and temporal heterogeneity. For example, epidemiological studies of dengue in both rural [7] and urban [8] settings in Thailand showed that transmission tends to occur focally at spatial scales of 1 km or less and can be much more intense at some times than others. Similar observations are being made for Zika and chikungunya [9,10]. Some of the complications posed to vaccine trials by such strong spatial and temporal heterogeneity include achieving balanced randomization with respect to exposure across trial arms and correctly anticipating exposure rates in advance of a trial.

Another complication for vaccine trials for these diseases also derives from heterogeneity in transmission, but at much finer scales. Field studies [11,12] have shown that patterns of Ae. aegypti blood-feeding are extremely heterogeneous; i.e., most of the mosquito bites are concentrated on a few, while other people are rarely bitten [13]. These heterogeneities apply to individuals within a single household and become compounded even further [14] when layered on top of house-to-house variation in Ae. aegypti densities [15] and complex patterns of human [16,17] and mosquito [18] movement, which affect human-mosquito contact. This fine-scale variation results in heterogeneity in the number of DENV exposures that trial participants experience, which can lead to bias in estimates of vaccine efficacy for susceptibility (VES) [19] for “leaky” vaccines, which confer partial protection to all vaccinees rather than full protection to a subset [20]. This type of bias applies to VES estimates derived using survival methods, such as Cox regression models, unless trial participants experience homogeneity in exposure [21,22]. In the presence of heterogeneities in exposure, frailty models have been proposed to reduce bias in VES estimates [21]. Measuring exposure or collecting data on risk factors can also help in reducing this bias [22].

Because heterogeneity in exposure is known to induce bias in VES estimates for leaky vaccines and because there is so much heterogeneity in DENV transmission that would be nearly impossible to account for explicitly in a trial, we used a simulation-based approach to assess the extent to which heterogeneous exposure may bias VES estimates for dengue vaccines. Leveraging detailed historical understanding of DENV transmission in Iquitos, Peru, we used this model to explore plausible outcomes for hypothetical vaccine efficacy trials that were simulated as if they took place at a specific point in time during the past in Iquitos. This exercise is an initial case study on which additional analyses can be developed for other specific settings of interest as data from other vaccine trial sites become available and amenable to modeling.

Methods

Trial overview

We designed a hypothetical, individually randomized phase-III trial of a generic dengue vaccine. The aim of the trial was to evaluate the vaccine efficacy for susceptibility (VES) as a measure of the protective effects of vaccination against the first symptomatic, virologically-confirmed case of dengue caused by any DENV serotype during simulated vaccine trials in Iquitos, Peru. A virologically-confirmed dengue case was defined as febrile illness with temperature ≥ 38°C for at least two consecutive days and a positive result from a reverse-transcriptase polymerase chain reaction (RT-PCR) from a blood sample. This endpoint has been used in phase-III dengue vaccine trials and is recommended by the WHO [23]. We chose to simulate the trial in Iquitos because of its long history of epidemiological studies of dengue, which provide a unique opportunity to parameterize and validate detailed models of DENV transmission [16,2427]. Iquitos is a city of approximately 370,000 inhabitants, located in the Amazon Basin and accessible only by boat or airplane.

We simulated the active phase of our hypothetical dengue vaccine trials comparable to previous and ongoing trials [5,2832] (Table 1), such that ours consisted of regular contacts with study participants for at least 13 months after completion of a full course of vaccination (two doses, two months apart). Simulated trial participants were randomly assigned to a placebo or vaccine arm in a 1:1 ratio. We assumed that the placebo conferred no protection against clinical endpoints and that the vaccine conferred some level of protection defined as an input of the model and estimated as VES with the data collected from the trial. In all simulations, we initialized the trial on June 26, 2009. The period thereafter was characterized by a relatively high force of infection of multiple DENV serotypes [27], which would have been ideal for a trial.

thumbnail
Table 1. Characteristics of four vaccine trial designs for dengue vaccines, in addition to our virtual trial.

https://doi.org/10.1371/journal.pone.0210041.t001

Vaccine model

We modeled a hypothetical vaccine with a per-exposure protection (PEP) that represents the reduction in risk of disease that the vaccine confers to an individual following a single exposure to DENV. The vaccine reduces the risk of two separate events: successful DENV infection conditional on exposure (RRinf|exp) and clinically apparent dengue fever conditional on DENV infection (RRdis|inf), such that (1) where RRinf|exp, RRdis|inf, and ρ are specified as inputs to the model. Our default assumption was that the vaccine protects equally against infection conditional on exposure and against disease conditional on infection (ρ = 0.5). Our default assumptions about RRinf|exp and RRdis|inf were that they both equaled 0.447, consistent with an exploratory PEP value of 0.8 as assumed in planning for the ongoing Butantan trial [31]. In our model, the vaccine does not modify the probability of progressing from symptomatic to severe disease, because the focus of our analysis is on the primary endpoint of virologically-confirmed disease rather than the more severe outcomes that can sometimes be associated with dengue. Even though dengue vaccines seem to have heterogeneous effects in protection [29], we assumed constant protection with respect to time since inoculation, serotype, and serostatus, to simplify the vaccine model and to isolate the effect of heterogeneous exposure from the effect of heterogeneous vaccine protection.

Virtual trial procedures

Our virtual phase-III trial featured three main processes: recruitment, vaccination, and surveillance. These processes comprise eight specific procedures, which are depicted in Fig 1. The recruitment phase included (1) enrollment of participants and (2) serological testing to assess for previous DENV infection. Vaccination included (3,4) administration of two doses of vaccine two months apart. Next, study personnel (5) captured cases during weekly phone calls to participants, and a proportion of (6) individuals with symptoms notified trial personnel. Confirmation of DENV infection (7) was attempted with a serotype-specific RT-PCR test, and (8) confirmed cases were recorded in surveillance reports. The trial considers only the first case of symptomatic dengue for each individual in estimates of VES.

thumbnail
Fig 1. Modeling framework and trial procedures.

DENV is transmitted by individual mosquitoes that are found indoors. Humans move daily to different types of locations within the city. Infectious mosquitoes transmit DENV through bites to individuals susceptible to the infecting serotype who are present at a location at the same time as mosquitoes seeking a blood meal. The trial consists of eight procedures listed in the figure. The top curve shows the timing of these procedures in reference to DENV transmission dynamics, and the bottom cartoon depicts each of the procedures.

https://doi.org/10.1371/journal.pone.0210041.g001

Recruitment.

We enrolled simulated trial participants during the first two months of the trial. We randomly selected a subset of the simulated population who were 5–45 years old (y.o.). The number of participants was consistent with the desired sample size to achieve a statistical power of 0.9[33], as described in S1 Text. Because DENV exposure differs between children and adults, we recruited participants in such a way that half were children (5–17 y.o.) and half were adults (18–45 y.o.). Within each of those two groups, each participant was randomly assigned to the vaccine or placebo arm on a 1:1 basis. Next, a participant’s pre-existing seropositivity to DENV was recorded, which we assumed could be measured accurately for all trial participants. Individuals without previous exposure to DENV were classified as seronegative and individuals with previous exposure to any of the four DENV serotypes were classified as seropositive, without specification of the number or type of the previous DENV infections.

Vaccination.

In the simulation, participants received the first dose of the vaccine or placebo on the day of enrollment. The vaccine conferred PEP defined by the vaccine model, whereas the placebo conferred no protection against infection or disease. Each participant received two doses of vaccine or placebo within sixty days of enrollment. This choice was intermediate between two dengue vaccines currently in phase-III trials (Takeda: two doses 90 days apart [32]; Butantan: one dose [31]). We assumed that the full PEP was conferred by the first dose, its longevity boosted by the second dose, and that all trial participants received both doses on schedule. Consistent with observations of modest efficacy despite high immunogenicity in the only dengue vaccine for which efficacy has been assessed [34] and with assumptions of a model developed by a dengue vaccine manufacturer [35], we assumed that the vaccine was leaky: i.e., partial protection applies uniformly to all vaccinees, as opposed to full protection of a subset of vaccinees.

Surveillance.

Surveillance started on the day on which the first dose of vaccine was administered, but cases used for the VES analysis were restricted to those that occurred 28 days or more after administration of the second dose, similar to the protocol for the CYD-TDV trials [36]. The surveillance system captured symptomatic cases in two different ways: weekly phone calls to trial participants by trial personnel and self-reporting. Weekly active surveillance contacted participants to inquire about disease symptoms experienced since the last contact with the surveillance system. Half of the individuals with symptoms of dengue fever reported their symptoms during the weekly surveillance phone calls or sought care, which allowed for laboratory diagnosis. Everyone with severe dengue reported their symptoms to the surveillance system. The probability of self-reporting mild dengue cases was higher than estimates of this probability under circumstances outside the context of a trial [37] (5–15%), given that we expect that people would be more likely to use free healthcare provided as part of their participation in the trial. Some DENV infections go undetected because they are asymptomatic or are not virologically-confirmed by the surveillance system. Because these infections are not detected by the surveillance system, their enrollment status in the trial remains unchanged. To confirm the presence of DENV, we simulated an RT-PCR test in participants with two consecutive days of symptoms and up to five days after symptom onset in agreement with current guidelines [38].

To simulate virological confirmation of cases, we modeled the sensitivity of a RT-PCR test as a function of time since symptom onset. To obtain such a function, we first obtained 3,000 simulated viremia trajectories from a model of viremia dynamics for primary and post-primary DENV infections that was fitted to data from individuals enrolled in a clinical study in Vietnam [39]. We then applied a fixed limit of detection (LOD) of 10 cDNA copies per ml to the simulated viremia trajectories and recorded the proportion of those 3,000 simulated trajectories that exceeded that LOD at a given time since symptom onset, which we regarded as the sensitivity of RT-PCR at a given time since symptom onset. We then fitted the function (2) to the simulated values of sensitivity t days after symptom onset in primary and post-primary infections i using the optim function in R [40], obtaining curves depicted in S1 Fig. We assumed that viremia trajectories corresponding to post-primary infections also applied to vaccinated individuals, given that both should exhibit an antibody response to DENV infection that would result in a shortening of DENV infection and, thus, a shorter window of detectability by RT-PCR.

Statistical analyses

We used surveillance records resulting from a given virtual vaccine trial to estimate VES based on the timing of when trial participants attained the primary endpoint as detected by the surveillance system. The model simulated censored observations; i.e., some individuals dropped out of the trial before being infected, whereas others finished the trial without being exposed to DENV. Ignoring this feature of the data can result in overestimation of vaccine efficacy [19]. We estimated the vaccine efficacy for susceptibility (VES) as a measure of how protective the vaccine was against disease [19]. VES was calculated based on three approaches: incidence rates, survival methods, and transmission probability. Based on incidence rates, VES,IR was calculated as (3) where c represents the number of cases detected in each arm of the study, y the number of person-years within the trial, and the subscripts v and p represent the vaccine and placebo arms, respectively [19]. To calculate VES based on survival methods, we estimated VES,Cox with a Cox model in R [40] using the coxph function with default settings [41]. The survival model was implemented with the data represented by the time to first event (or follow-up time) and a binary variable for PCR positivity. In an attempt to account for heterogeneous exposure within the trial, we also considered the addition of a frailty expression—in which heterogeneity in exposure was defined as a random effect by a gamma distribution—to the Cox-regression model and estimated VES,frailty using the same coxph function in R with an additional frailty term specified as frailty(ID, distribution = "gamma").

The above methods do not require information about the level of exposure of individuals in the trial. Such measures are often used in phase-III trials to measure VES because information on exposure to infection may be unfeasible to obtain in field trials. This task is particularly difficult for mosquito-borne diseases, because individual mosquito bites would need to be recorded to estimate the level of exposure of trial participants. Our model allowed us to record the exact number of exposures that every individual experienced during the trial, defined as an infectious bite received by a susceptible human. Knowledge on the exact number of exposures allowed us to estimate VES based on the probability of transmission ‘p’, which should be robust to heterogeneous exposure. The measure VES,p was defined as (4) where cv and cp represent the number of cases in the vaccine and placebo arm, respectively, and nv and np represent the total number of exposures in the vaccine and placebo arms, respectively [19]. Comparison between this and other VES measures was used to aid in the interpretation of our results.

Model overview

We simulated vaccine trials using an agent-based model of DENV transmission that simulates the transmission dynamics of DENV in a population of 200,000 in a core population area of Iquitos, Peru [42,43]. Our model includes simulated daily human movement patterns calibrated to data from retrospective, semi-structured interviews of residents in Iquitos [44,45]. This model has been demonstrated to reproduce the dynamics of all four DENV serotypes [42] and was used to investigate the epidemiological impacts of hypothetical vaccination campaigns to project the population-level impact of Dengvaxia [46,47]. Because a formal description of the version of the transmission model that was used here has been published elsewhere [42], we provide an overview of its key features and assumptions in S2 Text.

Model calibration

We used parameter estimates from the literature wherever possible (Table 2) and calibrated the values of unknown parameters so as to allow the model to generate patterns of transmission within the study area consistent with time-varying incidence of infection estimates from a longitudinal cohort study [27]. Three sets of parameters were calibrated through this process: (1) serotype-specific population immunity on 1 January 2000, when estimates of time-varying, serotype-specific incidence of infection began; (2) time-varying, serotype-specific patterns of unexplained infections from visitors to the study area or residents that were infected outside the study area; and (3) constants for scaling mosquito population density at two different time points during which mosquitoes were collected using different methods. We calibrated these parameters using a particle filtering algorithm described in detail elsewhere (Table 3) [42].

Simulation experiments

Our primary goal was to quantify bias in VES estimates insofar as they relate to estimates of individual protection afforded by the vaccine, as defined in our simulation model by PEP. This individual-level interpretation is significant, because VES estimates from standard methods represent a weighted average of vaccine effects at the population level. Therefore, whenever the assumption of homogeneous exposure is not met, VES estimates are specific to exposure patterns in the population where the vaccine was evaluated [22]. To the extent that VES estimates made in this way might be used to inform projections of vaccination impact in other populations (due to what is usually a lack of a direct estimate of PEP), understanding the extent of bias in estimates of PEP based on VES is important [22]. To address this driving question, we performed three simulation experiments under different assumptions about heterogeneity in DENV transmission patterns.

1. Baseline scenario.

The trial design under the baseline scenario was described in the Trial Overview and Sample Size sections. In this scenario, the attractiveness of humans to mosquitoes was proportional to the human’s body size [11]. The timeframe of the trial ranged one to three years depending on when 68 virologically-confirmed symptomatic dengue cases were captured. We simulated 1,000 replicates of this trial with an input PEP = 0.8 and quantified the bias of the estimated VES relative to the input PEP. We also simulated the trial with an alternative assumption about the viremia of infected vaccinees, in which their viremia followed that of someone experiencing a natural DENV infection. We quantified differences between estimated VES and simulated PEP under both assumptions of viremia of infected vaccinees.

We computed the probability that the point estimate of VES was lower than a specified minimum product profile (MPP) below which the vaccine would not be licensed or further developed. We varied the value of MPP (0.1–0.7) and calculated this probability given a PEP equal to the target product profile (TPP) (VE ≥ 0.8). We varied the value of PEP (0.1–0.95) and computed the proportion of trials resulting in VES falling below MPP = 0.3.

2. Effects of heterogeneous exposure due to heterogeneous biting.

Heterogeneity in the number of DENV exposures experienced by participants in each trial arm can affect trial results. In our model, an exposure is defined as a bite from an infected mosquito that would result in a human infection in the absence of vaccination. Hence, placebo participants can only experience one exposure before developing infection. Conversely, participants in the vaccine arm can experience multiple exposures before infection due to the assumption that the vaccine is leaky. We simulated four scenarios about the nature of heterogeneous exposure, illustrated in Fig 2, to investigate how different forms of exposure heterogeneity affect VES estimates.

thumbnail
Fig 2. Four distinct assumptions about heterogeneous DENV exposure.

Scenario A represents homogeneous transmission of DENV ignoring mosquitoes and human movement patterns, leading to everyone experiencing the same hazard of exposure to any serotype of DENV. In scenarios B, C, and D, DENV is transmitted by mosquitoes, and human and mosquito movement patterns are considered. Scenario B represents homogeneous attractiveness of different humans to mosquitoes at a given location. Scenario C represents human attractiveness to mosquitoes proportional to human body size. Scenario D represents heterogeneous human attractiveness to mosquitoes based on a gamma distribution with shape 1.43 and scale 1, regardless of their body size.

https://doi.org/10.1371/journal.pone.0210041.g002

The different forms of exposure heterogeneity that we considered (Fig 2) come about through:

  1. Homogeneous hazard of exposure, in which mosquitoes are bypassed altogether and individuals are infected directly based on a time-varying rate of DENV infection.
  2. Constant attractiveness, in which mosquitoes are equally attracted to all humans for blood feeding at a location.
  3. Attractiveness proportional to body surface area, in which mosquitoes are more likely to bite humans with larger bodies [11].
  4. Heterogeneous attractiveness, in which each individual’s attractiveness is drawn from a gamma distribution with shape and rate , as informed by data from de Benedictis et al. [12].

In addition to heterogeneity in attractiveness to mosquitoes, heterogeneous exposure in the transmission model results from spatial heterogeneity in contact between mosquitoes and humans, due to spatial patterns of mosquito abundance and human mobility. This is true for all scenarios except A. In scenario A, infections during a trial occur only by a time-varying hazard of DENV infection. We adjusted this hazard to be the same as the force of infection in the remaining three scenarios so that within the trial the AR would be comparable among scenarios. We compared the distributions of VES estimates under these four scenarios by performing 1,000 simulations, with analogous simulations in each set that had the same sequence of random number seeds to minimize differences due to chance across the four scenarios.

3. Effect of increased heterogeneity of exposure due to high transmission intensity.

We hypothesized that phase-III trials in sites with high transmission intensity yield more biased vaccine efficacy estimates than trials performed with moderate to low transmission intensity, due to an increased number of exposures in the vaccine arm. Our goal was to quantify the potential magnitude of this effect in our simulated trials by computing bias in VES estimates in a scenario with high transmission intensity. We achieved variation in transmission intensity by varying mosquito emergence rates, which scale the overall density of adult mosquitoes and have a linear effect on the force of infection. Specifically, we doubled mosquito emergence rates in the high-transmission scenario, and we calculated required sample sizes separately for each trial to attain the same statistical power for VES estimates under both scenarios. With these specifications, we simulated 1,000 replicates for each scenario with paired random seeds to allow the behavior of the simulations to be as comparable as possible.

Results

1. Baseline scenario

Analysis of the baseline simulations showed a general pattern of VES underestimating the simulated value of PEP (0.8) when using VES,IR (0.71) and VES,Cox (0.71). Compared to VES,IR and VES,Cox, the frailty model slightly reduced bias in its estimate of VES,frailty (0.73). Knowledge of the number of exposures in each trial arm was associated with a slight overestimation of efficacy by VES,p (0.81) (Fig 3A). Although these patterns held qualitatively across the full range of assumptions about the extent to which the vaccine protects against infection versus disease (modulated by the parameter ρ), the magnitude of bias in VES,IR, VES,Cox, and VES,frailty was somewhat sensitive to ρ (S3 Fig). Specifically, these biases were stronger when protection derived more from reducing the risk of infection conditional on exposure and less from reducing the risk of disease conditional on infection (i.e., ρ closer to 1). The slight upward bias of VES,p by 0.01 was unexpected, because VES,p should be capable of fully accounting for bias due to heterogeneous exposure [19]. We also noted that the extent of bias was different between seropositive (Fig 3B) and seronegative (Fig 3C) groups. In particular, bias towards the null was 0.02 higher in the seropositive group for all VES measures (Fig 3B). Even though this was a small difference, an upward bias in the seronegative group could partially mask the magnitude of the bias due to heterogeneous exposure in the overall VES estimates.

thumbnail
Fig 3. Overall VES and VES with stratification estimated for 1,000 simulations of the baseline scenario for one site.

In this scenario, attractiveness of humans is proportional to their body size. VES is calculated with four different methods: VES,Cox, VES,IR, VES,CI, and VES,p. Violin plots on the left panel show the shape of the distribution of VES,Cox estimates. Colored solid lines show the 25, 50, and 75 quantiles for VES,Cox, VES,p, VES,IR, and VES,frailty (gray, blue, red, and green, respectively). Black solid line represents the true PEP = 1—RRinf|exp RRdis|inf = 0.8. Left panel corresponds to the assumption of viremia in vaccinated individuals as secondary infections. Right panel corresponds to the assumption of viremia in vaccinated individuals as a natural infection.

https://doi.org/10.1371/journal.pone.0210041.g003

After reviewing aspects of the simulation model that could have differed between seropositive and seronegative individuals, we hypothesized that a bias away from the null that is distinct from the bias due to heterogeneous exposure occurred as a result of the way in which we simulated viremia trajectories in infections of vaccinated individuals. Because these infections were defined as secondary-like infections with shorter viremia periods (S1 Fig), fewer primary infections in the vaccine arm were detected by the RT-PCR than in the placebo arm. This resulted in VES estimates being approximately 0.01 higher on average than they would have been if the viremia trajectories in primary vaccinee infections had been the same as they were in unvaccinated individuals (Fig 3C and 3F). This bias increased to 0.06 (S2C and S2F Fig) in a scenario in which RT-PCR was used up to seven days after symptom onset, in contrast to the baseline of five days. This effect was apparent for seronegative individuals (for whom there were differences in RT-PCR sensitivity between vaccinated and unvaccinated) but not for seropositive individuals (for whom RT-PCR sensitivity was equal for vaccinated and unvaccinated) (Fig 3B and 3C). We interpret this bias to have occurred because differences in RT-PCR sensitivity among seronegative trial participants resulted in vaccinated individuals being perceived to experience an artificially low number of infections. As expected under this interpretation, this bias disappeared in simulations in which we assumed that RT-PCR sensitivity was the same for vaccinated and unvaccinated individuals (Figs 3F and S2F). Furthermore, these individuals were also protected from infection later in the trial, which was perceived to result from vaccination but in reality resulted from heterotypic immunity following natural infection. Assumptions about viremia did not affect VES estimates for the seropositive population (Fig 3B and 3E), because their viremia curves were the same for vaccinated and unvaccinated individuals.

Because differential detectability resulted in a bias away from the null under the baseline scenario, it had the effect of slightly reducing our estimate of the extent to which heterogeneous exposure induced bias toward the null. VES estimates from the seropositive group were unaffected by the bias due to differential detectability and, therefore, provided a clearer indication of the extent of bias due to heterogeneous exposure. Across all four measures of VES that we examined, bias was approximately 0.01 more toward the null in the seropositive group than in the trial as a whole (Fig 3F vs 3A). This relatively small difference between the overall VES estimate and the VES estimate for the seropositive group can be attributed to the majority of trial participants being seropositive (0.6, CI = [0.57,0.63]) and the majority of the cases coming from seropositive individuals (0.66, CI = [0.55, 0.76]). Consequently, the extent to which bias away from the null due to differential detectability counteracts bias toward the null due to heterogeneous exposure would be expected to differ in trials in which the proportion of seropositive trial participants differs from our simulations. In summary, using conventional measures of VES that do not account for these biases (i.e., VES,Cox and VES,IR), analyses of our simulations with PEP = 0.8 resulted in VES estimates that were biased toward the null by approximately 0.09 due to heterogeneous exposure. Estimates of VES were biased away from the null by as much as 0.01 due to differential detectability of primary infections between vaccinated and unvaccinated individuals.

As one way to place these biases into context, we examined the proportion of simulated trials in which the vaccine would not have been licensed on the basis of the VES point estimate falling below the minimum product profile (MPP). For an MPP of 0.3, this probability depended on the VES measure used at different values of PEP. This probability was always lower for the VES,p measure, compared to the other three measures used (Fig 4). For instance, for PEP = 0.4, the vaccine would not have been licensed in nearly 40% of simulated trials on the basis of the point estimates of VES,Cox, VES,frailty, and VES,IR. It would not have been licensed in almost all simulated trials if the lower bound of the confidence interval of the VES estimates were used as a criterion for licensure. For PEP = 0.4, the probability of a vaccine not being licensed on the basis of the point estimate of VES,p was only around 0.2, and around 0.8 when based on the lower bound of the confidence interval. Across these and other instances, there was a lower probability of rejecting a vaccine with PEP exceeding MPP whenever PEP was farther away from MEP and when bias toward the null was less severe.

thumbnail
Fig 4. Probability of not licensing a vaccine with MPP = 0.3 at different input PEP values.

Solid lines represent the probability of rejection calculated as the proportion of simulations where the lower bound of the confidence interval of VES was below MPP. Dotted lines represent the probability of rejection calculated as the proportion of simulations where the mean estimate of VES was below MPP.

https://doi.org/10.1371/journal.pone.0210041.g004

2. Effects of heterogeneous exposure due to heterogeneous biting

The scenario with homogeneous hazard of exposure was associated with less bias toward the null compared to the three scenarios that allowed for some form of heterogeneous hazard of exposure (Fig 5A compared with Fig 5B–5D). When the hazard of exposure was homogeneous, VES,p and VES,frailty were approximately equal to PEP (VES,p = 0.8, VES,frailty = 0.79), and the other two measures slightly underestimated it (VES,Cox = 0.77, VES,IR = 0.77) (Fig 5A). As in the baseline scenario described above, overall estimates of VES reflect a combination of distinct VES estimates for the seropositive and seronegative groups, due to a bias away from the null resulting from differential detectability of primary infections among vaccinated and unvaccinated individuals. Among seropositive individuals, VES,p was the only measure that appeared to be unbiased (blue in Fig 6A). The other three measures exhibited some bias toward the null (red, gray, green in Fig 6A), because a homogeneous hazard of exposure still results in some degree of heterogeneity in the numbers of exposures that individuals experience (S4 Fig). Among seronegative individuals in the vaccinated group, the downward bias of heterogeneous exposure canceled out with the upward bias of lower detectability of primary infections and VES,Cox and VES,IR showed unbiased estimates of VES, while VES,p and VES,frailty displayed bias away from the null for this group (Fig 6E).

thumbnail
Fig 5. Vaccine efficacy in four scenarios about heterogeneous exposure to DENV for seropositive and seronegative individuals.

Violin plots show the shape of the distribution of VES,Cox estimates. The colored solid lines show the 25, 50, and 75 quantiles for VES,Cox, VES,p, VES,IR, and VES,frailty (gray, blue, red, and green, respectively). Black solid line represents the individual input of vaccine efficacy: VE = 1 –PEP = 1—RRinf|exp RRdis|inf = 0.8. The left panel corresponds to VES in seropositives and the right panel corresponds to the VES in seronegatives.

https://doi.org/10.1371/journal.pone.0210041.g005

thumbnail
Fig 6. Vaccine efficacy in four scenarios about heterogeneous exposure to DENV for seropositive and seronegative individuals.

Violin plots show the shape of the distribution of VES,Cox estimates. The colored solid lines show the 25, 50, and 75 quantiles for VES,Cox, VES,p, VES,IR, and VES,frailty (gray, blue, red, and green, respectively). Black solid line represents the individual input of vaccine efficacy: VE = 1 –PEP = 1—RRinf|exp RRdis|inf = 0.8. The left panel corresponds to VES in seropositives and the right panel corresponds to the VES in seronegatives.

https://doi.org/10.1371/journal.pone.0210041.g006

The scenarios with heterogeneous hazard of exposure resulted in a larger bias toward the null when using VES,Cox, VES,IR, and VES,frailty than when using VES,p (blue closer to PEP than gray, red, or green in Fig 5B–5D). These estimates were all within approximately 0.01 of each other under all three scenarios that allowed for heterogeneous attractiveness of humans to mosquitoes (right three columns in S5 Fig), as opposed to a difference of approximately 0.05 between the homogeneous scenario and each of the three heterogeneous scenarios (left three columns in S5 Fig). This indicates that a substantial degree of heterogeneity in exposure derives from spatial heterogeneity in mosquito density among different houses and heterogeneity in how individual humans sample that heterogeneity through their movement patterns, whereas relatively little derives from heterogeneous blood feeding within a single house. Examination of the distribution of the number of exposures experienced by trial participants further supports this conclusion, with higher variance in the number of exposures under the three heterogeneous scenarios than under the homogeneous scenario (S4 Fig).

3. Effect of increased heterogeneity of exposure due to high transmission intensity

Higher transmission intensity increased the bias in VES estimates even further toward the null because of a higher chance of repeated exposure in the population at risk, which increased heterogeneity in the number of exposures. This increased heterogeneity also affected the scenario with homogeneous hazard of infection (Fig 7A), because homogeneity in the hazard of exposure does not ensure homogeneity in the number of exposures that individuals experience during the trial. Some individuals in this scenario were exposed up to five times during the trial (S6 Fig), compared to a maximum of three exposures under the baseline transmission intensity assumptions (S4 Fig). Under the homogeneous hazard scenario, the frailty model did a relatively good job of reducing bias toward the null (VES,frailty = 0.77, compared with VES,Cox = 0.64, VES,IR = 0.64), but it was less successful at doing so under the three heterogeneous hazard scenarios (VES,frailty = 0.69, compared with VES,Cox = 0.66, VES,IR = 0.66) (Fig 7B–7D). Accounting for exposure with VES,p resulted in unbiased estimates.

thumbnail
Fig 7. Overall VES estimates from 1,000 simulations of each setting with different assumptions about heterogeneous exposure in a high-intensity setting.

Sample size was adjusted to 800 participants to achieve 90% statistical power in a higher transmission setting. Each row represents an assumption about heterogeneous exposure to DENV: homogeneous hazard of exposure, constant attractiveness of humans, attractiveness of humans based on body size, and attractiveness of humans based on a gamma distribution (shape = 1.43 and scale = 1). VES is calculated with four different methods: VES,Cox, VES,IR, VES,CI, and VES,p. Violin plots on the left panel show the shape of the distribution of VES,Cox estimates. Colored solid lines show the 25, 50, and 75 quantiles for VES,Cox, VES,p, VES,IR, and VES,frailty (gray, blue, red, and green, respectively). Black solid line represents the true PEP = 1—RRinf|exp RRdis|inf = 0.8. Right panel corresponds to the number of cases captured in the trial.

https://doi.org/10.1371/journal.pone.0210041.g007

Discussion

Our results indicate that, under the assumption of a leaky vaccine (i.e., provides partial protection to all vaccinated individuals), VES is consistently underestimated by standard measures that do not account for the repeated, heterogeneous nature of DENV exposure. For instance, heterogeneity in exposure violates an assumption required by the Cox model to ensure unbiased estimates of VES [22]. Heterogeneous exposure has the effect of reducing efficacy because some individuals are highly protected from infection in both arms of the study, which makes the placebo arm of the study more similar to the vaccine arm. Under the default assumptions and scenarios that we examined, we observed that the extent of bias toward the null as a result of heterogeneous exposure could be as great as 21% of the value of simulated PEP, depending on the extent to which the vaccine protects against infection or disease. Depending on the simulated value of PEP and how close it was to MPP, we found that different methods for estimating VES have implications for the possibility of not licensing a vaccine that has desirable properties but is not perceived as such due to bias from heterogeneous exposure.

One possibility for mitigating bias due to heterogeneous exposure that we examined was incorporating a frailty expression, which accounts for unmeasured heterogeneity as a random effect [21], into our Cox survival analysis of simulated trial data. This step did help reduce VES bias, particularly under scenarios in which heterogeneity was more extreme. Although our results indicate that incorporation of a frailty term into estimation of VES from dengue vaccine trial data could help reduce bias toward the null, a caveat is that biased estimates of VES can still be obtained when the frailty distribution is not accurately specified [21]. Also, at high degrees of censoring, non-identifiability of the model parameters can be an issue with frailty models [21].

Another possibility for mitigating bias due to heterogeneous exposure is to incorporate information about the number of exposures in each trial arm. This approach successfully eliminated bias due to heterogeneous exposure in our simulations. A disadvantage, however, is that additional information pertinent to the number of exposures in each trial arm must be collected. In a real trial, doing so would amount to obtaining estimates of the total number of bites by infectious mosquitoes on all individuals in each trial arm. Even as an average across many individuals in a trial arm, estimating entomological risk of DENV infection is not straightforward [59]. Challenges to doing so include significant spatiotemporal heterogeneity in DENV transmission activity [7], shifting hotspots of mosquito density over timescales shorter than a trial [15], and the fact that individuals are subject to mosquito bites and DENV exposure at the numerous locations they visit during their daily routines [60]. Unfortunately, these are precisely the forms of heterogeneity that our results indicate contribute most to bias in VES estimates, rendering VES,p an ideal, but impractical, solution to mitigate bias due to heterogeneous exposure.

We also found an unexpected source of bias for seronegative individuals. This bias comes from our assumption of shorter viremia periods in primary infections for vaccinated compared to unvaccinated individuals. This assumption resulted in fewer detected cases between follow-up visits in the vaccine arm than in the placebo arm. As a consequence, estimates of VES in the seronegative group were biased away from the null. This bias was relatively small when the RT-PCR was used up to five days after symptom onset, as advised by the Centers for Disease Control and Prevention [38]. Not adhering to these guidelines could, however, result in larger bias away from the null, which we observed when we allowed use of RT-PCR up to seven days after symptom onset. Ignoring this type of bias could result in reducing the magnitude of the bias toward the null from heterogeneous exposure. Doing so may be acceptable from the perspective of estimating overall VES, but it has the potential to bias estimates of VES for seronegatives above that of seropositives. For biological reasons [61], estimates for Dengvaxia indicate that VES is higher for seropositives than seronegatives [29]. If analyses of Dengvaxia trial data are subject to this bias from differential detectability, that would indicate that the estimated difference in VES between seropositives and seronegatives could be somewhat greater than currently thought.

There are also sources of bias other than heterogeneous exposure and differential detectability that could manifest in dengue vaccine trials. For instance, infections among trial participants that go undetected could result in bias toward the null. This could occur if analyses of symptomatic endpoints fail to consider that unobserved infections in the placebo arm differentially reduce person-time at risk, increasing the apparent similarity between outcomes in vaccine and placebo arms [62]. This effect may have been moderated in our analyses because conditions that promote it—protection deriving from blocking infection rather than ameliorating disease, high baseline hazard—were not in effect to a large degree. Nonetheless, undetected DENV infections are common [63] and can cause other problems for the interpretation of dengue vaccine trial data [64]. Heterogeneity in susceptibility is a related source of bias that causes bias toward the null for reasons similar to why unobserved infections do [65]. Although pre-enrollment serological testing could be used to classify individuals as seropositive or seronegative for any dengue vaccine trial, it is likely that the serotype(s) to which an individual is susceptible would be undefined, which could be considered a potential source of unmeasured heterogeneity in susceptibility.

Recent decades have seen great elaboration of theory for vaccine trial design and analysis [19], which has increasingly embraced heterogeneity in data-generating processes such as exposure. This growing body of literature has resulted in a number of principles about potential sources of bias in VES estimation, highlighted through simulation experiments that have tended to make use of relatively simple models [66]. Because our primary goal in this study was to quantify the extent of a known bias for a specific disease, we opted for an agent-based model that uses highly detailed data to realistically simulate effects of heterogeneous exposure. In addition to achieving this explicit goal of our study, we uncovered an unexpected source of bias that could have implications for interpretation of Dengvaxia trial data. This discovery emerged from our analyses as a result of including detail in model structure beyond what was minimally necessary to meet our primary goal of quantifying bias due to heterogeneous exposure. This work adds to a growing list of applications of dynamic transmission models to planning for dengue vaccine trials (this study), interpreting their results [35,61], and making projections of their impacts when deployed at the population level [42,46,47,61].

Supporting information

S1 Text. Statistical estimation of sample size.

https://doi.org/10.1371/journal.pone.0210041.s001

(DOCX)

S2 Text. Agent-based model of dengue virus transmission.

https://doi.org/10.1371/journal.pone.0210041.s002

(DOCX)

S1 Fig. Sensitivity as a function of time since symptom onset for primary and post-primary DENV-infected individuals given a limit of detection of 10 cDNA copies/mL.

Dots represent the sensitivity curves obtained from 3,000 simulations of viremia. Solid lines represent the fitted curves described in Eq 1. For primary infections, and . For post-primary infections, and .

https://doi.org/10.1371/journal.pone.0210041.s003

(PDF)

S2 Fig. Overall VES and VES with stratification estimated for 1,000 simulations of the baseline scenario for one site with RT-PCR up to 7 days after symptom onset.

Attractiveness of humans is proportional to their body size. VES is calculated with four different methods: VES,Cox, VES,p, VES,IR, and VES,frailty (gray, blue, red, and green, respectively). Left panel shows results under the assumption that viremia in vaccinated individuals is similar to that in secondary infections. In the right panel, viremia in vaccinees is the same as a natural infection. Violin plots show the shape of the distribution of VES,Cox estimates. Colored solid lines show the 25, 50, and 75 quantiles for VES. Black solid line represents the individual input of PEP = 1—RRinf|exp RRdis|inf = 0.8.

https://doi.org/10.1371/journal.pone.0210041.s004

(EPS)

S3 Fig. Sensitivity of overall VES to changes in the proportion of protection from infection.

Each panel shows VES estimates across different values of the parameter p. Dots show the estimates for each of the four VES measures: VES,Cox, VES,p, VES,IR, and VES,frailty. The black line shows the regression obtained with a generalized additive model and the dark-gray band shows the 95% confidence interval. In all scenarios, PEP is represented by the black dotted line and is defined as PEP = 1 –RRinf|exp RRdis|inf = 0.8. The proportion of protection from infection, p, is defined such that RRinf = (1 –PEP)ρ, and RRdis|inf = (1 –PEP)1 –ρ.

https://doi.org/10.1371/journal.pone.0210041.s005

(EPS)

S4 Fig. Distribution of exposures within the vaccine group of the trial for each of four assumptions of heterogeneity of exposure.

The height of the bars represents the relative frequency of the number of exposures in the vaccine arm for a specific realization of the virtual trial. The four simulations used the same random number seed to increase comparability among simulations.

https://doi.org/10.1371/journal.pone.0210041.s006

(EPS)

S5 Fig. Pairwise difference of VES,Cox between different scenarios about heterogeneous exposure.

Each scenario was simulated 1,000 times with the same sequence of random number seeds to minimize differences caused by stochastic effects.

https://doi.org/10.1371/journal.pone.0210041.s007

(EPS)

S6 Fig. Distribution of exposures within the vaccine arm of the trial under different scenarios about heterogeneous exposure with high transmission intensity.

The height of the bars represents the relative frequency of the number of exposures in the vaccine arm for a specific realization of the virtual trial. The four simulations used the same random number seed to increase comparability among simulations.

https://doi.org/10.1371/journal.pone.0210041.s008

(EPS)

References

  1. 1. Bhatt S, Gething PW, Brady OJ, Messina JP, Farlow AW, Moyes CL, et al. The global distribution and burden of dengue. Nature. Nature Publishing Group; 2013;496: 504. https://doi.org/10.1038/nature12060 pmid:23563266
  2. 2. Gessner BD, Halsey N. Dengue vaccine safety signal: Immune enhancement, waning immunity, or chance occurrence? Vaccine. Elsevier; 2017;35: 3452–3456. pmid:28528764
  3. 3. Vannice KS, Durbin A, Hombach J. Status of vaccine research and development of vaccines for dengue. Vaccine. 2016;34: 2934–2938. pmid:26973072
  4. 4. Precioso AR, Palacios R, Thomé B, Mondini G, Braga P, Kalil J. Clinical evaluation strategies for a live attenuated tetravalent dengue vaccine. Vaccine. 2015;33: 7121–7125. pmid:26458796
  5. 5. Osorio JE, Partidos CD, Wallace D, Stinchcomb DT. Development of a recombinant, chimeric tetravalent dengue vaccine candidate. Vaccine. 2015;33: 7112–7120. pmid:26585500
  6. 6. World Health Organization. WHO vaccine pipeline tracker [Internet]. 2018 [cited 1 Aug 2018]. Available: http://www.who.int/immunization/research/vaccine_pipeline_tracker_spreadsheet/en/
  7. 7. Mammen MP, Pimgate C, Koenraadt CJM, Rothman AL, Aldstadt J, Nisalak A, et al. Spatial and temporal clustering of dengue virus transmission in Thai villages. PLoS Med. 2008;5: e205. pmid:18986209
  8. 8. Salje H, Lessler J, Maljkovic Berry I, Melendrez MC, Endy T, Kalayanarooj S, et al. Dengue diversity across spatial and temporal scales: Local structure and the effect of host population size. Science (80-). American Association for the Advancement of Science; 2017;355: 1302–1306. pmid:28336667
  9. 9. Salje H, Lessler J, Paul KK, Azman AS, Rahman MW, Rahman MW, et al. How social structures, space, and behaviors shape the spread of infectious diseases using chikungunya as a case study. Proc Natl Acad Sci. National Academy of Sciences; 2016;113: 13420–13425. pmid:27821727
  10. 10. Perkins A, Rodriguez-Barraquer I, Manore C, Siraj A, Espana G, Barker C, et al. What lies beneath: a spatial mosaic of Zika virus transmission in the 2015–2016 epidemic in Colombia. bioRxiv. Cold Spring Harbor Laboratory; 2018;
  11. 11. Liebman KA, Stoddard ST, Reiner RC, Perkins TA, Astete H, Sihuincha M, et al. Determinants of Heterogeneous Blood Feeding Patterns by Aedes aegypti in Iquitos, Peru. Barrera R, editor. PLoS Negl Trop Dis. Public Library of Science; 2014;8: 1–10. pmid:24551262
  12. 12. De Benedictis J, Chow-Shaffer E, Costero A, Clark GG, Edman JD, Scott TW. Identification of the people from whom engorged Aedes aegypti took blood meals in Florida, Puerto Rico, using polymerase chain reaction-based DNA profiling. Am J Trop Med Hyg. 2003;68: 437–446. pmid:12875293
  13. 13. Woolhouse ME, Dye C, Etard JF, Smith T, Charlwood JD, Garnett GP, et al. Heterogeneities in the transmission of infectious agents: implications for the design of control programs. Proc Natl Acad Sci U S A. 1997;94: 338–42. Available: http://www.ncbi.nlm.nih.gov/pubmed/8990210 pmid:8990210
  14. 14. Perkins TA, Scott TW, Le Menach A, Smith DL. Heterogeneity, Mixing, and the Spatial Scales of Mosquito-Borne Pathogen Transmission. Salathé M, editor. PLoS Comput Biol. Public Library of Science; 2013;9: 1–16. pmid:24348223
  15. 15. LaCon G, Morrison AC, Astete H, Stoddard ST, Paz-Soldan VA, Elder JP, et al. Shifting Patterns of Aedes aegypti Fine Scale Spatial Clustering in Iquitos, Peru. PLoS Negl Trop Dis. Public Library of Science; 2014;8: 1–13. pmid:25102062
  16. 16. Perkins TA, Garcia AJ, Paz-Soldán VA, Stoddard ST, Reiner RC, Vazquez-Prokopec G, et al. Theory and data for simulating fine-scale human movement in an urban environment. J R Soc Interface. The Royal Society; 2014;11. pmid:25142528
  17. 17. Perkins TA, Paz-Soldan VA, Stoddard ST, Morrison AC, Forshey BM, Long KC, et al. Calling in sick: impacts of fever on intra-urban human mobility. Proc R Soc London B Biol Sci. The Royal Society; 2016;283. pmid:27412286
  18. 18. Harrington LC, Scott TW, Lerdthusnee K, Coleman RC, Costero A, Clark GG, et al. Dispersal of the dengue vector Aedes aegypti within and between rural communities. Am J Trop Med Hyg. 2005;72: 209–220. doi:72/2/209 [pii] pmid:15741559
  19. 19. Halloran ME, Longini IM, Struchiner CJ. Design and Analysis of Vaccine Studies. Springer; 2010. https://doi.org/10.1007/978-0-387-68636-3
  20. 20. White MT, Griffin JT, Drakeley CJ, Ghani AC. Heterogeneity in malaria exposure and vaccine response: implications for the interpretation of vaccine efficacy trials. Malar J. 2010;9: 82. pmid:20331863
  21. 21. Halloran ME, Longini IM, Struchiner CJ. Estimability and interpretation of vaccine efficacy using frailty mixing models. Am J Epidemiol. 1996;144.
  22. 22. O’Hagan JJ, Lipsitch M, Hernán MA. Estimating the per-exposure effect of infectious disease interventions. Epidemiology. NIH Public Access; 2014;25: 134. pmid:24240656
  23. 23. Edelman R, Hombach J. “Guidelines for the clinical evaluation of dengue vaccines in endemic areas”: Summary of a World Health Organization Technical Consultation. Vaccine. 2008. pp. 4113–4119. pmid:18597906
  24. 24. Morrison AC, Minnick SL, Rocha C, Forshey BM, Stoddard ST, Getis A, et al. Epidemiology of dengue virus in Iquitos, Peru 1999 to 2005: interepidemic and epidemic patterns of transmission. PLoS Negl Trop Dis. Public Library of Science; 2010;4: 1–17.
  25. 25. Morrison a C, Sihuincha M, Stancil JD, Zamora E, Astete H, Olson JG, et al. Aedes aegypti (Diptera: Culicidae) production from non-residential sites in the Amazonian city of Iquitos, Peru. Ann Trop Med Parasitol. 2006;100 Suppl: S73–S86. pmid:16630393
  26. 26. Getis A, Morrison AC, Gray K, Scott TW. Characteristics of the spatial pattern of the dengue vector, Aedes aegypti, in Iquitos, Peru. Am J Trop Med Hyg. 2003;69: 494–505. pmid:14695086
  27. 27. Reiner RC, Stoddard ST, Forshey BM, King A a, Ellis AM, Lloyd AL, et al. Time-varying, serotype-specific force of infection of dengue virus. Proc Natl Acad Sci U S A. 2014;111: E2694–702. pmid:24847073
  28. 28. Capeding MR, Tran NH, Hadinegoro SRS, Ismail HIHJM, Chotpitayasunondh T, Chua MN, et al. Clinical efficacy and safety of a novel tetravalent dengue vaccine in healthy children in Asia: a phase 3, randomised, observer-masked, placebo-controlled trial. Lancet. 2014;384: 1358–1365. pmid:25018116
  29. 29. Hadinegoro SR, Arredondo-García JL, Capeding MR, Deseda C, Chotpitayasunondh T, Dietze R, et al. Efficacy and Long-Term Safety of a Dengue Vaccine in Regions of Endemic Disease. N Engl J Med. 2015; 1195–1206. pmid:26214039
  30. 30. Villar L, Dayan GH, Arredondo-García JL, Rivera DM, Cunha R, Deseda C, et al. Efficacy of a Tetravalent Dengue Vaccine in Children in Latin America. N Engl J Med. 2015;372: 113–123. pmid:25365753
  31. 31. ClinicalTrials.gov. Phase III Trial to Evaluate Efficacy and Safety of a Tetravalent Dengue Vaccine [Internet]. [cited 25 Mar 2018]. Available: https://clinicaltrials.gov/ct2/show/NCT02406729?cond=dengue+vaccine&phase=2&draw=2&rank=11
  32. 32. ClinicalTrials.gov. Efficacy, Safety and Immunogenicity of Takeda’s Tetravalent Dengue Vaccine (TDV) in Healthy Children [Internet]. [cited 25 Mar 2018]. Available: https://clinicaltrials.gov/ct2/show/record/NCT02747927?term=takeda+dengue&rank=3
  33. 33. Morrow RH, Smith PG, Special Programme for Research and Training in Tropical Diseases. Field trials of health interventions in developing countries: a toolbox. Macmillan; 1996.
  34. 34. Sabchareon A, Wallace D, Sirivichayakul C, Limkittikul K, Chanthavanich P, Suvannadabba S, et al. Protective efficacy of the recombinant, live-attenuated, CYD tetravalent dengue vaccine in Thai schoolchildren: A randomised, controlled phase 2b trial. Lancet. 2012;380: 1559–1567. pmid:22975340
  35. 35. Coudeville L, Baurin N, L’Azou M, Guy B. Potential impact of dengue vaccination: Insights from two large-scale phase III trials with a tetravalent dengue vaccine. Vaccine. 2016;34: 6426–6435. pmid:27601343
  36. 36. Hadinegoro S, Arredondo-García J, Capding M. Efficacy and safety of a novel tetravalent dengue vaccine in healthy children aged 2 to 14 years in Asia. Protocol. 2010.
  37. 37. Stanaway JD, Shepard DS, Undurraga EA, Halasa YA, Coffeng LE, Brady OJ, et al. The global burden of dengue: an analysis from the Global Burden of Disease Study 2013. Lancet Infect Dis. Elsevier; 2016;16: 712–723. pmid:26874619
  38. 38. Dengue Branch C. Criteria for the Processing of Dengue Samples at the CDC Dengue Branch, San Juan, Puerto Rico [Internet]. 2011.
  39. 39. Clapham HE, Tricou V, Van Vinh Chau N, Simmons CP, Ferguson NM, Bhatt S, et al. Within-host viral dynamics of dengue serotype 1 infection. J R Soc Interface. The Royal Society; 2014;11: 504–507. pmid:24829280
  40. 40. R Core Team. R: A Language and Environment for Statistical Computing [Internet]. Vienna, Austria, Austria; 2015. Available: https://www.r-project.org/
  41. 41. Therneau TM. A Package for Survival Analysis in S [Internet]. 2015. Available: http://cran.r-project.org/package=survival
  42. 42. Perkins A, Reiner R, España G, ten Bosch QA, Verma A, Liebman K, et al. An agent-based model of dengue virus transmission shows how multiple uncertainties about vaccine efficacy influence public health impact projections. bioRxiv. Cold Spring Harbor Laboratory; 2018;
  43. 43. Morrison AC, Gray K, Getis A, Astete H, Sihuincha M, Focks D, et al. Temporal and Geographic Patterns of Aedes aegypti (Diptera: Culicidae) Production in Iquitos, Peru. J Med Entomol. Entomological Society of America; 2004;41: 1123–1142. pmid:15605653
  44. 44. Paz-Soldan VA, Stoddard ST, Vazquez-Prokopec G, Morrison AC, Elder JP, Kitron U, et al. Assessing and Maximizing the Acceptability of Global Positioning System Device Use for Studying the Role of Human Movement in Dengue Virus Transmission in Iquitos, Peru. Am J Trop Med Hyg. 2010;82: 723–730. pmid:20348526
  45. 45. Paz-Soldan VA, Reiner RC, Morrison AC, Stoddard ST, Kitron U, Scott TW, et al. Strengths and Weaknesses of Global Positioning System (GPS) Data-Loggers and Semi-structured Interviews for Capturing Fine-scale Human Mobility: Findings from Iquitos, Peru. Jones MK, editor. PLoS Negl Trop Dis. Public Library of Science; 2014;8: 1–11. pmid:24922530
  46. 46. Flasche S, Jit M, Rodríguez-Barraquer I, Coudeville L, Recker M, Koelle K, et al. The Long-Term Safety, Public Health Impact, and Cost-Effectiveness of Routine Vaccination with a Recombinant, Live-Attenuated Dengue Vaccine (Dengvaxia): A Model Comparison Study. von Seidlein L, editor. PLOS Med. Public Library of Science; 2016;13: 1–19. pmid:27898668
  47. 47. España G, Yao Y, Anderson KB, Fitzpatrick MC, Smith DL, Morrison AC, et al. Model-based assessment of public health impact and cost-effectiveness of dengue vaccination following screening for prior exposure. bioRxiv. Cold Spring Harbor Laboratory; 2018;
  48. 48. Magori K, Legros M, Puente ME, Focks DA, Scott TW, Lloyd AL, et al. Skeeter Buster: A Stochastic, Spatially Explicit Modeling Tool for Studying Aedes aegypti Population Replacement and Population Suppression Strategies. PLoS Negl Trop Dis. 2009;3: 1–18. pmid:19721700
  49. 49. Yang HM, Macoris MLG, Galvani KC, Andrighetti MTM, Wanderley DM V. Assessing the effects of temperature on the population of Aedes aegypti, the vector of dengue. Epidemiol Infect. Cambridge Univ Press; 2009;137: 1188–1202. pmid:19192322
  50. 50. Reiner R, Stoddard S, Vazquez-Prokopec G, Astete H, Perkins TA, Sihuincha M, et al. Estimating the impact of city-wide Aedes aegypti population control: An observational study in Iquitos, Peru. bioRxiv. Cold Spring Harbor Laboratory; 2018;
  51. 51. Focks D a, Haile DG, Daniels E, Mount G a. Dynamic life table model for Aedes aegypti (diptera: Culicidae): simulation results and validation. J Med Entomol. 1993;30: 1003–1017. pmid:8271242
  52. 52. Otero M, Solari HG, Schweigmann N. A Stochastic Population Dynamics Model for Aedes Aegypti: Formulation and Application to a City with Temperate Climate. Bull Math Biol. 2006;68: 1945–1974. pmid:16832731
  53. 53. Scott TW, Amerasinghe PH, Morrison a C, Lorenz LH, Clark GG, Strickman D, et al. Longitudinal studies of Aedes aegypti (Diptera: Culicidae) in Thailand and Puerto Rico: blood feeding frequency. J Med Entomol. 2000;37: 89–101. pmid:15218911
  54. 54. Ellis AM, Garcia AJ, Focks DA, Morrison AC, Scott TW. Parameterization and Sensitivity Analysis of a Complex Simulation Model for Mosquito Population Dynamics, Dengue Transmission, and Their Control. Am J Trop Med Hyg. 2011;85: 257–264. pmid:21813844
  55. 55. Chan M, Johansson MA. The Incubation Periods of Dengue Viruses. PLoS One. 2012;7: 1–7. pmid:23226436
  56. 56. ten Bosch QA, Clapham HE, Lambrechts L, Duong V, Buchy P, Althouse BM, et al. Contributions from the silent majority dominate dengue virus transmission. PLOS Pathog. Public Library of Science; 2018;14: 1–20. pmid:29723307
  57. 57. Reich NG, Shrestha S, King AA, Rohani P, Lessler J, Kalayanarooj S, et al. Interactions between serotypes of dengue highlight epidemiological impact of cross-immunity. J R Soc Interface. The Royal Society; 2013;10. pmid:23825116
  58. 58. United Nations, Department of Economic and Social Affairs PD. World population prospects: The 2015 revision—SPecial Aggregates. DVD Editio. 2015.
  59. 59. Takken W, Scott TW. Ecological aspects for application of genetically modified mosquitoes. Springer Science & Business Media; 2003.
  60. 60. Reiner RC Jr., Achee N, Barrera R, Burkot TR, Chadee DD, Devine GJ, et al. Quantifying the Epidemiological Impact of Vector Control on Dengue. PLoS Negl Trop Dis. Public Library of Science; 2016;10: 1–11. pmid:27227829
  61. 61. Ferguson NM, Rodríguez-Barraquer I, Dorigatti I, Mier-y-Teran-Romero L, Laydon DJ, Cummings DAT. Benefits and risks of the Sanofi-Pasteur dengue vaccine: Modeling optimal deployment. Science (80-). 2016;353: 1033–1036.
  62. 62. Kahn R, Hitchings M, Wang R, Bellan S, Lipsitch M. Analyzing Vaccine Trials in Epidemics with Mild and Asymptomatic Infection. Am J Epidemiol. 2018; kwy239. pmid:30329134
  63. 63. Clapham HE, Cummings DAT, Johansson MA. Immune status alters the probability of apparent illness due to dengue virus infection: Evidence from a pooled analysis across multiple cohort and cluster studies. PLoS Negl Trop Dis. Public Library of Science; 2017;11: 1–12. pmid:28953902
  64. 64. Rodriguez-Barraquer I, Mier-y-Teran-Romero L, Burke DS, Cummings DAT. Challenges in the interpretation of dengue vaccine trial results. PLoS Negl Trop Dis. Public Library of Science; 2013;7: 1–3. pmid:24009782
  65. 65. Longini IM, Halloran ME. A frailty mixture model for estimating vaccine efficacy. J R Stat Soc Ser C (Applied Stat. 1996;45: 165–173.
  66. 66. Halloran ME, Auranen K, Baird S, Basta NE, Bellan SE, Brookmeyer R, et al. Simulations for designing and interpreting intervention trials in infectious diseases. BMC Med. 2017;15: 223. pmid:29287587