A series of VOI analyses were conducted using the analytic framework previously developed by EPA ORD to compare alternative toxicity testing methods (Hagiwara et al. 2022). The present report applies the framework in a case study comparing ETAP and THHA, with a goal of evaluating the contexts under which the less costly and timelier ETAP - despite being subject to greater uncertainty - may offer a viable alternative to THHA. The VOI analysis examines the trade-offs between timeliness of decision-making, cost of toxicity testing, and the amount of uncertainty reduced by these tests in order to determine which of the two tests provides the most 'value', and which test affords the greatest public health protection from the potential toxic effects of exposure to chemicals present in the human environment.

To conduct VOI analyses, the decision-making criteria are specified to determine whether or not regulatory action is required on the basis of results of toxicity testing and human health assessment. It is assumed that the chemical is or was in commerce with continued human exposure. Two types of decision makers are considered, specifically the BRDM and the TRDM. The BRDM seeks to minimize the TSC associated with chemical exposure. When the economic value of health benefits realized through the exposure mitigation action outweighs the cost of such action, the BRDM would decide to regulate the chemical of interest. In contrast, the TRDM focuses on the average population risk associated with the chemical exposure, without consideration of the cost of risk mitigation, and chooses to reduce exposure whenever that risk is greater than the target risk level. Although neither of these decision-making contexts fully emulates real-world decision-making practices, they do correspond to two of the ten decision-making principles discussed by Krewski et al. (2022).

Parameterization of the decision-making scenarios in the VOI analysis is informed by realistic data on toxicity and exposure. Specifically, the 1,522 chemicals and endpoints considered by Chiu et al. (2018) formed the foundation for gauging uncertainty in toxic potency prior to testing. The conversion of the animal-based toxicity testing results to an HED requires the application of a series of adjustment factors reflecting different sources of uncertainty and variability. Within-study variability for the THHA is estimated using 584 previously evaluated bioassays (Sand et al. 2011). The animal-to-human scaling factors²⁴ are based on recommendations from the IPCS (WHO 2017), which conducted an extensive evaluation of empirical data on factors affecting uncertainty in converting the results of animal toxicity tests to humans. Within-study variability for ETAP is taken from experimental data on 14 chemicals (EPA 2024c). Finally, exposure estimates from SHEDS-HT are used to construct the nine baseline exposure scenarios corresponding to a 3×3 grid defined by low, medium, and high average exposures and by low, medium, and high variation in population exposure.

As with the choice of parameters for toxicity and exposure, the health costs and control costs are also based on realistic data. The valuation of adverse health effects, ranging from short-term transient toxic responses to longer-term irreversible toxic effects to mortality, is based on examination of values applied by health economists to such effects. As discussed in Section 5.3.1, annualized per case health costs of $1K and $10K are used for non-fatal outcomes of lesser and greater severity, along with $110K for fatal outcomes. For the control costs, a maximum annualized cost of exposure mitigation is set to $23.1B in the baseline scenarios, based on estimated actual cost of major air pollution emission programs (EPA 2011b), extrapolated to the case of complete elimination of emissions. For the sensitivity analysis scenarios, this maximum annualized cost is modified to $578M, based on an examination of the cost of 33 chemical risk management programs implemented or proposed under the European Union's REACH program (ECHA 2021). The wide-range of annualized control costs examined in the baseline ($23.1B) and sensitivity analysis scenarios ($578M) was intended to capture the diversity in potential chemicals to which ETAP may be applied. In comparison, the recent total annualized control cost estimates from the final national primary drinking water rule for six PFAS was $1.55B (EPA 2024a) which falls within the range in control costs used in the case study and supports the use of the range for more generalizable conclusions. However, the relationship between the cost of exposure mitigation and its effectiveness was assumed to be continuous in this case study. In practice, there may be a finite number of control measures available, which translates to a stepwise relationship. Nonetheless, this option was not evaluated due to the complexity of pairing the cost and effectiveness of various control measures that is generalizable as well as owing to a lack of data to inform a stepwise function.

This report presents VOI analyses for a total of 360 scenarios under benefit-risk and target-risk decision-making contexts including 18 baseline scenarios (9 for BRDM and 9 for TRDM), and 342 sensitivity analysis scenarios (171 scenarios each for BRDM and TRDM). These sensitivity analysis scenarios investigate the impacts across a broad range of varying parameters including: the degree of uncertainty in exposure information, economic valuation of both adverse health effects and the cost of mitigation action, the time frame associated with the toxicity testing and assessment process, the study cost of the toxicity test, and the time horizon to realize potential health benefits, the size of the exposed population, the choice of the target risk level, and consideration of discordance with traditional toxicity testing results based on the chronic bioassay as an additional source of uncertainty for ETAP.

For the BRDM, the results from the baseline scenarios suggest that, when the ORE (based on the prior information) is over 78%, the cost associated with the 8-year delay in decision-making for the THHA leads to negative EVDSI values (along with negative ENBS and ROI values), indicating that timeliness is an important determinant of the value of information. In contrast, ETAP produces a positive ROI in all but one baseline scenario, with the benefit realized by collecting additional toxicity information via ETAP outweighing both the delay and the COT. In Baseline Scenarios 1 to 8, where either ETAP or THHA produced positive EVDSI values, ETAP produced greater EVDSI, ENBS, and ROI values as compared to THHA. Similar results are observed for the TRDM, where ROI_ETAP is at least 12-fold greater than ROI_THHA across the nine baseline scenarios included in the VOI analysis.

For the BRDM, Table 7-1 summarizes the number of scenarios where ETAP is preferred over THHA based on three key VOI metrics - EVDSI, ENBS, and ROI - across all 180 baseline and additional scenarios considered in the sensitivity analysis. Of these, 31 (17%) scenarios result in negative EVDSI values for both ETAP and THHA (resulting in negative ENBS and ROI values as well). These scenarios represent situations in which the ORE is 100%, either because μ_exp or AHC-to-ACC ratio is high. ETAP produces positive EVDSI, ENBS, and ROI values in the remaining 149 (83%) scenarios, whereas THHA produces positive ENBS values in 92 (51%) scenarios. In these 149 scenarios, ETAP always produced greater ENBS values than THHA, with the differences ENBS_Diff=ENBS_ETAP — ENBS_THHA ranging from as low as $4M to as high as $1T, with a median difference of $44B. Figure 7-1 shows boxplots of ENBS_Diff grouped by the baseline scenarios and the scenarios considered under the various sensitivity analyses for the BRDM. From Figure 7-1, it is apparent that the choice of the time horizon and the direct cost of testing do not result in notable changes in ENBS. On the other hand, the quality of the exposure information (in particular, the mean exposure level μ_exp), associated health and control costs, duration of the toxicity testing and assessment processes, size of the affected population, and the amount of uncertainty reduction all have an impact on the magnitude of the VOI. Nonetheless, the median differences favor ETAP over THHA within each of these six sensitivity analyses. Since ETAP would lead to earlier regulatory decisions than THHA in all scenarios, the results presented in this report underscore the importance of timeliness in toxicity testing and the associated human health risk assessment.

Table 7-1

Summary of baseline and sensitivity analysis scenarios in which ETAP is preferred for BRDM.

Figure 7-1

Boxplots of ENBS_Diff=ENBS_ETAP - ENBS_THHA for BRDM. Of 180 scenarios considered, 31 scenarios in which both ETAP and THHA produced negative ENBS values are excluded.

For the TRDM, Table 7-2 summarizes the number of scenarios where ETAP is preferred over THHA for the same set of VOI metrics and across all 180 baseline and sensitivity analysis scenarios. Of the 180 scenarios considered, 13 (7%) require no further toxicity testing. Of the 13 scenarios that required no additional toxicity testing, 12 of these can be explained by the smaller prior uncertainty in μ_exp and σ_exp compared to the baseline, with the remaining sensitivity scenario occurring because the TRL is increased to 10⁻⁴ compared to the baseline value of 10⁻⁶. Among the remaining 166 scenarios, ETAP produces greater ENBS values than THHA in 148 (89%) scenarios. In 14 scenarios where ENBS_THHA > ENBS_ETAP, ETAP has difficulty concluding that exposure mitigation action is required because the average population risk is low. Since ETAP is subject to greater uncertainty than traditional bioassay, obtaining accurate estimates of risk is more difficult with ETAP. This is particularly true when additional potential discordance between ETAP and THHA is taken into consideration. Despite these differences, when VOI is measured using ROI, THHA is superior only in Baseline Scenario 1 when discordance is included as additional source of uncertainty for ETAP. Overall, ENBS_Diff ranges from -$3B to as high as $12T (this maximum value is the extreme outlying value in the boxplots shown in Figure 7-2), with a median value of $81B. As with the BRDM, extending the time horizon beyond 20 years or changing the study cost did not markedly alter the difference in VOI between ETAP and THHA for the TRDM; changes in values of the remaining parameters did lead to some differences in the numerical values of the VOI metrics for ETAP and THHA, although ETAP generally remained preferrable to THHA for both the BRDM and the TRDM across the spectrum of sensitivity analyses conducted in this report.

Table 7-2

Summary of baseline and sensitivity analysis scenarios in which ETAP is preferred for TRDM.

Figure 7-2

Boxplots of ENBS_Diff=ENBS_ETAP - ENBS_THHA for TRDM. Of 180 scenarios considered, 13 scenarios where no additional toxicity testing was required are excluded.

Although additional exposure data may reduce uncertainty about exposure to the chemical of interest in the general population, assessment of the benefits of improved exposure data was outside of the scope of the case study. In order to determine whether additional toxicity testing or additional efforts to reduce the uncertainty about population exposure using more refined exposure ascertainment methods would be more valuable, the cost of specific exposure assessment approaches and the concomitant reduction in uncertainty would need to be incorporated into the VOI analytic framework. Since the overarching goal of this present case study was to evaluate the relative VOI provided by ETAP and THHA, the benefits of improved exposure assessment were not considered. However, assuming that the prior uncertainty distributions for toxicity and exposure are similar, a proportionate reduction in either source of uncertainty would be expected to result in a similar benefit in terms of VOI.

In the scenarios considered in this report, the ETAP was generally preferred over the THHA in terms of cost, timeliness, and public health benefit. This conclusion is remarkably robust in that VOI metrics favor ETAP over the THHA across a wide range of exposure scenarios reflecting a broad range of conditions. The robustness of this finding is further supported by the extensive series of sensitivity analyses examining the impact of changes in quality of the exposure information, associated health and control costs, timeliness of the toxicity testing and assessment processes, time horizon to realize potential health benefits, study cost of toxicity testing, size of the affected population, and amount of uncertainty reduction. However, there are several potential caveats to these conclusions. First, the experimental data used to inform some of the parameters for ETAP are limited (e.g., intra-study variability). The continued generation and application of ETAP data will serve to refine these parameter estimates and provide a more accurate VOI comparison. Second, the ETAP provides a reference value that is not linked to a specific hazard, whereas the traditional toxicity testing process identifies specific hazards that may be relevant to human health. The potential overall economic value of identifying specific hazards has not been incorporated into the analysis. Third, within the context of present VOI framework, 'value' is only realized when the collection of additional toxicity testing information results in a decision based on this additional information that is different from the decision that would be made in the absence of this information. An exception to this assertion that 'value' is only realized when decisions are altered would occur in an extended VOI framework which places value on knowing a chemical does not pose an appreciable population health risk. In this case, knowing that a chemical is safe with less uncertainty would have some value. Although economists have suggested approaches to valuing additional assurances of safety (Hanemann 1989), the current analysis has not attempted to incorporate such 'option values', in the absence of consensus on the methodological approach to assigning option values or consensus that such valuations should be included in VOI analysis.

VOI analysis provides a set of methods for organizing and evaluating efforts to collect and apply new information to regulatory decisions. However, application of VOI methods in chemical safety decisions in regulatory agencies, such as the EPA, has been underutilized (NASEM 2009; Yokota and Thompson 2004). To both support application of VOI to decision making and evaluate a draft new human health assessment product, a case study was conducted to evaluate the human health and economic trade-offs associated with the timeliness, uncertainty, and costs of the new toxicity testing and human health assessment process. Collectively, the VOI analyses conducted in the case study demonstrated that under the exposure scenarios and assumptions considered, the ETAP is the more frequently preferred approach for more rapidly and cost effectively evaluating chemicals with no existing toxicity testing or human health data. If applied to a significant number of chemicals lacking toxicity testing and human health assessments, the public health benefits of ETAP would become multiplicative. In addition, ETAP could be combined with other data gap filling methods by strategically selecting chemicals with structural and other characteristics that could serve as analogs for a broader collection of substances or anchor a chemical category. These considerations could be incorporated into the candidate identification step in the ETAP process, which could further enhance the public health benefits. Application of the VOI framework to other case studies and decision contexts should build further confidence in its broad utility and importance for policy and decision makers.

Footnotes

24

Specifically, the uncertainty associated with allometric scaling and differences in toxicokinetics and toxicodynamics between animals and humans was examined.