bebi103.hmc.inferential_calibration

bebi103.hmc.inferential_calibration(prior_predictive_stan_file=None, posterior_stan_file=None, prior_predictive_model_data=None, posterior_model_data=None, measured_data=None, var_names=None, ignored_var_names=(), measured_data_dtypes=None, sampling_kwargs=None, diagnostic_check_kwargs=None, cores=1, N=400, n_prior_draws_for_sd=1000, samples_dir='inferential_calibration_samples', remove_sample_files=True, df_package='polars', progress_bar=False, output_csv='inferential_calibration_results.csv', overwrite=False)

Perform simulation-based calibration on a Stan Model.

Parameters:

  • prior_predictive_stan_file (str) – Stan file for generating prior predictive data sets.

  • posterior_model (str) – Stan file for the posterior that allows sampling.

  • prior_predictive_model_data (dict) – Dictionary with entries specified by the data block of the prior predictive Stan model.

  • posterior_model_data (dict) – Dictionary with entries specified by the data block of the prior predictive Stan model. Measured data in this dictionary will be replaced in each simulation by what was generated by the prior predictive model.

  • measured_data (list) – A list of strings containing the variable names of measured data. Each entry in measured_data must be a key in posterior_model_data.

  • var_names (list of strings) – A list of strings containing parameter names to be considered in the inferential calibration analysis. Not all parameters of the model need be considered; only those in var_names have rank statistics calculated. Note that for multidimensional variables, var_names only has the root name. E.g., var_names=[‘x’, ‘y’], not something like var_names=[‘x[0]’, ‘x[1]’, ‘y[0,0]’].

  • ignored_var_names (tuple or list of strings, default ()) – Variable names to ignore in the inferential calibration procedure. These variables have no rank statistics computed and are omitted from the diagnostic checks. This is typically used for posterior retrodictive variables and log likelihood variables. As with var_names, only the root name is given for multidimensional variables, e.g. ignored_var_names=[‘x_prc’], not ignored_var_names=[‘x_prc[0]’, ‘x_prc[1]’].

  • measured_data_dtypes (dict, default None) – The key in the dtypes dict is a string representing the data name, and the corresponding item is its dtype, almost always either int or float.

  • sampling_kwargs (dict, default None) – kwargs to be passed to sm.sample().

  • diagnostic_check_kwargs (dict, default None) – kwargs to pass to check_all_diagnostics(). If verbosity and/or return_diagnostics are given, they are ignored. max_treedepth is inferred from sampling_kwargs.

  • cores (int, default 1) – Number of cores to use in the inferential calibration calculation.

  • N (int, 400) – Number of simulations to run.

  • n_prior_draws_for_sd (int, default 1000) – Number of prior draws to compute the prior standard deviation for a parameter in the prior distribution. This standard deviation is used in the shrinkage calculation.

  • samples_dir (str, default "inferential_calibration_samples") – Path to directory to store .csv and .txt files generated by CmdStan. The directory specified here will NOT be destroyed after a calculation, though the files within it may be, depending on the remove_sample_files kwarg.

  • remove_sample_files (bool, default True) – If True, remove .csv and .txt files generated by CmdStan.

  • df_package (str, either 'polars' (default) or 'pandas') – Which package to use for output data frame

  • progress_bar (bool, default False) – If True, display a progress bar for the calculation using tqdm.

  • output_csv (str, default 'inferential_calibration_results.csv') – Path to CSV file where results will be written. Results are written directly to this file as workers complete (instead of collecting in memory). This reduces memory usage and allows monitoring progress by examining the growing CSV file. The file includes metadata comments at the top with run parameters and start time.

  • overwrite (bool, default False) – If False, raises RuntimeError if output_csv already exists. If True, overwrites existing file with new results.

Returns:

output – Data frame with the output of the inferential calibration analysis. It has the following columns.

  • trial : Unique trial number for the simulation.

  • warning_code : Warning code based on diagnostic checks outputted by check_all_diagnostics().

  • parameter: The name of the scalar parameter.

  • prior: Value of the parameter used in the simulation. This value was drawn out of the prior distribution.

  • mean : mean parameter value based on sampling out of the posterior in the simulation.

  • sd : standard deviation of the parameter value based on sampling out of the posterior in the simulation.

  • L : The number of bins used in computing the rank statistic. The rank statistic should be uniform on the integers [0, L].

  • rank_statistic : Value of the rank statistic for the parameter for the trial.

  • shrinkage : The shrinkage for the parameter for the given trial. This is computed as 1 - sd / sd_prior, where sd_prior is the standard deviation of the parameters as determined from drawing out of the prior.

  • z_score : The z-score for the parameter for the given trial. This is computed as abs(mean - prior) / sd.

  • tail_xi_hat_left : Left tail xi_hat for the parameter, computed over all chains pooled together.

  • tail_xi_hat_right : Right tail xi_hat for the parameter, computed over all chains pooled together.

  • inc_tau_hat : Incremental empirical integrated autocorrelation time for the parameter, computed over all chains pooled together.

  • variance : Empirical variance of the parameter’s posterior samples, computed over all chains pooled together.

  • average_accept_proxy : Average proxy acceptance statistic for the trial, taken over all chains.

  • prior_predictive_seed : Random number seed used by Stan to generate the prior predictive data set for the trial.

  • posterior_seed : Random number seed used by Stan for the posterior MCMC sampling for the trial.

  • prior_predictive_prefix : Full-path prefix of the sampling files (.csv and .txt) generated by Stan when drawing the prior predictive data set for the trial. If remove_sample_files is True (so the files are deleted), every entry is null (Polars output) or np.nan (Pandas output).

  • samples_prefix : Full-path prefix of the sampling files (.csv and .txt) generated by Stan for the posterior MCMC sampling for the trial. If remove_sample_files is True (so the files are deleted), every entry is null (Polars output) or np.nan (Pandas output).

Return type:

Polars or Pandas DataFrame

Notes

prior distribution, using those parameters to generate data from the likelihood, and then performing posterior sampling based on the generated data. A rank statistic for each simulation is computed. This rank statistic should be uniformly distributed over its L possible values. See https://arxiv.org/abs/1804.06788, by Talts, et al., for details.