- Confidence interval and sample size
- ANOVA and Kruskal-Wallis on three diets
- Clinical trial 2x2 table
- Bayesian A/B test with Beta posteriors
- Before / after treatment: paired t-test or Wilcoxon
- z-test with known sigma, power and sample size
- Summary of a 5-point Likert survey item
- Which mean? Arithmetic, geometric and harmonic in practice
- The Student t density from the Gamma and Beta functions
- Mincer wage equation from a survey table in SQL
- Diagnostic test: AUC and Youden threshold
- How many populations? Gaussian mixture and BIC
- Correspondence analysis of a survey table
- Prediction interval of a calibration
- Is the best model really better?
- How many subjects for the study?
- Decide an A/B test
- Bayesian logistic regression by MCMC
- Repeated measures: each subject has its own slope
- Missing values: multiple imputation instead of deleting the rows
- Repeated measures: time, group and their interaction
- Small sample: Bayesian logistic regression and the probability of an effect
- Size of a survival trial: events, patients and power
- A/B test of a landing page: probability to win and expected loss
- Multi-center trial: partial pooling of the effects of 8 centers
- Effect of a speed limit on road accidents: interrupted time series
- Opinion poll: raking on sex and age, weighted estimate and margin of error
Confidence interval and sample size
A pilot study measures the drop of systolic pressure on 10 patients. The script gives the 95 % confidence interval of the mean and the number of patients needed for a margin of 1.5 mmHg.
#1. Pilot study: drop of systolic pressure in mmHg after 8 weeks, 10 patients; -> "[d]" "[8, 12, 5, 15, 10, 7, 13, 9, 11, 6]"; json load "rx_st_ci" (stat conf_interval_mean [d] 0.95); #2. Sample size for a margin of 1.5 mmHg at 95 percent: n = z s / E squared; -> "[z]" (stat dist_inv "normal" "[0, 1]" 0.975); -> "[n]" (math ceil (math calc (concat "([z]*" (stat stddev [d]) "/1.5)^2"))); concat "mean = " (json select "rx_st_ci" /mean) " mmHg, 95% CI [" (math round (json select "rx_st_ci" /lower) 2) ", " (math round (json select "rx_st_ci" /upper) 2) "], margin " (math round (json select "rx_st_ci" /margin) 2) " -> " [n] " patients needed for a margin of 1.5 mmHg"
mean = 9.6 mmHg, 95% CI [7.31, 11.89], margin 2.29 -> 18 patients needed for a margin of 1.5 mmHg
- Use the Student quantile with
stat dist_inv"student" and iterate on n - Add the power of a test: n = ((z_alpha + z_beta) s / delta)^2
- Check the hypothesis of the paired difference with
stat ttest_1
ANOVA and Kruskal-Wallis on three diets
Weight loss is measured on three diets of six subjects. The script runs a one-way ANOVA with its effect size and the non-parametric Kruskal-Wallis test as a check.
#1. Weight loss in kg with three diets; -> "[g]" "[[3.1, 4.2, 2.8, 3.9, 3.5, 4.0], [5.2, 4.8, 6.1, 5.5, 4.9, 5.8], [3.8, 4.5, 4.1, 3.6, 4.9, 4.4]]"; #2. Parametric one-way ANOVA; json load "rx_st_a" (stat anova [g]); #3. Non-parametric check: Kruskal-Wallis on the ranks; json load "rx_st_k" (stat kruskal [g]); concat "ANOVA F(" (json select "rx_st_a" /df_between) ", " (json select "rx_st_a" /df_within) ") = " (math round (json select "rx_st_a" /f) 2) ", p = " (math round_sig (json select "rx_st_a" /p_value) 3) ", eta2 = " (math round (json select "rx_st_a" /eta2) 3) " ; Kruskal-Wallis H = " (math round (json select "rx_st_k" /h) 2) ", p = " (math round_sig (json select "rx_st_k" /p_value) 3)
ANOVA F(2, 15) = 18.92, p = 0.0000791, eta2 = 0.716 ; Kruskal-Wallis H = 12.07, p = 0.00240
- Check the residuals with
stat shapiro - Compare the pairs with
stat ttest_2and a Bonferroni correction - Use
stat mann_whitneyfor two groups only
Clinical trial 2x2 table
A trial counts the events in a treatment group and a placebo group of 100 patients each. The script runs the chi-square and Fisher exact tests and computes the relative risk, the absolute risk reduction and the number needed to treat.
#1. Trial: events / no event, treatment 12 / 88, placebo 27 / 73; -> "[t]" "[[12, 88], [27, 73]]"; #2. Chi-square test of independence and Fisher exact test; json load "rx_st_c" (stat chi2_independence [t]); json load "rx_st_f" (stat fisher_exact [t]); #3. Relative risk, absolute risk reduction and number needed to treat; -> "[rr]" (math calc "(12/100)/(27/100)"); -> "[arr]" (math calc "27/100-12/100"); concat "chi2 = " (math round (json select "rx_st_c" /chi2) 2) ", p = " (math round_sig (json select "rx_st_c" /p_value) 3) " ; Fisher p = " (math round_sig (json select "rx_st_f" /p_value) 3) ", OR = " (math round (json select "rx_st_f" /odds_ratio) 3) " ; RR = " (math round [rr] 3) ", ARR = " (math round (math calc "100*[arr]") 1) " %, NNT = " (math ceil (math calc "1/[arr]"))
chi2 = 7.17, p = 0.00743 ; Fisher p = 0.0118, OR = 0.369 ; RR = 0.444, ARR = 15 %, NNT = 7
- Get the confidence interval of the log odds ratio with
stat dist_inv - Test the goodness of fit of a distribution with
stat chi2_test - Use
stat wilcoxonfor paired continuous outcomes
Bayesian A/B test with Beta posteriors
Conversions 120/1000 against 145/1000 with uniform priors give Beta posteriors. The script computes the probability that B is better than A by a double integral of the joint posterior (normalised with lgamma) and checks it in one dimension with the incomplete beta function.
#1. A/B test: A 120 conversions out of 1000, B 145 out of 1000, uniform priors: Beta posteriors; -> "[a1]" 121; -> "[b1]" 881; -> "[a2]" 146; -> "[b2]" 856; #2. Log normalising constants with lgamma: ln B of a,b = lgamma a + lgamma b - lgamma a+b; -> "[lB1]" (math calc "lgamma([a1])+lgamma([b1])-lgamma([a1]+[b1])"); -> "[lB2]" (math calc "lgamma([a2])+lgamma([b2])-lgamma([a2]+[b2])"); #3. P of pB > pA: double integral of the joint posterior over y > x; -> "[P]" (num integrate2 (concat "exp(120*ln(x)+880*ln(1-x)-(" [lB1] ")+145*ln(y)+855*ln(1-y)-(" [lB2] "))") "x" 0.05 0.25 "y" "x" 0.3); #4. Check in one dimension with the incomplete beta function; -> "[P1]" (num integrate (concat "exp(120*ln(x)+880*ln(1-x)-(" [lB1] "))*(1-beta_inc(146, 856, x))") "x" 0.05 0.25); concat "P(B better than A) = " (math round [P] 4) " (1D check " (math round [P1] 4) "), posterior means " (math round (math calc "[a1]/([a1]+[b1])") 4) " and " (math round (math calc "[a2]/([a2]+[b2])") 4)
P(B better than A) = 0.9502 (1D check 0.9502), posterior means 0.1208 and 0.1457
- Compare with the frequentist test
stat chi2_independenceon the same 2x2 table. - The expected loss of choosing B is another integral with
num integrate2.
Before / after treatment: paired t-test or Wilcoxon
The systolic pressure of 12 patients is measured before and after 8 weeks of treatment. Test the normality of the differences (Shapiro-Wilk), run the paired t-test and the Wilcoxon signed-rank test, and give the effect size Cohen dz. Here the differences are not normal (p < 0.05), so the Wilcoxon test is the one to report.
#1. Systolic blood pressure in mmHg of 12 patients before and after 8 weeks of treatment; -> "[before]" "[152, 148, 160, 145, 158, 151, 163, 149, 155, 147, 159, 150]"; -> "[after]" "[141, 145, 148, 140, 150, 147, 151, 146, 144, 145, 149, 148]"; #2. Differences and their normality, which the paired t-test assumes; -> "[d]" (vector sub [before] [after]); json load "rx_sw" (stat shapiro [d]); #3. Paired t-test and its non-parametric counterpart; json load "rx_tp" (stat ttest_paired [before] [after]); json load "rx_wx" (stat wilcoxon [before] [after]); #4. Effect size: Cohen dz = mean of d / sd of d; -> "[dz]" (/ (stat mean [d]) (stat stddev [d])); concat "mean drop " (math round (stat mean [d]) 2) " mmHg | Shapiro W = " (math round (json select "rx_sw" "/w") 3) ", p = " (math round (json select "rx_sw" "/p_value") 3) " | paired t = " (math round (json select "rx_tp" "/t") 3) ", df = " (json select "rx_tp" "/df") ", p = " (math round_sig (json select "rx_tp" "/p_value") 3) " | Wilcoxon W = " (json select "rx_wx" "/w") ", p = " (math round_sig (json select "rx_wx" "/p_value") 3) " (" (json select "rx_wx" "/method") ") | Cohen dz = " (math round [dz] 2)
mean drop 6.92 mmHg | Shapiro W = 0.848, p = 0.035 | paired t = 5.812, df = 11, p = 0.000117 | Wilcoxon W = 0, p = 0.000488 (permutation) | Cohen dz = 1.68
- Give a 95 pct confidence interval of the mean drop with
stat conf_interval_meanon d. - Compute the sample size for a power of 80 pct with
stat dist_inv.
z-test with known sigma, power and sample size
A standardised reading test has a known sigma of 15 and a national mean of 100. A school tested 12 pupils. The statistician runs the z-test, computes the power to detect a 5-point shift with this n, and the n needed for 80 % power, all from the error function.
#1. Scores of the class, H0: mu = 100, sigma = 15; -> "[x]" "[112, 98, 105, 121, 93, 108, 117, 101, 99, 110, 125, 104]"; json load "rx_p4g_z" (stat ztest [x] 100 15); -> "[t]" (concat "mean = " (math round (stat mean [x]) 2) ", z = " (math round (json select "rx_p4g_z" /z) 3) ", p = " (math round (json select "rx_p4g_z" /p_value) 4)); json unload "rx_p4g_z"; #2. Critical values: Phi^-1 of p = sqrt2 . erf_inv of 2p - 1; -> "[za]" (* (math sqrt 2) (math erf_inv 0.95)); -> "[zb]" (* (math sqrt 2) (math erf_inv 0.6)); #3. Power for delta = 5 with n = 12, Phi of x = 0.5 . erfc of -x / sqrt2; -> "[k]" (- (/ (* 5 (math sqrt 12)) 15) [za]); -> "[pw]" (* 0.5 (math erfc (/ (- 0 [k]) (math sqrt 2)))); #4. n for 80 % power; -> "[n]" (math ceil (math power (/ (* (+ [za] [zb]) 15) 5) 2)); concat [t] [_n_] "z 0.975 = " (math round [za] 4) ", z 0.80 = " (math round [zb] 4) [_n_] "power for +5 points with n = 12: " (math round (* 100 [pw]) 1) " %" [_n_] "n for 80 % power: " [n]
mean = 107.75, z = 1.79, p = 0.0735 z 0.975 = 1.96, z 0.80 = 0.8416 power for +5 points with n = 12: 21 % n for 80 % power: 71
- Compare with
stat ttest_1when sigma is estimated from the sample. - Plot the power curve against n with
plot chart.
Summary of a 5-point Likert survey item
An item of a satisfaction survey is ordinal (1 = strongly disagree to 5 = strongly agree). The statistician reports what fits ordinal data: the frequency table, the mode, the median, the top-2-box and the bottom-2-box, with the mean only for information.
#1. Answers of 24 respondents; -> "[a]" "[4, 5, 3, 4, 4, 2, 5, 4, 3, 1, 4, 5, 4, 3, 2, 4, 5, 4, 3, 4, 5, 2, 4, 3]"; json load "rx_p4g_lk" (stat frequency [a]); #2. Frequency per answer; -> "[tab]" ""; for (-> "[i]" 0) (< [i] (json count "rx_p4g_lk" /)) (++ "[i]") { -> "[tab]" (concat [tab] (json select "rx_p4g_lk" (concat "/[" [i] "]/value")) ": " (math round (* 100 (json select "rx_p4g_lk" (concat "/[" [i] "]/frequency"))) 1) " % "); }; #3. Bottom-2 box = cumulative frequency of answer 2, top-2 = 1 - cumulative of 3; -> "[b2]" (json select "rx_p4g_lk" "/[1]/cumulative_frequency"); -> "[t2]" (- 1 (json select "rx_p4g_lk" "/[2]/cumulative_frequency")); json unload "rx_p4g_lk"; concat [tab] [_n_] "mode = " (stat mode [a]) ", median = " (stat median [a]) ", mean = " (math round (math avg [a]) 2) [_n_] "top-2-box = " (math round (* 100 [t2]) 1) " %, bottom-2-box = " (math round (* 100 [b2]) 1) " %, net score = " (math round (* 100 (- [t2] [b2])) 1)
1: 4.2 % 2: 12.5 % 3: 20.8 % 4: 41.7 % 5: 20.8 % mode = [4], median = 4, mean = 3.63 top-2-box = 62.5 %, bottom-2-box = 16.7 %, net score = 45.8
- Compare two groups of respondents with
stat mann_whitney. - Test the distribution against last year with
stat chi2_test.
Which mean? Arithmetic, geometric and harmonic in practice
Three classic cases where the arithmetic mean misleads: the average yearly growth of an investment (geometric mean), the P/E ratio of an equal-amount portfolio (harmonic mean), and the population versus sample variance of a full census (divide by n).
#1. Yearly growth factors: +25 %, -20 %, +15 %, +5 %; -> "[g]" "[1.25, 0.80, 1.15, 1.05]"; -> "[ga]" (math avg [g]); -> "[gg]" (stat geo_mean [g]); #2. Same amount invested in 4 shares with these P/E ratios; -> "[pe]" "[12, 18, 30, 45]"; #3. Census of the 6 branches of a company: staff per branch; -> "[s]" "[14, 22, 9, 31, 18, 26]"; concat "growth: arithmetic " (math round (* 100 (- [ga] 1)) 2) " %/yr is wrong, geometric " (math round (* 100 (- [gg] 1)) 2) " %/yr, check final value " (math round (math power [gg] 4) 4) " = " (math round (* 1.25 (* 0.8 (* 1.15 1.05))) 4) [_n_] "portfolio P/E: harmonic " (math round (stat harm_mean [pe]) 2) " (arithmetic would say " (math avg [pe]) ")" [_n_] "census variance /n = " (math round (stat pvariance [s]) 3) ", sample variance /(n-1) = " (math round (stat variance [s]) 3)
growth: arithmetic 6.25 %/yr is wrong, geometric 4.83 %/yr, check final value 1.2075 = 1.2075 portfolio P/E: harmonic 20.57 (arithmetic would say 26.25) census variance /n = 53.667, sample variance /(n-1) = 64.4
- Show the inequality harmonic <= geometric <= arithmetic on any positive series.
- Use
stat weighted_meanwhen the amounts invested differ.
The Student t density from the Gamma and Beta functions
To teach where the t density comes from, the script evaluates f(t) = Gamma((v+1)/2) / (sqrt(v pi) Gamma(v/2)) (1 + t^2/v)^(-(v+1)/2) with the Gamma function, then with the Beta form, then with lngamma for v = 200 (where Gamma overflows), and compares with the built-in law.
#1. v = 5 degrees of freedom, t = 1.5; -> "[v]" 5; -> "[t]" 1.5; -> "[k]" (math power (+ 1 (/ (* [t] [t]) [v])) (- 0 (/ (+ [v] 1) 2))); -> "[f1]" (* (/ (math gamma (/ (+ [v] 1) 2)) (* (math sqrt (* [v] (math pi))) (math gamma (/ [v] 2)))) [k]); #2. Same with the Beta function; -> "[f2]" (/ [k] (* (math sqrt [v]) (math beta 0.5 (/ [v] 2)))); #3. v = 200: Gamma of 100.5 is huge, use lngamma; -> "[c200]" (math exp (- (math lngamma 100.5) (math lngamma 100))); -> "[f200]" (* (/ [c200] (math sqrt (* 200 (math pi)))) (math power (+ 1 (/ 2.25 200)) -100.5)); concat "v = 5: Gamma form " (math round [f1] 8) ", Beta form " (math round [f2] 8) ", stat dist_pdf " (math round (stat dist_pdf "student" "[5]" 1.5) 8) [_n_] "v = 200: lngamma form " (math round [f200] 6) ", stat dist_pdf " (math round (stat dist_pdf "student" "[200]" 1.5) 6) ", normal " (math round (stat dist_pdf "normal" "[0, 1]" 1.5) 6)
v = 5: Gamma form 0.12451734, Beta form 0.12451734, stat dist_pdf 0.12451734 v = 200: lngamma form 0.129445, stat dist_pdf 0.129445, normal 0.129518
- Integrate the density with
num integrateto get the p-value. - Do the same for the chi2 density with
math gamma.
Mincer wage equation from a survey table in SQL
Survey records sit in a database table (embedded H2 here). The statistician loads log hourly wage against years of education and experience straight from a SELECT into a multiple regression, and reports the coefficients, their standard errors and the adjusted R2.
#1. Survey database, embedded H2 file under tmp; cm set "tuto_p4g_svy" {execute "db.h2.embedded.config.get" "[database]" "./tmp/tuto_p4g_svy/survey" "[user]" "tuto" "[password]" "tuto_pwd" ;}; sql connect "tuto_p4g_s2" {cm get "tuto_p4g_svy";}; sql dml "tuto_p4g_s2" "create table resp (educ double, exper double, lnwage double)"; sql dml "tuto_p4g_s2" "insert into resp values (10, 5, 2.45), (12, 3, 2.52), (12, 10, 2.78), (14, 2, 2.70), (16, 4, 2.98), (16, 12, 3.21), (11, 20, 2.80), (18, 6, 3.25), (13, 8, 2.80), (15, 15, 3.12), (12, 25, 2.95), (17, 9, 3.20)"; sql disconnect "tuto_p4g_s2"; #2. Multiple regression straight from the query, unused X fields empty; pa rm load "rx_p4g_wage" "tuto_p4g_svy" "educ" "exper" "" "" "" "lnwage" "select educ, exper, lnwage from resp"; json load "rx_p4g_b" (pa rm estimate_regression_parameters "rx_p4g_wage"); json load "rx_p4g_se" (pa rm estimate_regression_parameters_standard_errors "rx_p4g_wage"); #3. Return to one year of schooling = exp of b1 - 1; -> "[b1]" (json select "rx_p4g_b" "/[1]"); -> "[r]" (concat "ln w = " (math round (json select "rx_p4g_b" "/[0]") 3) " + " (math round [b1] 4) " educ + " (math round (json select "rx_p4g_b" "/[2]") 4) " exper" [_n_] "SE: " (math round (json select "rx_p4g_se" "/[0]") 3) ", " (math round (json select "rx_p4g_se" "/[1]") 4) ", " (math round (json select "rx_p4g_se" "/[2]") 4) ", t educ = " (math round (/ [b1] (json select "rx_p4g_se" "/[1]")) 2) [_n_] "adjusted R2 = " (math round (pa rm calculate_adjusted_r_squared "rx_p4g_wage") 4) ", return to schooling = " (math round (* 100 (- (math exp [b1]) 1)) 2) " % per year"); #4. Close and remove the temporary database; json unload "rx_p4g_b"; json unload "rx_p4g_se"; pa rm close "rx_p4g_wage"; cm remove "tuto_p4g_svy"; file delete "tmp/tuto_p4g_svy"; [r]
ln w = 1.296 + 0.1018 educ + 0.0195 exper SE: 0.087, 0.0057, 0.0021, t educ = 17.93 adjusted R2 = 0.9695, return to schooling = 10.71 % per year
- Add experience squared as a third field to get the concave profile.
- Check the residuals with
pa rm estimate_residualsandstat shapiro.
Diagnostic test: AUC and Youden threshold
From the biomarker level of 12 patients with or without the disease, compute the ROC curve and its area, then the cut-off that maximizes the Youden index J = sensitivity + specificity - 1.
#1. Biomarker in ng/mL - disease 1 or 0; -> "[y]" "[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]"; -> "[score]" "[1.2, 2.3, 1.8, 3.1, 2.7, 4.2, 3.9, 5.5, 4.8, 6.1, 2.9, 7.0]"; json load "rx_roc" (ml metrics roc [y] [score] "1"); #2. Youden index on each threshold; -> "[best]" -1; -> "[cut]" 0; -> "[se]" 0; -> "[sp]" 0; for (-> "[i]" 1) (< [i] (json count "rx_roc" "/thresholds")) (++ "[i]") { -> "[tpr]" (json select "rx_roc" (concat "/tpr/[" [i] "]")); -> "[fpr]" (json select "rx_roc" (concat "/fpr/[" [i] "]")); -> "[j]" (- [tpr] [fpr]); if (> [j] [best]) {-> "[best]" [j]; -> "[cut]" (json select "rx_roc" (concat "/thresholds/[" [i] "]")); -> "[se]" [tpr]; -> "[sp]" (- 1 [fpr]);}; }; -> "[res]" (concat "AUC " (math round (json select "rx_roc" "/auc") 3) " | cut-off >= " [cut] " ng/mL | sensitivity " (math round [se] 3) " | specificity " (math round [sp] 3) " | Youden J " (math round [best] 3)); json unload "rx_roc"; [res]
AUC 0.917 | cut-off >= 3.9 ng/mL | sensitivity 0.833 | specificity 0.833 | Youden J 0.667
- Check the classification at this cut-off with
ml metrics report. ml metrics aucreturns only the area, to compare two biomarkers.
How many populations? Gaussian mixture and BIC
Fit gaussian mixtures with 1, 2 and 3 components on two biomarkers of 15 subjects and keep the number of populations with the lowest BIC, with the weight and mean of each population.
#1. Two biomarkers per subject; -> "[X]" "[[1.0, 2.1], [1.2, 1.9], [0.8, 2.3], [1.1, 2.0], [0.9, 1.8], [1.3, 2.2], [1.0, 2.4], [4.1, 6.0], [3.6, 6.4], [4.5, 5.7], [4.0, 6.3], [3.7, 5.8], [4.4, 6.2], [4.2, 5.6], [3.9, 6.1]]"; -> "[best]" 0; -> "[bic]" 1e300; -> "[report]" ""; for (-> "[k]" 1) (<= [k] 3) (++ "[k]") { json load "rx_g" (ml fit "rx_gmm" "gmm" [X] (concat "{\"k\": " [k] ", \"seed\": 1}")); -> "[b]" (json select "rx_g" "/bic"); -> "[report]" (concat [report] "k=" [k] " BIC " (math round [b] 1) "; "); if (< [b] [bic]) {-> "[bic]" [b]; -> "[best]" [k];}; }; json load "rx_g" (ml fit "rx_gmm" "gmm" [X] (concat "{\"k\": " [best] ", \"seed\": 1}")); -> "[res]" (concat [report] "populations: " [best] " | weights " (json select "rx_g" "/weights") " | means " (json select "rx_g" "/means")); json unload "rx_g"; ml model delete "rx_gmm"; [res]
k=1 BIC 87.4; k=2 BIC 45.4; k=3 BIC 60.4; populations: 2 | weights [
0.4666666666666667,
0.5333333333333333
] | means [
[
1.042857142857143,
2.1
],
[
4.05,
6.012500000000001
]
]ml model transformgives the probability of each subject to belong to each population.- The AIC is also returned; the BIC penalizes more the complex models.
Correspondence analysis of a survey table
Cross the hair color and the eye color of 592 students, test the independence and place the categories on the first axis of a correspondence analysis.
#1. Rows: black, brown, red, blond hair - columns: brown, hazel, green, blue eyes; -> "[t]" "[[68, 15, 5, 20], [119, 54, 29, 84], [26, 14, 14, 17], [7, 10, 16, 94]]"; json load "rx_ca" (ml ca [t] 2); -> "[res]" (concat "chi2 " (math round (json select "rx_ca" "/chi2") 1) " p = " (json select "rx_ca" "/p_value") " | axis 1: " (math round (* 100 (json select "rx_ca" "/explained_ratio/[0]")) 1) " % of the inertia | black hair " (math round (json select "rx_ca" "/row_coordinates/[0]/[0]") 3) ", blond hair " (math round (json select "rx_ca" "/row_coordinates/[3]/[0]") 3) " | brown eyes " (math round (json select "rx_ca" "/column_coordinates/[0]/[0]") 3) ", blue eyes " (math round (json select "rx_ca" "/column_coordinates/[3]/[0]") 3)); json unload "rx_ca"; [res]
chi2 138.3 p = 0 | axis 1: 89.4 % of the inertia | black hair -0.505, blond hair 0.835 | brown eyes -0.492, blue eyes 0.547
- The categories close on an axis are associated: blond hair with blue eyes, black hair with brown eyes.
stat chi2_independencegives the same chi2 test with the expected counts.
Prediction interval of a calibration
Fit the linear calibration of an instrument, then give the 95 % prediction interval of a new measure, exact Student interval, and compare with the Laplace interval of a support vector regression.
#1. Reference concentration in mg/L - signal; -> "[X]" "[[0], [5], [10], [15], [20], [25], [30], [35], [40], [45]]"; -> "[y]" "[0.02, 0.51, 0.98, 1.52, 2.05, 2.48, 3.01, 3.55, 3.98, 4.51]"; ml train "rx_cal" "linear" [X] [y]; ml train "rx_svr" "svm_regressor" [X] [y] "{\"kernel\": \"linear\", \"C\": 10, \"epsilon\": 0.01, \"probability\": true}"; json load "rx_i" (ml model predict_interval "rx_cal" "[[22]]" 0.95); json load "rx_j" (ml model predict_interval "rx_svr" "[[22]]" 0.95); -> "[res]" (concat "OLS at 22 mg/L: " (math round (json select "rx_i" "/[0]/prediction") 4) " [" (math round (json select "rx_i" "/[0]/lower") 4) " - " (math round (json select "rx_i" "/[0]/upper") 4) "] | SVR: " (math round (json select "rx_j" "/[0]/prediction") 4) " [" (math round (json select "rx_j" "/[0]/lower") 4) " - " (math round (json select "rx_j" "/[0]/upper") 4) "]"); json unload "rx_i"; json unload "rx_j"; ml model delete "rx_cal"; ml model delete "rx_svr"; [res]
OLS at 22 mg/L: 2.211 [2.1443 - 2.2777] | SVR: 2.2033 [2.1163 - 2.2904]
- The prediction interval is wider than the confidence interval of the line: it includes the noise of one new measure.
- Invert the calibration with
ml trainlinear on [signal] -> concentration to read unknown samples.
Is the best model really better?
Like the Weka Experimenter: the same 10 x 10-fold cross validation for five algorithms, and a corrected resampled paired t-test of each one against naive Bayes, to know which differences are significant and which are noise.
#1. Measures and species; -> "[X]" "[[-0.08, 1.37, -0.34, -1.78], [2.26, 1.1, 2.61, 0.57], [0.95, 1.53, 1.25, -0.14], [-0.26, 0.92, -0.03, -1.17], [2.72, 0.57, 1.91, 1.61], [1.6, 0.48, 1.0, -1.47], [1.52, -0.5, -0.79, 1.95], [0.5, 0.63, 0.02, -0.42], [0.37, 2.72, 0.95, 0.46], [0.68, -1.46, -0.54, -0.14], [0.59, 0.0, 0.3, 1.13], [0.47, 2.2, 1.15, 1.16], [-0.02, 1.0, -0.26, -0.62], [2.71, 1.27, 3.45, 0.73], [1.29, 2.81, 4.02, 0.64], [0.86, -0.26, -0.36, -2.28], [2.48, 0.63, 1.77, -1.79], [0.66, 1.59, 0.53, 1.04], [1.23, -1.06, -1.13, 1.88], [0.99, 1.51, 2.02, 2.02], [-0.15, 1.13, -0.07, 1.35], [-1.05, -0.59, 0.31, 1.25], [1.59, 1.13, 2.03, 1.3], [1.49, 2.42, 3.99, 0.6], [-1.06, -0.39, 0.73, 0.26], [1.78, 0.83, 2.53, 0.07], [1.55, 2.53, 3.87, 1.34], [0.9, 0.47, 0.04, 1.41], [2.81, 0.53, 0.69, 0.06], [-0.01, 1.89, -0.15, -0.73], [1.04, 0.21, 0.63, -0.87], [1.93, 1.86, 3.73, -0.63], [0.28, 1.56, 0.45, 0.82], [0.24, 0.8, 0.49, 0.24], [3.3, 1.29, 4.22, -0.05], [1.09, 1.42, 1.17, 0.15], [-0.37, 0.65, -0.39, 0.22], [2.29, 0.17, 0.48, -0.63], [0.71, 2.84, 2.16, 0.11], [-0.62, 0.74, 0.18, -2.12], [3.07, 0.8, 2.83, 1.31], [0.16, 3.71, 0.45, -0.15], [0.68, 0.02, 0.13, -0.42], [1.14, 0.61, 0.4, 1.49], [1.32, 3.4, 4.14, -2.03], [-2.01, -0.25, 0.14, 0.48], [0.75, -0.68, -0.39, -0.3], [1.72, 2.84, 5.04, -1.14], [-0.7, 2.19, -1.58, 0.41], [2.21, 0.57, 1.24, -0.47], [0.62, 0.24, 0.69, -0.0], [-1.12, 0.57, -0.4, -1.37], [1.62, 1.98, 3.39, 0.55], [2.02, 1.98, 3.92, 1.74], [-0.07, 0.02, 0.23, 0.14], [0.96, 2.34, 2.25, 1.95], [1.57, 1.26, 2.48, -0.07], [-0.7, 0.07, -0.68, 0.52], [1.42, 0.36, 0.28, 0.8], [1.43, 3.09, 4.96, 0.4], [0.18, 0.38, -0.15, -0.45], [2.79, -0.2, -1.11, -0.25], [1.33, 3.45, 4.41, -0.85], [0.18, -0.05, 0.1, -0.42], [2.28, 0.06, 0.47, 0.39], [1.62, 1.69, 2.33, 0.12], [0.04, 1.11, 0.38, 0.07], [2.05, -0.32, -0.64, 0.32], [0.71, 1.1, 1.2, -0.84], [1.61, 1.18, 2.4, -0.77], [2.67, -0.69, -1.91, -1.46], [0.42, 1.63, 1.0, 1.2], [-0.25, -0.17, -0.21, -1.07], [0.45, 0.39, 0.38, -0.69], [1.42, 1.03, 1.18, -0.45], [0.89, -0.4, -0.09, -0.27], [1.07, -1.26, -1.46, -0.89], [1.91, 2.48, 4.26, 0.94], [0.56, -2.57, -1.74, 0.33], [0.62, -0.09, 0.13, 0.34], [1.49, 2.63, 3.67, 2.5], [0.55, 2.02, 1.06, 0.65], [2.38, 0.44, 1.16, 0.34], [1.81, 2.36, 4.32, 0.57], [1.39, 0.14, 0.4, 0.22], [0.29, -0.25, -0.33, -1.32], [0.51, 0.47, 0.34, -0.46], [1.18, 0.42, 0.13, -0.14], [2.11, -0.05, -0.18, -1.01], [0.41, 2.32, 0.86, 1.29]]"; -> "[y]" "[\"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\", \"setosa\", \"versi\", \"virgi\"]"; json load "rx_c" (ml compare [X] [y] "[\"naive_bayes\", \"logistic\", {\"algo\": \"knn_classifier\", \"params\": {\"k\": 7}}, \"bayes_net\", \"one_r\"]"); -> "[res]" ""; for (-> "[i]" 0) (< [i] (json count "rx_c" "/results")) (++ "[i]") { -> "[p]" (concat "/results/[" [i] "]"); -> "[res]" (concat [res] (json select "rx_c" (concat [p] "/algo")) " " (math round (json select "rx_c" (concat [p] "/mean")) 3) (if (> [i] 0) {concat " " (json select "rx_c" (concat [p] "/vs_baseline")) " p=" (math round (json select "rx_c" (concat [p] "/p_value")) 3)} {" baseline"}) " | "); }; json unload "rx_c"; [res]
naive_bayes 0.731 baseline | logistic 0.702 same p=0.394 | knn_classifier 0.719 same p=0.756 | bayes_net 0.603 worse p=0.004 | one_r 0.572 worse p=0.017 |
- The corrected t-test (Nadeau and Bengio) takes into account that the training sets overlap: the naive t-test finds too many differences.
- Give {"k": 5, "repeats": 3} for the slow algorithms.
How many subjects for the study?
Plan a study before collecting the data: the number of subjects per group to detect a medium effect (d = 0.5) with a t test, a difference of proportions 65 % vs 45 %, and an ANOVA on 4 groups, with 80 % and 90 % power.
json load "rx_a" (stat sample_size "t_two" "{\"d\": 0.5}"); json load "rx_b" (stat sample_size "t_two" "{\"d\": 0.5, \"power\": 0.9}"); json load "rx_c" (stat sample_size "prop_two" "{\"p1\": 0.65, \"p2\": 0.45}"); json load "rx_d" (stat sample_size "anova" "{\"f\": 0.25, \"k\": 4}"); json load "rx_e" (stat power "t_two" "{\"d\": 0.5, \"n\": 30}"); -> "[res]" (concat "t test: " (json select "rx_a" "/n") " per group at 80 %, " (json select "rx_b" "/n") " at 90 % | proportions: " (json select "rx_c" "/n") " per group | ANOVA 4 groups: " (json select "rx_d" "/n") " per group | with only 30 per group, power = " (math round (json select "rx_e" "/power") 2)); json unload "rx_a"; json unload "rx_b"; json unload "rx_c"; json unload "rx_d"; json unload "rx_e"; [res]
t test: 64 per group at 80 %, 86 at 90 % | proportions: 96 per group | ANOVA 4 groups: 45 per group | with only 30 per group, power = 0.48
- d = difference of the means / standard deviation: 0.2 small, 0.5 medium, 0.8 large.
- Add 10 to 20 % for the subjects lost during the study.
Decide an A/B test
A new landing page (B) is tested against the current one (A): 120 conversions on 2400 visits against 156 on 2380. The z test, the 95 % interval of the difference and the Bayesian probability that B is better support the decision.
json load "rx_t" (analytics abtest 120 2400 156 2380); json load "rx_p" (stat power "prop_two" "{\"p1\": 0.05, \"p2\": 0.065, \"n\": 2400}"); -> "[res]" (concat "A " (math round (* 100 (json select "rx_t" "/rate_a")) 2) " %, B " (math round (* 100 (json select "rx_t" "/rate_b")) 2) " % | lift " (math round (* 100 (json select "rx_t" "/lift")) 1) " % [" (math round (* 100 (json select "rx_t" "/lower")) 2) ", " (math round (* 100 (json select "rx_t" "/upper")) 2) "] points | p = " (math round (json select "rx_t" "/p_value") 4) " | P(B > A) = " (math round (json select "rx_t" "/probability_b_better") 3) " | power " (math round (json select "rx_p" "/power") 2)); json unload "rx_t"; json unload "rx_p"; [res]
A 5 %, B 6.55 % | lift 31.1 % [0.23, 2.88] points | p = 0.0212 | P(B > A) = 0.989 | power 0.61
- Fix the sample size before the test with
stat sample_size "prop_two": stopping when p < 0.05 inflates the false positives. - The Bayesian probability is easier to explain than the p-value.
Bayesian logistic regression by MCMC
Three doses of a product are given to 20 animals each, with 2, 8 and 17 deaths: the log posterior of the logistic model is written directly, then sampled by MCMC (4 chains); the dose that kills half of the animals (LD50 = -a / b) is computed from the posterior means.
#1. Log likelihood of the binomial logistic model, flat priors; -> "[lp]" "2*(a+b*1) - 20*ln(1+exp(a+b*1)) + 8*(a+b*2) - 20*ln(1+exp(a+b*2)) + 17*(a+b*3) - 20*ln(1+exp(a+b*3))"; json load "rx_m" (ml mcmc [lp] "{\"a\": 0, \"b\": 0}" "{\"iterations\": 4000}"); -> "[a]" (json select "rx_m" "/parameters/[0]/mean"); -> "[b]" (json select "rx_m" "/parameters/[1]/mean"); -> "[res]" (concat "b = " (math round [b] 2) " [" (math round (json select "rx_m" "/parameters/[1]/lower") 2) ", " (math round (json select "rx_m" "/parameters/[1]/upper") 2) "] | LD50 about dose " (math round (/ (* -1 [a]) [b]) 2) " | R-hat " (math round (json select "rx_m" "/parameters/[1]/rhat") 3) ", ESS " (math round (json select "rx_m" "/parameters/[1]/ess") 0) " | converged " (json select "rx_m" "/converged")); json unload "rx_m"; [res]
b = 2.11 [1.21, 3.19] | LD50 about dose 2.16 | R-hat 1.002, ESS 2228 | converged true
- ln is the natural logarithm in the expressions (log is the base 10 logarithm).
- Add {"samples": 1000} to get draws and compute the posterior of any function of the parameters, like the LD50 itself.
Repeated measures: each subject has its own slope
Eight patients are measured every week for 6 weeks. A random intercept only lets each patient start at its own level; the random slope also lets each one progress at its own speed. The AIC of the two REML models tells whether the slopes differ between patients.
#1. Week, score and patient; -> "[X]" "[[0], [1], [2], [3], [4], [5], [0], [1], [2], [3], [4], [5], [0], [1], [2], [3], [4], [5], [0], [1], [2], [3], [4], [5], [0], [1], [2], [3], [4], [5], [0], [1], [2], [3], [4], [5], [0], [1], [2], [3], [4], [5], [0], [1], [2], [3], [4], [5]]"; -> "[y]" "[46.3, 43.5, 48.3, 46.8, 53.4, 53.7, 56.0, 57.4, 59.1, 62.8, 63.1, 66.9, 52.2, 59.1, 59.0, 61.1, 65.3, 67.3, 51.5, 51.1, 53.9, 57.2, 61.8, 61.0, 45.3, 52.9, 60.4, 68.8, 70.0, 73.7, 52.1, 55.7, 57.7, 60.2, 61.5, 66.5, 55.3, 60.2, 62.8, 63.6, 67.5, 72.8, 49.9, 52.2, 54.8, 54.5, 62.7, 62.9]"; -> "[g]" "[\"p1\", \"p1\", \"p1\", \"p1\", \"p1\", \"p1\", \"p2\", \"p2\", \"p2\", \"p2\", \"p2\", \"p2\", \"p3\", \"p3\", \"p3\", \"p3\", \"p3\", \"p3\", \"p4\", \"p4\", \"p4\", \"p4\", \"p4\", \"p4\", \"p5\", \"p5\", \"p5\", \"p5\", \"p5\", \"p5\", \"p6\", \"p6\", \"p6\", \"p6\", \"p6\", \"p6\", \"p7\", \"p7\", \"p7\", \"p7\", \"p7\", \"p7\", \"p8\", \"p8\", \"p8\", \"p8\", \"p8\", \"p8\"]"; json load "rx_1" (stat lmm [X] [y] [g]); json load "rx_2" (stat lmm [X] [y] [g] "{\"random_slopes\": [0]}"); -> "[res]" (concat "mean progress " (math round (json select "rx_2" "/fixed_effects/[1]/estimate") 2) " per week (se " (math round (json select "rx_2" "/fixed_effects/[1]/se") 2) ", random intercept only: se " (math round (json select "rx_1" "/fixed_effects/[1]/se") 2) ") | sd between patients: level " (math round (json select "rx_2" "/random_effects/[0]/sd") 2) ", slope " (math round (json select "rx_2" "/random_effects/[1]/sd") 2) " | AIC " (math round (json select "rx_1" "/aic") 1) " -> " (math round (json select "rx_2" "/aic") 1) " | p1 slope " (math round (+ (json select "rx_2" "/fixed_effects/[1]/estimate") (json select "rx_2" "/group_effects/p1/x0")) 2)); json unload "rx_1"; json unload "rx_2"; [res]
mean progress 2.93 per week (se 0.43, random intercept only: se 0.23) | sd between patients: level 3.9, slope 1.13 | AIC 259 -> 243.7 | p1 slope 1.83
- When the slopes really differ, the random intercept model underestimates the standard error of the mean slope: the random slope model is the honest one.
- Both models are fitted by REML with the same fixed effects, so their AIC can be compared.
Missing values: multiple imputation instead of deleting the rows
The blood pressure is explained by the age and the body mass index, but the BMI is missing for a third of the patients, more often for the older ones. Deleting these rows loses power; multiple imputation (MICE) fills the BMI 20 times from the other columns, fits the regression on each completed table and pools the results by Rubin's rules.
#1. Systolic pressure, age, BMI (null = missing); -> "[d]" "[[165.7, 76.0, 22.9], [148.4, 46.9, 21.5], [160.7, 54.6, null], [172.7, 63.0, null], [136.0, 30.5, 24.5], [141.1, 53.2, 21.2], [158.2, 61.2, 25.9], [135.5, 38.5, 23.6], [132.7, 31.2, 26.1], [152.4, 43.0, 26.7], [153.1, 53.2, null], [162.9, 53.6, 25.3], [149.9, 44.4, 29.1], [152.3, 59.9, null], [149.3, 54.0, 20.1], [146.4, 42.2, 30.3], [181.0, 66.2, null], [145.4, 54.3, 28.4], [123.6, 59.9, 17.3], [156.0, 41.4, 19.4], [148.9, 51.7, 27.6], [153.8, 51.7, null], [131.8, 42.8, 25.2], [146.3, 44.1, 18.1], [157.1, 36.1, 32.4], [135.8, 26.5, 19.6], [142.4, 44.3, 23.4], [149.8, 61.8, 21.9], [159.1, 63.2, 30.2], [179.8, 63.3, null], [140.7, 36.1, 23.4], [158.0, 65.3, null], [157.2, 34.9, 30.1], [157.9, 68.8, null], [146.4, 45.6, 26.6], [134.2, 37.0, 20.4], [170.0, 42.4, 33.8], [151.4, 62.7, 19.7], [140.4, 42.1, 23.6], [160.3, 60.1, null], [132.7, 30.7, 28.6], [155.6, 52.1, null], [155.3, 48.0, 33.7], [162.8, 49.9, 29.7], [155.7, 55.8, 23.9], [143.7, 42.7, 24.8], [163.7, 48.2, null], [154.9, 68.2, 26.6], [162.3, 48.7, 27.5], [153.2, 41.1, 34.3], [171.1, 66.4, null], [149.4, 64.9, 25.4], [144.0, 57.1, null], [163.9, 44.2, 32.3], [139.7, 40.6, 27.1], [143.3, 39.9, 22.0], [133.4, 49.1, 16.8], [168.5, 73.2, null], [152.3, 53.4, 18.7], [143.6, 53.0, null], [146.4, 46.5, 26.6], [150.3, 61.6, null], [148.4, 42.9, 31.1], [132.1, 52.5, 16.8], [127.7, 40.9, 22.6], [130.1, 33.1, 22.9], [149.4, 56.4, null], [156.7, 52.8, 17.8], [140.6, 45.4, 20.0], [166.0, 57.2, 31.3], [154.9, 44.1, 25.2], [146.9, 55.6, 24.1], [139.5, 50.0, 20.5], [153.1, 48.4, 29.1], [143.2, 34.9, 25.5], [155.2, 40.3, 32.5], [146.1, 55.6, 24.9], [153.3, 23.0, 31.5], [171.3, 67.3, 31.7], [130.8, 52.4, 24.2], [122.2, 40.5, 25.3], [138.8, 33.4, 27.1], [151.2, 49.5, 24.5], [157.7, 43.2, 31.1], [156.2, 50.1, 26.0], [124.8, 31.1, 18.6], [160.4, 63.7, 32.2], [158.9, 62.6, 30.0], [154.2, 59.3, 28.2], [157.8, 35.5, 27.2], [137.1, 54.1, 27.9], [139.4, 33.7, 20.9], [125.2, 35.3, 24.6], [137.7, 34.6, 26.9], [166.9, 58.5, null], [141.1, 42.1, 21.8], [128.8, 15.6, 24.2], [152.9, 44.6, 20.0], [161.7, 58.2, 30.7], [160.9, 53.4, 33.1], [153.3, 58.9, 30.6], [157.5, 34.7, 33.9], [146.5, 49.0, null], [143.2, 39.3, 21.3], [136.3, 35.4, 28.6], [155.5, 63.7, null], [152.1, 63.1, 21.3], [137.5, 44.7, 26.5], [163.6, 66.1, null], [141.6, 47.0, 21.1], [124.9, 39.0, 21.2], [126.8, 30.0, 24.6], [158.3, 46.3, 27.8], [161.0, 50.7, 30.1], [151.9, 68.7, null], [150.2, 30.0, 29.1], [163.6, 62.6, null], [133.3, 44.1, 21.4], [155.9, 78.9, 22.7], [145.6, 18.7, 28.3], [135.8, 48.8, 22.3], [155.2, 45.7, 29.5], [153.6, 51.5, null], [150.2, 52.2, 18.8], [159.8, 64.4, 30.3], [156.2, 63.6, 20.4], [163.5, 62.1, null], [134.9, 38.4, 18.5], [144.7, 47.2, 18.2], [150.1, 36.6, 29.7], [154.2, 69.9, 23.0], [147.2, 53.2, null], [173.1, 61.3, 22.9], [152.4, 73.1, null], [139.5, 42.7, 25.3], [151.8, 41.7, 27.6], [174.9, 62.3, null], [144.2, 44.4, 27.1], [144.9, 43.5, 23.1], [174.6, 54.8, null], [145.0, 42.0, 24.9], [148.7, 49.1, 25.9], [149.7, 43.7, 29.8], [161.5, 46.5, 25.0], [162.5, 56.5, null], [170.3, 54.6, 32.9], [142.8, 35.7, 25.4], [147.8, 46.6, 25.6], [141.8, 33.5, 19.7], [149.1, 50.1, 25.1], [151.6, 64.6, 25.4], [132.2, 30.3, 27.0], [144.4, 56.4, 18.9], [136.8, 40.1, 23.0], [157.9, 51.4, 29.0], [151.0, 47.7, 21.7], [161.8, 56.9, null], [156.1, 57.9, null], [147.4, 46.4, null], [146.4, 61.8, null], [149.8, 52.6, 23.8], [149.3, 45.8, 28.6], [151.6, 37.2, 32.0], [125.8, 43.1, 25.0], [169.8, 60.5, 31.9], [157.9, 51.2, 26.4], [127.9, 24.0, 27.0], [162.9, 42.5, 28.4], [146.7, 55.2, 22.2], [127.7, 48.8, 19.9], [163.6, 39.3, 31.2], [144.9, 25.9, 20.9], [149.3, 37.9, 23.6], [146.3, 38.8, 26.0], [149.9, 52.4, null], [155.6, 65.1, null], [140.6, 46.7, 23.0], [168.0, 63.5, null], [154.8, 44.5, 30.0], [145.3, 43.1, 20.9], [139.4, 44.0, 24.2], [137.2, 41.9, 22.2], [161.2, 50.9, null], [156.1, 62.8, null], [121.4, 48.2, 25.3], [166.8, 68.8, 20.2], [140.7, 54.1, 22.8], [154.3, 62.6, null], [161.9, 46.4, 28.5], [160.6, 63.3, 26.2], [172.3, 46.6, 24.7], [134.1, 38.0, null], [165.2, 48.7, 28.8], [157.7, 52.3, 21.8], [154.2, 48.9, 27.9], [149.7, 73.3, 26.5], [154.7, 29.8, 33.0], [146.5, 56.0, 24.8], [142.9, 51.0, 19.6], [158.1, 68.0, 29.5]]"; json load "rx_i" (stat mice [d] "{\"m\": 20, \"analysis\": {\"y\": 0}}"); -> "[res]" (concat "missing BMI: " (json select "rx_i" "/missing_by_column/[2]") " / " (json select "rx_i" "/n") " | pooled: age " (math round (json select "rx_i" "/pooled/[1]/estimate") 3) " (se " (math round (json select "rx_i" "/pooled/[1]/se") 3) "), BMI " (math round (json select "rx_i" "/pooled/[2]/estimate") 3) " (se " (math round (json select "rx_i" "/pooled/[2]/se") 3) ", fraction of missing information " (math round (json select "rx_i" "/pooled/[2]/fmi") 2) ")"); json unload "rx_i"; [res]
missing BMI: 42 / 200 | pooled: age 0.558 (se 0.054), BMI 1.332 (se 0.148, fraction of missing information 0.11)
- For another analysis, ask {"return_data": true}, run it on each completed table and pool the m results with stat pool.
- Predictive mean matching (pmm, the default) only imputes values seen in the data: no negative BMI, integers stay integers.
Repeated measures: time, group and their interaction
Sixteen subjects, half on a diet, have their weight measured 4 times. A mixed (split-plot) ANOVA tests the time, the group and the interaction time x group (do the groups evolve differently?); the Greenhouse-Geisser correction protects against a violation of sphericity.
#1. One row per subject: the 4 measures; -> "[w]" "[[118.3, 116.1, 114.2, 104.1], [114.5, 112.3, 109.1, 103.3], [118.5, 119.1, 112.6, 111.6], [122.2, 117.7, 113.8, 109.6], [129.4, 128.6, 122.0, 118.4], [116.6, 110.4, 112.2, 108.7], [131.0, 122.3, 115.7, 119.0], [119.4, 116.1, 116.2, 114.8], [121.0, 123.4, 123.2, 117.6], [122.1, 123.3, 123.5, 121.8], [113.0, 108.5, 112.1, 105.8], [116.5, 115.2, 116.3, 115.6], [130.6, 131.2, 125.7, 129.5], [127.6, 126.9, 121.2, 119.5], [114.7, 118.1, 110.2, 113.7], [125.0, 121.6, 121.9, 124.1]]"; json load "rx_r" (stat rm_anova [w] "{\"groups\": [\"diet\", \"diet\", \"diet\", \"diet\", \"diet\", \"diet\", \"diet\", \"diet\", \"control\", \"control\", \"control\", \"control\", \"control\", \"control\", \"control\", \"control\"]}"); -> "[res]" (concat "time: F " (math round (json select "rx_r" "/anova/[2]/f") 1) ", p GG " (math round (json select "rx_r" "/sphericity/conditions_p_gg") 6) " | time x group: F " (math round (json select "rx_r" "/anova/[3]/f") 1) ", p GG " (math round (json select "rx_r" "/sphericity/interaction_p_gg") 6) " | group: p " (math round (json select "rx_r" "/anova/[0]/p_value") 4) " | Mauchly p " (math round (json select "rx_r" "/sphericity/p_value") 3) ", epsilon GG " (math round (json select "rx_r" "/sphericity/greenhouse_geisser_epsilon") 2)); json unload "rx_r"; [res]
time: F 21.3, p GG 0 | time x group: F 6, p GG 0.00222 | group: p 0.1959 | Mauchly p 0.895, epsilon GG 0.93
- The interaction is the question of a trial: a significant time effect alone only says that the weights change.
- With several outcomes measured once (weight, fat mass, waist), compare the groups with stat manova.
Small sample: Bayesian logistic regression and the probability of an effect
Thirty patients, a dose and a genotype: with so few events the maximum likelihood estimates are unstable and the p-value answers an indirect question. The Bayesian logistic regression gives the posterior of each coefficient, its credible interval and directly the probability that the dose increases the response.
#1. Dose and genotype, response (0 / 1); -> "[X]" "[[0.15, 0], [1.48, 0], [1.99, 0], [2.31, 1], [0.23, 0], [1.5, 0], [3.34, 0], [2.39, 1], [1.51, 0], [0.75, 0], [1.18, 0], [0.41, 0], [2.49, 1], [2.66, 1], [3.39, 1], [2.37, 1], [2.69, 0], [2.21, 0], [3.6, 1], [0.17, 1], [0.33, 0], [3.49, 0], [1.23, 1], [0.15, 0], [3.17, 0], [0.07, 1], [2.13, 1], [2.81, 1], [1.54, 0], [0.82, 1]]"; -> "[y]" "[0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1]"; json load "rx_b" (stat bayes_glm [X] [y] "binomial"); json load "rx_g" (stat glm [X] [y] "binomial"); -> "[res]" (concat "dose: posterior median " (math round (json select "rx_b" "/coefficients/[1]/median") 2) " [" (math round (json select "rx_b" "/coefficients/[1]/lower") 2) ", " (math round (json select "rx_b" "/coefficients/[1]/upper") 2) "], P(effect > 0) = " (math round (json select "rx_b" "/coefficients/[1]/probability_positive") 3) " | maximum likelihood " (math round (json select "rx_g" "/coefficients/[1]/estimate") 2) ", p = " (math round (json select "rx_g" "/coefficients/[1]/p_value") 3) " | R-hat " (math round (json select "rx_b" "/coefficients/[1]/rhat") 3)); json unload "rx_b"; json unload "rx_g"; [res]
dose: posterior median 0.43 [-0.27, 1.11], P(effect > 0) = 0.872 | maximum likelihood 0.37, p = 0.287 | R-hat 1.001
- The prior normal(0, 2.5) on the standardized slopes pulls the extreme estimates toward 0; {"prior_sd": 1} pulls harder, {"prior_sd": 10} almost not.
- Check R-hat < 1.05 and ess > 400 before reading the results; increase iterations otherwise.
Size of a survival trial: events, patients and power
A trial must detect a hazard ratio of 0.75 between two arms with a power of 80 % (two-sided 5 %). The log-rank test needs a number of events (Schoenfeld); the number of patients depends on the share of patients with an event during the follow-up. The power of the trial if only 500 patients can be included is also computed.
#1. 35 % of the patients have an event during the follow-up; json load "rx_n" (stat sample_size "logrank" "{\"hr\": 0.75, \"p_event\": 0.35}"); json load "rx_n2" (stat sample_size "logrank" "{\"hr\": 0.75, \"p_event\": 0.35, \"ratio\": 0.667}"); json load "rx_w" (stat power "logrank" "{\"hr\": 0.75, \"p_event\": 0.35, \"n\": 500}"); -> "[res]" (concat "1:1 -> " (json select "rx_n" "/events") " events, " (json select "rx_n" "/n") " patients | 2:1 -> " (json select "rx_n2" "/n") " patients | power with 500 patients: " (math round (* 100 (json select "rx_w" "/power")) 1) " %"); json unload "rx_n"; json unload "rx_n2"; json unload "rx_w"; [res]
1:1 -> 380 events, 1084 patients | 2:1 -> 1220 patients | power with 500 patients: 47.7 %
- Add 10 to 20 % for the patients lost to follow-up.
- A longer follow-up raises p_event and lowers the number of patients for the same number of events.
A/B test of a landing page: probability to win and expected loss
Two versions of a landing page: A (current) converts 312 visitors out of 4 120, B (new) 368 out of 4 087. Instead of a p-value, the Bayesian A/B test gives the probability that B is better, the interval of the lift, and the expected loss of each decision: switch to B when its expected loss is below the threshold of caring (here 0.1 point of conversion).
json load "rx_b" (stat bayes_ab 312 4120 368 4087); -> "[loss]" (json select "rx_b" "/expected_loss_choosing_b"); -> "[decision]" "keep collecting data"; if (< [loss] 0.001) {-> "[decision]" "switch to B";}; -> "[res]" (concat "conversion A " (math round (* 100 (json select "rx_b" "/rate_a")) 2) " %, B " (math round (* 100 (json select "rx_b" "/rate_b")) 2) " % | P(B > A) = " (math round (json select "rx_b" "/prob_b_beats_a") 4) " | lift " (math round (* 100 (json select "rx_b" "/lift_median")) 1) " % [" (math round (* 100 (json select "rx_b" "/lift_lower")) 1) ", " (math round (* 100 (json select "rx_b" "/lift_upper")) 1) "] | expected loss of B " (math round (* 100 [loss]) 3) " point -> " [decision]); json unload "rx_b"; [res]
conversion A 7.59 %, B 9.02 % | P(B > A) = 0.9906 | lift 18.9 % [2.9, 37.5] | expected loss of B 0.002 point -> switch to B
- The probability is exact (Evan Miller formula); prob_b_beats_a_mc is its Monte Carlo check.
- With a prior from the past campaigns, give {"prior": [a, b]}; stat bayes_factor_t compares two means in the same spirit, stat prop_test gives the classical test.
Multi-center trial: partial pooling of the effects of 8 centers
Each of 8 hospitals estimated the effect of a treatment on the LDL cholesterol (mg/dL) with its standard error. The small centers give extreme and uncertain effects. The normal hierarchical model pools them partially: each center is pulled toward the common mean according to its precision, and tau measures the real heterogeneity between centers.
-> "[eff]" "[-6.2, -3.1, -9.8, -1.0, -4.5, -12.4, -2.7, -5.6]"; -> "[se]" "[2.8, 2.1, 4.5, 3.0, 2.4, 5.1, 2.6, 3.3]"; json load "rx_h" (stat bayes_hierarchical [eff] [se] (concat "{\"names\": " "[\"Paris\", \"Lyon\", \"Lille\", \"Nantes\", \"Bordeaux\", \"Toulouse\", \"Rennes\", \"Saint-Denis\"]" ", \"tau_prior\": \"half_cauchy\"}")); -> "[res]" (concat "common effect mu " (math round (json select "rx_h" "/mu/mean") 2) " [" (math round (json select "rx_h" "/mu/lower") 2) ", " (math round (json select "rx_h" "/mu/upper") 2) "] | tau median " (math round (json select "rx_h" "/tau/median") 2) " | Toulouse " (json select "rx_h" "/groups/[5]/estimate") " -> " (math round (json select "rx_h" "/groups/[5]/posterior_mean") 2) " (pooling " (math round (json select "rx_h" "/groups/[5]/shrinkage") 2) ") | Lyon " (json select "rx_h" "/groups/[1]/estimate") " -> " (math round (json select "rx_h" "/groups/[1]/posterior_mean") 2) " (pooling " (math round (json select "rx_h" "/groups/[1]/shrinkage") 2) ")"); json unload "rx_h"; [res]
common effect mu -4.61 [-7.28, -2.21] | tau median 1.24 | Toulouse -12.4 -> -5.41 (pooling 0.89) | Lyon -3.1 -> -4.08 (pooling 0.69)
- The half-Cauchy prior on tau (scale 5) is the recommendation of Gelman (2006) when there are few groups; stat meta gives the classical random effects meta-analysis.
- To compare models fitted with ml mcmc, give their pointwise log-likelihood to stat waic (WAIC and PSIS-LOO).
Effect of a speed limit on road accidents: interrupted time series
Monthly road injuries on a network over 3 years; the speed limit dropped at month 18. The segmented regression separates the immediate level change from the change of trend, with Newey-West standard errors because the monthly counts are autocorrelated.
-> "[y]" "[51.7, 51.1, 49.1, 49.2, 52.5, 49.4, 49.9, 49.7, 50.0, 50.5, 49.5, 48.1, 49.9, 51.8, 49.7, 47.8, 54.2, 48.2, 42.7, 41.1, 42.6, 41.5, 38.5, 42.0, 40.4, 36.8, 36.8, 36.0, 39.5, 37.3, 36.7, 37.6, 37.9, 34.9, 34.9, 33.4]"; json load "rx_i" (stat its [y] 18); -> "[res]" (concat "baseline trend " (math round (json select "rx_i" "/baseline_slope") 2) " per month | level change " (math round (json select "rx_i" "/level_change/estimate") 2) " [" (math round (json select "rx_i" "/level_change/lower") 2) ", " (math round (json select "rx_i" "/level_change/upper") 2) "], p = " (math round (json select "rx_i" "/level_change/p_value") 4) " | slope change " (math round (json select "rx_i" "/slope_change/estimate") 2) " per month, p = " (math round (json select "rx_i" "/slope_change/p_value") 4) " | Durbin-Watson " (math round (json select "rx_i" "/durbin_watson") 2)); json unload "rx_i"; [res]
baseline trend -0.04 per month | level change -7.51 [-8.85, -6.16], p = 0 | slope change -0.42 per month, p = 0 | Durbin-Watson 2.43
- With a comparable network where nothing changed, stat did or stat synthetic_control (several donors) remove the common trends.
- Seasonality: add a seasonal adjustment before (ts decompose) for monthly data with strong seasons.
Opinion poll: raking on sex and age, weighted estimate and margin of error
An online poll of 120 people over-represents the men and the young. Raking on the census margins (sex and age) corrects the sample; the weighted share of "yes" changes, and the margin of error must use the design effect of the weights (the effective sample size is smaller than 120).
-> "[rows]" "[{\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"18-34\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"F\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"F\", \"age\": \"35-64\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 0}, {\"sex\": \"M\", \"age\": \"65+\", \"yes\": 1}]"; json load "rx_w" (survey rake [rows] "{\"sex\": {\"M\": 0.48, \"F\": 0.52}, \"age\": {\"18-34\": 0.27, \"35-64\": 0.47, \"65+\": 0.26}}"); #The answers in the order of the rows; json load "rx_r" [rows]; json load "rx_y" "[]"; for (-> "[i]" 0) (< [i] (json count "rx_r" "/")) (++ "[i]") { json iarray "rx_y" "/" (json select "rx_r" (concat "/[" [i] "]/yes")) NUM; }; json load "rx_e" (survey estimate (json doc "rx_y") (json select "rx_w" "/weights")); json load "rx_m" (survey margin (json select "rx_e" "/estimates/[0]/mean") 120 (concat "{\"deff\": " (json select "rx_w" "/design_effect_kish") "}")); -> "[res]" (concat "raking converged in " (json select "rx_w" "/iterations") " iterations, weights " (math round (json select "rx_w" "/min_weight") 2) " to " (math round (json select "rx_w" "/max_weight") 2) ", effective n " (math round (json select "rx_w" "/effective_n") 1) " | yes: weighted " (math round (* 100 (json select "rx_e" "/estimates/[0]/mean")) 1) " % (se " (math round (* 100 (json select "rx_e" "/estimates/[0]/se")) 1) ") | margin of error with the design effect: +/- " (math round (* 100 (json select "rx_m" "/margin_of_error")) 1) " points"); json unload "rx_w"; json unload "rx_r"; json unload "rx_y"; json unload "rx_e"; json unload "rx_m"; [res]
raking converged in 5 iterations, weights 0.62 to 2.01, effective n 101.5 | yes: weighted 36.9 % (se 4.8) | margin of error with the design effect: +/- 9.4 points
- survey sample_size {"deff": ...} plans the next wave with the design effect observed here.
- With strata or clusters (face to face polls by area), give them to survey estimate {"strata": [...], "clusters": [...]}.
