Profiles / Engineering & data
DS

Data scientist

Regression, correlation, feature scaling and drift tests.

22 recipes on this page · open in the interactive view

Recipes
  1. Sales forecast from ad spend
  2. Log transform and standardization
  3. Data drift between training and production
  4. Histogram of a feature with its normal fit
  5. Price of flats by multiple linear regression
  6. Activation functions of a neural network, side by side
  7. Least squares by QR decomposition, then ridge regression
  8. Glorot uniform initialisation of a dense layer
  9. Regression through the origin: heating energy of a building
  10. Multivariate Gaussian log-likelihood with an LU decomposition
  11. From SQL rows to a trained model
  12. Select the useful variables of a regression
  13. Customer segments on principal components
  14. Product recommendations with matrix factorization
  15. Explain a model and one of its predictions
  16. Market basket analysis with association rules
  17. Features that only work together
  18. Choose the features with the model itself
  19. A monthly report without a database
  20. Is the model still valid?
  21. Choose a model automatically and export it in PMML
  22. Customer messages: routing, sentiment and recurring topics
executed on MentDB

Sales forecast from ad spend

Weekly ad spend and sales are available for ten weeks. The script measures the correlation, fits a line and forecasts the sales for a new budget.

Pipeline
Script
admin
#1. Weekly ad spend in kEUR and sales in kEUR;
-> "[ad]" "[12, 18, 25, 31, 40, 46, 52, 60, 67, 75]";
-> "[sales]" "[210, 245, 290, 318, 372, 398, 441, 470, 512, 548]";
#2. Strength of the link: Pearson and Spearman;
-> "[r]" (stat correlation [ad] [sales]);
json load "rx_ds_sp" (stat spearman [ad] [sales]);
#3. Least squares line and forecast for a budget of 85 kEUR;
json load "rx_ds_fit" (stat fit [ad] [sales] "linear");
-> "[pred]" (math eval (json select "rx_ds_fit" /expression) "{\"x\": 85}");
concat "r = " (math round [r] 4) ", rho = " (math round (json select "rx_ds_sp" /rho) 3) ", sales = " (math round (json select "rx_ds_fit" /a) 1) " + " (math round (json select "rx_ds_fit" /b) 3) " * ad, R2 = " (math round (json select "rx_ds_fit" /r2) 4) ", forecast at 85 kEUR: " (math round [pred] 0) " kEUR"
Result
mentdb
r = 0.9989, rho = 1, sales = 151.5 + 5.374 * ad, R2 = 0.9978, forecast at 85 kEUR: 608 kEUR
Go further
  • Get the confidence interval of the slope with pa rl_slope_confidence_interval
  • Try a log model with stat fit "log" for diminishing returns
  • Fit several features with pa rm_load (multiple regression)
executed on MentDB

Log transform and standardization

Incomes are strongly right-skewed. The script applies a log transform, standardizes the values with z-scores and checks the skewness and the histogram.

Pipeline
Script
admin
#1. Customer incomes in kEUR, right-skewed;
-> "[inc]" "[18, 22, 25, 27, 29, 31, 34, 38, 42, 47, 55, 68, 85, 120, 190]";
json load "rx_ds_inc" [inc];
#2. Log transform of each value;
-> "[lg]" "["; -> "[sep]" "";
json parse_array "rx_ds_inc" "/" "[v]" {
  -> "[lg]" (concat [lg] [sep] (math calc "ln([v])"));
  -> "[sep]" ",";
};
-> "[lg]" (concat [lg] "]");
#3. Standardize and check the shape;
-> "[z]" (stat zscore [lg]);
concat "skewness raw = " (math round (stat skewness [inc]) 2) ", after log = " (math round (stat skewness [lg]) 2) ", z-scores from " (math round (stat min [z]) 2) " to " (math round (stat max [z]) 2) ", histogram: " (stat histogram [z] 4)
Result
mentdb
skewness raw = 2.13, after log = 0.89, z-scores from -1.36 to 2.23, histogram: [
  {"from": -1.3589523213190358, "to": -0.4625482543642111, "count": 6},
  {"from": -0.4625482543642111, "to": 0.43385581259061357, "count": 5},
  {"from": 0.43385581259061357, "to": 1.3302598795454381, "count": 2},
  {"from": 1.3302598795454381, "to": 2.226663946500263, "count": 2}
]
Go further
executed on MentDB

Data drift between training and production

A model feature is compared between the training set and last week production data. The script runs a two-sample Kolmogorov-Smirnov test and measures the shift in standard deviations.

Pipeline
Script
admin
#1. A feature in the training set and in last week production data;
-> "[train]" "[0.42, 0.51, 0.38, 0.47, 0.55, 0.44, 0.49, 0.40, 0.53, 0.46, 0.50, 0.43]";
-> "[prod]" "[0.58, 0.62, 0.49, 0.66, 0.71, 0.55, 0.60, 0.68, 0.57, 0.63, 0.59, 0.65]";
#2. Two-sample Kolmogorov-Smirnov test of the distributions;
json load "rx_ds_ks" (stat ks_2 [train] [prod]);
#3. Size of the shift in standard deviations of the training data;
-> "[shift]" (math calc (concat "(" (stat mean [prod]) "-" (stat mean [train]) ")/" (stat stddev [train])));
-> "[p]" (json select "rx_ds_ks" /p_value);
concat "KS D = " (math round (json select "rx_ds_ks" /d) 3) ", p = " (math round_sig [p] 3) ", mean shift = " (math round [shift] 2) " sd -> " (if (< [p] 0.01) {"data drift: retrain the model"} {"no drift"})
Result
mentdb
KS D = 0.833, p = 0.000204, mean shift = 2.76 sd -> data drift: retrain the model
Go further
executed on MentDB

Histogram of a feature with its normal fit

Before modelling, check the shape of a feature: the script draws the histogram of 1000 session times and overlays the normal density with the same mean and standard deviation, scaled to the counts of each class.

Pipeline
Script
admin
#1. A feature of 1000 users: session time in min, simulated normal with mean 50 and sd 8;
-> "[x]" (stat dist_random "normal" "[50, 8]" 1000);
-> "[m]" (stat mean [x]); -> "[s]" (stat stddev [x]);
#2. Class width of a 25 bin histogram;
-> "[w]" (math calc (concat "(" (stat max [x]) "-" (stat min [x]) ")/25"));
#3. Histogram, then overlay the normal density with the same mean and sd, scaled to counts;
plot histogram [x] 25;
plot add (plot function "1000*[w]*exp(-(x-[m])^2/(2*[s]^2))/([s]*sqrt(2*pi))" "x" 20 80)
Result
mentdb
The chart is shown in the Plot tab of the editor.
Go further
  • Test the normality with stat shapiro before using a z-score.
  • A skewed feature (lognormal) shows the gap at once; try stat dist_random "lognormal".
executed on MentDB

Price of flats by multiple linear regression

Ten flats were sold with their surface, number of rooms and age. Fit price = b0 + b1 surface + b2 rooms + b3 age by ordinary least squares, compute the t statistic of each coefficient, the adjusted R2 and the residual standard error, and estimate the price of a new flat.

Pipeline
Script
admin
#1. Sold flats: surface m2, rooms, age in years, and price in kEUR;
-> "[X]" "[[45, 2, 30], [62, 3, 12], [80, 4, 5], [38, 1, 40], [95, 4, 2], [70, 3, 20], [55, 2, 8], [110, 5, 15], [66, 3, 35], [84, 4, 10]]";
-> "[y]" "[152, 228, 310, 118, 372, 241, 205, 398, 205, 315]";
#2. Ordinary least squares: price = b0 + b1 surface + b2 rooms + b3 age;
pa rm load_from_json "rx_flats" [X] [y];
json load "rx_b" (pa rm estimate_regression_parameters "rx_flats");
json load "rx_se" (pa rm estimate_regression_parameters_standard_errors "rx_flats");
#3. t statistics of the coefficients;
-> "[out]" "";
json load "rx_lbl" "[\"intercept\", \"surface\", \"rooms\", \"age\"]";
for (-> "[k]" 0) (< [k] 4) (++ "[k]") {
  -> "[b]" (json select "rx_b" (concat "/[" [k] "]"));
  -> "[out]" (concat [out] (json select "rx_lbl" (concat "/[" [k] "]")) " " (math round [b] 3) " (t = " (math round (/ [b] (json select "rx_se" (concat "/[" [k] "]"))) 2) ") | ");
};
#4. Fit quality and price estimate of a 75 m2 flat, 3 rooms, 10 years old;
-> "[r2a]" (pa rm calculate_adjusted_r_squared "rx_flats");
-> "[s]" (pa rm estimate_regression_standard_error "rx_flats");
-> "[p]" (pa rm predict "rx_flats" "[75, 3, 10]");
pa rm close "rx_flats";
concat [out] "adjusted R2 = " (math round [r2a] 4) ", residual std error = " (math round [s] 1) " kEUR | 75 m2, 3 rooms, 10 years: " (math round [p] 1) " kEUR"
Result
mentdb
intercept 26.475 (t = 1.67) | surface 3.793 (t = 7.37) | rooms -4.71 (t = -0.49) | age -1.404 (t = -4.99) | adjusted R2 = 0.9912, residual std error = 8.6 kEUR | 75 m2, 3 rooms, 10 years: 282.8 kEUR
Go further
executed on MentDB

Activation functions of a neural network, side by side

Before choosing the activation of a hidden layer, the data scientist plots sigmoid, tanh, softplus and ELU on [-4, 4]. Softplus uses log1p(exp z) and ELU uses expm1(z), the numerically safe forms used by deep learning libraries.

Pipeline
Script
admin
#1. Grid of pre-activations, step 0.5;
-> "[x]" ""; -> "[sig]" ""; -> "[th]" ""; -> "[sp]" ""; -> "[elu]" "";
for (-> "[i]" -8) (<= [i] 8) (++ "[i]") {
  -> "[z]" (/ [i] 2);
  -> "[x]" (concat [x] "," [z]);
  -> "[sig]" (concat [sig] "," (/ 1 (+ 1 (math exp (- 0 [z])))));
  -> "[th]" (concat [th] "," (math tanh [z]));
  -> "[sp]" (concat [sp] "," (math log1p (math exp [z])));
  -> "[elu]" (concat [elu] "," (if (> [z] 0) {[z];} {math expm1 [z];}));
};
#2. One chart, four series;
plot chart (concat "{\"title\": \"Activation functions\", \"x_label\": \"z\", \"y_label\": \"f(z)\", \"series\": [{\"name\": \"sigmoid\", \"x\": [" (string substr [x] 1) "], \"y\": [" (string substr [sig] 1) "]}, {\"name\": \"tanh\", \"x\": [" (string substr [x] 1) "], \"y\": [" (string substr [th] 1) "]}, {\"name\": \"softplus\", \"x\": [" (string substr [x] 1) "], \"y\": [" (string substr [sp] 1) "]}, {\"name\": \"ELU\", \"x\": [" (string substr [x] 1) "], \"y\": [" (string substr [elu] 1) "]}]}")
Result
mentdb
The chart is shown in the Plot tab of the editor.
Go further
  • Add the derivatives to see where the gradients vanish.
  • Check the identity tanh(z) = 2 sigmoid(2z) - 1 with math calc.
executed on MentDB

Least squares by QR decomposition, then ridge regression

With a small, slightly noisy dataset, the script fits y = b0 + b1 x through the QR decomposition (the stable way used by statistical software) and then the ridge solution (X'X + lambda D)^-1 X'y, where the diagonal matrix D leaves the intercept unpenalised.

Pipeline
Script
admin
#1. Design matrix with a column of ones, and the target;
-> "[X]" "[[1, 1], [1, 2], [1, 3], [1, 4], [1, 5], [1, 6]]";
-> "[y]" "[2.3, 4.1, 6.2, 7.8, 10.3, 11.9]";
#2. OLS through QR: keep the first 2 rows of R and of Q'y;
json load "rx_p4g_qr" (matrix qr [X]);
-> "[qty]" (matrix mul (matrix transpose (json select "rx_p4g_qr" /Q)) [y]);
json load "rx_p4g_qty" [qty];
-> "[R1]" (concat "[" (json select "rx_p4g_qr" "/R[0]") ", " (json select "rx_p4g_qr" "/R[1]") "]");
-> "[ols]" (matrix solve [R1] (concat "[" (json select "rx_p4g_qty" "/[0]") ", " (json select "rx_p4g_qty" "/[1]") "]"));
json unload "rx_p4g_qr"; json unload "rx_p4g_qty";
#3. Ridge with lambda = 5 on the slope only;
-> "[Xt]" (matrix transpose [X]);
-> "[A]" (matrix add (matrix mul [Xt] [X]) (matrix diag "[0, 5]"));
-> "[ridge]" (matrix mul (matrix inverse [A]) (matrix mul [Xt] [y]));
concat "OLS by QR: " [ols] [_n_] "ridge lambda 5: " [ridge] [_n_] "check with matrix solve: " (matrix solve [X] [y])
Result
mentdb
OLS by QR: [0.2799999999999971,1.9485714285714293]
ridge lambda 5: [1.7955555555555591,1.5155555555555553]
check with matrix solve: [0.2800000000000079,1.9485714285714266]
Go further
  • Choose lambda by leave-one-out cross validation in a for loop.
  • Compare the condition numbers of X'X and X'X + lambda D with matrix cond.
executed on MentDB

Glorot uniform initialisation of a dense layer

A dense layer has 64 inputs and 32 outputs. The script draws the weight matrix uniformly in [-L, L] with L = sqrt(6/(fan_in + fan_out)) from a random matrix, and checks the bounds, the mean and the variance against the theoretical 2/(fan_in + fan_out).

Pipeline
Script
admin
#1. Glorot limit for 64 -> 32;
-> "[fin]" 64; -> "[fout]" 32;
-> "[L]" (math sqrt (/ 6 (+ [fin] [fout])));
#2. W = L . (2U - 1);
-> "[U]" (matrix random [fin] [fout]);
-> "[W]" (matrix scalar_mul (matrix sub (matrix scalar_mul [U] 2) (matrix ones [fin] [fout])) [L]);
#3. Flatten and check the draw;
-> "[w]" (concat "[" (string regexp_replace [W] "[\[\]]" "") "]");
-> "[var]" (stat pvariance [w]);
-> "[target]" (/ 2 (+ [fin] [fout]));
concat "shape " (matrix size [W]) ", limit L = " [L] [_n_] "inside [-L, L]: " (and (>= (stat min [w]) (- 0 [L])) (<= (stat max [w]) [L])) ", mean close to 0: " (< (math abs (stat mean [w])) 0.02) [_n_] "target variance = " (math round [target] 5) ", draw within 10 %: " (< (math abs (- [var] [target])) (* 0.1 [target]))
Result
mentdb
shape [64,32], limit L = 0.25
inside [-L, L]: 1, mean close to 0: 1
target variance = 0.02083, draw within 10 %: 1
Go further
executed on MentDB

Regression through the origin: heating energy of a building

Heating energy is physically zero with zero degree-days and zero occupancy, so the energy model is fitted without intercept: kWh = a . HDD + b . occupied hours. The script compares it with the model with intercept and forecasts a cold week.

Pipeline
Script
admin
#1. Weekly heating degree-days and occupied hours, energy in kWh;
-> "[x]" "[[45, 50], [62, 55], [80, 52], [95, 60], [110, 58], [70, 40], [55, 62], [88, 45]]";
-> "[y]" "[1510, 1980, 2390, 2840, 3190, 2050, 1850, 2550]";
#2. Model with intercept;
pa rm load_from_json "rx_p4g_e1" [x] [y];
-> "[p1]" (string regexp_replace (pa rm estimate_regression_parameters "rx_p4g_e1") "\s+" "");
#3. Model through the origin;
pa rm load_from_json "rx_p4g_e0" [x] [y];
pa rm set_no_intercept "rx_p4g_e0" true;
-> "[p0]" (string regexp_replace (pa rm estimate_regression_parameters "rx_p4g_e0") "\s+" "");
-> "[se]" (string regexp_replace (pa rm estimate_regression_parameters_standard_errors "rx_p4g_e0") "\s+" "");
-> "[f]" (pa rm predict "rx_p4g_e0" "[120, 55]");
-> "[r]" (concat "with intercept [b0,a,b]: " [p1] [_n_] "through origin [a,b]: " [p0] ", std errors " [se] [_n_] "forecast HDD 120, 55 h: " (math round [f] 0) " kWh");
pa rm close "rx_p4g_e1"; pa rm close "rx_p4g_e0";
[r]
Result
mentdb
with intercept [b0,a,b]: [-6.7634917036253785,24.941766035780343,7.877581710857582]
through origin [a,b]: [24.927410991925406,7.771949937735065], std errors [0.10516038164074848,0.1547319810680258]
forecast HDD 120, 55 h: 3419 kWh
Go further
  • Test whether the intercept is significant with its t ratio from the standard errors.
  • Add the solar gains as a third variable.
executed on MentDB

Multivariate Gaussian log-likelihood with an LU decomposition

An anomaly detector scores a new sensor vector with the log density of a 3D Gaussian fitted on normal data. The log-determinant of the covariance comes from the diagonal of U in the LU decomposition, the quadratic form from the inverse, and 2 pi from the constants.

Pipeline
Script
admin
#1. Mean and covariance of temperature, vibration, current;
-> "[mu]" "[60, 2.0, 12]";
-> "[S]" "[[4, 0.3, 1.2], [0.3, 0.09, 0.15], [1.2, 0.15, 1.0]]";
-> "[xv]" "[64.5, 2.6, 13.9]";
#2. log det Sigma from the diagonal of U;
json load "rx_p4g_lu" (matrix lu [S]);
-> "[ld]" 0;
for (-> "[i]" 0) (< [i] 3) (++ "[i]") {
  -> "[ld]" (+ [ld] (math log (math abs (json select "rx_p4g_lu" (concat "/U[" [i] "][" [i] "]")))));
};
json unload "rx_p4g_lu";
#3. Squared Mahalanobis distance;
-> "[d]" (vector sub [xv] [mu]);
-> "[m2]" (vector dot [d] (matrix mul (matrix inverse [S]) [d]));
#4. log pdf = -1/2 . m2 + log det + k ln 2pi;
-> "[tpi]" (string regexp_replace (constant math_two_pi) "^\((\S+) .*$" "$1");
-> "[lp]" (math calc "-0.5*([m2]+[ld]+3*ln([tpi]))");
concat "det from LU = " (math round (math exp [ld]) 6) ", matrix det = " (math round (matrix det [S]) 6) [_n_] "Mahalanobis^2 = " (math round [m2] 3) ", log pdf = " (math round [lp] 3) [_n_] "anomaly at 99 %: " (> [m2] (stat dist_inv "chi2" "[3]" 0.99))
Result
mentdb
det from LU = 0.1584, matrix det = 0.1584
Mahalanobis^2 = 6.254, log pdf = -4.963
anomaly at 99 %: 0
Go further
executed on MentDB

From SQL rows to a trained model

Turn rows as returned by sql parse into a matrix, fill the missing values with the median, standardize, train a k nearest neighbors classifier on the scaled data and classify a new customer.

Pipeline
Script
admin
#1. Rows from a table: age, income in kEUR, visits - segment;
-> "[rows]" "[{\"age\": 23, \"income\": 21, \"visits\": 12, \"seg\": \"young\"}, {\"age\": 27, \"income\": 25, \"visits\": null, \"seg\": \"young\"}, {\"age\": 25, \"income\": null, \"visits\": 15, \"seg\": \"young\"}, {\"age\": 45, \"income\": 62, \"visits\": 4, \"seg\": \"senior\"}, {\"age\": 52, \"income\": 70, \"visits\": 3, \"seg\": \"senior\"}, {\"age\": 48, \"income\": 58, \"visits\": 5, \"seg\": \"senior\"}, {\"age\": 35, \"income\": 40, \"visits\": 8, \"seg\": \"family\"}, {\"age\": 38, \"income\": 45, \"visits\": 9, \"seg\": \"family\"}, {\"age\": 33, \"income\": null, \"visits\": 7, \"seg\": \"family\"}]";
#2. Matrix, labels and missing values;
-> "[X]" (ml prep impute (ml prep matrix [rows] "[\"age\", \"income\", \"visits\"]") "median");
-> "[y]" (ml prep vector [rows] "seg");
#3. Standardize, then train on the scaled matrix;
ml fit "rx_sc" "scaler_zscore" [X];
ml train "rx_knn" "knn_classifier" (ml model transform "rx_sc" [X]) [y] "{\"k\": 3}";
#4. A new customer: 40 years, 50 kEUR, 6 visits;
json load "rx_x" [X];
-> "[res]" (concat "filled row 2: " (json select "rx_x" "/[2]") " | new customer: " (ml model predict "rx_knn" (ml model transform "rx_sc" "[[40, 50, 6]]")));
json unload "rx_x"; ml model delete "rx_sc"; ml model delete "rx_knn";
[res]
Result
mentdb
filled row 2: [
  25,
  45,
  15
] | new customer: ["family"]
Go further
  • With a database connection, build [rows] with sql parse and the same fields.
  • The knn_classifier parameter scale: true standardizes inside the model, without a scaler.
executed on MentDB

Select the useful variables of a regression

Score 4 candidate variables of a house price model with an F-test, remove the constant one, keep the 2 best and compare their R2 with the full model in cross validation.

Pipeline
Script
admin
#1. Houses: surface m2, rooms, distance to center km, constant flag - price kEUR;
-> "[X]" "[[50, 2, 8, 1], [65, 3, 6, 1], [80, 3, 7, 1], [95, 4, 3, 1], [110, 4, 5, 1], [120, 5, 2, 1], [70, 3, 9, 1], [140, 5, 4, 1], [55, 2, 2, 1], [100, 4, 8, 1]]";
-> "[y]" "[150, 190, 215, 290, 300, 360, 180, 380, 200, 250]";
json load "rx_v" (ml prep variance_threshold [X] 0);
json load "rx_k" (ml prep select_k_best [X] [y] 2 "f_regression");
#2. Cross validated R2: all the variables against the 2 best;
json load "rx_all" (ml cross_validate "linear" (ml prep columns [X] (json select "rx_v" "/selected")) [y] 5 "{}" 3);
json load "rx_best" (ml cross_validate "linear" (ml prep columns [X] (json select "rx_k" "/selected")) [y] 5 "{}" 3);
-> "[res]" (concat "non constant columns " (json select "rx_v" "/selected") " | best 2: " (json select "rx_k" "/selected") " | CV R2 all " (math round (json select "rx_all" "/mean") 3) " / best 2 " (math round (json select "rx_best" "/mean") 3));
json unload "rx_v"; json unload "rx_k"; json unload "rx_all"; json unload "rx_best";
[res]
Result
mentdb
non constant columns [
  0,
  1,
  2
] | best 2: [
  0,
  1
] | CV R2 all 0.983 / best 2 0.825
Go further
  • ml train lasso selects the variables inside the model (key non_zero).
  • For a classification, use the score f_classif, chi2 or mutual_info.
executed on MentDB

Customer segments on principal components

Reduce 4 correlated purchase indicators to the principal components explaining 90 % of the variance, choose the number of segments with the silhouette, then run a k-means on the components.

Pipeline
Script
admin
#1. Customers: baskets per month, mean basket EUR, share of promotions, online share;
-> "[X]" "[[2, 35, 0.6, 0.1], [3, 30, 0.7, 0.2], [2, 40, 0.5, 0.1], [8, 90, 0.1, 0.8], [9, 95, 0.2, 0.9], [7, 85, 0.1, 0.7], [5, 60, 0.3, 0.4], [4, 55, 0.4, 0.5], [5, 65, 0.3, 0.4], [3, 33, 0.6, 0.2]]";
#2. Components explaining 90 % of the variance;
json load "rx_pca" (ml fit "rx_pca" "pca" [X] "{\"n_components\": 0.9, \"standardize\": true}");
-> "[Z]" (ml model transform "rx_pca" [X]);
#3. Silhouette of each k, keep the best;
json load "rx_el" (ml elbow [Z] 5 42);
-> "[k]" 2; -> "[s]" -1;
for (-> "[i]" 1) (< [i] 5) (++ "[i]") {
  -> "[si]" (json select "rx_el" (concat "/[" [i] "]/silhouette"));
  if (> [si] [s]) {-> "[s]" [si]; -> "[k]" (+ [i] 1);};
};
json load "rx_km" (ml fit "rx_km" "kmeans" [Z] (concat "{\"k\": " [k] "}"));
-> "[res]" (concat (json count "rx_pca" "/explained_variance") " components, " (math round (* 100 (json select "rx_pca" (concat "/cumulative_ratio/[" (- (json count "rx_pca" "/explained_variance") 1) "]"))) 1) " % of the variance | k = " [k] ", silhouette " (math round [s] 3) " | sizes " (json select "rx_km" "/sizes") " | labels " (json select "rx_km" "/labels"));
json unload "rx_pca"; json unload "rx_el"; json unload "rx_km"; ml model delete "rx_pca"; ml model delete "rx_km";
[res]
Result
mentdb
1 components, 95.3 % of the variance | k = 3, silhouette 0.848 | sizes [
  4,
  3,
  3
] | labels [
  0,
  0,
  0,
  1,
  1,
  1,
  2,
  2,
  2,
  0
]
Go further
  • ml fit gmm gives soft segments with a probability per customer (ml model transform).
  • ml hclust with the ward linkage gives the dendrogram of the customers.
executed on MentDB

Product recommendations with matrix factorization

Learn the tastes of 6 customers from 18 ratings with an ALS factorization, recommend 2 products to a customer, predict one rating and list the products closest to a best seller.

Pipeline
Script
admin
#1. Ratings: customer, product, stars;
-> "[r]" "[[\"c1\", \"tea\", 5], [\"c1\", \"coffee\", 2], [\"c1\", \"matcha\", 5], [\"c2\", \"tea\", 4], [\"c2\", \"matcha\", 5], [\"c2\", \"cocoa\", 2], [\"c3\", \"coffee\", 5], [\"c3\", \"espresso\", 5], [\"c3\", \"tea\", 1], [\"c4\", \"coffee\", 4], [\"c4\", \"espresso\", 5], [\"c4\", \"cocoa\", 3], [\"c5\", \"cocoa\", 5], [\"c5\", \"tea\", 3], [\"c5\", \"matcha\", 4], [\"c6\", \"espresso\", 4], [\"c6\", \"coffee\", 5], [\"c6\", \"matcha\", 1]]";
json load "rx_fit" (ml recommend fit "rx_rec" [r] "{\"factors\": 3, \"lambda\": 0.05, \"seed\": 1}");
#2. Recommendations for c1 and for c4;
json load "rx_c1" (ml recommend items "rx_rec" "c1" 2);
json load "rx_sim" (ml recommend similar "rx_rec" "espresso" 1);
-> "[res]" (concat "train RMSE " (math round (json select "rx_fit" "/train_rmse") 3) " | for c1: " (json select "rx_c1" "/[0]/item") " " (math round (json select "rx_c1" "/[0]/score") 2) ", " (json select "rx_c1" "/[1]/item") " " (math round (json select "rx_c1" "/[1]/score") 2) " | c4 on matcha: " (math round (ml recommend rating "rx_rec" "c4" "matcha") 2) " | closest to espresso: " (json select "rx_sim" "/[0]/item"));
json unload "rx_fit"; json unload "rx_c1"; json unload "rx_sim"; ml model delete "rx_rec";
[res]
Result
mentdb
train RMSE 0.071 | for c1: espresso 3.72, cocoa 2.15 | c4 on matcha: 4.36 | closest to espresso: coffee
Go further
executed on MentDB

Explain a model and one of its predictions

Train a gradient boosting on house prices, rank the variables by permutation importance, read the partial dependence of the price on the surface, then explain the price predicted for one house.

Pipeline
Script
admin
#1. Houses: surface m2, rooms, distance to center km - price kEUR;
-> "[X]" "[[50, 2, 8], [65, 3, 6], [80, 3, 7], [95, 4, 3], [110, 4, 5], [120, 5, 2], [70, 3, 9], [140, 5, 4], [55, 2, 2], [100, 4, 8], [85, 3, 4], [60, 2, 5]]";
-> "[y]" "[150, 190, 215, 290, 300, 360, 180, 380, 200, 250, 260, 185]";
ml train "rx_gb" "gradient_boosting_regressor" [X] [y] "{\"n_estimators\": 100, \"max_depth\": 2, \"seed\": 1}";
json load "rx_pi" (ml model permutation_importance "rx_gb" [X] [y] 20 1);
json load "rx_pd" (ml model partial_dependence "rx_gb" [X] 0 3);
json load "rx_ex" (ml model explain "rx_gb" [X] "[[90, 3, 3]]");
-> "[res]" (concat "ranking " (json select "rx_pi" "/ranking") " | price vs surface " (math round (json select "rx_pd" "/average_prediction/[0]") 0) ", " (math round (json select "rx_pd" "/average_prediction/[1]") 0) ", " (math round (json select "rx_pd" "/average_prediction/[2]") 0) " kEUR | house 90 m2: " (math round (json select "rx_ex" "/prediction") 0) " kEUR, average " (math round (json select "rx_ex" "/baseline") 0) ", effects surface " (math round (json select "rx_ex" "/effects/[0]") 1) ", rooms " (math round (json select "rx_ex" "/effects/[1]") 1) ", distance " (math round (json select "rx_ex" "/effects/[2]") 1));
json unload "rx_pi"; json unload "rx_pd"; json unload "rx_ex"; ml model delete "rx_gb";
[res]
Result
mentdb
ranking [
  0,
  2,
  1
] | price vs surface 194, 273, 322 kEUR | house 90 m2: 263 kEUR, average 247, effects surface 14.7, rooms -9.1, distance 7
Go further
  • Compute the permutation importance on a test set from ml prep split to measure the generalization.
  • The partial dependence works on any model: compare a linear model and a forest.
executed on MentDB

Market basket analysis with association rules

Find the products bought together in 10 receipts: the frequent pairs, then the rules with a confidence of at least 70 % sorted by lift, to place the products or build bundles.

Pipeline
Script
admin
#1. Receipts;
-> "[t]" "[[\"coffee\", \"milk\", \"sugar\"], [\"coffee\", \"milk\"], [\"tea\", \"lemon\"], [\"coffee\", \"sugar\", \"cookies\"], [\"tea\", \"lemon\", \"honey\"], [\"coffee\", \"milk\", \"cookies\"], [\"tea\", \"honey\"], [\"coffee\", \"milk\", \"sugar\"], [\"tea\", \"lemon\", \"cookies\"], [\"coffee\", \"milk\", \"sugar\", \"cookies\"]]";
json load "rx_is" (ml assoc itemsets [t] 0.3 "frequent");
json load "rx_r" (ml assoc rules [t] 0.3 0.7);
#2. The 3 rules with the highest lift;
-> "[res]" (concat (json select "rx_is" "/count") " frequent itemsets, " (json select "rx_r" "/count") " rules | ");
for (-> "[i]" 0) (< [i] (math min 3 (json select "rx_r" "/count"))) (++ "[i]") {
  -> "[p]" (concat "/rules/[" [i] "]");
  -> "[res]" (concat [res] (json select "rx_r" (concat [p] "/if/[0]")) " -> " (json select "rx_r" (concat [p] "/then/[0]")) " conf " (math round (json select "rx_r" (concat [p] "/confidence")) 2) " lift " (math round (json select "rx_r" (concat [p] "/lift")) 2) "; ");
};
json unload "rx_is"; json unload "rx_r";
[res]
Result
mentdb
12 frequent itemsets, 10 rules | lemon -> tea conf 1 lift 2.5; tea -> lemon conf 0.75 lift 2.5; milk -> coffee conf 1 lift 1.67;
Go further
  • A lift close to 1 means independence: the rule only reflects popular products.
  • ml assoc itemsets with the type maximal gives the largest baskets in a few lines.
executed on MentDB

Features that only work together

The class depends on the sign of x0 * x1: each feature alone says nothing, so the univariate tests miss them. ReliefF and the recursive elimination with a random forest find them.

Pipeline
Script
admin
#1. 5 random features, the class is the sign of x0 * x1;
-> "[X]" "[[-0.67, 0.38, 0.27, -0.04, -0.57], [0.59, 0.62, 0.02, 0.01, -0.53], [-0.99, -0.26, 0.17, -0.86, 0.59], [-0.54, -0.53, -0.92, 1.0, 0.48], [0.75, 0.23, -0.93, -0.34, -0.0], [-0.77, 0.9, -0.26, -0.69, 0.64], [-0.75, 0.86, -0.09, 0.1, -0.31], [-0.03, -0.63, -0.93, 0.73, -0.53], [0.56, -0.58, 0.94, 0.79, 0.51], [0.53, 0.16, 0.46, -0.76, -0.05], [0.65, -0.6, 0.85, 0.78, -0.18], [-0.27, -0.1, -0.23, 0.99, -0.23], [-0.95, -0.81, 0.44, 0.99, -0.01], [-0.02, 0.1, 0.5, -0.56, 0.22], [0.11, 0.25, -0.64, -0.66, -0.3], [0.85, -0.31, 0.19, -0.39, -0.79], [-0.87, 0.63, 0.55, -0.25, 0.37], [0.35, 0.8, 0.92, -0.19, 0.98], [0.92, 0.18, 0.77, -0.02, -0.25], [-0.51, 0.75, -0.68, -0.86, 0.61], [0.55, -0.03, 0.67, -0.9, -0.48], [-0.42, -0.16, -0.66, -0.86, -0.32], [0.8, 0.97, 0.49, -0.04, -0.5], [0.92, 0.05, 0.46, -0.94, -0.58], [0.68, 0.59, 0.76, 0.2, -0.15], [0.16, 0.7, -0.38, 0.07, -0.45], [0.54, -0.98, 0.37, 0.91, -0.95], [0.19, 0.29, 0.7, 0.17, -0.49], [-0.19, 0.92, 0.81, -0.73, -0.81], [-0.81, -0.95, -0.11, -0.72, 0.73], [0.31, -0.75, 0.08, -0.35, -0.01], [-0.56, -0.67, -0.63, -0.24, 0.42], [0.1, -0.91, 0.42, 0.75, 0.55], [0.58, -0.98, -0.07, -0.87, 0.53], [-0.65, 0.42, -0.49, 0.39, -0.37], [-0.92, -0.22, -0.38, 0.8, -0.41], [0.01, 0.1, 0.9, -0.8, 0.2], [0.02, 0.47, 0.93, -0.38, 0.4], [0.05, 0.12, -0.87, 0.63, 0.95], [-0.29, -0.36, 0.99, -0.78, -0.85], [-0.85, 0.63, 0.52, -0.1, 0.24], [-0.96, 0.29, -0.11, 0.03, -0.22], [-0.2, -0.55, -0.9, -0.13, -0.51], [0.41, 0.63, 0.45, -0.09, -0.53], [-0.23, -0.03, 0.26, 0.93, -0.83], [-0.04, -0.02, 0.43, 0.78, 0.56], [0.25, 0.52, 0.34, 0.72, 0.4], [0.37, -0.51, 0.83, -0.23, 0.23], [0.65, 0.93, 0.98, -0.99, -0.37], [-0.38, -0.22, -0.71, 0.11, -0.84], [-0.39, 0.89, 0.48, -0.08, -0.78], [0.91, 0.41, -0.59, -0.33, 0.28], [0.36, -0.48, 0.9, -0.36, -0.49], [0.69, -0.69, 0.36, -0.6, 0.2], [-0.45, -0.24, 0.88, -0.07, -0.13], [0.23, -0.44, -0.65, -0.23, -0.46], [-0.62, 0.92, -0.93, 0.62, 0.87], [0.41, -0.96, 0.0, -0.92, 0.22], [0.4, -0.52, -0.98, 0.09, -0.81], [-0.38, -0.09, 0.79, -0.26, -0.73], [0.58, 0.64, 0.06, -0.9, -0.62], [-0.19, 0.1, -0.55, 0.6, 0.69], [0.85, 0.77, -0.96, 0.16, -0.84], [0.8, 0.49, 0.52, -0.62, -0.88], [0.63, 0.21, 0.24, -0.06, 0.6], [-0.32, -0.46, -0.29, -0.4, 0.25], [0.78, -0.86, -0.86, 0.02, 0.36], [-0.62, -0.46, -0.88, -0.93, -0.01], [0.36, 0.31, -0.68, -0.79, 0.86], [0.08, -0.18, 0.38, 0.46, 0.05], [-0.84, 0.65, -0.57, -0.92, -0.3], [0.07, 0.26, -0.65, 0.42, -0.01], [-0.68, 0.7, 0.61, 0.81, 0.36], [-0.57, 0.56, 0.81, 0.07, 0.82], [0.94, -0.22, 0.88, 0.87, 0.59], [-0.98, -0.3, 0.23, 0.21, 0.51], [0.29, -0.68, 0.86, 0.07, 0.76], [-0.59, 1.0, 0.11, 0.55, 0.48], [0.77, -0.28, -0.83, -0.19, -0.64], [0.08, 0.63, -0.28, -0.74, -0.95], [-0.15, 0.16, -0.95, 0.33, -0.58], [-0.79, 0.86, 0.05, 0.79, 0.03], [-0.75, -0.52, -0.15, 0.28, 0.54], [0.5, 0.3, -0.49, -0.14, -0.24], [-0.02, -0.47, 0.08, -0.08, -0.93], [0.65, 0.17, 0.61, 0.47, 0.46], [0.99, -0.25, -0.21, -0.99, 0.33], [0.16, 0.15, 1.0, -0.61, -0.55], [0.47, 0.62, -0.03, -0.89, -0.26], [0.65, -0.0, -0.08, -0.22, 0.26], [0.2, 0.39, 0.73, 0.5, -0.25], [-0.5, 0.56, 0.59, 0.86, -0.6], [0.7, 0.78, -0.85, -0.38, 0.27], [-0.71, 0.66, -0.47, 0.36, -0.41], [-0.56, -0.59, -0.0, 0.43, -0.56], [0.37, -0.33, 0.3, -0.26, -0.61], [-0.72, -0.88, 0.39, -0.17, -0.75], [0.76, 0.31, -0.96, 0.45, 0.19], [-0.11, 0.0, 0.92, -0.25, 0.27], [0.27, -0.27, -0.31, 0.51, 0.27]]";
-> "[y]" "[\"B\", \"A\", \"A\", \"A\", \"A\", \"B\", \"B\", \"A\", \"B\", \"A\", \"B\", \"A\", \"A\", \"B\", \"A\", \"B\", \"B\", \"A\", \"A\", \"B\", \"B\", \"A\", \"A\", \"A\", \"A\", \"A\", \"B\", \"A\", \"B\", \"A\", \"B\", \"A\", \"B\", \"B\", \"B\", \"A\", \"A\", \"A\", \"A\", \"A\", \"B\", \"B\", \"A\", \"A\", \"A\", \"A\", \"A\", \"B\", \"A\", \"A\", \"B\", \"A\", \"B\", \"B\", \"A\", \"B\", \"B\", \"B\", \"B\", \"A\", \"A\", \"B\", \"A\", \"A\", \"A\", \"A\", \"B\", \"A\", \"A\", \"B\", \"B\", \"A\", \"B\", \"B\", \"B\", \"A\", \"B\", \"B\", \"B\", \"A\", \"B\", \"B\", \"A\", \"A\", \"A\", \"A\", \"B\", \"A\", \"A\", \"B\", \"A\", \"B\", \"A\", \"B\", \"A\", \"B\", \"A\", \"A\", \"B\", \"B\"]";
json load "rx_k" (ml prep select_k_best [X] [y] 2 "f_classif");
json load "rx_r" (ml prep relieff [X] [y] "{\"n_features\": 2}");
json load "rx_e" (ml prep rfe [X] [y] "{\"algo\": \"random_forest_classifier\", \"params\": {\"n_trees\": 50}, \"n_features\": 2}");
-> "[res]" (concat "ANOVA keeps x" (json select "rx_k" "/selected/[0]") " and x" (json select "rx_k" "/selected/[1]") " | ReliefF keeps x" (json select "rx_r" "/selected/[0]") " and x" (json select "rx_r" "/selected/[1]") " | RFE keeps x" (json select "rx_e" "/selected/[0]") " and x" (json select "rx_e" "/selected/[1]"));
json load "rx_a" (ml cross_validate "knn_classifier" (ml prep columns [X] (json select "rx_r" "/selected")) [y] 5 "{}" 1);
-> "[res]" (concat [res] " | k-NN on the ReliefF features: " (math round (json select "rx_a" "/mean") 3));
json unload "rx_k"; json unload "rx_r"; json unload "rx_e"; json unload "rx_a";
[res]
Result
mentdb
ANOVA keeps x2 and x4 | ReliefF keeps x0 and x1 | RFE keeps x0 and x1 | k-NN on the ReliefF features: 0.96
Go further
  • ml prep cfs searches a subset of features correlated with the class but not with each other.
  • With {"cv": 5}, ml prep rfe cross validates every number of features and keeps the best.
executed on MentDB

Choose the features with the model itself

The wrapper method scores each subset of features by the cross validation of the final model: a forward search with k nearest neighbors keeps the 2 useful columns out of 6, and the score improves.

Pipeline
Script
admin
#1. 6 measures, the target depends on x0 and x3 only;
-> "[X]" "[[0.49, 0.04, -1.19, 0.95, 0.31, 0.37], [1.67, 0.99, 1.31, -0.28, -0.46, -1.06], [-2.86, 1.75, 0.17, 0.11, 0.94, -0.38], [0.85, -1.65, 1.87, 0.17, 0.68, -1.08], [1.83, -0.55, -0.42, 0.53, 1.55, -2.57], [0.31, 0.98, -1.07, 0.35, 1.75, -0.08], [-0.45, -0.54, 0.87, 0.23, -0.51, 0.87], [-1.52, 2.66, -0.66, 1.76, -0.07, -0.22], [-0.9, -0.88, -0.2, -0.39, 0.79, 2.11], [-0.92, 0.83, 0.57, 1.22, -0.77, 0.66], [-1.72, 0.5, -1.66, 0.6, -0.46, -0.19], [1.66, 0.48, 0.45, -0.81, 1.36, 0.69], [-1.76, 0.02, -0.17, 0.95, -1.08, -0.93], [0.12, 1.04, 1.22, -0.94, -2.52, 0.08], [0.28, -2.89, -0.09, 0.12, -0.58, 0.58], [0.27, 1.42, 1.41, 1.31, 0.43, 0.18], [0.36, -0.76, 0.44, 1.01, -0.41, -1.15], [-0.78, -1.57, -0.24, -1.54, 0.38, -0.52], [1.25, 1.22, -0.08, -2.39, -0.48, -1.22], [1.69, 0.77, 0.71, 0.62, 0.76, -0.37], [-0.3, -1.21, 0.08, 1.74, -0.18, -1.87], [0.28, 0.61, -0.47, 0.54, 1.75, -0.66], [0.41, -0.9, 0.22, 0.66, -0.87, 1.28], [1.26, 0.69, 0.3, -0.81, -0.81, -2.34], [-0.11, 0.62, 1.12, 0.24, -0.33, 1.08], [0.0, -1.22, 1.15, -0.69, 2.1, -0.89], [0.12, -0.72, -1.53, -1.19, -0.85, -0.22], [0.74, 0.37, 0.89, -2.3, -0.63, -0.56], [0.54, -0.67, -1.14, 0.98, -0.67, -0.24], [0.99, 0.08, 1.56, 2.37, 0.74, -0.36], [1.77, -0.19, 0.7, 0.29, -3.55, 0.19], [0.24, -0.12, 0.28, -1.05, 0.67, -0.51], [1.78, 0.26, -1.06, 0.42, -1.51, 0.21], [-1.09, -0.13, -0.9, 0.21, 0.38, 0.22], [-1.99, -0.92, -0.44, 0.89, -2.09, 1.18], [-1.75, 1.57, 0.39, -0.97, 0.31, 0.66], [1.39, -0.87, 0.89, -0.57, -1.64, -0.61], [0.16, -1.74, -1.04, 0.36, -0.38, -0.53], [-1.37, 2.13, -0.76, 0.41, -0.15, 1.07], [-0.8, 0.28, 0.66, -0.14, 2.17, 0.81], [1.37, 1.1, -0.17, 1.73, -0.37, 0.17], [1.0, -1.11, -0.3, -0.92, 1.23, -0.36], [-0.7, -0.51, 0.72, -1.04, 0.08, -0.5], [-1.32, 1.23, -1.67, -1.27, -0.43, 0.08], [-1.69, -0.49, -1.04, -0.7, 1.68, -2.14], [0.39, -0.43, -0.37, 1.11, -0.22, -1.8], [0.38, -0.4, -0.6, 0.18, -0.52, 0.13], [-0.65, -1.64, 1.37, 0.4, -0.81, -1.48], [0.21, 0.5, 0.37, -0.46, -0.43, -1.58], [-0.5, 1.1, -1.37, 1.92, 0.92, -0.54], [0.22, 0.4, 0.73, -1.8, -0.36, 0.51], [0.46, -0.1, 0.31, -0.55, 1.77, 0.93], [-0.59, -0.05, 0.38, 0.09, 1.68, 2.13], [-1.19, 0.72, -0.52, 0.01, 2.02, -1.78], [-0.65, 0.8, -0.42, -2.03, 0.4, 0.67], [0.11, 1.15, -0.6, 0.91, 1.23, -0.28], [1.26, -0.75, -0.92, 0.05, 0.6, 0.95], [0.54, -0.89, -1.17, 2.52, -0.74, -1.55], [-0.75, 0.75, -2.57, -1.15, 1.08, 0.29], [1.55, 1.08, -0.35, 1.18, 0.65, -1.29], [0.19, -1.1, -1.63, -0.52, 0.73, 0.13], [-2.55, 0.76, -0.28, 0.8, -0.14, -0.56], [-0.8, -0.6, -1.0, -2.3, 1.02, 1.05], [-0.48, 2.31, 0.52, -0.47, -1.32, 1.72], [-0.9, 0.66, -1.02, -0.38, 1.08, -0.32], [-1.55, 0.56, -0.25, -1.31, 2.07, -0.76], [0.37, -0.34, -0.01, 0.38, -0.55, -1.01], [1.01, -0.03, 1.57, 0.06, -0.98, 0.21], [-0.53, -0.23, 0.95, 0.86, 0.34, 0.2], [0.73, 0.4, -0.72, 1.23, 1.19, 1.36], [0.57, 0.18, -2.82, 1.25, -2.01, -0.51], [1.11, -0.28, -0.66, -1.85, -0.26, 0.62], [1.58, -2.92, -0.6, -0.46, 0.06, 0.34], [-0.64, -1.62, -1.22, -0.36, -0.93, -0.36], [-0.02, 0.92, 0.41, -1.41, -0.8, -0.61], [0.79, 0.9, 0.29, 1.55, -0.99, 0.01], [-0.02, -0.67, -0.82, 1.04, 0.84, 2.03], [0.25, -0.49, -0.88, -0.48, 1.96, -0.16], [-0.18, 0.4, -0.72, -0.76, -0.64, 1.42], [0.55, 0.59, -2.0, 0.45, -1.2, -0.41], [0.82, 0.2, -1.0, 1.28, -0.87, 1.46], [2.42, -1.23, 0.53, -1.28, -1.34, -0.64], [-1.2, 0.47, 1.69, 0.83, -0.79, 2.11], [1.77, -0.18, -0.98, -0.01, -0.02, 1.14], [-0.36, 0.96, -0.31, -1.09, -1.25, -1.07], [-0.41, 0.78, -1.45, 0.06, 0.41, 0.15], [-2.95, -1.4, -0.8, 0.07, 1.29, 2.36], [-0.21, 0.88, -0.55, 1.19, -0.11, -0.01], [1.82, 0.19, -0.76, 1.43, 0.55, -0.85], [-0.11, -0.59, -0.74, 0.26, 0.78, 1.41], [-1.09, -1.33, -1.51, 0.6, -1.82, -0.71], [-0.47, -0.55, -0.4, 1.64, 0.86, -0.42], [0.54, -0.33, -0.13, -1.18, 0.41, 0.32], [0.1, 0.19, -1.12, 1.12, 0.34, 1.73], [-0.36, 1.09, -0.49, 0.54, -0.12, -0.18], [0.38, 0.06, -1.46, 1.21, -2.15, -0.31], [0.92, -0.72, 0.81, 0.93, -3.34, 0.07], [1.59, 0.08, -1.35, -0.46, 2.13, -0.06], [0.94, -0.79, 0.76, 0.26, 0.59, 0.26], [1.03, 0.1, -0.29, -0.02, -0.48, -0.03], [-1.42, 1.8, 0.23, -0.23, -1.69, 1.43], [0.99, -0.06, 0.84, -0.55, 0.04, 1.54], [0.11, -1.34, 0.35, 0.4, -1.0, -0.95], [0.17, -0.56, 0.56, -1.46, -1.67, -0.7], [1.07, 0.34, 1.9, -1.43, -1.17, 1.61], [-0.03, 0.5, -0.64, 2.1, -0.47, 1.01], [-0.61, -1.44, -0.28, -0.16, -0.9, -1.56], [-1.68, 1.39, -0.33, -0.74, 0.93, 0.29], [1.07, -0.14, -2.05, 0.43, 0.09, -0.29], [0.65, 0.14, 1.7, -1.42, -1.33, 0.43], [-0.41, 1.13, -0.03, 1.2, -0.2, 0.29], [-1.11, 0.98, -0.71, 0.87, -1.06, -2.37], [-1.77, 2.11, -2.07, -1.29, 1.04, 0.69], [1.69, 1.68, -1.05, -0.3, 0.25, 0.67], [0.62, -1.19, -0.18, -1.56, 0.08, 0.98], [1.04, -1.3, -2.76, -0.9, -0.22, 0.79], [-1.26, -0.86, 0.65, 0.12, -1.15, -1.33], [-0.19, -0.15, -1.08, -1.28, -0.82, -1.14], [0.2, -0.04, -3.89, -1.28, 0.54, -0.53], [-2.17, -1.06, 0.68, -0.96, 3.3, 0.62]]";
-> "[y]" "[0.13, 4.98, -8.97, 2.03, 6.2, 0.01, -2.49, -10.46, -2.19, -6.34, -6.23, 4.8, -8.62, 1.75, 0.7, -1.14, -0.25, 2.0, 6.03, 5.37, -4.23, 0.29, -0.4, 4.38, -0.83, 1.14, 3.14, 4.58, -0.6, 0.17, 5.34, 2.67, 6.01, -4.05, -9.91, -2.36, 4.26, 0.06, -5.71, -1.28, 3.76, 3.94, 0.53, -0.13, -2.5, -0.81, 1.18, -3.01, 1.07, -6.61, 4.33, 2.43, -1.5, -4.3, 4.07, -1.58, 3.5, -1.89, 0.45, 3.72, 0.98, -11.2, 3.72, 0.37, -1.98, -0.3, 0.31, 2.86, -3.44, -0.89, 0.01, 4.27, 5.5, -1.08, 2.49, 1.08, -1.14, 1.26, 1.11, 0.02, 0.89, 6.59, -7.11, 5.65, 1.61, -1.17, -9.55, -2.94, 4.4, -0.79, -5.83, -6.02, 3.66, -1.98, -3.48, 0.38, 1.78, 5.95, 2.14, 3.04, -3.23, 4.65, 0.86, 3.67, 4.96, -4.45, -2.09, -1.99, 2.29, 4.11, -3.83, -6.0, -0.82, 5.11, 3.43, 4.35, -3.92, 2.61, 3.13, -1.79]";
json load "rx_w" (ml prep wrapper [X] [y] "{\"algo\": \"knn_regressor\", \"params\": {\"k\": 5}}");
json load "rx_a" (ml cross_validate "knn_regressor" [X] [y] 5 "{\"k\": 5}" 42);
-> "[res]" (concat "kept x" (json select "rx_w" "/selected/[0]") " and x" (json select "rx_w" "/selected/[1]") " in " (json count "rx_w" "/steps") " steps | R2 " (math round (json select "rx_w" "/score") 3) " instead of " (math round (json select "rx_a" "/mean") 3) " with the 6 columns");
json unload "rx_w"; json unload "rx_a";
[res]
Result
mentdb
kept x0 and x3 in 2 steps | R2 0.952 instead of 0.733 with the 6 columns
Go further
  • {"direction": "backward"} starts from all the columns and removes them one by one.
  • The wrapper is slow on many columns: filter first with ml prep relieff or ml prep cfs.
executed on MentDB

A monthly report without a database

From 18 monthly sales rows, build a pivot table shop x month, the running total and the month-over-month change of each shop with window functions, and the best month of each shop.

Pipeline
Script
admin
#1. Sales by shop and month;
-> "[s]" "[{\"shop\": \"Paris\", \"month\": 1, \"amount\": 990.0}, {\"shop\": \"Lyon\", \"month\": 1, \"amount\": 1286.0}, {\"shop\": \"Nice\", \"month\": 1, \"amount\": 1573.0}, {\"shop\": \"Paris\", \"month\": 2, \"amount\": 1087.0}, {\"shop\": \"Lyon\", \"month\": 2, \"amount\": 1536.0}, {\"shop\": \"Nice\", \"month\": 2, \"amount\": 995.0}, {\"shop\": \"Paris\", \"month\": 3, \"amount\": 1380.0}, {\"shop\": \"Lyon\", \"month\": 3, \"amount\": 1641.0}, {\"shop\": \"Nice\", \"month\": 3, \"amount\": 1161.0}, {\"shop\": \"Paris\", \"month\": 4, \"amount\": 1686.0}, {\"shop\": \"Lyon\", \"month\": 4, \"amount\": 1024.0}, {\"shop\": \"Nice\", \"month\": 4, \"amount\": 1683.0}, {\"shop\": \"Paris\", \"month\": 5, \"amount\": 1550.0}, {\"shop\": \"Lyon\", \"month\": 5, \"amount\": 1003.0}, {\"shop\": \"Nice\", \"month\": 5, \"amount\": 1127.0}, {\"shop\": \"Paris\", \"month\": 6, \"amount\": 1606.0}, {\"shop\": \"Lyon\", \"month\": 6, \"amount\": 1755.0}, {\"shop\": \"Nice\", \"month\": 6, \"amount\": 1548.0}]";
json load "rx_p" (table pivot [s] "shop" "month" "[\"sum\", \"amount\"]");
json load "rx_w" (table window [s] "{\"partition\": [\"shop\"], \"order\": [\"month\"], \"functions\": {\"running\": [\"cumsum\", \"amount\"], \"change\": [\"pct_change\", \"amount\"]}}");
json load "rx_t" (table top [s] "[\"shop\"]" "[[\"amount\", \"desc\"]]" 1);
-> "[res]" (concat "Paris: total " (json select "rx_p" "/[0]/total") ", month 6: " (json select "rx_p" "/[0]/6") " | Paris running total at month 6: " (json select "rx_w" "/[5]/running") ", change " (math round (* 100 (json select "rx_w" "/[5]/change")) 1) " % | best months: ");
for (-> "[i]" 0) (< [i] (json count "rx_t" /)) (++ "[i]") {-> "[res]" (concat [res] (json select "rx_t" (concat "/[" [i] "]/shop")) " m" (json select "rx_t" (concat "/[" [i] "]/month")) " ");};
json unload "rx_p"; json unload "rx_w"; json unload "rx_t";
[res]
Result
mentdb
Paris: total 8299, month 6: 1606 | Paris running total at month 6: 8299, change 3.6 % | best months: Paris m4 Lyon m6 Nice m4
Go further
executed on MentDB

Is the model still valid?

A quality model was trained on last year production data: compare the data of this month with the training data, field by field (PSI, Kolmogorov-Smirnov, chi-square, new categories) to decide if the model must be retrained.

Pipeline
Script
admin
#1. Training data and current data;
-> "[ref]" "[{\"temp\": 71.9, \"speed\": 1568, \"material\": \"steel\"}, {\"temp\": 74.8, \"speed\": 1463, \"material\": \"steel\"}, {\"temp\": 66.6, \"speed\": 1477, \"material\": \"steel\"}, {\"temp\": 74.9, \"speed\": 1507, \"material\": \"steel\"}, {\"temp\": 69.2, \"speed\": 1511, \"material\": \"steel\"}, {\"temp\": 68.6, \"speed\": 1503, \"material\": \"steel\"}, {\"temp\": 70.1, \"speed\": 1534, \"material\": \"steel\"}, {\"temp\": 70.3, \"speed\": 1451, \"material\": \"steel\"}, {\"temp\": 71.4, \"speed\": 1478, \"material\": \"alu\"}, {\"temp\": 73.6, \"speed\": 1550, \"material\": \"steel\"}, {\"temp\": 70.8, \"speed\": 1509, \"material\": \"alu\"}, {\"temp\": 69.1, \"speed\": 1571, \"material\": \"steel\"}, {\"temp\": 69.9, \"speed\": 1456, \"material\": \"steel\"}, {\"temp\": 72.8, \"speed\": 1580, \"material\": \"steel\"}, {\"temp\": 68.0, \"speed\": 1499, \"material\": \"steel\"}, {\"temp\": 67.9, \"speed\": 1413, \"material\": \"steel\"}, {\"temp\": 75.3, \"speed\": 1586, \"material\": \"steel\"}, {\"temp\": 66.0, \"speed\": 1502, \"material\": \"alu\"}, {\"temp\": 65.1, \"speed\": 1512, \"material\": \"steel\"}, {\"temp\": 67.0, \"speed\": 1485, \"material\": \"steel\"}, {\"temp\": 70.7, \"speed\": 1492, \"material\": \"alu\"}, {\"temp\": 76.0, \"speed\": 1506, \"material\": \"alu\"}, {\"temp\": 68.3, \"speed\": 1456, \"material\": \"alu\"}, {\"temp\": 70.2, \"speed\": 1542, \"material\": \"steel\"}, {\"temp\": 70.6, \"speed\": 1458, \"material\": \"alu\"}, {\"temp\": 69.6, \"speed\": 1463, \"material\": \"steel\"}, {\"temp\": 69.5, \"speed\": 1510, \"material\": \"steel\"}, {\"temp\": 70.1, \"speed\": 1526, \"material\": \"steel\"}, {\"temp\": 73.2, \"speed\": 1525, \"material\": \"alu\"}, {\"temp\": 74.6, \"speed\": 1498, \"material\": \"alu\"}, {\"temp\": 75.0, \"speed\": 1483, \"material\": \"steel\"}, {\"temp\": 72.8, \"speed\": 1465, \"material\": \"steel\"}, {\"temp\": 66.5, \"speed\": 1514, \"material\": \"steel\"}, {\"temp\": 63.4, \"speed\": 1479, \"material\": \"steel\"}, {\"temp\": 73.0, \"speed\": 1529, \"material\": \"steel\"}, {\"temp\": 72.5, \"speed\": 1481, \"material\": \"alu\"}, {\"temp\": 73.5, \"speed\": 1528, \"material\": \"steel\"}, {\"temp\": 64.0, \"speed\": 1498, \"material\": \"steel\"}, {\"temp\": 71.2, \"speed\": 1556, \"material\": \"steel\"}, {\"temp\": 71.8, \"speed\": 1491, \"material\": \"steel\"}, {\"temp\": 70.5, \"speed\": 1514, \"material\": \"steel\"}, {\"temp\": 70.2, \"speed\": 1438, \"material\": \"steel\"}, {\"temp\": 72.5, \"speed\": 1581, \"material\": \"alu\"}, {\"temp\": 72.3, \"speed\": 1500, \"material\": \"steel\"}, {\"temp\": 68.7, \"speed\": 1418, \"material\": \"steel\"}, {\"temp\": 70.5, \"speed\": 1455, \"material\": \"steel\"}, {\"temp\": 75.7, \"speed\": 1560, \"material\": \"alu\"}, {\"temp\": 69.4, \"speed\": 1520, \"material\": \"steel\"}, {\"temp\": 70.2, \"speed\": 1478, \"material\": \"steel\"}, {\"temp\": 66.6, \"speed\": 1440, \"material\": \"steel\"}, {\"temp\": 64.7, \"speed\": 1448, \"material\": \"steel\"}, {\"temp\": 71.7, \"speed\": 1426, \"material\": \"steel\"}, {\"temp\": 70.6, \"speed\": 1478, \"material\": \"steel\"}, {\"temp\": 67.0, \"speed\": 1505, \"material\": \"steel\"}, {\"temp\": 62.6, \"speed\": 1487, \"material\": \"alu\"}, {\"temp\": 63.4, \"speed\": 1464, \"material\": \"alu\"}, {\"temp\": 64.5, \"speed\": 1485, \"material\": \"steel\"}, {\"temp\": 72.0, \"speed\": 1482, \"material\": \"steel\"}, {\"temp\": 65.0, \"speed\": 1566, \"material\": \"steel\"}, {\"temp\": 71.1, \"speed\": 1459, \"material\": \"steel\"}]";
-> "[cur]" "[{\"temp\": 80.0, \"speed\": 1493, \"material\": \"titanium\"}, {\"temp\": 70.6, \"speed\": 1503, \"material\": \"alu\"}, {\"temp\": 76.6, \"speed\": 1579, \"material\": \"titanium\"}, {\"temp\": 74.3, \"speed\": 1529, \"material\": \"steel\"}, {\"temp\": 71.5, \"speed\": 1542, \"material\": \"alu\"}, {\"temp\": 74.4, \"speed\": 1445, \"material\": \"alu\"}, {\"temp\": 74.9, \"speed\": 1489, \"material\": \"steel\"}, {\"temp\": 78.4, \"speed\": 1466, \"material\": \"titanium\"}, {\"temp\": 72.3, \"speed\": 1543, \"material\": \"titanium\"}, {\"temp\": 73.9, \"speed\": 1610, \"material\": \"titanium\"}, {\"temp\": 73.9, \"speed\": 1493, \"material\": \"steel\"}, {\"temp\": 70.9, \"speed\": 1478, \"material\": \"alu\"}, {\"temp\": 76.6, \"speed\": 1494, \"material\": \"titanium\"}, {\"temp\": 70.9, \"speed\": 1506, \"material\": \"alu\"}, {\"temp\": 72.6, \"speed\": 1472, \"material\": \"alu\"}, {\"temp\": 71.9, \"speed\": 1500, \"material\": \"steel\"}, {\"temp\": 75.5, \"speed\": 1566, \"material\": \"alu\"}, {\"temp\": 78.2, \"speed\": 1531, \"material\": \"alu\"}, {\"temp\": 73.2, \"speed\": 1532, \"material\": \"steel\"}, {\"temp\": 74.3, \"speed\": 1550, \"material\": \"alu\"}, {\"temp\": 73.7, \"speed\": 1495, \"material\": \"titanium\"}, {\"temp\": 74.3, \"speed\": 1551, \"material\": \"alu\"}, {\"temp\": 76.6, \"speed\": 1433, \"material\": \"titanium\"}, {\"temp\": 71.1, \"speed\": 1470, \"material\": \"titanium\"}, {\"temp\": 74.3, \"speed\": 1530, \"material\": \"alu\"}, {\"temp\": 76.1, \"speed\": 1527, \"material\": \"steel\"}, {\"temp\": 72.9, \"speed\": 1494, \"material\": \"alu\"}, {\"temp\": 76.0, \"speed\": 1537, \"material\": \"alu\"}, {\"temp\": 71.1, \"speed\": 1488, \"material\": \"alu\"}, {\"temp\": 76.9, \"speed\": 1589, \"material\": \"alu\"}, {\"temp\": 75.9, \"speed\": 1594, \"material\": \"alu\"}, {\"temp\": 74.0, \"speed\": 1479, \"material\": \"alu\"}, {\"temp\": 74.5, \"speed\": 1493, \"material\": \"steel\"}, {\"temp\": 75.5, \"speed\": 1491, \"material\": \"steel\"}, {\"temp\": 74.4, \"speed\": 1481, \"material\": \"alu\"}, {\"temp\": 73.5, \"speed\": 1496, \"material\": \"alu\"}, {\"temp\": 69.7, \"speed\": 1504, \"material\": \"alu\"}, {\"temp\": 75.7, \"speed\": 1411, \"material\": \"steel\"}, {\"temp\": 74.2, \"speed\": 1523, \"material\": \"alu\"}, {\"temp\": 79.5, \"speed\": 1542, \"material\": \"alu\"}, {\"temp\": 69.9, \"speed\": 1573, \"material\": \"alu\"}, {\"temp\": 75.9, \"speed\": 1542, \"material\": \"steel\"}, {\"temp\": 71.3, \"speed\": 1492, \"material\": \"titanium\"}, {\"temp\": 73.5, \"speed\": 1479, \"material\": \"steel\"}, {\"temp\": 75.1, \"speed\": 1483, \"material\": \"alu\"}, {\"temp\": 80.9, \"speed\": 1510, \"material\": \"alu\"}, {\"temp\": 73.5, \"speed\": 1426, \"material\": \"titanium\"}, {\"temp\": 76.0, \"speed\": 1498, \"material\": \"titanium\"}, {\"temp\": 74.8, \"speed\": 1549, \"material\": \"titanium\"}, {\"temp\": 74.4, \"speed\": 1546, \"material\": \"steel\"}, {\"temp\": 70.8, \"speed\": 1580, \"material\": \"alu\"}, {\"temp\": 75.8, \"speed\": 1510, \"material\": \"alu\"}, {\"temp\": 74.2, \"speed\": 1512, \"material\": \"titanium\"}, {\"temp\": 76.0, \"speed\": 1483, \"material\": \"alu\"}, {\"temp\": 76.1, \"speed\": 1469, \"material\": \"titanium\"}, {\"temp\": 78.1, \"speed\": 1511, \"material\": \"titanium\"}, {\"temp\": 72.1, \"speed\": 1477, \"material\": \"alu\"}, {\"temp\": 77.6, \"speed\": 1503, \"material\": \"alu\"}, {\"temp\": 72.7, \"speed\": 1442, \"material\": \"titanium\"}, {\"temp\": 70.3, \"speed\": 1512, \"material\": \"alu\"}]";
json load "rx_d" (drift compare [ref] [cur] "{\"bins\": 5}");
-> "[res]" (concat (json select "rx_d" "/drifted_fields") " fields drifted | ");
for (-> "[i]" 0) (< [i] (json count "rx_d" "/fields")) (++ "[i]") {
  -> "[p]" (concat "/fields/[" [i] "]");
  -> "[res]" (concat [res] (json select "rx_d" (concat [p] "/field")) ": PSI " (json select "rx_d" (concat [p] "/psi")) " " (json select "rx_d" (concat [p] "/drift")) (if (equal (json select "rx_d" (concat [p] "/type")) "numeric") {concat ", KS p=" (math round (json select "rx_d" (concat [p] "/ks_p_value")) 4)} {concat ", new: " (json select "rx_d" (concat [p] "/new_categories/[0]"))}) " | ");
};
json unload "rx_d";
[res]
Result
mentdb
3 fields drifted | temp: PSI 1.8731 high, KS p=0 | speed: PSI 0.2735 high, KS p=0.1821 | material: PSI 1.9235 high, new: titanium |
Go further
  • PSI below 0.1: stable; 0.1 to 0.25: watch; above 0.25: retrain the model.
  • Run it in a job every month and send an alert when a field drifts.
executed on MentDB

Choose a model automatically and export it in PMML

A production line records the temperature, the pressure and the speed of each batch, and whether the batch was ok or scrapped. AutoML compares a dozen algorithms by cross validation and keeps the best one; the model is then exported in PMML to be scored by the MES or by another tool.

Pipeline
Script
admin
#1. Batches: temperature, pressure, speed and result;
-> "[X]" "[[198.1, 3.12, 96], [203.5, 2.37, 76], [205.2, 3.59, 55], [212.4, 3.39, 52], [219.3, 3.93, 83], [180.6, 3.06, 53], [187.6, 2.48, 52], [213.7, 3.04, 82], [200.0, 3.32, 73], [219.8, 3.68, 85], [192.6, 2.46, 64], [196.0, 3.69, 69], [218.3, 3.69, 50], [198.8, 3.96, 70], [182.9, 3.26, 89], [193.3, 3.93, 88], [184.7, 2.49, 55], [187.1, 3.12, 72], [187.6, 3.46, 57], [196.8, 2.43, 63], [218.8, 3.61, 65], [195.8, 3.71, 82], [184.0, 3.98, 61], [193.2, 2.59, 54], [183.6, 3.17, 62], [198.1, 3.92, 74], [203.0, 3.73, 59], [212.7, 2.5, 59], [209.6, 3.88, 60], [204.1, 2.84, 55], [181.5, 3.93, 62], [212.9, 3.19, 65], [187.0, 3.44, 53], [214.1, 3.23, 64], [216.7, 2.41, 51], [182.4, 2.35, 68], [202.9, 2.26, 68], [206.3, 3.38, 79], [185.6, 2.07, 51], [218.5, 2.04, 82], [199.3, 3.46, 66], [201.8, 3.47, 95], [209.5, 3.41, 90], [207.4, 3.8, 94], [196.7, 3.58, 93], [195.3, 3.17, 80], [183.2, 3.28, 100], [195.5, 3.47, 79], [197.6, 3.68, 54], [204.1, 2.96, 62], [207.9, 2.99, 81], [180.5, 2.6, 84], [188.1, 2.34, 95], [215.7, 2.65, 83], [187.9, 2.86, 90], [195.4, 3.17, 66], [185.4, 2.99, 92], [218.0, 2.55, 58], [198.0, 2.55, 61], [199.8, 2.63, 92], [219.3, 2.9, 54], [181.7, 3.42, 79], [192.4, 3.58, 51], [181.0, 3.66, 62], [185.6, 2.09, 81], [206.2, 3.61, 98], [207.4, 2.4, 74], [198.9, 3.43, 59], [190.9, 2.69, 85], [210.2, 2.79, 90], [216.2, 2.17, 97], [198.1, 3.25, 95], [195.1, 3.14, 94], [198.5, 3.3, 60], [208.9, 3.64, 82], [216.0, 3.96, 99], [201.5, 3.58, 66], [193.9, 2.17, 72], [202.0, 3.54, 74], [182.6, 3.6, 59], [193.4, 3.58, 57], [208.9, 3.68, 84], [217.8, 2.99, 97], [201.1, 2.58, 86], [205.6, 3.05, 92], [195.2, 3.69, 95], [188.3, 3.7, 98], [188.0, 3.07, 75], [204.2, 2.06, 98], [212.0, 3.13, 75]]";
-> "[y]" "[\"ok\", \"ok\", \"scrap\", \"ok\", \"ok\", \"scrap\", \"scrap\", \"ok\", \"scrap\", \"ok\", \"scrap\", \"scrap\", \"ok\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"ok\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"ok\", \"ok\", \"ok\", \"ok\", \"scrap\", \"ok\", \"scrap\", \"ok\", \"ok\", \"scrap\", \"ok\", \"ok\", \"scrap\", \"ok\", \"ok\", \"ok\", \"ok\", \"ok\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"ok\", \"ok\", \"scrap\", \"scrap\", \"ok\", \"ok\", \"scrap\", \"scrap\", \"ok\", \"scrap\", \"ok\", \"ok\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"ok\", \"ok\", \"scrap\", \"scrap\", \"ok\", \"ok\", \"ok\", \"ok\", \"scrap\", \"ok\", \"ok\", \"ok\", \"scrap\", \"ok\", \"scrap\", \"scrap\", \"ok\", \"ok\", \"ok\", \"ok\", \"scrap\", \"scrap\", \"scrap\", \"scrap\", \"ok\"]";
#2. AutoML (the slow neural network and SVM are excluded);
json load "rx_a" (ml automl "rx_best" [X] [y] "{\"k\": 5, \"exclude\": [\"mlp_classifier\", \"svm_classifier\"]}");
#3. PMML with the real names of the columns;
-> "[pmml]" (ml model pmml "rx_best" "{\"features\": [\"temperature\", \"pressure\", \"speed\"], \"target\": \"quality\"}");
-> "[res]" (concat (json select "rx_a" "/task") ", best: " (json select "rx_a" "/leaderboard/[0]/algo") " (accuracy " (math round (json select "rx_a" "/leaderboard/[0]/score") 3) "), 2nd: " (json select "rx_a" "/leaderboard/[1]/algo") " (" (math round (json select "rx_a" "/leaderboard/[1]/score") 3) ") | prediction of a new batch: " (ml model predict "rx_best" "[[205, 3.0, 80]]") " | PMML: " (string length [pmml]) " characters");
json unload "rx_a"; ml model delete "rx_best";
[res]
Result
mentdb
classification, best: naive_bayes (accuracy 0.933), 2nd: random_forest_classifier (0.922) | prediction of a new batch: ["ok"] | PMML: 2420 characters
Go further
  • Write the PMML into a file (file writer or the file functions) and load it in the scoring engine: the decision trees, forests, boosting, linear, logistic, naive Bayes and k-means models are exported.
  • The scores of the leaderboard are cross validated: two algorithms closer than the std are not really different, ml compare gives the paired test.
executed on MentDB

Customer messages: routing, sentiment and recurring topics

A customer service receives free text messages. A naive Bayes model trained on a few labeled messages routes each new message to the right team (billing, delivery, technical), the lexicon sentiment ranks the angriest first, and the keywords of the whole batch show what customers talk about.

Pipeline
Script
admin
#1. Labeled history and the new messages;
-> "[train]" "[\"la facture est fausse, merci de corriger le montant\", \"je n'ai pas re\u00e7u le remboursement de ma commande\", \"double pr\u00e9l\u00e8vement sur ma carte bancaire\", \"relance pour le paiement de la facture de mars\", \"le colis est arriv\u00e9 cass\u00e9\", \"livraison en retard de deux semaines\", \"le livreur a laiss\u00e9 le colis devant la porte\", \"commande jamais re\u00e7ue, suivi bloqu\u00e9\", \"l'application plante \u00e0 l'ouverture\", \"impossible de me connecter \u00e0 mon compte\", \"le site affiche une erreur au paiement\", \"mot de passe refus\u00e9 apr\u00e8s la mise \u00e0 jour\"]";
-> "[labels]" "[\"billing\", \"billing\", \"billing\", \"billing\", \"delivery\", \"delivery\", \"delivery\", \"delivery\", \"technical\", \"technical\", \"technical\", \"technical\"]";
-> "[new]" "[\"Toujours pas de remboursement, c'est vraiment inadmissible !\", \"Le colis est arriv\u00e9 tr\u00e8s vite, livreur aimable, excellent service.\", \"L'application plante encore apr\u00e8s la mise \u00e0 jour, tr\u00e8s mauvaise exp\u00e9rience.\", \"Le montant de la facture ne correspond pas \u00e0 la commande.\", \"Commande arriv\u00e9e en retard et emballage ab\u00eem\u00e9, je suis d\u00e9\u00e7u.\"]";
json load "rx_c" (text classify [train] [labels] [new]);
#2. Team and sentiment of each message;
-> "[res]" "";
json load "rx_n" [new];
for (-> "[i]" 0) (< [i] (json count "rx_n" "/")) (++ "[i]") {
  -> "[s]" (text sentiment (json select "rx_n" (concat "/[" [i] "]")));
  json load "rx_s" [s];
  -> "[res]" (concat [res] "#" [i] " " (json select "rx_c" (concat "/predictions/[" [i] "]/label")) " (p " (math round (json select "rx_c" (concat "/predictions/[" [i] "]/probability")) 2) "), " (json select "rx_s" "/label") " " (math round (json select "rx_s" "/compound") 2) " | ");
  json unload "rx_s";
};
#3. What the batch is about;
json load "rx_k" (text keywords "Toujours pas de remboursement, c'est vraiment inadmissible ! Le colis est arrivé très vite, livreur aimable, excellent service. L'application plante encore après la mise à jour, très mauvaise expérience. Le montant de la facture ne correspond pas à la commande. Commande arrivée en retard et emballage abîmé, je suis déçu." "{\"top\": 4}");
-> "[res]" (concat [res] "keywords: " (json select "rx_k" "/keywords/[0]/keyword") ", " (json select "rx_k" "/keywords/[1]/keyword") ", " (json select "rx_k" "/keywords/[2]/keyword") ", " (json select "rx_k" "/keywords/[3]/keyword"));
json unload "rx_c"; json unload "rx_n"; json unload "rx_k";
[res]
Result
mentdb
#0 billing (p 0.5), negative -0.76 | #1 delivery (p 0.85), positive 0.61 | #2 technical (p 0.94), negative -0.61 | #3 billing (p 0.81), neutral 0 | #4 delivery (p 0.56), negative -0.61 | keywords: commande, vraiment, emballage, inadmissible
Go further
  • Add your own domain words to the sentiment with {"lexicon": {"plante": -2}}; text topics finds the themes of a large batch without labels.
  • For thousands of documents, text dtm gives the TF-IDF matrix for ml train (logistic regression, SVM).

Also in Engineering & data

TR
Telecom & radar engineer
Link budgets in dB, radar equation, detection thresholds and spectra of modulated signals.
11 recipes
CE
Control engineer
Transfer functions, margins, PID and state feedback: complex numbers, CAS, ODE and matrices.
10 recipes
RE
Robotics engineer
Kinematics, Jacobians and trajectories: matrices, vectors, non-linear systems and splines.
10 recipes
EE
Electrical engineer
Impedances, three-phase power and cable sizing with complex numbers and units.
10 recipes
ES
Electronics & signal engineer
Filter design, spectra, peak detection and SNR on sampled signals.
12 recipes
IS
IoT & sensor engineer
Sensor calibration, smoothing, drift detection and measurement uncertainty.
18 recipes
MC
Mechanical & civil engineer
Deflection, stresses, safety factors and concrete strength statistics.
13 recipes
IQ
Industrial quality engineer
SPC control limits, process capability and tests on machines.
20 recipes
SB
Statistician & biostatistician
Confidence intervals, sample size, ANOVA, chi-square and non-parametric tests.
27 recipes
DQ
Data quality analyst
Outliers, format checks, normalization and duplicate detection.
13 recipes