Profiles / AI & prediction
DL

Deep learning engineer

Text classifiers (OpenNLP), multilayer perceptrons on tabular data (ml train mlp_*), and the maths of the neural networks.

8 recipes on this page · open in the interactive view

Recipes
  1. Categorise bank statement lines
  2. Scaled dot-product attention of a transformer
  3. Xavier initialisation of a dense layer
  4. Train a sigmoid neuron by gradient descent
  5. Sentiment triage of customer reviews
  6. Predictive maintenance with a neural network
  7. Confusion matrix of an intent classifier
  8. A neural network learns the XOR truth table
executed on MentDB

Categorise bank statement lines

Write a labelled training file of bank statement lines, train a document classifier (OpenNLP, 10 iterations) into a model file, load it and categorise new lines as salary, cash, grocery or fees; the files are deleted at the end.

Pipeline
Script
admin
#1. Training file: one labelled bank statement line per row;
file create "tmp/rx_bank_train.txt" "salary TRANSFER RECEIVED ACME CORP SALARY JUNE
salary TRANSFER RECEIVED ACME CORP SALARY JULY
salary TRANSFER RECEIVED ACME CORP SALARY AUGUST
cash CARD 2052 ATM WITHDRAWAL PARIS ROCH
cash CARD 2052 ATM WITHDRAWAL LYON GARE
cash CARD 2052 ATM WITHDRAWAL NANTES CENTRE
grocery CARD 2052 PAYMENT SUPERMARKET CARREFOUR
grocery CARD 2052 PAYMENT SUPERMARKET LIDL
grocery CARD 2052 PAYMENT SUPERMARKET MONOPRIX
fees MONTHLY ACCOUNT FEE PREMIUM OPTION
fees FOREIGN PAYMENT FEE OUTSIDE EURO ZONE
fees CARD ANNUAL FEE PREMIUM";
#2. Train the document classifier, 10 iterations, then load it;
dl n_bayesian create_model "en" "tmp/rx_bank_train.txt" 10 "tmp/rx_bank_model.bin";
if (== (dl n_bayesian exist "rx_bank") 0) {dl n_bayesian load "rx_bank" "tmp/rx_bank_model.bin";};
#3. Categorise the new lines of the statement;
json load "rx_lines" "[\"CARD 2052 PAYMENT SUPERMARKET AUCHAN\", \"ATM WITHDRAWAL BORDEAUX\", \"TRANSFER RECEIVED ACME CORP SALARY SEPTEMBER\", \"ACCOUNT FEE OCTOBER\"]";
-> "[out]" "";
for (-> "[i]" 0) (< [i] (json count "rx_lines" /)) (++ "[i]") {
  json load "rx_p" (dl n_bayesian predict "rx_bank" (json select "rx_lines" (concat "/[" [i] "]")));
  -> "[out]" (concat [out] (json select "rx_p" "/input") " => " (json select "rx_p" "/prediction") " " (math round (* 100 (json select "rx_p" "/best_double")) 1) "% | ");
};
json load "rx_m" (dl n_bayesian show);
-> "[n]" (json count "rx_m" /);
#4. Free the model and delete the files;
dl n_bayesian delete "rx_bank";
file delete "tmp/rx_bank_train.txt";
file delete "tmp/rx_bank_model.bin";
json unload "rx_lines"; json unload "rx_p"; json unload "rx_m";
concat [out] "classifiers loaded: " [n]
Result
mentdb
CARD 2052 PAYMENT SUPERMARKET AUCHAN => grocery 100% | ATM WITHDRAWAL BORDEAUX => cash 99.9% | TRANSFER RECEIVED ACME CORP SALARY SEPTEMBER => salary 100% | ACCOUNT FEE OCTOBER => fees 99.8% | classifiers loaded: 1
Go further
  • Keep the model file and reload it at server start with dl n_bayesian load.
  • More labelled lines per category give more reliable probabilities.
executed on MentDB

Scaled dot-product attention of a transformer

The core of a transformer: for 3 tokens, the script computes the scores Q.Kt / sqrt(d_k), a stable softmax on each row (max subtracted) and the output A.V.

Pipeline
Script
admin
#1. Queries, keys and values of 3 tokens, d_k = 4;
-> "[Q]" "[[1, 0, 1, 0], [0, 1, 0, 1], [1, 1, 0, 0]]";
-> "[K]" "[[1, 0, 1, 0], [0, 2, 0, 2], [1, 1, 1, 1]]";
-> "[V]" "[[1, 0], [0, 1], [0.5, 0.5]]";
#2. Scores = Q.Kt / sqrt d_k;
json load "rx_p4h_s" (matrix scalar_mul (matrix mul [Q] (matrix transpose [K])) (/ 1 (math sqrt 4)));
#3. Softmax on each row, max subtracted for stability;
json load "rx_p4h_a" "[]";
for (-> "[i]" 0) (< [i] 3) (++ "[i]") {
	-> "[row]" (json select "rx_p4h_s" (concat "/[" [i] "]"));
	-> "[m]" (stat max [row]);
	-> "[e0]" (math exp (- (json select "rx_p4h_s" (concat "/[" [i] "]/[0]")) [m]));
	-> "[e1]" (math exp (- (json select "rx_p4h_s" (concat "/[" [i] "]/[1]")) [m]));
	-> "[e2]" (math exp (- (json select "rx_p4h_s" (concat "/[" [i] "]/[2]")) [m]));
	-> "[z]" (+ [e0] [e1] [e2]);
	json iarray "rx_p4h_a" / (concat "[" (/ [e0] [z]) "," (/ [e1] [z]) "," (/ [e2] [z]) "]") ARRAY;
};
#4. Output = A.V;
concat "attention weights of token 1: " (json select "rx_p4h_a" "/[0]") " | output A.V: " (matrix mul (json doc "rx_p4h_a") [V])
Result
mentdb
attention weights of token 1: [
  0.4223187982515182,
  0.15536240349696362,
  0.4223187982515182
] | output A.V: [[0.6334781973772773,0.3665218026227227],[0.21239480869777927,0.7876051913022206],[0.424522403214174,0.5754775967858261]]
Go further
  • Add a causal mask: set the scores above the diagonal to a large negative value with matrix set before the softmax.
  • Multi-head: run the block with several Q, K, V projections and concatenate the outputs.
  • Check that each row of A sums to 1 with stat sum.
executed on MentDB

Xavier initialisation of a dense layer

A dense layer 8 to 4 is initialised with the Xavier (Glorot) uniform law, bound a = sqrt(6 / (fan_in + fan_out)). The script draws the weights, checks they are inside [-a, a] and runs one forward pass with ReLU. The weights change at each run.

Pipeline
Script
admin
#1. Layer 8 inputs to 4 neurons: Xavier uniform bound a = sqrt 6 / fan_in + fan_out;
-> "[a]" (math sqrt (/ 6 (+ 8 4)));
#2. W = a . 2R - 1 with R uniform in 0 1;
-> "[R]" (matrix random 4 8);
-> "[W]" (matrix scalar_mul (matrix sub (matrix scalar_mul [R] 2) (matrix ones 4 8)) [a]);
-> "[flat]" (string replace (string replace (string replace [W] "],[" ",") "[[" "[") "]]" "]");
#3. Forward pass of one input and ReLU;
json load "rx_p4h_h" (matrix mul [W] "[0.2, -0.1, 0.5, 0.3, -0.4, 0.1, 0.0, 0.6]");
-> "[active]" 0;
for (-> "[i]" 0) (< [i] 4) (++ "[i]") {
	if (> (json select "rx_p4h_h" (concat "/[" [i] "]")) 0) {++ "[active]";};
};
concat "shape " (matrix size [W]) " | bound a = " (math round [a] 4) " | all 32 weights in [-a, a]: " (<= (math max (math abs (stat min [flat])) (stat max [flat])) [a]) " | ReLU neurons active on the input: " [active] "/4"
Result
mentdb
shape [4,8] | bound a = 0.7071 | all 32 weights in [-a, a]: 1 | ReLU neurons active on the input: 2/4
Go further
  • For ReLU networks, the He bound sqrt(6 / fan_in) keeps the variance better.
  • Check the empirical variance with stat pvariance: it should be close to a^2/3.
  • Draw normal weights with stat dist_random instead of the uniform law.
executed on MentDB

Train a sigmoid neuron by gradient descent

A failure model p = sigmoid(w.x + b) is trained on 8 machines (hours of use, failure 0/1) by batch gradient descent on the cross entropy, 100 epochs, learning rate 2. The script plots the learning curve with the final weights in the title.

Pipeline
Script
admin
#1. Hours of use of a machine and failure 0/1;
json load "rx_p4h_x" "[1, 2, 3, 4, 5, 6, 7, 8]";
json load "rx_p4h_y" "[0, 0, 0, 1, 0, 1, 1, 1]";
-> "[w]" 0; -> "[b]" 0; -> "[lr]" 2;
json load "rx_p4h_loss" "[]";
#2. Batch gradient descent of a sigmoid neuron, cross entropy loss;
for (-> "[ep]" 0) (< [ep] 100) (++ "[ep]") {
	-> "[gw]" 0; -> "[gb]" 0; -> "[L]" 0;
	for (-> "[i]" 0) (< [i] 8) (++ "[i]") {
		-> "[xi]" (/ (json select "rx_p4h_x" (concat "/[" [i] "]")) 8);
		-> "[yi]" (json select "rx_p4h_y" (concat "/[" [i] "]"));
		-> "[p]" (math calc "1/(1+exp(-([w]*[xi]+[b])))");
		-> "[gw]" (math calc "[gw]+([p]-[yi])*[xi]/8");
		-> "[gb]" (math calc "[gb]+([p]-[yi])/8");
		-> "[L]" (math calc "[L]-([yi]*ln([p])+(1-[yi])*ln(1-[p]))/8");
	};
	-> "[w]" (math calc "[w]-[lr]*[gw]");
	-> "[b]" (math calc "[b]-[lr]*[gb]");
	json iarray "rx_p4h_loss" / [L] NUM;
};
#3. Learning curve;
plot chart (concat "{\"title\": \"Training loss, w=" (math round [w] 2) " b=" (math round [b] 2) "\", \"x_label\": \"epoch\", \"y_label\": \"cross entropy\", \"series\": [{\"name\": \"loss\", \"y\": " (json doc "rx_p4h_loss") "}]}")
Result
mentdb
The chart is shown in the Plot tab of the editor.
Go further
  • Try a learning rate of 0.1 and 10 to see a slow and an unstable training.
  • Add an L2 penalty lambda.w^2 to the loss and to the gradient of w.
  • Score the trained neuron on new machines with the same math calc sigmoid and a 0.5 threshold.
executed on MentDB

Sentiment triage of customer reviews

Train a naive Bayes document classifier (OpenNLP) on eight labelled reviews and classify four new reviews with the probability of the best class, so that the negative ones go to the support team; the files are deleted at the end.

Pipeline
Script
admin
#1. Labelled reviews: the category, then the text;
file create "tmp/rx_reviews.txt" "positive great product, fast delivery
positive excellent quality, I love it
positive works perfectly, great value for money
positive very good product, fast and solid
negative broken on arrival, very disappointed
negative poor quality, it stopped working
negative late delivery and bad support
negative bad product, waste of money";
#2. Train the classifier, 20 iterations, then load it;
dl n_bayesian create_model "en" "tmp/rx_reviews.txt" 20 "tmp/rx_reviews.bin";
if (== (dl n_bayesian exist "rx_reviews") 1) {dl n_bayesian delete "rx_reviews";};
dl n_bayesian load "rx_reviews" "tmp/rx_reviews.bin";
#3. Triage of new reviews: the negative ones go to the support team;
json load "rx_new" "[\"great quality, works well\", \"bad quality, stopped after a week\", \"late delivery, disappointed\", \"I love it, excellent value\"]";
-> "[out]" "";
for (-> "[i]" 0) (< [i] (json count "rx_new" /)) (++ "[i]") {
  json load "rx_p" (dl n_bayesian predict "rx_reviews" (json select "rx_new" (concat "/[" [i] "]")));
  -> "[out]" (concat [out] (json select "rx_p" "/input") " => " (json select "rx_p" "/prediction") " " (math round (* 100 (json select "rx_p" "/best_double")) 1) "% | ");
};
#4. Free the model and delete the files;
dl n_bayesian delete "rx_reviews";
file delete "tmp/rx_reviews.txt";
file delete "tmp/rx_reviews.bin";
json unload "rx_new"; json unload "rx_p";
[out]
Result
mentdb
great quality, works well => positive 99.9% | bad quality, stopped after a week => negative 99.9% | late delivery, disappointed => negative 99.8% | I love it, excellent value => positive 100% |
Go further
  • Add labelled lines and train again: a document classifier needs many examples per class.
  • Measure the quality on a test set with ml metrics report.
executed on MentDB

Predictive maintenance with a neural network

Train a multilayer perceptron on 24 sensor records (temperature, vibration, motor current) labelled normal, bearing wear or overheating, check it with a 4-fold cross validation, then diagnose three motors with the probability of the diagnosis.

Pipeline
Script
admin
#1. Sensor history: temperature in C, vibration in mm/s, motor current in A - state;
-> "[X]" "[[61.5, 1.28, 7.9], [65.4, 4.13, 8.5], [91.3, 1.68, 12.0], [62.5, 1.26, 8.1], [62.7, 4.84, 8.8], [89.5, 1.26, 10.6], [60.2, 1.13, 8.1], [65.9, 4.71, 8.3], [88.9, 1.68, 11.2], [65.4, 1.28, 8.5], [64.8, 4.2, 8.5], [87.7, 1.73, 11.6], [61.1, 1.06, 7.8], [68.4, 4.18, 8.7], [89.3, 1.3, 11.5], [64.6, 0.9, 7.9], [65.8, 4.17, 8.8], [87.8, 1.31, 11.9], [63.3, 1.34, 8.6], [66.7, 4.55, 8.1], [89.8, 1.48, 11.3], [59.5, 1.05, 7.8], [68.6, 3.69, 8.0], [88.7, 1.89, 11.8]]";
-> "[y]" "[\"normal\", \"bearing_wear\", \"overheating\", \"normal\", \"bearing_wear\", \"overheating\", \"normal\", \"bearing_wear\", \"overheating\", \"normal\", \"bearing_wear\", \"overheating\", \"normal\", \"bearing_wear\", \"overheating\", \"normal\", \"bearing_wear\", \"overheating\", \"normal\", \"bearing_wear\", \"overheating\", \"normal\", \"bearing_wear\", \"overheating\"]";
#2. A network with one hidden layer of 8 neurons, inputs standardized inside the model;
-> "[p]" "{\"hidden\": [8], \"epochs\": 500, \"seed\": 1}";
json load "rx_cv" (ml cross_validate "mlp_classifier" [X] [y] 4 [p] 1);
ml train "rx_motor" "mlp_classifier" [X] [y] [p];
#3. Diagnosis of three motors;
json load "rx_pp" (ml model predict_proba "rx_motor" "[[62.0, 1.2, 8.0], [66.0, 4.4, 8.4], [90.0, 1.5, 11.5]]");
json load "rx_d" (ml model predict "rx_motor" "[[62.0, 1.2, 8.0], [66.0, 4.4, 8.4], [90.0, 1.5, 11.5]]");
-> "[res]" (concat "CV accuracy " (math round (json select "rx_cv" "/mean") 3) " | ");
for (-> "[i]" 0) (< [i] 3) (++ "[i]") {
  -> "[best]" 0;
  for (-> "[c]" 0) (< [c] 3) (++ "[c]") {-> "[best]" (math max [best] (json select "rx_pp" (concat "/proba/[" [i] "]/[" [c] "]")));};
  -> "[res]" (concat [res] "motor " (+ [i] 1) ": " (json select "rx_d" (concat "/[" [i] "]")) " " (math round (* 100 [best]) 1) "% | ");
};
json unload "rx_cv"; json unload "rx_pp"; json unload "rx_d"; ml model delete "rx_motor";
[res]
Result
mentdb
CV accuracy 1 | motor 1: normal 100% | motor 2: bearing_wear 100% | motor 3: overheating 100% |
Go further
  • Compare with ml train random_forest_classifier, often as good on small tables.
  • Save the trained network with ml model save and reload it in a web service.
executed on MentDB

Confusion matrix of an intent classifier

A help desk classifier routes messages to billing, technical or shipping. Train it (OpenNLP naive Bayes document classifier), run it on a held-out test set with the true classes, then read the accuracy, the recall of the billing class and the confusion matrix.

Pipeline
Script
admin
#1. Training file of the help desk: the class, then the message;
file create "tmp/rx_intent.txt" "billing I was charged twice on my invoice
billing refund the payment of my last invoice
billing wrong amount on my bill
billing the invoice amount is not correct
technical the application crashes at login
technical error message when I open the app
technical the device does not start after the update
technical the app shows an error after the update
shipping my parcel has not arrived yet
shipping change the delivery address of my order
shipping the parcel was delivered to the wrong address
shipping where is my parcel, the delivery is late";
dl n_bayesian create_model "en" "tmp/rx_intent.txt" 30 "tmp/rx_intent.bin";
if (== (dl n_bayesian exist "rx_intent") 1) {dl n_bayesian delete "rx_intent";};
dl n_bayesian load "rx_intent" "tmp/rx_intent.bin";
#2. Held-out test set: message and true class;
json load "rx_test" "[[\"double payment on the invoice\", \"billing\"], [\"the bill amount is wrong\", \"billing\"], [\"app error after the update\", \"technical\"], [\"login crashes every time\", \"technical\"], [\"where is my parcel\", \"shipping\"], [\"wrong delivery address\", \"shipping\"], [\"refund my order, the parcel never arrived\", \"shipping\"], [\"I paid but the app shows an error\", \"billing\"], [\"the update changed my delivery address\", \"shipping\"]]";
json load "rx_true" "[]"; json load "rx_pred" "[]";
for (-> "[i]" 0) (< [i] (json count "rx_test" /)) (++ "[i]") {
  json load "rx_p" (dl n_bayesian predict "rx_intent" (json select "rx_test" (concat "/[" [i] "]/[0]")));
  json iarray "rx_true" / (json select "rx_test" (concat "/[" [i] "]/[1]")) STR;
  json iarray "rx_pred" / (json select "rx_p" "/prediction") STR;
};
#3. Report: rows of the confusion matrix = true class, columns = predicted class;
json load "rx_rep" (ml metrics report (json doc "rx_true") (json doc "rx_pred"));
-> "[res]" (concat "accuracy " (math round (json select "rx_rep" "/accuracy") 3) " | recall billing " (math round (json select "rx_rep" "/classes/[0]/recall") 3) " | labels " (string replace (string replace (json select "rx_rep" "/labels") [_n_] "") " " "") " | confusion " (string replace (string replace (json select "rx_rep" "/confusion") [_n_] "") " " ""));
dl n_bayesian delete "rx_intent";
file delete "tmp/rx_intent.txt"; file delete "tmp/rx_intent.bin";
json unload "rx_test"; json unload "rx_true"; json unload "rx_pred"; json unload "rx_p"; json unload "rx_rep";
[res]
Result
mentdb
accuracy 0.889 | recall billing 0.667 | labels ["billing","shipping","technical"] | confusion [[2,0,1],[0,4,0],[0,0,2]]
Go further
  • The macro F1 of ml metrics report summarizes the 3 classes in one number.
  • A message mixing two intents, ex: payment and app error, is the typical error of a single-label classifier.
executed on MentDB

A neural network learns the XOR truth table

XOR is the classic problem a single neuron cannot solve. Train a logistic regression and a multilayer perceptron on the 4 cases of the truth table, then compare their predictions: only the network with a hidden layer finds the non linear frontier.

Pipeline
Script
admin
#1. The truth table of XOR;
-> "[X]" "[[0, 0], [0, 1], [1, 0], [1, 1]]";
-> "[y]" "[0, 1, 1, 0]";
#2. A linear model and a network with 8 tanh neurons;
ml train "rx_lin" "logistic" [X] [y] "{\"alpha\": 0.1}";
ml train "rx_xor" "mlp_classifier" [X] [y] "{\"hidden\": [8], \"activation\": \"tanh\", \"epochs\": 2000, \"learning_rate\": 0.05, \"batch_size\": 4, \"tol\": 0, \"seed\": 1}";
#3. Predictions of both models;
json load "rx_pl" (ml model predict "rx_lin" [X]);
json load "rx_pn" (ml model predict "rx_xor" [X]);
-> "[res]" "network: ";
for (-> "[i]" 0) (< [i] 4) (++ "[i]") {
  -> "[a]" (math floor (/ [i] 2)); -> "[b]" (math mod [i] 2);
  -> "[out]" (json select "rx_pn" (concat "/[" [i] "]"));
  -> "[res]" (concat [res] [a] " xor " [b] " = " [out] (if (== [out] (math abs (- [a] [b]))) {" ok"} {" WRONG"}) " | ");
};
-> "[res]" (concat [res] "logistic accuracy " (ml model score "rx_lin" [X] [y]) " | network accuracy " (ml model score "rx_xor" [X] [y]));
json unload "rx_pl"; json unload "rx_pn"; ml model delete "rx_lin"; ml model delete "rx_xor";
[res]
Result
mentdb
network: 0 xor 0 = 0 ok | 0 xor 1 = 1 ok | 1 xor 0 = 1 ok | 1 xor 1 = 0 ok | logistic accuracy 0.5 | network accuracy 1
Go further
  • ml prep polynomial adds the product x0*x1: then even a linear model solves XOR.
  • The same check, run after each training, is a regression test of the model.

Also in AI & prediction

ML
Machine learning engineer
Train, benchmark, tune and deploy models (forests, boosting, SVM, neural networks), density clustering and graph search, in memory on the server.
18 recipes
FE
Forecaster & econometrician
Linear and multiple regressions, confidence intervals, ANOVA, polynomial trends, Holt-Winters, stationarity tests and ARIMA.
16 recipes
CB
Conversational bot designer
Bots, intents, users, rights and releases between servers.
11 recipes