Data quality
Define data quality algorithms, generate the matching MQL script and run a data quality analysis on a database, then show its results. Typical uses: find invalid or inconsistent values before a migration, monitor the quality of a reference table.
dq algorithm showdq algorithm setdq algorithm existdq algorithm getdq algorithm removedq generatedq analysedq analyse show
dq algorithm show
Description
Show all algorithms
Show all algorithms
admin
dq algorithm show
mentdb
{ "is_null_or_empty": "\n\tis null or empty [VAR];\n", "is_hour_without_sec": "\n\ttype is_hour_without_sec [VAR];\n", "is_bool": "\n\ttype is_bool [VAR] 0 1;\n", "is_integer": "\n\ttype is_integer [VAR] 8;\n", "is_valid_timestamp": "\n\ttype is_valid_timestamp [VAR] \"yyyy-MM-dd HH:mm:ss\";\n", "is_hour": "\n\ttype is_hour [VAR];\n", "is_big_int": "\n\ttype is_big_int [VAR];\n", "is_matches_regex": "\n\ttype is_matches_regex [VAR] \".*ze.*\";\n", "is_empty": "\n\tis empty [VAR];\n", "is_time": "\n\ttype is_time [VAR];\n", "is_char": "\n\ttype is_char [VAR] 25;\n", "is_email": "\n\ttype is_email [VAR];\n", "is_enum": "\n\ttype is_enum [VAR] \"1,2,3\";\n", "is_decimal": "\n\ttype is_decimal [VAR] 4 5;\n", "is_null": "\n\tis null [VAR];\n", "is_double": "\n\ttype is_double [VAR];\n", "is_byte": "\n\ttype is_byte [VAR];\n", "is_varchar": "\n\ttype is_varchar [VAR] 15;\n", "is_float": "\n\ttype is_float [VAR];\n", "is_int": "\n\ttype is_int [VAR];\n", "is_date": "\n\ttype is_valid_date [VAR] \"yyyy-MM-dd\";\n", "is_small_int": "\n\ttype is_small_int [VAR];\n", "is_medium_int": "\n\ttype is_medium_int [VAR];\n", "is_number": "\n\ttype is_number [VAR];\n" }
Used in use cases (3 recipes)
CRUD & data API developer › Data quality check on the table
Data quality analyst › Apply the data quality rules of the server catalog
Machine learning engineer › Data quality gate before training
CRUD & data API developer › Data quality check on the table
Data quality analyst › Apply the data quality rules of the server catalog
Machine learning engineer › Data quality gate before training
Often used with
dq algorithm exist · dq algorithm get · json doc · json iobject · and · atom get · atom size · cm get
dq algorithm exist · dq algorithm get · json doc · json iobject · and · atom get · atom size · cm get
dq algorithm set <key> <mql>
Description
Set an algorithm
Set an algorithm
Parameters
key: The key - string - required
mql: The MQL source code - string - required
key: The key - string - required
mql: The MQL source code - string - required
admin
dq algorithm set "key" (mql { type is_char [VAR] 25; });
mentdb
1
Used in use cases (3 recipes)
Data validation developer › A shared catalogue of data quality rules
CRUD & data API developer › Data quality check on the table
Machine learning engineer › Audit a database column with a custom rule
Data validation developer › A shared catalogue of data quality rules
CRUD & data API developer › Data quality check on the table
Machine learning engineer › Audit a database column with a custom rule
Often used with
dq algorithm exist · dq algorithm get · dq algorithm remove · mql · type is_matches_regex · and · atom get · atom size
dq algorithm exist · dq algorithm get · dq algorithm remove · mql · type is_matches_regex · and · atom get · atom size
dq algorithm exist <key>
Description
Check if an algorithm already exists
Check if an algorithm already exists
Parameters
key: The key - string - required
key: The key - string - required
admin
dq algorithm exist "key"
mentdb
1
Used in use cases (5 recipes)
Data validation developer › A shared catalogue of data quality rules
CRUD & data API developer › Data quality check on the table
Data quality analyst › Apply the data quality rules of the server catalog
Machine learning engineer › Data quality gate before training
Machine learning engineer › Audit a database column with a custom rule
Data validation developer › A shared catalogue of data quality rules
CRUD & data API developer › Data quality check on the table
Data quality analyst › Apply the data quality rules of the server catalog
Machine learning engineer › Data quality gate before training
Machine learning engineer › Audit a database column with a custom rule
Often used with
dq algorithm get · dq algorithm remove · dq algorithm set · dq algorithm show · eval · json doc · mql · type is_matches_regex
dq algorithm get · dq algorithm remove · dq algorithm set · dq algorithm show · eval · json doc · mql · type is_matches_regex
dq algorithm get <key>
Description
Get an algorithm
Get an algorithm
Parameters
key: The key - string - required
key: The key - string - required
admin
dq algorithm get "key";
mentdb
type is_char [VAR] 25;
Used in use cases (5 recipes)
Data validation developer › A shared catalogue of data quality rules
CRUD & data API developer › Data quality check on the table
Data quality analyst › Apply the data quality rules of the server catalog
Machine learning engineer › Data quality gate before training
Machine learning engineer › Audit a database column with a custom rule
Data validation developer › A shared catalogue of data quality rules
CRUD & data API developer › Data quality check on the table
Data quality analyst › Apply the data quality rules of the server catalog
Machine learning engineer › Data quality gate before training
Machine learning engineer › Audit a database column with a custom rule
Often used with
dq algorithm exist · dq algorithm remove · dq algorithm set · dq algorithm show · eval · json doc · mql · type is_matches_regex
dq algorithm exist · dq algorithm remove · dq algorithm set · dq algorithm show · eval · json doc · mql · type is_matches_regex
dq algorithm remove <key>
Description
Remove an algorithm
Remove an algorithm
Parameters
key: The key - string - required
key: The key - string - required
admin
dq algorithm remove "key";
mentdb
1
Used in use cases (3 recipes)
Data validation developer › A shared catalogue of data quality rules
CRUD & data API developer › Data quality check on the table
Machine learning engineer › Audit a database column with a custom rule
Data validation developer › A shared catalogue of data quality rules
CRUD & data API developer › Data quality check on the table
Machine learning engineer › Audit a database column with a custom rule
Often used with
dq algorithm exist · dq algorithm get · dq algorithm set · mql · type is_matches_regex · and · atom get · atom size
dq algorithm exist · dq algorithm get · dq algorithm set · mql · type is_matches_regex · and · atom get · atom size
dq generate <cmId> <tablename> <fieldname> <jsonArrayAlgoId> <sqlSource>
Description
Generate the Data Quality MQL script
Generate the Data Quality MQL script
Parameters
cmId: The database connection id - string - required
tablename: The table name - string - required
fieldname: The field name - string - required
jsonArrayAlgoId: The JSON array of the selected algorithm ids - string - required
sqlSource: The select query (origin) - string - required
cmId: The database connection id - string - required
tablename: The table name - string - required
fieldname: The field name - string - required
jsonArrayAlgoId: The JSON array of the selected algorithm ids - string - required
sqlSource: The select query (origin) - string - required
admin
dq generate "demo_cm_mysql" "products" "quantity" "[ \"is_big_int\", \"is_date\" ]" "select * from products limit 0, 500"
mentdb
json load "dq" "[]"; json load "dq_quantity" "{}"; json iobject "dq_quantity" / field "quantity" STR; json iobject "dq_quantity" / algo "{}" OBJ; json iobject "dq_quantity" /algo "is_big_int" (mql { type is_big_int [T_quantity]; }) STR; json iobject "dq_quantity" /algo "is_date" (mql { type is_valid_date [T_quantity] "yyyy-MM-dd"; }) STR; json iarray "dq" / (json doc "dq_quantity") OBJ; dq analyse "demo_cm_mysql" (json doc "dq") "products" "select * from products limit 0, 500";
Used in use cases (2 recipes)
CRUD & data API developer › Data quality check on the table
Machine learning engineer › Audit a database column with a custom rule
CRUD & data API developer › Data quality check on the table
Machine learning engineer › Audit a database column with a custom rule
Often used with
dq algorithm exist · dq algorithm get · dq algorithm remove · dq algorithm set · dq analyse show · json doc · mql · type is_matches_regex
dq algorithm exist · dq algorithm get · dq algorithm remove · dq algorithm set · dq analyse show · json doc · mql · type is_matches_regex
dq analyse <cmId> <jsonArrayAlgoId> <title> <sqlSource>
Description
Run a data quality analysis on a database
Run a data quality analysis on a database
Parameters
cmId: The database connection id - string - required
jsonArrayAlgoId: The JSON array of the selected algorithm ids - string - required
title: The title - string - required
sqlSource: The select query (origin) - string - required
cmId: The database connection id - string - required
jsonArrayAlgoId: The JSON array of the selected algorithm ids - string - required
title: The title - string - required
sqlSource: The select query (origin) - string - required
admin
json load "dq" "[]"; json load "dq_name" "{}"; json iobject "dq_name" / field "name" STR; json iobject "dq_name" / algo "{}" OBJ; json iobject "dq_name" /algo "is_float" (mql { type is_float [T_name]; }) STR; json iobject "dq_name" /algo "is_null" (mql { is null [T_name]; }) STR; json iobject "dq_name" /algo "is_small_int" (mql { type is_small_int [T_name]; }) STR; json iobject "dq_name" /algo "is_valid_timestamp" (mql { type is_valid_timestamp [T_name] "yyyy-MM-dd HH:mm:ss"; }) STR; json iarray "dq" / (json doc "dq_name") OBJ; dq analyse "demo_cm_mysql" (json doc "dq") "products" "SELECT * FROM `products` LIMIT 0, 500";
mentdb
In editor ...
Often used with
and · atom get · atom size · cm get · dq algorithm exist · dq algorithm get · dq algorithm remove · dq algorithm set
and · atom get · atom size · cm get · dq algorithm exist · dq algorithm get · dq algorithm remove · dq algorithm set
dq analyse show <cmId> <jsonArrayAlgoId> <algoKey> <fieldKey> <title> <sqlSource>
Description
Show an analysis
Show an analysis
Parameters
cmId: The database connection id - string - required
jsonArrayAlgoId: The JSON array of the selected algorithm ids - string - required
algoKey: The selected algorithm key - string - required
fieldKey: The selected field key - string - required
title: The title - string - required
sqlSource: The select query (origin) - string - required
cmId: The database connection id - string - required
jsonArrayAlgoId: The JSON array of the selected algorithm ids - string - required
algoKey: The selected algorithm key - string - required
fieldKey: The selected field key - string - required
title: The title - string - required
sqlSource: The select query (origin) - string - required
admin
json load "dq" "[]"; json load "dq_name" "{}"; json iobject "dq_name" / field "name" STR; json iobject "dq_name" / algo "{}" OBJ; json iobject "dq_name" /algo "is_float" (mql { type is_float [T_name]; }) STR; json iobject "dq_name" /algo "is_null" (mql { is null [T_name]; }) STR; json iobject "dq_name" /algo "is_small_int" (mql { type is_small_int [T_name]; }) STR; json iobject "dq_name" /algo "is_valid_timestamp" (mql { type is_valid_timestamp [T_name] "yyyy-MM-dd HH:mm:ss"; }) STR; json iarray "dq" / (json doc "dq_name") OBJ; dq analyse show "demo_cm_mysql" (json doc "dq") "is_float" "name" "products" "SELECT * FROM `products` LIMIT 0, 500";
mentdb
In editor ...
Used in use cases (2 recipes)
CRUD & data API developer › Data quality check on the table
Machine learning engineer › Audit a database column with a custom rule
CRUD & data API developer › Data quality check on the table
Machine learning engineer › Audit a database column with a custom rule
Often used with
dq algorithm exist · dq algorithm get · dq algorithm remove · dq algorithm set · dq generate · json doc · mql · type is_matches_regex
dq algorithm exist · dq algorithm get · dq algorithm remove · dq algorithm set · dq generate · json doc · mql · type is_matches_regex
