Data quality
Define data quality algorithms, generate the matching MQL script and run a data quality analysis on a database, then show its results. Typical uses: find invalid or inconsistent values before a migration, monitor the quality of a reference table.
dq algorithm showdq algorithm setdq algorithm existdq algorithm getdq algorithm removedq generatedq analysedq analyse show

dq algorithm show


Description

    Show all algorithms

admin
dq algorithm show
mentdb
{ "is_null_or_empty": "\n\tis null or empty [VAR];\n", "is_hour_without_sec": "\n\ttype is_hour_without_sec [VAR];\n", "is_bool": "\n\ttype is_bool [VAR] 0 1;\n", "is_integer": "\n\ttype is_integer [VAR] 8;\n", "is_valid_timestamp": "\n\ttype is_valid_timestamp [VAR] \"yyyy-MM-dd HH:mm:ss\";\n", "is_hour": "\n\ttype is_hour [VAR];\n", "is_big_int": "\n\ttype is_big_int [VAR];\n", "is_matches_regex": "\n\ttype is_matches_regex [VAR] \".*ze.*\";\n", "is_empty": "\n\tis empty [VAR];\n", "is_time": "\n\ttype is_time [VAR];\n", "is_char": "\n\ttype is_char [VAR] 25;\n", "is_email": "\n\ttype is_email [VAR];\n", "is_enum": "\n\ttype is_enum [VAR] \"1,2,3\";\n", "is_decimal": "\n\ttype is_decimal [VAR] 4 5;\n", "is_null": "\n\tis null [VAR];\n", "is_double": "\n\ttype is_double [VAR];\n", "is_byte": "\n\ttype is_byte [VAR];\n", "is_varchar": "\n\ttype is_varchar [VAR] 15;\n", "is_float": "\n\ttype is_float [VAR];\n", "is_int": "\n\ttype is_int [VAR];\n", "is_date": "\n\ttype is_valid_date [VAR] \"yyyy-MM-dd\";\n", "is_small_int": "\n\ttype is_small_int [VAR];\n", "is_medium_int": "\n\ttype is_medium_int [VAR];\n", "is_number": "\n\ttype is_number [VAR];\n" }
Used in use cases (3 recipes)

    CRUD & data API developer › Data quality check on the table
    Data quality analyst › Apply the data quality rules of the server catalog
    Machine learning engineer › Data quality gate before training
Often used with

    dq algorithm exist  ·  dq algorithm get  ·  json doc  ·  json iobject  ·  and  ·  atom get  ·  atom size  ·  cm get

dq algorithm set <key> <mql>


Description

    Set an algorithm

Parameters

    key:   The key - string - required
    mql:   The MQL source code - string - required
admin
dq algorithm set "key" (mql { type is_char [VAR] 25; });
mentdb
1
Used in use cases (3 recipes)

    Data validation developer › A shared catalogue of data quality rules
    CRUD & data API developer › Data quality check on the table
    Machine learning engineer › Audit a database column with a custom rule
Often used with

    dq algorithm exist  ·  dq algorithm get  ·  dq algorithm remove  ·  mql  ·  type is_matches_regex  ·  and  ·  atom get  ·  atom size

dq algorithm exist <key>


Description

    Check if an algorithm already exists

Parameters

    key:   The key - string - required
admin
dq algorithm exist "key"
mentdb
1
Used in use cases (5 recipes)

    Data validation developer › A shared catalogue of data quality rules
    CRUD & data API developer › Data quality check on the table
    Data quality analyst › Apply the data quality rules of the server catalog
    Machine learning engineer › Data quality gate before training
    Machine learning engineer › Audit a database column with a custom rule
Often used with

    dq algorithm get  ·  dq algorithm remove  ·  dq algorithm set  ·  dq algorithm show  ·  eval  ·  json doc  ·  mql  ·  type is_matches_regex

dq algorithm get <key>


Description

    Get an algorithm

Parameters

    key:   The key - string - required
admin
dq algorithm get "key";
mentdb
type is_char [VAR] 25;
Used in use cases (5 recipes)

    Data validation developer › A shared catalogue of data quality rules
    CRUD & data API developer › Data quality check on the table
    Data quality analyst › Apply the data quality rules of the server catalog
    Machine learning engineer › Data quality gate before training
    Machine learning engineer › Audit a database column with a custom rule
Often used with

    dq algorithm exist  ·  dq algorithm remove  ·  dq algorithm set  ·  dq algorithm show  ·  eval  ·  json doc  ·  mql  ·  type is_matches_regex

dq algorithm remove <key>


Description

    Remove an algorithm

Parameters

    key:   The key - string - required
admin
dq algorithm remove "key";
mentdb
1
Used in use cases (3 recipes)

    Data validation developer › A shared catalogue of data quality rules
    CRUD & data API developer › Data quality check on the table
    Machine learning engineer › Audit a database column with a custom rule
Often used with

    dq algorithm exist  ·  dq algorithm get  ·  dq algorithm set  ·  mql  ·  type is_matches_regex  ·  and  ·  atom get  ·  atom size

dq generate <cmId> <tablename> <fieldname> <jsonArrayAlgoId> <sqlSource>


Description

    Generate the Data Quality MQL script

Parameters

    cmId:   The database connection id - string - required
    tablename:   The table name - string - required
    fieldname:   The field name - string - required
    jsonArrayAlgoId:   The JSON array of the selected algorithm ids - string - required
    sqlSource:   The select query (origin) - string - required
admin
dq generate "demo_cm_mysql" "products" "quantity" "[ \"is_big_int\", \"is_date\" ]" "select * from products limit 0, 500"
mentdb
json load "dq" "[]"; json load "dq_quantity" "{}"; json iobject "dq_quantity" / field "quantity" STR; json iobject "dq_quantity" / algo "{}" OBJ; json iobject "dq_quantity" /algo "is_big_int" (mql { type is_big_int [T_quantity]; }) STR; json iobject "dq_quantity" /algo "is_date" (mql { type is_valid_date [T_quantity] "yyyy-MM-dd"; }) STR; json iarray "dq" / (json doc "dq_quantity") OBJ; dq analyse "demo_cm_mysql" (json doc "dq") "products" "select * from products limit 0, 500";
Used in use cases (2 recipes)

    CRUD & data API developer › Data quality check on the table
    Machine learning engineer › Audit a database column with a custom rule
Often used with

    dq algorithm exist  ·  dq algorithm get  ·  dq algorithm remove  ·  dq algorithm set  ·  dq analyse show  ·  json doc  ·  mql  ·  type is_matches_regex

dq analyse <cmId> <jsonArrayAlgoId> <title> <sqlSource>


Description

    Run a data quality analysis on a database

Parameters

    cmId:   The database connection id - string - required
    jsonArrayAlgoId:   The JSON array of the selected algorithm ids - string - required
    title:   The title - string - required
    sqlSource:   The select query (origin) - string - required
admin
json load "dq" "[]"; json load "dq_name" "{}"; json iobject "dq_name" / field "name" STR; json iobject "dq_name" / algo "{}" OBJ; json iobject "dq_name" /algo "is_float" (mql { type is_float [T_name]; }) STR; json iobject "dq_name" /algo "is_null" (mql { is null [T_name]; }) STR; json iobject "dq_name" /algo "is_small_int" (mql { type is_small_int [T_name]; }) STR; json iobject "dq_name" /algo "is_valid_timestamp" (mql { type is_valid_timestamp [T_name] "yyyy-MM-dd HH:mm:ss"; }) STR; json iarray "dq" / (json doc "dq_name") OBJ; dq analyse "demo_cm_mysql" (json doc "dq") "products" "SELECT * FROM `products` LIMIT 0, 500";
mentdb
In editor ...
Used in use cases (1 recipe)

    CRUD & data API developer › Data quality check on the table
Often used with

    and  ·  atom get  ·  atom size  ·  cm get  ·  dq algorithm exist  ·  dq algorithm get  ·  dq algorithm remove  ·  dq algorithm set

dq analyse show <cmId> <jsonArrayAlgoId> <algoKey> <fieldKey> <title> <sqlSource>


Description

    Show an analysis

Parameters

    cmId:   The database connection id - string - required
    jsonArrayAlgoId:   The JSON array of the selected algorithm ids - string - required
    algoKey:   The selected algorithm key - string - required
    fieldKey:   The selected field key - string - required
    title:   The title - string - required
    sqlSource:   The select query (origin) - string - required
admin
json load "dq" "[]"; json load "dq_name" "{}"; json iobject "dq_name" / field "name" STR; json iobject "dq_name" / algo "{}" OBJ; json iobject "dq_name" /algo "is_float" (mql { type is_float [T_name]; }) STR; json iobject "dq_name" /algo "is_null" (mql { is null [T_name]; }) STR; json iobject "dq_name" /algo "is_small_int" (mql { type is_small_int [T_name]; }) STR; json iobject "dq_name" /algo "is_valid_timestamp" (mql { type is_valid_timestamp [T_name] "yyyy-MM-dd HH:mm:ss"; }) STR; json iarray "dq" / (json doc "dq_name") OBJ; dq analyse show "demo_cm_mysql" (json doc "dq") "is_float" "name" "products" "SELECT * FROM `products` LIMIT 0, 500";
mentdb
In editor ...
Used in use cases (2 recipes)

    CRUD & data API developer › Data quality check on the table
    Machine learning engineer › Audit a database column with a custom rule
Often used with

    dq algorithm exist  ·  dq algorithm get  ·  dq algorithm remove  ·  dq algorithm set  ·  dq generate  ·  json doc  ·  mql  ·  type is_matches_regex