ETL designer
etl design_sourcesetl design_transformsetl generateetl design_getetl generateetl space_saveetl space_getetl space_listetl space_viewetl space_deleteetl manual_checketl runetl mark_startetl join_armetl mark_doneetl join_readyetl join_readyetl error_marketl marksetl mark_reset

etl design_sources


Description

    The families that can be dropped as source or destination: rows {kind, family, cm_type, label, category, source, destination, source_needs, destination_needs, formats}. A field item is {name, label, kind (text, query, json, bool, choice, columns), required, default, values, help}. Only the families whose functions exist on this server are listed

admin
etl design_sources
mentdb
[{"kind": "files", "family": "file", "cm_type": "file", "label": "Local files", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Relative to the folder of the file connection (empty: that folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Relative to the folder of the file connection (empty: that folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "ftp", "cm_type": "ftp", "label": "FTP", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "ftps", "cm_type": "ftps", "label": "FTPS", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "sftp", "cm_type": "sftp", "label": "SFTP", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "ssh", "cm_type": "ssh", "label": "SSH (scp)", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "cifs", "cm_type": "cifs", "label": "CIFS / SMB share", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "mentdb", "cm_type": "mentdb", "label": "Remote MentDB server (tunnel)", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "s3", "cm_type": "config", "label": "S3 (AWS, MinIO, OVH...)", "category": "Cloud storage", "source": 1, "destination": 1, "source_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "azure_blob", "cm_type": "config", "label": "Azure Blob", "category": "Cloud storage", "source": 1, "destination": 1, "source_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "gcs", "cm_type": "config", "label": "Google Cloud Storage", "category": "Cloud storage", "source": 1, "destination": 1, "source_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "sql", "family": "db", "cm_type": "db", "label": "SQL database", "category": "Databases", "source": 1, "destination": 1, "source_needs": [{"name": "query", "label": "SELECT query", "kind": "query", "required": true}], "destination_needs": [{"name": "table", "label": "Table", "kind": "text", "required": true}, {"name": "mode", "label": "Mode", "kind": "choice", "required": false, "default": "insert", "values": ["insert", "upsert"]}, {"name": "keys", "label": "Key columns (upsert)", "kind": "columns", "required": false}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of sql bulk_insert / bulk_upsert (batch_size...)"}]}, {"kind": "rows", "family": "mongodb", "cm_type": "config", "label": "MongoDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'mongodb find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'mongodb find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'mongodb insert'"}]}, {"kind": "rows", "family": "couchdb", "cm_type": "config", "label": "CouchDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'couchdb find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'couchdb find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'couchdb insert'"}]}, {"kind": "rows", "family": "couchbase", "cm_type": "config", "label": "Couchbase", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'couchbase find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'couchbase find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'couchbase insert'"}]}, {"kind": "rows", "family": "surreal", "cm_type": "config", "label": "SurrealDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'surreal find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'surreal find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'surreal insert'"}]}, {"kind": "rows", "family": "ravendb", "cm_type": "config", "label": "RavenDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'ravendb find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'ravendb find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'ravendb insert'"}]}, {"kind": "rows", "family": "firestore", "cm_type": "config", "label": "Firestore", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'firestore find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'firestore find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'firestore insert'"}]}, {"kind": "rows", "family": "cosmos", "cm_type": "config", "label": "Cosmos DB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'cosmos find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'cosmos find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'cosmos insert'"}]}, {"kind": "rows", "family": "dynamodb", "cm_type": "config", "label": "DynamoDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'dynamodb find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'dynamodb find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}]}, {"kind": "rows", "family": "cassandra", "cm_type": "config", "label": "Cassandra / ScyllaDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'cassandra find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'cassandra find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'cassandra insert'"}]}, {"kind": "rows", "family": "hbase", "cm_type": "config", "label": "HBase", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'hbase find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'hbase find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'hbase insert'"}]}, {"kind": "rows", "family": "elastic", "cm_type": "config", "label": "Elasticsearch / OpenSearch", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'elastic find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'elastic find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'elastic insert'"}]}, {"kind": "rows", "family": "solr", "cm_type": "config", "label": "Solr", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'solr find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'solr find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'solr insert'"}]}, {"kind": "rows", "family": "meili", "cm_type": "config", "label": "Meilisearch", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'meili find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'meili find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'meili insert'"}]}, {"kind": "rows", "family": "typesense", "cm_type": "config", "label": "Typesense", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'typesense find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'typesense find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'typesense insert'"}]}, {"kind": "rows", "family": "qdrant", "cm_type": "config", "label": "Qdrant", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'qdrant find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'qdrant find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'qdrant insert'"}]}, {"kind": "rows", "family": "chroma", "cm_type": "config", "label": "Chroma", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'chroma find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'chroma find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'chroma insert'"}]}, {"kind": "rows", "family": "weaviate", "cm_type": "config", "label": "Weaviate", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'weaviate find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'weaviate find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'weaviate insert'"}]}, {"kind": "rows", "family": "odata", "cm_type": "config", "label": "OData (SAP, Dynamics, Dataverse...)", "category": "Applications", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'odata find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'odata find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'odata insert'"}]}, {"kind": "messages", "family": "kafka", "cm_type": "config", "label": "Kafka", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'kafka consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'kafka produce_batch'"}]}, {"kind": "messages", "family": "amqp", "cm_type": "config", "label": "AMQP (RabbitMQ...)", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Queue", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'amqp consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Queue / routing key", "kind": "text", "required": true}, {"name": "exchange", "label": "Exchange", "kind": "text", "required": false, "default": "", "help": "Empty: the default exchange (the routing key is the queue)"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'amqp publish_batch'"}]}, {"kind": "messages", "family": "mqtt", "cm_type": "config", "label": "MQTT", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'mqtt receive' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'mqtt publish_batch'"}]}, {"kind": "messages", "family": "sqs", "cm_type": "config", "label": "Amazon SQS", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Queue", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'sqs consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Queue / routing key", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'sqs publish_batch'"}]}, {"kind": "messages", "family": "nats", "cm_type": "config", "label": "NATS", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Subject", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'nats consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Subject", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'nats publish_batch'"}]}, {"kind": "messages", "family": "pubsub", "cm_type": "config", "label": "Google Pub/Sub", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Subscription", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'pubsub consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'pubsub publish_batch'"}]}]

etl design_transforms


Description

    The transformations of the middle block: rows {type, label, function, params: [{name, label, kind}]}

admin
etl design_transforms
mentdb
[{"type": "select", "label": "Keep some columns, in this order", "function": "table select", "params": [{"name": "columns", "label": "Columns", "kind": "columns"}]}, {"type": "rename", "label": "Rename columns", "function": "json to_rows (rename)", "params": [{"name": "map", "label": "Old name -> new name", "kind": "map"}]}, {"type": "where", "label": "Keep the rows that match", "function": "table where", "params": [{"name": "condition", "label": "Conditions: {\"field\": value} or {\"field\": {\"gt\": 10}} (eq, ne, gt, ge, lt, le, in, not_in, contains, starts_with, null)", "kind": "condition"}]}, {"type": "distinct", "label": "Distinct combinations of columns", "function": "table distinct", "params": [{"name": "columns", "label": "Columns", "kind": "columns"}]}, {"type": "sort", "label": "Sort the rows", "function": "table sort", "params": [{"name": "columns", "label": "Columns: [\"id\"] or [[\"amount\", \"desc\"]]", "kind": "order"}]}, {"type": "top", "label": "The first rows (per group)", "function": "table top", "params": [{"name": "n", "label": "Number of rows", "kind": "number"}, {"name": "columns", "label": "Order: [[\"amount\", \"desc\"]]", "kind": "order"}, {"name": "groups", "label": "Group columns (optional)", "kind": "columns"}]}, {"type": "mql", "label": "Free MQL on [rows] (must set [rows])", "function": "mql", "params": [{"name": "code", "label": "MQL code", "kind": "code"}]}, {"type": "script", "label": "Call a script with [rows] (returns the new rows)", "function": "execute", "params": [{"name": "name", "label": "Script name", "kind": "script"}]}]

etl generate <design>


Description

    The MQL of the whole flow (group, 5 script merge, grants), not run. With a session the family of each block is read from its connection; without one (or to check it) the block gives "family". Clear errors for a bad name, a kind that does not match the connection, an unknown field, a value that cannot be written in MQL

Parameters

    design:   The design JSON: {name, prefix, group (optional), source, transforms, destination} - String - required
admin
etl generate "{\"name\": \"orders_csv_to_json\", \"prefix\": \"demo.orders.csv_to_json\", \"source\": {\"cm\": \"demo_in\", \"kind\": \"files\", \"family\": \"file\", \"filter\": \"*.csv\", \"format\": \"csv\", \"delete_source\": true}, \"transforms\": [{\"type\": \"select\", \"columns\": [\"id\", \"client\", \"total\"]}, {\"type\": \"where\", \"condition\": {\"total\": {\"gt\": 100}}}, {\"type\": \"rename\", \"map\": {\"total\": \"amount\"}}, {\"type\": \"sort\", \"columns\": [[\"amount\", \"desc\"]]}], \"destination\": {\"cm\": \"demo_out\", \"kind\": \"files\", \"family\": \"file\", \"format\": \"json\", \"file_name\": \"orders_[SOURCE]_[FLOW_PID].json\"}}"
mentdb
# ETL flow 'orders_csv_to_json' generated by the ETL designer (etl generate): file (files) -> 4 transformation(s) -> file (files); if (not (group exist "demo_orders_csv_to_json")) { group add "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_1_source_connect.exe" true 5 (param ) "Connect to the source (orders_csv_to_json)" { # ETL_DESIGN eyJuYW1lIjogIm9yZGVyc19jc3ZfdG9fanNvbiIsICJwcmVmaXgiOiAiZGVtby5vcmRlcnMuY3N2X3RvX2pzb24iLCAic291cmNlIjogeyJjbSI6ICJkZW1vX2luIiwgImtpbmQiOiAiZmlsZXMiLCAiZmFtaWx5IjogImZpbGUiLCAiZmlsdGVyIjogIiouY3N2IiwgImZvcm1hdCI6ICJjc3YiLCAiZGVsZXRlX3NvdXJjZSI6IHRydWV9LCAidHJhbnNmb3JtcyI6IFt7InR5cGUiOiAic2VsZWN0IiwgImNvbHVtbnMiOiBbImlkIiwgImNsaWVudCIsICJ0b3RhbCJdfSwgeyJ0eXBlIjogIndoZXJlIiwgImNvbmRpdGlvbiI6IHsidG90YWwiOiB7Imd0IjogMTAwfX19LCB7InR5cGUiOiAicmVuYW1lIiwgIm1hcCI6IHsidG90YWwiOiAiYW1vdW50In19LCB7InR5cGUiOiAic29ydCIsICJjb2x1bW5zIjogW1siYW1vdW50IiwgImRlc2MiXV19XSwgImRlc3RpbmF0aW9uIjogeyJjbSI6ICJkZW1vX291dCIsICJraW5kIjogImZpbGVzIiwgImZhbWlseSI6ICJmaWxlIiwgImZvcm1hdCI6ICJqc29uIiwgImZpbGVfbmFtZSI6ICJvcmRlcnNfW1NPVVJDRV1fW0ZMT1dfUElEXS5qc29uIn19; # ETL_HASH 32a0d0eee72d3556566dbb601be38a3a32b70a03d6bf1768165befc5b077cc64 ec25d3e9117b87d9 667358107360f60f 8595e42b6fc97fc0 9b180e46364afc82 903b217d96d2685a; # WORK FOLDER: every file of a run is in home/[FLOW_PID]/, [FLOW_PID] being the PID of this step 1; # home/[FLOW_PID]/<file>: a file extracted at step 2, extract.json for a sql, rows or messages source; # home/[FLOW_PID]/<file>.rows.json: the rows after step 3; # home/[FLOW_PID]/<file>.out/<name>: the file built at step 5 for a files destination; # Step 5 deletes the files of its file once loaded, then home/[FLOW_PID]/ when it is empty: all the files are loaded; # step 2 deletes it when there is nothing to extract. After an error the files stay for the replay of the step; # a replay of step 1 is a new run: a new PID, a new folder; # CONFIGURATION ; -> "[CONF_NAME_OF_THE_FLOW]" "orders_csv_to_json"; -> "[CONF_SOURCE_CM]" "demo_in"; # INITIALIZATION ; -> "[FLOW_PID]" [PID]; -> "[etl_connected]" 0; # HANDLE ; try { # Flow initialization; stack flow_init [FLOW_PID] [CONF_NAME_OF_THE_FLOW] "{}"; stack flow_step [FLOW_PID] 1 "source_connect..."; # Start mark: the last run of this flow (etl space_view); try {etl mark_start "demo.orders.csv_to_json" [FLOW_PID];} {log write (concat "ETL start mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; # Local connection: nothing to open; log write "Source: local files" OK null null; #Step 1 is valid; stack flow_step [FLOW_PID] 1 "source_connect_ok"; -> "[etl_connected]" 1; include "demo.orders.csv_to_json.step_2_extract.exe"; } { #Step 1 is not valid (an error of step 2 is already marked by step 2); if (== [etl_connected] 0) {stack flow_step [FLOW_PID] 1 "source_connect_KO";}; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_1_source_connect.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_1_source_connect.exe" "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_2_extract.exe" true 1 (param ) "Extract from the source (orders_csv_to_json)" { # CONFIGURATION ; json load "etl_cm_source" {cm get [CONF_SOURCE_CM];}; -> "[CONF_DIR_SOURCE]" (json select "etl_cm_source" "/localDir"); json unload "etl_cm_source"; -> "[CONF_SOURCE_FILTER]" ".*[.]csv"; # HANDLE ; try { # Mark the flow as step 2; stack flow_step [FLOW_PID] 2 "extract..."; # Create the local directory PID; if (not (file exist (concat "home/" [FLOW_PID]))) {file mkdir (concat "home/" [FLOW_PID]);}; # Get files into PID directory; log write (concat "Get files from '" [CONF_DIR_SOURCE] "' (" [CONF_SOURCE_FILTER] ").") OK null null; json load "etl_names" (file dir_list_regex [CONF_DIR_SOURCE] [CONF_SOURCE_FILTER] true false); json load "valid_files" "[]"; json parse_array "etl_names" "/" "[etl_name]" { if (and (string matches [etl_name] "[^/\\\p{Cntrl}]+") (not (string matches [etl_name] "[.][.]?"))) {json iarray "valid_files" / [etl_name] STR;} {log write (concat "Skip the file name '" (string regexp_replace [etl_name] "\p{Cntrl}" "?") "' (not a plain file name).") OK null null;}; }; json unload "etl_names"; json parse_array "valid_files" "/" "[filename]" { file copy_file (concat [CONF_DIR_SOURCE] "/" [filename]) (concat "home/" [FLOW_PID] "/" [filename]); log write (concat "Get file 'home/" [FLOW_PID] "/" [filename] "'.") OK null null; }; # Delete the source files; json parse_array "valid_files" "/" "[filename]" { file delete (concat [CONF_DIR_SOURCE] "/" [filename]); log write (concat "Remove source file '" [CONF_DIR_SOURCE] "/" [filename] "'.") OK null null; }; } { #Step 2 is not valid; stack flow_step [FLOW_PID] 2 "extract_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; log write (concat (json count "valid_files" /) " file(s) to transform.") OK null null; if (> (json count "valid_files" /) 0) { #Step 2 is valid; stack flow_step [FLOW_PID] 2 (concat "extract_ok_" (json count "valid_files" /) "_files"); json parse_array "valid_files" "/" "[filename]" { stack (date now) "demo.orders.csv_to_json.step_3_transform.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [filename]; }; } { #Step 2 is valid; stack flow_step [FLOW_PID] 2 "extract_ok_zero_file"; # Nothing to transform; file delete (concat "home/" [FLOW_PID]); # END OF THE RUN (nothing to extract counts as a run ended OK): done mark for the joins; try {etl mark_done "demo.orders.csv_to_json";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_2_extract.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_2_extract.exe" "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_3_transform.exe" true 1 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") ) "Transform the rows (orders_csv_to_json)" { # CONFIGURATION ; -> "[CONF_SOURCE_FORMAT_OPTIONS]" "{}"; -> "[CONF_TRANSFORMED_FILENAME]" (concat [filename] ".rows.json"); # HANDLE ; try { # Mark the flow as step 3; stack flow_step [FLOW_PID] 3 "transform..."; -> "[etl_file]" (concat "home/" [FLOW_PID] "/" [filename]); log write (concat "Read the rows of '" [etl_file] "' (csv).") OK null null; -> "[rows]" (csv read [etl_file] [CONF_SOURCE_FORMAT_OPTIONS]); # Transformation 1: select; -> "[rows]" (table select [rows] "[\"id\", \"client\", \"total\"]"); # Transformation 2: where; -> "[rows]" (table where [rows] "{\"total\": {\"gt\": 100}}"); # Transformation 3: rename; -> "[rows]" (json to_rows [rows] "{\"rename\": {\"total\": \"amount\"}}"); # Transformation 4: sort; -> "[rows]" (table sort [rows] "[[\"amount\", \"desc\"]]"); json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) after the transformations.") OK null null; json unload "etl_rows"; file create (concat "home/" [FLOW_PID] "/" [CONF_TRANSFORMED_FILENAME]) [rows] "UTF-8"; #Step 3 is valid; stack flow_step [FLOW_PID] 3 "transform_ok"; stack (date now) "demo.orders.csv_to_json.step_4_destination_connect.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [CONF_TRANSFORMED_FILENAME] "[source_filename]" [filename]; } { #Step 3 is not valid; stack flow_step [FLOW_PID] 3 "transform_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_3_transform.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_3_transform.exe" "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_4_destination_connect.exe" true 5 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") (var "[source_filename]" {true} "The file extracted at step 2" is_null:true is_empty:true "file.xxx") ) "Connect to the destination (orders_csv_to_json)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "demo_out"; -> "[etl_step5]" 0; # HANDLE ; try { # Flow step 4; stack flow_step [FLOW_PID] 4 "destination_connect..."; # Local connection: nothing to open; log write "Destination: local files" OK null null; #Step 4 is valid; stack flow_step [FLOW_PID] 4 "destination_connect_ok"; include "demo.orders.csv_to_json.step_5_load.exe"; } { #Step 4 is not valid (an error of step 5 is already marked by step 5); if (not (== [etl_step5] 1)) {stack flow_step [FLOW_PID] 4 "destination_connect_KO";}; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_4_destination_connect.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_4_destination_connect.exe" "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_5_load.exe" true 1 (param ) "Load to the destination (orders_csv_to_json)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "demo_out"; json load "etl_cm_destination" {cm get [CONF_DESTINATION_CM];}; -> "[CONF_DIR_DESTINATION]" (json select "etl_cm_destination" "/localDir"); json unload "etl_cm_destination"; -> "[CONF_DESTINATION_FORMAT_OPTIONS]" "{}"; -> "[CONF_DESTINATION_FILENAME]" (concat "orders_" (string regexp_replace [source_filename] "\.[^.]*$" "") "_" [FLOW_PID] ".json"); -> "[etl_step5]" 1; -> "[etl_run_done]" 0; # HANDLE ; try { # Mark the flow as step 5; stack flow_step [FLOW_PID] 5 "load..."; -> "[rows]" (file load (concat "home/" [FLOW_PID] "/" [filename]) "UTF-8"); if (not (and (string matches [CONF_DESTINATION_FILENAME] "[^/\\\p{Cntrl}]+") (not (string matches [CONF_DESTINATION_FILENAME] "[.][.]?")))) {exception (1) (concat "The destination file name is not a plain file name: " (string regexp_replace [CONF_DESTINATION_FILENAME] "\p{Cntrl}" "?"));}; # The file is built in a folder of its own: home/[FLOW_PID]/<source file>.out/; -> "[etl_out_dir]" (concat "home/" [FLOW_PID] "/" [source_filename] ".out"); if (not (file exist [etl_out_dir])) {file mkdir [etl_out_dir];}; -> "[etl_out]" (concat [etl_out_dir] "/" [CONF_DESTINATION_FILENAME]); log write (concat "Build the file '" [etl_out] "' (json).") OK null null; file create [etl_out] [rows] "UTF-8"; log write (concat "Send '" [etl_out] "' to '" [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME] "'...") OK null null; if (not (file exist [CONF_DIR_DESTINATION])) {file mkdir [CONF_DIR_DESTINATION];}; file copy_file [etl_out] (concat [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME]); if (not (file exist (concat [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME]))) { exception (1) (concat "Cannot copy the file '" [CONF_DESTINATION_FILENAME] "'."); }; file delete [etl_out]; try {file delete [etl_out_dir];} {} "[err]"; log write "Sent." OK null null; # Remove the work files of this file, then the folder of the run when it is empty (all its files are loaded); try {file delete (concat "home/" [FLOW_PID] "/" [filename]);} {} "[err]"; try {file delete (concat "home/" [FLOW_PID] "/" [source_filename]);} {} "[err]"; try { json load "etl_left" (file dir_list_regex (concat "home/" [FLOW_PID]) ".*" true true); if (== (json count "etl_left" /) 0) {file delete (concat "home/" [FLOW_PID]); -> "[etl_run_done]" 1;}; json unload "etl_left"; } {} "[err]"; #Step 5 is valid; stack flow_step [FLOW_PID] 5 "load_ok"; # END OF THE RUN (its last file is loaded: the folder of the run is deleted): done mark for the joins; if (== [etl_run_done] 1) { try {etl mark_done "demo.orders.csv_to_json";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } { try {file delete [etl_out];} {} "[err]"; try {file delete [etl_out_dir];} {} "[err]"; #Step 5 is not valid; stack flow_step [FLOW_PID] 5 "load_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_5_load.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_5_load.exe" "demo_orders_csv_to_json"; }; # No join: step 1 runs as soon as it is in the stack; script set delay "demo.orders.csv_to_json.step_1_source_connect.exe" 0 day {1};
a SELECT on the internal database -> free MQL -> another table
admin
etl generate "{\"name\": \"orders_sql_to_sql\", \"prefix\": \"demo.orders.sql_to_sql\", \"source\": {\"cm\": \"MENTDB\", \"kind\": \"sql\", \"family\": \"db\", \"query\": \"select id, client, total from etl_orders where status = 'NEW'\"}, \"transforms\": [{\"type\": \"rename\", \"map\": {\"total\": \"amount\"}}, {\"type\": \"mql\", \"code\": \"-> \\"[rows]\\" (table where [rows] \\"{\\\\"amount\\\\": {\\\\"ge\\\\": 100}}\\");\"}], \"destination\": {\"cm\": \"MENTDB\", \"kind\": \"sql\", \"family\": \"db\", \"table\": \"etl_orders_new\", \"mode\": \"insert\"}}"
mentdb
# ETL flow 'orders_sql_to_sql' generated by the ETL designer (etl generate): db (sql) -> 2 transformation(s) -> db (sql); if (not (group exist "demo_orders_sql_to_sql")) { group add "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_1_source_connect.exe" true 5 (param ) "Connect to the source (orders_sql_to_sql)" { # ETL_DESIGN eyJuYW1lIjogIm9yZGVyc19zcWxfdG9fc3FsIiwgInByZWZpeCI6ICJkZW1vLm9yZGVycy5zcWxfdG9fc3FsIiwgInNvdXJjZSI6IHsiY20iOiAiTUVOVERCIiwgImtpbmQiOiAic3FsIiwgImZhbWlseSI6ICJkYiIsICJxdWVyeSI6ICJzZWxlY3QgaWQsIGNsaWVudCwgdG90YWwgZnJvbSBldGxfb3JkZXJzIHdoZXJlIHN0YXR1cyA9ICdORVcnIn0sICJ0cmFuc2Zvcm1zIjogW3sidHlwZSI6ICJyZW5hbWUiLCAibWFwIjogeyJ0b3RhbCI6ICJhbW91bnQifX0sIHsidHlwZSI6ICJtcWwiLCAiY29kZSI6ICItPiBcIltyb3dzXVwiICh0YWJsZSB3aGVyZSBbcm93c10gXCJ7XFxcImFtb3VudFxcXCI6IHtcXFwiZ2VcXFwiOiAxMDB9fVwiKTsifV0sICJkZXN0aW5hdGlvbiI6IHsiY20iOiAiTUVOVERCIiwgImtpbmQiOiAic3FsIiwgImZhbWlseSI6ICJkYiIsICJ0YWJsZSI6ICJldGxfb3JkZXJzX25ldyIsICJtb2RlIjogImluc2VydCJ9fQ==; # ETL_HASH f5afb9dcb788788ab6e7bd4fe4c6dd04e90398163c22bb1ab72312d60291cc67 c344ced2e4a9f42c 03232d42af1637ec b3a39ae6b1febd23 42b5d33342d896e1 9484834210f778d3; # WORK FOLDER: every file of a run is in home/[FLOW_PID]/, [FLOW_PID] being the PID of this step 1; # home/[FLOW_PID]/<file>: a file extracted at step 2, extract.json for a sql, rows or messages source; # home/[FLOW_PID]/<file>.rows.json: the rows after step 3; # home/[FLOW_PID]/<file>.out/<name>: the file built at step 5 for a files destination; # Step 5 deletes the files of its file once loaded, then home/[FLOW_PID]/ when it is empty: all the files are loaded; # step 2 deletes it when there is nothing to extract. After an error the files stay for the replay of the step; # a replay of step 1 is a new run: a new PID, a new folder; # CONFIGURATION ; -> "[CONF_NAME_OF_THE_FLOW]" "orders_sql_to_sql"; -> "[CONF_SOURCE_CM]" "MENTDB"; # INITIALIZATION ; -> "[FLOW_PID]" [PID]; -> "[etl_connected]" 0; # HANDLE ; try { # Flow initialization; stack flow_init [FLOW_PID] [CONF_NAME_OF_THE_FLOW] "{}"; stack flow_step [FLOW_PID] 1 "source_connect..."; # Start mark: the last run of this flow (etl space_view); try {etl mark_start "demo.orders.sql_to_sql" [FLOW_PID];} {log write (concat "ETL start mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; sql connect "session_remote" {cm get [CONF_SOURCE_CM];}; #Step 1 is valid; stack flow_step [FLOW_PID] 1 "source_connect_ok"; -> "[etl_connected]" 1; include "demo.orders.sql_to_sql.step_2_extract.exe"; } { try {sql disconnect "session_remote";} {} "[err]"; #Step 1 is not valid (an error of step 2 is already marked by step 2); if (== [etl_connected] 0) {stack flow_step [FLOW_PID] 1 "source_connect_KO";}; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_1_source_connect.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_1_source_connect.exe" "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_2_extract.exe" true 1 (param ) "Extract from the source (orders_sql_to_sql)" { # CONFIGURATION ; -> "[CONF_SOURCE_QUERY]" "select id, client, total from etl_orders where status = 'NEW'"; # HANDLE ; try { # Mark the flow as step 2; stack flow_step [FLOW_PID] 2 "extract..."; # Create the local directory PID; if (not (file exist (concat "home/" [FLOW_PID]))) {file mkdir (concat "home/" [FLOW_PID]);}; log write "Run the query of the source." OK null null; -> "[rows]" (json to_rows (sql to json "session_remote" "rows" [CONF_SOURCE_QUERY]) "{\"path\": \"$.data\"}"); json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) extracted.") OK null null; if (> (json count "etl_rows" /) 0) { file create (concat "home/" [FLOW_PID] "/extract.json") [rows] "UTF-8"; json load "valid_files" "[\"extract.json\"]"; } { json load "valid_files" "[]"; }; json unload "etl_rows"; sql disconnect "session_remote"; } { try {sql disconnect "session_remote";} {} "[err]"; #Step 2 is not valid; stack flow_step [FLOW_PID] 2 "extract_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; log write (concat (json count "valid_files" /) " file(s) to transform.") OK null null; if (> (json count "valid_files" /) 0) { #Step 2 is valid; stack flow_step [FLOW_PID] 2 (concat "extract_ok_" (json count "valid_files" /) "_files"); json parse_array "valid_files" "/" "[filename]" { stack (date now) "demo.orders.sql_to_sql.step_3_transform.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [filename]; }; } { #Step 2 is valid; stack flow_step [FLOW_PID] 2 "extract_ok_zero_file"; # Nothing to transform; file delete (concat "home/" [FLOW_PID]); # END OF THE RUN (nothing to extract counts as a run ended OK): done mark for the joins; try {etl mark_done "demo.orders.sql_to_sql";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_2_extract.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_2_extract.exe" "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_3_transform.exe" true 1 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") ) "Transform the rows (orders_sql_to_sql)" { # CONFIGURATION ; -> "[CONF_TRANSFORMED_FILENAME]" (concat [filename] ".rows.json"); # HANDLE ; try { # Mark the flow as step 3; stack flow_step [FLOW_PID] 3 "transform..."; -> "[etl_file]" (concat "home/" [FLOW_PID] "/" [filename]); log write (concat "Read the rows of '" [etl_file] "' (rows).") OK null null; -> "[rows]" (file load [etl_file] "UTF-8"); # Transformation 1: rename; -> "[rows]" (json to_rows [rows] "{\"rename\": {\"total\": \"amount\"}}"); # Transformation 2: mql; # Free MQL: it reads and sets [rows]; -> "[rows]" (table where [rows] "{\"amount\": {\"ge\": 100}}"); json load "etl_check" [rows]; json unload "etl_check"; json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) after the transformations.") OK null null; json unload "etl_rows"; file create (concat "home/" [FLOW_PID] "/" [CONF_TRANSFORMED_FILENAME]) [rows] "UTF-8"; #Step 3 is valid; stack flow_step [FLOW_PID] 3 "transform_ok"; stack (date now) "demo.orders.sql_to_sql.step_4_destination_connect.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [CONF_TRANSFORMED_FILENAME] "[source_filename]" [filename]; } { #Step 3 is not valid; stack flow_step [FLOW_PID] 3 "transform_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_3_transform.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_3_transform.exe" "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_4_destination_connect.exe" true 5 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") (var "[source_filename]" {true} "The file extracted at step 2" is_null:true is_empty:true "file.xxx") ) "Connect to the destination (orders_sql_to_sql)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "MENTDB"; -> "[etl_step5]" 0; # HANDLE ; try { # Flow step 4; stack flow_step [FLOW_PID] 4 "destination_connect..."; sql connect "destination_sess" {cm get [CONF_DESTINATION_CM];}; sql auto_commit "destination_sess" false; #Step 4 is valid; stack flow_step [FLOW_PID] 4 "destination_connect_ok"; include "demo.orders.sql_to_sql.step_5_load.exe"; } { try {sql rollback "destination_sess";} {} "[err]"; try {sql disconnect "destination_sess";} {} "[err]"; #Step 4 is not valid (an error of step 5 is already marked by step 5); if (not (== [etl_step5] 1)) {stack flow_step [FLOW_PID] 4 "destination_connect_KO";}; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_4_destination_connect.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_4_destination_connect.exe" "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_5_load.exe" true 1 (param ) "Load to the destination (orders_sql_to_sql)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "MENTDB"; -> "[CONF_DESTINATION_TABLE]" "etl_orders_new"; -> "[CONF_DESTINATION_OPTIONS]" "{}"; -> "[etl_step5]" 1; -> "[etl_run_done]" 0; # HANDLE ; try { # Mark the flow as step 5; stack flow_step [FLOW_PID] 5 "load..."; -> "[rows]" (file load (concat "home/" [FLOW_PID] "/" [filename]) "UTF-8"); log write (concat "Load the rows into the table '" [CONF_DESTINATION_TABLE] "'...") OK null null; json load "etl_rows" [rows]; if (> (json count "etl_rows" /) 0) { -> "[etl_result]" (sql bulk_insert "destination_sess" [CONF_DESTINATION_TABLE] [rows] [CONF_DESTINATION_OPTIONS]); log write (concat "Loaded: " [etl_result]) OK null null; } { log write "No row to load." OK null null; }; json unload "etl_rows"; sql commit "destination_sess"; sql disconnect "destination_sess"; # Remove the work files of this file, then the folder of the run when it is empty (all its files are loaded); try {file delete (concat "home/" [FLOW_PID] "/" [filename]);} {} "[err]"; try {file delete (concat "home/" [FLOW_PID] "/" [source_filename]);} {} "[err]"; try { json load "etl_left" (file dir_list_regex (concat "home/" [FLOW_PID]) ".*" true true); if (== (json count "etl_left" /) 0) {file delete (concat "home/" [FLOW_PID]); -> "[etl_run_done]" 1;}; json unload "etl_left"; } {} "[err]"; #Step 5 is valid; stack flow_step [FLOW_PID] 5 "load_ok"; # END OF THE RUN (its last file is loaded: the folder of the run is deleted): done mark for the joins; if (== [etl_run_done] 1) { try {etl mark_done "demo.orders.sql_to_sql";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } { try {sql rollback "destination_sess";} {} "[err]"; try {sql disconnect "destination_sess";} {} "[err]"; #Step 5 is not valid; stack flow_step [FLOW_PID] 5 "load_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_5_load.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_5_load.exe" "demo_orders_sql_to_sql"; }; # No join: step 1 runs as soon as it is in the stack; script set delay "demo.orders.sql_to_sql.step_1_source_connect.exe" 0 day {1};
a kind that does not match the connection
admin
etl generate "{\"name\": \"bad\", \"prefix\": \"demo.bad\", \"source\": {\"cm\": \"demo_in\", \"kind\": \"sql\", \"family\": \"file\", \"filter\": \"*.csv\", \"format\": \"csv\"}, \"destination\": {\"cm\": \"demo_out\", \"family\": \"file\", \"format\": \"json\"}}"
mentdb
ERROR: Sorry, source.kind is 'sql' but the connection 'demo_in' (file) is of kind 'files'.
free MQL that would close the script is refused
admin
etl generate "{\"name\": \"bad\", \"prefix\": \"demo.bad\", \"source\": {\"cm\": \"demo_in\", \"family\": \"file\", \"filter\": \"*.csv\", \"format\": \"csv\"}, \"transforms\": [{\"type\": \"mql\", \"code\": \"} script delete \\"x\\"; {\"}], \"destination\": {\"cm\": \"demo_out\", \"family\": \"file\", \"format\": \"json\"}}"
mentdb
ERROR: Sorry, transforms[0].code: the braces { } are not balanced.

etl design_get <prefix>


Description

    The design kept in step 1 of a generated flow, to reopen it in the designer

Parameters

    prefix:   The prefix of the scripts of the flow - String - required
admin
etl design_get "demo.orders.csv_to_json"
mentdb
{"name": "orders_csv_to_json", "prefix": "demo.orders.csv_to_json", "source": {"cm": "demo_in", "kind": "files", "family": "file", "filter": "*.csv", "format": "csv", "delete_source": true}, "transforms": [{"type": "select", "columns": ["id", "client", "total"]}, {"type": "where", "condition": {"total": {"gt": 100}}}, {"type": "rename", "map": {"total": "amount"}}, {"type": "sort", "columns": [["amount", "desc"]]}], "destination": {"cm": "demo_out", "kind": "files", "family": "file", "format": "json", "file_name": "orders_[SOURCE]_[FLOW_PID].json"}}

etl generate <design>


Description

    Links (ETL spaces): next and wait_for in the design add to the MQL the start and done marks, the flows put in the stack at the end of a run (success) or on an error, the delay of step 1 and the arming of the join. A success link that makes a cycle is refused

Parameters

    design:   The design JSON with next [{prefix, on: success|error}] and/or wait_for [prefixes] - String - required
admin
etl generate "{\"name\": \"orders_join\", \"prefix\": \"demo.orders.join\", \"wait_for\": [\"demo.orders.csv_to_json\", \"demo.orders.sql_to_sql\"], \"next\": [{\"prefix\": \"demo.ops.alert\", \"on\": \"error\"}], \"source\": {\"cm\": \"MENTDB\", \"kind\": \"sql\", \"family\": \"db\", \"query\": \"select id, amount from etl_orders_new\"}, \"destination\": {\"cm\": \"demo_out\", \"kind\": \"files\", \"family\": \"file\", \"format\": \"csv\", \"file_name\": \"orders_[DATE].csv\"}}"
mentdb
# ETL flow 'orders_join' generated by the ETL designer (etl generate): db (sql) -> 0 transformation(s) -> file (files); if (not (group exist "demo_orders_join")) { group add "demo_orders_join"; }; script merge "demo.orders.join.step_1_source_connect.exe" true 5 (param ) "Connect to the source (orders_join)" { # ETL_DESIGN eyJuYW1lIjogIm9yZGVyc19qb2luIiwgInByZWZpeCI6ICJkZW1vLm9yZGVycy5qb2luIiwgIndhaXRfZm9yIjogWyJkZW1vLm9yZGVycy5jc3ZfdG9fanNvbiIsICJkZW1vLm9yZGVycy5zcWxfdG9fc3FsIl0sICJuZXh0IjogW3sicHJlZml4IjogImRlbW8ub3BzLmFsZXJ0IiwgIm9uIjogImVycm9yIn1dLCAic291cmNlIjogeyJjbSI6ICJNRU5UREIiLCAia2luZCI6ICJzcWwiLCAiZmFtaWx5IjogImRiIiwgInF1ZXJ5IjogInNlbGVjdCBpZCwgYW1vdW50IGZyb20gZXRsX29yZGVyc19uZXcifSwgImRlc3RpbmF0aW9uIjogeyJjbSI6ICJkZW1vX291dCIsICJraW5kIjogImZpbGVzIiwgImZhbWlseSI6ICJmaWxlIiwgImZvcm1hdCI6ICJjc3YiLCAiZmlsZV9uYW1lIjogIm9yZGVyc19bREFURV0uY3N2In19; # ETL_HASH 886a918bc49d868ad5bece067dab940bb9b3bfebfcc4b15000157525747cb51c 9f4a0854b610fc6f 2a1d1d0733afac73 9cb3bdcb2c1cd001 5a8d08b33a2d46fd 0eeb259bda4911cd; # WORK FOLDER: every file of a run is in home/[FLOW_PID]/, [FLOW_PID] being the PID of this step 1; # home/[FLOW_PID]/<file>: a file extracted at step 2, extract.json for a sql, rows or messages source; # home/[FLOW_PID]/<file>.rows.json: the rows after step 3; # home/[FLOW_PID]/<file>.out/<name>: the file built at step 5 for a files destination; # Step 5 deletes the files of its file once loaded, then home/[FLOW_PID]/ when it is empty: all the files are loaded; # step 2 deletes it when there is nothing to extract. After an error the files stay for the replay of the step; # a replay of step 1 is a new run: a new PID, a new folder; # CONFIGURATION ; -> "[CONF_NAME_OF_THE_FLOW]" "orders_join"; -> "[CONF_SOURCE_CM]" "MENTDB"; # INITIALIZATION ; -> "[FLOW_PID]" [PID]; -> "[etl_connected]" 0; -> "[etl_err_next]" 0; # HANDLE ; try { # Flow initialization; stack flow_init [FLOW_PID] [CONF_NAME_OF_THE_FLOW] "{}"; stack flow_step [FLOW_PID] 1 "source_connect..."; # Start mark: the last run of this flow (etl space_view); try {etl mark_start "demo.orders.join" [FLOW_PID];} {log write (concat "ETL start mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; sql connect "session_remote" {cm get [CONF_SOURCE_CM];}; #Step 1 is valid; stack flow_step [FLOW_PID] 1 "source_connect_ok"; -> "[etl_connected]" 1; include "demo.orders.join.step_2_extract.exe"; } { try {sql disconnect "session_remote";} {} "[err]"; #Step 1 is not valid (an error of step 2 is already marked by step 2); if (== [etl_connected] 0) {stack flow_step [FLOW_PID] 1 "source_connect_KO";}; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_1_source_connect.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_1_source_connect.exe" "demo_orders_join"; }; script merge "demo.orders.join.step_2_extract.exe" true 1 (param ) "Extract from the source (orders_join)" { # CONFIGURATION ; -> "[CONF_SOURCE_QUERY]" "select id, amount from etl_orders_new"; # HANDLE ; try { # Mark the flow as step 2; stack flow_step [FLOW_PID] 2 "extract..."; # Create the local directory PID; if (not (file exist (concat "home/" [FLOW_PID]))) {file mkdir (concat "home/" [FLOW_PID]);}; log write "Run the query of the source." OK null null; -> "[rows]" (json to_rows (sql to json "session_remote" "rows" [CONF_SOURCE_QUERY]) "{\"path\": \"$.data\"}"); json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) extracted.") OK null null; if (> (json count "etl_rows" /) 0) { file create (concat "home/" [FLOW_PID] "/extract.json") [rows] "UTF-8"; json load "valid_files" "[\"extract.json\"]"; } { json load "valid_files" "[]"; }; json unload "etl_rows"; sql disconnect "session_remote"; } { try {sql disconnect "session_remote";} {} "[err]"; #Step 2 is not valid; stack flow_step [FLOW_PID] 2 "extract_KO"; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; log write (concat (json count "valid_files" /) " file(s) to transform.") OK null null; if (> (json count "valid_files" /) 0) { #Step 2 is valid; stack flow_step [FLOW_PID] 2 (concat "extract_ok_" (json count "valid_files" /) "_files"); json parse_array "valid_files" "/" "[filename]" { stack (date now) "demo.orders.join.step_3_transform.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [filename]; }; } { #Step 2 is valid; stack flow_step [FLOW_PID] 2 "extract_ok_zero_file"; # Nothing to transform; file delete (concat "home/" [FLOW_PID]); # END OF THE RUN (nothing to extract counts as a run ended OK): done mark for the joins; try {etl mark_done "demo.orders.join";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_2_extract.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_2_extract.exe" "demo_orders_join"; }; script merge "demo.orders.join.step_3_transform.exe" true 1 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") ) "Transform the rows (orders_join)" { # CONFIGURATION ; -> "[CONF_TRANSFORMED_FILENAME]" (concat [filename] ".rows.json"); -> "[etl_err_next]" 0; # HANDLE ; try { # Mark the flow as step 3; stack flow_step [FLOW_PID] 3 "transform..."; -> "[etl_file]" (concat "home/" [FLOW_PID] "/" [filename]); log write (concat "Read the rows of '" [etl_file] "' (rows).") OK null null; -> "[rows]" (file load [etl_file] "UTF-8"); json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) after the transformations.") OK null null; json unload "etl_rows"; file create (concat "home/" [FLOW_PID] "/" [CONF_TRANSFORMED_FILENAME]) [rows] "UTF-8"; #Step 3 is valid; stack flow_step [FLOW_PID] 3 "transform_ok"; stack (date now) "demo.orders.join.step_4_destination_connect.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [CONF_TRANSFORMED_FILENAME] "[source_filename]" [filename]; } { #Step 3 is not valid; stack flow_step [FLOW_PID] 3 "transform_KO"; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_3_transform.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_3_transform.exe" "demo_orders_join"; }; script merge "demo.orders.join.step_4_destination_connect.exe" true 5 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") (var "[source_filename]" {true} "The file extracted at step 2" is_null:true is_empty:true "file.xxx") ) "Connect to the destination (orders_join)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "demo_out"; -> "[etl_step5]" 0; -> "[etl_err_next]" 0; # HANDLE ; try { # Flow step 4; stack flow_step [FLOW_PID] 4 "destination_connect..."; # Local connection: nothing to open; log write "Destination: local files" OK null null; #Step 4 is valid; stack flow_step [FLOW_PID] 4 "destination_connect_ok"; include "demo.orders.join.step_5_load.exe"; } { #Step 4 is not valid (an error of step 5 is already marked by step 5); if (not (== [etl_step5] 1)) {stack flow_step [FLOW_PID] 4 "destination_connect_KO";}; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_4_destination_connect.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_4_destination_connect.exe" "demo_orders_join"; }; script merge "demo.orders.join.step_5_load.exe" true 1 (param ) "Load to the destination (orders_join)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "demo_out"; json load "etl_cm_destination" {cm get [CONF_DESTINATION_CM];}; -> "[CONF_DIR_DESTINATION]" (json select "etl_cm_destination" "/localDir"); json unload "etl_cm_destination"; -> "[CONF_DESTINATION_FORMAT_OPTIONS]" "{}"; -> "[CONF_DESTINATION_FILENAME]" (concat "orders_" (date systimestamp_min) ".csv"); -> "[etl_step5]" 1; -> "[etl_run_done]" 0; # HANDLE ; try { # Mark the flow as step 5; stack flow_step [FLOW_PID] 5 "load..."; -> "[rows]" (file load (concat "home/" [FLOW_PID] "/" [filename]) "UTF-8"); if (not (and (string matches [CONF_DESTINATION_FILENAME] "[^/\\\p{Cntrl}]+") (not (string matches [CONF_DESTINATION_FILENAME] "[.][.]?")))) {exception (1) (concat "The destination file name is not a plain file name: " (string regexp_replace [CONF_DESTINATION_FILENAME] "\p{Cntrl}" "?"));}; # The file is built in a folder of its own: home/[FLOW_PID]/<source file>.out/; -> "[etl_out_dir]" (concat "home/" [FLOW_PID] "/" [source_filename] ".out"); if (not (file exist [etl_out_dir])) {file mkdir [etl_out_dir];}; -> "[etl_out]" (concat [etl_out_dir] "/" [CONF_DESTINATION_FILENAME]); log write (concat "Build the file '" [etl_out] "' (csv).") OK null null; csv write [etl_out] [rows] [CONF_DESTINATION_FORMAT_OPTIONS]; log write (concat "Send '" [etl_out] "' to '" [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME] "'...") OK null null; if (not (file exist [CONF_DIR_DESTINATION])) {file mkdir [CONF_DIR_DESTINATION];}; file copy_file [etl_out] (concat [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME]); if (not (file exist (concat [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME]))) { exception (1) (concat "Cannot copy the file '" [CONF_DESTINATION_FILENAME] "'."); }; file delete [etl_out]; try {file delete [etl_out_dir];} {} "[err]"; log write "Sent." OK null null; # Remove the work files of this file, then the folder of the run when it is empty (all its files are loaded); try {file delete (concat "home/" [FLOW_PID] "/" [filename]);} {} "[err]"; try {file delete (concat "home/" [FLOW_PID] "/" [source_filename]);} {} "[err]"; try { json load "etl_left" (file dir_list_regex (concat "home/" [FLOW_PID]) ".*" true true); if (== (json count "etl_left" /) 0) {file delete (concat "home/" [FLOW_PID]); -> "[etl_run_done]" 1;}; json unload "etl_left"; } {} "[err]"; #Step 5 is valid; stack flow_step [FLOW_PID] 5 "load_ok"; # END OF THE RUN (its last file is loaded: the folder of the run is deleted): done mark for the joins; if (== [etl_run_done] 1) { try {etl mark_done "demo.orders.join";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } { try {file delete [etl_out];} {} "[err]"; try {file delete [etl_out_dir];} {} "[err]"; #Step 5 is not valid; stack flow_step [FLOW_PID] 5 "load_KO"; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_5_load.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_5_load.exe" "demo_orders_join"; }; # JOIN: step 1 waits for demo.orders.csv_to_json, demo.orders.sql_to_sql (each must end OK since the join was armed or last passed), checked again every minute; script set delay "demo.orders.join.step_1_source_connect.exe" 1 min {etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql" [PID]}; # Arm the join now: the flows ended before this generation do not count; etl join_arm "demo.orders.join";
a flow cannot be linked to itself
admin
etl generate "{\"name\": \"orders_csv_to_json\", \"prefix\": \"demo.orders.csv_to_json\", \"source\": {\"cm\": \"demo_in\", \"kind\": \"files\", \"family\": \"file\", \"filter\": \"*.csv\", \"format\": \"csv\", \"delete_source\": true}, \"transforms\": [{\"type\": \"select\", \"columns\": [\"id\", \"client\", \"total\"]}, {\"type\": \"where\", \"condition\": {\"total\": {\"gt\": 100}}}, {\"type\": \"rename\", \"map\": {\"total\": \"amount\"}}, {\"type\": \"sort\", \"columns\": [[\"amount\", \"desc\"]]}], \"destination\": {\"cm\": \"demo_out\", \"kind\": \"files\", \"family\": \"file\", \"format\": \"json\", \"file_name\": \"orders_[SOURCE]_[FLOW_PID].json\"}, \"next\": [{\"prefix\": \"demo.orders.csv_to_json\"}]}"
mentdb
ERROR: Sorry, next[0].prefix: a flow cannot be linked to itself.

etl space_save <space>


Description

    Create or replace a space: {name, description, nodes: [{prefix, x, y, draft (optional: the design of an object drawn but not generated yet), ui}], notes: [{text, x, y, ui}], view: {zoom, x, y}, ui}. updated and by are set by the server. Returns the new updated. To replace an existing space give the updated read with it: a space changed since (or a save without updated) is refused

Parameters

    space:   The space JSON. Name: letters, digits, _ . - (100 characters at most) - String - required
admin
etl space_save "{\"name\": \"demo_erp\", \"description\": \"Orders of the ERP\", \"nodes\": [{\"prefix\": \"demo.orders.csv_to_json\", \"x\": 40, \"y\": 40}, {\"prefix\": \"demo.orders.sql_to_sql\", \"x\": 40, \"y\": 180}, {\"prefix\": \"demo.orders.join\", \"x\": 320, \"y\": 110, \"draft\": {\"name\": \"orders_join\", \"prefix\": \"demo.orders.join\", \"wait_for\": [\"demo.orders.csv_to_json\", \"demo.orders.sql_to_sql\"], \"next\": [{\"prefix\": \"demo.ops.alert\", \"on\": \"error\"}], \"source\": {\"cm\": \"MENTDB\", \"kind\": \"sql\", \"family\": \"db\", \"query\": \"select id, amount from etl_orders_new\"}, \"destination\": {\"cm\": \"demo_out\", \"kind\": \"files\", \"family\": \"file\", \"format\": \"csv\", \"file_name\": \"orders_[DATE].csv\"}}}], \"notes\": [{\"text\": \"Joins at night\", \"x\": 40, \"y\": 300}], \"view\": {\"zoom\": 1.0, \"x\": 0, \"y\": 0}}"
mentdb
2026-11-19 13:11:57.462
the same space saved again without its updated
admin
etl space_save "{\"name\": \"demo_erp\"}"
mentdb
ERROR: Sorry, the space 'demo_erp' already exists (saved by 'mentdb' at 2026-11-19 13:11:57.462): give its 'updated' (etl space_get) to replace it.
a node twice
admin
etl space_save "{\"name\": \"demo_bad\", \"nodes\": [{\"prefix\": \"a.b\"}, {\"prefix\": \"a.b\"}]}"
mentdb
ERROR: Sorry, nodes[1]: the flow 'a.b' is already in the space.

etl space_get <name>


Description

    The space JSON (with updated and by); the nodes the user is not granted are {prefix, hidden: true}

Parameters

    name:   The name of the space - String - required
admin
etl space_get "demo_erp"
mentdb
{"name": "demo_erp", "description": "Orders of the ERP", "nodes": [{"prefix": "demo.orders.csv_to_json", "x": 40, "y": 40}, {"prefix": "demo.orders.sql_to_sql", "x": 40, "y": 180}, {"prefix": "demo.orders.join", "x": 320, "y": 110, "draft": {"name": "orders_join", "prefix": "demo.orders.join", "wait_for": ["demo.orders.csv_to_json", "demo.orders.sql_to_sql"], "next": [{"prefix": "demo.ops.alert", "on": "error"}], "source": {"cm": "MENTDB", "kind": "sql", "family": "db", "query": "select id, amount from etl_orders_new"}, "destination": {"cm": "demo_out", "kind": "files", "family": "file", "format": "csv", "file_name": "orders_[DATE].csv"}}}], "notes": [{"text": "Joins at night", "x": 40, "y": 300}], "view": {"zoom": 1, "x": 0, "y": 0}, "updated": "2026-11-19 13:11:57.462", "by": "mentdb"}

etl space_list


Description

    The spaces: rows {name, description, nodes (count), updated, by}. A user who is not sys sees the spaces he saved last and those whose generated nodes are all granted to him

admin
etl space_list
mentdb
[{"name": "demo_erp", "description": "Orders of the ERP", "nodes": 3, "updated": "2026-11-19 13:11:57.462", "by": "mentdb"}]

etl space_view <name>


Description

    Everything to draw a space in one call: {space, nodes: [{prefix, exists, draft, x, y, design: {name, description, source: {cm, kind, family, type}, destination: {cm, kind, family, type}, transforms (count)}, next, wait_for, manual_modified, state: {state (none, waiting, running, closed, error), pid, date, message}}], cycles: [[prefixes of a cycle of success links]]}. A node not generated gives its draft design; a node the user is not granted is {prefix, hidden: true}

Parameters

    name:   The name of the space - String - required
admin
etl space_view "demo_erp"
mentdb
{"space": {"name": "demo_erp", "description": "Orders of the ERP", "nodes": [{"prefix": "demo.orders.csv_to_json", "x": 40, "y": 40}, {"prefix": "demo.orders.sql_to_sql", "x": 40, "y": 180}, {"prefix": "demo.orders.join", "x": 320, "y": 110, "draft": {"name": "orders_join", "prefix": "demo.orders.join", "wait_for": ["demo.orders.csv_to_json", "demo.orders.sql_to_sql"], "next": [{"prefix": "demo.ops.alert", "on": "error"}], "source": {"cm": "MENTDB", "kind": "sql", "family": "db", "query": "select id, amount from etl_orders_new"}, "destination": {"cm": "demo_out", "kind": "files", "family": "file", "format": "csv", "file_name": "orders_[DATE].csv"}}}], "notes": [{"text": "Joins at night", "x": 40, "y": 300}], "view": {"zoom": 1, "x": 0, "y": 0}, "updated": "2026-11-19 13:11:57.462", "by": "mentdb"}, "nodes": [{"prefix": "demo.orders.csv_to_json", "exists": false, "draft": false, "x": 40, "y": 40, "design": null, "next": [], "wait_for": [], "manual_modified": false, "state": {"state": "none", "pid": null, "date": null, "message": null}}, {"prefix": "demo.orders.sql_to_sql", "exists": false, "draft": false, "x": 40, "y": 180, "design": null, "next": [], "wait_for": [], "manual_modified": false, "state": {"state": "none", "pid": null, "date": null, "message": null}}, {"prefix": "demo.orders.join", "exists": false, "draft": true, "x": 320, "y": 110, "design": {"name": "orders_join", "description": "", "source": {"cm": "MENTDB", "kind": "sql", "family": "db", "type": "sql"}, "destination": {"cm": "demo_out", "kind": "files", "family": "file", "type": "file"}, "transforms": 0}, "next": [{"prefix": "demo.ops.alert", "on": "error"}], "wait_for": ["demo.orders.csv_to_json", "demo.orders.sql_to_sql"], "manual_modified": false, "state": {"state": "none", "pid": null, "date": null, "message": null}}], "cycles": []}

etl space_delete <name>


Description

    Delete a space (not its flows). Returns 1

Parameters

    name:   The name of the space - String - required
admin
etl space_delete "demo_erp"
mentdb
1

etl manual_check <prefix>


Description

    Were the scripts of a generated flow modified by hand since their generation? {prefix, exists, hash (the flow has an ETL_HASH line), modified, scripts: [{name, exists, modified}]}

Parameters

    prefix:   The prefix of the scripts of the flow - String - required
admin
etl manual_check "demo.orders.csv_to_json"
mentdb
{"prefix": "demo.orders.csv_to_json", "exists": true, "hash": true, "modified": true, "scripts": [{"name": "demo.orders.csv_to_json.step_1_source_connect.exe", "exists": true, "modified": false}, {"name": "demo.orders.csv_to_json.step_2_extract.exe", "exists": true, "modified": false}, {"name": "demo.orders.csv_to_json.step_3_transform.exe", "exists": true, "modified": true}, {"name": "demo.orders.csv_to_json.step_4_destination_connect.exe", "exists": true, "modified": false}, {"name": "demo.orders.csv_to_json.step_5_load.exe", "exists": true, "modified": false}]}

etl run <prefix>


Description

    Put step 1 of a generated flow in the stack, now: its pid, or a message when it already waits there (put in the stack, not started yet: not added again). A flow with wait_for starts only when its join is ready

Parameters

    prefix:   The prefix of the scripts of the flow - String - required
admin
etl run "demo.orders.csv_to_json"
mentdb
w

etl mark_start <prefix> <pid>


Description

    Written by step 1 of a generated flow (just after stack flow_init): the start mark of the flow with the pid of the run (etl space_view finds the last run with it). Returns 1. The marks of a flow are written by the flow itself (its step 1 granted) or by sys

Parameters

    prefix:   The prefix of the flow - String - required
    pid:   The pid of the run ([FLOW_PID]) - String - required
admin
etl mark_start "demo.orders.join" "w"
mentdb
1

etl join_arm <prefix>


Description

    Written by the MQL of etl generate for a flow with wait_for: arms the join, the flows ended before do not count. Returns 1

Parameters

    prefix:   The prefix of the flow that waits - String - required
admin
etl join_arm "demo.orders.join"
mentdb
1

etl mark_done <prefix>


Description

    Written by step 5 of a generated flow at the end of a run that ended OK: the done mark of the flow. Returns 1

Parameters

    prefix:   The prefix of the flow - String - required
admin
etl mark_done "demo.orders.csv_to_json"
mentdb
1

etl join_ready <prefix> <wait_for>


Description

    Is the join ready? 1 when every flow of the list has a done mark more recent than the arming of the join and than its last pass, else 0 (nothing is recorded)

Parameters

    prefix:   The prefix of the flow that waits - String - required
    wait_for:   The prefixes of the flows waited for, separated by commas - String - required
admin
etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql"
mentdb
0
both flows have ended OK since the arming
admin
etl mark_done "demo.orders.sql_to_sql"; etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql"
mentdb
1

etl join_ready <prefix> <wait_for> <pid>


Description

    The delay condition of step 1 of a flow with wait_for (the stack gives [PID]): when the join is ready its pass is recorded with this pid, so the next run needs new completions while a retry of the same stack entry passes again

Parameters

    prefix:   The prefix of the flow that waits - String - required
    wait_for:   The prefixes of the flows waited for, separated by commas - String - required
    pid:   The pid of the stack entry ([PID]) - String - required
admin
etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql" "x"
mentdb
1
another entry y: it waits for new completions
admin
etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql" "y"
mentdb
0

etl error_mark <prefix> <flow_pid> <pid>


Description

    Used by the error links of a generated flow: 1 only for the first error of the run and at the last attempt of the stack entry (a step that will be tried again gives 0)

Parameters

    prefix:   The prefix of the flow - String - required
    flow_pid:   The pid of the run ([FLOW_PID]) - String - required
    pid:   The pid of the stack entry ([PID]); empty: taken as the last attempt - String - required
admin
etl error_mark "demo.orders.join" "w" ""
mentdb
1

etl marks <prefix>


Description

    The marks of a flow: {prefix, start, start_pid, done, armed, join, join_pid, error, error_pid} (dates of the database with milliseconds, null when none; a pid only when the mark has one). Needs the grant of the flow (or sys)

Parameters

    prefix:   The prefix of the flow - String - required
admin
etl marks "demo.orders.join"
mentdb
{"prefix": "demo.orders.join", "start": "2026-11-19 13:11:57.463", "start_pid": "w", "done": null, "armed": "2026-11-19 13:11:57.464", "join": "2026-11-19 13:11:57.467", "join_pid": "x", "error": "2026-11-19 13:11:57.468", "error_pid": "w"}

etl mark_reset <prefix>


Description

    Remove every mark of a flow (start, done, armed, join, error): its joins start again from now. Returns 1

Parameters

    prefix:   The prefix of the flow - String - required
admin
etl mark_reset "demo.orders.join"
mentdb
1