JSON Lines
Read and write JSON Lines (NDJSON) files and texts, one JSON value per line. Typical uses: load log exports or machine learning data sets, prepare files for bulk APIs.
jsonl readjsonl readjsonl writejsonl writejsonl to_rowsjsonl to_rowsjsonl from_rows

jsonl read <path>


Description

    Read a file of JSON Lines: a JSON array of the values of all the lines (objects or not). The first bad line is an error

Parameters

    path:   The file - String - required
admin
jsonl write "tmp/doc_api.jsonl" "[{\"id\": 1}, {\"id\": 2}]"; jsonl read "tmp/doc_api.jsonl"
mentdb
[{"id": 1}, {"id": 2}]

jsonl read <path> <json_params>


Description

    Read with parameters: objects_only (false; true: a line that is not an object is bad), skip_bad (false: the first bad line is an error naming its number; true: the result is {rows, count, bad_count, bad}, with bad = [{line, error, text}]), max_bad (100 lines kept in bad), max_lines (1000000, 10000000 at most: beyond it is always an error), max_line_bytes (16 MB), max_bytes (of the result, 64 MB, 1 GB at most), offset (data lines to pass over without reading them), limit (stop after this number of values)

Parameters

    path:   The file - String - required
    json_params:   The parameters - String - required
admin
jsonl read "tmp/doc_api.jsonl" "{\"offset\": 1, \"limit\": 1}"
mentdb
[{"id": 2}]
Keep the good lines and report the others
admin
jsonl write "tmp/doc_api2.jsonl" "[{\"id\": 1}, [1, 2]]"; jsonl read "tmp/doc_api2.jsonl" "{\"skip_bad\": true, \"objects_only\": true}"
mentdb
{"rows": [{"id": 1}], "count": 1, "bad_count": 1, "bad": [{"line": 2, "error": "a JSON object is expected", "text": "[1, 2]"}]}
Used in use cases (1 recipe)

    Integrator & ETL developer › Import a JSON Lines export and report the bad lines
Often used with

    file exist  ·  file mkdir  ·  jsonl to_rows  ·  jsonl write  ·  string char

jsonl write <path> <json_rows>


Description

    Write a JSON array as a file, one compact value per line, each line ended by a line feed. The file is replaced all at once: a failure leaves the old file as it was. On Linux and macOS an existing file keeps its rights and its owner, and a symbolic link stays a link. A path that exists and is not a regular file (named pipe, device) is refused. JSON {path, lines, bytes, append}

Parameters

    path:   The file (its folder must exist) - String - required
    json_rows:   The JSON array - String - required
admin
jsonl write "tmp/doc_api.jsonl" "[{\"id\": 1}, {\"id\": 2}]"
mentdb
{"path": "tmp/doc_api.jsonl", "lines": 2, "bytes": 20, "append": false}

jsonl write <path> <json_rows> <json_params>


Description

    Write with parameters: append (true: add at the end, with a line feed first if the last line had none)

Parameters

    path:   The file - String - required
    json_rows:   The JSON array - String - required
    json_params:   The parameters - String - required
admin
jsonl write "tmp/doc_api.jsonl" "[{\"id\": 3}]" "{\"append\": true}"
mentdb
{"path": "tmp/doc_api.jsonl", "lines": 1, "bytes": 10, "append": true}
Used in use cases (1 recipe)

    Integrator & ETL developer › Import a JSON Lines export and report the bad lines
Often used with

    file exist  ·  file mkdir  ·  jsonl read  ·  jsonl to_rows  ·  string char

jsonl to_rows <text>


Description

    Read a JSON Lines text as rows: a JSON array of objects. A line that is not an object is bad

Parameters

    text:   The text - String - required
admin
jsonl to_rows "{\"id\": 1, \"name\": \"Ana\"} {\"id\": 2, \"name\": \"Bob\"} "
mentdb
[{"id": 1, "name": "Ana"}, {"id": 2, "name": "Bob"}]
A line in error
admin
jsonl to_rows "{\"id\": 1} not json"
mentdb
ERROR: Sorry, an error appears (line 2: invalid JSON at position 0 (unexpected character)).

jsonl to_rows <text> <json_params>


Description

    The same with parameters: objects_only (true; false: any JSON value), skip_bad (false: the first bad line is an error naming its number; true: the result is {rows, count, bad_count, bad}, with bad = [{line, error, text}]), max_bad (100 lines kept in bad), max_lines (1000000, 10000000 at most: beyond it is always an error), max_line_bytes (16 MB), max_bytes (of the result, 64 MB, 1 GB at most), offset (data lines to pass over without reading them), limit (stop after this number of values)

Parameters

    text:   The text - String - required
    json_params:   The parameters - String - required
admin
jsonl to_rows "{\"id\": 1} not json [1, 2] {\"id\": 12345678901234567890} " "{\"skip_bad\": true}"
mentdb
{"rows": [{"id": 1}, {"id": 12345678901234567890}], "count": 2, "bad_count": 2, "bad": [{"line": 2, "error": "invalid JSON at position 0 (unexpected character)", "text": "not json"}, {"line": 3, "error": "a JSON object is expected", "text": "[1, 2]"}]}
The first line only
admin
jsonl to_rows "{\"id\": 1, \"name\": \"Ana\"} {\"id\": 2, \"name\": \"Bob\"} " "{\"limit\": 1}"
mentdb
[{"id": 1, "name": "Ana"}]
Used in use cases (1 recipe)

    Integrator & ETL developer › Import a JSON Lines export and report the bad lines
Often used with

    file exist  ·  file mkdir  ·  jsonl read  ·  jsonl write  ·  string char

jsonl from_rows <json_rows>


Description

    Write a JSON array as JSON Lines text, each line ended by a line feed. An empty array gives an empty text

Parameters

    json_rows:   The JSON array - String - required
admin
jsonl from_rows "[{\"id\": 1, \"price\": 1.50}, {\"id\": 2, \"price\": 12345678901234567890}]"
mentdb
{"id": 1, "price": 1.50} {"id": 2, "price": 12345678901234567890}