When you use the DLC console for task queries, the query results will be displayed in real-time below the editor.
A single SQL task in the console can display up to 1,000 rows of data. SQL tasks submitted via API and JDBC are not subject to this limitation.
You can view the query history for a single Session for up to 3 months by checking the running history. For more methods to query historical records, see History.
Output Format Configuration for Task Results
The results of data exploration are saved in CSV format by calling Spark's DataFrame.write. If the engine version is released later than April 2023, you can configure the output format of the exploration results.
1. Configure the format of the results output to CSV. The following parameters are supported:
Parameter
Default Value
Remark
livy.sql.result.format.option.sep
livy.sql.result.format.option.delimiter
,
The separator between columns when the result is stored in CSV, which is a comma by default.
livy.sql.result.format.option.encoding
livy.sql.result.format.option.charset
UTF-8
String encoding format.
For example: UTF-8, US-ASCII, ISO-8859-1, UTF-16BE, UTF-16LE, and UTF-16.
livy.sql.result.format.option.quote
\\"
Specifies whether to use single or double quotation marks, with attention to the use of escape characters.
livy.sql.result.format.option.escape
\\\\
Escape character. Ensure the proper use of escape characters.
The name for the column that cannot be converted. This parameter is influenced by spark.sql.columnNameOfCorruptRecord, with table configuration taking precedence.
livy.sql.result.format.option.nullValue
Specifies the storage format for null values. The default is an empty string, in which case it can specify other emptyValue types.
livy.sql.result.format.option.nanValue
NaN
The storage format for non-numeric values.
livy.sql.result.format.option.positiveInf
Inf
The storage format for positive infinity.
livy.sql.result.format.option.negativeInf
-Inf
The storage format for negative infinity.
livy.sql.result.format.option.compression or codec
The class name of the compression algorithm. By default, no compression is applied. Short names like bzip2, deflate, gzip, lz4, and snappy can be used.
livy.sql.result.format.option.timeZone
System default time zone
The default time zone, influenced by spark.sql.session.timeZone. For example, Asia/Shanghai. Table configuration takes precedence.
livy.sql.result.format.option.locale
en-US
Specifies the language type.
livy.sql.result.format.option.dateFormat
yyyy-MM-dd
The default format for dates.
livy.sql.result.format.option.timestampFormat
yyyy-MM-dd'T'HH:mm:ss.SSSXXX
The default format for time. In non-LEGACY mode, it follows the format of yyyy-MM-dd'T'HH:mm:ss[.SSS][XXX].
The maximum number of characters per column. -1 means no limit.
livy.sql.result.format.option.escapeQuotes
true
Escapes quotation marks.
livy.sql.result.format.option.quoteAll
quoteAll
Encloses the entire content in quotation marks when writing.
livy.sql.result.format.option.emptyValue
\\"\\"
The format used for reading and writing empty values.
livy.sql.result.format.option.lineSep
The newline character used for line separation.
2. Configure the output format to a non-CSV format. Note that in this case, the console will not be able to display the results. However, you can read the result path using other methods. For details on where the result path is saved, see the next section.
The configuration option livy.sql.result.format supports saving in formats such as text, ORC, JSON, and Parquet.
Task Result Storage Location Configuration
Note: The Standard Engine - Presto is not supported. Full results can be obtained via JDBC.
DLC supports automatically saving query results to a COS path or DLC's managed storage through configuration. The configuration steps are as follows:
1. Log in to the DLC console, select the service region, and ensure that the login account has necessary COS-related permissions.
2. Go to the Data Exploration Page, click Storage Configuration in the upper right corner, and configure the settings for saving query results.
3. You can save the results to DLC's managed storage or COS. If you want to configure the path to COS, the operating account should have necessary COS-related permissions. Data storage fees will be based on COS pricing.
The task results are stored in subfolders under the following COS path:
Data path for task results: COS directory path/DLCQueryResults/yyyy/mm/dd/[QueryID]/data/XXXX.csv
Metadata path for task results: COS directory path/DLCQueryResults/yyyy/mm/dd/[QueryID]/meta/result.meta.json
COS directory path: This is the COS directory path configured in the system settings.
/yyyy/mm/dd: The directory is organized based on the task execution date.
/data: This directory stores the query result data, with files in CSV format. DLC may generate multiple data files.
/meta: This directory stores the metadata for the queried data tables, with files in JSON format.
Note:
Storing SELECT query results in DLC's internal storage, with Cloud Object Storage as the underlying storage, and the results are retained for 36 hours.
When SELECT query results are stored in your COS bucket path, ensure that you have necessary COS-related permissions.
Downloading Task Results
Note: The Standard Engine - Presto is not supported. Full results can be obtained via JDBC.
DLC allows users to manually download query results to their local devices. If full result mode is not enabled, users can download the results of tasks with available query results to their local devices or manually save them to COS (COS permissions are required).
The data downloaded or saved to COS correspond to the query results of the current SQL task, with a maximum of 500 results.
The maximum size for the local download is 50 MB.
If the results are configured to be saved to COS, they will be automatically stored in the COS path without the need for manual downloads.
Yes
No
Was this page helpful?