CLS's data processing functions can be flexibly combined and used to cleanse, structure, filter, distribute, mask log data and more.
The figure below shows how a log containing JSON data is processed: the log data is processed and transformed into structured data, and then a field in the log is masked before the processed log is distributed.
The following is an overview of data processing functions:
Key-value extraction functions: extracting field-value pairs from log text
Function
Description
Syntax Description
Return Value Type
ext_sep
Extracts field value content based on a separator.
ext_sep("Source field name", "Target field 1,Target field 2,Target field...", sep="Separator", quote="Non-segmentation part"", restrict=False, mode="overwrite")
Log after extraction (LOG)
ext_sepstr
Extracts field value content based on specified characters (string).
ext_sepstr("Source field name","Target field 1,Target field 2,Target field...", sep="abc", restrict=False, mode="overwrite")
Log after extraction (LOG)
ext_json
Extracts field values from JSON data.
Log after extraction (LOG) ext_json("Source field name",prefix="",suffix="",format="full",exclude_node="JSON nodes not to expand")
Log after extraction (LOG)
ext_json_jmes
Extracts a field value based on a JMES expression.
Extracts field values by using two levels of separators.
ext_kv("Source field name", pair_sep=r"\\s", kv_sep="=", prefix="", suffix="", mode="fill-auto")
Log after extraction (LOG)
ext_regex
Extracts field values by using a regular expression.
ext_regex("Source field name", regex="Regular expression", output="Target field 1,Target field 2,Target field...", mode="overwrite")
Log after extraction (LOG)
ext_first_notnull
Returns the first non-null and non-empty result value.
ext_first_notnull(Value 1,Value 2,...)
The first non-null result value
Enrichment functions: Adding fields to existing fields according to rules
Function
Description
Syntax Description
Return Value Type
enrich_table
Uses CSV structure data to match fields in logs and, when matched fields are found, the function adds other fields and values in the CSV data to the source logs.
enrich_table("CSV source data", "CSV enrichment field", output="Target field 1,Target field 2,Target field....", mode="overwrite")
Mapped log (LOG)
enrich_dict
Uses dict structure data to match a field value in a log. If the specified field and value match a key in the dict structure data, the function assigns the value of the key to another field in the log.
enrich_dict("JSON dictionary", "Source field name", output=Target field name, mode="overwrite")
Mapped log (LOG)
Flow control functions: condition determination
Function
Description
Syntax Description
Return Value Type
compose
Combines multiple operation functions. Providing combination capabilities similar to those of branch code blocks, this function can combine multiple operation functions and execute them in sequence. It can be used in combination with branches and output functions.
compose("Function 1","Function 2", ...)
Log (LOG)
t_if
Executes a corresponding function if a condition is met and does not perform any processing if the condition is not met.
t_if("Condition", Function)
Log (LOG)
t_if_not
Executes a corresponding function if a condition is not met and does not perform any processing if the condition is met.
t_if_not("Condition",Function)
Log (LOG)
t_if_else
Executes a function based on the evaluation result of a condition.
t_if_else("Condition", Function 1, Function 2)
Log (LOG)
t_switch
Executes different functions depending on whether branch conditions are met. If all conditions are not met, the data is deleted.
t_switch("Condition 1", Function 1, "Condition 2", Function 2, ...)
Log (LOG)
Row processing functions: log distribution, deletion, and splitting
Function
Description
Syntax Description
Return Value Type
log_output
Outputs a row of log to a specified log topic. This function can be used independently or together with branch conditions.
log_output(Log topic alias) (The alias here is the target log topic alias specified when the data processing task is created.)
No return value
log_split
Splits a row of log into multiple rows of logs based on the value of a specified field by using a separator and JMES expression.
Field processing functions: field Create/Read/Update/Delete/Rename
Function
Description
Syntax Description
Return Value Type
fields_drop
Deletes the fields that meet a specified condition.
fields_drop(Field name 1, Field name 2, ..., regex=False,nest=False)
Log (LOG)
fields_keep
Retains the fields that meet a specified condition.
fields_keep(Field name 1, Field name 2, ..., regex=False)
Log (LOG)
fields_pack
Matches field names based on a regular expression and encapsulates the matched fields into a new field whose value is in JSON format.
fields_pack(Target field name, include=".*", exclude="", drop_packed=False)
Log (LOG)
fields_set
Sets field values or adds fields.
fields_set(Field name 1, Field value 1, Field name 2, Field value 2, mode="overwrite")
Log (LOG)
fields_rename
Renames fields.
fields_rename(Field name 1, New field name 1, Field name 2, New field name 2, regex=False)
Log (LOG)
has_field
If the specified field exists, returns `True`. Otherwise, returns `False`.
has_field(Field name)
Condition value (BOOL)
not_has_field
If the specified field does not exist, returns `True`. Otherwise, returns `False`.
not_has_field(Field name)
Condition value (BOOL)
v
Gets the value of a specified field and returns the corresponding string.
v(Field name)
Value string type (STRING)
Value structuring functions: JSON data processing
Function
Description
Syntax Description
Return Value Type
json_select
Extracts a JSON field value with a JMES expression and returns the JSON string of the extraction result.
json_select(v(Field name), jmes="")
Value string type (STRING)
xml_to_json
Parses and converts an XML-formatted value to a JSON string. The input value must be an XML string. Otherwise, a conversion exception will occur.
xml_to_json(Field value)
Value string type (STRING)
json_to_xml
Parses and converts a JSON string value to an XML string.
json_to_xml(Field value)
Value string type (STRING)
if_json
Checks whether a value is a JSON string.
if_json(Field value)
Condition value (BOOL)
Regular expression functions: matching and replacing characters in text by using regular expressions
Function
Description
Syntax Description
Return Value Type
regex_match
Matches data in full or partial match mode based on a regular expression and returns whether the match is successful.
regex_match(Field value, regex="", full=True)
Condition value (BOOL)
regex_select
Matches data based on a regular expression and returns the corresponding partial match result. You can specify the sequence number of the matched expression and the sequence number of the group to return (partial match + sequence number of the specified matched group). If no data is matched, an empty string is returned.
Matches data based on a regular expression and returns a JSON array of the matched data (partial match).
regex_findall(Field value, regex="")
Value string type (STRING)
Date value processing functions
Function
Description
Syntax Description
Return Value Type
dt_str
Converts a time field value (a date string in a specific format or timestamp) to a target date string of a specified time zone and format.
dt_str(Value, format="Formatted string", zone="")
STRING
dt_to_timestamp
Converts a time field value (a date string in a specified format; time zone specified) to a UTC timestamp.
dt_to_timestamp(Value, zone="")
STRING
dt_from_timestamp
Converts a timestamp field value to a time string in the specified time zone.
dt_from_timestamp(Value, zone="")
STRING
dt_now
Obtains the current datetime of the processing calculation.
dt_now(format="Formatted string", zone="")
STRING
String processing functions
Function
Description
Syntax Description
Return Value Type
str_count
Searches for a substring in a specified range of a value and returns the number of occurrences of the substring.
str_count(Value, sub="", start=0, end=-1)
INT
str_len
Returns the length of a string.
str_len(Value)
INT
str_uppercase
Converts a string to uppercase.
str_uppercase(Value)
STRING
str_lowercase
Converts a string to lowercase.
str_lowercase(Value)
STRING
str_join
Concatenates input values by using a concatenation string.
str_join(Concatenation string 1, Value 1, Value 2, ...)
STRING
str_replace
Replaces an old string with a new string.
str_replace(Value, old="", new="", count=0)
STRING
str_format
Formats strings.
str_format(Formatting string, Value 1, Value 2, ...)
STRING
str_strip
Deletes specified characters from a string concurrently from the start and end of the string, and returns the remaining part.
str_strip(Value, chars="\\t\\r\\n")
STRING
str_lstrip
Deletes specified characters from a string from the start of the string, and returns the remaining part.
str_strip(Value, chars="\\t\\r\\n")
STRING
str_rstrip
Deletes specified characters from a string from the end of the string, and returns the remaining part.
str_strip(Value, chars="\\t\\r\\n")
STRING
str_find
Checks whether a string contains a specified substring and returns the position of the first occurrence of the substring in the string.
str_find(Value, sub="", start=0, end=-1)
INT
str_start_with
Checks whether a string starts with a specified string.
str_start_with(Value, sub="", start=0, end=-1)
BOOL
str_end_with
Checks whether a string ends with a specified string.
str_end_with(Value, sub="", start=0, end=-1)
BOOL
Logical expression functions
Function
Description
Syntax Description
Return Value Type
op_if
Returns a value based on a specified condition.
op_if(Condition 1, Value 1, Value 2)
If the condition is `true`, `Value 1` is returned; otherwise, `Value 2` is returned.
op_and
Performs the AND operation on values. If all the specified parameter values are evaluated to true, `True` is returned; otherwise, `False` is returned.
op_and(Value 1, Value 2, ...)
BOOL
op_or
Performs the OR operation on values. If one or more of the specified parameter values are evaluated to false, `False` is returned; otherwise, `True` is returned.
op_or(Value 1, Value 2, ...)
BOOL
op_not
Performs the NOT operation on values.
op_not(Value)
BOOL
op_eq
Compares two values. If the values are equal, `True` is returned.
op_eq(Value 1, Value 2)
BOOL
op_ge
Compares two values. If `Value 1` is greater than or equal to `Value 2`, `True` is returned.
op_ge(Value 1, Value 2)
BOOL
op_gt
Compares two values. If `Value 1` is greater than `Value 2`, `True` is returned.
op_gt(Value 1, Value 2)
BOOL
op_le
Compares two values. If `Value 1` is less than or equal to `Value 2`, `True` is returned.
op_le(Value 1, Value 2)
BOOL
op_lt
Compares two values. If `Value 1` is less than `Value 2`, `True` is returned.
op_lt(Value 1, Value 2)
BOOL
op_add
Returns the sum of two specified values.
op_add(Value 1, Value 2)
Calculation result
op_sub
Returns the difference between two specified values.
op_sub(Value 1, Value 2)
Calculation result
op_mul
Returns the product of two specified values.
op_mul(Value 1, Value 2)
Calculation result
op_div
Returns the quotient of two specified values.
op_div(Value 1, Value 2)
Calculation result
op_sum
Returns the sum of multiple specified values.
op_sum(Value 1, Value 2, ...)
Calculation result
op_mod
Returns the remainder of a specified value divided by the other specified value.
op_mod(Value 1, Value 2)
Calculation result
op_null
Checks whether a value is `null`. If so, `true` is returned; otherwise, `false` is returned.
op_null(Value)
BOOL
op_notnull
Checks whether a value is not `null`. If so, `true` is returned; otherwise, `false` is returned.
op_notnull(Value)
BOOL
op_str_eq
Compares string values. If they are equal to each other, `true` is returned.
op_str_eq(Value 1, Value 2, ignore_upper=False)
BOOL
Type conversion functions
Function
Description
Syntax Description
Return Value Type
ct_int
Converts a value (whose base can be specified) to a decimal integer.
Ya
Tidak
Apakah halaman ini membantu?