tencent cloud

All product documents
TDMQ for CKafka
DocumentationTDMQ for CKafkaGeneral ReferencesConnectorData ProcessingJSONPath Description
JSONPath Description
Download PDF
Last updated: 2024-11-07 11:39:19
JSONPath Description
Last updated: 2024-11-07 11:39:19
Download PDF

Overview

JSON is currently one of the most commonly used format protocols for Internet information transmission. Data processing mainly focus on parsing and handling JSON data.
JSONPath is a message query syntax specification designed for the JSON format. For data processing, simple JSONPath syntax can be used to quickly retrieve the value of a member in a complex nested JSON struct, and extension functions in the JayWay library can also be used to aggregate or operate on a certain type of member fields.

Basic Features

Basic Syntax

$ is a root node operator and represents the root node of the current JSON struct.
.<childName> is a dot operator, and ['<childName>'] is a bracket operator. They represent the selected child member named childName of the current object.
.. is a recursive operator and indicates recursively obtaining all child members of the current object.
[<index>] is a selection operator and indicates obtaining the No.index child member of the current iterable object.

Obtaining Specific Member Variable of Nested JSON Struct

The following figure shows the structure of container standard output logs collected by TKE:
{
  "@timestamp": 1648803500.63659,
  "@filepath": "/var/log/tke-log-agent/test7/xxxxxxxx-adfe-4617-8cf3-9997aea90ded/c_tke-es-xxxxxxxx57-n29jr_default_nginx-xxxxxxxx49626ef42d5615a636aae74d6380996043cf6f6560d8131f21a4d8ba/jgw_INFO_2022-02-10_15_4.log",
  "log": "15:00:00.000[4349811564226374227] [http-nio-8081-exec-64] INFO  com.qcloud.jgw.gateway.server.topic.TopicService",
  "kubernetes": {
    "pod_name": "tke-es-xxxxxxxxxx-n29jr",
    "namespace_name": "default",
    "pod_id": "xxxxxxxx-adfe-4617-8cf3-9997aea90ded",
    "labels": {
      "k8s-app": "tke-es",
      "pod-template-hash": "xxxx95d557",
      "qcloud-app": "tke-es"
    },
    "annotations": {
      "qcloud-redeploy-timestamp": "1648016531476",
      "tke.cloud.tencent.com/networks-status": "[{\n    \"name\": \"tke-bridge\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"172.16.x.xx\"\n    ],\n    \"mac\": \"xx:xx:xx:4a:c2:ba\",\n    \"default\": true,\n    \"dns\": {}\n}]"
    },
    "host": "10.0.xx.xx",
    "container_name": "nginx",
    "docker_id": "xxxxxxxx49626ef42d5615a636aae74d6380996043cf6f6560d8131f21a4d8ba",
    "container_hash": "nginx@sha256:xxxxxxxx7b29b585ed1aee166a17fad63d344bc973bc63849d74c6452d549b3e",
    "container_image": "nginx"
  }
}
When users need to obtain the current Pod name, that is, the qcloud-app member field, they can use the $.kubernetes.labels.qcloud-app or $.['kubernetes'].['labels'].['qcloud-app'] JSONPath syntax.
The running result is as follows. The figure shows that the corresponding logs have been successfully read using JSONPath:


Note
When JSONPath is used to handle parameters, only the square bracket operator can be used if a JSON variable name contains special characters such as ..
For example, for a JSON struct like {"key1.key2":"value1"}, $.['key1.key2'] must be used to obtain the corresponding member fields.

Advanced Features

Advanced Syntax

* is a wildcard operator and indicates obtaining all child objects of the current object.
*~ is a built-in function and indicates obtaining the names of all child objects of the current iterable object.
min() is a built-in function and indicates obtaining the minimum value of child objects of the current iterable object.
max() is a built-in function and indicates obtaining the maximum value of child objects of the current tterable object.
sum() is a built-in function and indicates obtaining the sum of child objects of the current iterable object.
concat() is a built-in function that concatenates multiple objects into a string.

Aggregating Data of Specific Fields

When there is an object list in a JSON struct, the list length is usually variable. Take the request response log in the figure below as an example:
{
  "data": {
    "Response": {
      "Result": {
        "Routers": [
          {
            "AccessType": 0,
            "RouteId": 81111,
            "VpcId": "vpc-xxxxxxxx",
            "VipType": 3,
            "VipList": [
              {
                "Vip": "10.0.0.189",
                "Vport": "9xxx"
              }
            ]
          },
          {
            "AccessType": 0,
            "RouteId": 81112,
            "VpcId": "vpc-r5sbavzp",
            "VipType": 3,
            "VipList": [
              {
                "Vip": "10.0.0.248",
                "Vport": "9xxx"
              }
            ]
          },
          {
            "AccessType": 0,
            "RouteId": 81113,
            "VpcId": "vpc-xxxxxxxx",
            "VipType": 3,
            "VipList": [
              {
                "Vip": "10.0.0.210",
                "Vport": "9xxx"
              }
            ]
          }
        ]
      },
      "RequestId": "20e74750-ca40-403d-9ea9-d3f63b5415d2"
    }
  },
  "code": 0
}
When you need to aggregate the member attributes of a variable-length list, you cannot use the processing chain for aggregation. Instead, you can use the JSONPath syntax * to match all elements in the list.
For example, if you want to obtain all the Vip elements in VipList, you can use the JSONPath syntax $.data.Response.Result.Routers[*].VipList[0].Vip.
The running result is shown below. It shows that all Vip elements in the struct are successfully obtained.



Concatenating and Modify Struct Members

In some scenes, multiple objects in a JSON struct need to be concatenated during data processing to facilitate transfer to the downstream for further operations. The following syntax can be used:
{
  "data": {
    "Response": {
      "SubnetSet": [
        {
          "VpcId": "vpc-xxxxxxxx",
          "SubnetId": "subnet-xxxxxxxx",
          "SubnetName": "ckafka_cloud_subnet-1",
          "CidrBlock": "10.0.0.0/19",
          "Ipv6CidrBlock": "",
          "IsDefault": false,
          "IsRemoteVpcSnat": false,
          "EnableBroadcast": false,
          "Zone": "ap-changsha-ec-1",
          "RouteTableId": "rtb-xxxxxxxx",
          "NetworkAclId": "",
          "TotalIpAddressCount": 8189,
          "AvailableIpAddressCount": 8033,
          "CreatedTime": "2021-01-25 17:31:00",
          "TagSet": [],
          "CdcId": "",
          "IsCdcSubnet": 0,
          "LocalZone": false,
          "IsShare": false
        }
      ],
      "TotalCount": 1,
      "RequestId": "705c4955-0cd9-48b2-9132-79eadae2e3e6"
    }
  },
  "code": 0
}
If the downstream has no computing capability, and the VPC and subnet attributes need to be concatenated during data processing, the JSONPath function concat() can be used to concatenate multiple fields and modify the output string.
For example, the syntax $.concat($.data.Response.SubnetSet[0].VpcId,"#",$.data.Response.SubnetSet[0].SubnetId,"#",$.data.Response.SubnetSet[0].CidrBlock)) can be used to concatenate the VPC and subnet attributes, which can be separated with the character #.
The running result is shown below. It shows that the information on concatenated VPC resources are successfully obtained:



Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon