tencent cloud

Feedback

Push to HTTP Server

Last updated: 2024-06-28 14:21:49
    EdgeOne Real-time Log Push supports sending logs to a Custom Interface address, and you can configure it through the console or API. EdgeOne can use an HTTP POST request to call the backend interface address you provide, transferring logs in the HTTP Body to the server you specify.

    Operation step

    1. Access the EdgeOne console, locate the left-hand menu, and select the Site List. Within this listing, click on the Site requiring configuration to enter its detailed page.
    2. On the Site Details Page, click Log Service > Real-time Logs.
    3. On the Real-time Logs page, click Create shipping task.
    4. On Select the log source Page, fill in the Task name, select the Log type Service area, the Domain name/Layer 4 Proxy Instance for which the logs need to be pushed, and click Next.
    5. On the Define delivery content page:
    (Required) Check the log fields to be pushed from the Predefined field list;
    (Optional) Add a Custom Log Field, supporting the extraction of specified field names from the Request Header, Response Header, and Cookie Header;
    (Optional) Configure the Push Log Filter Criteria, the default is to push full logs;
    (Optional) In advanced configuration, set the sampling ratio. By default, sampling is not enabled, and 100% of logs are pushed to the destination.
    (Optional) In advanced configuration, set the log output format. The default format is JSON Lines.
    6. On the Select a destination address page, select HTTP (POST), click Next .
    7. On the Destination Information Page, enter the related destination and parameter information.
    Parameter name
    Description
    Address
    Enter your log receiving interface address, for example: https://www.example.com/edgeone-logs
    File compression
    To reduce the size of log content and save on traffic costs, you can enable content compression by checking Use Compress log files with gzip. EdgeOne will use the gzip format to compress and then transmit the logs, and it will add the HTTP header Content-Encoding: gzip to indicate the compression format.
    Origin authentication
    When Encryption Authentication is selected, the push logs will carry authentication information for the origin server to verify, ensuring the security of the data source identity. For the authentication algorithm, see: Authentication Algorithm Reference.
    Custom HTTP header
    Add the HTTP headers that need to be carried when EdgeOne initiates a request. For example:
    Identify the log source as EdgeOne by adding the header log-source: EdgeOne.
    Obtain the number of log entries pushed in each POST request by adding the header BatchSize: ${batchSize}
    Note
    If the header name you fill in is a default header carried by EdgeOne log push, such as Content-Type, the header value you specify will override the default value.
    8. Click Ship, confirm the related cost alert in the popup, and click Confirm Creation.
    9. During the configuration phase of the Real-time Log Push Task, test data will be sent to the interface address to verify interface connectivity. The data format is as follows:
    { "ClientState": "CH-AH", "EdgeResponseTime": 366, "RequestID": "13515444256055847385", "ClientRegion": "CN", "RemotePort": 443, "RequestHost": "www.tencent.com", "RequestMethod": "GET", "RequestUrlQueryString": "-", "RequestUrl": "/en-us/about.html", "RequestProtocol": "HTTP/2.0", "EdgeServerID": "336d5ebc5436534e61d16e63ddfca327-d41d8cd98f00b204e9800998ecf8427e", "RequestTime": "2022-07-01T02:37:13Z", "EdgeCacheStatus": "-", "EdgeResponseBytes": 39430, "EdgeResponseStatusCode": 200, "ClientIP": "0.0.0.0", "RequestReferer": "https://www.tencent.com/", "RequestUA": "Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Mobile/15E148 Safari/604.1", "EdgeServerIP": "0.0.0.0", "RequestRange": "0-100/200", "EdgeInternalTime": 334, "RequestBytes": 237 }

    Related References

    Server-side Log Parsing Code Example

    When origin server authentication is not enabled, you can refer to the following Python code to parse the log content in the request body on the server side.
    # Import modules from the Python standard library
    import time # Used to get the current time
    import gzip # Used to handle Gzip compressed data
    # Import HTTPServer and BaseHTTPRequestHandler classes from the http.server module
    from http.server import HTTPServer, BaseHTTPRequestHandler
    import json # Used to handle JSON data format
    # Define a class that inherits from BaseHTTPRequestHandler, used to handle HTTP requests
    class Resquest(BaseHTTPRequestHandler):
    # Override the do_POST method, which is called when the server receives a POST request
    def do_POST(self):
    # Print request header information
    print(str(self.headers))
    # Print the command from the HTTP request (e.g., POST)
    print(self.command)
    # Read the request body content, determining the length to read as specified in the Content-Length field
    req_datas = self.rfile.read(int(self.headers['content-length']))
    try:
    # Attempt to decode the request body content and print it
    print(req_datas.decode())
    except Exception as e:
    # If an exception occurs during decoding, print the exception information
    print(e)
    # Check if the request header contains Content-Encoding: gzip, if so, decompress the request body
    if self.headers['Content-Encoding'] == 'gzip':
    data = gzip.decompress(req_datas)
    # Print the decompressed gzip content
    print('---------------decompress gzip content-------------------------')
    print(data.decode())
    # Check if the request path is '/edgeone-logs', if not, return a 404 error
    if self.path != '/edgeone-logs':
    self.send_error(404, "Page not Found!")
    return
    # If the request path is correct, prepare the response data
    data = {
    'result_code': '1',
    'result_desc': 'Success',
    'timestamp': int(time.time()) # Respond with the current timestamp
    }
    # Send an HTTP response status code 200, indicating the request was successful
    self.send_response(200)
    # Set the response header Content-type to application/json
    self.send_header('Content-type', 'application/json')
    # End the sending of response headers
    self.end_headers()
    # Write the response data in JSON format to the response body
    self.wfile.write(json.dumps(data).encode('utf-8'))
    # Check if the current script is running as the main program
    if __name__ == '__main__':
    # Define the server listening address and port. You can replace 9002 with your own port
    host = ('', 9002)
    # Create an HTTPServer object, passing in the listening address, port, and the request handler class
    server = HTTPServer(host, Resquest)
    # Print server startup information
    print("Starting server, listen at: %s:%s" % host)
    # Start the server and keep it running until externally interrupted
    server.serve_forever()

    Request authentication algorithm

    If you select Encryption Signature for origin server authentication in the Push Destination Information, you can enter the SecretId and SecretKey configured by yourself in Definition. EdgeOne will add auth_key and access_key to the request URL. The details of the signature algorithm are as follows:
    1. Request URL Composition
    As shown below, the request URL will carry auth_key and access_key after ?.
    http://DomainName[:port]/[uri]?auth_key=timestamp-rand-md5hash&access_key=SecretId
    Parameter description:
    timestamp: The current request time, using Unix second-level 10-digit timestamp.
    rand: Random number.
    access_key: Used to identify the identity of the interface requester, that is, the SecretId configured by your definition.
    SecretKey: Fixed length of 32 characters, that is, the SecretKey configured by your definition.
    uri: Resource identifier, for example: /access_log/post.
    md5hash: md5hash = md5sum(string_to_sign), where string_to_sign ="uri-timestamp-rand-SecretKey". A verification string calculated through the MD5 algorithm, consisting of digits 0-9 and lowercase letters a-z, with a fixed length of 32 characters.
    2. Computational Example
    Assume the parameters are filled as: API Address: https://www.example.com/access_log/post SecretId = YourID SecretKey = YourKey uri = /access_log/post timestamp = 1571587200 rand = 0
    string_to_sign = "/access_log/post-1571587200-0-YourKey"
    Based on this string, calculate:
    md5hash=md5sum("/access_log/post-1571587200-0-YourKey")=1f7ffa7bff8f06bbfbe2ace0f14b7e16
    Then the final URL for the push request is:
    https://www.example.com/cdnlog/post?auth_key=1571587200-0-1f7ffa7bff8f06bbfbe2ace0f14b7e16&access_key=YourID
    After the server receives the push request, it extracts the value of auth_key. Split the value of auth_key to obtain timestamp, rand, and md5hash. First, check whether the timestamp has expired; the recommended expiration time is 300s. Then assemble the encrypted string based on the aforementioned rules and use the SecretKey to construct the string to be encrypted. After encryption, compare it with the auth_key value in the md5hash. If they match, the authentication is successful.
    3. Server-side Code Example for Parsing Authentication Requests
    Python
    Golang
    import hashlib
    
    from flask import Flask, request
    
    app = Flask(__name__)
    
    
    def get_rsp(msg, result={}, code=0):
    return {
    "respCode": code,
    "respMsg": msg,
    "result": result
    }
    
    
    def get_secret_key(access_key):
    return "secret_key"
    
    
    @app.route("/access_log/post", methods=['POST'])
    def access_log():
    if request.method == 'POST':
    if request.content_type.startswith('application/json'):
    current_time_ts, rand_num, md5hash = request.args.get("auth_key").split("-")
    # Determine if the request time is within the validity period
    if time.time() - int(current_time_ts) > 300:
    return get_rsp(msg="The request is out of time", code=-1)
    
    access_key = request.args.get("access_key")
    # Retrieve the secret_key using access_key (SecretId)
    secret_key = get_secret_key(access_key)
    raw_str = "%s-%s-%s-%s" % (request.path, current_time_ts, rand_num, secret_key)
    auth_md5hash = hashlib.md5(raw_str.encode("utf-8")).hexdigest()
    if auth_md5hash == md5hash:
    # Authentication successful
    if request.headers['content-encoding'] == 'gzip':
    # Decompress data
    pass
    # Data processing
    return get_rsp("ok")
    return get_rsp(msg="Please use content_type by application/json", code=-1)
    return get_rsp(msg="The request method not find, method == %s" % request.method, code=-1)
    
    if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8888, debug=True)
    
    package main
    
    import (
    "context"
    "crypto/md5"
    "fmt"
    "log"
    "net/http"
    "os"
    "os/signal"
    "strings"
    "syscall"
    )
    
    func main() {
    mux := http.NewServeMux()
    mux.Handle("/access_log/post", &logHandler{})
    
    server := &http.Server{
    Addr: ":5000",
    Handler: mux,
    }
    
    // Create system signal receiver
    done := make(chan os.Signal)
    signal.Notify(done, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
    go func() {
    <-done
    
    if err := server.Shutdown(context.Background()); err != nil {
    log.Fatal("Shutdown server:", err)
    }
    }()
    
    err := server.ListenAndServe()
    if err != nil {
    if err == http.ErrServerClosed {
    log.Print("Server closed under request")
    } else {
    log.Fatal("Server closed unexpected")
    }
    }
    }
    
    type logHandler struct{}
    
    func (*logHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if r.Method == "POST" {
    query := r.URL.Query()
    authKey := query.Get("auth_key")
    accessKey := query.Get("access_key") // access_key is your provided SecretId
    authKeys := strings.Split(authKey, "-")
    if len(authKeys) == 3 {
    currentTimeTs := authKeys[0]
    // Perform timestamp validity judgment
    RandNum := authKeys[1]
    md5Hash := authKeys[2]
    secretKey := getSecretKey(accessKey)
    authStr := fmt.Sprintf("%s-%s-%s-%s", "/access_log/post", currentTimeTs, RandNum, secretKey)
    data := []byte(authStr)
    has := md5.Sum(data)
    authMd5 := fmt.Sprintf("%x", has) // Convert to string for comparison
    if authMd5 == md5Hash {
    // TODO Authentication successful
    if r.Header.Get("Content-Encoding") == "gzip" {
    // Decompress data
    }
    // Data processing
    }
    } else {
    // Exception Handling
    }
    }
    }
    
    // Retrieve SecretKey
    func getSecretKey(accessKey string) string {
    if accessKey != "" {
    // Retrieve Secret_Key using Access_key (SecretId)
    return "secret_key"
    }
    return ""
    }
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support