Through a practical example, this document will demonstrate how to use GoReplay to record requests received by a Nginx gateway and save the various fields of these requests into a CSV file. In PTS, by uploading the CSV parameter file and specifying the expected concurrency, you can playback the requests to the user-specified addresses in a distributed manner.
Introduction to GoReplay
GoReplay is an open-source tool for traffic recording and playback. It is mainly used to capture live traffic and replicate it in a testing environment.
Since GoReplay does not provide a distributed execution solution by itself, it can only run on a single machine. After the traffic recording is completed, it becomes challenging to playback recorded traffic on a large scale due to the limitations of single-machine resource bottlenecks. This makes it difficult to effectively simulate the performance testing behavior of real user traffic and extreme testing. Tencent Cloud's PTS is a distributed performance testing service that can simulate real business scenarios of massive users. Therefore, we can introduce PTS and use it to playback the real traffic recorded by GoReplay.
Common GoReplay Application Scenarios
Performance testing: By replicating production traffic in a test environment, applications can be stress tested and performance evaluated without impacting real users.
Troubleshooting and debugging: When issues arise in the production environment, relevant traffic can be captured and replayed in an isolated environment, allowing developers to securely debug problems without affecting the actual service.
Regression testing: Before releasing new versions, GoReplay-captured traffic can be used to verify that changes do not introduce new bugs or performance issues.
A/B testing: Traffic can be simultaneously directed to two different versions of a service to compare their performance, enabling data-driven decision making.
By increasing the multiplier of replayed requests during playback, high-traffic scenarios can be simulated. This helps in determining the required resource amounts under different load levels.
Principles of GoReplay Traffic Recording
GoReplay traffic recording involves listening for traffic on the specified port and capturing it into a GOR file (or sending it to another destination) for easier subsequent playback.
sudo gor --input-raw :8080--output-file requests.gor
Starting Recording and Playback of User Nginx Gateway
This document uses the recording and playback of an Nginx gateway as an example. For all other types of gateways, you can use the same method to record requests, and then use PTS to playback user requests.
Environment Preparation
Nginx gateway: The Nginx gateway continuously receives a steady stream of user requests. It is necessary to record these requests on the Nginx gateway.
GoReplay: A tool for recording and playback of requests.
Install GoReplay on the machine where the gateway resides. If the machine is running Linux or macOS, you can use the following command:
# Download the latest binary files from the official GitHub repository.
curl -L https://github.com/buger/goreplay/releases/download/1.3.3/gor_1.3.3_x64.tar.gz | tar xz
# Move binary files to your PATH directory, such as /usr/local/bin.
mv gor /usr/local/bin/
Note:
Make sure to replace the version number in the URL above with the latest version. Repository address: https://github.com/buger/goreplay.
CSV generation service: Receive HTTP requests and write the received request fields into a CSV file.
PTS: Playback all user-recorded requests based on user-uploaded CSV files.
Experimental Process
Recording Requests from Nginx into a GOR File
Components involved in this section (other components are shown for a complete scenario): Nginx Gateway and GoReplay.
The overall architecture diagram is as follows:
Before starting to record traffic, you need to run GoReplay on the server where the gateway is located. Below is a basic command example that listens on port 80 of the gateway and saves the captured traffic to a file:
sudo gor --input-raw :80--output-file requests.gor
This command captures all traffic passing through port 80 and saves it to a requests.gor file in the current directory.
Note:
The sudo permissions are required to listen on port 80.
Converting a GOR File to a CSV Parameter File
This section describes using GoReplay to playback the requests recorded in a GOR file to a CSV generation service. Components involved: GoReplay and CSV generation service.
The overall architecture is as follows:
In the CSV file, the request fields such as scheme, host, uri, method, and base64Body would be recorded. Below is a simple example of what such a CSV file might look like:
Using base64Body instead of directly recording the body is because some request bodies send binary files, which would appear as garbled characters if directly written to a CSV file. By encoding it in base64, it facilitates subsequent conversion back to the original structure.
CSV generation service code
Write the service code and save it as a main.go file.
package main
import(
"encoding/base64"
"encoding/csv"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
)
funcmain(){
http.HandleFunc("/", requestHandler)// Set up the handler function.
log.Println("Server starting on port 8080...")
log.Fatal(http.ListenAndServe(":8080",nil))
}
funcrequestHandler(w http.ResponseWriter, r *http.Request){
// Obtain request information.
scheme :="http"// Default to http, as Go's http package does not support directly obtaining the scheme.
if r.TLS !=nil{
scheme ="https"
}
host := r.Host
uri := r.RequestURI
method := r.Method
// Convert headers to JSON format.
headersJson, err := json.Marshal(r.Header)
if err !=nil{
http.Error(w,"Error converting headers to JSON", http.StatusInternalServerError)
Playback traffic to the CSV generation service to generate CSV files.
gor --input-file requests.gor --output-http "http://csv-server-address:8080"--http-original-host true
This command will read the traffic from the requests.gor file and playback it to the CSV generation service. The CSV generation service will write the received requests into a requests.csv file by default; the host of the generated traffic will be the original host of the requests, not the address of the CSV service.
Replaying Requests Using a CSV Parameter File on PTS
PTS supports users uploading CSV files as parameter files. You can dynamically reference the test data within for use with variables in your scripts. This way, when the load generator executes this code concurrently, each request can dynamically fetch each row of data from the CSV file to use as request parameters. For specific usage of parameter files, see Using Parameter Files.
1. Log in to the TCOP Console > PTS, PTS provides a free performance testing resource package for first-time users.
2. In the left sidebar, select Test Scenario, click Create New Scenario, and choose Script Mode.
PTS script mode supports native JavaScript ES2015 (ES6)+ syntax and provides additional functions to help you quickly orchestrate performance testing scenarios under script mode. You can describe the logic required for your performance testing scenarios using JavaScript code in the online editor available in the console, such as request orchestration, variable definitions, result assertions, and common functions. (For detailed API documentation, see PTS API).
3. Upload the previously recorded CSV file as a parameter file.
4. Write a performance testing script so that each time the load generator executes the script, it reads the next row from the CSV file and uses the fields recorded in the CSV to reconstruct the original request.
The performance testing script is as follows:
// Send a http get request
import http from 'pts/http';
import { check, sleep } from 'pts';
import util from 'pts/util';
import dataset from 'pts/dataset';
export default function () {
// Read various fields of the CSV file.
var method = dataset.get("method")
var scheme = dataset.get("scheme")
var host = dataset.get("host")
var uri = dataset.get("uri")
var jsonHeaders = dataset.get("jsonHeaders")
var base64Body = dataset.get("base64Body")
var headers = JSON.parse(jsonHeaders)
var body = util.base64Decoding(base64Body, "std", "b")
// Construct a request.
var req = {
method: method,
url: scheme + "://" + host + uri,
headers: headers,
body: body
}
// Send the request.
var resp = http.do(req)
// simple get request
console.log(resp.body);
check('status is 200', () => resp.statusCode === 200, resp);
// sleep 1 second
sleep(1);
}
5. Click Save and Run to execute the performance testing script and playback the traffic. Review the performance testing report and Request Sampling to monitor if the requests meet the expectations.
