tencent cloud

Feedback

EdgeOne Anti-fraud Practice Tutorial

Last updated: 2024-08-27 10:54:56
    This document is expected to take 20 minutes to study. By studying this document, you can learn about:
    1. What is CDN fraud, its common types and harms.
    2. How to set up traffic alarms and usage cap policies on the EdgeOne platform, enable real-time log push, and prevent CDN fraud.
    3. Using EdgeOne's Traffic Analysis and Log Analysis features, recognize and locate CDN fraud attack traffic.
    4. Configuration recommendations for EdgeOne anti-CDN-fraud practical tutorials for small and medium websites and enterprise-level business platforms.

    What is CDN fraud?

    CDN fraud refers to unauthorized users using illegal means to massively obtain website resources, consuming website bandwidth and server resources. Compared to DDoS attacks, which directly affect website availability, CDN fraud primarily consumes the website's bandwidth and other computing resources, causing sudden high bandwidth or high traffic and leading to fees higher than usual daily consumption, thereby significantly increasing operational costs. Common methods of CDN fraud include:
    Sending large volumes of false requests through automation tools, proxy servers, or botnets;
    Continuously downloading large files or transferring large amounts of data using automation tools;
    Send a large number of concurrent requests through the load testing tool to perform overload testing on the server.
    Note:
    We strongly recommend that Tencent Cloud CDN users upgrade to EdgeOne and implement appropriate security measures within the platform. By setting up robust protection strategies, you can significantly mitigate the risk of CDN fraud attacks, safeguard your normal business operations, and prevent unexpected high bills. For migration steps, please refer to CDN services migration to EdgeOne Tool Usage Guide.

    Precautions

    Set Usage Cap Policy

    Implementing a usage capping policy for key website metrics (such as bandwidth, traffic, and request volume) is an effective strategy to prevent excessive bills caused by CDN fraud attacks. We recommend setting reasonable usage limits and alert thresholds for these indicators. If an alert is triggered, promptly investigate the real-time requests using the outlined investigation measures and respond according to the prescribed countermeasures.
    Note:
    Please be aware that it takes approximately 10 minutes for usage capping policy to activate. During this period, usage will be billed normally.
    Usage capping policies are calculated on a per-subdomain basis. When the policy is applied to an entire site or all subdomains, all subdomains under that site will share a single capping policy.
    If multiple policies (traffic, bandwidth, or request number) are set for the same domain, service suspension will be triggered as soon as any one of these metrics reaches its threshold.
    Currently, the usage capping policy only supports L7 (application layer) traffic/bandwidth and HTTP/HTTPS request configurations. L4 (transport layer TCP/UDP applications) traffic and other value-added services such as QUIC and BOT Management are not yet supported.

    Configuration Samples

    For detailed steps on configuring usage cap strategies, please refer to Usage Cap Strategy. In the Add Capping Policy window, select the effective site and configure the capping strategy based on the following suggestions:
    Configuration Dimension
    Configuration Options
    Corresponding Suggestions
    Use cases
    Statistical Periods
    5 minutes (recommended choice)
    Set a lower threshold to quickly detect and respond to abnormal traffic or requests.
    This enables the timely detection of abnormal traffic or request peaks within a short period, allowing for quick preventive measures, suitable for real-time monitoring and immediate response needs.
    Hour
    Set a medium threshold, combining data from normal business peak periods to avoid false cap triggers during short-term traffic surges.
    Capture short-term traffic fluctuation trends, providing some response time for protection adjustment.
    Days (24 hours)
    Set a higher threshold, based on 2-3 times the normal daily business traffic, to ensure the recognition of abnormal traffic over a long period.
    Provides a global perspective to identify abnormal traffic or request patterns throughout the day, suitable for formulating long-term protection policies and resource planning.
    Cap Configuration
    Seven-layer traffic (recommended)
    Set the traffic threshold based on 2-3 times the normal business traffic to handle traffic surges and prevent the cap from being mistakenly triggered by short-term normal traffic growth.
    Effectively prevent attackers from consuming bandwidth resources through massive downloads of large files.
    HTTP/HTTPS Number of requests
    Set the threshold based on 2-3 times the normal request count to ensure the peak period of normal business does not mistakenly trigger the cap.
    Effectively prevent request flooding attacks by consuming resources with numerous false requests.
    Layer 7 Bandwidth
    Set the bandwidth threshold based on 2-3 times the normal bandwidth usage to handle bandwidth usage surges.
    Effectively prevent excessive bandwidth consumption and avoid resource waste caused by large traffic download attacks.
    Exceed the threshold
    Stop the service, and you need to re-enable it in the domain list.
    Alarm Threshold
    50% (recommended), an alarm message is sent when the usage reaches 50% of the configured alarm threshold.
    Note:
    If the alarm threshold is enabled: As the scan interval is five minutes, if the usage surges significantly in a short period, the previous scan may not trigger the alarm threshold, and the next scan directly reaches the access threshold. In this case, both percentage alarm and access threshold alarm notifications will be sent.

    Enable Real-time Log Push

    For more granular protection, we recommend enabling the Real-time Log Push feature. This tool delivers access logs to your specified destination with minimal latency and can be configured via the console or API. Logs typically arrive within 5 minutes of the initial request, making this feature ideal for real-time monitoring and rapid troubleshooting, especially in preventing CDN fraud attacks. By analyzing access patterns in real time, you can swiftly identify characteristics of CDN fraud attacks and configure targeted interception policies.
    Here's a breakdown of the request types recorded by each log category:
    Site Acceleration Log: Captures domain access logs, including all L7 requests through CDN. By default, it only logs requests that pass protection, not blocked requests. These comprehensive logs help identify abnormally high-frequency requests, unusual traffic patterns, and potential CDN fraud behavior.
    Note:
    The Site Acceleration Log's full L7 Request Log feature, including L7 Protection Blocked Log, is currently in beta. Please Contact Us if you need access.
    Rate Limiting and CC Attacks Protection Log: Only logs requests that hit the L7 Protection - Rate Limiting and CC Attack Protection Module security rules, whether they are blocked or not. It helps identify behaviors attempting to swipe traffic through high-frequency requests.
    Managed Rules Log: Only logs requests that hit the L7 Protection - Managed Rules Module security rules, whether they are blocked or not. It helps detect protection based on managed rules, identifying potential attacks and CDN fraud behavior.
    Custom Rules Log: Only logs requests that hit the L7 Protection - Custom Rules Module security rules, whether they are blocked or not. It helps identify abnormal requests that match custom rules, preventing specific types of CDN fraud behavior.
    Bot Management Log: Only logs requests that hit the L7 Protection - Bot Management Module security rules, whether they are blocked or not. It helps identify CDN fraud behavior triggered by automated scripts or malicious bots.
    Note:
    Bot Management Log is supported only after Bot Management capability is enabled for the site domain. For pricing details, please refer to Value-Added Service Usage Unit Fee (Pay-as-You-Go).
    If you need to include specific field values from HTTP Request Headers, HTTP Response Headers, or Cookies in your logs, you can use the Custom Push Log Fields feature to accurately capture this information.

    Investigation Measures

    After implementing the preventive measures described earlier, if you receive an alarm indicating a sudden, significant increase in usage, it's time to conduct a thorough investigation. This section focuses on using EdgeOne's Traffic Analysis and Log Analysis features to perform multi-dimensional analysis and pinpoint suspected unauthorized traffic causing excessive charges.

    Analytics

    Analytics is a powerful data analysis service offered by EdgeOne, aimed at helping users gain deep insights into business operations and security status. By monitoring and analyzing key indicators in real time, users can quickly identify issues, optimize resource allocation, and enhance the stability and security of their business. In the context of investigating brushing attacks, it is recommended to focus on the following data through How to use filter condition, in combination with TOP rankings:
    Referer Distribution: A high concentration of blank or illegitimate Referers often indicates credential stuffing attempts, crawler activity, or other malicious requests.
    Changes in URL Resource Type Visit Volume: If the request volume for a small number of URLs or resource types spikes dramatically compared to others, this may indicate targeted unauthorized usage.
    Client IP TOP Rankings: Observe whether a small number of IPs are responsible for a disproportionate number of requests. This information can help you evaluate the feasibility of implementing IP-based request frequency controls.

    Operation step

    1. Log in to the EdgeOne console. In the left sidebar, click Analysis.
    2. On the Analysis page, click Add filter to include sites with usage alerts in the filter.
    3. Select the date range during which the suspected CDN fraud activity occurred.
    
    4. In the L7 client traffic section, check the TOP rankings for the following dimensions:
    Hosts: Subdomains requested by the client.
    URL paths: Specific resource paths requested by the client.
    Resource Types: Types of resources requested by the client, such as: ".png", ".json", etc.
    Client IPs: The specific source IP address of the client request.
    Referers: Referer information of the client's request.
    Client Device Types:
    Device Types: Type of hardware device used for the client request, including:
    TV: Television.
    Tablet: Tablet.
    Mobile: Mobile phone.
    Desktop: Computer.
    Other: Other.
    Browser: The type of browser used by the client.
    Operating System: The type of operating system used by the client.
    
    5. Click Add filter to add the following suggested filter conditions. Focus on abnormal traffic, click OK.
    Referer: Identify empty Referer requests;
    URL: Includes TOP 5 URLs, identify suspicious hotspot resources;
    Resource types: Includes TOP 5 resource types, identify the type distribution of hotspot resources;
    Client Device/Browser/Operating System: Equal to Other; Empty, identify suspicious unconventional clients.
    
    6. Observe the distribution of each indicator after filtering, identify data that deviates significantly from normal levels, and analyze its correlation with fraud.

    Offline Log Analysis

    To further discover more characteristics of CDN fraud requests, an in-depth analysis of the alert occurrence period offline logs is required. Through a comprehensive analysis of fields, we can depict the profile of CDN fraud requests from multiple dimensions such as source IP, URL path, request parameters, User-Agent, and Referer source, laying the data foundation for Next to formulate precise countermeasures. The following are the key log fields and corresponding descriptions for offline log analysis in the investigation of fraud:
    Field Name
    Data Type
    Note
    Does Offline Logging Support This Field?
    Does Real-time Logging Support This Field?
    RequestUrl
    String
    The URL path of the client request, excluding query parameters. This field is a key analysis dimension for traffic attacks.
    RequestUrlQueryString
    String
    Query parameters in the client's URL request. If the query parameters of spam requests are fixed or have obvious characteristics, you can set a blocklist for the source IP of the requests or for requests that match the parameters.
    RequestUA
    String
    User-Agent information of the client request. Simple spam tools often use the same User-Agent. If access is concentrated on a specific and uncommon User-Agent, consider blocking it.
    RequestReferer
    String
    Referer information of the client request. The Referer of a normal request is usually the URL of another page on the site or a search engine URL, while curl or other TCCLI might forge the Referer. If the URL being spammed is not actually referenced by other sites but a Referer appears, it can be deemed abnormal. You can block this by configuring Referer Hotlink Protection.
    ClientIP
    String
    The Client IP connected to the EdgeOne node, i.e., the source IP of the request. If a small number of IPs far exceed others in access volume, consider blocking them.
    EdgeResponseBodyBytes
    Integer
    The response body size returned by the node to the client, measured in bytes. Malicious traffic often repeatedly downloads large files, and analyzing the statistical results of EdgeResponseBodyBytes is a critical step in traffic analysis.
    For more fields and their corresponding descriptions, please refer to Layer 7 Access Log Field Description.
    For detailed steps on downloading offline logs, please refer to Offline Logs.

    Countermeasures

    Addressing website fraud is a complex and ever-evolving challenge that requires a tailored approach. There's no one-size-fits-all solution. EdgeOne offers a comprehensive suite of protection features, including access control and rate limiting, which can be flexibly combined to create a robust defense strategy. To effectively combat fraud, it's crucial to select the optimal protection configuration based on both the specific characteristics of the attacks you're facing and your unique business requirements. We've developed a detailed EdgeOne CDN Fraud Prevention Best Practice Tutorial from the perspectives of personal site operators and online business sites to guide you through this process.

    Small and Medium Website Platforms

    Scenario One: Block of Abnormal Source IP Based on Traffic Analysis

    Example
    During the suspected fraudulent traffic period, by analyzing the L7 access traffic resource type ranking metrics, an anomaly was found in the proportion of access to a 5MB file. Further investigation revealed the file path as /test/installer.apk, with requests primarily coming from the 1.1.1.0/24 network segment's Client IPs. Based on this clue, an IP blacklist policy can be quickly created to intercept this malicious network segment and curb potential fraudulent traffic behavior.
    Recommended Configuration
    We recommend you use EdgeOne Web Security Feature's Custom Rules to configure protection policies. For detailed steps, please refer to Custom Rules.
    For individual users, you can configure a rule type as Basic Access Control in, with Client IP Control, and select the matching method as Client IP is 1.11.32.0/24 and the action as Block.
    
    For basic and higher level users, exact matching rules can be configured in Precise Matching Rules, with the matching field as Client IP Matches 1.1.1.0/24 and the request path includes /test/installer.apk, the action taken as JavaScript Challenge.
    

    Scenario Two: Block of Abnormal User-Agent Based on Log Analysis

    Example
    Real-time logs show that within a certain period, the distribution of RequestUA is abnormally concentrated. Further analysis found that the most frequent access ispython-requests/2.22.0, and there are also a large number of requests using python-requests/ and other Python script-specific User-Agent identifiers. Since these requests significantly deviate from the regular browser's User-Agent characteristics, they can be identified as automated requests or even malicious bots. Based on this, you can configure a User-Agent blocklist rule to precisely intercept suspicious requests containing specific User-Agent identifiers.
    Recommended Configuration
    We recommend using the EdgeOne Web Security feature with custom rules to set up protection policies. For specific operations, please refer to Custom Rules.
    For personal edition users, in Basic Access Control, you can configure a rule with the type User-Agent Management, match method as request User-Agent Is, match content as python-requests/2.22.0, and action as Block.
    
    For basic edition and above users, in Precise Access Control, you can configure the match field as User-Agent Include python-requests, and the action as JavaScript Challenge.
    

    Scenario three: Prevention and Interception Based on Known Malicious User-Agent

    Example
    For known common scraping tools, you can configure their characteristic User-Agent strings into your Custom rules in advance. By preemptively activating this rule globally on your site or key paths, you can significantly reduce the risk of being scraped by such tools. Common scraping User-Agent include:empty User-Agent; curl/xx.xx; Wget/xx.xx; ApacheBench/xx.xx; python-requests/xx.xx.
    Recommended Configuration
    We recommend you use EdgeOne Web Security Feature's Custom Rules to configure protection policies. For detailed steps, please refer to Custom Rules.
    For personal edition users, you can configure two rules in Basic Access Control;
    Rule 1: Configure the rule type as User-Agent Management, with the matching method as request User-Agent is empty, and action as Block.
    Rule 2: In Basic Access Control, configure the rule type as User-Agent Management, matching method as request User-Agent Matches wildcard pattern curl/; Wget/; ApacheBench/; python-requests/, and action as Block.
    
    For Basic Edition and above users, you can configure the following two rules in Precise Access Control:
    Rule 1: In Precise Access Control, configure the match field as User-Agent Containscurl/; Wget/; ApacheBench/; python-requests/ requests, with action as JavaScript Challenge.
    
    Rule 2: In Precise Access Control, configure the match field as User-Agent is empty, with action as JavaScript Challenge.
    

    Scenario 4: Allowing Only Common User-Agents (Temporary High Defense)

    When experiencing large-scale, dispersed User-Agent scraping, if it is difficult to identify malicious User-Agent characteristics one by one, you can use reverse logic to allow access only to common legitimate User-Agents from normal browsers and apps. This method can filter a large number of suspicious requests at once, but due to the strictness of the rules, there is a risk of misjudgment, so it should be used cautiously in conjunction with other dimensional features.
    Recommended Configuration
    We recommend using the EdgeOne Web Security feature with custom rules to set up protection policies. For specific operations, please refer to Custom Rules.
    For personal users, you can configure the rule type as User-Agent Management in Basic Access Control, with the matching method as Request User-Agent Does not match pattern, matching content as *Linux*; *Macintosh*; *Android*; *iPhone*; *iPad*; *Windows*, and disposal method as Block.
    
    For basic and above users, you can configure the match field as User-Agent Does not match wildcard patternAndroid, iPhone, iPad, Mac, Windows, Linux requests in Precise Access Control, with the action as JavaScript Challenge.
    
    Note:
    This strategy is not applicable for app scenarios where normal operations have an empty User-Agent.
    If the User-Agent value is the App name, you need to add the App name of normal operations into the matching content.
    Higher intensity, configure cautiously. To avoid false blocking, please use joint judgment with other features.

    Scenario Five: Set CC Attack Single IP High-Frequency Access Restriction (Temporary High Defense)

    CC Attack Protection identifies CC attacks through rate baseline learning, header characteristic statistical analysis, and Client IP intelligence, among other methods, and addresses them. EdgeOne offers three preset CC attack protection strategies:
    Adaptive frequency control: used to combat CC attacks that occupy server resources through high-frequency and large concurrency connection requests, and can limit access frequency based on a single IP source.
    Slow attack defense: used to combat CC attacks that occupy server resources through a large number of slow connection requests, and can limit the minimum connection rate for access based on a single session, eliminating slow connection clients.
    Client filtering: integrates rate baseline learning, head feature statistical analysis, and client IP intelligence, and dynamically generates protection rules in real-time. Requests from high-risk clients or those with high-risk head features are subject to human-machine verification. Intelligent client filtering is enabled by default and performs a JavaScript Challenge for clients that meet the criteria.
    In case of a suspected website scraping attack or abnormal usage alarm, it is recommended to temporarily set Adaptive Frequecy Control to Adaptive - Emergency level, with the handling method being JavaScript Challenge. This measure can effectively block a large number of malicious IP requests, preventing scraping and other attacks efficiently. For specific operations, please refer to CC Attack Protection.
    
    Note:
    Please revert the Adaptive frequency control level to the recommended configuration, Adaptive - Loose, promptly after dealing with the scraping attack, to ensure smooth access to normal business traffic. For more details on the descriptions of various limitation levels, refer to CC Attack Protection.

    Scenario Six: Personalized Frequency Control Based on Business Water Level

    Distinct from DDoS strong attacks, CDN fraud tends to be more covert and requires judgment based on specific business scenarios to formulate Personalized Rate Strategy to avoid mistakenly intercepting legitimate users. Whether the interception strategy is targeting IP or User-Agent, it falls under precise interception. However, in actual attacks, the attack characteristics might not be obvious, especially when the request volume from a source IP could be as high as hundreds of thousands.
    Defense strategies combined with the business scenario first require the site administrators to assess the normal access pattern of the business and determine the business traffic baseline. For example, in App download or upgrade scenarios, most IPs usually perform only one or two downloads. In rare cases, there might be multiple attempts due to failures, but usually within a reasonable frequency range. Anomalously high-frequency access is likely a sign of an attack or malicious CDN fraud.
    When a website suffers from scraping attacks, the domain's bandwidth will significantly increase. To address this, it is recommended to use EdgeOne Web Security feature's rate limiting, set thresholds based on normal business levels, configure speed limit policies, or monitor and adjust policies through Real-time Logs. For specific operations, please refer to Rate Limiting.
    Note:
    When configuring frequency control rules, they should be dynamically adjusted based on the actual defense effectiveness. Initially, you can set the frequency thresholds based on empirical values to quickly achieve defense. If these are found to be ineffective, gradually tighten them; conversely, if the rules affect normal business, they should be appropriately loosened.
    Game Package Download Frequency Limit based on the Business Baseline
    Example
    A game platform offers multiple game installation and update package download services through EdgeOne Accelerated Distribution. The download URL of the game package has a fixed pattern, for example:
    Game A Installation Package: https://cdn.example.com/games/A/installer_v1.0.zip
    Game A Update Package: https://cdn.example.com/games/A/patch_v1.1.zip
    Game B Installation Package: https://cdn.example.com/games/B/installer_v2.0.exe
    Game B Update Package: https://cdn.example.com/games/B/patch_v2.1.exe
    On the day of the game's version release, the number of downloads per single IP is usually 1, with retry downloads due to individual network issues not exceeding 3. However, certain IPs frequently download installation and update packages after the version release, far exceeding the behavior of normal users, thus triggering a usage alarm. It is speculated that this could be due to piracy websites or sharing communities capturing game packages, or attackers intending to consume bandwidth. These malicious requests can be promptly blocked by configuring Rate Limiting rules.
    Recommended Configuration
    In Rate limit, click Add rule. Set the condition where the Request URL Containsgames/; installer/; patch/AND Request method is GET. Set rate limiting for individual client IPs, counting requests (client to EdgeOne). Trigger the action JavaScript Challenge for client IPs that exceed 3 requests within 10 minutes. Set the action duration to 1 hour. For detailed steps, please refer to Rate Limiting.
    
    Abnormal User-Agent Rate Limiting Based on Log Analysis
    Example
    A website experienced a surge in traffic from malicious actors, triggering usage alerts. Analysis of real-time logs revealed that while the attacking IP addresses were widely dispersed, suggesting a distributed attack, the User-Agents were suspiciously uniform.
    The vast majority of requests came from a single User-Agent string: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0). This pattern deviated significantly from normal traffic, which typically exhibits a diverse range of User-Agents representing various browsers and devices. Under normal circumstances, this particular User-Agent would account for only a small fraction of traffic. However, during the attack, it dominated the traffic profile. This anomaly strongly indicated a CDN fraud attack. These malicious requests can be promptly blocked by configuring Rate Limiting rules.
    Recommended Configuration
    In Rate limit, click Add rule. Set the condition where the User-Agent Is/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0). Set rate limiting for individual client IPs, counting requests (client to EdgeOne). Trigger the action JavaScript Challenge for client IPs that exceed 400 requests per minute. Set the action duration to 30 minutes. For detailed steps, please refer to Rate Limiting.
    
    Note:
    You need to adjust the threshold and disposal duration for triggering protection based on the normal business water level of your own operations, attacker characteristics in real-time logs, and frequency assessment.

    EdgeOne Hotlink Protection Practical Tutorial

    In addition to direct protection measures against unauthorized charging, websites should pay attention to protecting the resources themselves and adopt proactive defense. Hotlink protection is an important means of preventing unauthorized use of website resources.
    Hotlinking refers to the act of illegally referencing resources (such as images, videos, software packages, etc.) from the original site on other sites without the permission of the website owner, consuming the original site's bandwidth and resources. It not only infringes the legal rights of the original site but may also have adverse SEO impacts. Therefore, it is necessary to actively implement hotlink protection measures.
    EdgeOne offers a comprehensive hotlink protection solution, which includes Referer Hotlink Protection, Token Hotlink Protection, and Remote Authentication, among others, to control hotlinking behavior. This protects your content from unauthorized hotlink access and enhances the security of acceleration services. For more details, please refer to the EdgeOne Hotlink Protection Practical Tutorial.

    Enterprise-grade business platform

    For online business sites facing fraud threats, in addition to using general protection measures commonly adopted for personal sites, it is recommended to choose EdgeOne Standard or Enterprise Edition and enable the Bot Management feature. With its built-in artificial intelligence engine and extensive behavioral feature analysis, you will gain a smarter, more effortless Bot management experience, effectively addressing various CDN fraud attacks.
    Bot intelligence Analysis module employs advanced machine learning algorithms to form a threat identification model trained with massive data. This model extracts key behavioral characteristics from multiple dimensions such as request rate, IP intelligence, URL sequence, and SSL/TLS fingerprint. Techniques like cluster analysis and similarity comparisons accurately determine whether the request origin is automated and whether it has malicious intent, reducing false positives for legitimate requests with a comprehensive and multi-dimensional analytical approach.
    Additionally, the EdgeOne Enterprise Edition also supports JA3 fingerprint characteristics. Website administrators can preset fingerprint conditions for high-risk Bots based on their business scenarios, achieving precise interception of specific attack tools. For example, incorporating fingerprints of commonly used malicious crawler Python libraries and headless browsers into fraud defense rules can automatically intercept related traffic, making protection more proactive and efficient.
    Note:
    The Bot Management feature is supported only after the site domain has enabled Bot Management capability. The pricing standard for Bot Management after enabling can be found in: Value-Added Service Usage Unit Fee (Pay-as-You-Go).
    
    
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support