tencent cloud

All product documents
Stream Compute Service
DocumentationStream Compute ServiceOperation GuideDeveloping JobsConfiguring Job Resources
Configuring Job Resources
Download PDF
Last updated: 2023-11-08 10:05:51
Configuring Job Resources
Last updated: 2023-11-08 10:05:51
Download PDF

Overview

You can configure the JobManager spec, TaskManager spec, and default operator parallelism in Job parameters > Resources. In the Resources section, you can configure compute resources suitable for your job.
Stream Compute Service provides three CU specs: 0.5 CU, 1 CU, and 2 CUs. The JobManager spec and TaskManager spec can be set to different values. 0.5 CU represents fine-grained resources, which are available only to some new clusters by default. The JobManager and TaskManager spec options are not displayed for existing clusters. If you need to configure these two options for an existing cluster, please contact us.

CUs used by a job

CUs used by job = JobManager spec + TaskManager spec x maximum parallelism of all operators in job.
You can adjust the specs and the default operator parallelism based on the maximum ‍number of CUs available to the job given in the Resources section. For example, the figure below shows that a maximum of 8 CUs are available to the current job, so if you set the TaskManager spec to 2 CUs, the default operator parallelism can be set to 3 at most.

Resources for scenarios with low resource consumption

Some jobs may have no data sync most of the time, and using 1-CU resources causes a waste of resources. For such jobs, you can adjust the JobManager and TaskManager specs to 0.5 CU (1 CU in total) and the default operator parallelism to 1, reducing waste.

FAQs

1. What are fine-grained resources? Fine-grained resources refer to resources whose compute unit specs can be smaller than 1 CU (1 core and 4 GB memory). Stream Compute Service currently provides three CU specs: 0.5 CU, 1 CU, and 2 CUs. The JobManager and TaskManager specs can be set to different values.
2. Why does the actual job parallelism fail to reach the maximum parallelism of the job? If fine-grained resources are used, there is a remote possibility that resource fragmentation affects the job running, leading to an actual parallelism of the job smaller than the maximum one. In this case, you can select appropriate resource specs to avoid resource fragmentation as much as possible. If you need any help, please contact us.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon