Stream Compute Service
- Releases Notes and Announcements
- Getting Started
- Operation Guide
- Managing Jobs
- Developing Jobs
- Monitoring Jobs
- Events and Diagnosis
- Managing Metadata
- Tuning Jobs
- Managing Clusters
- Managing Permissions
- SQL Developer Guide
- DDL Statements
- Connectors
- SET Statement
- Operators and Built-in Functions
- Identifiers and Reserved Words
- ETL Developer Guide
Creating an ETL Job
Last updated: 2023-11-08 10:51:17
Prerequisites
ETL jobs must be executed in a private cluster. If you don't have one yet, see Creating a Private Cluster.
Step 1. Create an ETL job
Log in to the Stream Compute Service console. Enter a workspace and click Jobs on the left menu. On the Jobs page, click Create job, select ETL job as the type, enter a job name, select a running cluster (where the job will be executed), and click Confirm.
Note
Currently, ETL jobs only support Flink 1.13. If your cluster does not support Flink 1.13, please submit a ticket to update it first.
Step 2. Grant access to related services
Select the job created and click Development & Testing. If you haven't granted Stream Compute Service access to related services, in the window that pops up, click Authorize now to allow Stream Compute Service to access resources such as message queues and cloud databases. For detailed directions, see Granting access to other services.
Step 3. Configure the data source
Click Job development > Draft and drag the MySQL data source from the left to the canvas.
Click to configure the data source.
If you don't have an available database instance, click Manage connections and then click Create connection to add a new data source.
If you select Tencent Cloud instance, you can add a Tencent Cloud instance of the current account (which must have permission to query the corresponding Tencent Cloud product).
If you select IP connection, the IP address entered will be connected. Make sure the address is in the same VPC as the cluster or is connected with the cluster.
After selecting the data source, from the drop-down list of Database/table, choose a database and table to synchronize.
Click Next, select the fields to synchronize, and click Confirm.
Return to the canvas. You will see that the data source has been configured.
Step 4. Configure the sink
Drag a data sink from the left to the canvas and connect it to the data source.
Click the sink to configure the database instance and select the database/table.
Click Next and map the fields.
Click Confirm.
Step 5. Configure job parameters
Step 6. Publish and run the ETL job
Click Run to check and run the job.
Step 7. View logs
Click Log to view the logs of the job.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No