Container-based EMR clusters allow you to submit Spark jobs and view job information in the console.
Note:
A job should be submitted as a YAML file of up to 10 MB in size.
Directions
1. Log in to the container-based EMR console and click the ID/Name of the target cluster in the Cluster list to enter the cluster details page.
2. On the cluster details page, click Job Management to submit and query jobs.
3. You can submit YAML job files through CRD in the EMR console after compiling the files.
4. Click Submit Job above the Job List to pop up the Submit Job window. Then, select the job file to be submitted and click Confirm.
5. Click Details in the Job List to enter the Spark HistoryServer UI to view the job details.
6. Click Delete in the Job List. Then, in the Delete job pop-up window, confirm the information of the job to be deleted and click Confirm.
Sample Job
The process of submitting a Spark job through CRD is as follows:
1. Write a Spark program.
2. Compile and package the program into a JAR package and put the package in the COS or HDFS file system, or write a Dockerfile to create an image for the package.
3. Write a YAML file and submit it in the console.
For more information on the parameters used in this sample, visit GitHub.
apiVersion and kind are the resource version and type in K8s, which cannot be changed here.
metadata.name defines the job name, which is test1 here and can be customized.
spec.hadoopConf defines the configuration information of Hadoop. Interacting with COS requires configuring the key information, which can be obtained on the Manage API Key page. The $SecretId and $Secretkey in the code should be replaced with your actual SecretId and Secretkey.
type defines the type of the Spark program, which can be Java, Scala, Python, or R. It is Scala here and can be selected as needed.
mode defines the deployment mode of sparkApplication, which can be cluster or client. It is cluster here and can be selected as needed.
driver and executor define the Spark driver and executor respectively. They are automatically generated on the backend as follows by default:
driver:
cores: 1
memory: 512m
executor:
cores: 1
instances: 2
memory: 512m
You can customize the driver and executor parameters and add them to the sample YAML job file 1. Then, the custom parameters will overwrite the default parameters. Below is a sample:
driver:
cores: 1
coreLimit: "1200m"
memory: "512m"
executor:
cores: 1
instances: 1
memory: "512m"
Sample 2. Compiling and packaging a Spark program and putting the JAR package in COS (recommended)
The following sample shows the complete process of compiling a Spark program, packaging it into a JAR package, and writing and submitting a YAML job file.
1. Prepare for development.
You need to have a COS bucket for this job, which can be the bucket you selected when creating the cluster or a new bucket created in the same region as the previously selected bucket.
2. Create a project with Maven.
You need to create a project and then compile, package, and upload it. Maven is recommended because it can help you manage project dependency more easily.
3. Write a WordCount program and add the following sample code:
Here, arguments is the parameters passed to the main class and indicates the input and output directories of the WordCount program. The mainApplicationFile and the input and output directories of the WordCount program here are examples and can be customized.
Sample 3. Compiling and packaging a Spark program into a JAR package and putting it in HDFS
Write a Spark program and package it into a JAR package as shown in sample 2. Then, upload the package to HDFS and write a YAML file as follows:
Yes
No
Was this page helpful?