tencent cloud

All product documents
Elastic MapReduce
Dumping YARN Job Logs to COS
Last updated: 2021-07-01 15:34:12
Dumping YARN Job Logs to COS
Last updated: 2021-07-01 15:34:12
By default, Hadoop stores YARN job logs in HDFS. Tencent Cloud EMR also provides the ability to store YARN job logs in external storage (COS).

Prerequisites

The EMR cluster needs to support COS. For more information, please see Analyzing Data in HDFS/COS with API.

Directions

1. Modify the configuration in yarn-site.xml and deliver the configuration to all nodes.
yarn.nodemanager.remote-app-log-dir=cosn://[bucket_name]/[logs_dirs]
2. Add a new configuration item to core-site.xml and deliver it to all nodes.
fs.AbstractFileSystem.cosn.impl=org.apache.hadoop.fs.cosnative.COS
3. Restart all the nodemanager/datanode services in the cluster.
4. Run the hive/spark job to view the job logs stored in COS.
hdfs dfs -ls cosn://[bucket_name]/[logs_dirs]
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon