spark 无法加载主类 org.apache.spark.deploy.yarn.ApplicationMaster

2019-09-23 11:17:43 | 编辑

切换spark环境后出现无法加载主类 org.apache.spark.deploy.yarn.ApplicationMaster的异常

2019-09-23 11:09:42 INFO  Client:54 - 
	 client token: N/A
	 diagnostics: Application application_1568261674011_0088 failed 2 times due to AM Container for appattempt_1568261674011_0088_000002 exited with  exitCode: 1
For more detailed output, check the application tracking page: http://slave203:8088/cluster/app/application_1568261674011_0088 Then click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e07_1568261674011_0088_02_000001
Exit code: 1

Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.spark.deploy.yarn.ApplicationMaster

Failing this attempt. Failing the application.
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time: 1569208181406
	 final status: FAILED
	 tracking URL: http://slave203:8088/cluster/app/application_1568261674011_0088
	 user: root
2019-09-23 11:09:42 INFO  Client:54 - Deleted staging directory hdfs://master:9000/user/root/.sparkStaging/application_1568261674011_0088
Exception in thread "main" org.apache.spark.SparkException: Application application_1568261674011_0088 finished with failed status
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1159)
	at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1518)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2019-09-23 11:09:42 INFO  ShutdownHookManager:54 - Shutdown hook called

 

登录后即可回复 登录 | 注册
    
  • admin
    admin

    spark.yarn.jars、spark.yarn.archive这连个配置可能导致加载不到类,切换环境要记得修改这两个配置,以及yarn-site的配置yarn.resourcemanager.address

     

关注编程学问公众号