spark on yarn 异常 spark_shuffle does not exist

spark | 2019-09-13 10:02:39

1.spark 应用场景

我的是Spark Streaming程序，spark on yarn 模式

我配置了 spark动态资源分配

spark-defaults.conf

spark.executor.cores              10
spark.dynamicAllocation.enabled               true
spark.shuffle.service.enabled                 true
spark.dynamicAllocation.minExecutors        1
spark.dynamicAllocation.maxExecutors        8
spark.dynamicAllocation.initialExecutors    1

2.异常信息

启动之后不久Streaming程序久挂掉了，在yarn的8088界面，进到application界面

Diagnostics:
reason: Max number of executor failures (16) reached

然后我还是把yarn log日志下载下来了

yarn logs applicationId application_1560>application_1560

然后有大量同样的异常信息：

19/06/12 11:03:18 ERROR YarnAllocator: Failed to launch executor 23 on container container_1558494893857_0406_02_000031
org.apache.spark.SparkException: Exception while starting container container_1558494893857_0406_02_000031 on host slave2
at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:125)
at org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:65)
at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$runAllocatedContainers$1$$anon$1.run(YarnAllocator.scala:534)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:spark_shuffle does not exist
at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:205)
at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:122)
... 5 more

3.解决方法

所有节点都要配置auxService

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle,spark_shuffle</value>
  </property>
  <property>
     <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
     <value>org.apache.spark.network.yarn.YarnShuffleService</value>
  </property>
  <property>
     <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
     <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

然后所有节点都要把spark-2.3.0-yarn-shuffle.jar从spark复制到hadoop

cp /opt/hadoop/spark-2.3.0-bin-hadoop2.7/yarn/spark-2.3.0-yarn-shuffle.jar /opt/hadoop/hadoop-2.7.7/share/hadoop/yarn

注意：记住是所有节点都要配置！

登录后即可回复登录 | 注册

spark on yarn参数优化配置解决spark hive插入数据异常spark currently does not populate bucketed output spark hive 异常version information not found in metastore spark on yarn资源分配性能调优静态和动态 spark操作mongodb异常 cannot cast bsonvalue spark异常 could not locate executable null bin winutils.exe in the hadoop binaries spark submit yarn提交任务异常error initializing sparkcontext spark hive 元数据异常 filenotfoundexception spark 异常 missing an output location for shuffle spark on yarn 异常 spark shuffle does not exist spark 异常 spark conf / hadoop conf bad substitution spark hive 异常 could not connect to meta store using any of the uris provided php sqlite 函数 opens an sqlite database and create the database if it does not exist php sqlite 函数 opens a persistent handle to an sqlite database and create the database if it does not exist php sqlite 函数 execute a query that does not prefetch and buffer all data php wincache 函数 adds a variable in user cache only if variable does not already exist in the cache php mongocollection creates an index on the specified field s if it does not already exist php mongocollection creates an index on the specified field s if it does not already exist Chrome DevTools Protocol异常PrintToPDF is not implemented elasticsearch异常query does not support [auto_generate_synonyms_phrase_query]