问题

spark提交任务失败出现下面异常

java.net.BindException: Address already in use: Service 'SparkUI' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'SparkUI' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:433)
	at sun.nio.ch.Net.bind(Net.java:425)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.spark_project.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:351)
	at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:319)
	at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
	at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$newConnector$1(JettyUtils.scala:352)
	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$httpConnect$1(JettyUtils.scala:381)
	at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:384)
	at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:384)
	at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2275)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2267)
	at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:384)
	at org.apache.spark.ui.WebUI.bind(WebUI.scala:130)
	at org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:451)
	at org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:451)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:451)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
	at com.report.tool.datacenter.engine.spark.analysis.DataAnalysisMain$.main(DataAnalysisMain.scala:43)
	at com.report.tool.datacenter.engine.spark.analysis.DataAnalysisMain.main(DataAnalysisMain.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)

原因

每一个Spark任务都会起一个Driver端口，即SparkUI，默认为23000，如果被占用则随机选取端口重试，默认会重试16次。16次重试都失败后，会放弃任务的运行。

解决

使用以下步骤中的任何一个可以解决。

1. 初始化SparkConf时，添加conf.set(“spark.port.maxRetries”,“100”)语句
2. 使用spark-submit提交任务时，在命令行中添加 --conf spark.port.maxRetries=100
3. 在spark-defaults.conf中添加spark.port.maxRetries 100

可以将100替换为任何想要的数字，数字越大，允许同时运行的任务越多。