spark异常 Could not locate executable null\bin\winutils.exe in the Hadoop binaries

2019-09-13 10:02:39 | 编辑

idea中开发spark程序出现异常:

18/12/10 11:09:36 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:378)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:393)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:386)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:116)
at org.apache.hadoop.security.Groups.<init>(Groups.java:93)
at org.apache.hadoop.security.Groups.<init>(Groups.java:73)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:293)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:789)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2464)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:292)
at SparkTestMain$.main(SparkTestMain.scala:6)
at SparkTestMain.main(SparkTestMain.scala)

原因是程序找不到HADOOP_HOME里面的winutils.exe。

解决方法

先下载:https://github.com/srccodes/hadoop-common-2.2.0-bin

然后再程序中设置代码: System.setProperty("hadoop.home.dir", "D:\\bigdata\\hadoop-2.7.3")


最佳办法是判断是否开发环境来设置:

if (args(5).equals("deve")) {

            System.setProperty("hadoop.home.dir", "D:\\bigdata\\hadoop-2.7.3")

            masterUri = "local[*]"

        }


登录后即可回复 登录 | 注册
    
关注编程学问公众号