• Unable to load native-hadoop library for your platform

    windows 下在 eclipse 执行spark相关的代码时,比如:

    def main(args: Array[String]) {

    val logFile = “D:/app/spark-2.1.1-bin-without-hadoop/README.md”;

    val conf = new SparkConf().setAppName(“Simple Application”)
    val sc = new SparkContext(conf)

    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains(“a”)).count()
    val numBs = logData.filter(line => line.contains(“b”)).count()
    println(s”Scala: Lines with a: $numAs, Lines with b: $numBs”)
    sc.stop()
    }

    会提示没有hadoop 本地库:

    WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

    解决办法,把 hadoop.dll 复制到 eclipse.exe 所在目录即可,类似 tomcat 的 tcnative-1.dll 的做法。
    而 hadoop 相关的windows 相关的二进制文件默认是没有提供下载的,可以在这里下载到,根据不同的hadoop版本复制到对应的bin目录即可。
    主要是 dll 文件和 exe 文件。
    Windows binaries for Hadoop versions

    https://github.com/steveloughran/winutils