• Unable to load native-hadoop library for your platform

    windows 下在 eclipse 执行spark相关的代码时,比如:

    def main(args: Array[String]) {

    val logFile = “D:/app/spark-2.1.1-bin-without-hadoop/README.md”;

    val conf = new SparkConf().setAppName(“Simple Application”)
    val sc = new SparkContext(conf)

    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains(“a”)).count()
    val numBs = logData.filter(line => line.contains(“b”)).count()
    println(s”Scala: Lines with a: $numAs, Lines with b: $numBs”)
    sc.stop()
    }

    会提示没有hadoop 本地库:

    WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

    解决办法,把 hadoop.dll 复制到 eclipse.exe 所在目录即可,类似 tomcat 的 tcnative-1.dll 的做法。
    而 hadoop 相关的windows 相关的二进制文件默认是没有提供下载的,可以在这里下载到,根据不同的hadoop版本复制到对应的bin目录即可。
    主要是 dll 文件和 exe 文件。
    Windows binaries for Hadoop versions

    https://github.com/steveloughran/winutils

     

  • 在windows下配置spark运行环境

    将以下内容保存为 spark-env.cmd 放到 spark-2.1.1-bin-without-hadoop\bin 目录。

    前提(安装好hadoop[比如 hadoop-2.6.3 ] 并配置环境好变量)

    @echo off
    rem ###################  SET ENV ##################
    rem  set SPARK_MASTER_IP=localhost
    rem  set SPARK_WORKER_CORES=1
    set SPARK_WORKER_MEMORY=1g

    echo HADOOP_HOME: %HADOOP_HOME%
    set HADOOP_CONF_DIR=”%HADOOP_HOME%\hadoop\etc\hadoop”
    echo HADOOP_CONF_DIR: %HADOOP_CONF_DIR%

    for /f %%i in (‘hadoop classpath’) do set SPARK_DIST_CLASSPATH=%%i

    echo SPARK_DIST_CLASSPATH: %SPARK_DIST_CLASSPATH%
    mkdir -p %SPARK_HOME%\temp
    SET temp=%SPARK_HOME%\temp

    echo %temp%

    然后先执行 spark-env

    在执行 spark-shell 即可成功。

  • 配置 scala sbt 工程

    配置 scala sbt 工程

    以下描述中的版本根据自己的情况进行修改,下载不了的可能需要挂代理。

    1、 下载sbt sbt-0.13.15.zip, 解压缩至 d:\app\sbt-0.13.15
    下载 scala 并解压缩到 d:\app\scala-2.12.1
    下载地址如下:

    https://github.com/sbt/sbt/releases/download/v0.13.15/sbt-0.13.15.zip
    https://downloads.lightbend.com/scala/2.12.1/scala-2.12.1.zip

    2、 修改缓存仓库配置,修改 conf/sbtopts 中的3个路径相关的配置:

    # Path to global settings/plugins directory (default: ~/.sbt)
    #
    -sbt-dir  e:/maven/.sbt

    # Path to shared boot directory (default: ~/.sbt/boot in 0.11 series)
    #
    -sbt-boot e:/maven/.sbt/boot

    # Path to local Ivy repository (default: ~/.ivy2)
    #
    -ivy e:/maven/.ivy2

    在系统中设置环境变量(有则增加):

    SBT_OPTS= -Dsbt.global.base=e:/maven/.sbt -Dsbt.ivy.home=e:/maven/.ivy2
    PATH=d:\app\scala-2.12.1\bin;d:\app\sbt-0.13.15\bin

    3. 在新建的 scala 工程的 project/plugins.sbt 中增加 eclipse 插件配置
    参考  https://github.com/typesafehub/sbteclipse

    addSbtPlugin(“com.typesafe.sbteclipse” % “sbteclipse-plugin” % “5.1.0”)

    4、通过cmd进入工程目录(包含 build.sbt 的目录), 执行 sbt eclipse

    输出没错误,且看到

    [info] About to create Eclipse project files for your project(s).

    [info] Successfully created Eclipse project files for project(s)

    则表示创建 scala 工程成功

  • windows增加右键进入dos或linux子系统菜单

    另存 为  .reg 文件双击执行即可,不需要重启或注销

    Windows Registry Editor Version 5.00

    [HKEY_CLASSES_ROOT\Directory\Background\shell\OpenDosHere]
    @=”Dos Here”
    “Icon”=”cmd.exe”

    [HKEY_CLASSES_ROOT\Directory\Background\shell\OpenDosHere\command]
    @=”cmd.exe /s /k pushd \”%V%\””

    或者下载以下附件:

    right_menu_context_win

  • How to find auto startup folder in Windows 10

    The startup folder in Windows 10 is a good way to auto start the programs with Windows 10, here is the solution how to find the folder in Windows-10!

    The best way to open the Startup folder in Windows 10 is with the address shell:startup, you can enter this path in address bar of MS Explorer.

    Or use the Run-Dialog Box [Windows-logo] + [R] in Windows-10 and enter the command shell:startup (… see Image-1 Arrow-1)

    The second startup folder in Windows 10 shell:common startup is responsible for all users, when here, an entry is created, or deleted, this is valid for all users on the Windows-10 PC.

    Here is the alternative addresses, this can you use to create a new Windows 10 desktop shortcut.

    Autostart for currently logged-on user:
    shell:startup = %appdata%\Microsoft\Windows\Start Menu\Programs\Startup
    And startup folder all users:
    shell:common startup = %programdata%\Microsoft\Windows\Start Menu\Programs\Startup

    TIP:
    shell:start menu = start menu in Windows-10
    shell:common start menu =All user start menu in Windows-10

  • ThinkPad t440p Win10指纹解锁问题

    环境: Win10 32/64

    Win10下 联想只提供了指纹驱动,并无单独的指纹管理界面软件,需要使用Windows Hello 来管理指纹数据。
    需要正常安装 Lenovo Preferred Pro USB Fingerprint Keyboard
    驱动名称:指纹驱动程序和软件(Windows 10):  L67Y47A0.exe

    安装完成后在设备管理器能看到设备:


     

    1、安装完成重启后,需要保证Windows服务中的以下服务设置为自动启动:

    Synaptics FP WBF Policy Service  (C:\WINDOWS\system32\valWBFPolicyService.exe)
    BiometricSensorDataSynchronization (C:\WINDOWS\system32\valWbioSyncSvc.exe)

    2、在 Windows 设置 –> 账户 –> 登录选项 中开启 Pin,设置pin登录密码。

    3、在 Windows 设置 –> 账户 –> 登录选项 中 使用 Windows Hello 添加指纹。

  • 部署 artifact jar 到maven中央仓库

    部署 artifact jar 到maven中央仓库

    参考文档:
    http://central.sonatype.org/pages/apache-maven.html
    http://central.sonatype.org/pages/working-with-pgp-signatures.html

    导出创建的公钥和私钥
    gpg –export –armor –output /e/appframework.pub EBDCBB3F
    gpg –export-secret-keys –armor –output /e/appframework.key

    导入
    gpg –import e:/appframework.pub
    gpg –import e:/appframework.key

    发送公钥的服务器
    gpg –keyserver hkp://pgp.mit.edu –search-keys 【yourPublicKeyId】
    gpg –keyserver hkp://pool.sks-keyservers.net –search-keys 【yourPublicKeyId】

    发布  pgp 的密钥公钥后可以查询  http://pgp.mit.edu:11371
    gpg –list-keys

    git tag -a appframework-2.3.1 -m “release appframework-2.3.1”
    git push –tags

    mvn -Dmaven.test.skip=true clean package site

    验证公钥,看是否能对生成的jar生成asc签名文件

    mvn  -Dmaven.test.skip=true verify -Dgpg.passphrase=xxx

    如果verify 过程中产生如下错误,则表示命令行中没有导入公钥和私钥。
    [INFO] — maven-gpg-plugin:1.6:sign (sign-artifacts) @ appframework —
    gpg: no default secret key: Bad passphrase
    gpg: signing failed: Bad passphrase

    Windows 命令行里面没有导入公钥和私钥

    通过的话,直接发布,发布的用户名和密码需要在自己本地的maven的setting.xml进行配置

    mvn  -Dmaven.test.skip=true deploy

    生成 maven 站点

    mvn  -Dmaven.test.skip=true site

  • 使用python ssh 执行远程命令

    python ssh 模块: paramiko

    CentOS 组件安装:

    yum install -y python-devel
    yum install -y python-pip
    yum install -y libffi-devel
    pip install –upgrade pip
    pip install paramiko
    pip install paramiko –upgrade

    为执行此脚本的用户配置好目标机器的ssh免登录,编写python 脚本:

    import paramiko, base64
    paramiko.common.logging.basicConfig(level=paramiko.common.ERROR)
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    client.connect(‘your server ip’,username=’login name’)

    #  此命令查询目标机器上storm的6704执行进程的ID
    cmd=”ps -ef | grep -v grep | grep 6704 | grep org.apache.storm.LogWriter | awk  -F\” \” ‘{print $2}’ ”
    stdin, stdout, stderr = client.exec_command(cmd)
    for line in stdout:
    print ‘storm executor pid for xxx 6704 : ‘ + line.strip(‘\n’)
    client.close()

     

     

  • 在windows7下安装php7和apache2.4

    Windows 下
    安装 VS2015
    安装 Apache 2.4.18 Win64
    安装php 线程安全版本 php-7.0.3-Win32-VC14-x64
    复制 php.ini-development 为 php.ini, 可直接开启以下扩展:
    extension_dir = “ext”
     
    extension=php_bz2.dll
    extension=php_curl.dll
    ;extension=php_fileinfo.dll
    extension=php_gd2.dll
    ;extension=php_gettext.dll
    ;extension=php_gmp.dll
    ;extension=php_intl.dll
    ;extension=php_imap.dll
    ;extension=php_interbase.dll
    ;extension=php_ldap.dll
    extension=php_mbstring.dll
    ;extension=php_exif.dll      ; Must be after mbstring as it depends on it
    extension=php_mysqli.dll
    ;extension=php_oci8_12c.dll  ; Use with Oracle Database 12c Instant Client
    ;extension=php_openssl.dll
    ;extension=php_pdo_firebird.dll
    extension=php_pdo_mysql.dll
    ;extension=php_pdo_oci.dll
    extension=php_pdo_odbc.dll
    ;extension=php_pdo_pgsql.dll
    extension=php_pdo_sqlite.dll
    ;extension=php_pgsql.dll
    ;extension=php_shmop.dll

    ; The MIBS data available in the PHP distribution must be installed.
    ; See http://www.php.net/manual/en/snmp.installation.php
    ;extension=php_snmp.dll

    extension=php_soap.dll
    ;extension=php_sockets.dll
    extension=php_sqlite3.dll
    ;extension=php_tidy.dll
    extension=php_xmlrpc.dll
    extension=php_xsl.dll

    找到php7目录下的:libeay32.dll和ssleay32.dll,复制并替换到apache\bin目录下
    在apache 的 httpd.conf 增加如下内容:
    LoadFile “D:/x64/php-7.0.3/php7ts.dll”
    LoadFile “D:/x64/php-7.0.3/libpq.dll”
    LoadModule php7_module “D:/x64/php-7.0.3/php7apache2_4.dll”
    <IfModule php7_module>
        PHPIniDir “D:/x64/php-7.0.3/”
        #AddType application/x-httpd-php .php .phtml
        <FilesMatch \.php$>
              SetHandler application/x-httpd-php
        </FilesMatch>
    </IfModule>

    启动即可
  • linux 安装 nginx

    —- install nginx

    # yum install -y httpd-devel pcre perl pcre-devel zlib zlib-devel GeoIP GeoIP-devel
    wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.36.tar.gz
    wget http://zlib.net/zlib-1.2.8.tar.gz
    wget http://www.openssl.org/source/openssl-1.0.1m.tar.gz

    ./configure –prefix=/data/lib/zlib-1.2.8
    make & make install

    安装pcre报错You need a C++ compiler for C++ support
    解决办法 sudo apt-get install build-essential

    ./configure –prefix=/data/lib/pcre-8.36
    make & make install

    apt-get install libxml2 libxslt1-dev libxml2-dev
    apt-get install libgd2-xpm-dev
    apt-get install geoip-database libgeoip-dev
    ##apt-get install libpcre3 libpcre3-dev

    yum list | grep geo
    yum install gcc-c++

    # error: C compiler cc is not found
    yum install gcc
    # the HTTP XSLT module requires the libxml2/libxslt
    yum install libxml2-devel libxslt-devel
    # the HTTP image filter module requires the GD library.
    yum install gd-devel
    # the GeoIP module requires the GeoIP library.
    # –with-http_geoip_module
    yum install GeoIP GeoIP-data GeoIP-devel

    #首先定义新的Nginx名称:
    NGINX_BANNER=”superServer”
    #执行更名操作

    sed -i “s#\”NGINX\”#\”$NGINX_BANNER\”#” src/core/nginx.h
    sed -i “s#\”nginx/\”#\”$NGINX_BANNER/\”#” src/core/nginx.h
    sed -i “s#Server: nginx#Server: $NGINX_BANNER#” src/http/ngx_http_header_filter_module.c
    sed -i “s#\”<hr><center>nginx<\/center>\”#\”<hr><center>$NGINX_BANNER<\/center>\”#” src/http/ngx_http_special_response.c

     

    export DATA=/data/download
    export NG_HOME=/data/server/nginx
    ./configure –prefix=$NG_HOME –user=nobody –group=nobody –sbin-path=$NG_HOME/sbin/nginx –conf-path=$NG_HOME/conf/nginx.conf –error-log-path=$NG_HOME/logs/error.log –pid-path=$NG_HOME/var/nginx.pid –lock-path=$NG_HOME/var/nginx.lock –http-log-path=$NG_HOME/logs/access.log –http-client-body-temp-path=$NG_HOME/temp/client_body_temp –http-proxy-temp-path=$NG_HOME/temp/proxy_temp –http-fastcgi-temp-path=$NG_HOME/temp/fastcgi_temp –http-uwsgi-temp-path=$NG_HOME/temp/uwsgi_temp –http-scgi-temp-path=$NG_HOME/temp/scgi_temp –with-rtsig_module –with-select_module –with-poll_module –with-threads –with-file-aio –with-ipv6 –with-http_ssl_module –with-http_spdy_module –with-http_realip_module –with-http_addition_module –with-http_xslt_module –with-http_image_filter_module –with-http_geoip_module –with-http_sub_module –with-http_dav_module –with-http_flv_module –with-http_mp4_module –with-http_gunzip_module –with-http_gzip_static_module –with-http_auth_request_module –with-http_random_index_module –with-http_secure_link_module –with-http_degradation_module –with-http_stub_status_module –with-zlib=$DATA/zlib-1.2.8 –with-openssl=$DATA/openssl-1.0.1m –with-pcre=$DATA/pcre-8.36

    make && make install