This page is for educational use only.

Used information from rockfishsec and the hashcat wiki.

• Go to Amazon's web services, EC2.
• Click Launch Instance.
• Click AWS Marketplace.
• Search for 'Linux NVIDIA', select “Amazon Linux AMI with NVIDIA GRID and TESLA GPU Driver”.
• Click Continue.

• Filter by 'GPU instances'.
• Select g2.8xlarge
• Click 'Review and Launch'.
• Select: Make General Purpose (SSD) the default boot volume for all instance launches from the console going forward (recommended).

• Edit your security group.
• Review the details / 16 GB storage, don't delete on termination.
• Create a new key pair.
• Launch the instance. At first i was not able to launch the instance. Opened a support case. Error: You have requested more instances (1) than your current instance limit of 0 allows for the specified instance type.

• Log in with your key:

  chmod 400 <keyfile>
  ssh -i <key> ec2-user@<hostname>
  
         __|  __|_  )
         _|  (     /   Amazon Linux AMI
        ___|\___|___|
  
  https://aws.amazon.com/amazon-linux-ami/2016.09-release-notes/
  6 package(s) needed for security, out of 11 available
  Run "sudo yum update" to apply all updates.

• Update the instance and reboot if needed.
• Quick hardware check (32 CPU's and 4 GPU's):

  $ lscpu
  Architecture:          x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
  CPU(s):                32
  On-line CPU(s) list:   0-31
  Thread(s) per core:    2
  Core(s) per socket:    8
  Socket(s):             2
  NUMA node(s):          2
  Vendor ID:             GenuineIntel
  CPU family:            6
  Model:                 45
  Model name:            Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
  Stepping:              7
  CPU MHz:               2600.068
  BogoMIPS:              5268.46
  Hypervisor vendor:     Xen
  Virtualization type:   full
  L1d cache:             32K
  L1i cache:             32K
  L2 cache:              256K
  L3 cache:              20480K
  NUMA node0 CPU(s):     0-15
  NUMA node1 CPU(s):     16-31
  
  $ lspci
  00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
  00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
  00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
  00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
  00:02.0 VGA compatible controller: Cirrus Logic GD 5446
  00:03.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
  00:04.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
  00:05.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
  00:06.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
  00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)

• Install packages and install from source:

  sudo yum install git
  
  git clone https://github.com/hashcat/hashcat.git
  cd hashcat
  mkdir -p deps/OpenCL-Headers
  git clone https://github.com/KhronosGroup/OpenCL-Headers deps/OpenCL-Headers/CL
  cd ../..
  make
  sudo make install

• See if hashcat sees the GPU's:

  ./hashcat --opencl-info
  hashcat (v3.10-809-g7fe575e+) starting...
  
  OpenCL Info:
  
  Platform ID #1
    Vendor  : NVIDIA Corporation
    Name    : NVIDIA CUDA
    Version : OpenCL 1.2 CUDA 7.5.30
  
    Device ID #1
      Type           : GPU
      Vendor ID      : 32
      Vendor         : NVIDIA Corporation
      Name           : GRID K520
      Version        : OpenCL 1.2 CUDA
      Processor(s)   : 8
      Clock          : 797
      Memory         : 1023/4095 MB allocatable
      OpenCL Version : OpenCL C 1.2
      Driver Version : 352.99
  
    Device ID #2
      Type           : GPU
      Vendor ID      : 32
      Vendor         : NVIDIA Corporation
      Name           : GRID K520
      Version        : OpenCL 1.2 CUDA
      Processor(s)   : 8
      Clock          : 797
      Memory         : 1023/4095 MB allocatable
      OpenCL Version : OpenCL C 1.2
      Driver Version : 352.99
  
    Device ID #3
      Type           : GPU
      Vendor ID      : 32
      Vendor         : NVIDIA Corporation
      Name           : GRID K520
      Version        : OpenCL 1.2 CUDA
      Processor(s)   : 8
      Clock          : 797
      Memory         : 1023/4095 MB allocatable
      OpenCL Version : OpenCL C 1.2
      Driver Version : 352.99
  
    Device ID #4
      Type           : GPU
      Vendor ID      : 32
      Vendor         : NVIDIA Corporation
      Name           : GRID K520
      Version        : OpenCL 1.2 CUDA
      Processor(s)   : 8
      Clock          : 797
      Memory         : 1023/4095 MB allocatable
      OpenCL Version : OpenCL C 1.2
      Driver Version : 352.99
      
  
  
  When starting a run, the following warning appears:
  
  * Device #1: Outdated or broken NVIDIA driver detected!
  
  
  
  You are STRONGLY encouraged to use the official supported NVIDIA driver
  
  See hashcat's homepage for official supported NVIDIA drivers
  
  Also see: https://hashcat.net/wiki/doku.php?id=frequently_asked_questions#i_may_have_the_wrong_driver_installed_what_should_i_do
  
  You can use --force to override this but do not post error reports if you do so

• Uninstall current drivers:

  sudo yum erase nvidia cuda -y
  sudo reboot

• Download legacy driver: http://us.download.nvidia.com/XFree86/Linux-x86_64/367.57/NVIDIA-Linux-x86_64-367.57.run
• Install needed packages and the driver:

  sudo yum groupinstall -y "Development tools"
  sudo yum install kernel-devel-`uname -r`
  
  sudo /bin/bash NVIDIA-Linux-x86_64-367.57.run
  
    - Accept the license
    - Yes to registering the kernel module sources with DKMS
    - Yes to installing the 32-bit compatibility libraries
    - OK to the disclaimer about the libvdpau and libvdpau_trace libraries
    - Yes to running the nvidia-config utility
    - OK to acknowledge the X config file was updated
  
  sudo reboot

• Check correct driver version:

  nvidia-smi -q | head -10
  
  ==============NVSMI LOG==============
  
  Timestamp                           : Tue Nov 22 20:55:03 2016
  Driver Version                      : 367.57
  
  Attached GPUs                       : 4
  GPU 0000:00:03.0
      Product Name                    : GRID K520
      Product Brand                   : Grid

• md5crypt:

  $ sudo /usr/local/bin/hashcat -b -m 500
  hashcat (v3.10-809-g7fe575e+) starting in benchmark mode...
  
  * Device #1: Old CUDA chipset 3.0 detected, OpenCL performance is reduced.
               For ideal hashcat performance on NVIDIA GPU you need Shader Model 5.0 or higher
  * Device #2: Old CUDA chipset 3.0 detected, OpenCL performance is reduced.
               For ideal hashcat performance on NVIDIA GPU you need Shader Model 5.0 or higher
  * Device #3: Old CUDA chipset 3.0 detected, OpenCL performance is reduced.
               For ideal hashcat performance on NVIDIA GPU you need Shader Model 5.0 or higher
  * Device #4: Old CUDA chipset 3.0 detected, OpenCL performance is reduced.
               For ideal hashcat performance on NVIDIA GPU you need Shader Model 5.0 or higher
  nvmlDeviceGetFanSpeed(): Not Supported
  
  nvmlDeviceGetFanSpeed(): Not Supported
  
  nvmlDeviceGetFanSpeed(): Not Supported
  
  nvmlDeviceGetFanSpeed(): Not Supported
  
  OpenCL Platform #1: NVIDIA Corporation
  ======================================
  * Device #1: GRID K520, 1009/4036 MB allocatable, 8MCU
  * Device #2: GRID K520, 1009/4036 MB allocatable, 8MCU
  * Device #3: GRID K520, 1009/4036 MB allocatable, 8MCU
  * Device #4: GRID K520, 1009/4036 MB allocatable, 8MCU
  
  Hashtype: md5crypt, MD5(Unix), FreeBSD MD5, Cisco-IOS MD5
  
  Speed.Dev.#1.....:   927.5 kH/s (69.69ms)
  Speed.Dev.#2.....:   927.3 kH/s (69.69ms)
  Speed.Dev.#3.....:   927.0 kH/s (69.74ms)
  Speed.Dev.#4.....:   927.0 kH/s (69.73ms)
  Speed.Dev.#*.....:  3708.9 kH/s
  
  Started: Tue Nov 22 21:17:17 2016
  Stopped: Tue Nov 22 21:17:26 2016

• Download latest OpenCL runtime: https://software.intel.com/en-us/articles/opencl-drivers#latest_CPU_runtime:

  sudo yum install redhat-lsb-core
  
  
  wget http://registrationcenter-download.intel.com/akdlm/irc_nas/9019/opencl_runtime_16.1.1_x64_rh_6.4.0.25.tgz
  tar zxf opencl_runtime_16.1.1_x64_rh_6.4.0.25.tgz
  cd opencl_runtime_16.1.1_x64_rh_6.4.0.25
  sudo ./install.sh
  
    - skip unsupported OS
    - accept the license

• Test:

  hashcat -I
  hashcat (v3.10-809-g7fe575e+) starting...
  
  OpenCL Info:
  
  Platform ID #1
    Vendor  : Intel(R) Corporation
    Name    : Intel(R) OpenCL
    Version : OpenCL 1.2 LINUX
  
    Device ID #1
      Type           : CPU
      Vendor ID      : 8
      Vendor         : Intel(R) Corporation
      Name           :        Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
      Version        : OpenCL 1.2 (Build 25)
      Processor(s)   : 32
      Clock          : 2600
      Memory         : 2047/60387 MB allocatable
      OpenCL Version : OpenCL C 1.2
      Driver Version : 1.2.0.25
  
  Platform ID #2
    Vendor  : NVIDIA Corporation
    Name    : NVIDIA CUDA
    Version : OpenCL 1.2 CUDA 8.0.46
  
    Device ID #2
      Type           : GPU
      Vendor ID      : 32
      Vendor         : NVIDIA Corporation
      Name           : GRID K520
      Version        : OpenCL 1.2 CUDA
      Processor(s)   : 8
      Clock          : 797
      Memory         : 1009/4036 MB allocatable
      OpenCL Version : OpenCL C 1.2
      Driver Version : 367.57
  
    Device ID #3
      Type           : GPU
      Vendor ID      : 32
      Vendor         : NVIDIA Corporation
      Name           : GRID K520
      Version        : OpenCL 1.2 CUDA
      Processor(s)   : 8
      Clock          : 797
      Memory         : 1009/4036 MB allocatable
      OpenCL Version : OpenCL C 1.2
      Driver Version : 367.57
  
    Device ID #4
      Type           : GPU
      Vendor ID      : 32
      Vendor         : NVIDIA Corporation
      Name           : GRID K520
      Version        : OpenCL 1.2 CUDA
      Processor(s)   : 8
      Clock          : 797
      Memory         : 1009/4036 MB allocatable
      OpenCL Version : OpenCL C 1.2
      Driver Version : 367.57
  
    Device ID #5
      Type           : GPU
      Vendor ID      : 32
      Vendor         : NVIDIA Corporation
      Name           : GRID K520
      Version        : OpenCL 1.2 CUDA
      Processor(s)   : 8
      Clock          : 797
      Memory         : 1009/4036 MB allocatable
      OpenCL Version : OpenCL C 1.2
      Driver Version : 367.57

• Test:

  Session..........: hashcat
  Status...........: Running
  Hash.Type........: md5crypt, MD5(Unix), FreeBSD MD5, Cisco-IOS MD5
  Hash.Target......: 
  Time.Started.....: Tue Nov 22 21:51:59 2016 (39 secs)
  Time.Estimated...: Tue Nov 22 22:27:02 2016 (34 mins, 24 secs)
  Input.Mask.......:  
  Input.Queue......: 1/1 (100.00%)
  Speed.Dev.#1.....:   126.1 kH/s (64.21ms)
  Speed.Dev.#2.....:   922.8 kH/s (69.76ms)
  Speed.Dev.#3.....:   923.2 kH/s (69.77ms)
  Speed.Dev.#4.....:   922.8 kH/s (69.74ms)
  Speed.Dev.#5.....:   922.8 kH/s (69.79ms)
  Speed.Dev.#*.....:  3817.6 kH/s
  Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
  Recovered/Time...: CUR:N/A,N/A,N/A AVG:0.00,0.00,0.00 (Min,Hour,Day)
  Progress.........: 150454272/8031810176 (1.87%)
  Rejected.........: 0/150454272 (0.00%)
  Restore.Point....: 4399104/308915776 (1.42%)
  Candidates.#1....:  
  Candidates.#2....:  
  Candidates.#3....:  
  Candidates.#4....:  
  Candidates.#5....:  
  HWMon.Dev.#1.....: N/A
  HWMon.Dev.#2.....: Temp: 49c Util: 99% Core: 797Mhz Mem:2500Mhz Lanes:16
  HWMon.Dev.#3.....: Temp: 46c Util: 99% Core: 797Mhz Mem:2500Mhz Lanes:16
  HWMon.Dev.#4.....: Temp: 52c Util:100% Core: 797Mhz Mem:2500Mhz Lanes:16
  HWMon.Dev.#5.....: Temp: 44c Util:100% Core: 797Mhz Mem:2500Mhz Lanes:16
  
  
  htop:
  
    1  [|||||||||||||||||98.0%]    9  [|||||||||||||||||98.0%]     17 [|||||||||||||||||97.4%]    25 [|||||||||||||||||98.1%]
    2  [|||||||||||||||||98.7%]    10 [|||||||||||||||||98.1%]     18 [|||||||||||||||||98.0%]    26 [|||||||||||||||||97.5%]
    3  [|||||||||||||||||98.0%]    11 [|||||||||||||||||98.1%]     19 [|||||||||||||||||98.0%]    27 [|||||||||||||||||98.2%]
    4  [|||||||||||||||||97.4%]    12 [|||||||||||||||||97.5%]     20 [|||||||||||||||||98.7%]    28 [|||||||||||||||||97.4%]
    5  [|||||||||||||||||98.0%]    13 [|||||||||||||||||98.1%]     21 [|||||||||||||||||98.0%]    29 [|||||||||||||||||98.1%]
    6  [|||||||||||||||||98.0%]    14 [|||||||||||||||||98.1%]     22 [|||||||||||||||||98.0%]    30 [|||||||||||||||||98.1%]
    7  [|||||||||||||||||98.0%]    15 [|||||||||||||||||98.1%]     23 [|||||||||||||||||98.0%]    31 [|||||||||||||||||98.1%]
    8  [|||||||||||||||||98.0%]    16 [|||||||||||||||||97.4%]     24 [|||||||||||||||||98.0%]    32 [|||||||||||||||||97.5%]
    Mem[||||                                     1803/60387MB]     Tasks: 34, 71 thr; 34 running
    Swp[                                                0/0MB]     Load average: 27.78 11.48 4.39
                                                                   Uptime: 00:59:20
  
  $ uptime
   21:55:29 up  1:00,  2 users,  load average: 28.19, 15.47, 6.41