I decided recently to write a technical report to help administrators understand how write scripts for automation of tasks on a NetApp AltaVault appliance. The scripts contain sequences that execute AltaVault CLI command. The scripts are executed from a server, not from the AltaVault appliance. For example, a centralized Linux server is used to run commands on a remote AltaVault appliance. The commands for the AltaVault CLI are sent from the Linux server using secure shell (SSH). The technical report provides examples in BASH and Python.

For an additional level of protection, key based authentication is implemented. Key based authentication with SSH is preferred to prevent scripts from containing login credentials including username and passwords in the script files. The scripts execute as a local Linux user. The Linux user’s public key for SSH is stored on the AltaVault appliance to provide the authentication mechanism. So it was important that the library support the use of keys for client connections.

One challenge is the CLI for AltaVault is not a Linux type shell. It is a network device CLI interface similar to the CLI on a Cisco switch. It is not a big challenge, it just means that it is not possible to simply pass commands to execute as you would from a Linux server to another Linux server. Sending commands and receiving output are a little more complicated and require some trial and error to get things working well.

I researched several libraries before I found Paramiko. Paramiko is a nicely done Python library that provides functionality of both client and server SSH.

Initial Test Using Python Interactive Mode

This first test is just to load the Paramiko library and see how to execute a command using ssh. This is a quick way to check to see if this library will work like I expect it to after reading the documentation for it.

The initial test is simplified and sends both the user name and password. After working out some details on using keys, the final script will rely on key based authentication.

After a few attempts, I worked out what was needed to have the library use the existing known_hosts file so that it would complete a connection. The known_hosts file is typically populated interactively the first time you connect to a new host. The purpose is to warn you if something changes with the host you are connecting to in case its not the actual host you want to communicate with.

[root@svr1 ~]# python
Python 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
>>> username = 'admin'
>>> rem_con_pre = paramiko.SSHClient()
>>> rem_con_pre.load_system_host_keys()
>>> rem_con.connect('ava40mb1',username=username)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'rem_con' is not defined
>>> rem_con_pre.connect('ava40mb1',username=username)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 348, in connect
    server_key)
  File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 635, in missing_host_key
    raise SSHException('Server %r not found in known_hosts' % hostname)
paramiko.ssh_exception.SSHException: Server 'ava40mb1' not found in known_hosts
>>>

OK, now I figured out how to give the information for known_hosts. But there is a problem. The command output hangs.

>>> rem_con_pre.connect('ava40mb1',username=username,key_filename='/root/.ssh/id_rsa')
>>> rem_con = rem_con_pre.invoke_shell()
>>> output = rem_con.recv(1000)
>>> print output
Last login: Mon Mar 21 18:14:47 2016 from 10.1.2.172
ava40mb1 >
>>> rem_con.send("show info")
9
>>> output = rem_con.recv(1000)
>>> print output
show info
>>> output = rem_con.recv(1000)
^C

^C
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 596, in recv
    out = self.in_buffer.read(nbytes, self.timeout)
  File "/usr/lib/python2.7/site-packages/paramiko/buffered_pipe.py", line 147, in read
    self._cv.wait(timeout)
  File "/usr/lib64/python2.7/threading.py", line 339, in wait
    waiter.acquire()
KeyboardInterrupt
>>>
>>> rem_con.send("\n")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 698, in send
    return self._send(s, m)
  File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 1058, in _send
    raise socket.error('Socket is closed')
socket.error: Socket is closed

This is pretty simple, the command needs to terminate with a new line. When you are on a console you end the command with a return key. This is easily accomplished by sending a new line escape code \n.

>>> rem_con_pre.connect('ava40mb1',username=username,key_filename='/root/.ssh/id_rsa')
>>> rem_con = rem_con_pre.invoke_shell()
>>> rem_con.send("show info\n")
10
>>> output = rem_con.recv(1000)
>>> print output
Last login: Mon Mar 21 18:19:07 2016 from 10.1.2.179
ava40mb1 > show info
Current User:      admin

Status:            Healthy
Config:            initial
Appliance Up Time: 13d 20h 13m 28s
Service Up Time:   13d 19h 48m 9s
Number of CPUs:    4
CPU load averages: 0.06 / 0.04 / 0.00
Temperature (C):   0

Serial:            1234567890
Model:             AVA-v2
Revision:          A
Version:           4.0.1
ava40mb1 >
>>> rem_con.close()
>>>

OK, so now that its working, lets view the commands in a clean session.

Initial Test Cleaned Up

# python

import paramiko
username = 'admin'
rem_con_pre = paramiko.SSHClient()
rem_con_pre.load_system_host_keys()
rem_con_pre.load_host_keys('/root/.ssh/known_hosts')
rem_con_pre.set_missing_host_key_policy(paramiko.AutoAddPolicy())
rem_con_pre.connect('ava40mb1',username=username,key_filename='/root/.ssh/id_rsa')
rem_con = rem_con_pre.invoke_shell()
output = rem_con.recv(1000)
print output
Last login: Mon Mar 21 18:14:47 2016 from 10.1.2.172
ava40mb1 >

rem_con.send("show info\n")
10
output = rem_con.recv(1000)
print output
Last login: Mon Mar 21 18:19:07 2016 from 10.1.2.179
ava40mb1 > show info
Current User:      admin

Status:            Healthy
Config:            initial
Appliance Up Time: 13d 20h 13m 28s
Service Up Time:   13d 19h 48m 9s
Number of CPUs:    4
CPU load averages: 0.06 / 0.04 / 0.00
Temperature (C):   0

Serial:            1234567890
Model:             AVA-v2
Revision:          A
Version:           4.0.1
ava40mb1 >

rem_con.close()

I also checked the known_hosts file a few times to see if it would add new hosts. It does not. If you want to be able to add hosts, you have to do that programmatically in your Python script.

[root@svr1 ~]# wc -l ./.ssh/known_hosts
16 ./.ssh/known_hosts

Additional Test with more Complex Commands

Now it is time to test with more complex commands. The CLI has modes, the same way a Cisco Catalyst switch does.

  • User mode
  • Enable mode
  • Configure mode

The default mode is user mode. Switching to a new mode is done by entering the name of the mode to get a higher level prompt.

This example changes from user mode to enable mode, then executes a command. The output is collected, then another command is issued and its output is collected.

rem_con.send("en\nshow cifs shares\n")

output = rem_con.recv(1000)
print output

rem_con.send("show int prim brief\n")
output = rem_con.recv(1000)
print output

[root@svr1 ~]# python
Python 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
>>> username = 'admin'
>>> rem_con_pre = paramiko.SSHClient()
>>> rem_con_pre.load_host_keys('/root/.ssh/known_hosts')
>>> rem_con_pre.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> rem_con_pre.connect('ava40mb1',username=username,key_filename='/root/.ssh/id_rsa')
>>> rem_con = rem_con_pre.invoke_shell()
>>> output = rem_con.recv(1000)
>>> print output
Last login: Mon Mar 21 19:03:29 2016 from 10.1.2.179

>>> rem_con.send("en\nshow cifs shares\n")
20
>>> output = rem_con.recv(1000)
>>> print output
ava40mb1 > en
ava40mb1 # show cifs shares
Share: My Dir
     Path:               /mydir1
     Comment:
     Read only:          no
     Pinned:             no
     No Dedup:           no
     No Compression:     no
     Early Eviction:     no

Share: share1
     Path:               /share1
     Comment:
     Read only:          no
     Pinned:             no
     No Dedup:           no

>>>  Early Eviction:     no
>>>
>>> print output
>>> e is no -o option  (press RETURN)
>>> rem_con.send("\n")re
1    Read only:          no
>>> output = rem_con.recv(1000)
>>> print output
...skipping...
NetApp ssh: ssh remote command is not allowed.
~>> rem_con.send("show int prim brief\n")
~0
~>> output = rem_con.recv(1000)
~>> print output
~here is no -o option  (press RETURN)
~
~
~
ava40mb1 # ow int prim brief
% Unrecognized command "ow".
Type "?" for help.
ava40mb1 #
ava40mb1 #
>>> rem_con.send("show int prim brief\n")
20
>>> output = rem_con.recv(1000)
>>> print output
show int prim brief
% Unknown interface prim
ava40mb1 #
>>> rem_con.send("show int primary brief\n")
23
>>> output = rem_con.recv(1000)
>>> print output
show int primary brief
Interface primary state
   Up:                 yes
   Interface type:     ethernet
   IP address:         10.1.2.192
   Netmask:            255.255.254.0
   IPv6 link-local address: fe80::250:56ff:fe9c:f460/64
   Speed:              10000Mb/s
   Duplex:             full
   MTU:                1500
   HW address:         00:50:56:9C:F4:60
   Link:               yes
   Counters cleared date:  2016/03/07 21:50:16

ava40mb1 #
>>> rem_con.close()
>>> quit()
[root@svr1 ~]#

This test brought up a few things, but the most important is that paging needs to be handled. In the normal CLI, if a full page of text output is printed to the terminal, it prompts to continue printing the next terminal window full. Since we are not interacting with the terminal session, paging needs to be disabled.

It also didn’t like the abbreviated command show int prim brief so I corrected that, too.

Next Test - Disable Terminal Paging

Using a regular interactive ssh session, look at the command like help to see what option disables paging. There are two actually. Setting terminal length persists across sessions. I decided to use another method in the technical report that is kept just for a session so that it doesn’t affect other scripts or administrative users.

ava40mb1 (config) # terminal length 0

ava40mb1 # show term
CLI current session settings
  Terminal width:     98 columns
  Terminal length:    39 rows
  Terminal type:      xterm
ava40mb1 #

ava40mb1 > terminal ?
length           Set the number of lines for this terminal
type             Set the terminal type
width            Set the width of this terminal in characters
ava40mb1 > show terminal
CLI current session settings
  Terminal width:     98 columns
  Terminal length:    39 rows
  Terminal type:      xterm
ava40mb1 > terminal length 0
ava40mb1 > exit
Connection to ava40mb1 closed.
[root@svr1 ~]#

From the above, its possible to use terminal length 0. But that is a persistent change. I opted instead to use:

enable
no cli session paging enable

This disabled paging for a single session and is a much better way to do this.

Making the Python Script

This script is based on the example scripts from the Paramiko documentation.

I also learned that a delay is needed between commands. Some commands take a while to process and if the delay isn’t used the output will be missed.

import paramiko
import traceback
import time

username = 'admin'

try:
    client = paramiko.SSHClient()
    client.load_system_host_keys()
    client.load_host_keys('/root/.ssh/known_hosts')
#    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    client.set_missing_host_key_policy(paramiko.WarningPolicy())

    client.connect('ava40mb1',username=username,key_filename='/root/.ssh/id_rsa')

    channel = client.invoke_shell()
    output = channel.recv(1000)
    print output
    time.sleep(1)

    channel.send("show info\n")
    time.sleep(1)
    output = channel.recv(1000)
    print output

    channel.close()
    client.close()

except Exception as e:
    print('** Exception: %s: %s' % (e,__class__, e))
    traceback.print_exec()
    try:
        client.close()
    except:
        pass
    sys.exit(1)

testing with exec, doesn’t work

[root@svr1 ~]# python
Python 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
username = 'admin'
rem_con_pre = paramiko.SSHClient()
rem_con_pre.load_system_host_keys()
rem_con_pre.load_host_keys('/root/.ssh/known_hosts')
rem_con_pre.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> username = 'admin'
>>> rem_con_pre = paramiko.SSHClient()
>>> rem_con_pre.load_system_host_keys()
>>> rem_con_pre.load_host_keys('/root/.ssh/known_hosts')
>>> rem_con_pre.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> rem_con_pre.connect('ava40mb1',username=username,key_filename='/root/.ssh/id_rsa')
>>> stdin, stdout, stderr = rem_con_pre.exec_command('show info')

>>> stdout=stdout.readlines()
>>> print stdout
[u'NetApp ssh: ssh remote command is not allowed.\n']

Output from the working script.

[root@svr1 ~]# python ./ava_example.py
Last login: Mon Mar 21 23:34:28 2016 from 10.1.2.179

show info
ava40mb1 > show info
Current User:      admin

Status:            Healthy
Config:            initial
Appliance Up Time: 14d 0h 44m 52s
Service Up Time:   14d 0h 19m 34s
Number of CPUs:    4
CPU load averages: 0.08 / 0.02 / 0.01
Temperature (C):   0

Serial:            1234567890
Model:             AVA-v2
Revision:          A
Version:           4.0.1
ava40mb1 >

This is the updated script.

[root@svr1 ~]# cat ava_example.py
import paramiko
import traceback
import time

username = 'admin'

try:
    client = paramiko.SSHClient()
    client.load_system_host_keys()
    client.load_host_keys('/root/.ssh/known_hosts')
#    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    client.set_missing_host_key_policy(paramiko.WarningPolicy())

    client.connect('ava40mb1',username=username,key_filename='/root/.ssh/id_rsa')

    channel = client.invoke_shell()
    output = channel.recv(1000)
    print output
    time.sleep(1)

    channel.send("show info\n")
    time.sleep(1)
    output = channel.recv(1000)
    print output

    channel.close()
    client.close()

except Exception as e:
    print('** Exception: %s: %s' % (e,__class__, e))
    traceback.print_exec()
    try:
        client.close()
    except:
        pass
    sys.exit(1)

[root@svr1 ~]#

Summary

The final technical report was published on the NetApp field portal. In the end, the obvious conclusion was that BASH scripts were OK for some small tasks, but to really automate a complex workflow it was much better to use Python. There are so many things you can do with a programming language like Python that just are not possible with BASH scripts.

References

Paramiko
OpenSSH
NetApp AltaVault Appliance Documentation
NetApp AltaVault Command-Line Reference Guide 4.2.2