Cluster management

Goal of this article is to give you a couple of tools to remotely manage multiple systems without having to do all the sneaker support.

Managing Multiple clusters with ClusterSSH also known as CSSH.

(Parts of this article adapted from: https://www.linux.com/learn/tutorials/413853:managing-multiple-linux-servers-with-clusterssh)

If you're a Linux system administrator, chances are you've got more than one machine that you're responsible for on a daily basis. You may even have a bank of machines that you maintain that are similar — a farm of Web servers, for example. If you have a need to type the same command into several machines at once, you can login to each one with SSH and do it serially, or you can save yourself a lot of time and effort and use a tool like ClusterSSH.

ClusterSSH is a Tk/Perl wrapper around standard Linux tools like XTerm and SSH. As such, it'll run on just about any POSIX-compliant OS where the libraries exist — I've run it on Linux, Solaris, and Mac OS X. It requires the Perl libraries Tk (perl-tk on Debian or Ubuntu) and X11::Protocol (libx11-protocol-perl on Debian or Ubuntu), in addition to xterm and OpenSSH.

Installation


Installing ClusterSSH on a Debian or Ubuntu system is trivial — a simple
sudo apt-get install clusterssh will install it and its dependencies. It is also packaged for use with Fedora, and it is installable via the ports system on FreeBSD. There's also a MacPorts version for use with Mac OS X, if you use an Apple machine. Of course, it can also be compiled from source.
Configuration

$ sudo apt-get install clusterssh


ClusterSSH can be configured either via its global configuration file — /etc/clusters, or via a file in the user's home directory called .csshrc. I tend to favor the user-level configuration as that lets multiple people on the same system to setup their ClusterSSH client as they choose. Configuration is straightforward in either case, as the file format is the same. ClusterSSH defines a "cluster" as a group of machines that you'd like to control via one interface. With that in mind, you enumerate your clusters at the top of the file in a "clusters" block, and then you describe each cluster in a separate section below.

For example, Substitute "test" for "srvr" where necessary. let's say I've got two clusters, each consisting of two machines. "Cluster1" has the machines "srvr1" and "srvr2" in it, and "Cluster2" has the machines "srvr3" and "srvr4" in it. The ~.csshrc (or /etc/clusters) control file would look like this:

clusters = cluster1 cluster2

cluster1 = srvr1 srvr2
cluster2 = srvr3 srvr4


You can also make meta-clusters — clusters that refer to clusters. If you wanted to make a cluster called "all" that encompassed all the machines, you could define it two ways. First, you could simply create a cluster that held all the machines, like the following:

clusters = cluster1 cluster2 all

cluster1 = srvr1 srvr2
cluster2 = srvr3 srvr4
all = srvr1 srvr2 srvr3 srvr4


However, my preferred method is to use a meta-cluster that encompasses the other clusters:

clusters = cluster1 cluster2 all

cluster1 = srvr1 srvr2
cluster2 = srvr3 srvr4
all = cluster1 cluster2





By calling out the "all" cluster as containing cluster1 and cluster2, if either of those clusters ever change, the change is automatically captured so you don't have to update the "all" definition. This will save you time and headache if your .csshrc file ever grows in size.
Using ClusterSSH

Using ClusterSSH is similar to launching SSH by itself. Simply running cssh -l <username> <clustername> will launch ClusterSSH and log you in as the desired user on that cluster. In the figure below, you can see I've logged into "cluster1" as myself. The small window labeled "CSSH [2]" is the Cluster SSH console window. Anything I type into that small window gets echoed to all the machines in the cluster — in this case, machines "srvr1" and "srvr2". In a pinch, you can also login to machines that aren't in your .csshrc file, simply by running cssh -l <username> <machinename1> <machinename2> <machinename3>.

If I want to send something to one of the terminals, I can simply switch focus by clicking in the desired XTerm, and just type in that window like I usually would. ClusterSSH has a few menu items that really help when dealing with a mix of machines. As per the figure below, in the "Hosts" menu of the ClusterSSH console there's are several options that come in handy.

"Retile Windows" does just that if you've manually resized or moved something. "Add host(s) or Cluster(s)" is great if you want to add another set of machines or another cluster to the running ClusterSSH session. Finally, you'll see each host listed at the bottom of the "Hosts" menu. By checking or unchecking the boxes next to each hostname, you can select which hosts the ClusterSSH console will echo commands to. This is handy if you want to exclude a host or two for a one-off or particular reason. The final menu option that's nice to have is under the "Send" menu, called "Hostname". This simply echoes each machine's hostname to the command line, which can be handy if you're constructing something host-specific across your cluster.




Caveats with ClusterSSH

Like many UNIX tools, ClusterSSH has the potential to go horribly awry if you aren't very careful with its use. I've seen ClusterSSH mistakes take out an entire tier of Web servers simply by propagating a typo in an Apache configuration. Having access to multiple machines at once, possibly as a privileged user, means mistakes come at a great cost. Take care, and double-check what you're doing before you punch that Enter key.
Conclusion

ClusterSSH isn't a replacement for having a configuration management system or any of the other best practices when managing a number of machines. However, if you need to do something in a pinch outside of your usual toolset or process, or if you're doing prototype work, ClusterSSH is indispensable. It can save a lot of time when doing tasks that need to be done on more than one machine, but like any power tool, it can cause a lot of damage if used


CSSH

 CSSH(1p)              User Contributed Perl Documentation             CSSH(1p)



NAME
       cssh, crsh, ctel, ccon - Cluster administration tool

SYNOPSIS
       cssh [options] [[user@]<server>[:port]|<tag>] [...]
       crsh [options] [[user@]<server>[:port]|<tag>] [...]
       ctel [options] [<server>[:port]|<tag>] [...]
       ccon [options] [[user@]<server>[:port]|<tag>] [...]

DESCRIPTION
       The command opens an administration console and an xterm to all
       specified hosts.  Any text typed into the administration console is
       replicated to all windows.  All windows may also be typed into
       directly.

       This tool is intended for (but not limited to) cluster administration
       where the same configuration or commands must be run on each node
       within the cluster.  Performing these commands all at once via this
       tool ensures all nodes are kept in sync.

       Connections are opened via ssh so a correctly installed and configured
       ssh installation is required.  If, however, the program is called by
       "crsh" then the rsh protocol is used (and the communications channel is
       insecure), or by "ctel" then telnet is used, or by "ccon" then console
       is used.

       Extra caution should be taken when editing system files such as
       /etc/inet/hosts as lines may not necessarily be in the same order.
       Assuming line 5 is the same across all servers and modifying that is
       dangerous.  Better to search for the specific line to be changed and
       double-check before changes are committed.

   Further Notes
       Please also see "KNOWN BUGS".

       ·   The dotted line on any sub-menu is a tear-off, i.e. click on it and
           the sub-menu is turned into its own window.

       ·   Unchecking a hostname on the Hosts sub-menu will unplug the host
           from the cluster control window, so any text typed into the console
           is not sent to that host.  Re-selecting it will plug it back in.

       ·   If your window manager menu bars are obscured by terminal windows
           see the "screen_reserve_XXXXX" options in the .clusterssh/config
           file (see "FILES").

       ·   If the terminals overlap too much see the "terminal_reserve_XXXXX"
           options in the .clusterssh/config file (see "FILES").

       ·   If the code is called as crsh instead of cssh (i.e. a symlink
           called crsh points to the cssh file or the file is renamed) rsh is
           used as the communications protocol instead of ssh.

       ·   If the code is called as ctel instead of cssh (i.e. a symlink
           called ctel points to the cssh file or the file is renamed) telnet
           is used as the communications protocol instead of ssh.

       ·   If the code is called as ccon instead of cssh (i.e. a symlink
           called ccon points to the cssh file or the file is renamed) console
           is used as the communications protocol instead of ssh.

       ·   When using cssh on a large number of systems to connect back to a
           single system (e.g. you issue a command to the cluster to scp a
           file from a given location) and when these connections require
           authentication (i.e. you are going to authenticate with a
           password), the sshd daemon at that location may refuse connects
           after the number specified by MaxStartups in sshd_config is
           exceeded.  (If this value is not set, it defaults to 10.)  This is
           expected behavior; sshd uses this mechanism to prevent DoS attacks
           from unauthenticated sources.  Please tune sshd_config and reload
           the SSH daemon, or consider using the ~/.ssh/authorized_keys
           mechanism for authentication if you encounter this problem.

       ·   If client windows fail to open, try running:

           "cssh -e {single host name}"

           This will test the mechanisms used to open windows to hosts.  This
           could be due to either the "-xrm" terminal option which enables
           "AllowSendEvents" (some terminal do not require this option, other
           terminals have another method for enabling it - see your terminal
           documention) or the "ConnectTimeout" ssh option (see the
           configuration option "-o" or file .clusterssh/config below to
           resolve this).

OPTIONS
       Some of these options may also be defined within the configuration
       file.  Default options are shown as appropriate.

       --action,-a '<command>'
           Run the command in each session, i.e. "-a 'vi /etc/hosts'" to drop
           straight into a vi session.  NOTE: not all communications methods
           support this (ssh and rsh should, telnet and console will not).

       --autoclose,-A <seconds>
           Number of seconds to wait before closing finished terminal windows.

       --autoquit,-q|--no-autoquit,-Q
           Enable|Disable automatically quiting after the last client window
           has closed (overriding the config file)

       --cluster-file,-c <file>
           Use supplied file as additional cluster file (see also "FILES")

       --config-file,-C <file>
           Use supplied file as additional configuration file (see also
           "FILES")

       -d  DEPRECATED.  See '--debug'.

       -D  DEPRECATED.  See '--debug'.

       --debug [number].
           Enable debugging.  Either a level can be provided or the option can
           be repeated multiple times.  Maximum level is 4.

       --evaluate,-e [user@]<hostname>[:port]
           Display and evaluate the terminal and connection arguments so
           display any potential errors.  The <hostname> is required to aid
           the evaluation.

       --font,-f "5x8"
           Specify the font to use in the terminal windows. Use standard X
           font notation.

       --help,-h|-?
           Show basic help text, and exit

       --list, -L
           List available cluster tags.

       --man,-H
           Show full help test (the man page), and exit

       --master,-M <master>
           The console client program polls master as the primary server,
           rather than the default set at compile time (typically
           ``console'').

       --options,-o "-x -o ConnectTimeout=10" - for ssh connections
       --options,-o ""                        - for rsh connections
           Specify arguments to be passed to ssh or rsh when making the
           connection.

           NOTE: any "generic" change to the method (i.e. specifying the ssh
           port to use) should be done in the medium's own config file (see
           "ssh_config" and $HOME/.ssh/config).

       --output-config,-u
           Output the current configuration in the same format used by the
           $HOME/.clusterssh/config file.

       --port,-p <port>
           Specify an alternate port for connections.

       --show-history,-s
           IN BETA: Show history within console window.  This code is still
           being worked upon, but may help some users.

       --term-args,-t ""
           Specify arguments to be passed to terminals being used

       --tile,-g|--no-tile,-G
           Enable|Disable window tiling (overriding the config file)

       --title,-T "CSSH"
           Specify the initial part of the title used in the console and
           client windows

       --use_all_a_records,-A
           If a hostname resolves to multiple IP addresses, toggle whether or
           not to connect to all of them, or just the first one (see also
           config file entry)

       --username,-l $LOGNAME
           Specify the default username to use for connections (if different
           from the currently logged in user).  NOTE: will be overridden by
           <user>@<host>

       --version,-v
           Show version information and exit

ARGUMENTS
       The following arguments are support:

       [user@]<hostname>[:port] ...
           Open an xterm to the given hostname and connect to the
           administration console.  An optional port number can be used if
           sshd is not listening on standard port (e.g not listening on port
           22) and ssh_config cannot be used.

       <tag> ...
           Open a series of xterms defined by <tag> within either
           /etc/clusters or $HOME/.clusterssh/clusters (see "FILES").

           Note: specifying a username on a cluster tag will override any
           usernames defined in the cluster

KEY SHORTCUTS
       The following key shortcuts are available within the console window,
       and all of them may be changed via the configuration files.

       Control-q
           Quit the program and close all connections and windows

       Control-+
           Open the 'Add Host(s) or Cluster(s)' dialogue box.  Mutiple host or
           cluster names can be entered, separated by spaces.

       Alt-n
           Paste in the hostname part of the specific connection string to
           each client, minus any username or port, i.e.

           "scp /etc/hosts server:files/<Alt-n>.hosts"

           would replace the <Alt-n> with the client's name in each window

       Alt-r
           Retile all the client windows

EXAMPLES
       Open up a session to 3 servers
           $ cssh server1 server2 server3

       Open up a session to a cluster of servers identified by the tag 'farm1'
       and give the controlling window a specific title, where the cluster is
       defined in one of the default configuration files
           $ cssh -T 'Web Farm Cluster 1' farm1

       Connect to different servers using different login names.  NOTE: this
       can also be achieved by setting up appropriate options in the
       .ssh/config file. Do not close cssh when last terminal exits.
           $ cssh -Q user1@server1 admin@server2

       Open up a cluster defined in a non-default configuration file
           $ cssh -c $HOME/cssh.config db_cluster

       Use telnet on port 2022 instead of ssh
           $ ctel -p 2022 server1 server2

       Use rsh instead of ssh
           $ crsh server1 server2

       Use console with master as the primary server instead of ssh
           $ ccon -M master server1 server2

FILES
       /etc/clusters
           This file contains a list of tags to server names mappings.  When
           any name is used on the command line it is checked to see if it is
           a tag.  If it is a tag, then the tag is replaced with the list of
           servers.  The formated is as follows:

           <tag> [user@]<server> [user@]<server> [...]

             i.e.

             # List of servers in live
             live admin1@server1 admin2@server2 server3 server4

           All comments (marked by a #) and blank lines are ignored.  Tags may
           be nested, but be aware of recursive tags which are not checked
           for.

           Clusters may also be specified either directly (see "clusters"
           configuration options) or indirectly (see "extra_cluster_file"
           configuration option) in the users $HOME/.clusterssh/clusters file.

           NOTE: there is a special cluster tag called "default" - any tags or
           hosts included within this tag will be automatically opened if no
           other tags are specified on the command line.

       /etc/csshrc & $HOME/.clusterssh/config
           This file contains configuration overrides - the defaults are as
           marked.  Default options are overwritten first by the global file,
           and then by the user file.

           NOTE: values for entries do not need to be quoted unless it is
           required for passing arguments, i.e.

             terminal_allow_send_events="-xrm '*.VT100.allowSendEvents:true'"

           should be written as

             terminal_allow_send_events=-xrm '*.VT100.allowSendEvents:true'

           always_tile = yes
               Setting to anything other than "yes" does not perform window
               tiling (see also -G).

           auto_close = 5
               Close terminal window after this many seconds.  If set to 0
               will instead wait on input from the user in each window before
               closing. Can be overridden by "-K" on the command line

           auto_quit = yes
               Automatically quit after the last client window closes.  Set to
               anything other than "yes" to disable.  Can be overridden by
               "-Q" on the command line.

           clusters = <blank>
               Define a number of cluster tags in addition to (or to replace)
               tags defined in the /etc/clusters file.  The format is:

                clusters = <tag1> <tag2> <tag3>
                <tag1> = host1 host2 host3
                <tag2> = user@host4 user@host5 host6
                <tag3> = <tag1> <tag2>

               As with the /etc/clusters file, be sure not to create
               recursivly nested tags.

           comms = ssh
               Sets the default communication method (initially taken from the
               name of program, but can be overridden here).

           console_position = <null>
               Set the initial position of the console - if empty then let the
               window manager decide.  Format is '+<x>+<y>', i.e. '+0+0' is
               top left hand corner of the screen, '+0-70' is bottom left hand
               side of screen (more or less).

           extra_cluster_file = <null>
               Define an extra cluster file in the format of /etc/clusters.
               Multiple files can be specified, seperated by commas.  Both ~
               and $HOME are acceptable as a to reference the users home
               directory, i.e.

                extra_cluster_file = ~/clusters, $HOME/clus

           ignore_host_errors
               THIS OPTION IS DEPRECATED.  It has been left in so current
               systems continue to function as expected.

           key_addhost = Control-Shift-plus
               Default key sequence to open AddHost menu.  See below notes on
               shortcuts.

           key_clientname = Alt-n
               Default key sequence to send cssh client names to client.  See
               below notes on shortcuts.

           key_paste = Control-v
               Default key sequence to paste text into the console window.
               See below notes on shortcuts.

           key_quit = Control-q
               Default key sequence to quit the program (will terminate all
               open windows).  See below notes on shortcuts.

           key_retilehosts = Alt-r
               Default key sequence to retile host windows.  See below notes
               on shortcuts.

           max_addhost_menu_cluster_items = 6
               Maximum number of entries in the 'Add Host' menu cluster list
               before scrollbars are used

           max_host_menu_items = 30
               Maximum number of hosts to put into the host menu before
               starting a new column

           menu_host_autotearoff = 0
           menu_send_autotearoff = 0
               When set to non-0 will automatically tear-off the host or send
               menu at program start

           mouse_paste = Button-2 (middle mouse button)
               Default key sequence to paste text into the console window
               using the mouse.  See below notes on shortcuts.

           rsh_args = <blank>
           ssh_args = "-x -o ConnectTimeout=10"
               Sets any arguments to be used with the communication method
               (defaults to ssh arguments).

               NOTE: The given defaults are based on OpenSSH, not commercial
               ssh software.

               NOTE: Any "generic" change to the method (i.e. specifying the
               ssh port to use) should be done in the medium's own config file
               (see "ssh_config" and $HOME/.ssh/config).

           screen_reserve_top = 0
           screen_reserve_bottom = 60
           screen_reserve_left = 0
           screen_reserve_right = 0
               Number of pixels from the screen side to reserve when
               calculating screen geometry for tiling.  Setting this to
               something like 50 will help keep cssh from positioning windows
               over your window manager's menu bar if it draws one at that
               side of the screen.

           rsh = /path/to/rsh
           ssh = /path/to/ssh
               Depending on the value of comms, set the path of the
               communication binary.

           terminal = /path/to/terminal
               Path to the x-windows terminal used for the client.

           terminal_args = <blank>
               Arguments to use when opening terminal windows.  Otherwise
               takes defaults from $HOME/.Xdefaults or $<$HOME/.Xresources>
               file.

           terminal_font = 6x13
               Font to use in the terminal windows.  Use standard X font
               notation.

           terminal_reserve_top = 5
           terminal_reserve_bottom = 0
           terminal_reserve_left = 5
           terminal_reserve_right = 0
               Number of pixels from the terminal side to reserve when
               calculating screen geometry for tiling.  Setting these will
               help keep cssh from positioning windows over your scroll and
               title bars or otherwise overlapping the windows too much.

           terminal_colorize = 1
               If set to 1 (the default), then "-bg" and "-fg" arguments will
               be added to the terminal invocation command-line.  The terminal
               will be colored in a pseudo-random way based on the host name;
               while the color of a terminal is not easily predicted, it will
               always be the same color for a given host name.  After a while,
               you will recognize hosts by their characteristic terminal
               color.

           terminal_bg_style = dark
               If set to dark, the the terminal background will be set to
               black and the foreground to the pseudo-random color.  If set to
               light, then the foreground will be black and the background the
               pseudo-random color.  If terminal_colorize is zero, then this
               option has no effect.

           terminal_size = 80x24
               Initial size of terminals to use (note: the number of lines
               (24) will be decreased when resizing terminals for tiling, not
               the number of characters (80))

           terminal_title_opt = -T
               Option used with "terminal" to set the title of the window

           terminal_allow_send_events = -xrm '*.VT100.allowSendEvents:true'
               Option required by the terminal to allow XSendEvents to be
               received

           title = cssh
               Title of windows to use for both the console and terminals.

           unmap_on_redraw = no
               Tell Tk to use the UnmapWindow request before redrawing
               terminal windows.  This defaults to "no" as it causes some
               problems with the FVWM window manager.  If you are experiencing
               problems with redraws, you can set it to "yes" to allow the
               window to be unmapped before it is repositioned.

           use_all_a_records = no
               If a hostname resolves to multiple IP addresses, set to "yes"
               to connect to all of them, not just the first one found.

           use_hotkeys = yes
               Setting to anything other than "yes" will disable all hotkeys.

           user = $LOGNAME
               Sets the default user for running commands on clients.

           window_tiling = yes
               Perform window tiling (set to "no" to disable)

           window_tiling_direction = right
               Direction to tile windows, where "right" means starting top
               left and moving right and then down, and anything else means
               starting bottom right and moving left and then up

           NOTE: The key shortcut modifiers must be in the form "Control",
           "Alt", or "Shift", i.e. with the first letter capitalised and the
           rest lower case.  Keys may also be disabled individually by setting
           to the word "null".

       $HOME/.csshrc_send_menu
           This (optional) file contains items to populate the send menu.  The
           default entry could be written as:

             <send_menu>
               <menu title="Hostname">
                   <command>%s</command>
                   <accelerator>ALT-n</accelerator>
               </menu>
             </send_menu>

           Submenus can also be specified as follows:

             <send_menu>
               <menu title="Default Entries">
                 <detach>yes</detach>
                 <menu title="Hostname">
                     <command>%s</command>
                     <accelerator>ALT-n</accelerator>
                 </menu>
               </menu>
             </send_menu>

           Caveats:

           There is currently no strict format checking of this file.
           The format of the file may change in the future
           If the file exists the default entry (Hostname) is not added

           The following replacement macros are available:

           %s  Hostname part of the specific connection string to each client,
               minus any username or port

           %u  Username part of the connection string to each client

           %h  Hostname of server where cssh is being run from

           %n  <RETURN> code

           NOTE: requires XML::Simple to be installed

KNOWN BUGS
       1.  Catering for IPv6 addresses is minimal.  This is due to a conflict
           between IPv6 addresses and port numbers within the same server
           definition since they both use the same seperator, i.e. is the
           following just an IPv6 address, or an address + port number of
           2323?

               2001:db8::1428:2323

           Exactly - I cannot tell either.  the IPv6 address without a port is
           assumed in those cases where it cannot be determined and a warning
           is issued.

           Possible work arounds include:

           a.  Use square brackets around the IPv6 address, i.e.
                   [2001:db8::1428]:2323 or
                   [2001:db8::1428:2323] as appropriate so there is no
               ambiguity

           b.  Use the full IPv6 address if also using a port number - the 8th
               colon is assumed to be the port seperator.

           c.  Define the IPv6 address in your /etc/hosts file, DNS or other
               name service lookup mechanism and use the hostname instead of
               the address.

       2.  Swapping virtual desktops can a redraw of all the terminal windows.
           This is due to a lack of distinction within Tk between switching
           desktops and minimising/maximising windows.  Until Tk can tell the
           difference between the two events, there is no fix (apart from
           rewriting everything directly in X)

       Anyone with any good ideas to fix the above bugs is more than welcome
       to get in touch and/or provide a patch.

REPORTING BUGS
       · If you have issues running cssh, first try:

         "cssh -e [user@]<hostname>[:port]"

         This performs two tests to confirm cssh is able to work properly with
         the settings provided within the .clusterssh/config file (or internal
         defaults).

                 1. test the terminal window works with the options provided

                 2. test ssh works to a host with the configured arguments

         Configuration options to watch for in ssh are

                 - Doesn't understand "-o ConnectTimeout=10" - remove the option
                   in the F<.clusterssh/config> file

                 - OpenSSH-3.8 using untrusted ssh tunnels - use "-Y" instead of "-X"
                   or use "ForwardX11Trusted yes' in ssh_config (if you change the
                   default ssh options from -x to -X)

       · If you require support, please run the following commands and post it
         on the web site in the support/problems forum:

         "perl -V"

         "perl -MTk -e 'print $Tk::VERSION,$/'"

         "perl -MX11::Protocol -e 'print $X11::Protocol::VERSION,$/'"

         "cat /etc/csshrc $HOME/.clusterssh/config"

       · Use the debug switches (-d, -D, or -dD) will turn on debugging
         output.  However, please only use this option with one host at a
         time, i.e. "cssh -d <host>" due to the amount of output produced (in
         both main and child windows).

SEE ALSO
       <http://clusterssh.sourceforge.net/>, "ssh", Tk::overview,
       X11::Protocol, "perl"

CREDITS
       A web site for comments, requests, bug reports and bug fixes/patches is
       available at <http://clusterssh.sourceforge.net/>

AUTHOR
       Duncan Ferguson, "<duncan_j_ferguson at yahoo.co.uk>"

LICENSE AND COPYRIGHT
       Copyright 1999-2010 Duncan Ferguson.

       This program is free software; you can redistribute it and/or modify it
       under the terms of either: the GNU General Public License as published
       by the Free Software Foundation; or the Artistic License.

       See http://dev.perl.org/licenses/ for more information.



perl v5.14.2                      2012-06-24                          CSSH(1p)

-------------------------------------------------
For Microsoft users, consider the powershell and Systems Internal utilities.