Hello all,
I've been googling around for some answers and I thought I would ask

the list as well.
In the past I have used different compters for different tasks.  I

would have many different installs of OpenBSD on many different

platforms.
However, i am moving some stuff into a data center and am getting a

blade server with 10 blades (up to 20 total).  I have been playing

with this and it is running great, but as each blade has exactly the

same specs (same drive, ram, processor etc) I was wondering about

improving my skills on handling a lot of identical computers on the

administration side.
There are basically three types of blade, database, app server and

front end world facing (proxy, mail, dns).
I want to automate handling them as much as possible and would like

some list suggestions on reading materials, software, or web howtos.
Examples of what I am after:
1) Create images or post install diffs so that if I need to add a

blade to expand, I put it in, connect via the console, install via PXE

and then download the diff - I know you can do this wth the post

install scrips in OpenBSD's install script, but any real world use of

this, things to avoid or good things to do?
2) Keeping 10 - 20 copies of OpenBSD up to the latest patch levels

without having to do more than trial on one (for each type) and then

for the rest type something as trivial as "/bin/sh -x update.sh"

rotating through the servers and testing as you go?  I can see myself

spending two days a month otherwise doing upgrades on all the servers.
3) Guides on how to manage the logs of this many servers.  Any

experiences with splunk on this sort of environment, other options?
4) Anything else I should think about / avoid?
I know this is a bit of a broad ranging question, but I am looking for

general gaps in my sys admin knowledge at the moment so I apologise

for any vagueness.
Regards
Mikel

Hi,
The multixterm program that comes with expect is useful for ssh'ing to

lots of machines and running the same commands on them all.
Unfortunately vi has stopped working for me in multixterm. Might be

something to do with the value of $TERM.
--

Best Regards
Edd
---------------------------------------------------

http://students.dec.bournemouth.ac.uk/ebarrett

multixterm is a bit strange, the started xterms inherit the settings

from the xterm multixterm was started from, but the new xterms have

default settings. So if you start it from a resized xterm, the new

xterm will probably have the have wrong settings.
	-Otto

Just my idea (never had more than 3 boxes at once):
On my main box, I'd have a separate copy of /etc and other files for

each class of box, where a class is a group of boxes with the same

configs.  Perhaps each of your boxes are different so the class concept

is irrelevant.  To change a box's config, I'd change these files in the

normal way, then use rsync (via ssh) to update the altered boxes.
To run a command-line command on a group of boxes, I'd likely write a

script that took the command line and a class of machines on which to

run it.  The script would send back any error messages recevied and from

which box it was received.  This script would be useful for making

changes other than to individual files, e.g. deleting files (more

direct than relying on rsync --delete), or chmod/chown.
Unless all the boxes are using the same OS, I'd probably write a script

for handling adding and removing users and groups (since each OS does it

slightly differently).
Then again, this is probably reinventing the wheel.
Doug.

http://tentakel.biskalar.de/

http://www.garbled.net/clusterit.html

http://fermitools.fnal.gov/abstracts/rgang/abstract.html

http://www.netfort.gr.jp/~dancer/software/dsh.html.en

http://sourceforge.net/projects/dcmd/
http://www.cfengine.org/

http://puppet.reductivelabs.com/

http://www.lcfg.org/

http://quattor.web.cern.ch/quattor/

http://www.hpl.hp.com/research/smartfrog/
...probably several more.
DS

-----BEGIN PGP SIGNED MESSAGE-----

Hash: SHA1
Too much work. Use cfengine. It is on ports.
Regards,
        Alvaro

Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHTNbxh0VmNM2kfikRAsugAJ9hsbhF+aanFn2bJ6sKop/oHr8X2wCePR1z

NtJq74d4/a0cP7IvwhR5nbA=

=aDn5

-----END PGP SIGNATURE-----

See also sysutils/clusterit, which has several tools useful for this

purpose. I use dsh to run oneliners on groups of machines, though

you can use it interactively, too.
-- 
o--------------------------{ Will Maier }--------------------------o

| web:.......http://www.lfod.us/ | email.........willmaier@ml1.net |

*------------------[ BSD Unix: Live Free or Die ]------------------*

cfengine[0] (which we use at work to manage ~500 Linux machines) or

radmind (which I use at home to manage my OpenBSD servers,

workstations and laptops).
There are at least two schools of thought on how one should manage >

1 machine. cfengine is the most popular convergent tool, where you

specify an ideal state using a declarative language and the clients

iterate towards that state. radmind is the most useful congruent

tool, where you specify (or directly imply) the exact sequence of

operations that each client must perform to reach an ideal state.
I've used both approaches for years, and I greatly prefer radmind

both for its simple design and the implicit guarantees of

congruence. See below for references on each:
    http://www.cfengine.org/papers.html
Log centrally using syslog; syslog-ng is a well-used central log

server.
[0] http://www.cfengine.org/

[1] http://rsug.itd.umich.edu/software/radmind/
-- 
o--------------------------{ Will Maier }--------------------------o

| web:.......http://www.lfod.us/ | email.........willmaier@ml1.net |

*------------------[ BSD Unix: Live Free or Die ]------------------*