run_radmind script (updated)


Here is the run_radmind script I use. It's called on demand at logout, and as a periodic task nightly.

First the script itself:
run_radmind.pl


Now, some commentary (in blue):

#!/usr/bin/perl
################################################################################
# run_radmind
#
# This script runs radmind.
#
# Requires the radmind client tools (and a running server obviously).
# (See http://rsug.itd.umich.edu/software/radmind)
#
# Can also be used with iHook
# (See http://rsug.itd.umich.edu/software/ihook)
#

This script started out as a light edit of University of Utah's run_radmind script, and I've added my own tweaks and changes. Of particular coolness was UofU's execute_command subroutine, which allowed the output of the radmind tools to appear in iHook's drawer AND be captured to a file, which is invaluable for troubleshooting.

# Copyright (c) 2002 University of Utah Student Computing Labs.
# All Rights Reserved.
#
# Many, many modifications by Greg Neagle, Walt Disney Feature Animation
#
# Permission to use, copy, modify, and distribute this software and
# its documentation for any purpose and without fee is hereby granted,
# provided that the above copyright notice appears in all copies and
# that both that copyright notice and this permission notice appear
# in supporting documentation, and that the name of The University
# of Utah not be used in advertising or publicity pertaining to
# distribution of the software without specific, written prior
# permission. This software is supplied as is without expressed or
# implied warranties of any kind.
#
################################################################################
################################################################################
# change these to match your installation
$rserver = "-h radmind";
$cksum = "";
$fsdiffpath = "/";
$ktcheckoutput = "/usr/local/radmind/tmp/ktcheck_output.log";
$ktcheckoutput_backup_number = 5;
$fsdiffoutput = "/usr/local/radmind/tmp/fsdiff_output.log";
$fsdiffoutput_backup_number = 5;
$ignoreList = "/usr/local/radmind/etc/ignore";
$lapplyinput = "/usr/local/radmind/tmp/diff.T";
$lapplyoutput = "/usr/local/radmind/tmp/lapply_output.log";
$lapplyoutput_backup_number = 5;
$max_tries = 1;

This next bit of code lets me use the same script when it is run at logout and when it is run by cron overnight. In the first instance, I run without checksums so fsdiff is faster. In the second instance, I can turn checksums on and presumably no-one will be around to complain how long fsdiff takes.

# if the checksum trigger file exists, run with checksumming on
$checksumTriggerFile = "/var/radmind/client/.radmindWithChecksums";
if (-f $checksumTriggerFile) {
    $cksum = "-c sha1";
    unlink $checksumTriggerFile;
}

$radmind_error = "/usr/local/radmind/tmp/radmind_error";
$radmind_log = "/var/log/run_radmind.log";
$errorlog = "/usr/local/radmind/tmp/radmind_error.log";
$ktcheck = "/usr/local/bin/ktcheck -c sha1 $rserver 2> \"$errorlog\"";

The fsdiffCommand variable below takes advantage of a new fsdiff feature (-v) which gives percent-done feedback:

$fsdiffCommand = "/usr/local/bin/fsdiff -A $cksum -o $fsdiffoutput -v $fsdiffpath 2> \"$errorlog\"";
$lapply = "/usr/local/bin/lapply -F $cksum $rserver $lapplyinput 2> \"$errorlog\"";

$restartCheckFile = "/usr/local/radmind/tmp/restartCheck.T";
$rebootList = "/usr/local/radmind/etc/rebootList";
$rebootIgnoreList = "/usr/local/radmind/etc/rebootIgnoreList";

$updatesNeededFile = "/var/radmind/client/.updatesNeeded";

$extensionsCheckFile = "/usr/local/radmind/tmp/extensionsCheck.T";
$prefsPanesCheckFile = "/usr/local/radmind/tmp/prefsPanesCheck.T";

$ktcheck_pic = "/usr/local/radmind/images/ktcheck.tif";
$fsdiff_pic = "/usr/local/radmind/images/fsdiff.tif";
$lapply_pic = "/usr/local/radmind/images/lapply.tif";

I discovered that when using radmind to upgrade a machine from 10.2.x to 10.3, that sometimes it locked up during the lapply. Since (for me at least), upgrading from 10.2.x to 10.3 via radmind takes over an hour, and I have cron set to run tasks every 15 minutes, I thought perhaps something cron ran was crashing as libraries and code was swapped out from under it. So I added code in the run_radmind script to turn cron off. This seems to have improved the reliability of the upgrade from 10.2.x to 10.3.

################################################################################
# Stop cron so it doesn't call stuff we don't want to run during radmind
#
$cronMsg = `/System/Library/StartupItems/Cron/Cron stop`;
print "$cronMsg\n";

I can't really tell if this next bit is doing anything: I still see the output from fsdiff and lapply come in chunks with pauses in between, which looks like buffering to me...
################################################################################
# Turn buffering off
#
$|++;
$oldhandle = select( STDERR );
$|++;
select( $oldhandle );

print "%UIMODE AUTOCRATIC\n"; # iHook directive (This removes the menu bar in Panther - which appears even when no-one is logged in!)
print "%BACKGROUND $ktcheck_pic\n"; # iHook directive.
print "%BEGINPOLE\n"; # iHook directive.

Note the indeterminant progress bar, since there's really no way to tell how far along ktcheck is.
################################################################################
# Keep track of time, write to log
#
$rightnow = time; # keep track of time
system "echo \"--------------------------------\" >> $radmind_log";
system "date >> $radmind_log";
system "echo \"run_radmind started\" >> $radmind_log";

################################################################################
# Roll logs
#
&roll_log($ktcheckoutput, $ktcheckoutput_backup_number);
&roll_log($fsdiffoutput, $fsdiffoutput_backup_number);
&roll_log($lapplyoutput, $lapplyoutput_backup_number);

################################################################################
# prep stuff
chdir ("/");

# radmind loop
$current_try = 1;
while (1) {
    ################################################################################
    # run ktcheck
    #
    print "Task 1 of 3: Checking for updates on the server\n";
    $result = execute_command ("$ktcheck", $ktcheckoutput);
    if ($result == 0) {
        print "No updates from server\n";
    } elsif ($result == 1) {
        print "Updates found\n";
        system "/usr/bin/touch $updatesNeededFile";
    } else {
        print "ktcheck encountered a fatal error: $result\n";
        system "echo \"ktcheck encountered a fatal error: $result\" >> $radmind_log";
        &radmindFailed;
    }
    
    print "%1\n"; # iHook directive.
    ################################################################################
    # run fsdiff
    #
    print "Task 2 of 3: Examining the filesystem for differences\n";
    print "%BACKGROUND $fsdiff_pic\n"; # iHook directive.
    $result = system "$fsdiffCommand";
    if ($result != 0) {
        print "fsdiff encountered a fatal error: $result\n";
        system "echo \"fsdiff encountered a fatal error: $result\" >> $radmind_log";
        &radmindFailed;
    }
    
This next bit filters out any lines you don't want in the final lapply. I originally used it to filter out .DS_Store files. The problem came when later you needed to remove a directory. If that directory had any .DS_Store files within, they had been removed from lapply's transcript and therefore lapply would try to remove a directory that wasn't empty and it then failed. I planned to try to write some code to work around that issue, but never got around to it.

    ################################################################################
    # filter output from fsdiff
    #
    system "cat $fsdiffoutput | fgrep -v -f $ignoreList > $lapplyinput";

    print "%1\n"; # iHook directive.
We reset the progress bar to 1% (Setting it to 0% removes it from the window, which is visually jarring!

Now we run lapply.

    ################################################################################
    # run lapply
    #
    print "Task 3 of 3: Applying changes\n";
    print "%BACKGROUND $lapply_pic\n"; # iHook directive.
    $fsdsize = ( stat( $lapplyinput ))[ 7 ];
    if ( $fsdsize > 0 ) {

We get the number of lines in the applicable transcript so we can calculate (roughly) how far done we are. See the execute_lapply subroutine for more of this.

        $lapplyLineCount = int `wc -l $lapplyinput`;
        $result = execute_lapply ("$lapply", $lapplyoutput, $lapplyLineCount, "1");
        if ($result == 0) {
            &radmindSuccessful;
            last;
        } elsif ($result == 1) {
        if ($current_try >= $max_tries) {
            print "lapply failed too many times: $result\n"; 
            system "echo \"lapply failed too many times: $result\" >> $radmind_log";
            &radmindFailed;
            } else {
                print "lapply failed, trying again: $result\n";
                system "echo \"lapply failed, trying again: $result\" >> $radmind_log";
                next;
            }
        } else {
            print "lapply encountered a fatal error - see log for errors.\n";
            system "echo \"lapply failed: $result\" >> $radmind_log";
            &radmindFailed;
        }
    } else {
        print "No changes found!\n";
        system "echo \"No changes found\" >> $radmind_log";
        &radmindSuccessful;
        last;
    }
    $current_try++;
}

This is UofUtah's subroutine. I had never figured out pipe handles in Perl, so this was an education for me:
################################################################################
# This subroutine executes the radmind commands and saves the output to a log,
# prints it to stdout, and saves stderr to a log as well.
#
sub execute_command {
    local ($thecommand, $path_to_log) = @_;
    open (COMMAND, "$thecommand |");
    open (LOG, ">>$path_to_log");
    while (<COMMAND>) {
        print STDERR;
        print LOG;
    }
    close (LOG);
    close (COMMAND);
    return $? >> 8;
}

This subroutine, based on execute_command, has the changes needed to give percent-done feedback to the user:
################################################################################
# This subroutine executes lapply and saves the output to a log,
# prints it to stdout, and saves stderr to a log as well.
# It also provides percent-done feedback for use by iHook.
#
sub execute_lapply {
    local ($thecommand, $path_to_log, $fsdiff_line_count, $startPercent) = @_;
    open (COMMAND, "$thecommand |");
    open (LOG, ">>$path_to_log");
    $lapply_line_count = 0;
    while (<COMMAND>) {
        print STDERR;
        print LOG;

For each line that comes back from lapply, we increment our counter. Since we know how many lines are in the applicable transcript, and lapply returns one line of output for (basically) each line in the applicable transcript, we have a pretty good idea how far along we are.

        $lapply_line_count++;
        if ( int ($lapply_line_count/10) == ($lapply_line_count/10) ) (This way we only update every ten files or so)
        {
            $ratio = $lapply_line_count/$fsdiff_line_count;
            if ( $ratio > 1 ) { $ratio = 1; }
            $percentDone = $startPercent + int ($ratio*(99-$startPercent));
            print "%$percentDone\n";
Or you can get fancy and do this: print "%percentDone Updated item $lapply_line_count of $fsdiff_line_count\n";
        }
    }
    close (LOG);
    close (COMMAND);
    return $? >> 8;
}
################################################################################

# If radmind was successful, do these things
#
sub radmindSuccessful {
    print "Finishing up\n";
    #print "%99\n"; # iHook directive

    # How long did radmind take?
    $rightnowwer = time;
    $radmindtime = $rightnowwer - $rightnow;

    $message = "Radmind successful and took $radmindtime seconds";
    system "echo \"$message\" >> $radmind_log";
    print "%100\n"; # iHook directive

In Jaguar, we'd delete /System/Library/Extensions.kextcache and /System/Library/Extensions.mkext to get the OS to rebuild its extensions cache on restart. I've found that method to be unreliable with Panther. A method that seems to work is to touch the /System/Library/Extensions directory. So we grep the applicable transcript for "/System/Library/Extensions/", which would catch any changes to the directory. It's important to grep for "/System/Library/Extensions/", and not "/System/Library/Extensions" - since we are touching /System/Library/Extensions, on the next radmind run, fsdiff will notice that /System/Library/Extensions has changed, and will change it back. Then our script would see /System/Library/Extensions in the transcript, touch it, and so on...

    #do we need to update extensions cache?
    #were any kernel extensions added, removed, or changed?
    system "grep /System/Library/Extensions/ $lapplyinput > $extensionsCheckFile";
    #check the extensionsCheckFile filesize. If it's greater than zero, touch /System/Library/Extensions
    $extcfsize = ( stat( $extensionsCheckFile ))[ 7 ];
    if ( $extcfsize > 0 ) {
        system "/usr/bin/touch /System/Library/Extensions";
    }

We do something similar with Preference Panes. The OS keeps a cache of Pref Panes. When radmind adds or deletes a pane from either /System/Library/PreferencePanes or /Library/PreferencePanes we need to touch /System/Library/PreferencePanes to signal to the OS to update its cache. (Preference panes can also go in ~/Library/PreferencePanes. By grepping for "/Library/PreferencePanes/" we actually catch all possible locations, though in practice we don't manage user space)

    #do we need to invalidate Preferences Panes cache?
    #were any prefs panes added, removed, or changed?
    system "grep /Library/PreferencePanes/ $lapplyinput > $prefsPanesCheckFile";
    #check the prefsPanesCheckFile filesize. If it's greater than zero, touch /System/Library/PreferencePanes
    $ppcfsize = ( stat( $prefsPanesCheckFile ))[ 7 ];
    if ( $ppcfsize > 0 ) {
        system "/usr/bin/touch /System/Library/PreferencePanes";
    }

I don't like to restart after a radmind run unless it is needed. But how to tell if it's needed? I grep through the applicable transcript and look for items that trigger a restart. First we find all the items in the $rebootList, then eliminate the items in the $rebootIgnoreList.

Here's my current $rebootList:
/mach_kernel
/System/Library/
/Library/StartupItems/
/Library/Preferences/DirectoryService/
/Library/Application\bSupport/Norton\bSolutions\bSupport/

Here's my current $rebootIgnoreList
/Library/Application\bSupport/Norton\bSolutions\bSupport/Scheduler/SymSecondaryLaunch.app/Contents/MacOS/SymSecondaryLaunch
/Library/Application\bSupport/Norton\bSolutions\bSupport/Scheduler/schedLauncher
/Library/Preferences/DirectoryService/.DSRunningSP
/Library/StartupItems/NortonMissedTasks/NortonMissedTasks
/System/Library/CoreServices/.disk_label

The idea here is to reboot if anything in the rebootList has changed, unless it is in the rebootIgnoreList.
So if radmind just installs an application with all its files under /Applications, the machine will not reboot. But if stuff in /System/Library/ changes, it generally will reboot.

    #check to see if we need a restart
    system "fgrep -f $rebootList $lapplyinput | fgrep -v -f $rebootIgnoreList > $restartCheckFile";
    #check the restartCheckFile filesize. If it's greater than zero, restart
    $rscfsize = ( stat( $restartCheckFile ))[ 7 ];
    if ( $rscfsize > 0 ) {
        print "Items installed require a restart. Restarting now...\n";
        system "sleep 2";

Note that I don't actually restart in this script - instead I let the calling script restart. This allows me to run this script manually remotely via SSH and not get kicked off at the end.

        # we'll just fall through and exit normally and let the logout hook script
        # notice that the restartCheckFile exists and actually do the restart.
    } else {
        #if we are not restarting we should restart cron
        $cronmsg = `/System/Library/StartupItems/Cron/Cron start`;
        print "$cronmsg\n";
    }

Since we were successful, we can remove the trigger file that tells us updates were needed. If the update failed, the file would not be removed.

    if (-e $updatesNeededFile) { unlink $updatesNeededFile }

I use a simple CGI to report success to a webserver. I can then see when the last successful radmind was for a given host. You can find details about this CGI elsewhere on this website.

    #check in with the radmind check-in cgi
    system "curl http://radmind/cgi-bin/radmindCheckIn";
    exit 0;

}
################################################################################
# If radmind died, this is the cleanup subroutine.
#
sub radmindFailed {
    # print error log to stdout
    open (LOGGY, "<$errorlog");
    while (<LOGGY>) {
        print;
    }
    close (LOGGY);
    #system "touch $radmind_error";
    
    # How long did radmind take?
    $rightnowwer = time;
    $radmindtime = $rightnowwer - $rightnow;
    
    #
    $message = "Radmind failed and took $radmindtime seconds";
    system "echo \"$message\" >> $radmind_log";
    system "cat \"$errorlog\" >> $radmind_log";

We're not restarting, so we should start cron back up:

    # restart cron
    $cronmsg = `/System/Library/StartupItems/Cron/Cron start`;
    print "$cronmsg\n";

If radmind fails, I send mail to root - you may (probably will) want to send it elsewhere. Of course this means sendmail/postfix must be properly configured on the system.

    # notify the authorities
    $hostname = `hostname`;
    chomp $hostname;
    system "cat $errorlog | mail -s \"Radmind failed on $hostname.\" root";

exit 1;
}

Unchanged from UofUtah:
################################################################################
# This script renames $path_to_log so that the lowest number is always the
# newest. The oldest that is greater than $number_of_backups is deleted.
#
sub roll_log {
    local ($path_to_log, $number_of_backups) = @_;
    if (-e $path_to_log) {
        if (-e $path_to_log.".bak$number_of_backups") {
            unlink $path_to_log.".bak$number_of_backups";
        }
        for ($i = $number_of_backups ; $i > 1 ; $i-- ) {
            if (-f $path_to_log.".bak".($i-1)) {
                system "/bin/mv -f \"$path_to_log".".bak".($i-1)."\" \"$path_to_log".".bak$i\"";
            }
        }
        system "/bin/mv -f \"$path_to_log\" \"$path_to_log".".bak1\"";
    }
} 

Here's the pictures referenced by the script, which because they are TIFFs with transparency, look awful on a web page, but quite nice in iHook:
images.zip





Posted: Mon - May 10, 2004 at 09:22 PM      

©