Sat - May 22, 2004

Web access to radmind data (updated)

It's useful to be able to quickly view radmind data via a webpage. Kris Steinhoff has written a radmind management module for Webmin. This module allows one to perform many common radmind management tasks via a webpage. We wrote a standalone CGI that gives us read-only access to radmind data. (With the help of Chris Buskirk, I've added some files the CGI depends on and updated the script to fix the viewing of check-ins in domains other than .com - you should be able to get this up and running very easily on a Mac OS X radmind/web server)

In our organization, only a few people are authorized to edit radmind data, but it was useful to provide quick read-only access so that a tech could verify what applications a user had assigned to him or her in his/her command file. We also wanted code that could do some sanity checking for us - if a command file was referenced in the config file - did that command file actually exist? If a hostname was referenced in the config file, was that hostname actually in DNS? Did the transcripts referenced in a command file actually exist? This data helps any tech quickly determine the source of many radmind errors.

Our CGI allows the user to view the config file, command files, and transcripts. These are all hyperlinked, so if you are viewing the config file, you can click on a command filename to view it. When viewing a command file, you can click on a transcript to view its contents. Any item that doesn't exist is flagged in red and not clickable, so you know there is a problem.

Additionally, the CGI knows about the /var/radmind/client/updated directory that is managed by the radmindCheckin cgi we use to report successful radmind runs. It can display the last time any given machine successfully ran radmind, sorting by hostname, command file name , or date.

Without further ado, here are the files needed to implement the CGI on Mac OS X (I'd include the CGI in the text of this post, but iBlog insists on rendering the HTML that is embedded in the script) :

radmind cgi.zip

And some screen shots:

Main menu: (note an additional option that is not in the script provided here. It's an option to view our Makefile, which we use to build our command files from "base" command files.)



View Config file:




View command file list:




View transcript list. The script also looks to see how many command files reference each transcript. This way you can figure out which transcripts may no longer be needed:




Viewing an individual transcript:




Viewing the Check-in report. You can click any column heading to resort the list by hostname, command file name, or date.

Posted at 11:41 PM     Read More  

Mon - May 10, 2004

Running radmind via cron (by way of periodic) and on-demand

Here are some more of the scripts I use to automate radmind...

Rather than editing crontab files, I put a script in /private/etc/periodic/daily, called 900.autoradmind:
900.autoradmind


This script checks to see the last time radmind ran; sees if there are any needed updates on the server (via ktcheck), and checks to see if anyone is logged in at the console.
If no-one is logged in, it runs a radmind session via a script at /usr/local/radmind/scripts/radmindNow
If someone is logged in and there are updates on the server, or it has been more than seven days since the last radmind session, it attempts to log the user out gracefully. If there are any unsaved documents, this will fail.
If it fails to run radmind and it's been more than seven days since the last run, it notifies the admin and alerts the user. I'll share this alert app at a later date (once I've made it easily customizable...)

When the 900.autoradmind script does run radmind, it calls this script:
radmindNow


#!/bin/sh

radmindTriggerFile="/var/radmind/client/.radmindOnLogout"
restartCheckFile="/usr/local/radmind/tmp/restartCheck.T"

touch $radmindTriggerFile
/usr/local/radmind/scripts/logoutHook

exit 0

Which as you can see, simply calls the logoutHook:
logoutHook


#!/bin/sh

radmindTriggerFile="/var/radmind/client/.radmindOnLogout"
restartCheckFile="/usr/local/radmind/tmp/restartCheck.T"

if [ -f $radmindTriggerFile ]; then
   rm $radmindTriggerFile
   rm -f $restartCheckFile
   /Applications/Utilities/radmind/iHook.app/Contents/MacOS/iHook --no-titlebar --script=/usr/local/radmind/scripts/run_radmind.pl
   if [ -s $restartCheckFile ]; then
      shutdown -r now
   fi
fi
exit 0


Which in turn calls iHook and my run_radmind script, which is detailed elsewhere.

This arrangement gives me a lot of flexibility. I can ssh in and run the run_radmind script if I want to see the output, but it's not important (or desirable) for the user at the machine to see the output. (Typically I'll do this if I know no-one is logged in and no-one is likely to log in.) Or, if I want iHook to come up and give the user some indication of what is going on, I can run the "radmindNow" script. Finally, since I am using a logoutHook script, I can let the user trigger a radmind session by running this app:
Update this Mac.zip




Which is just an AppleScript applet:

activate

set theMessage to "Warning: update this machine?" & return & return & "You will be logged out and software on this machine will be updated. This may take several minutes and may require a restart. Are you sure you want to continue?"

display dialog theMessage buttons {"No", "Yes"} default button "Yes" giving up after 15 with icon caution

if (button returned of result is equal to "Yes") then

       try

              do shell script "touch /var/radmind/client/.radmindOnLogout"

              ignoring application responses

                     tell application "loginwindow" to «event aevtrlgo»

              end ignoring

       on error theError

              display dialog "Error: " & theError giving up after 60

       end try

end if


If the user launches this application and clicks yes, they will be logged out and a radmind session will run.

Posted at 10:10 PM     Read More  

run_radmind script (updated)

Here is the run_radmind script I use. It's called on demand at logout, and as a periodic task nightly.

First the script itself:
run_radmind.pl


Now, some commentary (in blue):

#!/usr/bin/perl
################################################################################
# run_radmind
#
# This script runs radmind.
#
# Requires the radmind client tools (and a running server obviously).
# (See http://rsug.itd.umich.edu/software/radmind)
#
# Can also be used with iHook
# (See http://rsug.itd.umich.edu/software/ihook)
#

This script started out as a light edit of University of Utah's run_radmind script, and I've added my own tweaks and changes. Of particular coolness was UofU's execute_command subroutine, which allowed the output of the radmind tools to appear in iHook's drawer AND be captured to a file, which is invaluable for troubleshooting.

# Copyright (c) 2002 University of Utah Student Computing Labs.
# All Rights Reserved.
#
# Many, many modifications by Greg Neagle, Walt Disney Feature Animation
#
# Permission to use, copy, modify, and distribute this software and
# its documentation for any purpose and without fee is hereby granted,
# provided that the above copyright notice appears in all copies and
# that both that copyright notice and this permission notice appear
# in supporting documentation, and that the name of The University
# of Utah not be used in advertising or publicity pertaining to
# distribution of the software without specific, written prior
# permission. This software is supplied as is without expressed or
# implied warranties of any kind.
#
################################################################################
################################################################################
# change these to match your installation
$rserver = "-h radmind";
$cksum = "";
$fsdiffpath = "/";
$ktcheckoutput = "/usr/local/radmind/tmp/ktcheck_output.log";
$ktcheckoutput_backup_number = 5;
$fsdiffoutput = "/usr/local/radmind/tmp/fsdiff_output.log";
$fsdiffoutput_backup_number = 5;
$ignoreList = "/usr/local/radmind/etc/ignore";
$lapplyinput = "/usr/local/radmind/tmp/diff.T";
$lapplyoutput = "/usr/local/radmind/tmp/lapply_output.log";
$lapplyoutput_backup_number = 5;
$max_tries = 1;

This next bit of code lets me use the same script when it is run at logout and when it is run by cron overnight. In the first instance, I run without checksums so fsdiff is faster. In the second instance, I can turn checksums on and presumably no-one will be around to complain how long fsdiff takes.

# if the checksum trigger file exists, run with checksumming on
$checksumTriggerFile = "/var/radmind/client/.radmindWithChecksums";
if (-f $checksumTriggerFile) {
    $cksum = "-c sha1";
    unlink $checksumTriggerFile;
}

$radmind_error = "/usr/local/radmind/tmp/radmind_error";
$radmind_log = "/var/log/run_radmind.log";
$errorlog = "/usr/local/radmind/tmp/radmind_error.log";
$ktcheck = "/usr/local/bin/ktcheck -c sha1 $rserver 2> \"$errorlog\"";

The fsdiffCommand variable below takes advantage of a new fsdiff feature (-v) which gives percent-done feedback:

$fsdiffCommand = "/usr/local/bin/fsdiff -A $cksum -o $fsdiffoutput -v $fsdiffpath 2> \"$errorlog\"";
$lapply = "/usr/local/bin/lapply -F $cksum $rserver $lapplyinput 2> \"$errorlog\"";

$restartCheckFile = "/usr/local/radmind/tmp/restartCheck.T";
$rebootList = "/usr/local/radmind/etc/rebootList";
$rebootIgnoreList = "/usr/local/radmind/etc/rebootIgnoreList";

$updatesNeededFile = "/var/radmind/client/.updatesNeeded";

$extensionsCheckFile = "/usr/local/radmind/tmp/extensionsCheck.T";
$prefsPanesCheckFile = "/usr/local/radmind/tmp/prefsPanesCheck.T";

$ktcheck_pic = "/usr/local/radmind/images/ktcheck.tif";
$fsdiff_pic = "/usr/local/radmind/images/fsdiff.tif";
$lapply_pic = "/usr/local/radmind/images/lapply.tif";

I discovered that when using radmind to upgrade a machine from 10.2.x to 10.3, that sometimes it locked up during the lapply. Since (for me at least), upgrading from 10.2.x to 10.3 via radmind takes over an hour, and I have cron set to run tasks every 15 minutes, I thought perhaps something cron ran was crashing as libraries and code was swapped out from under it. So I added code in the run_radmind script to turn cron off. This seems to have improved the reliability of the upgrade from 10.2.x to 10.3.

################################################################################
# Stop cron so it doesn't call stuff we don't want to run during radmind
#
$cronMsg = `/System/Library/StartupItems/Cron/Cron stop`;
print "$cronMsg\n";

I can't really tell if this next bit is doing anything: I still see the output from fsdiff and lapply come in chunks with pauses in between, which looks like buffering to me...
################################################################################
# Turn buffering off
#
$|++;
$oldhandle = select( STDERR );
$|++;
select( $oldhandle );

print "%UIMODE AUTOCRATIC\n"; # iHook directive (This removes the menu bar in Panther - which appears even when no-one is logged in!)
print "%BACKGROUND $ktcheck_pic\n"; # iHook directive.
print "%BEGINPOLE\n"; # iHook directive.

Note the indeterminant progress bar, since there's really no way to tell how far along ktcheck is.
################################################################################
# Keep track of time, write to log
#
$rightnow = time; # keep track of time
system "echo \"--------------------------------\" >> $radmind_log";
system "date >> $radmind_log";
system "echo \"run_radmind started\" >> $radmind_log";

################################################################################
# Roll logs
#
&roll_log($ktcheckoutput, $ktcheckoutput_backup_number);
&roll_log($fsdiffoutput, $fsdiffoutput_backup_number);
&roll_log($lapplyoutput, $lapplyoutput_backup_number);

################################################################################
# prep stuff
chdir ("/");

# radmind loop
$current_try = 1;
while (1) {
    ################################################################################
    # run ktcheck
    #
    print "Task 1 of 3: Checking for updates on the server\n";
    $result = execute_command ("$ktcheck", $ktcheckoutput);
    if ($result == 0) {
        print "No updates from server\n";
    } elsif ($result == 1) {
        print "Updates found\n";
        system "/usr/bin/touch $updatesNeededFile";
    } else {
        print "ktcheck encountered a fatal error: $result\n";
        system "echo \"ktcheck encountered a fatal error: $result\" >> $radmind_log";
        &radmindFailed;
    }
    
    print "%1\n"; # iHook directive.
    ################################################################################
    # run fsdiff
    #
    print "Task 2 of 3: Examining the filesystem for differences\n";
    print "%BACKGROUND $fsdiff_pic\n"; # iHook directive.
    $result = system "$fsdiffCommand";
    if ($result != 0) {
        print "fsdiff encountered a fatal error: $result\n";
        system "echo \"fsdiff encountered a fatal error: $result\" >> $radmind_log";
        &radmindFailed;
    }
    
This next bit filters out any lines you don't want in the final lapply. I originally used it to filter out .DS_Store files. The problem came when later you needed to remove a directory. If that directory had any .DS_Store files within, they had been removed from lapply's transcript and therefore lapply would try to remove a directory that wasn't empty and it then failed. I planned to try to write some code to work around that issue, but never got around to it.

    ################################################################################
    # filter output from fsdiff
    #
    system "cat $fsdiffoutput | fgrep -v -f $ignoreList > $lapplyinput";

    print "%1\n"; # iHook directive.
We reset the progress bar to 1% (Setting it to 0% removes it from the window, which is visually jarring!

Now we run lapply.

    ################################################################################
    # run lapply
    #
    print "Task 3 of 3: Applying changes\n";
    print "%BACKGROUND $lapply_pic\n"; # iHook directive.
    $fsdsize = ( stat( $lapplyinput ))[ 7 ];
    if ( $fsdsize > 0 ) {

We get the number of lines in the applicable transcript so we can calculate (roughly) how far done we are. See the execute_lapply subroutine for more of this.

        $lapplyLineCount = int `wc -l $lapplyinput`;
        $result = execute_lapply ("$lapply", $lapplyoutput, $lapplyLineCount, "1");
        if ($result == 0) {
            &radmindSuccessful;
            last;
        } elsif ($result == 1) {
        if ($current_try >= $max_tries) {
            print "lapply failed too many times: $result\n"; 
            system "echo \"lapply failed too many times: $result\" >> $radmind_log";
            &radmindFailed;
            } else {
                print "lapply failed, trying again: $result\n";
                system "echo \"lapply failed, trying again: $result\" >> $radmind_log";
                next;
            }
        } else {
            print "lapply encountered a fatal error - see log for errors.\n";
            system "echo \"lapply failed: $result\" >> $radmind_log";
            &radmindFailed;
        }
    } else {
        print "No changes found!\n";
        system "echo \"No changes found\" >> $radmind_log";
        &radmindSuccessful;
        last;
    }
    $current_try++;
}

This is UofUtah's subroutine. I had never figured out pipe handles in Perl, so this was an education for me:
################################################################################
# This subroutine executes the radmind commands and saves the output to a log,
# prints it to stdout, and saves stderr to a log as well.
#
sub execute_command {
    local ($thecommand, $path_to_log) = @_;
    open (COMMAND, "$thecommand |");
    open (LOG, ">>$path_to_log");
    while (<COMMAND>) {
        print STDERR;
        print LOG;
    }
    close (LOG);
    close (COMMAND);
    return $? >> 8;
}

This subroutine, based on execute_command, has the changes needed to give percent-done feedback to the user:
################################################################################
# This subroutine executes lapply and saves the output to a log,
# prints it to stdout, and saves stderr to a log as well.
# It also provides percent-done feedback for use by iHook.
#
sub execute_lapply {
    local ($thecommand, $path_to_log, $fsdiff_line_count, $startPercent) = @_;
    open (COMMAND, "$thecommand |");
    open (LOG, ">>$path_to_log");
    $lapply_line_count = 0;
    while (<COMMAND>) {
        print STDERR;
        print LOG;

For each line that comes back from lapply, we increment our counter. Since we know how many lines are in the applicable transcript, and lapply returns one line of output for (basically) each line in the applicable transcript, we have a pretty good idea how far along we are.

        $lapply_line_count++;
        if ( int ($lapply_line_count/10) == ($lapply_line_count/10) ) (This way we only update every ten files or so)
        {
            $ratio = $lapply_line_count/$fsdiff_line_count;
            if ( $ratio > 1 ) { $ratio = 1; }
            $percentDone = $startPercent + int ($ratio*(99-$startPercent));
            print "%$percentDone\n";
Or you can get fancy and do this: print "%percentDone Updated item $lapply_line_count of $fsdiff_line_count\n";
        }
    }
    close (LOG);
    close (COMMAND);
    return $? >> 8;
}
################################################################################

# If radmind was successful, do these things
#
sub radmindSuccessful {
    print "Finishing up\n";
    #print "%99\n"; # iHook directive

    # How long did radmind take?
    $rightnowwer = time;
    $radmindtime = $rightnowwer - $rightnow;

    $message = "Radmind successful and took $radmindtime seconds";
    system "echo \"$message\" >> $radmind_log";
    print "%100\n"; # iHook directive

In Jaguar, we'd delete /System/Library/Extensions.kextcache and /System/Library/Extensions.mkext to get the OS to rebuild its extensions cache on restart. I've found that method to be unreliable with Panther. A method that seems to work is to touch the /System/Library/Extensions directory. So we grep the applicable transcript for "/System/Library/Extensions/", which would catch any changes to the directory. It's important to grep for "/System/Library/Extensions/", and not "/System/Library/Extensions" - since we are touching /System/Library/Extensions, on the next radmind run, fsdiff will notice that /System/Library/Extensions has changed, and will change it back. Then our script would see /System/Library/Extensions in the transcript, touch it, and so on...

    #do we need to update extensions cache?
    #were any kernel extensions added, removed, or changed?
    system "grep /System/Library/Extensions/ $lapplyinput > $extensionsCheckFile";
    #check the extensionsCheckFile filesize. If it's greater than zero, touch /System/Library/Extensions
    $extcfsize = ( stat( $extensionsCheckFile ))[ 7 ];
    if ( $extcfsize > 0 ) {
        system "/usr/bin/touch /System/Library/Extensions";
    }

We do something similar with Preference Panes. The OS keeps a cache of Pref Panes. When radmind adds or deletes a pane from either /System/Library/PreferencePanes or /Library/PreferencePanes we need to touch /System/Library/PreferencePanes to signal to the OS to update its cache. (Preference panes can also go in ~/Library/PreferencePanes. By grepping for "/Library/PreferencePanes/" we actually catch all possible locations, though in practice we don't manage user space)

    #do we need to invalidate Preferences Panes cache?
    #were any prefs panes added, removed, or changed?
    system "grep /Library/PreferencePanes/ $lapplyinput > $prefsPanesCheckFile";
    #check the prefsPanesCheckFile filesize. If it's greater than zero, touch /System/Library/PreferencePanes
    $ppcfsize = ( stat( $prefsPanesCheckFile ))[ 7 ];
    if ( $ppcfsize > 0 ) {
        system "/usr/bin/touch /System/Library/PreferencePanes";
    }

I don't like to restart after a radmind run unless it is needed. But how to tell if it's needed? I grep through the applicable transcript and look for items that trigger a restart. First we find all the items in the $rebootList, then eliminate the items in the $rebootIgnoreList.

Here's my current $rebootList:
/mach_kernel
/System/Library/
/Library/StartupItems/
/Library/Preferences/DirectoryService/
/Library/Application\bSupport/Norton\bSolutions\bSupport/

Here's my current $rebootIgnoreList
/Library/Application\bSupport/Norton\bSolutions\bSupport/Scheduler/SymSecondaryLaunch.app/Contents/MacOS/SymSecondaryLaunch
/Library/Application\bSupport/Norton\bSolutions\bSupport/Scheduler/schedLauncher
/Library/Preferences/DirectoryService/.DSRunningSP
/Library/StartupItems/NortonMissedTasks/NortonMissedTasks
/System/Library/CoreServices/.disk_label

The idea here is to reboot if anything in the rebootList has changed, unless it is in the rebootIgnoreList.
So if radmind just installs an application with all its files under /Applications, the machine will not reboot. But if stuff in /System/Library/ changes, it generally will reboot.

    #check to see if we need a restart
    system "fgrep -f $rebootList $lapplyinput | fgrep -v -f $rebootIgnoreList > $restartCheckFile";
    #check the restartCheckFile filesize. If it's greater than zero, restart
    $rscfsize = ( stat( $restartCheckFile ))[ 7 ];
    if ( $rscfsize > 0 ) {
        print "Items installed require a restart. Restarting now...\n";
        system "sleep 2";

Note that I don't actually restart in this script - instead I let the calling script restart. This allows me to run this script manually remotely via SSH and not get kicked off at the end.

        # we'll just fall through and exit normally and let the logout hook script
        # notice that the restartCheckFile exists and actually do the restart.
    } else {
        #if we are not restarting we should restart cron
        $cronmsg = `/System/Library/StartupItems/Cron/Cron start`;
        print "$cronmsg\n";
    }

Since we were successful, we can remove the trigger file that tells us updates were needed. If the update failed, the file would not be removed.

    if (-e $updatesNeededFile) { unlink $updatesNeededFile }

I use a simple CGI to report success to a webserver. I can then see when the last successful radmind was for a given host. You can find details about this CGI elsewhere on this website.

    #check in with the radmind check-in cgi
    system "curl http://radmind/cgi-bin/radmindCheckIn";
    exit 0;

}
################################################################################
# If radmind died, this is the cleanup subroutine.
#
sub radmindFailed {
    # print error log to stdout
    open (LOGGY, "<$errorlog");
    while (<LOGGY>) {
        print;
    }
    close (LOGGY);
    #system "touch $radmind_error";
    
    # How long did radmind take?
    $rightnowwer = time;
    $radmindtime = $rightnowwer - $rightnow;
    
    #
    $message = "Radmind failed and took $radmindtime seconds";
    system "echo \"$message\" >> $radmind_log";
    system "cat \"$errorlog\" >> $radmind_log";

We're not restarting, so we should start cron back up:

    # restart cron
    $cronmsg = `/System/Library/StartupItems/Cron/Cron start`;
    print "$cronmsg\n";

If radmind fails, I send mail to root - you may (probably will) want to send it elsewhere. Of course this means sendmail/postfix must be properly configured on the system.

    # notify the authorities
    $hostname = `hostname`;
    chomp $hostname;
    system "cat $errorlog | mail -s \"Radmind failed on $hostname.\" root";

exit 1;
}

Unchanged from UofUtah:
################################################################################
# This script renames $path_to_log so that the lowest number is always the
# newest. The oldest that is greater than $number_of_backups is deleted.
#
sub roll_log {
    local ($path_to_log, $number_of_backups) = @_;
    if (-e $path_to_log) {
        if (-e $path_to_log.".bak$number_of_backups") {
            unlink $path_to_log.".bak$number_of_backups";
        }
        for ($i = $number_of_backups ; $i > 1 ; $i-- ) {
            if (-f $path_to_log.".bak".($i-1)) {
                system "/bin/mv -f \"$path_to_log".".bak".($i-1)."\" \"$path_to_log".".bak$i\"";
            }
        }
        system "/bin/mv -f \"$path_to_log\" \"$path_to_log".".bak1\"";
    }
} 

Here's the pictures referenced by the script, which because they are TIFFs with transparency, look awful on a web page, but quite nice in iHook:
images.zip

Posted at 09:22 PM     Read More  

Fri - May 7, 2004

Sat - January 17, 2004

Wed - November 12, 2003

Tue - November 11, 2003

Mon - November 10, 2003