22 September 2000
Now automatically "sign" all classes so that they can have native methods. Rebuilt the mrj and as tools to no longer manually sign their classes with native methods. The signing is now done by ToolClassLoader.
Enhanced the setfile tool to permit mass setting of all files in a directory.
Updated the ant tool to the latest sources as of 22 September. Includes the Bean Scripting Framework, and the Rhino JavaScript classes.
10 August 2000
Bundled the "ant" tool, an XML-based Make utility. Included a demo build.xml in JShell's default directory, which runs a JavaScript. For more information about ant, see the Ant Homepage.
Fixed a bug in the set command, which was enumerating the system properties using System.getProperties().keys(), rather than System.getProperties().propertyNames().
24 July 2000
Implemented a workaround for a bug introduced in MRJ 2.2 when using Swing JFrame/JTextArea in combination where switching JShell to the background or simply bringing another window to front causes any Swing text area to lose keyboard focus. The workaround fixes this by patching HiliteWindow to detect when a different Swing window comes to front, and then calling requestFocus() on the root frame followed by JTextArea.requestFocus().
Added the "wrap" command, which wraps its input to the visible width of the console (which is indicated in the "window.width" property.
The "ls" command now recognizes the "-l" option, which provides more extensive information about files. Thanks to Stephyn G. W. Butcher for the useful contribution.
14 May 2000
Generalized handling of URL protocol handling. Kernel now installs a URLStreamHandlerFactory to handle all creation of URLStreamHandlers. For the "resource:" protocol, a specialized handler is used as before, to navigate JShell's class loader hierarchy. For all other protocols, the current tool is allowed to define the "java.protocol.handler.pkgs" property and provide its own classes. To accomplish this, the KernelURLStreamHandler delegates to tool provided protocol handlers or to the default handlers in "sun.net.www.protocol".
jEdit is a good example of a tool that defines its own resource handlers. jEdit defines the "jeditplugins:" protocol, which enumerates all known installed plugins, and the "jeditresource:" protocol, which reads a resource out of a given plugin. Enhanced the Window Proxy plugin and Drag & Drop plugins so that they show up in jEdit's plugins menu and in the "Documentation For Installed Plugins" links.
Also rebuilt jEdit as a compressed archive using trusty jlink. Found some problems with InflaterInputStream.read(), which doesn't return -1 when it hits EOF, but instead throws an EOFException. This breaks the "resource:" protocol when the stream is consumed by Properties.load() which is used by jEdit. Worked around by reading small compressed resources entirely into memory, and wrapping large resource streams in a FilterInputStream that maps EOFException to -1 in its read() method.
12 May 2000
Haven't been documenting the new features as the releases have gone out lately. Hopefully better documentation will materialize soon. For now, here's what new:
Support for compressed jar files should be pretty much there. There are some mighty strange cases to support when reading data out of compressed jars, such as when a ZipInputStream is used, and the length of the uncompressed entry is apparently unknown. This effects both the internal class loader (ToolClassLoader) and the one included in the java command.
Added a -jar option to the java command, which will read the META-INF/MANIFEST.MF file
looking for a line of the form "Main-Class:
The jlink tool now supports the following argument types: (1) .jar file, which is merged into the resulting output file as is; (2) plain file, which is added to the top of the output file; (3) directory, whose contents are recursively added to top-level of the output file. The 3rd option gives a lot more flexibility than the JDK's jar command, which always adds items relative to the current directory, which makes it really quite difficult to build jar files from disparate sources. In addition, when any file is added to the output file, if it doesn't end with ".class" but it is in fact found to be a .class file, its internal class name is used as the name of the entry in the output file. This gets around problems where long filenames are truncated on the Mac OS file system (no longer a problem with MRJ 2.2 and Mac OS 9, but helpful on older Mac OS installations).
February 15, 2000
JShell now has its own icon. I don't really like this one that much, but it's supposed to evoke a "shell" connected with the letter J. If you have an idea or artistic talent, to help me create a better icon, please speak up!
Added support for AppleScript files, which have the ".as" extension. When a script is compiled, an extra global variable, "arguments" is compiled in as well, that is an AppleScript list containing all of the arguments passed to the script. Take a look at JScripts/date.as for an example. A interface to ToolServer is provided in ts.as. Look at JScripts/perl.jxs for an example that uses ToolServer to run perl scripts. The tscd command synchronizes JShell's and ToolServer's current working directory (requires perl).
Added a history property to tool.property files. This provides an automatic way to give tools their own unique input history. The as and js tools are now both using this.
Improved the shell's parser by allowing anything to appear in quoted strings. This lets you pass just about anything through to ToolServer, etc. Some command may have to be rewritten to handle their own backslash escapes, but this makes much more sense this way.
Deferred loading of a Tool's main method, until ToolThread.run(), so that a tool's static initializer will run in the correct ToolContext (thread group). This is a subtle change, and there are still problems with tools that cache too many things in globals, but it should make many more tool happy. If you experience problems with any tools with this change, please let me know.
August 15, 1999
Removed dependence on JDirect1 unsupported classes (ProcessSerialNumber, OSUtils), for MRJ 2.2 compatibility.
Enhanced the jlink tool to expand truncated class file names when merging .jar files. Can now build the shell.jxe tool entirely within JShell on the MacOS (yay!). Requires MRJ 2.2 or later. Looking forward to OS 9's longer file name support.
May 6, 1999
Lots of changes to fundamental architecture of JShell, JXE itself. First, obtained the internet domain "jxe.org" which effectively reserves all Java package names in "org.jxe.*" for JXE's use. JXE stands for "Java Execution Environment." Most classes formerly in the "jxe" package are now in "org.jxe.kernel." Mac OS specific classes are in "org.jxe.mac." JShell classes are in "org.jxe.jshell." Finally, all core JShell tools are in "org.jxe.tools."
Trying to use JShell to build more of itself. Simple, single file tools can be built fairly automatically using "javac -d . toolName.java ; jar cf toolName.jxe org." More complex tools with differing classpath needs are currently still built with CodeWarrior.
Now using Netscape's Rhino JavaScript implementation as a shell scripting language.
Console window title also tracks the name of the current working directory.
New commands, gzip, gunzip, ln (makes aliases).
History command (thanks to Brian Nenninger <bwn@kreative.net> for the initial implementation). Up/down arrows now access history elements in circular buffer fashion. Added new interface, Console.InputHistory, to allow console clients to advertise a history to the console, which does the command line editing.
New command, jlink, combines .jar files. Really want a better jar tool.
Can now run jEdit within JShell. Many changes to resource handling to enable that. Still leaking ToolContexts in ToolClassLoader.
Added tool building scripts, buildtool and build1tool to JScripts. These build simple tools from .java sources, and install the tool in JTools.
Checked in baseline versions of the tools, to make it easier to bootstrap JShell on a new machine.
April 18, 1999
Added new interfaces jxe.console.Console, and jxe.console.ConsoleStream. This allows a running tool to discover the current console by checking System.in for instanceof ConsoleStream. If a console exists, Tool.ToolThread.run() attaches to it. This provides a way for the console to know which tool is currently running. Both the AWT console and the telnetd "virtual" console have been updated to use this. Can now kill tools with ^C, and signal EOF with ^D in both contexts. More sophisticated signal handling will now be possible, watch for ^Z (suspend) and others in the future. 1970 here we come!
Modernized the trap patching code, using JDirect 2, jump vectors, and ensuring the patches are removed by using termination services.
April 16, 1999
New experimental feature: added a Mac OS 8.5 window proxy to the console window, which provides a popup path menu (command click on the title or icon), and a draggable icon representing the current path. As you move around using cd, the window proxy tracks your location, and the folder icon changes if it's a special folder (i.e. System Folder, etc.). If you choose an item from the path popup, just like the finder (and many other Mac applications) that folder is made visible, and the Finder is brought to front. To get the events, I'm using a WaitNextEvent patch again. This could go away if MRJ had an event filtering API.
April 15, 1999
Fixed a bug in ToolClassLoader.getResource(), which was throwing an exception during static initialization of jxe.protocols.resource.Handler. This was trying to create a helper SecurityManager to get the current class loader. Changed Handler.ResourceConnection.getInputStream() to use ToolContext.getCurrentContext() to get the current context, and to get the class loader from the context's tool via Tool.getToolClassLoader().
April 7, 1999
Added preliminary support for the EOF condition when a tool is
reading from the console. Typing ^D will signal the EOF condition for
the current tool, and will make the tool read an empty line, followed
by a null line (e.g. when using BufferedReader). The Command class
conditionally clones the I/O streams when it inherits them from the
current ToolContext, by checking instanceof
CloneableObject. Currently only the console InputStream is a
CloneableObject, because its EOF condition must be unique.
Try it with the cat command. Thanks to Waldemar Horwat for suggesting it.
April 5, 1999
Added top-level window creation tracking. This allows tools that create windows and call System.exit() in their WindowListener.windowClosing() method to work more or less as expected.
February 24, 1999
Fixed a problem discovered when trying to run the ICE browser under JShell. Wasn't initializing the security manager properly, so ICE's static initializers would fail with a NullPointerException. Now guarantee that the System class is initialized before any static initializers of a tool are run.
Was getting a verify error when running the java command on JDK 1.1.6 on Windows. Seemed to be a problem with defining the JavaClassLoader in the java.lang package. Moved back to top-level and it works again.
February 17, 1999
Added a font control pop-up menu, with two sub-menus for size and font face. There's no persistent storage of this yet. Eventually there will be a JShell.properties file that will contain default settings. Question, how does one bring up a popup menu on Windows? I can't make it happen under Virtual PC.
Cleaned up command line editing a lot. The JShell console will no longer allow you to back up with the backspace key before the current prompt. You can also quickly insert some text by selecting it and hitting return, and the text will be appended to the current command line. You can still edit text after the current prompt. Also completely buffers all input while a command is running.
Improved filename completion enormously. Now checks to see if part of the prefix you type is a pathname, and if so, attempts to match the part after the path name in that directory. So, you can start in "/" and type the name of your hard disk, and hit tab and it should match it. Also expands to longest common prefix if there are multiple matches and beeps if there aren't any.
February 2, 1999
Lots of changes to the console window, hopefully some improvements.
Added file name completion -- if you hit tab while typing a filename, if that prefix is unique in the current directory, it will be completed.
Now providing a generic AWT TextArea console if swing isn't available. JShellConsole was restructured to work with the interface "ConsoleFrame" instead of subclassing JFrame. This should keep JShell running on slower speed machines.
Improved the shell's behavior when wildcards don't match anything are used. I've followed tcsh's handling of this, here's an example:
> ls *.foo
ls: No match.
This is done by the CommandParser class throwing an exception when Command.globArg returns false.
Fixed a bug in the java command where it would inappropriately find classes using findSystemClass. Now it uses the class loader that loads the java command, so that substitute classes (such as System) will be used. This corrects a problem when loading two copies of Rhino (JS in Java), where they both synchronize on the same System.in object, causing one instance to deadlock the other.
January 3, 1999
Fixed some bugs with telnetd, had to override InputStream.read() to prevent endless blocking.
Made an experimental change to ToolContext, ToolThread, and ToolClassLoader, so that when a thread isn't running in a Tool's thread group the current ToolContext is determined by the current class loader. This becomes necessary now that tools are allowed to use AWT. Typically tools will get their methods called in an AWT thread, and not in their own thread group. This brings up an issue: if tools call System.exit() in response to closing a window, and JShell doesn't allow the tool to do that, what should happen? Some interactive tools such as IBM's beanery don't work correctly in this case, they expect to be able to just shut down the JVM when their window closes.
Enhanced the java command to accept a -classpath argument, which allows directories and .jar files to be specified in the usual way. Java now keeps a vector of the most recently used class path elements, and uses the same class loader over and over, so as classes are loaded from the specified class path, the program runs faster. Try it it with the new "javac.jxs" script. The first time you run it, it loads the classes for the first time. Subsequent runs are much faster. You can then force unloading by using the "unload java" command. Run javac again, and it is slow the first time. The java command also special-cases classes that start with "java." and only loads these as system classes, because, for example, it's really not wise to load another copy of "java.lang.Object" and expect the program to continue functioning.
January 1, 1999
Well I finally bit the bullet and learned enough about Swing JTextAreas to craft a minimal console window. It's not perfect, but it is pretty fast (flushes minimally) and it allows other tools to run that use AWT, since JShell is now a JBindery application. In addition, you can create multiple console windows (each with an independent shell) using the new "console" command. To create a new console window with the name "Fred" use the command:
console "Fred" &
The "&" runs the command in the background; otherwise the initial shell would wait for the "Fred" console command to finish. This is pretty powerful stuff.
I fixed a variety of places in the code where conversions between UNICODE and single bytes weren't being done correctly. This amounted to changing DataInputStream to BufferedReader, and making sure to use the proper java.lang.String constructors. There's still a bug lurking where a lot of option characters can't be typed into the shell, because the ANTLR lexical analyzer can't handle UNICODE characters > 0xff. Characters like • (option-8) don't get handled correctly when typed in. Perhaps I could fix this using HTML-like character entities (•) or UNICODE hex escapes.
12/8/98
This release provides preliminary support for Windows 95/98/NT, and probably UNIX. The current working directory problem is solved by substituting custom versions of the java.io classes File, FileInputStream, FileOutputStream, and RandomAccessFile (thanks to Kelly Jacklin for suggesting this). The way files are opened has been changed to always use a File object to get the absolute path of a file, and File itself has been changed to always convert itself to an absolute path, so all relative pathname interpretation is done in the File constructors, using the system property "user.dir" which is changed by the cd tool. The batch file "JSHELL.BAT" is provided to run JShell on Windows, assuming JDK 1.1.6 is in C:\jdk1.1.6. I've only verified that this runs on Virtual PC using Windows 95, your mileage may vary.
To build substitutes.jar and files.jar, you need to take my diffs (System.java.diff, File.java.difm FileInputStream.java.diff, FileOutputStream.java.diff, and RandomAccessFile.java.diff) and apply them to the JDK 1.1 versions of these files (the versions that come with CW Pro 4 will do).
9/20/98
More radical revisions: this version provides a custom version of the System class for each running command. This java.lang.System class is initialized to a known state prior to the tool's invocation. Since each running tool has its own copy of System, each is guaranteed to have unique values for the globals System.in/out/err/props. This has performance advantages, as these I/O channels don't have to consult the current ToolContext to redirect I/O. The redirection I/O channels are still present for the system to use, so that when system classes print, they will go to the correct context, but tool classes don't have to use this.
A complication of this architecture is that cacheable tools can't be shared by multiple ToolContexts. Therefore, a new ToolCache class has been introduced that hands out Tool instances one to a ToolContext. When the tool finishes, it is returned to the cache ready to be used again. So, in the simple case of just a single thread running the tool, this is as efficient as before, only a single class loader ever needs to be created. However, unique copies of the shell will be loaded for each telnet session. The unload command can be used to unload all instances of a named tool, as before, but the details of unloading are different, and are known only by the ToolFactory class.
Modified the command parser to handle the macro $* in a shell script, which evaluates to all the arguments passed to the script itself. Therefore, a script can pass along all of its arguments to a tool. Take a look at cwie.jxs, which uses this:
setfile -t TEXT -c CWIE $*
9/17/98
Revised the method by which tools exit, and how the cleanup is performed. The Tool class tracks the running commands by storing them in a vector (instead of using a counter, see below). Instead of synchronizing/waiting on the ToolThread, the Tool now waits on the Command object. Tool.finished() is now called by the ToolThread when it exits. Added the new method, Command.exit(), which is called by KernelSecurityManager.checkExit() when a running tool calls System.exit(). Command.exit() also calls Tool.finished(), and then stops the ToolThread explicitly, instead of the current thread, so a multithreaded tool can call System.exit() from any of its threads. Also, if a tool catches ThreadDeath, the finally clause of ToolThread.run() may never run; this is why Command.exit() calls Tool.finished(), to ensure the proper notifications are delivered, and that the command's I/O channels are closed.
If a tool calls System.out.close() or System.err.close(), the request is ignored. The tool sun.tools.native2ascii.Main does this, and it was giving me fits. It would terminate the currently running telnet session, or make the main console inoperable. Command.close() has always closed the redirected I/O channels, so it should be fine to ignore this. KernelSecurityManager.checkExec() now checks for "setio" so that a tool can't change the redirecting I/O channels.
Changed the shell to work a little better with scripts. The shell looks in the current directory for a file called "command.jxs" before looking in the "JScripts" folder. "#" is now recognized (at the beginning of lines only) as a comment character. If a tool in a script calls System.exit() with a non-zero status, the shell will exit (useful for build scripts). Finally, if the property "shell.echo" is set to "1", the shell will echo its commands as they are executed (again for build scripts).
7/16/98
You can now make an alias to a .jar file, that is named after the fully qualified name of the class you want to run. For example, I've included an alias called "sun.tools.javac.Main" as an example.
Thanks go to David Smiley <dsmiley@ccs.neu.edu> for finding a huge memory leak involving ToolContext objects (which are ThreadGroups), which kept accumulating, thus keeping Command and Tool objects alive in the heap forever. Luckily, the fix was simple, calling setDaemon(true) in the ToolContext constructor causes the ToolContext to have its destroy() method called automatically after all of its threads terminate. David found this while implementing a "ps" command, which I am eagerly awaiting.
6/8/98
Fixed a bug when handling I/O redirection, would generate empty arguments in the argument list. Improved error reporting when a file cannot be used for standard input.
6/2/98
The ANTLR command parser is now much more robust. Handles arbitrary concatenation of shell variables in arguments, so you can say ${user.dir}temp/$user if you want.
Added a java command, to make it easier to try out examples in books. To try it out, go to the directory "test", type "javac test.java", and "java test". The java command doesn't let you specify the class path, so classes must either be in the system classes or they must be in the current directory (or a sub-directory determined by the package name). I've been using this to test out Sun's JavaCC parser generator (included in the JavaCC directory for your convenience).
Gosh, next I'll be needing a make utility.
5/21/98
Changed command parsing to use an ANTLR generated parser. This will make it much easier to maintain the command syntax as it grows more complex (I hope). It also gives me a chance to use the environment to work on itself in a very real way. The command language should now be as complete as before plus some new capabilities have been added. A new syntax has been added for concatenating shell variables with other arguments, e.g. echo ${user.dir}/test. It's not quite macro expansion, instead it right associates with the next argument.
5/21/98
Overhauled how I/O redirection works: was using a single global PrintStream through which all output was being buffered. This would needlessly serialize output from tools sending to different files/telnet streams. System.out/err are now replaced by a new RedirectedPrintStream class, which overrides all public PrintStream methods, and delegates to the current context's out/err stream. This should reduce tool thread contention and prevent any buffer sharing. The primordial shell command now has I/O streams that point to the default console streams. This means that all tools running in the shell window inherit a default stream, so Console.Input.getInput(), Console.Output.getOutput() and Console.Error.getOutput() no longer have to test for the null case. This should speeds things up a tad.
May 7, 1998
Telnetd has been improved substantially. It now handles several of the telnet options specified in the various RFCs (echo, status, naws). It features a simple command-line editor, which implements a circular buffer history mechanism (^N goes to the next command in the history, ^P to the previous). It also knows how large a telnet window it is currently in and reflects this in the System properties "window.width" and "window.height".
Simple regular expression matching of file names has been implemented. File expressions of the form "*.java" and "*.?ava" are recognized. All other patterns are assumed to be literal. I'm using the OROMatcher-1.1b2 PERL5 classes to implement the regular expression handling, substituting ".*" for "*" and "." for "?", and quoting all other strings so they will match literally.
April 30, 1998
Unload waits for running tools to finish (unload unload won't work, it will hang). Tool keeps a count of the number of threads that are running the tool (toolCount), and Tool.ToolThread decrements this counter and calls Tool.notifyAll() to inform Tool.close() that all threads are finished.
A simple telnetd tool has been implemented. This starts a thread in the background that opens a ServerSocket on port 23 (telnets). When it receives a new connection, it spawns a thread for that session which simply starts a shell with the socket's input/output streams as stdin. This required changing the system property "line.separator" to "\r\n" to make the telnet protocol happy.
April 22, 1998
Multiple tools can now be stored in a single .jxe file. :JTools:SunTools.jar contains the classes for javac, javadoc, jar, and polardoc (I'll repackage that later). Inside SunTools.jar is a file called "jxe.properties". This names all the tools contained in the .jxe file, and names the classes whose main should be called for that tool. All of the sun tools are aliases to SunTools.jar.
Antlr has also been repackaged this way -- antlr.jxe contains a jxe.properties that maps "antlr" to "antlr.Tool".
This multiple tool per .jxe capability required changes to the ToolFactory. It now maintains a separate cache of open .jxe files, keyed by tool name. It also keeps a map from the tool name to the class name. There is a new command, unload, which calls ToolFactory.unload(), which unloads all tools associated with the .jxe file of the named tool, and closes the .jxe file. This command is experimental and probably shouldn't be used while a tool is running in the background.
The tools javac, javadoc, and jar, have been experimentally marked as cacheable, so subsequent uses of them are much faster. I understand that javac leaks memory, so to release its memory just use unload.
April 21, 1998
This version now replaces the System properties with an instance of KernelProperties, a sub-class of "Properties" that uses the current ToolContext's properties. This allows you to run a time-consuming tool in the background (i.e. jar or javac), without disturbing its concept of the current directory ("user.dir" property). That said, this version also provides pretty complete support for running commands in the background using "&".
To allow certain tools the ability to change the parent's environment, a new interface "ShellCommand" is used to prevent cloning of the environment, so that the command can side-effect the parent's environment. So, cd, set, etc. all implement this interface, which extends the "Cacheable" interface.
For the MacOS only, WaitNextEvent is patched to give time while SIOUX is waiting for keyboard input. This should go away when we have a better console, but works for now. The shell runs at priority 6, foreground commands run at priority 5, and background commands run at priority 4. This keeps jar from just swamping interactive shell performance.
The JIT is now turned on (duh!). All tools are verified before running. Making your tool Cacheable makes everything faster, so I encourage it! All tools are now stored in a global cache rather than one per shell. I'll probably write an unload command to unload cached commands on-demand.
April 20, 1998
This version includes the following new commands, ls (thanks to zellers@basilsoft.com), and sort. As promised, this version supports pipes. For a simple demonstration, try "set | sort".
Changes to support pipes: System.in/out/err are no longer "slammed" when a tool starts executing. This can't work in general because tools running concurrently between pipes would not be able to concurrently use System.in or System.out independently. Instead, System.in/out/err streams delegate to a stream found in the current ToolContext (using ToolContext.getCurrentContext()). Each tool is run in an independent ToolContext which is a subclass of ThreadGroup. With this mechanism in place, it was simple to use java.io.PipedInputStream/PipedOutputStream to do the actual piping. One crucial observation is that when a command finishes writing output to a PipedOutputStream, it must close it, otherwise the reader of the pipe never gets all of the input.
Other changes: Tools folder renamed JTools, scripts folder renamed to JScripts. The shell no longer calls Tool.close() if the command running it hasn't finished yet, intend ToolThread.run() always does this.
April 17, 1998
JShell is a first stab at an open environment for Java-based command line tools. This first incarnation supports simple shell scripts, cd, cat, echo, set, shell and I/O redirection. The goal is to be able to run more complex tools, such as antlr/yacc (try "antlr.Tool"!).
The next version will support pipes.
Why do this? To harness the creative energy of programmers worldwide. I want to be able to run anybody's Java-based tool.