About
Exploring Solution Spaces, Keith Ray's blog on Software development and other topics.
Send comments to:
keithray@mac.com
| For Agile Training, eLearning, or Coaching contact: |
| Industrial Logic, Inc. |
| 866-540-8336 (toll free) |
| 510-540-8336 (Berkeley, California) |
| Links |
| xpminifaq |
| Résumé |
| “Adopting XP” Article 2002 (pdf) |
| “ Refactoring” Article 2006 |
| AYE Conference |
| Lucien W. Dupont |
| Elisabeth Hendrickson |
| Johanna Rothman's Managing Product Development |
| Brian Marick's Exploration Through Example |
| Esther Derby's Insights You Can Use |
| Laurent Bossavit's Incipient(thoughts) |
| Dale Emery's Conversations with Dale |
| Martin Fowler's Bliki |
| Creating Passionate Users |
Archives
Subscribe
RSS Exploring Solution Spaces XML
2003.Dec.31 Wed
The lean principle "See the Whole" is about seeing the whole of the project - not just the whole product, but the whole process. Ideally, Lean companies are lean all the way from Marketing to Human Resources, not just the development process. One way to see what waste there is in the development process is to compress it down to a week or three. If there are delays (wasting time) in going from requirements to design to deployment and testing, it will be more obvious -- a big, slow company that takes weeks for someone to approve something will have to change, or reject going lean.
Similarly to making the whole process smaller to make things clearer, XP recommends working on user-centric stories -- implementing a "whole" feature all the way from bottom layer to top layer (or back end to front end). Many people would rather implement the bottom layer first, and the top layer last, or vice versa, but then you don't have a "whole" feature to test and demonstrate and learn from. Building the software "in layers" risks building a LOT of stuff before being able to demonstrate and test a single user-centric story or feature.
On the Lean Development mailing list, John Roth summarizes the heart of Lean: (my emphasis)
Lean is the opposite of fat, and the only basic principle of lean thinking is to eliminate waste. As the saying goes, all else is commentary, or at least an elaboration on that one basic thought: find a faster, better, cheaper way to add value to the product, and do it at all scales, from the total, end to end process on down to the individual practices used to construct it.
[...] In manufacturing, the first exercise is to squeeze as much in-process inventory out of the value chain as possible. Doing that will force a huge number of other beneficial changes.
In Lean Software Development, the first exercise is to go to fixed length iterations, where you have a production quality deployable piece of software at the end of each iteration. Doing that one thing will force a huge number of other changes in the process. I can't emphasize those two criteria too much: production quality and deployable. If you need further testing, signoffs or work from some other department after the iteration ends, you don't have it.[...]
Let's look at the seven core principles of lean, and compare them to Extreme Programming practices:
1. Eliminate Waste. Waste is anything that doesn't contribute to adding value to the product. One-hour status meetings waste time, so XP has 15-minute daily standup meetings instead. Excessive documentation and planning wastes time and energy, so XP recommends creating just enough documentation and planning, at appropriate levels of detail. Debugging is a waste, so XP does test-driven-development, which reduces debugging time and reduces time spent manually testing. Slow communication to/from the Customer/Domain Expert is a waste, so XP recommends having the Customer on-site.
2. Amplify Learning. Creating software is a 'learning' process. Typically developers do this learning alone, and that knowledge doesn't spread around to other developers very quickly. XP does coding in pairs to spread knowledge around faster. (Code reviews can also do this, but with higher overhead.) The Customer/Domain Expert is learning, too, as he sees his ideas implemented and developers ask him for details needed for acceptance tests. XP teams often do Retrospectives each iteration to bring learning to the whole team. Having all the developers and Customer working in a "war-room" also helps the whole team learn together.
3. Decide as Late as Possible. In Release Planning, stories are "promises for future communication" - the Customer/Domain Expert describes each feature/story in just enough detail for the engineers to provide a rough estimate. In Iteration Planning, the Customer describes the story in more detail, so the engineers can break it down into a task-list and/or rough design. The Customer/Tester specifies acceptance tests, which test the "what" not the "how", and then the programmers do test-driven-development where "what" tests and "how" coding are interleaved in a short cycle.
4. Deliver as Fast as Possible. "Simple Code" does everything necessary for the current requirements, but not more, so the team can ship it quickly. Keeping the quality high via test-driven-development avoids coding on an unstable foundation, wasting time debugging. Having the Customer/Domain Expert within range of spoken questions allows getting requirements into code in a minimum of time.
5. Empower the Team. XP encourages all team members to design, test, and code, instead of restricting those activities to a few. It encourages people to sign up for tasks, instead of assigning them to those who may not be as willing or as able. Anyone can improve code at any time, as long as the tests continue to run - allowing new learnings to be reflected in the code/design.
6. Build Integrity In. With short iterations where something isn't "done" until it passes (automated and manual) acceptance testing, and test-driven-development insuring that every line of code is tested, the XP team always builds on top of working software. In every iteration, all automated testing is repeated frequently, detecting as soon as possible any "breakage" of code. Any bugs found by manual testing or automated tests failing are not just fixed, but also force the creation of new automated tests.
7. See the Whole. Before beginning the first iteration, the XP team has a release plan. The Customer/Domain Expert has created that plan with the development team, and together they maintain and update the plan throughout the project, measuring and charting progress. Code is only implemented according to the current and previous requirements, so there is no work wasted on a requirement that may be dropped... however, engineers are not going to write code that will be difficult to adapt to that future requirement in that future comes around. To use the (not very good bridge-building analogy): the draw-bridge may not have motors in this iteration, but the builders are not going to build the bridge out of stone in this iteration, ignoring the need for mobility in the future.
By the way, I hate the "bridge-building" analogy, because writing software isn't building, it is designing. Running the compiler and linker associated "build" scripts, etc., are the "building". If building a bridge were as quick and easy as running a compiler and linker, we'd build bridges multiple times in order to test them. Any comparison of software development to bridges should be to designing bridges - an iterative, learning activity where multiple solutions are designed and tested (via mathematical/computer models or physical models) until all the requirements are met.
2003.Dec.29 MonYour team is handed a huge pile of legacy code, to enhance or maintain. What are the first things you should do?
First, get it under source-code-control, if it isn't already.
Second, build it and run it. If it has tests, build and run those.
Third... review the code. Since your team is unfamiliar with the code, they will be reading it all anyway, just to figure out what it does and what they have to modify. It would be wise to organize that effort. If you can get the original writers of the code involved in the reviews, that will help, as long your company culture is mature enough to do reviews without blaming, put-downs, or criticisms that sound like attacks.
For agility, don't print out sources and hold whole-team reviews away from the computers. Have the team do the reviews in pairs and trios at computers that can build and run the source code. The reviewers should draw rough diagrams on paper to help document their understanding of the code, and have the reviewers present their findings to the whole team in verbal summaries.
Unlike reviews written up in the computer science literature, these are not reviews to find bugs, though no doubt with most legacy code, you will find lots of bugs. Don't fix them yet! The intention is to try to understand how the features are implemented, where the data flows (inputs, modification/processing, outputs), and what classes or functions are responsible for various things. When you find a possible bug, add a comment with an easily searchable tag, and/or add an assertion. Careful renaming of functions, files, and classes can be done here, when you identify names that don't explain the intentions of the code. If there are no layers, moving files/classes/modules into "Domain", "Infrastructure", "UI" and so on can help with understanding their roles.
You will probably identify lots of redundancies - several classes, modules, or functions that do the same thing. Move those things closer to each other in the source code directories and files, but don't yet do any refactoring that changes the function-bodies or class structures. You and your team should identify which functions/classes/modules would be the "preferred" ones to use for new code, and which ones you will avoid using (and eventually refactor away).
Identify and remove dead code, but be sure it's actually dead - if you're using a static language like C or C++, having a successful compile, link, run usually indicates that you didn't remove any live code.
You want a stable foundation on which to build your modifications, so now write unit tests that detect the bugs you identified in the reviews. Work on one bug at a time, with minimal refactoring to make the code testable. Fix the bug, and test the entire application to make sure the fix doesn't have any side-effects. Fixing uninitialized variables can create unexpected side-effects because the code may be relying on the (coincidental) value that the uninitialized variable had.
If you have requirements documents or some other description of the desired features, review those and the running application with your Customer (if you have one) to find out which features are actually present and working as desired, and what new features are desired.
Use XP-style story cards and release/iteration planning for new features (or features to be reworked) and if possible, write automated acceptance tests and unit tests when implementing the new features. If you don't have QA people to write the acceptance tests, write them yourselves.
All this advice supposes that you have some time to do the right things. If you don't have time to the right things, plan to spend time fixing bugs and stepping through the debugger trying figure out what's going on -- which will probably take longer than doing the reviews and bug-fixing up-front. Measure your progress against estimates - sometimes it's better to throw away bad code than to try to fix it, but that decision must rest on measured costs. Throwing away code doesn't have to an all-or-nothing situation. Replacing home-grown string classes with a standard string class, (ditto for container classes) and so on can increase the maintainability of the code without significantly affecting the domain-related design.
2003.Dec.28 SunFor entertainment - drama/action/comedy, science fiction, and "old west" bad guys and much badder guys, in a package that sometimes turns clichés upside down (or at least provides a fresh vision), check out Firefly - Joss Whedon's too-short series that aired on Fox -- a network that couldn't understand it, promoted it poorly, and didn't give it a chance to find an audience. The DVDs have episodes that were never aired, and several episodes have commentary audio tracks. Excellent cast. Excellent writing, set design, costume design, very good special effects... Unfortunately for us (and Joss and crew), much of the back-story of these interesting and sometimes mysterious characters remains unexplored, though the commentary hints that there might be some kind of revival/continuation for Firefly - perhaps a movie, a revived TV show, or something in another medium - Joss has been involved in comic books, and perhaps could be involved in writing novels... I haven't check the rumor sites yet, so you may know more than me.
For the spirit: my wife and I enjoyed Awakening the Buddhist Heart by Lama Surya Das on tape. The author has a Long Island accent - I've heard that his mother calls him "the Deli Lama". We also read the book Awakening the Buddha Within by the same author. The "Heart" book is about becoming compassionate, the Buddha book is a clear expression of what Buddhism is: wisdom, awareness, love, compassion, balance.
Jack Kornfield's book After the Ectasy, the Laundry talks about the often difficult spiritual path of several traditions: Christian, Jewish, Buddhism, Zen, Sufi, and so on.
Trivia fact: Ron Glass, who plays a Christian preacher in Firefly, is a Buddhist.
2003.Dec.24 WedI occasionally see advice to wrap 'legacy code' behind a facade or wrapper, putting new, tested, code "in front" of the wrapper, and leaving the old untested legacy code behind the wrapper. The recommendation is to "gradually" replace functionality behind the wrapper with new code.
I'm sure this comes from good intentions, but I see this kind of thing in legacy code itself -- layers and layers of code written by different people over the years, large portions of it being 'dead' code, and the one thing all these layers have in common is that their authors avoided changing any of the previous layers of code.
The Lava Flow antipattern describes this kind of layering -- good for building mountains and the Hawaiian Islands in real life, bad for building mountains of code in software.
It seems to me that "wrapping" should be a last resort, and continuing to engage in the legacy code, simplifying it, removing dead code and unnecessary layers, adding tests and refactoring, should be preferred.
I have to add some disagreements to the Lava Flow write-up that I linked to above. That author writes "There is only one sure-fire way to prevent the Lava Flow AntiPattern. That is by ensuring that sound architecture precedes production code development." There is more than one way to skin a cat. The other, less phasist way, is to apply Merciless Refactoring as you evolve the architecture as reflected in the production code, so that you have a good architecture at all times, even if it isn't the architecture you had planned to have three months ago, before you understood the domain as well as you do today.
Another point "As suspected dead code is eliminated bugs will be introduced. When this happens, resist the urge to immediately fix the symptoms..." I would say, "undo". Only remove dead code, not nearly dead code. Write unit tests and acceptance tests to detect bugs that may show up from a refactoring, and undo your last code change when the tests detect an undesired change.
2003.Dec.23 TueRegarding Value objects and Entity objects mentioned in Eric Evan's Domain Driven Design...
I think of value objects (mutable and immutable), entity objects, and "Other" including "stateless" objects...
For example, a "configuration object" may be a platform-independent wrapper for the Registry API on Windows, the User Defaults API on MacOS X, and reading/writing ~/.somefile on unix/linux. None of the platform-dependent versions of this object necessarily need to have their own state, though caching the state might be useful if profiling determines that there's too much overhead in using the native APIs all the time. I like the stateless object for this particular case, because we can have many of them, possibly in multiple threads, without having to synchronize them -- the APIs we're calling are already synchronized (at least on Windows and MacOS X). Being a normal object, we can pass around Mock versions in our tests.
Note that a generic preferences object would not be in the Domain layer - it would be part of the infrastructure layer.
Eric does mention Policy objects as stateless objects implementing the Strategy pattern in examples of the freight-routing model; these would neither be Value objects nor Entity objects, as far as I can tell, but they would reside in the Domain layer.
SWIG allows Java code (etc.) call C/C++ code, not the other way 'round. That's the opposite of JACE, which lets C/C++ call Java. Maybe SWIG can be extended...
"Doctor, it hurts when I do this," says the patient. "Don't do it," replies the doctor. Here's a language translation task that makes me say "double-ouch" - some guy in India has to translate legacy (undocumented, untested) C++ into COBOL!
So much of language trends in the software industry come from marketing and price. Objective-C is a language that easily allows mixing Object Oriented programming with C. Back in the day when the major languages were C, Pascal, FORTRAN, BASIC, and COBOL, and they were interoperable, Objective-C continued to be interoperable. Unfortunately Objective-C was initially rather expensive, and had the reputation for being slow, because of one or two poor implementations. At the same time, C++ was gaining ground, because it was essentially being given away for free to universities and cheaply to companies that licensed CFront from AT&T; it was often used as a "better C" and had a (not always deserved) reputation for high performance. By the time that Objective-C had a free (gcc) implementation, the momentum for commercial compiler vendors and GUI programming environments (other than NeXTStep/Cocoa) was firmly in the C++ camp.
Xerox didn't market Smalltalk well, and later the Smalltalk vendors had a meltdown from mergers and loss of focus, and many people decided the language was dead. Smalltalk also didn't encourage interoperability - mixing C and Smalltalk is difficult in many Smalltalk environments. (Dolphin Smalltalk seems to be one of the exceptions, though their use of the phrase "Pure Object Oriented Programming" may be scaring away potential customers.) Now there are some free versions of Smalltalk, and several good commercial versions, but no major company (like Sun, IBM, or Microsoft) is hyping Smalltalk, even though IBM has a commercial Smalltalk implementation, and Sun and Microsoft are hawking virtual-machine-based languages that owe a large debt to Smalltalk virtual-machine research.
Like Smalltalk of old, Java makes it difficult to interoperate with C. I've used JNI fairly extensively for calling Java from C and vice-versa, and it is incredibly painful to use. Calling Java from C seems to be particularly problematic on MacOS 9 - all the threads that are supposed to be running inside the JVM are only active during the course of the call into the JVM, and maybe not even then, and one of those threads is the garbage-collection thread.
One tool for making JNI work easier is JACE, by generating (and using) C++ template-based code. Unfortunately, the most recent version of JACE is using templates that I can't compile properly with CodeWarrior -- when I told the author this, he told me to get a better compiler! I tried to suggest to the author that instead of relying on templates so much he could generate code without templates, because templates essentially are code-generators after all. He said something about wanting type-safety, which completely misses the point -- if you specify types as part of the code-generation process, then you can have exactly as much type-safety as you would get from templates. Given the arrogance of the author ("get a better compiler!"), I am not going to link to JACE, and would recommend that you use SWIG instead, for your JNI and other language-interoperability tasks. Once you've gotten SWIG as an executable tool, it generates C code, compatible with a large number of compilers.
2003.Dec.22 MonSome time ago, Todd Blanchard wrote:
Some time before C++ started to proliferate, the developers of most operating systems defined a standard binary object file format that all compilers would generate. It was common to use FORTRAN math libraries from C, Pascal, or COBOL programs. You could mix or match to your heart's content. The object formats were the same so the linker didn't care.
The introduction of C++, in a misguided effort to make use of existing linker technology, introduced a technique called name mangling and essentially broke the guarantee that object files were interchangable regardless of which compiler were used. C++ code compiled with different compilers couldn't even interoperate.
The end result was projects now had to select a single language for the entire project and stay with it. The cost of calling out to code written in other languages went up. Thus began the porting wars.
Companies now spend money to port significant portions of their existing software assets to the current darling in the language world. Even conservative aerospace companies have risked introducing new defects by porting reliable and fast math libraries written in FORTRAN to C++ just because "everything is going towards C++".
Suckers.
They did it again with Java for the same reasons. I expect they'll do it yet again with C# and .NET.
The real fact of the matter is that there has been no real progress in the software industry in something like 10 years. In fact we've lost some interoperability capabilities among programming languages.
As a side effect of changing the rules every year, experience is marginalized and hiring low cost unskilled programmers labor begins to look attractive. After all, even the old timers are going to have to learn everything all over. Thus, the endless march of new stuff for the sake of being new stuff nullifies the value of experience and hinders the development of masters in the field.
Fortunately some languages are easier to interoperate with than others:
Bruce Eckel: "Zope is the enterprise application server I use for my website. Zope is virtually all Python, but includes some C code, because they profiled and optimized Zope so it would run faster..."
Questions for Designing a Language
Questions to ask yourself when designing a programming language, via reflectivesurface:
- What need are you trying to fill?
- What's the metaphor? (OO, logic, imperative, functional, etc.)
- Is high performance an issue?
- Is high programmer productivity an issue?
- How portable across platforms do you want the language to be?
- and so on...
There are over 7000 programming languages, so think twice before creating a new one. If you're looking at creating a way to script your application there are several alternatives to creating a new language:
Python can be "embedded" in an application, and is free and cross-platform.
On Windows, you can export functionality in your application via COM, and thus any COM-compatible scripting language can then script your application -- VBScript, VisualBasic, JScript, Python, Perl, and so on.
On MacOS X, programming in Cocoa or PowerPlant make it fairly easy to make your application scriptable via AppleScript, and via AppleEvents (the supporting layer for AppleScript) your app can be scripted via Python and other languages.
Because Cocoa/Objective-C has bridges to/from Java, Ruby, Python, and FScript, your MacOS X application implemented with Cocoa can be scripted or interfaced with several other languages.
Java, Python, Squeak Smalltalk, and so on, are very 'portable' languages, but they are implemented in C, so C is even more portable (though portable GUI frameworks via C/C++ are problematic -- sometimes more trouble than they are work). Smalltalk and many other languages which have environments "built-in" actually restrict portability -- I can't write a TWAIN plugin in a portable dialect of Smalltalk, for example.
Unfortunately, popularity of a language does not correspond to fitness of purpose. Many people program in languages that are optimized for machine-performance rather than programmer-performance. CPUs are 100's of times faster now, compared to when those languages were designed, and for the most part these days, the CPU is idle most of the time.
Rather than trying to find cheaper programmers, businesses should be looking at a less "expensive" programming languages/environments/methods: ones that provide power and speed to the programmers who are the most expensive "part" of the process.
The obstacle is the path. -- Zen Proverb
2003.Dec.21 Sun"What am I resisting doing, right now?" From reading Mark Forster's weblog, that seems to be the key to his Resistance Principle. The other half of the equation is, after asking yourself this question, and coming up with an answer, actually doing the thing I know I should be doing. We usually know what we need to be doing "now" to fulfill our desired life-plan (or just get the house clean), and we don't need to buy a DayTimer™ and make a list to figure out that one thing.
At work, I can get myself (and others) to do the things that need doing, even though sometimes it's hard. At home, sometimes I know what I'm resisting doing, and yet I still watch TV or whatever. I can have an excuse like "I'm tired", but I know it's an excuse. I'm not convinced that I need to search for deeper reasons -- for some resisted tasks, I'm comfortable enough with things the way they are now; for others, there's some discomfort involved, or I don't perceive enough benefit for the effort.
Mark has a few tricks to help yourself get started, like "I'll just open the file" (or get out whatever materials you need to do the task)... once it's easy to get started on the task, you often do the task. This trick sounds like something Theodore Sturgeon once did to get over his frequent bouts of writer's block: he put a piece of paper in the typewriter, and would write something, anything until the paper was filled. He ended up writing a story this way, and he said about this experience: "it was so effective, that I never did it again."
I was a bit worried that Mark Forster had died or gotten sick (he's not a young man); he had been putting something in his blog every day, but nothing had appeared for over a week! Finally he wrote this:
Thanks to everyone who has written in asking where I have been for the last ten days. The answer is that I have been experimenting to see what happened if I stopped using the Resistance Principle. [...].
Using the Resistance Principle, I had survived the difficult period after breaking my hip at the beginning of November, built up my web traffic to be the highest rated coaching website on the net, attracted many new clients, and produced a daily weblog and a weekly newsletter without fail. All without any form of to do list or goal setting.
As soon as I gave up using the Resistance Principle all this went into reverse!
By the way, I have a place to put blogging ideas, a "to blog" file, but so far, I have just about never written a blog entry based on something in my "to blog" file. Maybe putting something in the "to blog" file gives me just enough feeling of "having taken care of it" that I never take care of it for real.
I'm going to blog every day, and if I can't think of something to blog about, I will dig into the "to blog" file. And if I see something I want to put into the "to blog" file, I will instead blog about it immediately, instead. So you may be see me blog more than once a day.
Grumble. There are about three or four non-blogging writing tasks I am resisting doing... maybe I should just open the file, right now...
2003.Dec.20 SatChristian Sepulveda observed the following:
I was working with a team that had been gradually adopting agile practices, but the developers were resistant to practicing Test Driven Development. At one point, I decided to start tracking how much time was spent compiling code, running the application/debugger, re-establishing the context in the application, observing a result and then closing the application. The developer would think about what he observed, make code edits, start the debugger and repeat the cycle. I observed that it took four minutes on average to run the cycle of compile, launch debugger, find context, etc. This would be done an average of eight times per hour; the remaining hour was spent thinking and making edits.
4 minutes compiling and debugging * 8 times/hour = 32 minutes/hour. Imaging how much worse this is in C/C++ environments where just compiling and linking costs 5 minutes or more before getting into the debugger.
One half of each day was completely wasted. The developer couldn't really doing anything productive during the four-minute debug cycle because he had to navigate to the correct place in the application and execute the right steps so he could observe the results of the change he just made; he was forced into tedium. In other words, half the development budget was being spent on performing repeated, monotonous tasks. This isn't a very effective use of resources or talent.
[...]Faced with such results, the team in my example did adopt TDD. Their feedback loop was reduced to thirty seconds. [...] The same team could theoretically accomplish the same work in one hour each day instead of eight hours. [...]
[...] I have observed, with the specific team, that it isn't quite 8x acceleration, but it is greater than 2. Isn't twice the speed worth the effort to learn TDD?
Add in pair programming, which provides instant code review, finding those problems which are obvious to a person who didn't write the buggy piece of code, and XP-style development really rocks! The pair partner helps you find problems before you run your code, (or helps you diagnose why a test didn't perform as expected), so again you spend less time in the debugger.
2003.Dec.18 ThuCheck it out: http://www.nasa.gov/externalflash/m2k4/trailers.html A lot of sizzle, but where's the beef?
2003.Dec.17 WedBecause I keep reminding people that Smalltalk (and C) begat Objective-C, and Objective-C (and C++) begat Java, James Bullock let me know of a good article on reasons to love and hate Objective C. Check it out: http://rentzsch.com/papers/loveHateObjC.
All of the writer's points are good, except #14 "id should be id*" is really a quibble, not worthy of "hate". He mentions PyObjC, which allows writing Cocoa apps in Python, and don't forget RubyCocoa as well.
On hate #1, "Method-call Syntax", I'd like to see Objective-C's brackets become optional (I think it could be done, and still have a simpler parser than C++), but I think more people should be forced to love Smalltalk, and so of course they will then love Objective-C by extension. ;-)
In regards to Hate #8, "Hard to Write Good Getters/Setters," part of the problem is Cocoa's semi-automated memory management - hopefully someday solved by using real garbage collection. The other part of this problem happens in any language where objects variables are only references (in other words, just about any language other than C++, which can copy objects around on the stack "by value"). You have to decide if you want to share references to your class's private members with the outside world, or make copies to give to the outside world. At least Cocoa/Objective-C has an immutable string class, so you can return references to member string variables without risk, like Java. More programmers should be writing immutable value classes to avoid copying objects in getters and setters.
This getter/setter issue reminds me of a the distinction between "value objects" and "entity objects" described in the book Domain-Driven Design by Eric Evans. "Entity" objects have the responsibility of maintaining an identity representing real-world thing, like a person. A person's age changes, weight changes, names changes; they change jobs, they change addresses, and so on, but their "identity" -- as far some software is concerned -- doesn't change.
Few programmers make the distinction between value and entity objects, and failing to make this distinction can create problematic OO designs/implementations. Unfortunately, no programming language is helping us here. It would be great if we could tag a class as an "entity," and have the compiler/run-time system let us know if we ever try to create non-identical duplicates of what is supposed to be a unique object. Tagging a class as "value" and "immutable value" would also be nice, and better than C++'s "const" syntax, which only tags variables and methods, not classes and objects.
2003.Dec.13 SatSo many groups doing software development are just groups of soloists, even though we call them "teams". James Bullock made a comment on SHAPE on creating and sustaining teams of software developers:
2003.Dec.12 Fri- To make a great project team, it helps if making a great project team is part of your agenda. Better if this is part of everyone's agenda.
- To sustain a great team, it helps if sustaining the team is part of your agenda. Better if this is part of everyone's agenda.
Reflecting on Variable Initialization
I dreamed I was in a high-school class, studying Smalltalk, and there was only one other non-high-school-age student in the class. My dream-interpretation is that "attending a class" represents that I am learning, and the lack of other people my age in the class represents how few people continue to learn after they leave school, as evidenced by the code I still see people writing.
There are "experienced" programmers who still declare local variables without initializing them. After spending hours (or days) tracking down a bug due to an uninitialized variable, a programmer who reflects on his or her work should realize the cost of always initializing variables is very small compared to the cost of debugging.
Variables that are not automatically initialized to 0 or null are a symptom of a bad programming language. Maybe they made sense in the old days of 1 Megahertz CPUs and languages that didn't allow combining declaration and initialization, but not today.
C and C++ default to not initializing variables. The exceptions are variables declared "static" -- those are initialized to zeroes -- and reference variables in C++. A good general tactic in using C and C++ in particular is to pick the "non-default" behavior whenever there is a choice -- it is almost always safer than the default behavior. (I could give plenty of examples, uninitialized variables are just the tip of the iceberg.)
Java and Objective-C also do not initialize local (stack) variables, though they do initialize member variables to 0 or null, so this lesson applies to those languages too. Smalltalk and probably several other languages do initialize local variables as well as member variables.
Here is a little C++ program to illustrate non-initialized variables. Try translating this to your favorite language and see what happens.
#include#include using namespace std; const int kTextSize = 27; struct BlockTest { int ival; char text[kTextSize]; double dval; }; void PrintBlock( const BlockTest& block ) { cout << "ival = " << block.ival << '\n'; for ( int i = 0; i < kTextSize; i++ ) { cout << " text[" << i << "]= '" << block.text[i] << "'" << '\n'; } cout << "dval = " << block.dval << '\n'; cout << endl; } void initerFunc( BlockTest& blockOut ) { BlockTest local = { 1234, "abcdefghijklmnopqrstuvwxyz", 1234.56 }; blockOut = local; } void nonIniterFunc( BlockTest& blockOut ) { BlockTest local; blockOut = local; } int main (int argc, const char * argv[]) { BlockTest mainBlock; cout << "main's uninitialized block" << endl; PrintBlock( mainBlock ); nonIniterFunc( mainBlock ); cout << "nonIniterFunc's uninitialized block" << endl; PrintBlock( mainBlock ); initerFunc( mainBlock ); cout << "initerFunc's initialized block" << endl; PrintBlock( mainBlock ); nonIniterFunc( mainBlock ); cout << "nonIniterFunc's uninitialized block" << endl; PrintBlock( mainBlock ); return 0; }
And here's the output as generated in ProjectBuilder (gcc) on MacOS X 10.2.x:
main's uninitialized block ival = 8 text[0]= ' ' text[1]= ' ' text[2]= ' ' text[3]= ' ' text[4]= ' ' text[5]= ' ' text[6]= ' ' text[7]= '' text[8]= ' ' text[9]= ' ' text[10]= ' ' text[11]= '' text[12]= ' ' text[13]= ' ' text[14]= ' ' text[15]= '' text[16]= ' ' text[17]= ' ' text[18]= ' ' text[19]= '' text[20]= '\277' text[21]= '\377' text[22]= '\376' text[23]= '$' text[24]= '\277' text[25]= '\377' text[26]= '\376' dval = -3.46217e-232 nonIniterFunc's uninitialized block ival = 1 text[0]= '\277' text[1]= '\377' text[2]= '\376' text[3]= ',' text[4]= '\277' text[5]= '\377' text[6]= '\376' text[7]= '$' text[8]= '\217' text[9]= '\341' text[10]= '3' text[11]= 'X' text[12]= ' ' text[13]= ' ' text[14]= ' ' text[15]= '' text[16]= ' ' text[17]= '' text[18]= '' text[19]= ' ' text[20]= '\217' text[21]= '\341' text[22]= '3' text[23]= '\350' text[24]= ' ' text[25]= ' ' text[26]= ' ' dval = 5.7294e-313 initerFunc's initialized block ival = 1234 text[0]= 'a' text[1]= 'b' text[2]= 'c' text[3]= 'd' text[4]= 'e' text[5]= 'f' text[6]= 'g' text[7]= 'h' text[8]= 'i' text[9]= 'j' text[10]= 'k' text[11]= 'l' text[12]= 'm' text[13]= 'n' text[14]= 'o' text[15]= 'p' text[16]= 'q' text[17]= 'r' text[18]= 's' text[19]= 't' text[20]= 'u' text[21]= 'v' text[22]= 'w' text[23]= 'x' text[24]= 'y' text[25]= 'z' text[26]= ' ' dval = 1234.56 nonIniterFunc's uninitialized block ival = 1234 text[0]= 'a' text[1]= 'b' text[2]= 'c' text[3]= 'd' text[4]= 'e' text[5]= 'f' text[6]= 'g' text[7]= 'h' text[8]= 'i' text[9]= 'j' text[10]= 'k' text[11]= 'l' text[12]= ' ' text[13]= ' ' text[14]= ' ' text[15]= '' text[16]= ' ' text[17]= '' text[18]= '' text[19]= ' ' text[20]= 'u' text[21]= 'v' text[22]= 'w' text[23]= 'x' text[24]= 'y' text[25]= 'z' text[26]= ' ' dval = 5.7294e-313 initTest has exited with status 0.
I tried to do "Replace All" in ProjectBuilder (replacing each tab with four spaces, preparing to paste the program text into my web-page) and got this error message after all my tabs were removed (and not replaced by spaces):
Uncaught Exception:
*** NSRunStorage, _NSBlockNumberForIndex(): index (1096) beyond array
bounds (1096)
Stack Backtrace:
The stack backtrace has been logged to the console.
(button) Quit
(button) Continue
And this is visible in the Console application:
Stack:
0 0x97e547e8 +[NSException raise:format:]
1 0x9307937c __NSBlockNumberForIndex
2 0x93086e90 -[NSLayoutManager textContainerForGlyphAtIndex:
effectiveRange:]
3 0x930c7304 -[NSTextView(NSPrivate) _scrollRangeToVisible:forceCenter:]
4 0x914c9828 -[NSTextView(PBXTextViewFindExtensions)
replaceAllStringsMatchingString:withString:inSelection:
ignoreCase:matchStyle:]
5 0x914c8e6c -[PBXSimpleFinder _replaceAll]
6 0x930f9ca8 -[NSApplication sendAction:to:from:]
7 0x9316ec58 -[NSMenu performActionForItemAtIndex:]
8 0x931af2f8 -[NSCarbonMenuImpl
performActionWithHighlightingForItemAtIndex:]
9 0x930ec72c __NSHandleCarbonMenuEvent
10 0x9308e228 __DPSNextEvent
11 0x930a0154 -[NSApplication
nextEventMatchingMask:untilDate:inMode:dequeue:]
12 0x930b1d84 -[NSApplication run]
13 0x9315fc54 _NSApplicationMain
14 0x00013a94
15 0x00013914
Continue actually works, so the 'uncaught' exception was actually caught safely, and one would hope that this output would help Apple's engineers prevent this problem from happening again.
The stack output is Objective-C / Cocoa: "+" indicates a 'class method', "-" indicates an 'instance method', "__" indicates a function not in a class. The first piece of text in the [] is the class name, the second piece of text is the method name (and yes, colons are part of the method names in Objective-C). The names in parenthesis indicate extensions/replacements to existing classes without subclassing - a powerful feature not found in many environments. Almost all the methods/functions on the stack are part of the Cocoa framework, the only items specific to Project Builder are the class/method PBXSimpleFinder _replaceAll and the 'PBXTextViewFindExtensions' method replaceAllStringsMatchingString....
2003.Dec.10 WedWinter is "Glory Season" in David Brin's science fiction novel of the same name. And winter is a good time to read or re-read this under-appreciated page-turner. I like most of Brin's science fiction, and this is one of my favorites. Primarily, it is follows the adventure of a young person trying to find her way in her society from a disadvantaged social position. It also explores a society that is female-dominated, voluntarily low-tech, and favors clone-children over non-clone children ("vars" - varients).
The protagonist is one of those non-clone children, ejected from her the clan of her birth, so she views her world partly from an outsider perspective -- but not one as disenfranchised as the males on her world. Men in the society portrayed in this book are viewed as too emotional and too immature to run businesses, or be involved in politics or science or sports. For some reason, my wife likes this book a lot, even though she doesn't in general enjoy science fiction as much as I do. ;-) One character in the book is an unprecedented visitor from off-world, who tends to be under-estimated because he's male (and he gets kidnapped, part of the adventures we follow).
The "odd" aspects of the world portrayed in this book are the result of genetic engineering as well as social engineering. Sexual arousal is triggered by seasonal environmental clues - men are aroused by the summer auroras, and women by the winter frosts (glory frost). The world is oddly non-sexual, compared to ours. Women can conceive clone-children in winter, and varient-children in summer. The participation of men is required in both cases, though in winter the men don't make a genetic contribution.
Like other Brin books, there are many layers and currents and ideas to explore. Enjoy!
2003.Dec.08 MonI like these quotations from Mark Forster's blog:
"We judge ourselves by what we feel we are capable of doing while others judge us by what we have already done." (Longfellow)
"People have opinions on everything -- especially things they know nothing about." (William Bonner)
"The problem is not that there are problems. The problem is expecting otherwise and thinking that having problems is a problem." (Theodore Rubin)
"After I'm dead I'd rather have people ask why I have no monument than why I have one." (Cato the Elder)
"People don’t want their lives fixed. Nobody wants their problems solved. Their dramas. Their distractions. Their stories resolved. Their messes cleaned up. Because what would they have left? Just the big scary unknown." (Chuck Palahniuk's Survivor)
"No man ever had a point of pride that wasn't injurious to him." (Emerson)
A reader of Mark Forster's book "How to Make Your Dreams Come True", writes: "One surprising outcome of using your techniques . . . was how I became aware of other people's negative thought cycles and how they trapped their life experience into a grey, miserable and claustrophobic frame. I could see the patterns, always focusing on what was wrong or bad or what they hadn’t done, and I realised that I used to be like that. I also recognised how other people's attitudes affected my thoughts as well. That knowledge was worth its weight in gold alone."
"Many people are given good advice but the ones who profit from it are those who are prepared to take action." Karl George.
"Today there is a failure of what I call authoritarian leadership . . . Rather than look for new leaders, I believe we need to find our own inner leadership. When more people begin to discover the deeper part of themselves, then the world will be a better place." (Sir John Whitmore)
"Don't say you don't have enough time. You have exactly the same number of hours per day that were given to Helen Keller, Pasteur, Michaelangelo, Mother Teresea, Leonardo da Vinci, Thomas Jefferson, and Albert Einstein." (H. Jackson Brown, Jr.)
Mark Forster: "The expression 'time management' is in itself an example of the wrong mental attitude which prevents people from getting things done. It's as if a fish in the ocean were to say that it had a problem with 'water management'. Time is simply the medium in which we exist.
"It is said that power corrupts, but actually it's more true that power attracts the corruptible. The sane are usually attracted by other things than power." (David Brin)
"The problem is never that we don't have enough time. It is always that we have too much to do, and the answer is always to do less : to only do the things that matter." (Morendil)
"You don't get in life what you want; you get in life what you are." (Les Brown)
"Realize that now, in this moment of time, you are creating. You are creating your next moment. That is what's real." (Sara Paddison)
"Every difficulty slurred over will be a ghost to disturb your repose later on." (Frederick Chopin)
"If you want to build a ship, don't herd people together to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea." (Antoine de Saint-Exupery)
"If your success is not on your own terms, if it looks good to the world but does not feel good in your heart, it is not success at all." (Anna Quindlen)
"Putting off an easy thing makes it hard, and putting off a hard one makes it impossible." (George H. Lonmer)
"Mediocrity is the parent of regret." (R. Thompson)
"What is boredom but disengagement with reality? It is the feeling of wanting to be somewhere else at another time. There is no destination specified. Yet reality is only here and only now." Mark Forster
"Each problem that I solved became a rule which served afterwards to solve other problems." (Descartes)
"What I am doing is to ask myself the very simple question: "What am I resisting?". Then answer the question, do whatever it is, and ask the question again." Mark Forster
"A man may fail many times but he isn't a failure until he begins to blame somebody else." (J. Paul Getty)
"I know quite certainly that I myself have no special talent; curiosity, obsession and dogged endurance, combined with self-criticism have brought me to my ideas." (Albert Einstein)
"Life may have no meaning. Or even worse, it may have a meaning of which I disapprove." (Ashleigh Brilliant)
"I was interested in your comments about 'overlearning' and think this is a valuable concept and one which it is easy to overlook. I was taking part in a singing masterclass some years ago, and the distinguished singer taking the class said 'Amateurs practise until they get it right, professionals practise until they can't get it wrong' Those words have stuck with me and I find they apply in many ways - language learning as well as music. Of course, it is important to know when perfectionism is not helpful or not necessary, but my experience is that the only way to get 100% is to aim even higher." One of Mark Forster's readers.
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man." (George Bernard Shaw)
"Nothing is easier than being busy - and nothing more difficult than being effective." (R. Alec MacKenzie)
"Ninety percent of the politicians give the other ten percent a bad reputation." (Henry Kissenger)
"Those who are too smart to engage in politics are punished by being governed by those who are dumber." (Plato)
"If a million people believe a foolish thing, it is still a foolish thing." (Anatole France)
"It's not worth an intelligent man's time to be in the majority. By definition, there are already enough people to do that". (G. H. Hardy)
"Discovery consists of seeing what everybody has seen and thinking what nobody has thought." (Albert Gyorgyi)
"We can be knowledgeable with other men's knowledge, but we cannot be wise with other men's wisdom." (Montaigne)
"Only the mediocre are always at their best." (Jean Giraudoux)
"One way to create momentum is by creating a series of small successes. So pick small goals, goals that are easy to create and then create them. Think of each success you create as a warm up exercise for bigger creations. Just like musicians practice and athletes practice, practice creating. All of the things you create will give you even more skill and more momentum." (Robert Fritz)
2003.Dec.06 SatSometimes someone says that they can't do an iterative and incremental software development process, like Extreme Programming with its "frequent releases" practice, because the product is "all or nothing". They can't imagine a "partial" product being useful. This is, of course, a failure of imagination. A thread on this topic is on the XP mailing list.
Jeff Grigg made the analogy to building a fence around the back yard of a house (separating our yard from three neighbors). We estimate that it would take two days to build the entire fence, but we don't say we can't do it because we can't build it one day - nor do we need to redefine the length of the day to be 48 hours long. A partial fence can deliver value - we could build just one or more sections of fence each day.
We could divide the fence-building into post-hole-digging, post-planting, cross-beams, vertical slats, and so on. This means that the end of day one, we might have all the post-holes dug, and one-third of the posts planted. This would be the "infrastructure-first" approach that XP does NOT recommend, because this doesn't allow changing requirements easily, nor does it take in account learning that comes from completing a "small" feature.
Steve Howell expands on this 'learning' point: by building one section first, "to separate my yard from Bob's yard", we not only learn fence-building techniques on a smaller "sub-project", but also may learn that fencing-building takes longer than expected - maybe it took two days to do that one section, when we thought it would take a half-day. That learning allows us to change requirements. Instead of the next story being "build a fence to separate my yard from Mary's yard," we might opt for a "plant a hedge to separate my yard from Mary's yard" instead.
Note that if we had dug all the post-holes for the entire fence first, followed by post-planting, we'd have a less predictable situation for estimating the end of the entire job. The portions we have done so far do not form a good basis for estimating how long the next portions we're going to do, because the activities are so different. If we had done a 'complete' fence for just Bob's yard, we would have done ALL of the activities for building a fence, and thus estimating the fencing-building for Mary's yard would be more accurate.
If we had dug all the post-holes for the entire fence, and then decided not to fence away two of the three neighbors, (business requirements have changed), then we've wasted our effort, and he'd have to fill in those holes. If we did a partial fence first, that might be valuable enough "as is" not to remove, even though our plans have changed. The post-holes by themselves have little value.
Now, construction analogies to software are NOT my favorite thing. So let's talk about breaking down requirements for a software application. Steve Howell proposes thinking about the requirements for a coffee-house order system. "Until you put the entire menu on the system, you can't really use the system... How do you break this down?"
Jeff Grigg proposes the following sequence of stories, saying that many real diners work just this way:
- Server types in the total of the food items. System computes tax and total.
- System computes change.
- System tracks tips.
- System tracks which servers did which orders.
- System accepts some simple and very common orders.
John Roth proposes a couple of different stories, prioritized early:
...many places include the ability to insert orders that aren't on the official menu. That's what I'd do first, and then add in the menu items roughly in order of volume...
...I'd put in the ability to process credit (and debit) cards somewhere before tracking the servers and tips...
Now an infrastructure-first approach might have decided that a full-fledged (*) relational database with transaction processing would be a necessary first step, and a distributed system of order-station-clients a necessary second step. This is a bit like digging all the post-holes before planting any posts. However, that's not the XP way. The first several stories could have been done with a single station with a few simple text-files instead of a relational database.
The XP practices of Refactoring, Simple Design, and once and only-once would put all the code for reading and writing the text files in one or a few classes, which can be changed to deal with a relational database a later date. A little re-work would be involved, but good cohesion and loose coupling allow the change to be done without rewriting the entire application.
(*) Footnote: how may of us recall that the literal meaning of "full-fledged" is having all of your feathers -- "having fully developed adult plumage". Birds don't get all their feathers/features at once, neither do software applications.
2003.Dec.03 WedMary Pat, quoted on Learn About Lean:
2003.Dec.02 Tue1. There is a crisis - we just can't see it. Our assumptions about the stability of our jobs, our work environment, and our own expertise are all pure fantasy. [...]
2. [...] Job security is a moving target. You may not be able to control which base is closed, but you can add to the employability of your skills. [...]
3. [...] Watching successful teams is a multi-billion dollar entertainment industry. Groups of people overcoming outrageous odds are the highlight of the human condition. Think about how fun it would be to be a part of a great team. [...]
4. [...] Wasted work creates its own false reality. [...] We are trapped in our assumption that the recipients of our goods and services must be satisfied, because they keep coming back. Therefore, it "ain't broke, so don't fix it." [...] But our customers only come back until they can find satisfaction somewhere else.
To the Smalltalkers who keep using this phrase, please change it to "writing code in your IDE." Because that's what you're doing. It just happens that your program is "live" while you're doing it. Other environments have something like this, too, but people using those environments don't call their "fix-and-continue" feature "writing code in the debugger".
More on this later... (or maybe not).
2003.Nov.30 SunThe XP2004 conference on Extreme Programming is scheduled for June 6-10, 2004 in Garmisch-Partenkirchen, Germany, 90 km from Munich.
Trivia fact: In 1968 the legendary Nato Software Engineering Conference was held in Garmisch-Partenkirchen, where for the first time the term 'software engineering' was used.
There's a call for experience papers, tutorials, workshops, panels, PhD Symposium, etc.
Chris Gardner on the XP mailing list points to author Mark Forster's blog on "growing articles". He presents six drafts of an article on this subject. The first draft simply consists of the words "Growing articles". You can see the final draft in his November 28th blog entry, and all six drafts here.
2003.Nov.29 SatJames A. Robertson linked to an article in Industry Week, which describes five things unhappy companies have in common:
- A belief that employees are dangerous and lazy.
- A conviction that customers cannot be trusted.
- A focus on policies, not principles.
- An obsession with today, not tomorrow.
- Leadership in all the wrong places.
Check it out.
2003.Nov.27 ThuDid you know there are 50 members of Congress cosponsoring a bill to form a Department of Peace? I had to hear about it from a lecture televised on C-SPAN2 - an audience member asked Al Franken if he had heard of it. Neither of the two newspapers I read regularly have mentioned it.
2003.Nov.26 WedJohn Tarrant in the January 2004 issue of Shambala Sun (a magazine from the future!), writes:
2003.Nov.23 SunEveryone wants to use happiness as a fix for problems, yet happiness is its own, very big thing, and it is selling happiness short to make it a fix for problems. To be happy is to experience life not as a series of struggles but as a gift, one that has no known limit. This doesn't mean ignoring your difficulties: it means not assuming that they are what you think they are. If you throw away everything you believe about your difficulties you will notice that many of them disappear and the rest become interesting.
Martin Fowler writes of the CommandOrientedInterface and command objects and executors:
Command oriented interfaces have a number of benefits. One of the primary ones is the ability to easily add common behavior to commands by decorating the command executor. This is very handy for handling transactions, logging, and the like. Commands can be queued for later execution and [...] be passed across a network. Command results can be cached [...]
And, of course, it is relatively easy to add "Undo/Redo" support to command objects and store stacks of command objects to be undone/redone.
In a dynamic language that supports reflection, one could create generic command objects by creating a class that retains a reference to an arbitrary object [call it a 'target'], a reference to an arbitrary method of that object [call that an 'action'], and a list of arguments to pass into that method [or perhaps only one argument, the 'sender'].
Instances of this class could be created by reading a configuration file or deserializing instances that had been created in an Interface Builder. I'm not saying that Apple's Interface Builder does exactly like this, but it is doing something kind of like this. See this page for more.
Cocoa and Interface builder allow creating powerful programs without writing (or generating) a lot of code. It's been one of the most advanced development environments for at least a decade (it used to be part of NextStep), and it still is one of the most advanced development environments.
Smalltalk, Objective-C, and no doubt Python and Ruby could generic command-object classes like this very succinctly. Java would be more difficult, but it can be done in Java, as well.
Perhaps if methods and method-calls (and classes) were more easily thought of as first-class objects [which is very far from the case in C++], tools for Aspect-Oriented programming would be easier or unnecessary.
2003.Nov.18 TueI think the power of a Smalltalk environment is not just the language, and not just the environment. It's the combination of those and the Object Model - the concepts that underlay the syntax. (And a good class library helps a lot.)
I'd like to use a programming environment which has a choice of syntaxes but which uses the same object model. I could enter code in any of the following syntaxes, and see existing code decompiled into any of the following syntaxes:
- JavaScript
- Python
- Smalltalk
- Ruby
- Object Pascal
- RealBasic
- Dylan
- AppleScript
- Eiffel
Interestingly, there is one environment that is approaching the nirvana of multiple-syntaxes with a common object model. Apple's Cocoa, which is implemented in Objective-C, which has an object model inspired by Smalltalk, interoperates with Python, Ruby, AppleScript, and Java. Apple's XCode/ProjectBuilder environment allows writing code in these scripting languages and C/Objective-C/C++/Java/etc.
There are a lot of good Cocoa books out there. My favorite so far is by Aaron Hillegass: Cocoa Programming for Mac OS X It's not a lengthy as the others, and it covers a lot of what makes up a Cocoa application.
2003.Nov.16 SunAfter my wife read the book A Housekeeper Is Cheaper Than a Divorce, we hired a maid-service to clean our place once a week. Marina has been reliable and helpful. Now she has her own website: http://www.myhousecleaner.com/.
Innovator's Dilemma and Solution
I've read The Innovator's Dilemma and I'm reading The Innovator's Solution. I recommend reading these books.
The authors main points are that (1) established companies almost always get "trapped" by their customers into only supporting "sustaining" innovations - incremental improvements in their products or services, and (2) only (for the most part) new companies are able to take advantage of "disruptive" innovations that create new markets and eventually take over established markets "from below".
For example, in the steel-mill arena, Minimills using a new, cheaper, process were at first only able to sell into the lowest-quality product area: rebar. The established companies stopped producing rebar because of the low margins, and concentrating on satisfying their most "demanding" customers in profitable high-quality product areas. But the minimills continued to improve their quality, and eventually took over more and more product areas - always coming in from below, always taking lower-margin business away from established players. At any time, the established traditional steel mill companies could have bought one of the existing minimill companies, or established their own minimills, but the lower margins are too unattractive for people used to higher margins. It seems likely that minimills will eventually take over the entire steel industry, and the people used to high margins will have to get used to no margins at all.
Disruptive Innovation can also create new markets where none were before. The established companies, particularly large ones, do not have an incentive to sell into a new market, because the new market is so small compared to their existing markets. Sales would be 1/100th or smaller in comparison, and not attractive to people used to making larger sales numbers. Apple experienced this with the Newton. It pretty much established a new market and sold more Newton MessagePads in two years than they had sold Apple IIs in its first two years. The difference is that Apple was a much smaller company when it was selling Apple IIs (and had invested less money in Apple IIs than it had invested in Newtons.) A small company (like Palm originally was) would spend less on the product's development, and be more satisfied with sales totals that a large company would find to be too small.
2003.Nov.15 SatOne difference between Pokayoke in manufacturing compared to software, is that software is always unique and fresh (if it isn't, some re-use is lacking). To "manufacture" a software product, simply copy the CD-ROM or post the installer on a web-site. However, pokayoke is always about quality assurance in either domain; either detecting problems as soon as possible, or preventing problems from happening.
Harry Robinson has an article about a Pokayoke software tool here: http://www.campbell.berry.edu/faculty/jgrout/pokasoft.html, along with some background material about Pokayoke. Quotes:
Shigeo Shingo was a leading proponent of statistical process control in Japanese manufacturing in the 1950s, but became frustrated with the statistical approach as he realized that it would never reduce product defects to zero....
While visiting the Yamada Electric plant in 1961, Shingo was told of a problem that the factory had with one of its products. Part of the product was a small switch with two push-buttons supported by two springs. Occasionally, the worker assembling the switch would forget to insert a spring under each push-button.... This problem of the missing spring was both costly and embarrassing. Management at the factory would warn the employees to pay more attention to their work, but despite everyone's best intentions, the missing spring problem would eventually re-appear.
Shingo suggested a solution that became the first poka-yoke device:
In the old method, a worker began by taking two springs out of a large parts box and then assembled a switch.
In the new approach, a small dish is placed in front of the parts box and the worker's first task is to take two springs out of the box and place them on the dish. Then the worker assembles the switch. If any spring remains on the dish, then the worker knows that he or she has forgotten to insert it.
The new procedure completely eliminated the problem of the missing springs.
Shingo went on to develop this mistake-proofing concept for the next three decades. One crucial distinction he made was between a mistake and a defect. Mistakes are inevitable; people are human and cannot be expected to concentrate all the time on the work in front of them or to understand completely the instructions they are given. Defects result from allowing a mistake to reach the customer, and defects are entirely avoidable. The goal of poka-yoke is to engineer the process so that mistakes can be prevented or immediately detected and corrected. Poka-yoke devices proliferated in Japanese plants over the next three decades, causing one observer to note:
It is not one device, but the application of hundreds and thousands of these very simple "fail-safing" mechanisms that day after day has brought the quality miracle to Japan. Each one is relatively simple -- something you easily could do on your own. It is the totality, the hundreds of devices, that is almost frightening to behold.
Let me rant about bit a the Standard C Library. This library represents not the best or most appropriate set of functions that C programmers need to use, just the the ones that were most commonly implemented by C compiler vendors when the standardization process was begun. The string functions provided by the library are dangerous. For example, the string-copy function "strcpy(destBuffer,srcBuffer)" doesn't check the size of the destination buffer, so copying a too-large string can overwrite the stack or memory, provided us with countless bugs and security holes. Also countless lines of duplicate code where the programmer checked the size of the source string against the destination buffer-size before doing the copy.
The Standard C Library contains what might be considered a safer string-copy function "strncpy(destBuffer,srcBuffer,destBufferSize)" that will not copy characters beyond last byte of the destination buffer. But it will not terminate the string with a null-byte in the destination buffer if the source string is longer than the destination buffer. This lack of string termination means that functions expecting a well-formed string in that destination buffer will misbehave: accessing memory beyond the end of the buffer. A programmer who does not know this "gotcha" will have created another bug and/or security defect. Once again we will have countless amounts of duplicated code as the some programmers either write a "safe strncpy" function, or add a line of code to insure the destination buffer has a terminating null-byte every time they use strncpy.
And guess what? There's another problem with strncpy (and most "safe strncpy" implementations). It doesn't copy characters, it copies bytes. In multiple-byte encodings like SJIS and UTF8, half of a multiple-byte character could be left at the end of the buffer, if the buffer just happens to be one byte too short.
I avoid most of the problems with strcpy/strncpy, etc., by using the C++ std::string class. It dynamically allocates memory for strings, so I never have to consider buffer-sizes. This helps mistake-proof string coding. One frustration I have with this class, is that it is possible to pass a null-pointer into the constructor. The standard says that a character-pointer passed into the constructor must not be a null-pointer, but the implementations I use will not detect this situation. Generally, if this happens, there will be a crash sooner or later, and some debugging will be required to find out how a null-pointer got passed into a std::string. If the implementation of std::string had a simple line of code -- "assert(ptr != NULL);" -- the debugging time would be reduced.
C and C++ have a problem where its static type-checking falls down: integers are freely (and silently) convertible to pointers and bools, and vice-versa. In this domain, the language is "loosely-typed". The bool value 'false' can be passed into a std::string constructor because the compiler changes it character-pointer (whose value is the null-pointer). A character-pointer can be assigned to a bool variable - if the character-pointer is the null-pointer, then the bool variable takes on the value 'false', otherwise it takes on the value 'true'. Microsoft's CString class used in MFC has a "type-cast" operator method that returns a character-pointer to the CString's internal buffer. I recently found some code that "assigned" a CStdString (a cross-platform class with an API similar to CString) to a bool variable -- this silently called the character-pointer type-cast operator and assigned 'true' to that variable, since CStdString allocates its own non-null buffer. Not at all what I wanted.
Fortunately, I can avoid std::string and CString and use (and customize) CStdString. My version of CStdString has an assertion to detect being passed a null-pointer. My version of CStdString does not define a character-pointer type-cast operator. These two changes help mistake-proof my code.
2003.Nov.13 ThuPokayoke is a way to prevent mistakes from being made, or to detect a mistake immediately. It is "fool-proofing".
You might think that explicit/static typing is all we need for fool-proofing, but we know from experience that the presence of explicit typing does not eliminate bugs. Let's show this with an example in Java.
Requirement: "The game displays the winning player's name in the game-log."
class Player
{
public int score;
public string name;
};
class GameLog
{
public string logText;
public void appendText( string textIn )
{
// implementation left for the reader.
}
};
class Game
{
public void displayWinningPlayer( Player playerOne, Player playerTwo, GameLog log )
{
}
}
Now the above code satisfies the compiler's explicit type-checking, but it doesn't satisfy the requirement. Let's build a way of detecting this failure to meet the requirements - using Test Driven Development.
// ignore other JUnit scaffolding... just show the test...
public void TestDisplayWinningPlayer() throws Exception
{
Player one;
one.score = 12;
one.name = "stan";
Player two;
two.score = 13;
two.name = "kyle";
GameLog log;
Game game;
game.displayWinningPlayer( one, two, log );
assertEquals( "kyle won with 13 points", log.text );
game.displayWinningPlayer( two, one, log );
assertEquals( "kyle won with 13 points", log.text );
}
So how do we know the test is a valid one for the requirements? Well, it looks good to me. I ask my pair partner; he thinks it looks good, too. Since we're doing Extreme Programming, we ask our on-site customer what the log should look like. She looks at our test and says "This displayed text should should have an exclamation mark at the end." So we add "!" to the end of the strings in our tests.
We also validate the test by running it now, even though the displayWinningPlayer function is empty. The test fails (as we expected) because the necessary code to make it pass hasn't been written yet. If it had passed, then we would know something is wrong with our test.
assertion failure: expected 'kyle won with 13 points!' is not equal to actual ''. (line number and stack crawl follows.)
Now we write just enough code to pass the tests. Unlike some examples in the TDD literature, I'm going to write the 'right' code directly, instead of taking tiny steps.
class Game
{
public void displayWinningPlayer( Player playerOne, Player playerTwo, GameLog log )
{
if ( playerOne.score > playerTwo.score )
{
log.appendText( playerOne.name + " won with " + playerOne.score + " points!" );
}
else if ( playerTwo.score > playerOne.score )
{
log.appendText( playerTwo.name + " won with " + playerOne.score + " points!" );
}
}
}
We run the test and get this output:
assertion failure: expected 'kyle won with 13 points!' is not equal to actual 'kyle won with 12 points!'. (line number and stack crawl follows.)
Whoops. Our second assertion failed. We can immediately see that the failure has something to do the points, since it got the name right. We don't even have to get into the debugger. Doh! Look at the source: when player-two has the higher score, we still append player one's score. We fix it, and the now the test passes. Pokayoke.
We ask the customer about ties, and he says that it can't happen according to the rules of the game. I would probably document that kind of 'negative requirement' with an assertion in the code. (The assertion has no effect in 'release builds' but it does have an effect in 'debug builds' that we use in test-driven development and manual pre-release testing).
public void displayWinningPlayer( Player playerOne, Player playerTwo, GameLog log )
{
assert playerOne.score != playerTwo.score : "ties are not supposed to happen - fix your test/code";
if ( playerOne.score > playerTwo.score )
{
log.appendText( playerOne.name + " won with " + playerOne.score + " points!" );
}
else if ( playerTwo.score > playerOne.score )
{
log.appendText( playerTwo.name + " won with " + playerTwo.score + " points!" );
}
}
And one last point. I could have written all of the above in Python, which does not have explicit/static typing. The tests would still detect my failure to meet the requirement, and the code would be a lot shorter and more succinct.
Extreme Programming has another way of detecting errors relatively quickly: "Frequence Releases." If possible, some end-users of this game will see versions of the game perhaps as often as every iteration. But that goes beyond pokayoke into the general area of feedback.
2003.Nov.11 TueRon Jeffries wrote something quotable today:
Calling something you don't understand and don't know how to do "religion" is an entirely specious argument. There are people all around you doing these things and having them work. Tell us what happened when you tried it. Until then, it seems fatuous to tell us that we can't be doing what we are doing.
2003.Nov.08 SatSomeone wrote to Bruce Eckel asking for help:
I'm trying to develop a business application in C++. I want to be able to build a framework of the business objects which can be extended later without touching the framework source.
My main problem (one of them anyway) has been designing a way for these objects to be persistent. I want to use a relational database product....
The first thing wrong is using C++. I've used C++ for close to 16 years, but for the purposes he wants (persistence, extending a system without modifying code), it is the wrong tool. This is a case where you need a dynamic language with good reflection capabilities. Python or Smalltalk (or even Objective C or Java in some circumstances) would be better choices.
The second thing possibly wrong is very likely using the relational database. Unless interoperability or scalability issues require it, it would probably be better to use persistence/serialization options available in Python or Smalltalk instead of trying to get around the relational-object impedance mismatch.
The third thing, which Bruce goes into, is premature framework building. What is the problem being solved? Maybe it doesn't need a full-blown, extensible framework.
The writer also complained about Visual Basic applications being too rigid to re-use. There is another tool can help for this problem: separate the GUI from the model by incrementally implementing the model test-first (Test Driven Development) and implementing thin GUI code afterwards.
Principles are Constant, Practices Can Vary
Curtis Cooley writes (book links are my addition):
Womack and Jones opened my eyes in Lean Thinking to the fact that practices don't map from project to project. They discovered after their landmark book The Machine That Changed The World that manufacturing companies were having little or no success implementing Lean Manufacturing even though the company embraced and believed it. The problem was that the companies were trying to map the practices that Toyota used to their particular manufacturing process. This doesn't work. The principles of lean manufacturing are constant, so you take the principles and adopt your own practices, possibly from a list which includes practices known to work.
The interesting thing is that Toyota's practices don't even stay the same from month to month. Their continual search for improvement (driven by all levels of employees - not just management or a few driven people), means trying things and fixing things all the time. See what Joe Ely has written here:
Several years ago, I read a discussion of Toyota's openness in conducting tour after tour of their facility. The reason is that most visitors see things as they are. They take a "mental photograph", if you will, of a static system, resolving to try to do what they see. But the real power in Toyota's system is in its dynamic nature. It is impossible for the outsider to see just how rapidly and repeatedly the Toyota folks change the system. It is anything but static...it is a living, breathing, dynamic system that both changes constantly and is amazingly stable.
PS: We as programmers will be much better off if we could tune and periodically modify not only our practices, but the IDE and programming language we use as well. (Toyota can refine and build new tools for making cars, we should make it easy for us to do refine and build new tools for making software solutions.) Smalltalk is one particularly malleable system. In Squeak and many other implementations of Smalltalk, the compiler is written in Smalltalk and is something you can modify (as well as the IDE). [Unlike for complicated languages like C++, a Smalltalk compiler is succinct and relatively more understandable.] I don't have it handy right now, but some time ago I was given a piece of code, less than third of a page in size, that adds an "increment assignment" operator to the language, so you can write something like "a :+= b" instead of "a := a + b". Many other things that would be additions to the language in other languages, are just new methods or classes in Smalltalk.
2003.Nov.07 FriThe days of the week - (mostly) forgotten gods:
- Sun's day
- Moon's day
- Tiw's day (a Saxon war-god)
- Woden's day (Anglo-Saxon/Teutonic god)
- Thor's day (Norse god)
- Freya's day (Teutonic goddess)
- Saturn's day (Roman god)
See also http://www.crowl.org/Lawrence/time/days.html.
2003.Nov.06 Thu
Cost of Low Quality
Johanna Rothman's What Does it Cost You to Fix a Defect? And Why Should You Care":
Nynke Fokma's Are We Solving the Real Problem?: The global organization had a list of 5 company values for all thousands of employees all over the world. One read "Exceed Customer Expectation." How many projects had we shipped in the last year that did not live up to customer expectation, let alone "exceed" it? I had no idea. None of us was directly in touch with customers and users so how will we know how we are doing? Elizabeth Hendrickson's You Can't Test the Wings Back On an Airplane:After release, Company A estimated that it spent 20 person-days per fix. At $500 per person-day, this comes to $10,000 per fix. Company A typically had 20 "emergency" fixes after each release, for a total post-release fix cost of about $200,000.
Company B tracked the time it took to fix a defect during implementation (before system test), and found the average time to be about two hours, or about $125 per fix. Company B generally didn't release fixes to its products once the product was in the field, so it had no extra post-release costs.
Like the others, Company C did not track defect fix time during the requirements, design, and implementation phases. However, Company C engineers used a build-every-night and fix-any-defects-immediately approach to product development. In effect, the engineers were finding and fixing defects throughout the entire lifecycle. They spent an average of one person-hour each day fixing defects from the previous night's build, or about $60 per defect. Using this approach — along with code walk-throughs and peer reviews — they generally shipped a very high-reliability, low-defect product, and were able to avoid most post-release defects costs. When they did have an "emergency fix," their average time to fix was about 5 person-days per fix, or $2500 per fix.
Company A had the highest cost, over $400,000 just to fix defects.... Company C had the largest, most complex project, with the most people, and was able to contain its defect costs to about $135,000.
2003.Nov.05 WedTesting can tell you when something isn't right, but it can't fix what's wrong.
When I talk with individuals, most understand the difference. Yet the common behavior of software development groups shows that some people still believe that a test group really can ensure quality. The biggest indicator that this attitude persists lies in the name given to test groups: QA, Quality Assurance. Beyond that, there is an attitude, common in many organizations, that foists blame for bad releases on QA.
Quality problems originate long before the testers get their hands on the product. Even in an environment where testers are involved early, testers without training in quality techniques are unlikely to lend much to a quality effort. In short, naming a group QA seems to make some people in other groups think they are officially absolved of the responsibility to promote quality practices. "That's QA's job." I've heard more than one programmer, technical writer, and project manager utter those words.
Actually, quality is everyone's job. It begins with the requirements and ends with a solid product shipping out the door: a product that meets, exceeds, or redefines the customers needs and expectations.
Test-Driven Design/Development
I'm about two-thirds done reading Unit Testing In Java: How tests drive the code by Johannes Link. It's a very good book. Go buy it now, even if you're not programming in Java, and then finish reading my blog. :-)
The summary form of test-driven development is:
- Think about what we want to the program to do.
- Write a test that shows some aspect is done.
- Run the test, see it fail because we haven't written the code to get the thing done yet (this tests our test, giving us some confidence that our test is written correctly).
- Write the smallest amount of code to make that test pass.
- Run the test, hopefully see if pass. (If it doesn't, fix the problem.)
- Refactor to clean up the code.
- Run the test again, to make sure refactoring didn't break anything.
- Repeat [sometimes skipping 6 and 7] until you can't think of any new tests, or until the code stops changing as a result of adding tests.
So what if we don't know what we want the program to do? Writing a test can help us figure that out. Since the code hasn't been done yet, the test is "black-box" and does not necessarily commit us to any particular way of implementing the solution. We can write "Exploratory Tests" to investigate existing code - does calling "X" do "Y"? Write a test and find out.
I don't have time to write lots of tests. Well, you only have to test code that you want to work. If you don't test it before you write, you'll be testing it in some fashion after you write it. Or someone else will test it, file a bug report because you didn't test it, and then you have to not only interrupt whatever you're doing and fix it, but also test it to confirm that you fixed it. And the time-delays between your writing the code, the tester finding the bug, and your trying to fix the bug means that you will be less familiar with the code, and thus fixing it will be that much harder and slower.
There is, of course LOTS of code out there that was not written this way (some of it is mine). Just browse some of the code on the internet or in your company's source-code-control system - probably most of it isn't written using test-driven techniques.
You'll probably find these things in non-test-driven code (particularly in projects that don't do code-reviews or pair-programming and refactoring):
- Dead code -- code that is never invoked.
- Duplicate code.
- Unreferenced parameters and variables.
- Variables that never vary: 'constant' variables.
- Toughly-coupled code. You can't use a class in isolation because it depends on lots of other concrete classes.
- Non-cohesive classes and methods. (Classes and methods that do "too much")
- BUGS
There is in fact a whole list of "code smells" (possible design problems) at http://c2.com/cgi/wiki?CodeSmell. TDD helps prevent some of them, and refactoring with the safety net of tests can allow you to to fix the others. With TDD and Refactoring, you don't have to live with smelly code.
I find that once my tests are passing, I almost never have to go back and fix any bugs later. This saves me a lot of time. At the worse, it takes about the same time as coding + debugging, but it's less stressful. If a bug arises during TDD, it is most likely in code that I wrote less than five minutes ago -- easily found and fixed.
For those people who say they don't have time to write tests: why do you have time to write dead code, duplicated code, and where does the time to fix bugs come from?
In traditional development and testing, you create a lot of defect-ridden code, and then test the defects out. The underlying assumption is that this is the only way it can be done. Test-Driven-Design works with a different assumption: You can put the quality in first, and not have to spend time getting defects out.
Traditional (maintenance) development, since it usually isn't supported by a suite of tests, eventually grinds to a halt: the code becomes so fragile that it can't be modified without breaking something. I don't know if the IRS has depreciation tables for source-code assets, but the reality is that code eventually loses value unless you take steps to keep it from losing value. Sometimes, code becomes unmaintainable while the company is still trying to make money off of it - the result is the company loses money or doesn't get the opportunity to modify the code to make more money. Test-Driven-Design provides a suite of tests that allows you to refactor - keeping the code maintainable so it is easy to enhance, re-use, etc., so you can make money.
2003.Nov.01 SatJoe Ely mentions a Lean Conference. While most of the sessions are aimed at Lean Manufacturing, and a few aimed at Lean Office work, I found this session description particularly interesting and applicable to Lean Software Development:
2003.Oct.31 FriGS4 - Lean Liberates Leadership: Flowing Through the Top of the Bottle Jim Greer, CEO, The Greer Company
There is a common misconception among even experienced leaders that their job is to lead. To lead would then mean to control. Lean leadership is a giving away of control and discovering leadership in others. When I exercise lean leadership I inspire leadership in others to find and eliminate waste, to create a more direct route, to decide and move in more efficient ways. When one person controls leadership there is by definition a bottleneck. Most bottlenecks are at the top of the bottle. This talk is designed to help those who are positioned at the top of the bottle to help others to be liberated to lead without putting the whole business in jeopardy. It is also designed to help those who are stuck in the bottle to develop strategies to appropriately influence those at the top.
Dominique Plante has posted notes from a BayXP meeting in which Ron Jeffries talked about planning XP projects. (These notes have been revised from a previous posting.) Quoting:
2003.Oct.30 ThuBased on what Ron learned from how people played the planning game, Ron described a simple Microsoft Excel-based simulation that he developed that attempted to simulate the planning of real world projects. In the simulation, a project would consist of a set of tasks, where each task was given a value and a difficulty. Using the simulation, a new project could be simulated at the push of a button. Ron observed that all projects he simulated seemed to follow a pattern - assuming the development team worked at the same speed, it seemed possible to determine how long it would take to complete a project just from tracking the simulated team's velocity after only a couple of iterations. The whole talk was geared to get input from people to see what their thoughts were on how realistic / possible this might be.
Thus, Ron had developed a theory on how projects could be planned so that they succeed. He wanted peoples' thoughts on that theory.
Donnella Meadows wrote an article Places to Intervene in a System [pdf] that was published in Whole Earth Winter 1997 where her number one intervention point was "The mindset or paradigm out of which the goals, rules, feedback structure arise."
I've recently reread an article which describe a process-adoption experience, which was viewed by some as a success and others as not so successful. Those who saw the new process as a success felt that they had gone through a paradigm change, and the ones who didn't think it was so successful were still operating in the old paradigm. Their summary:
If you believe that quality comes from inspecting defects out, you don't need a tool for building quality in. If you believe that man-hours are the measure of success, you don't need a tool for delivering more value with fewer man-hours. If you believe projects are basically predictable, you don't need a tool for managing unpredictability. If you believe that social interaction is a small variable in project success, you don't need a tool for improving communication.
This paper is Learning by Doing: Why XP Doesn't Sell [pdf] by Kay Johansen, Ron Stauffer, and Dan Turner.
2003.Oct.22 WedBurn-Down Charts graph the work left to do, which is reduced by having done work, or by reducing scope. See the "progress" chart here. The project is done when the work left to do reaches zero (or time runs out).
John Brewer describes Burn-Up Charts here and here. The "finish line" -- "total work to be done", is drawn as separate line from the graph of the "work done so far". The project is done when the work done so far reaches up to the finish line. If the finish line is changed by reducing scope, the finish line is moved. The burn-down chart confuses velocity (work done per iteration) with reduction of scope; the burn-up chart keeps those ideas separate.
Here's an ascii art burn-up chart where, about half-way through the project, scope was reduced in order to be able to finish by the ship date:
| FINISH
| ------+ |
| | |
S| +-----+----
C| . |
O| . |S
P| . |H
E| . |I
| . |P
| . |
+-------------+----
TIME
2003.Oct.21 Tue
"When I use a word," Humpty Dumpty said, in a rather scornful tone, "it means just what I choose it to mean—neither more nor less."
"The question is," said Alice, "whether you can make words mean so many different things."--Lewis Carroll
One of the problems with Metaphor is that literal-minded people can always find cases where a metaphor doesn't apply to a situation. When Shakespeare wrote "Shall I compare thee to a summer's day," he didn't intend for us to think about a daylight savings time, or people dying from the heat. Industrial XP has replaced the XP Metaphor practice with "Domain Driven Design". There's a book with title that many people have recommended. I bought it, but I haven't read it yet. Check it out: Domain Driven Design by Eric Evans.
David J. Anderson reports that Arlen Bankston has replaced XP's Metaphor with "Shared Vision" from Peter Senge's "The Fifth Discipline". Anderson also points to Jesse James Garrett on "visual vocabulary" at http://www.jjg.net/ia/visvocab/ and Constantine and Lockwood's User Role and Task Models from usage-centered design.
2003.Oct.19 Sun
More Testing Support Means Less Need for Debugging...
James Robertson quotes Steven J. Vaughan-Nichols as writing "Enough is enough. It's time to make debugging code job No. 1. People have put up with bad programs for far too long."
Test-Driven Development can making stepping through code to understand what's going wrong mostly unnecessary. Here's what people on the XP mailing list have said recently:
I haven't used a debugger since 1999. But I guess there are lots of people who haven't written a test since 1999 either. I wish IDE makers would focus on profilers instead of debuggers. -- Bill de hÓra
Writing a test rig for 'strcpy()' is obvious. Writing a test rig for a WTL ActiveX Host window containing an embedded control under apartment model threading and multiple toolbars that hook into window events is ... non-trivial. [...] We must stop blaming ourselves and start blaming the tool vendors. If they can't test it, they shouldn't ship it. -- Phlip
Hmmm... actually, you just need a better test runner. The test runner knows what test failed, and it should be able to show it to you. Why isn't it? -- Robert Watkins.
Because its vendor has sunk millions into polishing an elaborate and full-featured debugger, to clean up the mess after making it. -- Phlip
If IDE makers focused on tests...
Imagine all the effort the past 2 decades spent on debuggers, spent instead on test rigs. -- Phlip
- wizards would create both code and the test-first rig the wizard author used to generate that code
- the source browser would distinguish between production code & test code, and would flip automatically from a test case to its target code
- the Go button would not ask any stupid questions, such as "you want to save files?" or "you want to compile?"
- Undo would have the ability to roll back to any previous GreenBar
- there'd be an Integrate button - NOT a bunch of goofy checkmarks on each file stating if it's "checked out".
- integrating would run the tests, and would back the integration out if the tests failed.
- Extract Method would be on the toolbar
- at assertion failure time, the IDE would know both the call stack of the production code and the fixture of the assertive code
- the Workspace panel on the left would have a third tab, containing the test suites and cases
- one could run any subset of tests, or any superset of folders with tests, from the IDE
I've never used a debugger for Java much... there were never very good tools.. by the time there were (that I had access to) I was practicing TDD. Back in '98 I was on a team that had a "test everything that could possible break" mentality (with a homegrown test framework), and code wasn't complete without tests, etc. We also evolved our design, refactored as needed, and on occasion threw out subsystems when it turned out that we had gone down a dead end. All this without any "Agile" buzzwords (none of us knew about XP at that point). Best team I've ever been on... so far. --Dave Astels
If I have a "bug" I write a test to find the bug. -- Nathan Paris
I don't like the debugger at all. It's a weird sort of personal failure mentality, where if I have to debug then maybe I don't understand the code enough to be writing it in the first place. -- Curtis R Cooley
Yes, the debugger is a valuable tool. I only use it as a last resort, when I can't get to the green bar from a failing test that I think should be passing. -- Kathie Drake
And what is a real shame, is that in some shops, it is quite acceptable for someone to be sat their wasting time in a debugger, not pairing or writing tests, but not acceptable for two people to be sat at a machine. -- Nick Robinson
I program in C++ for work and am very familiar with the debugger there. Unfortunately, even with rigorous tests it seems the debugger is almost unavoidable in C+. [...] On the other hand, I program my own for fun programs in either Java or Python, all test first with merciless refactoring. I don't even know how to use the debugger in those languages. I've never needed one. -- Brian Christopher Robinson
2003.Oct.17 FriJohanna Rothman mentions a few ways to produce software faster. A few more ways are:
- Don't use the conventional solutions that everybody else uses - use the tools that can allow developers to go up to seven times faster. For example, use Smalltalk (see also Cincom) or Python or Cocoa instead of Visual Basic or Java or C++. Use WebObjects instead of EJB. If you do use Java, use Eclipse or buy IntelliJ IDEA.
- "Sharpen the Saw" -- provide training for the above tools for your developers
- Don't use virus-ridden servers and clients. MacOS X is more productive for most users, MacOS X Server is cheaper than Windows servers, and MacOS X clients have a lower cost of ownership than Windows clients
- "Sharpen the Saw" -- provide training in object oriented design, test-driven development, design patterns, and refactoring for your developers -- don't assume they know this stuff already.
- Find out what the most productive developers use by asking the developers - don't believe marketing hype from the big companies who want to sell you their tools.
- Beware "resume-driven-development." XML may be trendy, but may not be the simplest (and fastest) thing to use in your application.
For Managers - Tracking Progress
How do you (a non-technical manager) track how much of the project has been done? Have your programmers implement a few features at a time, and have QA verify them with automated tests. A feature is done when its tests pass. Do manual testing as well, but you need automated tests to make sure that working features don't get broken when later features are added. Track your progress with a "burn-down" chart. Robert Martin explains it (with two simple graphs) here: http://www.artima.com/weblogs/viewpost.jsp?thread=16880.
2003.Oct.16 ThuBill Caputo sums it up well: The real problem is bad coders.
Joel says exceptions are just as bad as "Go To"s.
What was the danger of "Go To"? Too many of them would make the control-flow hard to understand. The the whole point of "while", "do", "for", "break", "switch", "if/else if/else" and "return" was to allow changing the control-flow in easier to understand ways than "Go To". Of couse, over-usages of these can also make the control-flow hard to understand. In particular the following piece of code is just as bad a using "go to," and can be confusing if you expect "do...while" to loop. (This example doesn't loop.)
do { statements; moreStatements; if ( cond ) break; // equivalent of "go to" otherStatements; } while ( false ); // destination of "go to." Does NOT loop.A "Go To" links a source point to a destination point -- this increases coupling between two points of the code. In C++, a "Go To" can be dangerous because it could skip initialization or destruction of local variables.
What is the danger of exceptions? Too many of them can make the control-flow hard to understand -- but so can other control-flow statements. An exception does not link a source point to a destination point. It does not increase coupling between two points of code (generally speaking). In C++, the compiler (and the scope rules for try/catch) insures that throwing an exception does not skip initialization or destruction of local variables.
The key point of using exceptions is to only use them for exceptional situations. If you're calling a search function, you expect to either find or not find find what you're searching for, so the search function should not be throwing an exception when something is not found (or when something is found). If the search function runs out of memory, an unexpected situation, then throwing an exception would be appropriate. Some place higher up the call-chain, perhaps in the user-interface code, should catch "out of memory" exceptions -- no matter who throws them -- and safely handle the situation. In the "old" days on MacOS pre-X, our "out of memory handler" would free up a reserve of memory, display a pre-fetched error alert, and then the user was in control again -- hopefully able to save and quit safely.
I've seen very readable C code that used "setjump/longjmp" to do the same thing as throwing an exception -- to exit from any of several places, to an exception-handler, in the case of an unexpected or unrecoverable error situation.
2003.Oct.15 WedMy reply to James A. Robertson:
Joel may be working in C++, perhaps with a lot of legacy C code... the only problem with exceptions I have seen, is that mixing exception-safe code with non-exception-safe code is ... unsafe. If the bottom-most layer returns errors code, then all layers above it can be written to use exceptions and be exception-safe. If any code calling exception-throwing code is not itself exception-safe... then you've got problems.
Exception-safety requires, in C++, a lot of reliance on "construction is resource-acquisition/destruction is resource-releasing" idioms. However, making that effort to be exception-safe turns out to be a good place to use XP's "once and only once" rule. Calling "delete" on an dynamically-allocated object should be done only once in a class that pretty much exists only to handle delete -- a smart pointer template class. Calling the operating system's "close file" function should only be done once, in a class that handles files.
When you use objects "by value" (allocated on the stack), C++ insures that their destructors are called when an exception is thrown - allowing you to close files, delete objects, and perhaps even roll-back a partially-completed database transation.
In Java and other languages that have garbage collection, most of the need for "resource-releasing" goes away, and the remaining can be taken care of by occasional "catch" or "finally" blocks. Not sure what the Smalltalk equivalent is (has this been ANSI-Smalltalk standardized?)
My reply to Joel's Oct 15 blog entry:void InstallSoftware() { DoCopyFiles(); MakeRegistryEntries(); } void InstallSoftwareGUI() { try { InstallSoftware(); } catch( NoDiskSpaceException& ex ) { // don't log - no space DiskErrorMessage( ex.DiskName(), ex.DiskSpace() ); } catch( RegistryException& ex ) { Log( ex.SrcFilename(), ex.SrcLineNum(), ex.what() ); ErrorMessage( kNoRegistryPermission ); } catch( std::exception& ex ) { try { Log( ex.SrcFilename(), ex.SrcLineNum(), ex.what() ); } catch ( std::exception& ex2 ); { // ignore problem with logging } ErrorMessage( kUnknownError ); } } Curious about recovery for copying files...? void DoCopyFiles() { FileCopierWithRollback appCopier( "app.exe" ); FileCopierWithRollback libCopier( "lib.dll" ); appCopier.DoCopy(); libCopier.DoCopy(); appCopier.CopyIsDone(); libCopier.CopyIsDone(); }These File Copier With Rollback objects will remove the file they copied in their destructor, unless you call CopyIsDone() to indicate that the copy of all the objects was successful.
If DoCopy throws an exception, the CopyIsDone() methods will not be invoked. When handling an exception, C++ automatically calls the destructors of local (stack-allocated) objects of all the functions that are being exited or "by-passed" between the throw and the catch, so, since CopyIsDone() hasn't been called, the File Copiers remove the files that they tried to copy.
This is very similar to Command Objects that support Do, Undo, and Redo. The C/C++ User's Journal has described a way to write templates to generically support specifying the action and roll-back action without writing new classes... but I don't recall the syntax.
Please ignore the absence of a destination directory parameter in the code, this is something I wrote quickly. In real life, I'd use Test Driven Development, which would flesh out the interfaces more realistically than this sample.
Really good post by John Roth on the XP Mailing List. Snippets:
Lean thinking didn't take over manufacturing because everyone saw it and said: "oh, wow!" It took over maufacturing because when Toyota started importing significant numbers of cars to the US, it started eating Detroit for lunch and asking for seconds.
Someone on a blog stated their opinion that software development going overseas was equivalent to the Japanese car manufacturers taking business from Detroit. I think it's much more like the garment industry moving its work from the U.S.A. to overseas -- the work going overseas is not being done differently (for the most part), it's just being done cheaper. When it is being done differently, it's because the management is different.
For example, see Johanna Rothman's blog: "I'm convinced that the reasons outsourcing works is that it forces organizations to document requirements and the outsourcers work on only one project at a time. The outsourcers' management can then choose any number of useful product development practices that increase the outsourcers' productivity. Management can't change their minds and refocus the outsourced project(s) in the same way they feel free to refocus the internal projects.".
To continue John Roth's posting about Lean Manufacturing and Lean Development:
With that kind of example, the early adopters tried it and found that it solved a lot of endemic problems. In the last two decades it's become the 'heads up' thing to do in manufacturing, but only now is the effect on inventory turns becoming aparent in the US national statistics.
And that is without full participation from the industrial establishment. If manufacturing was fully Lean, you wouldn't see jobs going overseas. It's impossible to operate a pull environment when one of your processing steps takes a month to transport goods by ship. That's old Henry Ford / Frederik Taylor thinking.
[...] I didn't start to see the difficulties with what the Poppendieck's had done before I read "Lean Thinking" by Womak and Jones.
[...] XP turns the fundamental process motif of plan driven development on its head. Plan driven development tries to take a work unit and run it through several processing stages: Requirements, Design, Coding, Testing, Deployment. XP runs smaller work units through a single developer who is responsible for taking it from requirements through a fully integrated, production quality deployable.
Everything else, and by that I mean *everything* else, in XP is a supporting practice to that one fundamental difference. The fact that most of those supporting practices have been in the environment for decades was simply luck: Kent didn't have to invent them.
You know you're doing "plan driven development" when each stage is done by different people. In XP, the practices for eliciting Requirements from the person(s) who know the domain/requirements, for doing Designing and Coding and Testing, are done all the time in small batches, by the same small group of people (maybe a few specializing in testing), and Deployment is (ideally) automated to the point of painlessness so it could be done after each iteration.
2003.Oct.13 Mon[...] In manufacturing, sooner or later you've got to take the plunge of making a fundamental reorganization of the factory floor. In software development, you've got to take that same fundamental plunge of reorganizing your development process. The Poppendiecks put it quite well: "in manufacturing, inventory control is the starting place that always works. In software development, the short, fixed length iteration is the starting place that always works."
Once you make the fundamental committment that you are going to take some part of the system you are building from requirements to fully integrated, production quality deployable in two weeks (or one week or one month,) all your other practices have to change. You cannot have handoffs and approvals for requirements analysis, nor can you have a separate, end of the line QA department. You can't have a godlike Data Base Administration department that takes months to optimize your data model either.
David Anderson, who wrote a book on the subject, announces:
I'm going to be co-authoring a regular column at the Borland Developer Network website called (unsurprisingly) "Agile Management". I'll be working with Mike Watson, a developer manager I work with daily. His hands on, first line management, downward focus will compliment my second/third line manager upward focus.
The URL for the site is http://bdn.borland.com/coadletter/agilemanagement/
I have posted a draft of the first article to the files section of the group, http://groups.yahoo.com/group/agilemanagement/files/ [ZenOfAgileManagement.pdf]
I recently finished reading the Cutter Consortium's "Executive Report on Agile Project Management, volume 4, number 6" on "Challenging the Fundamental Notions of Software Development" by Robert N. Charette. Lots of good stuff. I got this report for free, but the subscription (which I don't have) costs a whopping $2400.
Charette writes, in a summary of the Japanese experience with Lean Manufacturing of automobiles, "In performing lessons-learned studies, the Japanese discovered later that process and technology improvements on the factory floor contributed only 25% towards achieving their six original targets. Humanware issues account for the rest."
I found another interesting read while Looking for Robert N. Charette on the web: Center For Quality of Management Journal. Check it out.
2003.Oct.11 SatSteve McConnell on Extreme Programming
Dave Hoover compares advice from Steve McConnell's Code Complete with advice from XP and TDD books, prompted by Steve dissing XP in a Wired magazine article.
Notice Steve's self-contradictions in his Wired quote; he calls it a fad and also calls it nothing new (he also uses the term "sloppy" and yet says "the rules are too rigid":
This industry has a long history of latching onto fads," says programmer and consultant Steve McConnell [...] "and few [practices] are new; project management guru Tom Gilb came up with similar guidelines years ago."Steve McConnell's web site has a slightly more balance appraisal of XP here: http://www.construx.com/docs/open/CxOneXpWhitepaper.pdf. In that paper, he writes "XP is based on longstanding industry best practices, including evolutionary prototyping, short release cycles, and active end-user involvement in requirements definition."
That paper is also promoting McConnell's own methods toolbox, "CxOne", of which he writes: "CxOne tailoring must be performed by highly trained and experienced staff [...]. Once tailored, CxOne can be used by staff at any skill level." Which is the same claim made by RUP, which also has an "tailored" set of methods equivalent to XP.<.p>
He claims XP can't work in several areas because it is "inflexible", but ignores the fact that XP is supposed to adapted to the local situation once the team using it has experienced its synergistic practices. As Kent Beck wrote: "XP is a starting line. It asks the question, 'How little can we do and still build great software?'" - XP does not disallow adding practices appropriate to each team's circumstances.
Ron Jeffries wrote "[T]he twelve practices are intended to be a starting point. Projects who want to know how to start XP are pointed to the twelve practices and told to do them for a couple of iterations and then to modify and adapt them to their local circumstances." in "When is it not XP".
Martin Folwer reminds us that there are principles of XP, not just practices in "Principles Of XP" from which I'll quote the list here:
2003.Oct.09 ThuFundamental Principles:
- Rapid Feedback
- Assume Simplicity
- Incremental Change
- Embracing Change
- Quality Work
Further Principles:
- Teach Learning
- Small Initial Investment
- Play to Win
- Concrete Experiments
- Open, honest Communication
- Work with people's instincts - not against them
- Accepted Responsibility
- Local Adaptation
- Travel Light
- Honest Measurement
Superstitious code is code that works, but the author doesn't know why it works, and doesn't know why alternative pieces of code that should do the same thing don't work. It may also be called "programming by coincidence".
The driving force behind superstitious code is ignorance. In some cases, the programmer doesn't understand programming very well. In other cases, the problem is that there are too many "black boxes" -- too many layers of code that are not available in source form, not well-documented, and which may be buggy. Sometimes the source code is correct, but the compiler is buggy.
I'm seeing a pure Java program that crashes the virtual machine. It crashes under MacOS 9.2.2, using Apple's Java VM (MRJ 2.2.5), on a dual-processor Mac with 1000 MHz G4 CPUs. My code is simple, but it's calling someone else's Java code that is multi-threaded and doing network I/O.
Since JVMs are never supposed to crash, I can assume that the problem lies in MRJ JVM or in one of many other black-boxes: the operating-system, any one of many system extensions, any one of many other processes that are running concurrently. The bug may even lie in the disk-driver software, or in one of many pieces of firmware, such as the firmware on the ethernet card.
And by the way, setting a break-point in my code using a debugger makes the crash go away, and I don't see the problem on a 500 MHz dual-G4 Mac, nor on single-processor Macs. It happens with virtual memory turned on or turned off. (I may also try running it on MacOS X, though this code isn't intended to run on MacOS X.)
MacOS 9 is particularly hard to debug because it doesn't have protected memory -- any running process could write in any other process's memory-space. Multiple-thread code can be hard to debug, because threads are not necessarily scheduled deterministically, and a buggy thread could set up a problem that is not visible until later, possibly in a different thread.
I really admire people who can use a low-level debugger to reverse-engineer and debug multiple layers of black-box code... it not only takes a lot of knowledge, but it also takes a lot of time and effort.
However, I don't have a lot of time, and since Apple held a funeral for MacOS 9, complete with coffin and eulogy by Steve Jobs, I probably can't expect to see Apple fixing MRJ or MacOS 9, even if I identify a bug in their code. So the only solution for this problem may be changing the Java code to work around the JVM's problem, without ever knowing exactly what that problem is. (I'm guessing that a network-call-back function is calling JVM code that has a stale pointer to an object that was garbage collected, but it could be something else.)
Some other time I may write about the misconception that Test-Driven Development is a form of Superstitious Coding.
And isn't it interesting how scientists are opening up the black-boxes of humans, plant, bacteria, viruses, and animals, by examining the source-code -- the DNA -- by poking values into the code and seeing what happens?
2003.Oct.03 FriHow much time do you allocate for learning new things? Have you been told you don't have the budget for training? Learning doesn't require classes and workshops. Jim Little tells his experience with "continuous learning" on the Industrial XP mailing list:
2003.Oct.01 WedBack in 2000, I saw a post John Brewer made to the Extreme Programming mailing list. Every week, he said, his team took half a day to "play." They could use that time for anything they wanted.
I saw that post and thought it was a great idea. I was leading an XP project at the time, so I went to my manager and proposed it. She thought it was a great idea. [...]
Eager to try it out, I told the team that I had convinced management to give us free time. Every Thursday afternoon, we would experiment with any technical concept that interested them. The only rule was that it couldn't be project related. Using spike solutions to demonstrate the concepts was highly recommended.
I thought the team would jump at the opportunity. Instead, they balked. "That's a waste of time," they complained. "We're here to write software for our company, not play."
I pushed the issue anyway, and the team grudgingly agreed. People went off and learned about XML, XSLT, JavaMail, and other technologies. The effort paid off within weeks as we incorporated the new knowledge back into our product. For example, we used our newly acquired JavaMail knowledge to start sending HTML email.
It took a month or so, but people's objections to continuous learning faded. [...]
I think the initial reluctance to play time came more from a sense of being lost. There's so much to learn how do you find something that's really interesting? Half a day isn't really all that much time. Working in pairs helped people get over the hump, as did plain old pushing from the coach. Once people got used to it, they liked it.
I've heard companies claim that they allocate 20% of their time for learning. Half a day every week is 10%. Doing all 20% in one chunk would be ten straight weeks of learning. (Yeah, like that's going to happen.) If you're serious about that much learning, the only practical approach is continuously.
On the XP mailing list, Nancy Van Schooenderwoert described her successful embedded software project that used XP. Her team started out with good (XP-ish) practices and then went full-bore XP.
[...] In 1999 I got my embedded software project staffed at a large company in the Boston area. At the time I knew nothing of XP, but I was determined to apply good practices I'd picked up in the school of hard knocks, as they call it. I instituted strong unit tests for each module, coding standards, source code management, common code ownership, frequent code reviews (more as knowledge sharing than as a police action), solid build control, and iterative development. I was also very conscious of good naming & shared design concepts - i.e. we were using what XP calls 'metaphor'.
This all worked quite well. [...] Our bug rates were very low - about 1 bug per 2000 lines of code. Bugs were tallied at integration test, not unit test. [...]
In spring 2000, I found out about XP and later we tried the Planning Game, Pair Programming, and made more effort at 40-hour week. We worked for smaller releases, and were glad to see Refactoring brought out of the closet (we'd been doing it but feeling guilty because it didn't add features). Our bug rate stayed low - at the very same level in fact. But our releases became much more regular and all the developers learned to estimate.
The key practices we added to deal with the embedded realities were:
- Leave a bread crumb trail - a trouble log that is always "on" so it doesn't distort your troubleshooting by having to be enabled.
- Dual targetting. Make your app run on a desktop as well as on the target CPU to isolate board-specific problems.
- Stand-alone module execution. Each loosely-coupled domain of the app should build and execute alone on either target.
The above 3 practices allowed us to cut through the toughest bugs like a chainsaw! They are incredibly powerful especially when used together. At the start of the project, 3 of the team members had no experience with real-time or embedded software. The team practices had the effect of putting a safety net under everyone, so they could learn without getting hopelessly tripped up (or wrecking the code). I also taught them all I knew about multitasking - another skill that was absent from all the staff except me. The funny thing is that the worst multitasking problem we had was a deadlock that held us up for a few hours, and it was my bug.
Ron Morsicato (a team member) and I wrote an article for Cutter's IT Journal last year about our experience. Here's a link (it was mentioned in a previous posting by Charles Poole) -- http://www.cutter.com/itjournal/xp.html
Phlip adds his comments:
2003.Sep.30 Tue
Note what they did, compared to the "traditional" way:
You used more code to solve the root problems, not cause more problems. You did that by obeying the emergent principles - when in doubt test; when testing is hard make it easy.
These are more important than the grunt work of writing production code. Some bosses see all the bugs, and decide that nobody should write >more< code. They only predict that infrastructure would bring its own burden.
Thanks to Martin Fowler for providing a link to his description of Jim Johnson's speech at the XP2002 conference, vaguely mentioned in my previous blog entry. Fowler wrote:
Johnson's comparison of two SACWIS systems. SACWIS is a child welfare system that all the states in the US must implement. He stated that Florida began its SACWIS system in 1990 with an original cost estimate of $32 million for delivery in 1998 with a development team of around a 100 people. When they last looked the new cost was $170 million due to ship in 2005. He pointed out the state of Minnesota has to build the same capability with pretty much the same demographic issues as Florida - leading to very much the same broad system requirements. Minnesota started work in 1999, finished in 2000 with eight people costing $1.1 million.Measuring by effort, 16 person-years versus 1500 (projected) person-years, shows the Minnesota project was 93 times more productive.
James A. Robertson links to a comparison of two implementations of an example web application. It compares source code rather than effort... the Java/Struts sources are six times larger than the Smalltalk implementation, and have three times more classes. The author of the Smalltalk implementation says "I do recall the folks at the users group saying that they typically spent about 5 to 10 times as long to do wafer in the Java based frameworks (1 day vs 1 - 2 weeks)".
2003.Sep.29 MonMartin Fowler has blogged on this subject, saying that we can talk sensibly about the effectiveness of tools and methodologies because "we can't measure the productivity of development teams. Strictly the problem is that we can't measure their output - how much stuff gets cranked out."
I suggest we can measure productivity, but only at a higher level: how much time and effort does it take to satisfy the customer? Not just for the initial product, but also for changes to the product.
For example, if the customer only needs a simple word processor, and team A delivers a simple one like MacWrite in three weeks, and team B delivers a complicated one like current versions of MS Word in three months, and both satisfy the customer, I would say that team A was more productive, regardless of number of lines of code generated or the number of function points implemented.
I'd like to quote the example of two states' implementations of the same federally-mandated tracking system (Florida and Minnesota?), where one was developed in two years or less with less than a dozen people, and the other (over-budget) one was predicted to take five or more years and employ up to a hundred people, but I can't find a link. I think it was first mentioned in a keynote speech of an agile conference.
2003.Sep.22 MonSome quotes from a MacIntouch page on justifying buying Macs...
Converting from Windows to Mac server:
Actually a 100% Windows client company that I'm familiar with went the other way. They bought a Mac OS X Server to replace their farm of Windows servers. If "it all comes down to $", this was much cheaper as X Server has an unlimited license version that is more cost-effective than the Microsoft offerings. Windows clients seem to work as well with the Mac server as with a Windows server.
Their previous Windows servers crashed regularly, caught viruses on occasion, were rather slow and cost a lot of money and time to buy and maintain. Patches every week and constant trouble calls from users. This kept the admin more than well employed.
The single Mac server that replaced them is blindingly fast, can't catch Windows viruses and hasn't been rebooted in all the months since it was installed. The admin now only has maintenance on the Windows client machines; the server requires little more than changing backup tapes....
"Today there is not a single known virus on the Mac OS X -platform"
Virus woes
I teach at the University of Texas in Austin. Recently at a faculty meeting we heard a report from our school's computer support director. She was relating the experience of her department in the weeks before school started, when the university was plagued with the now-familiar series of Windows-related viruses and worms.
It was a nearly unbelievable tale of woe and hair-tearing frustration....
For a couple of weeks, it was a common sight to see computer techs rolling around large carts filled with PCs headed for "remediation."
The head of IT services later told me that another weird twist to this tale is that students, and sometimes parents, who brought a computer from home and then started using the campus network with that computer, would reintroduce a virus/worm to the campus network. And sometimes these computers were wireless, which meant that the IT services techs were literally chasing users around a building, up and down stairs or elevators, trying to find the infected machine and get it off the network.
Our school's support tech said to me, "If it weren't for the fact that a third of our computers are Macs, I would have jumped out a window."
None of the Macs on campus needed to be "remediated." Zero. After I heard this, I asked my fellow faculty members, "Why does anyone still use Windows?" Moreover, for institutions that are facing budget constraints and a shortage of trained personnel, why on earth would any institution think that it's fiscally responsible to depend on Windows machines?
Homogeneous computing is not only stupid, it's irresponsible.
"If you look at the IT departments at prestigious private schools -- Harvard, for instance -- standardization is a dirty word."
InfoWorld's Tom Yager: "Windows doesn't live here anymore" The client switch away from Windows is permanent, and the server switch is next.... I am turning away from two core assumptions: that all x86 machines are destined to run Windows and that all worthwhile entry-level servers, business desktops, and serious notebooks use Intel CPUs. "
Cost of support
I am an "army of one" supporting 20 Windows PC's, 67 Macs for teachers, and 64 iBook laptops for students. For the past two years I have documented my service calls and support issues and I have found the split to be almost exactly 50/50. This means that 20 Windows PC's require as much support as 131 macs.Regarding Microsoft Access (the only part of Microsoft Office that isn't available on the MacOS X platform)
2003.Sep.21 SunAbsolutely nothing has 100% compatibility with Access, not even Microsoft's own Visual Fox Pro for developers. Access is the worst database I have used. It has no clean migration pathway to any other database application. You can transfer data by exporting Access database tables, but you can not transfer Access forms, macros, or Visual Basic customizations. If you jump through some hoops, you can convert Access queries to SQL and copy them to another database. That's it.
If you need an easy to use, crossplatform database, I recommend FileMaker Pro (relational) or Panorama....
I looked up Kaizen Events because of a blog entry by Joe Ely. He emphasizes that kaizen events take practice, must adapt to local conditions. They reshape expectations about change, and help you re-think your job. He says that goal clarity is critical, simple real-time documentation is essential, and visual tools are critical too.
Google finds this definition:
The Kaizen method is a "rapid improvement process" utilizing a cross-functioning group of managers and employees working as a team to meet targets in a results-oriented focus on a predefined project area. The process may take the following steps: define the problem/opportunity, choose the best people, and correct the problem in one week or less using Kaizen tools and techniques. The ultimate goal is to significantly reduce costs, reduce lead times, reduce required inventory space, enhance workforce empowerment, eliminate waste, and focus on continuous improvement. The Kaizen process may include: new product development, robotics, total quality control, Just-in-Time, statistical quality control, labor and management relations, or other concepts.Another page claims these results from Kaizen Events
- 30-50% Productivity Improvements
- 30-50% Reductions in Floor Space Requirements
- 50-70% Quality Improvements
- 70-80% Reductions in WIP [Work-In-Process] Inventory
- 40-50% Reductions in Lead Time
Of course, those results came from improving conditions and processes of manufacturing (not the processes of designing and engineering), but I think most typical software processes, from requirements gathering to delivery, could obtain a 50% gain in productivity and an even higher reduction in defects by adapting a lean / agile software development process.
How would a company do this? Perhaps by starting with a facilitated retrospective. Maybe followed or preceded by a few observant people walking through the entire requirements-to-delivery process - counting how many information-lossy hand-offs there are between workers, how often work sits in queues waiting for actions or approvals, and other efficiency-killers.
Just found another definition, here:
2003.Sep.20 SatKaizen Event
Any action whose output is intended to be an improvement to an existing process. Kaizen Events are commonly refered to as a tool that:
- Gathers operators, managers, and owners of a process in one place
- Maps the existing process (using a deployment flowchart, in most cases)
- Improves on the existing process
- Solicits buy-in from all parties related to the process
Kaizen Events are an extremely efficient to quickly improve a process with a low Sigma score. Kaizen Events are also useful for convincing organizations new to Six Sigma of the methodology's value.
The true intent of a kaizen event is to hold small events attended by the owners and operators of a process to make improvements to that process which are within the scope of the process participants
Science Daily: "Testing Information Systems During Development Will Prevent Problems"
A story at http://www.sciencedaily.com/releases/2003/09/030918093222.htm> says:
"Rather than waiting to test projects until they are completed, evaluating them at increments can prevent such troubles. That can keep projects from having cost overruns and needing time extensions."Unfortunately, this news report is vague, mostly talking about metrics:
"The researchers developed a mathematical framework that groups metrics around the stages of project requirements; design specifications; implementation; and operation in the software system's intended environment. The framework revealed that some metrics are applicable only at certain product stages while others are applicable at multiple times. The researchers also realized few metrics exist for measuring the processes employed and resources expended during software development."The original news release is here: http://www.psu.edu/ur/2003/testinginfosystems.html. The researchers' results are published in "Product Metrics for Object-Oriented Systems," ACM Computing Surveys, June edition.
Some other news releases at Penn State:
2003.Sep.15 MonMarathons and Software Development
Dave Hoover has a nice comparison of running/training for marathons and software development at http://www.redsquirrel.com/dave/work/marathon.html.
2003.Sep.13 SatOn the subject of separating code into loosely-coupled components... see PicoContainer. (This may be ancient history [June 2003] for some people.) Some quotes:
"PicoContainer is very simple container for very simple components. It honors the Inversion of control pattern (IoC) in a way that we calling it type 3 IoC. [...] The idea is that this might scale from embedded containers for simple beans to enterprise and distributed applications.""Inversion of Control: What is this all about? IoC is a design pattern that is also sometimes called the Hollywood Pattern ("Don't call us we'll call you") or the Chain of Command pattern. Simply put, an IoC component does not go off and get other components that it needs on instantiation. It instead declares them to its boss, and the boss supplies them.""Pico components (to remind you) are simple java classes that declare their dependencies in the constructor(s). No meta XML for dependencies. No classes/interfaces to extend/implement."Note that the link to composite components is incorrect, it should link to http://www.picocontainer.org/compositecomponent.html
A blogger named Rickard wrote:
"PicoContainer: Here's something on the bright side. I just found the PicoContainer/NanoContainer projects, and to me they look like the perfect way to build loosely coupled component-oriented architectures. It's just bloody brilliant. WebWork used the type 2 IoC, and XWork currently does too, but they always felt a little awkward due to the extra coding needed. But this type 3 IoC stuff is just the way it should be. Nice and tidy, with flexibility for the implementation, and supports unit testing quite nicely. Just bloody brilliant. I can't wait to toss out our own kludgey service architecture in SiteVision in favor of this."Rickard has an example of the usefulness of constructor-specified inversion of control here.
Last word: I was expecting a paradigm shift, and all I got was a lousy constructor
2003.Sep.06 SatIf you have insights about the following ways of organizing and documenting code, or recommendations for reading about these subjects, please email me. How do you structure your source code or otherwise document the following aspects of your projects?
separating libraries, packages, or modules for dependency management hierarchical decomposition of applications, modules, libraries, etc. multiple teams dividing up a project into modules, libraries, packages, etc. categorizing packages, libraries, modules and classes in non-exclusive categories documenting data-flow across modules, applications, etc. I have read Lakos's book on Large Scale C++ Design and his recommendation for acyclic dependancies of packages and bottom-up implementation and testing of packages.
I've also read Robert Martin's book on Agile Software Development that has also recommends acyclic dependencies of classes and packages, and has a chapter on package design for managing dependences.
There's hasn't been much written about "refactoring to packages" and I'd like to know why.
2003.Sep.03 WedBook Review: Extreme Programming Refactored
See Ron Jeffries' review here: http://www.xprogramming.com/xpmag/books20030904.htm.
Some quotes:
2003.Sep.01 MonIt is dangerous to read for anyone who does not understand what XP is, because the authors have gone to such pains not to understand, to take out of context, and to distort.
A book satirizing XP might have been amusing to some people, but would have been informative to none. A book addressing XP's limitations and how to deal with them would have been valuable to a wide range of people and might even have been made entertaining as a side effect. The intimate mixture of satire, error, and somewhat good advice is unmanageable by anyone who doesn't already know the material, and largely unnecessary for anyone who does.
Almost half the book relies on demolishing the reputation of the C3 project through mockery and satire. Unfortunately, much of the discussion is as poorly researched [...].
The authors raise one of the most important issues with XP, namely that it won't work if you don't do it. As far as I know, no process will work if you don't do it, including the authors' own favorite. While they are quick to point out, and rightly, that a so-called XP project will go off the rails if the team doesn't do XP, they do not address the same issue for other proposed processes and practices.
For more valuable discussion of the limitations of XP and what to do about them, I would recommend either Pete McBreen's Questioning Extreme Programming, or Barry Boehm and Richard Turner's Balancing Agility and Discipline.
Marco Abis on the XP mailing list mentions a message-thread on RUP and Scrum/XP on the Scrum mailing list... here are some snippets:
Ken Schwaber: "Let us hypothesize that having a Scrum plug-in for RUP is an idea worth pursuing.[...] Is this type of metaphor appropriate for agile processes, or does this level of delineation lead to them being fodder for M/S project,for 'hands-off' management, and for robotic tracking of plans while ignoring realities?"
Ron Jeffries: "The comments from Rational clearly did not reflect the soul [of XP]. They do not get 'agile' in my opinion.[...] Their philosophy, again in my opinion, is based in large-systems experience, old-style 'up front' thinking, and a command and control orientation.[...] My involvement with the RUP XP plugin was based on my belief that it would be better with me than without me, and I think it is. Even so, it is still a long way from expressing what XP is."
Adriano Comai: "having been involved in many RUP customizations, I think RUP_Scrum makes sense (much more, in my opinion, than RUP-XP). RUP needs _exactly_ the Scrum practices to become agile. As you know, many shops with an existing waterfall culture buy RUP as an help to move towards a more effective development approach, and then misuse it."
Alan Shalloway: "You can take a committed dev group (most are) and have them do RUP/Scrum in a way that an upper management that doesn't want to hear about agile will accept.[...] here's a chapter discussing this I've just put on our public wiki (Dan Rawsthorne and I are writing a book on effective software development and this is a likely chapter). See http://www.netobjectivesbooks.com/N_O_BookFeedback_Wiki/owbase/ow.asp?DontTryToSellXP"
Marco Abis: " will try to explain why I think so (after years of effective RUP implementations!): RUP is fundamentally a very, very big framework [...] I will no spend time to write why managers like RUP but Scrum [...] makes me think that what managers want is something that isn't really useful to deliver the right project and often is the first impediment to the project success."
Adriano Comai: "Sometimes it's difficult to tell these organizations, who made huge investments into RUP: throw it off, there are more effective approaches out there, try them instead. In such cases, I believe, it is better to suggest an improvement using agile practices in the context of the existing process."
Marco Abis: "I experienced a lot of clients (of mine, of course) professing they were implementing an "Agile version" of RUP but when I analysed their work I found they were just cutting off some documents feeling "more lightweight". I definitly think this is one of the main issues: a lot of people believe Agile means just less docs, they try to cut off some of their docs and then affirm Agile is nothing more than this. As stated by Jim [Highsmith]: they are missing fundamental understanding of the unpredictability of project activities. [...] For those who haven't read it I suggest 'Agile Revolution' by Mike Beedle.
Marco referred to http://www.crystalmethodologies.org/cgi/wiki?AgileVsSelfAdapting, from which I quote Jim Highsmith: "So self-adapting, by itself isn't agile." and Alistair Cockburn: "criticisms of RUP [...] Doing the change case is very laborious. Just heard of an airline company that wants to use RUP, so they spent 6 months creating a RUP-for-them. Now every project has to spend 2 months turning that into a RUP-for-the-project. Problem is when the project is only 6 months long, this is too long."
Another Highsmith quote on that page: "But lightweight is only one aspect of agile, the other two are collaborative values and principles and a fundamental understanding of the unpredictability of project activities (although outcomes can be achieved). Agile is much more than lighter documentation and fewer processes. Rational is trying to have it both ways--still telling corporate managers what they want to hear in terms of predictability, control, productivity, etc. while at the same time touting that they are 'agile' also."
2003.Aug.31 SunI'm stating the obvious here, but it needs to be said: there is learning by doing, and learning by reading. This is the core, I believe, of the perpetual re-implementation of ideas in the software industry, and it is the bane of those trying to port software rather than re-implement it. (By coincidence, Joe Ely also writes about "the Knowing-Doing Gap" on his "Learning About Lean" blog.)
Consider that the first portable virtual machine environment and programming language designed to "run anywhere" (that I know of) was the UCSD P-System, an extended Pascal language that compiled to byte-code, a byte-code interpreter, and various other tools that ran on several machines in the early 80's including the Apple IIe. Later there was Smalltalk and its virtual machine, now there is Java and other languages on the JVM, C# and other languages on dot-net virtual machine, Perl, Python and Ruby on their own virtual machines. Lots of re-implementations of the same basic concepts, each with their own variations.
Back to reading and doing... This difference between learning-by-doing and learning-by-reading is the reason software keeps being re-invented. If I'm confronted by a bunch of code, even if it is moderately well-written and documented, I'm only reading it. My understanding will not be as deep as it would be if I had written it myself, or participated in writing it. For this reason, my recommendation to anyone confused by a test framework like JUnit or CppUnit, would be to study the documentation about JUnit, and then write your own test framework from scratch.
Consider an analogy: In learning by reading, I can read about how to rebuild an automobile engine, but that isn't going to give me the skills to actually do it. In learning by doing, I can have someone guide me through the steps of rebuilding several automobile engines, until I'm able to do it by myself, even with an engine-design similar but not necessarily identical to the ones I've practiced on.
To redesign an engine, say to burn a different kind of fuel, to fit into a different space, or to use a different number of pistons, the book-learning is far from sufficient, though books on engine theory could supplement the learning-by-doing of building and rebuilding engines. I would need know how to design engines in general, to be able to redesign an engine in particular.
Now when we write software, we learn a lot of things, not only about our design and the domain, but also about the environment our design has to live in. Particularly the "corner cases" where the platform dictates aspects of the design. What happens when you want to port that software to another platform?
In the best case, you actually implement your project on all the desired platforms at once. Just add "the integration is not complete until all tests pass on all platforms" to Extreme Programming's "Continuous Integration" and "Testing" practices. You end up learning about all the corner cases of all the target platforms by doing the work necessary to complete a story or task on all the platforms. You do this a little bit every day.
In the worst case, one team developed software on platform A, and another team has to port the software to platform B, and the B-team doesn't have the A-team helping them understand their code. The code whose design was influenced by the "corner-cases" of platform A's environment.
The B-team reads the code, but since they didn't write it, they can't easily tell which parts are of the originally-intended design, and which parts had their design modified by the target environment. The B-team has to know platform A well enough to figure that out (which may require more in-depth knowledge of platform A than the original coders had). The B-team also has to know platform B well enough to modify the design to conform to the "corner-cases" of platform B. That's a whole lot of knowledge required, and a lot of it has to come from doing, because reading isn't enough -- much of the "common knowledge" of programming for a platform isn't written down anywhere.
The B-team really needs to know more than just the source-code. They also need to know the requirements because sometimes reading the code isn't always enough to figure out what requirement it was trying to implement. (And the code could be incorrect -- not fulfilling a requirement.)
If the software hadn't been tested thoroughly, some of the code (individual functions, for example) may not even work properly for all inputs/situations on platform A. Trying to get it to work properly on platform B may not even be necessary, but reading the code may not answer that question. Trying to figure out if the original programmer intended some "weird" behavior, or was just having a bad coding day, can be frustrating.
In a non-test-first code-base, there is probably dead code that isn't actually called, and thus porting it is unnecessary. There is probably code that was prematurely generalized, and thus only specific cases need to be made to work on the second platform. There is probably code that was prematurely optimized, or correctly-optimized for platform A, but which needs a different form to be correctly-optimized for platform B.
The porting job can be made much easier if we have programmer-tests that came from Test-Driven Development. Test code is usually pretty simple compared to the code-under-test, so it is usually easy to port. If you can get the programmer tests ported over to the target environment, then porting the code-under-test is optional: you could instead write fresh code to satisfy all the tests. You would mostly likely be implementing a similar design, but you would not be having to concern yourself with the corner-cases of platform A, only those of platform B. This means that you don't have to understand the corner-cases of platform A, except as they might be reflected in the tests. If you do port the code-under-test, getting it to pass the tests in platform B can reveal the corner-cases of platform B, in a testing situation where the code under test is isolated and "exposed" and therefor easier to debug.
Some might say "write more documentation." But that relies too much on learning by reading again. The programmer tests and acceptance tests actually execute and demonstrate pass/failure. You can "do" the tests by running them. Documentation is passive, can be misleading, and rarely is there enough detail to make porting software easy. Test-driven development provides a level of detail that was essential to write the software, and was painless to write, but would have been mind-deadeningly too massive to write in a non-executable (non-automatically-validating) prose format.
To sum up: writing fresh code on platform A only requires knowing the requirements and the foibles of platform A. Ditto for platform B. Implementing the requirements helps you learn the problem domain, and to learn the target platform. To port code from A to B requires knowing the requirements, the design, the problem domain, the foibles of platform A, and the foibles of platform B, which adds up to too much.
2003.Aug.30 SatIn the XP community, names are often re-thought, in an attempt to more clearly and succinctly convey their true meaning. So "Unit Tests" evolved into "Programmer Tests" and sometimes "Test Driven Development Tests" (my preference leans to "Test Driven Design"). The latest phrasings seems to be "code-guiding examples" and "checked examples" as well as "change-detectors".
(Tim Bacon summarizes many of the names here.)
I admit that the word "tests" seems to evoke something counter-productive when trying to discuss or teach Test Driven Development. I was trying to explain TDD to someone yesterday, and he started objecting about the masses of data his tests would have to shovel through his code. I explained that if his code can handle one, two, and three pieces of data, he would not necessarily need to write additional tests for a hundred or a thousand... and the point of the tests in TDD is to drive the design, not to test the code.
As precise as we want our jargon to be, we will have to live with defining our terms. Consider that "rose", according to http://dictionary.reference.com/search?q=rose has more than 48 definitions, if you include the verb "rise". See also "Rose is Rose".
2003.Aug.23 SatNice example of Test-Driven Development in Ruby: http://jimweirich.umlcoop.net/articles/tdddemo/index.html.
I'm not a big fan of Ruby. I'd rather have a method named "isFoo" than "foo?", and member variables named "bar" (or "mBar", or "myBar", depending on coding standard) instead of "@bar".
One problem of small examples of TDD is that people look at the tiny steps being taken and assume TDD will be very tedious. In real life, your steps are most likely going to be bigger than these small examples, and you will have lots of other (already-TestDriven) code to build on.
2003.Aug.22 FriWireless networks, palm and laptop computers, and internet chat are beginning to change the lecture experience for the audience, and soon, the presenter. Audience members can chat with each other without disturbing the speaker, commenting on or refuting what the speaker says in real-time.
Apparently some conferences have also enabled the chat to be displayed on a screen behind or in front of the speaker. Kevin Marks refers to a blog entry describing his own "heckling" of a speaker. Kevin Marks has a "HeckleBot" an LED screen for displaying live text, with internet/web access.
Quotes:
2003.Aug.18 MonAn excerpt from Conferenza, which provides a tad more traditional paid research coverage of trade shows, contains this golden nugget of controversy:
Asserting that the largest e-commerce software supplier is Amazon.com, Benioff pointed toward co-panelists from IBM and Sun Microsystems and said, "None of these companies has any position in [that] market at all. Even Apple's iTunes music store was built on Amazon," [....]
We thought this was news, until [...] an online chat charge showing that most of this was apparently untrue: Amazon uses standard XML out-of-the-box stuff, and Apple's iTunes doesn't use Amazon's software at all, the chatters charged. As Benioff continued, the audience watched as a group of online contributors disputed fact after fact, input Benioff apparently did not see. "It was sort of like a 'Saturday Night Live' skit," said one attendee. "As Mark spoke, we could see his nose growing longer, like Pinocchio."
How it played out in the Chat (Archive) was Kevin Marks did the fact checking, which was simultaneously projected on to the big screen:
[11:51] KevinMarks: no he didn't [...] [11:51] KevinMarks: he licensed the patent [11:51] KevinMarks: iTunes backend is not Amazon [...] [11:51] Ross: Amazon's real smart move was an API for developers [...] [11:52] Ross: but they dont get decentralization. witness http://www.allconsuming.com [...] [11:52] KevinMarks: Apple had ahuge online store already selling Macs [11:52] KevinMarks: they built on that for iTunes [...] [11:54] DariusD: Do you know that the Apple onnline store was not built on Amazon technology? [11:54] KevinMarks: It is built on WebobjectsLaurent wrote about Lost In La Mancha, the documentary about Terry Gilliam unsuccessfully trying to make a movie. I have only watched the first hour so far, but I can see lots of awareness of potential and actual problems, but no addressing of those problems, reminding me of that people usually know that their project is doomed from the start.
I rented and saw American Movie, another documentary about making a movie. In the case of American Movie, the director/writer Mark Borchardt actually achieved a kind of low-budget success -- he managed, in the course of two or three years, to finish a short film and show it at a theater to a sold-out audience. He probably didn't make a profit, and he and his unskilled laborers were probably unpaid, but he was able to film and re-film, record sound, and re-record sound, edit, and re-edit, while having a day job delivering newspapers and/or working at a cemetery. The documentary is fascinating. The short film that Borchardt created, "Coven", is included on the DVD and is comparable to the films of Ed Wood, but is probably even worse, though quite fascinating if used to psycho-analyze Borchardt.
It was quite the movie weekend. I also rented the Jackie Chan movie Shanghai Knights and listened to the director's commentary and the writer's commentary AND read Jackie Chan's on-line diary he wrote during the filming. One thing about this movie, which is better than Shanghai Noon, is that the director allowed Chan more time to choreograph and film the fight-sequences that Chan is famous for.
Unlike the rest of the movie - very much planned up-front, with a lot of parallel activities - the fight scenes are choreographed and filmed in a "agile/just-in-time" manner: Chan would make up a short part of the fight sequence, rehearse it with the other actors/stunt-men, instruct the directory/camera-man where to position the camera for that sequence, film it, and then move on the next bit.
This movie was an American production, filmed mostly in the Czech Republic, and the differences in process from a Hong Kong film continue to amaze Jackie Chan -- the enormous amounts of catered food and time off for lunches, the expense in set-building/special effects/costuming, shooting in two units to reduce calendar time, and the reasonable working hours. For the sake of one (not that funny) joke, they created a replica of Stonehenge in a field somewhere in Czech Republic. It was probably made of styrofoam, but still that means a day or two of filming (and having to cope with rain) - one pretty costly joke that a Hong Kong or independent movie-maker could not have afforded.
2003.Aug.08 FriErik Benson points to a page on Flash Mobs. This is apparently derived a concept from science fiction, being made real (in a way).
The original concept, described in a short story "Flash Crowds" by Larry Niven, was that live news reporting of some event, combined with instantaneous, free transportation, would result in huge crowds arriving at the location of the event while it was being reported.
In Flash Mobs, people use the internet to announce a time and location, and people show up, perform some attention-getting non-political activities, and then leave. Apparently a flash mob in Boston attracted a lot of attention from the police.
While these "performance-art" flash mobs are innocuous, I am thinking that police have a good reason to worry -- a mob with terrorist intentions could organize, appear somewhere, do something damaging, and then disappear quickly. Such a group could do something very damaging in the USA, where we have a large supply of easily-concealed hand-guns.
See also the slash-dot effect. Thanks to google and wikipedia for letting me find the Niven reference without having to search my bookcase.
2003.Aug.06 WedJoshua Kerievsky writes on the IndustrialXP mailing list:
I'm reading Tom Gilb's Principles of Software Engineering Management. I came across the following principle, which reminded me of some discussions we've had here:
The Juicy Bits First Principle, by Tom Gilb: "If you deliver the juiciest bits of the project first, you will be forgiven for not providing all they dreamt about, or for not doing it as cheaply and quickly as they hoped."
Laurent Bossavit blogged about this concept, which read like a typical XP success story... I initially didn't notice that he was quoting Jerry Weinberg's 1988 book Rethinking Systems Analysis and Design.
XP requires a person or sub-team to prioritize requirements and calls that person or sub-team the "Customer". Ideally the customer is the person paying for this project, because they are the most motivated to prioritize requirements accurately. Often, the person playing the customer role is a project manager, product manager, business analyst, or some other job title. Successful projects have someone doing this job, whether they call their process XP or not.
If you structure your project in such a way that you spend the first 80% of your project duration developing "infrastructure" (whether that is back-end code without a user-interface, or user-interface code without a back-end, or anything else that isn't fully functional part of the application from the user's perspective) and only complete the user-visible requirements in the last 20% of the project duration, you will not be able to deliver just the "juicy bits" of a project early.
XP's requirement to build releasable software in every iteration forces you to develop just enough infrastructure per requirement -- probably less infrastructure than a plan-driven approach would do -- saving time and avoiding wasted work, and getting those juicy bits to the real customer sooner.
2003.Aug.01 FriThanks to Ron Jeffries, William E. Caputo, and others on the XP mailing list for looking this up: a segment on Marketplace broadcast on Public Radio -- "Overtime's not good for your health."/p>
Researchers at the University of Arkansas are set to report that the number of hours worked has a very small effect on life satisfaction, sick days and worker stress [Note that this ignores productivity or the impact of long hours on the quality of the work being done.] -- but don't try selling that to commentator Joe Robinson, who has seen some other studies.... "Robinson, author of the book Work To Live says that a Japanese study of overtime shows that 60-hour work weeks can double or triple the risk of heart attacks. Over the last 30 years, the Japanese have recorded thirty thousand deaths from overwork. The USA has less vacation time than the Japanese, but we don't record deaths from overwork, if we did, it could be a very scary number. Quoting from the broadcast:
"Almost 40% of us, now, work more than 50 hours a week, the vast majority of it completely counterproductive.... [More hours are not better,] neither for your arteries, nor for your productivity. A study by the Business Roundtable found that excess overtime not only reduces productivity during the excess hours, but during regular hours, too... People working seven 50-hour weeks don't get any more work done than people working seven 40-hour weeks... We actually work better when we're rested, not fried. Long hours are a tragic failure of the knowledge economy."Listen (using RealOnePlayer) to the link half-way down the page at http://www.marketplace.org/shows/2003/07/31_mpp.html
More info here: http://www.worktolive.info/index.html, such as: "MRI images of fatigued brains look exactly like ones that are sound asleep."
2003.Jul.29 TueI've been on vacation and otherwise occupied recently, so I'm catching up. I've got more than 70 emails to read, SHAPE forum, AYE wiki, and these blogs I'm reading this morning:
2003.Jul.12 Sat
- http://erikbenson.com/index.cgi
- http://www.testing.com/cgi-bin/blog
- http://www.fortune.com/fortune/technology/articles/0,15114,466180,00.html (referenced by http://www.cincomsmalltalk.com/blog/blogView )
- http://www.jrothman.com/weblog/htpblogger.html
- http://bossavit.com/thoughts/
- http://joeelylean.blogspot.com/
- http://www.jrothman.com/weblog/blogger.html
- http://martinfowler.com/bliki/
- http://www.estherderby.com/weblog/blogger.html
- http://www.artima.com/forums/flat.jsp?forum=106&thread=7827
- http://www.artima.com/forums/flat.jsp?forum=106&thread=8142
- http://www.artima.com/forums/flat.jsp?forum=106&thread=8248
- http://www.artima.com/forums/flat.jsp?forum=106&thread=8098
- http://www.artima.com/forums/flat.jsp?forum=106&thread=7852
- http://www.artima.com/forums/flat.jsp?forum=106&thread=7600
- http://www.artima.com/forums/flat.jsp?forum=106&thread=7590
- http://www.artima.com/forums/flat.jsp?forum=106&thread=7351
- http://www.artima.com/forums/flat.jsp?forum=106&thread=8216
- http://www.artima.com/forums/flat.jsp?forum=106&thread=7993
- http://www.artima.com/forums/flat.jsp?forum=106&thread=8343
- http://www.pycs.net/bbum/
Spaghetti Objects and Packages
I'm not a fan of "anti-patterns", but sometimes seeing badly designed classes and packages helps you appreciate well-designed classes and packages. About the only source of advice on package design that I've seen appears in Robert Martin's excellent book Agile Software Development: Principles, Patterns, and Practices, which also has good advice on class design and test-first programming.
We often are advised to have cohesive classes, and it isn't too hard to figure out that a method which doesn't reference any member variables of its own class is a candidate for belonging to another class. This and other heuristics help us break down a too-large class into smaller, more cohesive classes.
Sometimes you have lots of cohesive classes, but they depend on each other in a tangle... it is particularly nasty if the tangle has cyclic dependencies. Robert Martin's tool for untangling dependent classes is the "Dependancy Inversion Principle" [pdf]. Classes are cohesive and independent if you only have to change one class when you want to change one thing.
An aside: Of course, with manifestly-typed languages like Java, changing one thing (like changing coordinate values from ints to floats) could bubble up and across many classes purely for keeping the compiler happy. Advocates of dynamically-typed languages say that manifestly-typed languages promote leaky abstractions because of this need to keep the compiler happy.
Perhaps even worse than classes tangled within one package, is having your packages tangled. Ouch, that hurts! Martin helps us figure out what constitutes a cohesive package by asking us to think of a package as a unit of release - what is the smallest group of classes that has to be re-released if one thing changes? This can lead to small packages, which is probably better than the alternative.
Of course, just as tangled and cyclic class dependencies make classes hard to modify or even compile, tangled and cyclic package dependencies make packages hard to modify or compile, and hard to use in isolation. The dependency inversion principle again helps break these dependency traps.
Imagine that an application (which is one of your packages) depends directly on fourteen other packages. If ANY of those other packages changes, you have to (at the minimum) rebuild your application, and potentially modify it. If each of those other packages depend on five other packages, you may have to recompile your application if any of 84 packages changed.
By using principles of framework design (the "don't call us, we'll call you" principle) and the Dependency Inversion Principle, as well as the other good advice in Robert Martin's book, you can break the nightmare of spaghetti package design.
2003.Jul.11 FriI picked up a tape of the Mike Wallace Interviews of Frank Lloyd Wright the other day. It consists of two 25-minute interviews that were recorded around 1957, when Wright was around 88 years old. The first interview says a LOT about 1950's art of "the celebrity TV interview" and not much about Wright. The second interview was better, but neither one is the place to go to hear Wright say anything specific about any of his buildings.
"My name is Mike Wallace; the cigarette is Philip Morris". In the first interview, Wallace is apparently trying to get to know "Wright, the man" by asking controversial questions like "what is your opinion on organized Christianity?" and "What do you think about mercy killing?" and questions about the "Mobocracy". Wright answers the questions very carefully and reasonably, well aware that he's on nation-wide TV. Unlike modern-day interviews, the interviewee isn't prompted or allowed to tell stories about his life.
Sometimes the first interview degenerates into asking about other celebrities: "What do you think about Salvador Dali?" and Picasso? Wallace tries to get Wright to say something about Charlie Chaplin's "anti-Americanism" (Chaplin was forced to leave the US because of accusations of Communistic tendencies), but changes the subject when Wright mentions McCarthyism. The last question of the first interview is: what do you think of Marilyn Monroe's architecture?
Quote from the first interview:
"I don't think Architecture is for the Mob, it certainly isn't for education. Education certainly knows nothing of it. And very few architects in the world know anything about it. I've been accused of saying I was the greatest architect in the world, and if I had said so, I don't think it would be very arrogant because I don't believe there are many, if any. For 500 years architecture has been phony... in the sense that it was not innate, it was not organic, it didn't have the character of Nature."
In the second interview, (several months later, back by popular demand) Wright is more comfortable, and can express his philosophy at more length. Still almost nothing "concrete" about architecture, though.
Quotes from FLW in the second interview:
2003.Jul.07 Mon"I would like to make architecture appropriate to the Declaration of Independence, to the center-line of our freedom; I would like to have a free architecture. I would like to have architecture that belong to where you see it standing, an architecture that was a grace to the landscape rather than a disgrace."
[The New York skyline] "never was planned; it's all a race for rent, and it is a great monument I think to the power of money and greed, trying to substitute money for ideas. I don't see an idea in the whole thing anywhere, do you?"
Answering "give me something to live by" (the question young students ask of Wright): "The answer is within yourself. In the nature of thing that you represent as yourself. Jesus said it, I think, when he said the kingdom of God is within you. That's where Architecture lies, that's where Humanity lies, that's where the future we're going to have lies. If we're ever going to amount to anything, it's there, now, and all we have to do is develop it."
Is "passive" the opposite of "aggressive"? In software development, is "disciplined" the opposite of "agile"? I don't think so.
A passive person may be inactive, but still denies or resists others and context, just like an aggressive person. (Virginia Satir "self, other, context" concepts.) A healthier alternative to passive is "accepting" -- not active, but not in denial. A healthier alternative to aggressive is "assertive" - actively protecting one's self, but not denying the rights of others nor denying the reality of the context.
Dr. Barry Boehm (with Richard Turner) has a book coming out titled Balancing Agility and Discipline: A Guide for the Perplexed. The title does not reassure me that Boehm really understands agile development (in spite of the fact that he invented the "Spiral Development" model - an incremental/iterative process). We know that XP is the one of most disciplined of the agile methods, and that no process will work well with undisciplined people. The blurb from the back cover sounds better than what the title suggests:
Agile and disciplined: These apparently opposite attributes are, in fact, complementary values in software development. Plan-driven developers must also be agile; nimble developers must also be disciplined. The key to success is finding the right balance between the two, which will vary from project to project according to the circumstances and risks involved.Fine, but then it goes back to saying these are opposites:
Developers, pulled toward opposite ends by impassioned arguments, ultimately must learn how to give each value its due.... The authors describe a day in the life of developers who live on one side or the other. They expose the bureaucracy and stagnation that mark discipline without agility, and they liken agility without discipline to unbridled and fruitless enthusiasm.Alistair Cockburn, on the XP Mailing list, says this about "agile":
"Agile" is a value-set, a prioritization of whatever you think it means over other things.
Therefore, it is not so meaningful to ask, "Can agile be used in this situation?" Rather, the question is, "How agile can we be in this situation --- How can we be agile in this situation?" ... and then you get various interesting answers.
Picking up the theme of the value sets from my previous append, one might look at a missile guidance system and say, "time to market not crucial. people-centric not crucial. cost not crucial. correctness is crucial." In that situation, one could look to blend correctness-prioritized development with agile-prioritized development to get an appropriate mixture of the two.
Cockburn also wrote:
Consider "agile" development as a particular specie of development, e.g., a Bengal tiger. Then asking "what is the opposite of agile development?" is like asking "What is the opposite of a Bengal tiger?" Is it an elephant, a mosquito, or perhaps a non-Bengal tiger, or a Bengalese non-tiger?Brad Appleton came up with a nice definition of agile (notice the first letter of each point):
2003.Jul.04 Fri
- Adaptive - plans, designs, and processes are regularly tuned and adjusted to adapt to changing needs and requirements (as opposed to predictive methods that attempt to develop comprehensive and detailed plans/designs/requirements "up front").
- Goal-driven - focus on producing end-results (working functionality) in order of highest business value/priority (as opposed to being task-driven, document-driven, process-driven, or risk-driven).
- Iterative - not just iterative, but *highly* iterative, with very short development cycles, very frequent releases, and exremely frequent+regular feedback.
- Lean - simple design, streamlined processes, elimination of redundant information, and "barely sufficient" documentation and methodology.
- Emergent behavior - quality systems (requirements, architecture, and design) all "emerge" from highly collaborative, self-organizing teams following an initially minimal set of simple, generative rules, and working in very close interaction with stakeholders.
Peter Lindberg wrote
The assumption that lies behind the question You have documented this, haven't you? is that code is inherently unreadable and that there must be a manual to explain it. Many projects indeed are like that. Many projects have grand vapor architectures that exists only in the minds of those who have devised them, and can't be found anywhere in the code. Any document in such a case only reflects the vapor architecture, and is of little help to understanding the code. Let's hope there are still survivors to answer your questions.and
My goal donor asked me, You have documented this, haven't you? to which I replied, No, except for the 570 or so automated test cases Malte and I have written during the course of the project. Each test case captures a micro-scenario in the life of the system, by setting up the objects involved, telling them to do something together, and verifying the outcome. And the fact that they are executable means that a programmer new to a team, besides having an explanation of the code to read, written in code itself, can execute that codee to see if the system still works. So it's better than documentation.2003.Jul.03 ThuA quote from Seth Godin's Purple Cow:
If you're remarkable, it's likely that some people won't like you. [...] Criticism comes to those who stand out.
Where did you learn to fail? If you're like most Americans, you learned in first grade. That's when you started figuring out that the safe thing to do was to fit in. The safe thing to do was to color inside the lines, don't ask too many questions in class [....]
A quote from Eliyahu M. Goldratt's Theory of Constraints:
We assume that the openness of these functions [marketing, distribution, etc.] will be enhanced by the positive results achieved in the other functions [production and material...]
What we found out was this is certainly not the case. [...] The emotional resistance, which already exists, prevents almost any meaningful dialog. The only way to open the Throughput channel in these case was through personnel changes.
The lesson today is loud and clear. Before any function can go on an ego trip, demonstrating and waving results (and by that digging its own grave) -- before any function can start individual improvements, all functions should decide together on a common way.
Extreme Programming is one of those "Purple Cows" that Seth Godin talks about -- it is "remarkable". Its value comes from not fitting in with what the majority of programmers do. It does more testing and more design, and requires more discipline. The documented bug-rate reductions of projects that converted to XP and their improved time-to-market are actually not believed by many, if not most managers, even when the project is within their own company.
"Selling" XP also has to recognize the manager's fears of not fitting in. Heed Weinberg's Laws of Consulting: "In spite of what your client may tell you, there is always a problem. No matter how it looks at first, it is always a people problem.... Never promise more than a ten percent improvement. (If it were possible to achieve more than a ten percent improvement, there must have been a problem, but there isn't a problem, so...)"
I'm looking to the "project community" practices of IXP to help keep an organization from rejecting XP. Retrospectives and Readiness Assessments that include more than just the programmers. Management Tests. The value of Learning.
2003.Jul.02 WedJoshua is adding content on Industrial XP here: http://industrialxp.org/description.html. On the topic of Continuous Risk Management he recommends Waltzing With Bears and Radical Project Management.
2003.Jun.30 MonSocratic Questions and Experiential Learning
One way of teaching, developed 2500 years ago by Socrates, is to ask questions and let the student work out the answers. In Theory Of Constraints by Eliyahu M. Goldratt, the author credits Socratic questions for the success of his previous book The Goal. In The Goal, a consultant helps the main character with advice in the form of questions. The reader follows the main character as he tries to come up answers to the questions; and, Goldratt says, the readers come up with the answers themselves a page or two before the main character does. (I know I did.)
Inventing their own answers motivated the readers to see if the main character came up with the same answers, and in many cases this excitement further motivated readers to apply the lesson of The Goal to their own factories.
Goldratt also talks about the stages of "science" - classification, correlation, and effect-cause-effect. In classification, we could observe that students learn things in various ways. In correlation, we could observe that some students learning via Socratic questions were more excited about their learning than others. In effect-cause-effect, we could theorize why this happens, and use the theory to make predictions which could be invalidated (or validated) by experiment.
Goldratt goes on to say that an experiment may not completely invalidate a theory, but may constrain it. For example, Newton's laws of motion are valid for speeds we see on Earth, but are constrained by other effects when the speeds are near the speed of light.
We can have a theory that Socratic questions can be effective, because someone inventing their own answers feels ownership and the excitement of invention. In some cases they've done a lot of work, and finding the answer is a big reward. However, we can also observe that Socratic questions often fail in practice... the students get irritated by that smug teacher who obviously knows the answers, but isn't giving them out. The resentment can be very strong. The questions can have responses like "Why are you asking us?" followed by arguments about motivations and agendas.
Another style of teaching that also lets the students find their own answers is called "experiential learning." In a typical case, the teacher (or, as some preferred to be called, the facilitator), sets up a "simulation" where the students can try something out in a safe environment, followed by discussion. There's no initial question, just "try this out". In the discussion afterwards, the questions could be "What do you think about X after doing this exercise" or "What were you feeling?" These are non-threatening questions, and students don't feel that the teacher already knows the answer.
In the Agile Development Conference I just attended, some of the workshops used experiential learning. In XP Coaching workshop, facilitated by Ron Jeffries and William Wake, the exercise involved some people coaching other people doing origami (from an origami instruction manual). Some other workshops also involved simulation/practice and discussion.
Ron tells us that XP Immersions that he participates in are doing more experiential learning and less lecture. Instead of lecturing about pair programming, they have the students pair program, probably followed by discussion and reflection on what happened when they pair programmed. The whole point of many training courses is to just get the students to do reflection and discussion - and this is the point of Retrospectives... enabling learning in the organization and team.
Think about learning some skill or getting answers to some of your harder problems, not necessarily a problem with technology; in your own experience, when someone told you the answer to a problem or question, did you value it as much as when you came up with the answer yourself, or practiced the skill by yourself?
2003.Jun.23 MonCem Kaner's Blog has as its first entry an call to arms on the SWEBOK. Here are few snippets:
In retrospect, I think that keeping away from SWEBOK was a mistake. I think it has the potential to do substantial harm. I urge you to get involved in the SWEBOK review, make your criticisms clear and explicit, and urge them in writing to abandon this project. Even though this will have little influence with the SWEBOK promoters, it will create a public record of controversy and protest. Because SWEBOK is being effectively pushed as a basis for licensing software engineers and evaluating / accrediting software engineering degree programs, a public record of controversy may play an important role.
I am most familiar with SWEBOK's treatments of software testing, software quality and metrics. It endorses practices that I consider wastefully bureaucratic, document-intensive, tedious, and in commercial software development, not likely to succeed. These are the practices that some software process enthusiasts have tried and tried and tried and tried and tried to ram down the throats of the software development community, with less success than they would like.
Only 500 people participated in the development of SWEBOK and many of them voiced deep criticisms of it. The balloted draft was supported by just over 300 people (of a mere 340 voting). Within this group were professional trainers who stand to make substantial income from pre-licensing-exam and pre-certification-exam review courses, consulting/contracting firms who make big profits from contracts that (such as government contracts) that specify gold-plated software development processes (of course you need all this process documentation - the IEEE standards say you need it!), and academics who have never worked on a serious development project. There were also experienced, honest people with no conflicts of interest, but when there are only a few hundred voices, the voices of vested interests can exert a substantial influence on the result.See also my entry on this subject.
2003.Jun.22 SunFirst there was the Gettysburg Address. Then Edward R. Tufte wrote The Cognitive Style of PowerPoint. For those of us who haven't read Tufte's book, here's a summary of it in PowerPoint form: http://www.aaronsw.com/weblog/000931. (Thanks to Jim Bullock for mentioning this on the AYE wiki.)
2003.Jun.21 SatConsulting Essay and Testing FAQ
Danny R. Faught has a short paper on being a consultant. He's also maintaining Software Testing FAQs.
2003.Jun.19 ThuJames Robertson's notes on talk about CMMI and Agile:
[...] Not safety critical, but unpredictable change - appropriate for Agile. Again, IMHO, this sets up a straw man that Agile is unsafe. I don't think that's fair, and I think that the overall failure rate in the software business - including in the safety critical side - is not encouraging. [...]2003.Jun.18 Wed
A Level 5 Shop
Improvements are selected based on quantitative measurements. How do you get that? It could easily be via Agile methods. [...]Conflict Haiku and other Writings
Some good papers by Rick Brenner of Chaco Canyon consulting... the Point Lookout email newsletter.
Conflict Haiku: "When tempers flare, or tension fills the air, many of us contribute to the stew, often without realizing that we do. Here are some haiku that describe some of the many stances we choose that can lead groups into tangles, or let those tangles persist once they form."
When Naming Hurts: "One of our great strengths as Humans is our ability to name things. Naming empowers us by helping us think about and communicate complex ideas. But naming has a dark side, too. We use naming to oversimplify, to denigrate, to disempower, and even to dehumanize. When we abuse this tool, we hurt our companies, our colleagues and ourselves."
Let Me Finish, Please "We use meetings to exchange information and to explore complex issues. In open discussion, we tend to interrupt each other. Interruptions can be disruptive, distracting, funny, essential, and frustratingly common. What can we do to limit interruptions without depriving ourselves of their benefits?"
Full list of past issues: http://www.ChacoCanyon.com/pointlookout/topicalarchive.shtml
2003.Jun.15 SunI just read Now Discover Your Strengths by Buckingham and Clifton, and I'm currently reading First, Break All the Rules by Buckingham and Coffman. The books are based on extensive data collection and analysis by Gallup.
The point of First, Break All the Rules is that great managers treat each of their employees as unique individuals, encouraging their strengths, and working-around (rather than fixing) their weaknesses. The "rules" to be broken are "treat everyone the same" and "concentrate on fixing flaws".
Gallup has evidence that good managers are the primary reason for employee job-satisfaction, and poor managers are a primary reason for employees leaving a company. Managers are also a primary reason for productivity and profits (or losses) at the worker level. Of the Twelve Questions that measure employee satisfaction, the top five are directly related to how managers work with their employees.
- Do I know what is expected of me at work?
- Do I have the materials and equipment I need to do my work right?
- At work, do I have the opportunity to do what I do best every day?
- In the last seven days, have I received recognition or praise for doing good work?
- Does my supervisor, or someone at work, seem to care about me as a person?
Now Discover Your Strengths identifies thirty-four "themes" or basic strengths (which I'll call "StrengthFinder types") that people have in varying degrees, and provides an on-line test to find your top five strengths. It says that the strength that managers must have to be good managers is "Individualization," which is the natural ability to observe and identify people's strengths, style, motivation, and other unique qualities. Ironically, in a book about 34 generalized "types", it says that people with the talent for "Individualization" are impatient with generalizations and "types."
Unlike MBTI, the StrengthsFinder types are not pairs of opposites. Both type systems can help people who are not strong in Individualization to learn to see people as different from each other. There's probably some statistically-relevant relationships between MBTI types and StrengthsFinder types, but I'm not aware of any documentation on that yet.
This link explains the 34 types and give advice to students with those strengths http://student.gallup.com/strategies.html.
2003.Jun.14 SatDon't Impose Improved Practices; The Team Must Improve Their Own Practices
Reading the book Lean Development by Mary and Tom Poppendieck has reminded me that successful group-improvement comes from self-directed (self-improving) groups -- and that the principle of self-improvement is more important than the specific practices.
They have an example of a software team within a large company that decided to improve their own processes. That improvement was noticed, and a committee formed to copy their improved practices to other teams in the company -- which never worked as well for those other teams as it did for the first team.
They also say the same thing happened (is happening) as NUMMI - the auto plant owned by GM and run by Toyota. Originally it was GM's worst plant, but within a year of Toyota taking over its management, it became one of GM's most productive plants - low absenteeism, high quality (almost as high as Toyota's plants) - with substantially the same people. GM tried to copy the practices to other plants, never with much success. The principle of continuous, self-directed-improvement is what makes NUMMI different, and which GM doesn't seem able to copy.
2003.Jun.12 ThuIn Fear Of Refactoring, I wrote:
Micheal Features is working on a book about making legacy testable here: http://groups.yahoo.com/group/welc/.That should be "Michael Feathers". Thanks to Brad Appleton for the feedback.
Ron Jeffries has written on his discomfort with the Prime Directive of Project Retrosepectives here: http://www.xprogramming.com/xpmag/jatPrimeThis.htm.
The Prime Directive of Project Retrospectives is "Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand."
Jeffries writes:
We wound up, not in a "wonderful place", but in an OK place.... I don't get it. I don't get how accepting badness as "we did our best" leads to learning. There must be something after the Prime Directive. What's the Second Directive?The prime directive isn't setting the stage to accept that nothing could be improved, it is to set the stage to look for improvements without getting stuck blaming people for what they or others think they did wrong. Once you're in fault-finding mode, everyone will try to defend themselves, trying to re-direct fault-finding onto others, and will stop sharing experiences that the group could be learning from.
We can apply Dale Emery's excellent essay "Creating Empathy" to the example posed in Ron's essay: Ron wrote some code non-test-first. It took longer than expected to debug, so that it could have been done test-first in the same amount of time. Why would he do this? He says that he had a deadline; people were asking "where the heck" is this code? He didn't know how to throw characters at the GUI code in his tests... and so on. His deeper reasons were to not let down those people; to deliver as soon as possible. The prime directive helps us assume the intentions were good, even though the results were not.
Perhaps a facilitator in a retrospective would remind the team to not worry too much why Ron didn't write that code test-first, and help them realize what the consequences were, and let the team identify things to do differently in the future.
I know of a "kick-butt" manager whose every question seems like an attack. When he says "Why did you do this?" It doesn't come out as "I'd sincerely like to understand reasons why someone would do this," it comes out as "Why would anyone be so stupid to do this?" This attitude is very demoralizing to his employees. He may not realize that when he focuses a barrage of these kinds of questions onto his employees, they become demoralized and ineffective for hours or days afterwards. This is not a way to keep the project work going smoothly. If the job market was better, many of those employees would quit.
If a retrospective was attempted with that "kick-butt" manager involved, I hope the facilitator would be strong and experienced enough to kick that manager out of the retrospective - no learning is going to happen otherwise.
The XP mailing list had many good responses to Ron's essay. To quote just one, George Dinwiddie wrote
The reason they didn't do a better job is perhaps not in the list of items you're considering (knowledge, skills and abilities, resources and situation) but certainly there was something that led them to do what they did. Of course, you can attribute it to their laziness or orneriness, but in my experience this tends to lead to non-productive confrontations.
It can be much more productive to search for what can be done better in the future rather than searching for the reason(s) that one couldn't do better in the past. Just because, for reasons that are not clear, this was the best I could do at the time, is no excuse to quit striving to do better in the future. And that striving can be done without identifying character flaws or laying blame.
Ron asked what is the "second directive?" Several people identified that as talking about the future, identifying what things you would do in the future that would be different from what you did in the past.
2003.Jun.10 TueDave Hoover is writing his experiences in an e-commerce project adopting Extreme Programming. http://redsquirrel.com/dave/work/babySteps/.
2003.Jun.09 MonCode Reviews versus Pair Programming
On the XP mailing list today, Ron Jeffries responds to a question about code reviews:
Frankly, code reviews are so much worse than pair programming that a dose of them would make me fly to pair. Let's see if we can replicate my experience.
Here's one path through a network of a million decision points:
To do code reviews, everyone has to read the code beforehand, unless you're doing a walkthrough, see below. I'd ask everyone to come together physically to the review. Then I'd ask them to report truthfully how much time they spent reviewing the code. Early on, I would report truthfully that I had spent zero or very little time, in hopes of getting others to admit the truth. When they admit the truth, I'd dismiss the meeting and reschedule it.
Dismiss and reschedule is the appropriate response to lack of preparation, according to Freedman and Wienberg.
Then, after a while, the only alternative is a walkthrough, since no one is preparing effectively for the review. So we do walkthroughs for a while. They are intensely boring, and few people stay involved. Note in your mind the people who are present but not involved. At the end of the session, say, holding your hand up, "Who else had a real problem staying engaged with this walkthrough?" If there's honesty in the room, hands will go up. Prompting may be necessary. Then: "Any ideas?"
Surely someone will think of "doing this in smaller groups or one on one". Try it. Ask the team whether "we should empower the one-on-one folks to change the code, and under what circumstances." Don't mention that this is pair programming.
Try an experiment. You're "interested in collaborative programming". Interested parties should come to the room to help. On the screen, start writing a program. Ask for help with it, get the room to pair with you. Get stuck (no need to fake this if you are me). Someone will start telling you what to do. Don't get it (no need to fake this either). Get them to come up and do it ... grabbing the chair that is accidentally beside you, while you move over.
Note that reviews often find things. Observe how many of them are resisted by the original programmer, or are "too much trouble to fix now".
Build a few BVCs [Big Visible Charts] relating to time spent prepping, in the meetings, number of useful suggestions (by person if you can do it without problems), number of changes made in response to suggestions, ...
Code reviews are intensely painful, in my experience, and we were trained by Freedman himself. There will be no need to set them up to be perceived that way, though it will take honesty among the group to express it. After doing enough code reviews, which take way more than half the groups' time by the way, a team who has heard of pair programming should be begging to pair. About all you have to do is make sure that no one treats the review session as nap time, and that you are early in recognizing the people who think it's a waste of time. Because they're right.
I have done reviews on a project that was building software for a medical device. We prepared for the code review (and also reviewed unit tests). We found issues (like not enough unit tests for specific parts of the code). The programmer corrected issues after the review, and had a second review (a walkthrough with the coder and one reviewer) to confirm that the issues were corrected. We signed off pieces of paper to have the paper trail that the FDA would expect. It wasn't the most exciting work, but we were one of the best teams, people-wise, that I've ever been privileged to work with. Truly respecting each other and working together very well.
Coding and reviewing this way was very slow - for eight person-hours of programming, expect at least a half an hour of preparation (coder and three reviewers - so two person-hours of preparation), one hour of review (times four people - so four person-hours of review), one person-hour correcting issues, and another half-hour of walkthrough (times two people - one person-hour). So for 8 person-hours of programming, there was another 8 person-hours of overhead: reviews, correcting, and walkthrough.
As is often said, "basic XP" isn't intended for safety-critical applications. But I could see the following working in that environment: pair-programming and test-driven development for four hours (eight person-hours), and a one-hour walk-through by two people who didn't do the pair-programming (two person-hours). I would expect very few issues to be found in the walkthrough that hadn't already been found (or avoided) and corrected in the pair programming, and therefore there would be much less need for correcting issues. So 8 person-hours of (pair) programming would have an overhead of 2 person-hours of walkthroughs.
While I was working on the medical device project, we didn't have acceptance testers nor automated acceptance tests. If we had been doing XP, we would have had better testing -- better unit tests, and real acceptance tests. I didn't stay on that project through completion - when our (well-liked but not very political) manager was removed from the project, and our team leader quit, almost everyone else on the team quit too. I was one of the last ones to leave, accepting an offer from Apple Computer, Inc. and moving to California. I heard the the project eventually finished, with half the man-power and twice the planned schedule. Very likely the acceptance testing was done at the end of that project.
2003.Jun.08 SunWhen I was in high school, in the 1980's, I discovered a book on Esperanto at a library and studied it. I bought a few books, and even acquired (I don't recall how) a record album of folk songs recorded in Esperanto. It was interesting, but I never had anyone to speak it with. For academic credits in high school and college, I took Latin, another language I didn't use for speaking purposes. In case you haven't heard of it, Esperanto was one of the first "artificial languages," a language created to be culture-neutral, to enable international communication. It has very simple rules of grammar, and a vocabulary adopted from root words of many European languages. Esperanto is reported to be ten times easier to learn than a typical European language.
Created over a hundred years ago, Esperanto was not just a language, but a movement for international understanding and peace. It was denounced by Hitler and banned in Nazi Germany. The USSR in 1938 shot or deported Esperanto speakers. Pre-WWII Japan mistreated and sometimes executed Esperantists. In the USA during the McCarthy era, Esperanto was associated with Communism (and perhaps some American Communists were using it as a 'secret language'.)
Perhaps at its height of popularity, between one and ten million people spoke Esperanto world-wide. In recent years, it has been reported that Klingon and Elvish are two artificial languages with more speakers than Esperanto, though I think more people are actually fluent in Esperanto than Klingon.
Why am I talking about Esperanto? Partly because I just read an article about it in the San Francisco Chronicle, which reports that about 25 people in the Bay Area get together to speak it monthly. It quotes ELNA vice president of public relations Charlie Galvin as saying "For me, I like Esperanto because it's like a secret club. I like the fact that few people know it. It makes it special." That is so depressing, in light of the intent of Esperanto's creator.
Esperanto will always be a little-known language because there is no economic reason to learn it - and that's just fine with the Esperantists. The Esperanto-enabled travel service is free and non-commercial. In this day of international organizations (both commercial and non-commercial), Esperantists are no longer making the case that that a simple-syntax, easy-to-learn language could help people communicate with each other on a level playing field, with value gained in commercial as well as non-commercial arenas.
Another reason I'm talking about Esperanto is that it reminds me of Smalltalk. Like Esperanto, Smalltalk has a simple syntax that is pretty easy to learn. Esperanto was invented by an idealist wanting international communication. Smalltalk was invented by idealists who thought it could be a computer language easy enough for children. Like Esperanto, few people today have an economic reason to use Smalltalk. No big companies are hyping Smalltalk the way Sun hypes Java, or Microsoft hypes C#. Even IBM, which sells a Smalltalk environment, is putting more energy into Java than Smalltalk.
The rules of grammar for Esperanto can be written down in few pages, including all the exceptions. The rules of Smalltalk's grammar are also simple. This PDF explains Smalltalk syntax in relatively few pages: http://www.speakeasy.org/~podenski/stug/reading-smalltalk.pdf.
Smalltalk syntax is unusual today, though I find it easier to read than Perl or Ruby. Objective-C is a language that adds object-oriented constructs to the C language using a syntax similar to Smalltalk. One of my coworkers complained "why isn't Objective C using Java or C++ syntax?" Because it predates C++. When Objective C was invented, Smalltalk was the leading OO language. Now Smalltalk is one of the language reported as dead or near-dead by Wired magazine.
Objective-C is regaining mind-share these days because it is the recommended language for implementing GUI applications on MacOS X using the Cocoa class library. There is an economic reason for people to adopt that language -- it is the easiest way to put a really nice GUI on non-gui unix-compatible code on MacOS X. At least five books on Cocoa programming have been published in the last year on Cocoa, and every one teaches the OO parts of Objective C in the opening chapters.
Unlike some Smalltalk implementations, Objective C on MacOS X plays well with others. Linking C libraries and Objective code together is trivial. Even though the Cocoa class library is implemented in Objective C, you can write Cocoa-using applications in Java, Python, AppleScript, Ruby, and a Smalltalk-like scripting language called FScript. (C++ is too "static" to easily interoperate with classes written in dynamic languages, but Apple's version of 'gcc' allows mixing C++ and Objective C code if one is careful to cope with the differences in object allocation and deallocation.)
I would use Smalltalk it if had native GUIs on MacOS X and Windows. If it interoperated with Cocoa on MacOS X. If it easily interoperated with C libraries. If it could be used to implement shared libraries (TWAIN plugins) and other non-application code on the two platforms I need to support, and not have to be written in "dialect dependent" forms on each platform. If it didn't freak out my coworkers. If it wasn't a "secret language."
2003.Jun.07 SatFrom Rob Thomsett's page "Project Pathology: Causes, patterns and symptoms of project failure"... "We observed this pattern of failure in every one of the 20 major projects we reviewed. Fifteen of the 20 projects we reviewed had degraded to Step 4 and the remaining 5 were at Step 3."
The four stages of failure:
- Development team unilaterally de-scopes the project.
- Project manager requests additional people.
- Unpaid overtime work becomes the norm.
- Team has lost control of the project, but the team is in denial.
For an example of stage 4: "This denial was well evidenced in a project where all team members and the project manager felt that the project could meet a deadline 3 months out and that our review was a waste of time. In one day, we identified that the project was over 12 months behind schedule and would deliver [at best] in 15 months not 3! Our findings were wrong - the project finally delivered 18 months later."
2003.Jun.06 FriSome people very experienced in software development looked at my Extract Method refactoring example and wondered what was so unusual about doing that. One asked "Isn't this just obvious, and what any sane person would do?" Apparently not.
There are tons of "insane", unrefactored code in companies everywhere. Even in my own current XP-ish project, there are members who sometimes write (or more likely copy-paste-modify) long methods with duplicate code, and they have more than five years of experience developing software.
For many people, it seems like their tolerance level for hard-to-read, hard-to-maintain code is high, possibly because their high intelligence lets them sort things out in their minds - at least when they're familiar with the code. If only they would sort it out in the code itself.
I imagine those "tolerant" people get irritated by us "intolerant" Extreme Programmers, who want the code to be so easy to read and enhance that it looks like it was easy to write. There's an old joke in programmers circles - "If it was hard to write, it should be hard to read" - that is unfortunately too true.
If you really want to get insane code - long methods, lots of duplication - measure your programmer's productivity by counting lines of code. You'll get what you measure.
2003.Jun.05 ThuAndy Hunt blogs about shopping, not being able to find a shampoo that doesn't have fruit juice or herbal essences in it. He calls this "herd marketing" and writes: "It should be [...] the things I want. Isn't that supposed to be the essence of marketing? Give the people what they want -- not what everyone else wants to give them, or even what you want to give them."
Well, my understanding of typical marketing is that it's all about getting people to want what you have to offer, not offering what people want. Harry Beckwith's book Selling The Invisible was refreshing in that he goes against the grain, and says that selling the steak is more important than selling the sizzle.
And by the way, I don't mind herbal essences in my shampoo. My wife says she likes the way it smells.
Laurent Bossavit quoted Beck in his blog, referring to http://c2.com/cgi/wiki?GettingBetter. Thanks, Laurent.
2003.Jun.04 WedHow good the design is doesn't matter near as much as whether the design is getting better or worse. If it is getting better, day by day, I can live with it forever. If it is getting worse, I will die.
Ken Arnold points out Martin Fowler commenting on Cringley's post on Refactoring.
Somebody, possibly Kent Beck, said something very much like "If the code quality is going down, you're doomed. If the quality, however bad to start with, is always going up, you have a chance at success." (I can't find the quote. I need google to Find What I mean, not what I ask it to find.)
Every company that has invested in software has invested in an asset that they can choose to improve or let decline in value. If those software assets never need to change - no bug-fixes, no enhancements, no adapting to changes in the software ecology (new versions of libraries, new OSes) - then don't refactor.
Fear of changing legacy code is healthy, if the legacy code is not fully covered by automated tests. You need something to rapidly tell you if you've broken something when you're doing a code improvement. Micheal Features is working on a book about making legacy testable here: http://groups.yahoo.com/group/welc/.
Probably a good motivator for improving code is measuring the cost of changing code for bug-fixes and feature additions (and measuring the Fault Feedback Ratio). The older and cruftier the code is, the higher the cost of changing it. The cost of refactoring (which will reduce maintenance costs in the long run) has to be weighed against the increasing costs of maintenance when no refactorings are being done.
Perhaps the best way of addresssing fear of refactoring is by practicing it on toy projects. William C. Wake has a book called The Refactoring Workbook due in July. Preview chapters are on-line: http://www.xp123.com/rwb/.
The other key thing about refactoring to improve the design by taking small steps. For example, most code has functions and methods that are too long, containing multiple 'bundles' of lines, sometimes with comments describing the intent of each bundle. Use the extract method refactoring to move each little bundle of lines into a new function, named after that comment. (See my example below.) When big methods have been broken down into small methods, duplicate code becomes more obvious, and can be removed, reducing the amount of code to be maintained and raising the level of abstraction in the code.
Big refactorings are much easier if lots of small refactorings have already been done. It is important that the team members agree on the value of the small refactorings - you don't want inviduals to 'undo' each other's code improvements.
Example of "extract method" refactorings - pseudocode
"before"
procedure testVerylongmethod var result = Verylongmethod var expectedResult = 2398 assertEqual( expectedResult, result ) end procedure function Verylongmethod // set up foos var foos = array [1..3] of foobar foos[1] = new foobar(2,2,3,4) foos[2] = new foobar(3,4,6,3) foos[3] = new foobar(4,67,83,3) // calculate muckiness var muckiness = 0 for index = 1 to foos.length muckiness = foos[ index ] . muckinessQuotient( index ) + muckiness end for return muckiness end function"after"
(the test doesn't change, so I don't reproduce it here.)
function setupfoos var foos = array [1..3] of foobar foos[1] = new foobar(2,2,3,4) foos[2] = new foobar(3,4,6,3) foos[3] = new foobar(4,67,83,3) return foos end function function calculatemuckiness( foos ) var muckiness = 0 for index = 1 to foos.length muckiness = foos[ index ] . muckinessQuotient( index ) + muckiness end for return muckiness end function function Verylongmethod var foos = setupfoos() var muckiness = calculatemuckiness( foos ) return muckiness end functionNow imagine that there is Anotherlongmethod similar to the original function Verylongmethod, where the only difference in how it sets up the foos array. It can be refactored to call the same calculatemuckiness method that had been extracted from Verylongmethod.
"before"procedure testAnotherlongmethod var result = Anotherlongmethod var expectedResult = 9487 assertEqual( expectedResult, result ) end procedure function Anotherlongmethod // set up foos var foos = array [1..4] of foobar foos[1] = new foobar(2,9,3,4) foos[2] = new foobar(3,9,6,3) foos[3] = new foobar(4,9,83,3) foos[4] = new foobar(9,9,9,9) // calculate muckiness var muckiness = 0 for index = 1 to foos.length muckiness = foos[ index ] . muckinessQuotient( index ) + muckiness end for return muckiness end function"after"
(the test doesn't change, so I don't reproduce it here.)
function setupfourfoos var foos = array [1..4] of foobar foos[1] = new foobar(2,9,3,4) foos[2] = new foobar(3,9,6,3) foos[3] = new foobar(4,9,83,3) foos[4] = new foobar(9,9,9,9) return foos end function function Anotherlongmethod var foos = setupfourfoos() var muckiness = calculatemuckiness( foos ) return muckiness end functionPlease forgive all the magic numbers; I wanted to keep the example self-contained. While the safety of these refactorings can often be determined by code inspection, they are best determined by tests - the test don't change during (most) refactorings, and they should be passing before and after the refactoring.
2003.Jun.03 TueData-Driven GUI Code and YAGNI
Ron Jeffries has written an essay refuting the myth that YAGNI ("You Ain't Gonna Need It) prohibits data/table-driven code. He's coding in C# and his examples are for menu items.
2003.Jun.02 MonResponse to Unit Testing In Java
Previously I used my review of the sample chapter of the book Unit Testing In Java by Link and Frohlich to launch my diatribe against the typical Java-book method of creating widgets in windows/dialogs. Johannes Link send me this comment:
Subject: Unit Testing in Java
Hi Keith,
Compliments on being so quick in reviewing the sample chapter of "Unit
Testing in Java". That's really impressing.
One thing I'd like to comment on: You're perfectly right when saying that
implementing ActionListener in the window class itself usually leads to loads
of duplicated code. The code in the book, however, is in a temporary state. As
long as there are only two buttons the code is easier to read (IMO) than
when using anonymous inner classes. Refactoring to a different solution would
certainly take place during one of the next enhancements. The point is: Having
the tests in place allows you to perform this refactoring on stable
ground...
BTW, your blog really is worth reading.
best wishes,
Johannes Link
When to do refactoring is certainly worth further exploration. Martin Fowler has the "three strikes" rule to remind him to refactor away duplication when a third instance is seen. Extreme Programming recommends "once and only once" to refactor after the second instance is seen. But when your current task is test-driving a dialog or widget-filled window, at what point do you refactor to a dialog-creator class - after the second button, or after the second dialog, or never?
2003.Jun.01 SunSo-Called Software Engineering Body of Knowledge (SWEBOK)
Grady Booch has this to say about the SWEBOK: "The SWEBOK I reviewed was well-intentioned but misguided, naive, incoherent, and just flat wrong in so many dimensions."
Cem Kaner writes "I think SWEBOK promotes a set of practices that are sometimes appropriate, but in many contexts thay are as outrageously expensive as they are remarkably ineffective. The idea of adopting these as a standard of care for our profession, is, at least to me, abhorrent."
It seems like the SWEBOK group ignored Grady Booch and many other reviewers three years ago. It attempts to be an authoritative reference of "generally accepted" practices, but the SWEBOK does not reflect principles and practices of agile software development, and otherwise misrepresents itself when claiming to be "generally accepted" by our diverse "industry". Do game developers, web-app developers, AI researchers, and shrink-wrap software developers need to use the same practices? SWEBOK may be representative of Department of Defense contracting practices (which is far from "general"), but even the DoD is trying to develop software systems in more agile ways these days.
The SWEBOK is again open to review. Cem Kaner recommends that we become reviewers to highlight that the practices the SWEBOK preaches are not accepted by all. I recommend that you not only review the SWEBOK and provide your comments to that group, but you also post your comments on the web, so that everyone using a search engine can see the controversy around the SWEBOK.
One danger of the SWEBOK is that it would be used to license software developers. Software development is too broad a field, too immature, and too rapidly changing a profession to have meaningful tests for licensing developers. Cem Kaner was part of the ACM task force that considered and rejected the SWEBOK and licensing based on it. Check out what the ACM task force had to say about it at http://www.acm.org/serving/se_policy/. The ACM task force writes:
- Licensing as Professional Engineers would be impractical for software engineers, because it would require examinations over subjects most software engineers neither study in their formal education nor need in order to practice competent software engineering.
- Licensing software engineers as Professional Engineers would have no or little effect on the safety of the software produced.
- The SWEBOK effort, which specifically excludes from the body of knowledge the special knowledge required for most safety-critical systems (such as real-time software engineering techniques), will have little relevance for safety-critical systems, and it dangerously excludes the most important knowledge required to build these systems.
- Each industry and software engineering domain will need to determine an appropriate mix of approaches that work together to solve their particular problems and fit within the cultural context of the particular industry. There are no simple and universal fixes to solve the problem of ensuring public safety. Effective approaches will involve establishing accountability, competency within specific application domains and job responsibilities, liability, regulation where appropriate, standards, voluntary product certification and warranties, and industry-specific requirements. Licensing as Professional Engineers would not be an effective way to accomplish any of these goals.
The task force recommended that:
When reviewing, please include positive comments as well as negative comments. If it's all negative, they will feel tempted to dismiss all that you write. George Dinwiddie had a good example of both positive and negative in the xp mailing list:
- ACM withdraw from efforts to license software engineers as Professional Engineers.
- ACM take a stand against government efforts to require the licensing of software engineers as impractical, ineffective with respect to protecting public safety, and potentially detrimental with respect to economic and other societal and technological factors.
- ACM not support the SWEBOK activities, but consider supporting other efforts to validate and codify basic knowledge in various aspects of software engineering. [...]
- The professional societies, including ACM, must pursue every possible means towards improving the current state of affairs. At the same time, they must refrain from pursuing activities like SWEBOK that have a significant chance of reducing the public's understanding of, confidence in, and assurances about key properties of software.
They say, "One of the fundamental tenets of good software engineering is that there is good communication between system users and system developers." That's very good, but the next sentence says, "It is the requirements engineer who is the conduit for this communication." I don't see any justification for the assumption that there must be an intermediary.
Not only could a reviewer say that good communication is a good thing to emphasize, but the reviewer could then follow that by commenting that the assumption of a "conduit" is not generally accepted, perhaps referencing materials that document the "telephone game" effect of having someone between the system developers and the system users. Sure, a facilitator of some kind ("business analyst" or "human interface designer") is very helpful, but as an aid to the communication, not as a substitute for it. Extreme Programming, for example, says the "customer" should "speak with one voice", but not that the "customer" be a single person -- and if you call him a "requirements engineer", does that mean that person will have to a licensed engineer as well? How much of a priesthood do we need?
2003.May.31 SatI found the book Selling the Invisible by Harry Beckwith to be a very interesting and inspiring read. Those of us working in software development deal with the invisible all the time. While some of us may write shrink-wrap software, and can point to the box as proof that our product exists, even that product can be mostly invisible: anti-virus software, for example, should be invisible until it tells you it found and eliminated a virus (and maybe even then).
Selling the Invisible is about marketing and a lot more: some of the best marketing comes from having a good product or service, and happy clients or buyers. It comes from maintaining good relationships. I recommend this book. It gave me an idea for my company that could be profitable, and I've only read the book once through so far.
"Management" is mostly invisible - the worse management is, the more visible it is. The best managers (I'm told) often seem like they are neither busy nor doing "any work," because they've done a lot of work to keep potential problems from turning into actual problems. Managers could do with reading this book, considering their bosses, peers, customers, suppliers, and direct reports as "customers", all of whom to maintain good relationships with.
Software development methods can be invisible too. Particularly agile methods that don't produce many artifacts other than the software itself. The recommended "big visible charts" (in particular the scrum burn-down chart adapted for XP) help make the invisible stuff visible, and help keep a good relationship with the customer.
In fact, I thought Kent Beck wrote a message in the IXP mailing list that sounded like the "positioning statement" exercise of Selling the Invisible, but now I can't find it. I'll attempt to do the exercise here, using me as an XP coach/project leader. The positioning statement answers the questions of Who, what, for whom, what need, against whom, what's different, and what benefit.
Keith Ray is a coach and project leader for small to medium-sized projects (up to 10 programmers) which need to ship good quality software, quickly and/or frequently, whether the desired features are vaguely defined, changing, or very specific. Unlike many in this industry, Keith Ray has been developing software for over 15 years, and has participated in many successful projects in several product areas, creating and shipping good software at a predictable pace. Keith Ray uses good, context-specific, practices to enable you to get the important features you want as soon as possible.
You notice I don't say "XP" or "agile" in this statement - people are looking for solutions to their problems, and don't care about the techniques I use (I hope). What's your positioning statement?
2003.May.28 WedUnit Testing In Java, Duplicate Code in Java GUIs
Johannes Link announces that the book Unit Testing In Java by Link and Frohlich, is now available for purchase, even though Amazon says it isn't yet. Link says the book teaches Test Driven Development in Java, covering GUIs, web-applications, databases, multi-threaded applications and other advanced topics. A sample chapter is available here: [PDF, 28 pages] http://www.bhusa.com/bookscat/samples/1558608680/1558608680.pdf?mscssid=W2B78S2RV83D8K9W260JPX1W0DUJ9DC7.
The sample chapter on test-first Swing GUI programming isn't that different than other writings on Java GUI test-first. Near the end of the chapter they discuss JFCUnit, and briefly mention AWT Robot, neither of which they recommend for typical test-first unit testing. I will order the book and read it.
One nit I'd like to pick: the authors make the JFrame subclass an ActionListener, listening to the "Delete" button and the "Add" button. This is very common in Java books, but that doesn't make it right. Making the JFrame an ActionListener on multiple widgets means we have some tightly coupled pieces of code (reformatted - I like the braces to line up):
public class CatalogEditor extends JFrame implements ActionListener, ListSelectionListener { ... private void addAddButton() { addButton = new JButton("Add"); addButton.addActionListener(this); getContentPane().add(addButton); } ... etc. (similar code to add "Delete" button)... public void actionPerformed( ActionEvent e ) { // blech... nasty. if ( e.getSource() == deleteButton ) { deleteButtonClicked(); } else { addButtonClicked(); } } ...This is nasty because it couples the JFrame too tightly with the buttons it is listening to. If you add another button, you also have to change the actionPerformed method. You would rather not have to change multiple pieces of code when you add "just one more thing". There is a better way (using anonymous subclassing) - don't declare the JFrame to be an ActionListener; instead, pass anonymous implementations of ActionListener to the buttons like so:
public class CatalogEditor extends JFrame { ... private void addAddButton() { addButton = new JButton("Add"); addButton.addActionListener ( new ActionListener() { public void actionPerformed( ActionEvent e ) { addButtonClicked(); } } ); getContentPane().add(addButton); } ... etc. (similar code to add "Delete" button)... // now there is no need for an actionPerformed method that uses // cascading if statements.I also want to point out the code duplication we have: creating the "Add" button is almost the same code as creating the "Delete" button (not shown). That kind of duplication seems to be common in Java GUI programming, but it is really unforgivable when you have lots of dialogs and buttons. What can we do about it?
First, we could have a DialogCreator class that reads the button names from a text file, creates them, and adds them to the ContentPane of the JFrame subclass (which it could also instantiate). It could also read the layout name and other information needed to lay out the GUI. In order to get the "Add" button's listener to call the method "addButtonClicked" in the JFrame subclass, we need to use the Java reflection API. Here's the pseudo-code:
class ListenerCaller implements ActionListener { public ListenerCaller( Object objectToCall, String methodNameToCall ) .... public void actionPerformed( ActionEvent e ) { ... reflection to call methodNameToCall of objectToCall... } }Notice that the ListenerCaller class doesn't care what kind of class it is calling. You could connect your GUI buttons to your model classes directly this way. Apple's Cocoa does something similar - the Cocoa widget classes (menus, buttons, etc.) take an object (called "target") and a method selector (easily created from a string) and call whatever object and method that you specify. You would normally do this specification in the Interface Builder.
class DialogCreator { ... public void CreateDialog( textfile ) { String jfClassName = read JFrame subclass name from textfile; JFrame jfObj = ... reflection to create object given jfClassname; String layoutName = read layout name from textfile; // also read number of arguments, etc... this can get complicated. Layout lay = ... reflection to create object given layoutName, number of arguments, argument types, etc.; jfObj.getContentPane().setLayout( lay ); while not end of buttons { String buttonName = read button name from textfile; String methodToCall = read method to call name from textfile; JButton aButton = new JButton( buttonName ); aButton.addActionListener( new ListenerCaller( jfObj, methodToCall ) ); jfObj.getContentPane().add( aButton ); } } }A single DialogCreator class like this removes button-creation code from all your Swing GUI code. Given that GUIs have a nested hierarchical structure, you probably want your text file to reflect that structure as well, using XML, and creating all kinds of widgets, not just buttons. It turns out that someone has already done all this work (and more) and called it XMLTalk.
I haven't used XMLTalk myself, because I haven't needed to do very much Java GUI programming, and I'm not sure about its license agreement. I haven't downloaded the source, but I have looked at the white papers and other documentation. It is worth studying to see how ValueModel adapters and other helper classes can be used to decouple your GUI code from the model code, and to decouple some of your model classes from each other. I should point out that XMLTalk is based on concepts first(?) implemented in VisualWorks Smalltalk. You should check out their product as well.
2003.May.26 MonI'm concerned that people are getting sloppy about their usage of the word "refactoring." I had a short dialog with someone asking about refactoring, bug-fixing, and testing, where the questioner revealed that he had not read Martin Fowler's book "Refactoring: Improving the Design of Code." If he had read the book, then he would have known that refactoring is improving the design without changing behavior, and could have avoided problems from combining bug-fixing with design improvements.
If you're writing or fixing code, particularly in an object oriented language, you must buy and read
Refactoring: Improving the Design of Existing Code by Martin Fowler.
That book and Design Patterns by Gamma, et. al. are essential reading no matter what methodology you might be using.
In test-driven design, refactoring is the design step: test, code, refactor, repeat.
Martin Fowler has a blog/wiki (a bliki) here: http://martinfowler.com/bliki/, which I'm adding to my blog-roll. (I'm delighted to see I'm already on Fowler's blog-roll.) I'd love to make my site into a bliki, but unfortunately, my site is static (updated only when I'm adding a new entry) - there's no cgi. My time is full with working, reading, and blogging, so I probably won't get around to writing my own bliki software (though it would be a good exercise.)
That questioner I mentioned was asking about refactoring, bug-fixing, and testing legacy code. I forgot to point him to Michael Feathers, currently working on a book titled Working Effectively with Legacy Code, and the mailing list discussing the book, http://groups.yahoo.com/group/welc/, but I'll do that after finishing this blog entry.
Here's my advice on this topic.
First, keep refactoring and rewriting (bug-fixing) separate. These two activities may be done only five minutes apart, but you're wearing a different "hat" during each activity. Refactoring is improving the design of the code, while preserving its behavior. Bug-fixing is changing the behavior.
Since unit-testing legacy code is almost always difficult, try writing some automated acceptance tests before doing other changes. The acceptance tests will hopefully not need to be changed much during refactoring and bug-fixing activities.
When confronted by legacy code, my first step would be to review the sources, fixing up formatting problems - indentation and spacing. Badly-designed code is often badly formatted. This gets me familiar with the code without changing its behavior. This helps me understand the code. Doing this might seem too boring to do with a pair programmer, but the alternative would be to do code reviews, which are even more boring. If automatically reverse-engineering class diagrams and other UML diagrams is possible, I would do that.
If I have a bug to fix, I recommend writing an acceptance test or unit test that shows the bug's effect. Then track down where the bug is - what defects in the code cause the bug. I know from experience that it is really hard to get into the habit of writing the test before doing the debugging; I do think this test-first habit is a valuable one.
When you've found the code-defect, it can help a lot to do a little refactoring to make fixing the bug easier. Run the few tests that you now have and do manual testing to make sure that the refactoring hasn't changed any behaviors or broken anything. Then fix the bug, and run the tests to confirm that the bug is fixed. Then maybe do a little more refactoring in that area of the code and re-confirm that no behavior has changed from that refactoring.
At the beginning of this bug-fix/refactoring process, you will have to move very slowly - manual testing will take a lot of time. Trying to do too much at once will only slow you down when you find you've broken something and need to back-track. Don't forget to use source-code-control - check in frequently.
Resist the urge to refactor working code until at least the worst known bugs have been fixed.
For more on refactoring, check out http://www.refactoring.com/.
2003.May.23 FriFirst Draft: I want to work in a happy team writing software that helps creative people make stuff.
What kind of stuff? art, 3d graphics, movies, games. I'd also like to help ordinary people be more creative as well.
Software developers are creative people too. Hmm.
Second Draft: I want to help people be more creative and productive in making stuff, working in a happy team, producing software that helps people be creative.
I don't have to be writing the software myself, I could be a coach or mentor, or project lead. I've been a successful project lead on a few occasions, and I think I would be a better project lead today. Happy doesn't mean there are no disagreements, but it does mean that people value and respect each other.
An interesting analogy to programmers is honeybees. Orson Scott Card writes about it here: http://www1.things.org:8080/~muffy/pages/life_work.html. Card writes:
"If a software company provides such a hive, the coders will give up sleep, love, health, and clean laundry, while the company keeps the bulk of the money.... All successful software companies had, as their dominant personality, a leader who nurtured programmers. But no company can keep such a leader forever.... The shock is greater for the coder, though. He suddenly finds that alien creatures control his life. Meetings, Schedules, Reports. And now someone demands that he PLAN all his programming and then stick to the plan, never improving, never tweaking, and never, never touching some other team's code."
The goal of the honeybee is to produce honey. The delight of a programmer is to produce code. He/she wants to be appreciated for that skill. Beekeepers may move a hive to be closer to the flowers (making it easier for the bees to make honey), but the beekeeper doesn't tell the bees to fill a quota, backed up with threats of punishment. Make it easier for the programmers to program and the bees to find honey, and then step out of the way.
There are ways to get the programmer honeybees and those alien 'marketing' creatures more cooperative. Plans are OK, as long as they have appropriate levels of detail, and are not used as whips in a master-slave relationship. Reporting can be done in ways that don't impose burdens on the programmers. Let business steer at the non-technical levels and let the technical people do the things they need to do, to produce good software that supports that business.
2003.May.20 TueEsther Derby reminds us that learning comes from practice and feedback, (not just a training course) and links to one of her articles at StickyMinds.
Aside: StickyMinds is the on-line forum associated with Software Testing and Quality Engineering (STQE) magazine, which published my article on adopting XP last year (July/August 2002, order the back-issue!).
Extreme Programming defines "12" practices, though different ways of counting takes it up to 13 or more. Don Wells lists way more than that at http://www.extremeprogramming.org/rules.html. You should know that "they are just rules" meaning that you won't lose your license to program if you break a rule accidentally or with good reason, though failing to follow all of the practices often makes more work for yourself and your team.
Practicing is becoming a popular topic. Ron Jeffries wrote of "etudes" and Dave Thomas is writing about "katas". Other web-logs have discussed this topic, with analogies to music practices, authors writing drafts of papers or books, and so on.
William C. Wake has posted test-first "challenges" on the XP mailing list in the past, for people to practice on, and now has a book called The Refactoring Workbook due in July. Preview chapters are on-line: http://www.xp123.com/rwb/.
On other topics,Mike Cohn is writing a book on User Stories (Requirements/Features in XP lingo) and set up a web-site on this topic at http://www.userstories.com/. I haven't read the chapters that have been made available yet, but I plan to soon.
AndDavid Anderson recommends a book on the agile management mailing list: What Management Is by Joan Magretta. He's enthusiastic: "This book was nominated by no less an authority than The Economist as one of the 10 best books of 2002. That's what prompted me to read it. In summary - it is worth every cent! It's management summed up in a nutshell."
2003.May.19 MonTest Driven site, Blogger Security, Corporate Cultures
Dave Astels points out another TDD resource: http://www.testdriven.com/.
James Robertson points out a security issue with the blogger API: "This is why my posting API uses encryption, and why I don't support the blogger api on this blog - that pathetic excuse for an API passes usernames and passwords in the clear." My blogger friends need to make sure they change their passwords frequently.
Interesting article by Art Kleiner about multiple corporate cultures here: http://www.well.com/user/art/s%2Bb32001.html. He describes three cultures that live within single companies:
The first, the operational culture, is the culture of day-to-day line managers - the people who get products and services out, procure supplies, process bills, and make delivery trucks run on time. Operations people appreciate teams; they understand, as nobody else does, how to get a bunch of disparate individuals to pull together. If you want to find facility with deals, leverage, and capital flow, you have to look to the second corporate culture, the executive culture. Members of this culture typically include the CEO, the board, the business-unit leaders, and the finance-oriented staff.... their jobs and passions lie in shepherding the cash flow that keeps the organization alive. The third corporate culture, the engineering culture, is personified by engineers and technical specialists, particularly in information technology and process engineering. They are stimulated by puzzles and problems, and by the design challenge of creating an ideal world of elegant machines that operate in harmony. He also describes briefly how corporate cultures at the CEO level have changed over time. The most successful companies were "monopolist" until anti-trust laws were passed; "manufacturing" after that -- that is vertically integrated "operations" cultures; "marketing-oriented" after the Great Depression and up to the "golden age of advertising in the 1960's"; and "finance-oriented" oriented after inflation hit in the 1970's.
Now that we are living in what seems to be deflationary times, what CEO-level culture will become successful?
2003.May.17 SatA Series on Test Driven Develolpment
Software Development Magazine has been publishing a column "The Craftsman" by Robert Martin. A series of essays in the form of a conversation between apprentice and master, with code snippets in Java. Check it out: http://www.sdmagazine.com/columnists/martin/. Thanks to "Marc Hamann" and "Demyanovich, Craig" for mentioning it on the Test Driven Development Mailing List.
David Astels is holding tutorials on TDD at various conferences. Check out his list here. He also is offering training according to this page.
Robert Martin's Object Mentor does training on TDD, XP, and other topics. Check out the course catalog.
Joshua Keriefsky's Industrial Logic does workshops and "experiences" on TDD, XP, and other topics. Check out their workshop page.
This page lists training and trainers, but some of it is out of date: http://c2.com/cgi/wiki?XpTrainingClasses.
I can teach (one person at a time) an introduction to test-driven development if the student is willing to spend a an hour or two in front of a computer with me, on a Saturday or Sunday. (C, C++, Java, Objective C.) Send me an email if you're in the SF Bay area ("Silicon Valley" area). If I get lots of responses, I'll have to charge for this tutoring. (Perhaps you can teach me how to use VisualWorks Smalltalk.)
Of course, the books on TDD are presently few:
Test Driven Development By Example by Kent Beck Agile Software Development: Principles, Patterns, and Practices by Robert Martin. Test-driven Development: A Practical Guide by Dave Astels, which should start shipping summer 2003. It goes without saying that TDD is cool, useful, productive, fun, challenging, rewarding, and the way everyone will be developing software in the future.
Oh. I forgot Ron Jeffries's essays on TDD in C# here: http://www.xprogramming.com/xpmag/index.htm
2003.May.16 FriAn Example of the Power of Smalltalk
Here's a tidbit from James Robertson's http://www.cincomsmalltalk.com/blog/blogView.
James is using a binary file format to support his blog code (which is written in VisualWorks Smalltalk, natch.) Apparently the "class library" he is using is called BOSS, comes with VisualWorks, and it supports schema migration.
In the code snippets here, he has two very powerful lines:
oldObj become: newObj.
oldObj changeClassToThatOf: inst.
If I'm reading this right, this is transforming the old object (of an old class type) into a new object (of a new class type). All references to the old object will now be referring to the new object. (This is a big deal!)
Java can't do this (and perhaps shouldn't, since it could be a security risk when running code received over the internet in one's browser - if anyone does that anymore). C++ can't even get close. The closest that Objective-C comes is to have a class "pose as" another class.
From what I've read, the message "become:" is not used much in everyday Smalltalk programming, but it required by the Smalltalk IDE for letting users modify existing classes -- classes are objects too, so you want to make the existing class object "become" the modified (new) class object.
James points out that his blog's files are stored via BOSS are in a variety of old formats, since he's added categories, trackbacks, and so one over the last year or so. He doesn't have to read those files and save them in the current format; he just lets their data get converted at run-time when (and if) they need to loaded into memory.
2003.May.15 ThuAgile Project Management Paper and Statistics
Get a free copy of a Cutter Consortium report: "Agile Project Management: Principles and Tools", by Jim Highsmith, at http://www.cutter.com/offers/APM.html. Forty pages, well worth reading (and a "$150 value!").
A few quotes from that report:
On agility: "We are no longer talking about 15%-20% scope creep on projects; we are talking about everything - scope, features, technology, architecture - changing within the span of six months.... The traditional project management maxim of 'conforming to plan' fails dramatically in these situations."
On traditional project management, he quotes someone in the construction industry saying "The ... PMBOK... is based on two underlying theories: management-as-planning, ... and the thermostat model... both theories can be shown to be heroically simplistic and insufficient from the point of view of project management reality."
Highsmith describes the "agile project management model" with a graphic that I'm trying to reproduce in text form here:
1. Envision 2. Speculate (create Feature List and Feature Plan) 3. Iteratively Deliver Features (from Feature Plan, create Tested Features) 4. Monitor and Adapt (from Tested Features, create Final Product, Adaptations jump back to Speculate) 5. Close Traditional project management focuses on activities, while agile project management focuses on delivery. Highsmith writes "Teams that focus on activities get lost, and even worse, they often don't realize it. Earned value analysis ... has nothing to do with value but with cost accounting based on planned tasks versus actual tasks. The value of what those tasks actually deliver gets lost."
Cutter also has a survey paper here that I summarize below:
Survey of 200 IS/IT managers
Location
33% North America 20% Europe 10% Australia 8% India 8% Asia Industry
39% software companies 11% financial services 9% consulting or like services 6% state or national governments 5% telecommunications Heavy Methods in use...
51% RUP 27% CMM-compliant (!?) 26% ISO-9000 compliant (!?) Agile Methods in use...
54% in-house developed agile method 38% Extreme Programming 23% Feature-Driven Development 22% Adaptive Software Development 19% Dynamic Systems Development 9% other 8% Crystal Clear 7% Lean Development 3% Scrum This article predicts that 50% of companies will have some agile projects "by 2003".
The bit about CMM and ISO-9000 as heavy-weight methods is a bit confusing, since compliance with those standards doesn't necessarily require forgoing an agile process.
2003.May.14 WedDinosaur Brains versus Survival Rules
Way back in 1989, there was a book I read called Dinosaur Brains by Bernstein and Rozen. I thought it was a good read at that time. Their thesis is that irrational behavior, including territoriality, fear, flight, and aggression, are remnants of our reptilian ancestry, and the book gives advice on appeasing and working around these behaviors. By the way, the book is still in print, in paperback.
The problem with this idea is that reptile behaviors are hard-wired... they can't be changed. A better idea is "survival rules" that we learn as children (or later) and apply in sometimes inappropriate situations. When we identify our survival rules, we can modify them.
Wayne Strider's book Powerful Project Leadership talks about survival rules, as does Jerry Weinberg in Becoming a Technical Leader: An Organic Problem-Solving Approach and More Secrets of Consulting: The Consultant's Tool Kit.
An example I read about on a mailing list recently was pair programming. Someone (I think it was Robert Martin, but it could have been Joshua Kerievsky, or someone else), described a programmer that had some difficulty adapting to pair programming.
To make her comfortable with it, she would study the task/problem for some time by herself before pairing. The writer observed that she wasn't comfortable with being asked questions that she didn't know the answer to. Eventually, she got comfortable with pair programming, and no longer needed to study before pairing. It seems to me that her survival rule was something like "I can't show anyone that I don't know the answers" and she successfully modified the rule.
2003.May.13 TueThe Standish group, in their Chaos Report, reports that 85% of the features implemented in the average IT project are NOT not used by the customer.
If you knew which features the customers were really going to use, You could avoid wasting 85% of your budget, and you could ship 85% earlier than the average software project, getting revenue sooner. Sounds like a win-win for both customer and supplier.
This applies to shrink-wrap software as well as in-house projects. id software (if I recall correctly) released limited-features (free) early versions of their breakthrough game Doom to get feedback from their users and get them addicted to their software.
How do you find out which features the customer really need? Ask them. Get them involved in the product with early releases. If possible, get a representative customer involved in the planning and feature definition.
Eliminating waste (like that unnecessary 85%) is one part of Lean Manufacturing. Like myself, Kent Beck did not view "manufacturing" as an appropriate metaphor for software development until he started reading about Taiichi Ohno's Toyota Production Method. Here's a small write-up: by R.Balakrishnan.
Lean Manufacturing, like XP, often encounters what seems to be irrational rejection (even after demonstrating success). On the XP mailing list Kent Beck writes:
I'm reading Lean Transformation, which is full of stories of dramatic improvements in manufacturing quality and productivity that are immediately dismantled as soon as the consultants leave the building, even at the eventual cost of the jobs of the people doing the dismantling. Which is to say, you are not alone.
Harvard Business Review has an article this month about Bill Bratton, who has turned around 5 police organizations, including the whole of NYPD. You should read the article, but the sequence I took away was:
Put the managers in direct contact with the problem Focus you resources on the single worst problem Enlist powerful allies Marginalize or eliminate naysayers Beck also points to a paper [a zip file containing a pdf] on adopting Lean Manufacturing which is titled "First, Do No Harm!" by Michel Baudin. The paper recommends converting to Lean with pilot projects that show rapid improvement and pay for themselves. At no time should converting to Lean cause a drop in revenue or delays in delivering product. Baudin also points out that you can't convert to Lean instantly, because it requires learning new skills. Not unlike the new skills needed for XP.
2003.May.11 SunMy mother is a very organized person, or at least, that's how I perceived her to be during my childhood. When we went on camping trips, we had everything we needed, and we knew we had everything because she made checklists. I think she prepared the lists long before the trip, was able to check off items as they got packed into the car, and kept the list to reference and update during the trip. (They may have even been laminated.) If anything was found to be needed or desired during the trip, I'm sure Mom added it to the list, so it would be packed on the next camping trip. I can't imagine the settlers in their Conestoga wagons being more organized.. or packing more stuff in a relatively small area. Mom not only had checklists, but she had customized trunks for food, cooking-gear, and other specialized containers for other stuff.
This organizational skill might reflect a preference to "J" in the Myers Briggs Type Indicator. I come up as a "J" in the MBTI as well, but not a strong "J". When my wife and I go on camping trips, our checklist, if we have one at all, is written at the last minute, and lost once we're on our way. We haven't forgotten the can opener, but on one occasion we forgot sleepwear. We can add items to our next camping-trip checklist only if we remember what we forgot last time. We are in fact, sometimes relying on checklists in a camping book, because the book doesn't get lost as easily as a scrap of paper does. To pack our stuff, we have to rely on commercial containers, or let stuff just pile up in the back of the station wagon. We do have, thanks to Mom, a very nice, portable, container filled with paper plates, paper napkins, salt and pepper, plastic cups and utensils -- picnic supplies, because she heard our need and responded. Thanks, Mom.
Now why do I and my wife (both J's according to MBTI) have so little organizing skills for our vacations? My wife says she works hard at organizing at the office, and prefers to be lazier at home. (Don't tell her I just published that.) Maybe it is because in my childhood, I was not part of that organizing - I didn't help develop the check-lists or build the specialized containers. Maybe its because I didn't take classes on office management, like my mother did in college. Or maybe we're really Myers Briggs type "P" and have been trained to act like "J"s.
David Smaltz in his book "The Blind Men and The Elephant" makes a point of saying that a team can produce a plan or an organization that seems self-evident to them, but which would not be understood by another team without explanation. In my workplace, I know of a lot of people who would like to know more about the manager's plans for projects, particularly to be able to cooperate better with other teams. One employee told me his manager only provided information on a "need to know" basis, which made working for that manager more difficult than it had to be. Quite often I don't know that a project at my company has chosen to use one of the components I'm responsible for, until I get bug-reports at the end of the project. I'd really like to know before they write the contract with the customer, so I can make sure that we have time and resources to support whatever changes they are planning, and at least get them to pick the right version of the component.
For more on Myers Briggs, see Naomi Karten's book on Communication Gaps.
Also check out Gerald Weinberg on MBTI and other preferences in his Quality Software Management series, particularly volumes 3 and 4: Quality Software Management: Congruent Action (vol. 3), Quality Software Management: Anticipating Change (vol. 4). 2003.May.10 SatFeedback, the Secret of Agile Development
Tomorrow... a special blog entry for Mom.
A paper of mine, about feedback in Agile / XP projects, has been published on the AYE Conference website: http://www.ayeconference.com/Articles/Secretofagile.html. That paper concentrated on testing and customer involvement, but there is more feedback possible: there is feedback from members of the developer team, management, QA, and other stake-holders associated with the project.
The Standish group, in their Chaos Report, reports that the top three project success factors, in order of most important to less important are:
User Involvement Executive Management Support Clear Statement of Requirements Other success factors that also support my point are:
Realistic Expectations Smaller Project Milestones Competent Staff Ownership Clear Vision and Objectives In typical waterfall projects, user involvement is only at the first and last stages: requirements gathering and final testing (and sometimes users are not involved even in those two stages!). This obviously isn't enough, since IT projects, according to the Standish group, are only successful an average of 12% of the time.
Agile methods aim for success by requiring more user involvement through the life of the project: the practices of On-Site Customer and Frequent Releases, among others. User involvement also helps with maintaining realistic expectations and getting those requirements clearly understood.
Even with user involvement, there are project failures from lack of management support. Most of the stories of unsuccessful projects I've heard at BayXP are due to lack of executive management support - failure to get buy-in from the rest of the company. In some cases, the project was on-time and on-budget, but the rest of the company rejected the process and/or the people involved.
Successfully adopting XP requires a company culture compatible with its values, and many companies just don't have that culture. To get to XP, the company would need a culture change, which requires skilled, dedicated, change artists and a company-wide perceived need for change. Thus, Industrial XP does a Readiness Assessment to see if there is a culture fit.
Agile methods have smaller project milestones, a proven success factor, as well as frequent releases. Getting the product out in front of real users not only enables more feedback, but can be financially rewarding as well, by reducing time-to-market or opening a new market. For internal projects, "small releases" to some portion of the end-users is a proof of concept, demonstrating to management the viability of the project.
Competent staff is a sticky point. No method can work well if the people don't know how to do it or don't understand it. Many XP project failures result from staff that had no training on the practices and didn't practice all of the recommended practices. I've already written that XP's test-driven-development requires testing skills and refactoring skills that both inexperienced and experienced programmers may be lacking. Training and practice, whether by intensive, disciplined, self-training, or training from good teachers, is necessary for those new to XP.
Lack of competence in test-driven-development shows up in unexpectedly-failing acceptance tests, bugs slipping through to users, and decreasing velocity (features done per iteration). Developers puzzling over the bug-reports, unhappy users and managers, and increasingly-unmaintainable code probably realize that something is going wrong, but without a coach or someone experienced in software development, may not know how to fix it. In the worst cases, they ignore this feedback and deny that anything is going wrong.
Industrial XP's additional practices - Project Chartering, Project Community, Test-Driven Management, and so on - address the factors of ownership and clear vision. If the project starts to drift off course, members of the project community can point to the Charter and Manager-Tests and ask "why are we moving in this direction?" IXP's practice of doing Retrospectives regularly (mini-retrospective each iteration, as well as larger ones at longer intervals) will also help with clear vision of the objectives and ownership of the process, providing feedback to the participating members of the project community.
The hardest kind of feedback to specify in a "process" is personal feedback from people in the team, whether in pair programming, retrospectives, or otherwise.
Naomi Karten's book on Communication Gaps can help.
Many people I respect recommend What Did You Say?: The Art of Giving and Receiving Feedback by Charles N. Seashore, Edith Whitfield Seashore and Gerald M. Weinberg.
Norman L. Kerth wrote the definitive book on Project Retrospectives, but you shouldn't try to do a large retrospective without a skilled facilitator. Read the book to see why.
The larger community is also a valuable source of feedback and learning. Go to your local Agile or XP group meetings, join or at least read the XP, IXP, and/or TDD mailing lists. Read the books. Read the web. Go to conferences if you can. 2003.May.09 FriOpen, Collaborative, Project Planning
Mary Poppendieck on the Lean Development mailing list writes:
The difference between a planned and a market economy is rooted in two different management philosophies:
Management-as-planning/adhering focuses on creating a plan that becomes a blueprint for action, then managing implementation by measuring adherence to the plan. Management-as-organizing/learning focuses on organizing work so that intelligent agents know what to do by looking at the work itself, and improve upon their implementation through a learning process. and links to Laurie Koskela and Greg Howell's article at: http://www.cpgec.ufrgs.br/norie/iglc10/papers/47-Koskela&Howell.pdf.
I've recently read two books on project management:
David A. Schmaltz's book The Blind Men and the Elephant: Mastering Project Work Eliyahu M. Goldratt's book The Goal: a Process of Ongoing Improvement (a business novel written with Jeff Cox). David Schmaltz makes a point in his book that a plan or organization done by one person or group will not be self-evident to another person or group. With that in mind, The Goal was somewhat frustrating, because the main character, who is changing the way his factory works, isn't involving the people on the factory floor in designing his process changes (he is involving other managers). In fact, of the few factory-floor union workers depicted in the novel, all but one of them are depicted as stupidly getting in the way of the process changes. I suppose one man against the world makes for some drama in a novel, but doesn't work that well in real life.
Goldratt writes in the after-word of The Goal that major obstacles of process improvement discovered by his readers were:
Lack of ability to propagate the message throughout the company. Lack of ability to translate what they learned from the book into workable procedures for their plant Lack of ability to persuade decision makers to allow the change of some of the measurements which certainly shows that involving others is necessary for plans to work.
I also found The Goal frustrating because he made no mention of Deming, whose collaborative techniques for process improvement were proven "long ago" in Japan, and predated Goldratt's techniques.
2003.May.08 ThuWhat's the difference in a bug in a new feature, and a bug that suddenly appears in an old feature? In a new feature, one is getting added value imperfectly. In an old feature, one that used to work bug-free, but is now broken, someone on your team is subtracting value.
For a very long time, experts like Jerry Weinberg have been recommending careful code reviews of maintenance changes, because a bug in a deployed system (like payroll or billing) can result in multiple millions of dollars in losses.
Someone inserting defects into working code is creating waste: not just the risk of printing bad checks or bills, but creating work for themselves and others to identify and fix the problem they created.
The danger in maintenance or incremental development is that an innocuous change breaks an existing feature. Extreme Programming deals with this danger in three ways: pair programming (continuous code review), extensive automated unit tests, and automated acceptance tests. Plus, most if not all XP teams have manual testing as well.
Many teams doing test-driven-development (a core practice of XP) report that their unit-test code coverage is nearly 100% statement/branch coverage without taking extra effort. This gives them the freedom to make an innocuous change (as part of a refactoring, for example), and see within minutes if it breaks anything. If it does, they can Undo their change, or evaluate the change more carefully.
2003.May.07 WedBrad Appleton, on the XP mailing list, recommends "appreciative inquiry" http://www.appreciative-inquiry.org/ and http://www.thinbook.com/chap11fromle.html. This technique lets people feel they and their ideas are appreciated, and focus on what's right rather than on what's wrong.
On Hal Macomber's blog Reforming Project Management, he wrote on Tuesday, April 29, 2003:
My favorite management author is Ken Blanchard author of The One Minute Manager and dozens of other books. [...] Blanchard implores people to focus on positive feedback. In his book Whale Done! he goes into how trainers never use negative feedback when working with dangerous animals. If killer whales can be trained to do the spectacular things that they do with only positive feedback, then why would we want to use negative feedback with the even more dangerous human beings?
Apparently, "Whale Done!" is also on video, dvd, etc... http://www.rctm.com/app/Product/71727.html
2003.May.06 TueExtreme Planning - the Home Game
Extreme Programming has planning at several levels.
Release Planning, where the entire project is divided into small user-centered features (stories), written onto index cards, estimated by the developers, and prioritized by the story-writer. The story-writer tentatively assigns the stories to iterations (tentative, because the amount of work that can be done per iteration is only a guess when they do their first release plan), with an end-user release planned in three months or so, when enough features will have been implemented to make a release worth-while.
Iteration Planning, where a set of stories is selected by the story-writer (just enough to fit into one iteration as measured by the previous iteration). The developers flesh out the stories with the story-writer and (in some teams) break down the stories into tasks, writing the tasks onto index cards. Many teams put the cards and tasks are put on the wall.
When two developers get together to pair program, they may break down the task (or story) that they are working on into list of tests to do, and a design in UML or whatever notation they think best. This may be written on index cards or a white board. Then they implement one test at a time in test-driven-development.
I asked on the IXP mailing list if people had used the Extreme Programming's Planning techniques (sometimes called the Planning Game) for organizing home chores or other non-work things. Yes they have.
Jiva DeVoe planned a cross-country move on index cards, including estimates, posting them on the wall, etc.
Joshua Kerievsky tells us that Francesco Cirillo, who does XP training in Italy, uses XP-style planning to organize and execute a weekly dinner, with student volunteers doing most or all of the cooking.
Amy Schwab planned a wedding using her techniques of "project community" and "adaptive project methodology". She wasn't familiar with XP planning, but she also used "cards on the wall" (post-it notes instead of index cards). She let go of traditional project "control-oriented" planning, and let family and friends be in charge of "their own quality of experience."
Diana Larsen tells us that http://www.shesintouch.com/ does home chore planning similar to XP's planning, including using index cards, estimates, and measured velocity (and training workshops) [and this predates XP by 20 years].
It seems to work for lots of people, so I will give it a try with my wife. Guess who's be the "customer" in this planning session? Not me!
2003.May.04 SunExperience-based list of traps for those new to XP
Alex Chaffee has started a wiki page http://c2.com/cgi/wiki?XpPitfalls, making a list of pitfalls often encountered by teams new to XP. Erik Hanson made "major additions".
For example: "Problem: Refactoring is hard. Really hard. Solution: Practice, practice, practice."
This item is probably meant more for entertainment: "Problem: OAOO can lead to a nest of small objects. Writing new code or debugging you end up getting lost in a Smalltalky maze of pointers. Solution: Stop the madness. Trust your design instincts. Solution: Don't hire former Smalltalk programmers :-) "
Wyatt Sutherland reports a XP+Scrum success story for Blue Cross Blue Shield of Tennessee, on the XP Mailing List. A software project that supports BCBSoT's customer base of millions, managed via index cards and agile techniques.
2003.May.03 SatExtreme Programming requires new skills for both Managers and Developers; in today's entry, I write about some of the new skills required for Developers.
In typical "traditional" development (which is often waterfall / code-n-fix), a developer needs the skill to design from scratch. To some extant, this skill is taught in school and in books. Some developers master it better than others. Unfortunately, it doesn't help too much with maintenance, and about 90% of developer work is maintenance. Debugging is a very valuable skill for this kind of work, but only learned from experience. Testing is done after development, if it is done at all, (typically just manual testing is done) and very few projects keep automated tests running in maintenance mode.
Traditional Development State of Code Design from scratch Good "Patch" for new feature Less Good "Patch" for bug fix Even Less Good Repeat Patching for years Eventually Declared Unmaintainable In Extreme Programming, a developer doesn't have to make a good design from scratch. During the Test-Code-Refactor cycle, the developer's first stab at code doesn't have to have a really good design, it just has to pass the test. Then the developer "puts on his refactoring hat", and improves the code to good design. The project is in maintenance mode after the first week, and the discipline of refactoring keeps the design good throughout its life. Keeping the tests up-to-date is also required. It also turns out that pair programming and this test-first approach mostly removes the need for debugging.
Extreme Programming State of Code Write a Test No Code Yet Code To Pass Test Minimal Code, Possibly Poor Design Refactoring Good Design Write another Test No Changes to Code Yet Code To Pass Test "Patched" Code, Possibly Poor Design Refactoring Good Design If you compare the two methods, you see that XP requires testing skills, which are not emphasized in traditional methods, and it requires refactoring skills, which isn't (or wasn't until recently) taught in school, and is only described in very few, recently-published books.
Experienced developers, particularly those who have spent time in maintenance, know that patching leads to unmaintainable code, but many of them haven't learned refactoring, the "cure" for patching. Younger developers, particularly those that got started in the dot-com boom, haven't even had time in maintenance, and of course they are still learning how to do design from scratch.
I've spent the last several days teaching a developer how to write code test-first. Writing the test for a nonexistent class or method is fairly easy, and writing the code to pass that test is fairly easy. It can even have a good design, since it isn't being patched (yet).
Writing a test to drive modifying the behavior of an existing class can be more tricky. In some cases (equality functions), we had to regard a "passing" test as a failure, in order to follow the XP rule of never modifying the code unless there is a failing test.
One of my exercises for teaching is to have the student write a "prime factors" function, where I supply each test, and the student writes the simplest code to pass the test. The hard part in that lesson is learning to refactor "if" statements away to keep the design good. Remember the XP rule for good design: "no duplicate logic".
Sometimes, at the end of the day, getting the tests passing is such hard work, that we get tired and don't do the refactoring. We check in working code, and then the next day arrives and we forget to do the refactoring, moving onto new tasks instead. Pair programming is tiring enough that the experts don't recommend pairing more than six hours a day. The XP practice of "sustainable pace" is meant to keep you from programming while tired, when your judgment is poor.
An XP team might want to have a practice where developers coming in the next morning review the code that was checked in the previous day, looking for opportunities for refactoring, and doing it first thing, while the energy is available to do it. The next-day perspective could help provide the distance needed to recognize that yesterday's wonderful code isn't as wonderful as we thought.
Refactoring is similar to rewriting, but more disciplined: you can only make small, behavior-preserving changes. This discipline is a new skill.
Brooks said "expect to throw one away", Clarifying later that he meant small modules not entire systems, because throwing away entire systems leads to waterfall.
Cockburn said says it is easier to modify a working system than to write a new system from scratch. I say that it is easier to modify a working system than a non-working system.
Ed Yourdon said that any piece of code needs to be rewritten three or four times to be constituted as an elegant, professional piece of work.
Mary and Tom Poppendieck said that for prose, we take this [rewriting] for granted, why not for code?
2003.May.02 FriRewards, Teamwork, and Ropes Courses
Esther Derby, on her blog, mentions a few reward-structures that work against teamwork. She writes:
Many companies throw up organizational barriers to collaboration:
reward structures that focus on individual achievement recognition of individuals for group efforts disbanding groups that do work well together (the "seeding" principle) promotion of individual stars And most "team building" misses the mark. Ropes courses, pep talks, and inspirational posters are a crock.
Once upon a time, I and my coworkers were sent to a Ropes Course once. One thing being emphasized was the fact that old people, when asked what they would have done more of, never said they'd spend more time at the office. (Those who sent us on this course probably didn't know about this lesson ahead of time. Or maybe they did.) We also learned about providing "back up" to each other (support), not using joking put-downs on each other, and were given some time to think about our individual priorities.
Many of the employees were energized and tried to improve things at the office when they came back from the Ropes Course, but became frustrated and bitter because while many of the employees who came back did change (more teamwork and respect among coworkers), but the managers who came back from the course didn't change -- they continued a long history of backstabbing and turf-building.
Some employees quit to get better jobs and spend more time with their families. The company's president, who wanted this course to help turn the company into a focused team, did not achieve her goal: the managers under her did not change their behavior, even though the employees did change, for a short while, until they realized that the managers were not providing "back up" to their workers. They probably could have used Retrospectives, Charters, and all sorts of good things, but as long as the managers were rewarded for back-stabbing and turf-building, the situation would not have improved.
One of my priorities was/is "fun". (The IXP value of "Enjoyment".) The IXP mailing list has been discussing this, including other names like "Satisfaction". The most fun I have had in work comes from working in teams of respectful coworkers, with collaborative design (and since I've discovered XP, pair programming).
Pairing can be tiring, and with the some people it can be VERY tiring, but the fun comes from getting work done quickly, with someone to provide "back up" (catching my mistakes), and with me providing "back up" to the pair when he's keyboarding. Getting code to pass the latest test (or sometimes, getting the test to fail) is a shared triumph, repeated many times during the day. Getting our changes checked into source code control several times a day is also a shared triumph, particularly when you can announce that this check marks the completion of a story/feature.
Even formal code reviews can be a little bit fun, if the people in the review respect each other, and have a sense of humor about the mistakes we can all make.
Check out more about Pair Programming at http://www.pairprogramming.com 2003.May.01 ThuIs your company's reward structure counter-productive?
A recent issue of Inc. magazine had a article where a sales manager said, based on his experiences, that he'll never recommend using commissions to reward salesmen, because it damages the company that does it; specifically it works against team-work and causes the company owners to not trust their salesmen. Check it out: http://www.inc.com/magazine/20030501/25416.html.
Some managers complain that pair-programming prevents them from seeing clearly which programmers are accomplishing more than others, to be able to reward their individual accomplishments. Emphasizing individuals over teams rewards territoriality, rewards competition more than cooperation. Cooperation is more productive than competition at getting products developed and shipped. Do you really want to discourage cooperation?
When programmers are working solo, how do you know they are actually doing something useful? Could it be that someone is working alone because what he's doing is NOT useful to anyone else?
Software is a team effort. Even when its done by a bunch of soloists, they are still using each other's work by building on each other's libraries and code. Could it be that the guy who demos a flashy application has only done some of the GUI, and the functional parts of the application were done by his coworkers? Are you going to reward the flashy demo guy because he can show you something more visible than the other programmers who wrote the functionality?
Robert Martin wrote on the IXP mailing list:
In one case we transitioned a team that had technology project leads. There was a lead for the database, a lead for the middleware, and a lead for the UI. (I'm not kidding!) When we transitioned them to XP these folks rapidly realized that the practices of pairing and collective ownership were diluting their power base. Two of them made abortive attempts to seize power back, and wound either resigning or being fired. The team was much happier once they were gone.
Are you rewarding last-minute, late-night heroics? When you do XP well, you have measured progress every iteration, and you can predict how much scope can be accomplished a long time before the end of the project. Check out what happens when rewarding heroism meets XP...
In another case we had a cowboy. This fellow used to rescue the project at the eleventh hour by pulling all nighters. His manager learned to depend upon this behavior and just "knew" that the cowboy would save him at the last minute. As we transitioned to XP the cowboy actively sabotaged the planning meetings and the iterations. He did not want the team to make progress. We went through three iterations of *zero* velocity. (Count them, zero.) And then the manager dissolved the team and reformed it months later in a different way. (I'll leave it to you to predict how it was restructured.) -- Robert Martin.
Do you think that even before XP came along, that this cowboy might have been doing some sabotage to make sure his heroism would always come into play? I think XP just made it more visible.
Ken Auer, who runs a RoleModel Software, a consulting company that does XP, writes:
I've always had a reward structure here that was based on the concept "share the risk share the reward" while still acknowledging that "socialism doesn't work... Some people do provide more value than others". At first, I could only come up with a "how much money did we make due to your billable hours" measures, with a few other criteria to make sure it wasn't abused. Worked fine while everyone was billable most of the time. When it wasn't, it had people lobbying for billable time against each other e.g. "I think I should be on this project (instead of Joe) because...".
Finally, I came up with a formula for individual value to the company that could be identified, peer-reviewed, and steps taken to move the number up. That number was used to figure out shares in the pool of money. It wasn't perfect, but it completely got rid of all of the bad behavior (fighting against each other) and produced good behavior, i.e. I got help figuring out how to divvy up resources for the best of our company, the client, and the individuals.
Amy Shwab reminds us: In the January Harvard Business Review (all about motivation) there was a wonderful piece about the longer term effects of performance metrics and tying results to those metrics. The company, indeed, got what they were rewarding but it did not translate into what they wanted or needed. It fragmented the organization and served the customers very poorly. The most important things are often the most difficult to measure -- like 'fun'.
Developing intellectual property like software is always a learning experience. External rewards can hinder learning. Kohn said "Rewards are most damaging to interest when the task is already intrinsically motivating." Check out "Punished by Rewards? A conversation with Alfie Kohn".
Rachel Davies and John Nolan will hold a workshop on this topic at the Agile Development Conference
. Thanks go the people mentioned above, and Brad Appleton and Steve Berczuk on the XP mailing list. 2003.Apr.30 WedDiana Larsen on Changing Your Organization
Diana Larsen's article on change and learning (and XP) http://www.cutter.com/itjournal/change.html 12 pages (PDF).
She quotes Beckhard's formula for change (my paraphrasing): If dissatisfaction with status quo, plus desirability of change, plus clear definition of what/how to do, are greater than the resistance to change, then you can achieve the desired change.
She says to encourage change, market it by increasing awareness of the problems with the status quo (I see a risk of being called names like "negative" or "not a team player") and the communicating the desirability of getting to a better situation. "When you are implementing change, there is no such thing as too much communication."
Some of this runs counter to Jerry Weinberg and another (name forgotten) book. Jerry says "don't promise more than a 10% improvement." A manager doesn't want to admit that more improvement is possible, because then they would have to admit that they were not doing a "good job" before. The (name forgotten) book pointed out that too clear a picture of the future can be paralyzing because people can see the perceived drawbacks of that situation too visibly, while not appreciating the benefits.
She writes "XP has the advantage over many change efforts in that fast iterations build in the feedback loop for short-term success. While floundering through the chaos, nothing bolsters the participants in a change effort like the sense of progress from a quick 'win.'"
Larsen recommends Chartering to start a project, and agrees with Lindstrom and Beck on "Hold two-day professionally facilitated retrospectives each quarter." (And at project end.)
Change takes time. "Putnam points out the need for patience with change efforts as he maps out six months' worth of defect tracking and shows its consistency with Satir's [change] model. He notes that if you had an evaluation of success or failure after three months, you might have come to an erroneous conclusion."
Also check out Rick Brenner's "Fifteen Tips for Change Leaders" here: http://www.chacocanyon.com/essays/tipsforchange.shtml
2003.Apr.29 TueRelationships, Traditional vs Lean Training
Very funny web-page: "Things my girlfriend and I have argued about" http://homepage.ntlworld.com/mil.millington/things.html. This is one bit my wife and I found funny: "Just for reference; if Margret returns from having her hair cut and says, 'What do you think?' and you reply, 'I'd love you whatever your hair was like,' well, that's very much The Wrong Answer, OK?"
Rus Rufer, on the IXP mailing list, mentioned two lists comparing traditional and lean project manager training that were in a draft of Lean Software Development, but which did not make it to the final version:
Traditional Project Manager Training
Software Development Life Cycle (SDLC) PMI Knowledge Areas Schedule and Cost Estimating Critical Path Analysis, PERT/Gantt Charts Using Project Management Software Project Scope and Change Control Project Tracking and Schedule Control Testing and Quality Assurance Deployment and Support Risk Management Resource Management Contract/Vendor Management Lean Project Manager Training
Seeing Waste Value Stream Mapping Feedback Iterations Synchronization Emergence Options Thinking Last Responsible Moment Set-Based Development Pull Systems Queuing Theory Cost of Delay Self Determination Motivation Leadership Expertise Perceived Integrity Conceptual Integrity Testing Refactoring Contracts On the XP mailing list, there has been some unhappiness at the name "Industrial XP", fearing that it will divide the XP community, and perhaps weaken attempts to "sell" XP into companies.
The IXP web page says "Industrial XP is tuned to handle the needs of large scale, mission critical and enterprise applications" which could be taken to imply that "Classic" XP (I might get some hate-mail for that name, which I didn't make up) hasn't had success in mission critical and enterprise applications (which would be wrong). I think the emphasis I heard at the BayXP presentation, that IXP is turned for "highly political organizations" is actually the correct differentiator between IXP and "Classic" XP, but that doesn't make the best advertising copy.
Ron Jeffries would like the IXP web page to say something like this:
IXP is Extreme Programming.
Extreme Programming, like any good software development method, is always adapted to the context. As a project gets more connections into the enterprise, it needs different practices, and for best results, these need to be consistent both with the enterprise needs and the principles and values of Extreme Programming.
XP and Agile software leaders, including Industrial Logic, have been applying Extreme Programming to larger scale, mission-critical, distributed, and highly-coordinated projects for some time now. We offer here a summary of the approaches and practices that we have used, and that our colleagues have used, in adapting XP to larger-scale situations.
And, Mark Simmonds wants us to know that DSDM 4.2 (not yet released), which blends DSDM and XP, is not the same as EnterpriseXP, which is supposed to be a web-portal to discuss ways to make XP more commercially appealing. Mark also says "One other point I'd like to clarify is that when using DSDM and XP together we do not advocate getting rid of the planning game, far from it. In fact I was delighted to see how closely the Planning Game matched the Timebox planning process I have used in DSDM projects for a number of years." 2003.Apr.27 SunQuotes and Evolutionary Design Practices of Industrial XP
David Schmaltz said on the IXP mailing list: "Change never rests on the permission of the willing, but in the hearts of the brave and foolhardy." Let's hope we have supporting practices to not be too foolhardy. On the XP mailing list, Joshua wrote: "...most people react to change as if they are losing something. It's wired in to our human nature. I introduce XP into environments all the time. People think they're gonna lose something rather than gaining something with XP. I help them learn that they will be gaining a great deal."
On the IXP mailing list, Russ Rufer has provided the list of practices of IXP's Evolutionary Design:
Rapid Return on Investment Risk Reduction Backtracking Selective Automation Team Intelligence Walkthrough Spanning System Small Iterations Multiplicity & Selection Dead Reckoning I snipped quotations from this report http://www.sdmagazine.com/documents/s=7928/sdmsdw3d/sdmsdw3d.html, onto the IXP mailinsg list, and no one contradicted me, so here are some stabs at defining what some of the practices may be:
Rapid Return on Investment - Developing only what needs to be done at the moment, leaving the rest to be filled in later, allowing early releases that can prove themselves quickly.
Risk Reduction - Striving for design simplicity is a factor for reducing risk.
Backtracking - Stepping back to find a simpler solution to a problem. "Backtracking not only helps you to consider other alternatives, it allows you to rewrite, aggressively refactor and prune any dead code."
Selective Automation - "Quantity bows to quality: It's not about writing tests; it's about writing good tests"
Team Intelligence - "Developers should devote maximum attention to improving the code."
Walkthrough - "Studying, living and breathing code is at the heart of evolutionary design"
Spanning System - "Evolving the code from a rudimentary system that, though primitive, provides end-to-end functionality. This simple working application is a thin, vertical slice of the project that offers insight into both essential and unnecessary features. Illustrated with a simple blackjack problem. To span the system, they chose just one case with two known hands and incrementally built the system to accommodate the full deck. "
Small Iterations - "To implement a hotel reservation system, you might first implement a program that reserves just one room before developing the whole system. These small iterations can be viewed as embryonic versions of the system, and can be taken to the customer for feedback...this is the antithesis of RADinstead of throwing your code away, you evolve it."
Multiplicity & Selection - "Consider a multiplicity of design and selection, like the photographer who takes 10 rolls of film to find the perfect shot. Survival of the fittest."
Dead Reckoning - "Navigating without explicit instructions, by heading in roughly the right direction, and using feedback to make adjustments and to motivate backtracking."
2003.Apr.26 SatOn the Agile Modeling mailing list, Ken Hilving wrote in response to a thread on software Architects being labeled as "Ivory Tower" dwellers:
Time, money, knowledge, security, and prestige appears to be the five currencies we all deal in. Understanding this explains why others take an "ivory tower" position when an architect is brought into a project. The architect offers little time or money to the others, has new knowledge that they lack, and threatens their security. They maintain a little prestige within their group by taking the negative attitude.
The trick is using these currencies to provide value to them. Time and money may be outside your control, so deal with the other three. That additional knowledge needs to be freely shared. If the others on the team have a chance to bring their own knowledge into the sharing process, they gain security and prestige at the same time. I believe that the prestige aspect, the opportunity for recognition of their contribution, becomes the most critical factor towards changing the ivory tower label. When it works, the approach becomes theirs instead of yours and they defend it against any challenge.
Maybe this is the key aspect of leadership.
2003.Apr.24 ThuWhat is Industrial Extreme Programmging?
At the BayXP meeting last night, Joshua Kerievsky, Russ Rufer, Somik Raha, and Tracy Bialik of Industrial Logic gave a presentation on their version of XP that they have developed over the last several years. They named it "Industrial Extreme Programming" (IXP). What follows here are taken from my notes. Any errors are my own.
IXP is what Industrial Logic has been doing the past few years as they work with their clients in training and coaching XP projects. Joshua said he was concerned with recent "blendings" of XP and other methods (DSDM, FDD, Scrum, Crystal) because some of those blendings were throwing away XP's planning practices (one of the most valuable aspects of XP). Many of these blendings were for the most part untried and unproven, as well, though the unblended methods have records of success.
IXP doesn't remove any of the core practices of XP (except Metaphor, and few teams have really felt like they successfully used XP's Metaphor practice). IXP builds on XP, adapting it for survival in larger companies, highly political companies, and large teams.
Kent Beck defined four values of Extreme Programming, values he felt were essential... other values were good, but he wanted to emphasis four in particular. XP's values are Communication, Courage, Feedback, and Simplicity. Agile Modeling adopted those four and added Humility.
Joshua and his team have chosen five values, which they not only want to emphasize, but insist that the absence of these values in the project or company will cause failure and unhappiness. The IXP values are: Communication, Simplicity, Learning, Quality, and Enjoyment.
The value of Enjoyment is sometimes deemed controversial. Joshua considered Fun, and probably felt Enjoyment sounded better. People who enjoy their work are more likely to want to learn I've always said that XP requires a Learning Organization). People who enjoy their work and enjoy working together are more likely to have the teamwork that XP requires.
Quality is "we know it when we see it." Quality products, quality code, a quality process, quality people.
These are the original XP practices that IXP includes (more or less), sometimes with modified names and meanings: [names in brackets are the original XP names, or the names I prefer over Kent's names.]
Sustainable Pace Planning Game [Release Planning, Iteration Planning] Frequent Releases [Small Releases] Refactoring [Merciless Refactoring] Story Test-Driven Development [Programmer Tests, Acceptance Tests, TDD] Continuous Integration Pairing [Pair Programming] Collective [Code] Ownership Coding Standard Domain-Driven Design [replaces Metaphor] Evolutionary Design [replaces Simple Design?] The name changes are for clarity and to expand things beyond just coding -- people can pair on other things besides code, collective ownership can extend beyond code.
The new practices are:
Readiness Assessment Viability Assessment Project Community Project Chartering Test-Driven Management Storytelling Storytesting Small Teams Sitting Together Continuous Learning Iterative Usability Retrospectives Readiness Assessment answers the question "Are they able to transition to IXP?" See http://www.industriallogic.com/xp/assessment.html.
Viability Assessment answers the question "Is the project idea viable? Profitable? Feasible? Does the project have the necessary resources?"
Project Community expands on Kent Beck's "Whole Team" concept. "People who are affected by the project and who effect it." (Hope I got that quote right.) This includes QA staff, middle and upper level managers, tech support staff, programmers, DBAs, customers, end-users, and probably marketing and sales. (Reference to David Schmaltz / True North Consulting's Project Community Forum.)
Project Chartering provides the Vision and Mission, as well as the definition of who is in the Project Community. A light-weight exercise that seems to be necessary for clarifying the project's goals.
Test-Driven Management requires objective measures be defined for the success of the project. External results like "support 100 users by December 2003." The Whole Team cooperates to achieve this goal. Also defines return on investment.
Sustainable Pace. They considered renaming this to "Slack" (see the book by Tom DeMarco). An example of the value of slack is that it can provide the time for someone to write the tool needed to increase development speed -- too much focus on getting stories implemented quickly can be sub-optimal.
Storytelling. I think Joshua separated this out from Planning Game in order to emphasize that story-telling is a natural way to get requirements (sometimes after a bit of coaxing). IXP stories are not necessarily "user-centered" stories, since they may address concerns of administrators, maintainers, etc. "A story is an excuse to have a conversation." Conversation is required to understand some stories -- a story that can't be understood can't be implemented. Five words for a story title was also mentioned.
Storytesting. One word, to parallel Storytelling. This is defining the acceptance tests, but not writing them. IXP coaches help their clients in both Storytelling and Storytesting. Ideally, you do want "executable documentation" and they talked up Fit by Ward Cunningham - a framework that allows anyone using any editor capable of creating HTML tables to be able to specify acceptance tests. (Programmer help is still required to plug an application into Fit's acceptance test framework.)
Planning Game. Joshua says that it is very weird that some of the hybrid methods are throwing away the planning game. This practice is so useful that many of Industrial Logic's clients, who did not adopt all of XP, did adopt the Planning Game. Still, the concept of "velocity" (work done per iteration) seems to elude some clients
Frequent Releases - frequent end-user releases -- same as XP's practice. Enables rapid return on investment. Releasing to end-users provides opportunity for feedback, to find issues in deployment, issues raised by real live users. "Without learning, feedback does no good".
Small Teams -- for large projects, set up networks of small teams, with their own code-bases and coding rooms. A 30-person project might consist of teams as large as ten people and as small as three. Sometimes there might be a testing team and/or refactoring team that join the each of other teams at various times and then move on. Industrial Logic practices Pair Coaching, which does not require that both coaches be together at all times. Pair Coaching does enable coaching larger projects than a single coach could cope with.
Sitting Together -- Joshua says that the term "Open Workspace" turns some people off, but it is the same concept. He has seen a 40-person XP team in one very large room, but that's unusual. He has also seen one or more people give up the office they worked hard to get, because pairing in the same room as other people let them focus better and learn more. Sitting together / pair-programming can be done via internet collaboration, so it isn't limited to open workspaces. The gave an example of a team split in two time-zones, who decided to synchronize their hours to allow more "virtual pairing".
Continuous Learning. I've always said that XP requires a Learning Organization, and this practice make it explicit. Examples... Study groups who are not just allowed, but encouraged to get together for three hours a week, during office hours, because they know this helps them advance in their careers. XP Coaches who assign practice drills to the programmers or QA testers. "Lunch Break" learning groups show that management doesn't care enough about their employees learning. An XP coach in Italy spends an hour a day teaching his junior programmers -- whose skills are rapidly advancing. I think an member of the audience said "If everybody isn't learning, then learning becomes a subversive activity." Joshua also said that "resume-driven-design" tends to happen because programmers are starving to learn, but not given opportunities to do so.
Iterative Usability. The UI must be usable and tested regularly. Management-Tests should be tied into Iterative Usability. Redesign the UI as soon as feedback shows its flaws. Paper-based GUI design was also mentioned.
Time was running out, so the remaining practices were discussed quickly...
Evolutionary Design. Drives all design. Their tutorial has ten practices for this. (http://www.sdmagazine.com/documents/s=7928/sdmsdw3d/sdmsdw3d.html.)
Coding Standard. Have one.
Pairing. As per XP, but not just programmers.
Collective Ownership. As per XP, supported by tests, pairing, etc.
Retrospectives are a critical practice. Some clients are reluctant to get 40 people together for 2 or 3 days for a full project retrospect, but they should do it for the unexpected learnings that come from it. Also do mini-retrospectives each iteration.
Refactoring. Early and often as per XP. Don't let "refactoring debt" accumulate.
Domain Driven Design. Even though never officially a part of XP, it has been done by every good XP programmer that Joshua knows. The Model objects are kept separate from the rest of the code (GUI, etc.) The acceptance tests normally operate on the model objects, skipping the GUI. See the book on this subject at http://domainlanguage.com/. See also Erik Evan's "Ubiquitous Language".
Story-Test-Driven-Development. First write a failing acceptance tests. Then use the TDD cycle (failing programmer test, code to make programmer test pass, refactor) until the acceptance test passes. This is "top-down" TDD, and it best avoids writing unnecessary code.
Continuous Integration. As per XP.
See http://www.industrialxp.org/ for more information. Check out these papers, too: http://industriallogic.com/papers/index.html
2003.Apr.23 WedDale writes in "Dale Emery, Bureaucrat" that his department was being changed from serving others to ruling others.
I think this increase in command and control is a recent trend in the industry, a fear reaction to the current economic climate. But remember: "The more you tighten your grasp, the more star systems will slip through your fingers." -- Princess Leia, Star Wars.
Hmm. I suppose a quote from a fictional character isn't the most effective. How about this: "If you are distressed by anything external, the pain is not due to the thing itself but to your own estimate of it; and this you have the power to revoke at any moment." -- Marcus Aelius Aurelius (121-180 AD), Roman emperor. And this: "An intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction" -- Hoshang N. Akhtar
I'm going to have to read more about Deming. These are his "14 points":
1. Create constancy toward improvement 2. Adopt new philosophy 3. Cease dependence on inspection for quality 4. Minimize cost 5. Improve constantly 6. Institute job training 7. Institute leadership 8. Drive out fear 9. Break down barriers 10. Eliminate slogans 11. Eliminate management by objective 12. Right to pride of workmanship 13. Institute self improvement program 14. Accomplish transformation 2003.Apr.20 SunUpdate on Lean's "Do it Right the First Time"
One of the practices of Lean Manufacturing listed here http://www.poppendieck.com/lean.htm is "Do it Right the First Time".
When questioned about the appropriateness of that to software development. Mary Poppendieck wrote on the Lean Development mailing list:
I regret now the application of "Do It Right the First Time: from manufacturing to programming. I generally use the phrase "Test First" or "Build Integrity In".
"Do It Right the First Time" has been widely misinterpreted and I dont think it is a good way to look at a design process.
The impact of "Do It Right the First Time" in manufacturing is that every time a design is delivered to the plant to manufacture, it is delivered with a complete set of tests so that the manufacturing process can be checked every step of the process. Designs of manufactured products are changed repeatedly, even after a product goes into production. With every change, the tests are changed also.
The fact that we do not consider a test suite to be a part of a delivered software product is where we have a lot to learn from manufacturing.
Tom Poppendieck adds more on the "test-first" used by Toyota and in Lean software development:
Toyota's design process uses set-based design which employs check sheets to communicate gradually narrowing design options. These check sheets are equivalent to tests. Set based design basically suggests trying several ways of solving a problem in parallel to learn enough to make a fact based decision rather than speculative up-front decisions if you permit yourself only one shot.
The tests are the key mechanism for keeping options open. Without comprehensive tests, refactoring [software] is far too risky and rework would be expensive. The tests are a record of what you have learned their automation and very frequent execution guarantee that you will not accidentally forget what you have learned. Thats why the first step in fixing a bug is to write a test that fails as long as the bug is present.
To my mind, rework does not refer to recoding or recompiling. Code is pretty easy to generate via TDD. It is the learning that is expensive and forgetting and relearning is to be avoided.
I ... argue that an application that has been aggressively refactored throughout its evolution will have much more flexibility than one that is architected once....
Mary and Tom Poppendieck on Lean Development
Peter Lindberg's Irrational Software blog has book reviews on the book Lean Development by Mary and Tom Poppendieck. It's available for pre-order on amazon.com.
A draft of the book is also on-line at http://www.poppendieck.com/ld.htm. I have pre-ordered the book, but haven't read the on-line version yet. I'll be quoting Peter Lindberg in what follows.
They cite a study by Raymonde Guindon, which "found that cycling between high level design and detailed solution was typical of good designers when dealing with ill-structured problems, that is, problems which do not have a single right answer or a best way to arrive at a solution...."
That sounds like what we do with Test Driven Development (TDD), doesn't it? Write a small test (define some part of the problem); write the simplest code to pass the test; when the test is passing, if refactoring could improve the design of the code, do it; repeat.
The Poppendiecks also quote Ed Yourdon as saying that any piece of code needs to be rewritten three or four times to be consistuted as an elegant, professional piece of work. For prose, we take this for granted, he said, why not for code?
Again, we do this in TDD... as more and more tests are written, the code is made more and more functional, with frequent refactoring to keep it clean. The difference between rewriting and refactoring is that rewriting isn't constrained to keep the same functionality.
Aside: Dale Emory, Michel Tsai, and I, recommend books by Joseph Williams whose titles start with "Style". Peter Lindberg reports that Dale called it "Refactoring for writers" and I have said that it is the perfect writing book for programmers. Check it out: Style: Ten Lessons in Clarity and Grace and Style: Toward Clarity and Grace.
Peter also makes a comparison to XP: Software development projects are often managed as push systems, where plans and schedules drive the process. Extreme Programming projects, could perhaps be said to be more akin to pull systems, [pulled by demand, like the Kanban systems described by the Poppendiecks] where features are represented by cards, preferably kept visible on a wall divided into three sections representing what's currently worked on, what's finished, and what still remains to be done. Testers, for instance, would pick cards from the section with completed features, and managers and customers could tell from looking at the wall how things are going - instead of calling to a meeting to update the schedule.
Peter describes the Kanban system as "a Kanban card is left on the shelf as a part is taken from inventory. The cards identify the parts that were taken, and these are collected and replaced by newly manufactured parts. A part might consist of subparts, for which Kanban cards are left, thus causing demand for new parts to ripple across the system."
Isn't that a delightfully low-tech system for updating inventories? Who says you have to have computers to do everything?
2003.Apr.19 SatMore on Reset, Encapsulation, Value and Immutable Objects
(Sorry for the repost of yesterday's post -- I just had to correct a missing parenthesis, and this is what happens when you do that.)A reader suggests that I could use reference counted smart pointers to avoid problems I described in my previous essay.
That would not fix the problem of violating encapsulation -- retaining one object's member data in multiple other objects. In fact, in this application, if we were using boost::shared_ptr or our own reference counted smart pointer, and did the 'delete'/'new' approach, the result would be multiple "platform independent document objects" in a program designed to have only one document object. The various distinct views of the document would get out of synch. (I do use boost::shared_ptr in my application, to enable passing around large image objects among image processing functions as if they were Value objects -- I don't have to worry about premature deallocation.)
The same reader suggests that a Reset method isn't that bad... He writes "Functional requirements for cleaning self up logically belong in the object, not in delete/new."
I would say that in C++, the requirement for an object cleaning self up belongs in the destructor, by definition of "destructor". Whether the coder does the same cleaning up in Reset is up to the coder.
Probably the real reason for my dislike of Reset is that some coders using it seem to have confused "variables" with "objects". You reset a variable. You create and delete objects. In the application I was talking about, the object has effectively become a global variable, with all the problems that globals have, even though only member variables are being used.
In some ways it is even worse, because these variables are actually pointers to a global object: those pointers can become dangling pointers if the object is deleted by what is supposed to be its sole owner. Using Reset hides the fact that this is a global... better to make it a real global variable, to avoid the dangling pointer problem, or not pass it around at all (which is what LoD recommends). The application I was describing is a single-document application, so the MFC document object is effectively a global variable/global object.
My other point about Reset is that Value objects don't need it, and Immutable objects can't have it.
Imagine a Dimension object. In Java or Smalltalk, you might want to make it immutable, so you can safely return Dimension member values without making copies. This assumes that you don't do lots of math on Dimension objects -- because that would require making copies. It is a choice of which is more efficient, and/or safer.
In C++, I would implement Dimension as a Value object - one that implements the copy constructor and the assignment operator (and default constructor for STL compatibility). Returning this kind of object "by value" automatically makes a copy. If you want to reset a Dimension variable, just assign Dimension(0,0) to that variable. You minimize the number of methods to write, and you make it very clear what you're doing.
Law of Demeter and Encapsulation
The Law of Demeter (LoD) is a heuristic for good object encapsulation. Ignore the Law of Demeter (and other advice on encapsulation), and you'll find yourself in a debugging and refactoring hell. Applied too strictly, LoD forbids container objects and external iterators (there may be a loop-hole for that, which I'll get back to later).
Yesterday, I wanted to eliminate a "reset" function in one object, and instead have its owning object delete and re-create that object. That should be straightforward, but it didn't work because encapsulation and the Law of Demeter were being violated.
The problem here was that a pointer to member variable of this "reset-able" object was being passed around and retained by other objects, whose lifetimes are longer-lived than that member variable would be if I implemented the delete and re-create strategy.
I've trained myself to never give out a member variable's address, just like a TV network never gives out a TV star's address. It's dangerous. It causes the program to be more brittle. It violates encapsulation. You never know what those obsessive fans or programmers might do. Other objects become too dependent on the internal state of another object -- changing the internals of an object becomes difficult. Polymorphism is overly restricted because any "replacement" class must also give out a member variable's address, and that variable's type must be compatible. As I said, I trained myself, but sometimes I forget, and I'm working with other people who sometimes also forget about this danger.
In C++, an alternative to returning a pointer to a member object is returning a reference to a member object, but that turns out to be just as bad. I'll illustrate with some code.
Pointer* ptr = obj.GetMemberAddress(); delete obj; // ptr is now "dangling" - pointing to deleted memory. Reference& ref = obj.GetMemberReference(); delete obj; // ref is now "dangling" - it also points to deleted memory.No one would write code that obviously bad, but put in an event-loop, lots of intervening functions and other objects, and maybe some threading, and the same thing can happen without anyone realizing it until the crash occurs.
You could make a rule to never retain such a pointer or reference... that is an improvement (as long as no one breaks the rule), but it is awkward. It also requires that you never pass the pointer or reference into functions. It's too easy forget where the pointer came from, and create a persistent object holding that pointer. And that dangerous "persistence" include multi-threading as well as objects - a thread's lifetime is even less predictable than an object's, and it becomes much harder to diagnose dangling pointer problems in multi-threading programs. Unfortunately, I sometimes want "values" returned by my member functions to be polymorphic. In C++ that requires a pointer (which brings in memory management issues) or a "smart-pointer" (which I'm using more of lately, but is a bit cumbersome.)
So LoD forbids this:
memberPtr = obj.GetMemberAddress(); memberPtr->DoSomething(); // potentially changing obj's member state.It also forbids this:
obj.GetMemberAddress()->DoSomething();What you should do, is either return a copy of the member, and the copy's lifetime is no longer under the control of "obj", or incorporate DoSomething() into the API of "obj".
So we can write:
memberValue = obj.GetMemberValue(); // returns copy memberValue.DoSomething(); // doesn't affect obj's original member state.We can even write:
obj.GetMemberValue().DoSomething();because DoSomething is operating on a copy. NOTE: if you write this sequence of calls more than once, XP requires that you remove this duplication, most likely by incorporating DoSomething into the API of "obj".
Returning a copy is particularly useful for 'basic' types like String, Date, and so on. The safe programmer will return a copy of a string or date member variable, so that callers can not change the state of the member variable "behind the owner's back'".
Some of the more rabid fans of the Law of Demeter say that even this operating on a copy is too fragile, and you really should do this:
obj.DoSomething();The danger of over-applying this idea is that your object interfaces get really fat. You really don't want to re-implement all of the member functions of String for each of the String members in your EmployeeData class just because you think LoD tells you to. Because of this, I think of the "Law" as more of a "Recommendation".
I assert that immutable objects are an exception to the Law of Demeter.
Java's String class is immutable (once the object is created, it can't be changed), so Java programmers don't have to make copies of String member variables in their accessor functions.
Some people have recommended declaring mutable and immutable interfaces, declaring the mutable object to implement both of those interfaces, and declaring this "accessor" function's return type be just the immutable interface, so that you can return a mutable object through that immutable interface. Of course, a programmer could "down-cast" back to the mutable type, but then all sorts of bad things can be done if you work at it. Probably better to create a copy of the object to avoid the down-casting trick. And, in languages like Smalltalk and Python, you don't have variable and function type declarations to make this immutable interface idea work (though you could create and return an Immutable Adapter to enclose your mutable member).
And what about that loophole for containers and external iterators?
LoD says you can't return references or pointer to your object's own member data, but containers are given data to hold, and so they can return that data.
External Iterators are new objects, not member objects, created when you call the a function that returns the iterator.
So what am I going to do about my hard-to-modify program?
Well, changing it to conform to LoD is going to be at least a day's worth of work. And we violated LoD on purpose, though now I regret that decision. We have a MFC Document object that owns a platform-independent "document" object. We pass the MFC Document object to other MFC classes, and pass the platform-independent "document" object around to the rest of our code. But we're not consistent about that.
To make this conform to LoD, the platform-independent object must never be passed around to other objects at all -- everywhere we currently do this, we should be passing around that MFC Document object instead. That means that the complete interface of platform-independent "document" object must be implemented in the MFC Document object, delegating to the member object. However, the "type" we pass into various part of our program doesn't have to always be the MFC Document type, it can be a base-class type -- the platform-independent "document" interface -- we just have to declare the MFC Document type to subclass from that interface.
Then, and only then, could I have the MFC Document object have full control over the lifetime of its member objects (and even then, I have to be careful about threads - I can't delete an object if another thread is still using it.)
And why would I want to eliminate a "reset" member function and instead delete/re-create the object? Because it's too easy for a reset member function to forget to clean up all of its state. I object to "reset" functions generally, for both small objects and large objects like documents. For small objects, I prefer immutable objects that I can easily recreate on demand, because then copies don't have to made in accessor methods, and immutable objects are more easily made multi-thread safe.
For future thinking: how can LoD work with threads? Can we think of the thread as an object? Some platforms do.
Here's a "formal" version of LoD: A method "M" of an object "O" should invoke only the the methods of the following kinds of objects:
itself ("this" in Java and C++) its direct member objects (this.member in Java, this->member in C++) its parameters any objects "M" or "O" creates/instantiates [my extension] any objects created but not retained by a member or parameter method Here's the informal version attributed to Peter Van Rooijen:
You can play with yourself. You can play with your own toys (but you can't take them apart). You can play with toys that were given to you. And you can play with toys you've made yourself. See http://c2.com/cgi/wiki?LawOfDemeter for more discussion on this topic.
See http://www.ccs.neu.edu/home/lieber/LoD.html for a list of LoD links.
2003.Apr.16 WedMore Defects from MultiTasking?
Ron Jeffries posted his email conversation with Jon Eaves on XP mailing list in "Bug tracking vs user stories"
[ jon ]
"Projects that have developers working on multiple simultaneous projects have a 30% higher defect rate than projects with developers assigned specifically to it"[ ron ]
Fascinating if true.[ jon ]
2003.Apr.15 Tue
I don't remember the actual number, this is again from a previous life, but there was a higher defect incidence rate. We stopped this practice (or at least attempted to minimise it) and life was better for everybody.Michael Feathers has a start on a book currently titled Working Effectively with Legacy Code, the mailing list to discuss the book is http://groups.yahoo.com/group/welc/ and drafts of the book are in that yahoo group's file section.
David J. Anderson has a yahoo message group at http://groups.yahoo.com/group/agilemanagement/ to discuss his book (almost final now), the book is in pdf form here: http://www.uidesign.net/
Here's the message group's description
Agile Management is a group to discuss the content and thesis of the forthcoming book, "Agile Management for Software Development: applying the Theory of Constraints for Business Results", by David J. Anderson, to be published by Prentice Hall in summer 2003.
Agile Management uses a combination of Eli Goldratt's Theory of Constraints, its management accounting method, Throughput Accounting, Lean Thinking and Systems Thinking ("The Fifth Discipline") to demonstrate how agile software development methods can be controlled, managed, cost justified and summarized for executive reporting.
The book postulates 4 agile management roles: development manager; program manager; project manager; and product manager. It then provides definitions for the areas of responsibility and describes the work of each manager. The topics of Agile Development Management, Agile Product Management, Agile Project Management and Software Resource Planning (the job of the Agile Program Manager) are explained in detail.
Discussion is invited on this list regarding the application of Theory of Constraints, Lean Thinking, Learning Organizations and Failure Tolerant Management, to the field of software development.
2003.Apr.14 MonOne of the best books on Extreme Programming is Planning Extreme Programming by Kent Beck and Martin Fowler. It doesn't tell the customer how to gather requirements, but it gives lots of advice on how to do XP from the customer side... particularly Release Planning and Iteration Planning, and why, of four variables (Time, Scope, Resources, and Quality), he suggests only letting Scope vary.
Kent Beck's prose is highly factored -- he tends to say things once and only once -- so you have to read carefully. He doesn't explicitly define what a "release" is, but there are clues that he really does mean releasing to the end-user... when he talks about releasing to internal testing, or to the Customer (standing in for the end-user), he uses the phrase "interim, internal release". Here are few quotes from the book about how often to release:Often the dates for a project come from outside the company:
* The date on the contract.
* COMDEX
* When the vc money runs out.Even if the date is of the next release is internally generated, it will be set for business reasons. You want to release often to stay ahead of your competition, but if you release too often, you won't ever had enough new functionality to merit a press release, a new round of sales calls, or champagne for the programmers. (page 40)
Short Releases
Sometimes you can release much more often, maybe every iteration[....] However there is danger to never having "a release". The customer may lose strategic vision of where the software needs to go. (page 79)
Long Releases
What happens if you can only release once a year? [...] Another case is shrink-wrap software [...] look for a way to send intermediate releases to those customers that may be more interested in these versions. Call them service packs or something[....] Frequent releases are good, but if you can't release frequently you don't have to abandon XP completely. You may need to create interim releases that are only available internally. (page 80) [Bold emphasis is mine.]
2003.Apr.13 SunProposal: an Information Radiator for Feature Requests
Consider tracking feature requests using a cork board... (one cork board per product) each time a feature request comes in, compare it to what's already on the cork board. If it's not there, find a place on the cork board for it, write it on a sticky note or index card, and post it. If it's already there, add a hashmark to indicate multiple requests for that feature.
At some point, start grouping the feature requests that are on the cork board... this set of 9 feature requests can be grouped under "scriptability", this set can be grouped under "import/export", this group can be grouped under XX or YY.
Imagine this cork board visible to everyone at the company... other people could add cards to the cork board, because they see a missing area. The VP could add a comment to the XX grouping, saying that these features are not valuable to the more profitable customer base they want to target, only those laggards that haven't upgraded to the latest product version. Customer Support representatives could add to the cork board those off-hand comments said by end-users that they didn't think important enough to file as a formal feature requests.
Imagine being able to see patterns and order emerge out of the chaos of individual feature requests... someone sees this and says "I have an idea for a new product line!"
2003.Apr.12 SatYesterday afternoon, the Bay Area weather was so nice that I dragged myself to work with some effort. I had previously signed up and worked on a story that I knew I could finish off by the end of the day, and I didn't want to not finish it and loose the three story points from my team's velocity. So I do the hard thing and get the work done. And today, Saturday, it is raining. Bleh.
I'm sure that the weekly traffic patterns affect the weather -- all that automobile pollution keeps the area warmer than normal, and then on Saturday the cold and rain comes in because there's little traffic. Or something like that. I've read that California gets some of the pollution generated by China, so maybe Beijing's weekly traffic/pollution patterns are at fault.
Another bit on agile requirements... XP doesn't say how to gather requirements. RUP does. I attended one of the presentations by Dan Rawsthorne of Net Objectives, a presentation called Comparing RUP, XP, and Scrum: Mixing a Process Cocktail for Your Team (Check that link for their slides in PDF form). Dan recommends combining RUP's Use Case creation techniques, with Net Objective's own "Ever Unfolding Story" technique, to create stories for XP's release plan/iteration plan. He says that a typical Use Case can create up to forty XP stories. Maybe ten of the stories are the "core" of the Use Case and can be delivered in one or two iterations. Another ten may be important for the final release but of lower priority. The other twenty stories may not need to be implemented at all -- but a typical heavyweight RUP project probably would have implemented them. If you have question about this or the "Ever Unfolding Story", please talk to Dan at www.netobjectives.com... I'm just reporting here.
I do want Dan to correct one of the slides... the one about XP and the "business levels". The slide about RUP shows RUP "touching" the business level at "kickoff", "delivery 1.0", "delivery 2.0", and so on... Dan asserted that XP doesn't "touch" the business level, and drew the slide on page 22 of the PDF file that way. That's WRONG. XP's "Small Releases" correspond exactly to the RUP process slide's "delivery 1.0" and "delivery 2.0". Those releases every three or six months are at the business level, to get feedback from actual users.
I also attended a one-day class on Project Management recently. The simulated project was waterfall, but the word "waterfall" was never mentioned. There was no mention of the possibilities for iteration, incremental development, or even feedback. I don't think any of those terms were in the vocabulary taught as part of class. The instructor had never even heard of Rational Unified Process ("Rational Who?"), much less Extreme Programming, Scrum, Feature Driven Development, etc. That's scary.
So what did I learn in this project management class? Well, the project process has Initiating, Planning, Executing, Controlling, and Closing. No mention of maintenance. Not much about people issues.
Planning included the "Work Breakdown Structure" (a rather unfortunately overly task-oriented breakdown of features that does have some resemblance to XP's "Release Plan", sorta.) We saw a "Responsibility Matrix" correlating tasks with "do, review, and approve" columns. We learned about finding the Critical Path on a Gantt Chart. A little bit of Risk Management and mitigation planning. And of course, "resource" allocation and management. [I'm having a vision ala Solyent Green: "Resources are people!"]
The "trade-off triangle" of Time, Cost, and Scope/Requirements was mentioned. No mention of the zeroth law of software engineering: "If you do not care about quality, you can meet any other requirement."
We learned a lot of vocabulary, with emphasis on "crashing" (adding resources to decrease the project schedule -- hmm -- like getting nine women to have have a baby in one month? No mention of Brook's law.) and "fast-tracking" (defined as "compressing the schedule by overlapping activities normally performed in sequence").
My hand-out from this one-day class says "the PMBOK is the standard knowledge source for project management," and lists a bunch of books. Some of the books I've heard of -- Critical Chain by Goldratt, some of them I've read -- The Deadline by DeMarco (I wish he could go back and write an "agile" second edition or sequel), some of them sound good Customer-Driven Project Management by Barkley & Saylor, and some of them are just scary -- The Complete Idiot's Guide to Project Management.
Bryan Dollery's article The Buck Stops Here and Scott Ambler's Agile Requirements: The Real Message are two responses to an article by someone who'd never experienced an agile project, who wrote a rejection of the agile approach based on misunderstanding how it works.
My web-log entry Incremental Requirements was also a response to this article.
2003.Apr.08 TueThis is a dialog I created in July 2002, from messages on the Test Driven Development mailing list -- an email conversation between Robert Blum and Roger Lipscombe. After my editing, they didn't recognize their words, so perhaps I did a little more writing than editing. Any similarity to my favorite comics characters is purely coincidence.
Calvin: I don't do test driven programming. I write some code, and test it manually, walking through it in the debugger, checking that it works.
Hobbes: How do you know that it works? Could you express that as a test before you run your code?
Calvin: I can't think about the test until I've thought about the code
Hobbes: Changing the habit of writing code before the test can be a bother. Try this: Write the code on paper. Then write a test.
Calvin: Paper?!
Hobbes: What if you just outlined the code on a whiteboard and didn't write any real code? Could you still write a test first?
Calvin: I'd feel silly doing that alone.
Hobbes: What if you explained it to your pair partner? Could you still write a test first, or maybe have your pair partner write the test?
Calvin: That could work.
Hobbes: I notice you're refactoring without tests. What's that like?
Calvin: After I refactor, I have to test it again manually, in case I make a mistake doing the refactoring.
Hobbes: Wouldn't it be easier to run some quick automated tests before and after you refactor?
Calvin: Well duh! If I had the tests.
Hobbes: If you do test-driven development, your tests drive the design of your code, and as a side-benefit, you get a nice small set of automated tests that make refactoring safe.
Calvin: Perhaps if I start by thinking about the interfaces that my class or module would need to support, I would be able to write the unit test first? Does this sound like the kind of thing I should be trying more of?
Hobbes: I think so. If I were given just the task of writing a class or module, my first test would describe a very simple case of what I wanted from that code. The interface of this not-yet-written code is evolving, of course, during my test-driven development process. Since I'm calling the functions that I need at each moment, without a care how (or if) they are implemented, it tends to create a pretty useable interface.
Calvin: So you find out quickly if your interface functions are no good when you're forced to use them as client, in the tests, before investing a lot of work at writing them!
Hobbes: Now you're getting it.
Calvin: Tell me, when you think of a feature, is your next thought about testing or implementation?
Hobbes: I just think about using that feature. Which leads me to an example of how I would use it. Now I just write down the results I expect from that, and voila, there's the test. I'm not even actively thinking 'I have to write a test now. How can I test it?'. My main focus is how I can use it.
J. B. Rainsberger added a few more lines:
Hobbes: Have you written tests for that code yet?
Calvin: Tests?! TESTS?! I don't need tests! What do I need tests for? Don't you think I can write something this simple without using tests as a crutch?! I know it's right!
Hobbes: (rolling his eyes) Of course you do.
Calvin: Right. Now help me fix these last four bugs.
And Laurent Bossavit capped it off:
Hobbes: I thought I was doing that.
2003.Apr.07 MonA writing collaboration is taking place in the Agile Modeling mailing list. Scott Ambler posted a draft of a paper he's writing, about the rights and responsibilities of stakeholders and developers.
Scott: So, I'd appreciate any feedback that you might have. Thanks in advance.
Ron Jeffries: Here it is. It was wise to thank me in advance, because in a few minutes you may no longer feel that way.
Anne & Larry Brunelle's idea of no longer separating developers from the other stakeholders has resulted in a new paradigm. Ron's combined-rights draft has phrases like:
You have the right and responsibility to show and to observe progress in a running system, proven to work by repeatable tests specified by customers and programmers alike.
You have the right and responsibility to know what is needed, with clear declarations of priority. You have the right and responsibility to contribute as much as you can to determining needs and priorities.
You have the right and responsibility to ask for and receive help from everyone on the project, peers, superiors, subordinates, programmers, or customers. You have the responsibility to give help when it is asked for. 2003.Apr.05 SatThe great thing about agile projects, is that you don't have to have ALL the requirements up-front, just enough to get started. Once you see the software working (minimally), you can change your mind about the requirements that are not yet implemented or have been implemented.
Consider a non-XP project, one that is not incrementally implementing features. If the customer decides in the middle of the project to change some requirements, the project may have to throw away lots of partially-completed code.
In the middle of an XP project, a requirement is either implemented or not -- the only time a requirement may be partially implemented is during the two-week iteration it was schedule for. Any requirements that haven't been implemented yet can be changed at zero cost. A change for requirements that have already been implemented is essentially the same as adding a new requirement -- it will cost something to be re-implement, and an equivalent amount of other non-implemented requirements should be dropped to keep the project on schedule.
Developers are the genie in the magic lamp - delivering any features that the customer wishes - so long as the customer is willing to pay the price (time and money). With this power comes rights and responsibilities.
In Extreme Programming, rights and responsibilities are divided between "business" and "developers". The "business" side has the business analysts, the QA testers, domain experts, users, stakeholders, and so on. All wrapped up in the single word "Customer". They are all involved in defining, specifying, and creating tests for the requirements. If you have business analysts in your team, it is the job of business analysts to help the stakeholder come up with requirements.
The "developer" side has the programmers, DBA, and so on. They translate requirements into code, databases, web servers, and so on. User interface experts may be either side, depending on whether they do programming as well as user interface design.
The business side is responsible for the requirements, the wishes they make of the genie, because they are responsible for running their business. To avoid this responsibility is to let the developers run your business. Unless the business is making tools for other software developers, it's unlikely that the developers have the expertise to satisfy your customers and make a profit.
The responsibilities of the two sides are clearly divided to reduce risks: if the business guys (who are not programmers) start telling the developers how to do programming, they risk technical failure. If the programmers tell the business guys what features should be implemented, there is a risk that project will not meet business needs.
When talking about what to ask the genie in the magic lamp, I've never seen anyone with a lack of ideas. They may not know HOW to get what they want, but they always know enough of what they want to start a conversation. Like the following:
"Wouldn't it be great if we could do x?" "Could you make it so that y happens when I do z?" "I want this..." (drawing on whiteboard). "Our customers are doing this manually, and we want to sell them software that does it for them." "This is what I do all day, I want it simpler and faster." "Our competitor has product x with feature y, we need something like that." In XP a requirement is expressed as one or more stories. I've heard that one UML Use Case, RUP style, can translate into 40 XP stories, of which 20 stories could be optional. An XP story is a promise for future conversation about the details of that requirement. You don't have to think them up all at once, you don't have to have all the details up-front, you don't have to write up a thick document. You do talk with the developers, other stakeholders, and business analysts if you have them.
Recommended reading:
Exploring Requirements: Quality Before Design by Gause and Weinberg
Planning Extreme Programming by Beck and Fowler
I don't like hearing people refer to managing programmers as "herding cats". Probably because I am a programmer and I have cats (at one time, living in a really nice house in a suburb in Texas, my wife and I had five cats).
It is easy to lead cats... in fact, when I'm trying to take a picture of one, it's hard to keep him from following me around. Trying to make cats go somewhere, without leading them, can be very hard, if you don't consider what motivates them.
So when I hear someone comparing programmer management as "herding cats", it makes me think that they're trying to make them go somewhere, not leading them.
A particularly good book on project leadership is Powerful Project Leadership by Wayne Strider. (Order direct or via amazon). It's good enough that I just bought a second copy for lending to others. This book's first lessons in leadership are to help you become aware about yourself, others, and your shared context. In fact, the book is divided into three divisions: leading yourself, leading others, shaping your project's context.
Check out a sample of Wayne's writing here: Leading Projects in Stressful and Chaotic Situations.
2003.Apr.03 ThuTraining? We don't need no stinking training!
A manager told me the other day that his annual budget for training was $800. For him and the four or five people under him.
With most classes and conferences costing over $1200, there's no way that he or anyone working for him will get ANY training this year.
What's up with that?
2003.Apr.02 WedTWELVE|71 wrote a bit about reuse and modularity. "Modularity is the idea of having a tire that can be swapped on and off without affecting the car. Reuse is the idea that we can take a tire from an old car and use it without thought on a new car."
Off the top of my head, I think software reuse projects often fail because (1) the needs of all the client projects are not considered when making the 'reusable' module, (2) the module is not documented well enough, (3) people on the client projects are motivated to not reuse modules, perhaps because they are rewarded for hours worked or lines of code written (or some other counter-productive measure), or (4) they are under time pressure, and don't have time to rewrite their app to use a module that they seem to be able to do without.
Take the reverse approach. Instead of creating a module and telling projects to use it, create a small "Extreme Reuse" team: one to four people who join projects to (1) help them get things done (2) look for code that could be extracted for use by other teams (3) refactor project code to create and reuse shared parts. The Extreme Reuse team needs to join several projects before extracting code from any of them, in order to know all of their needs. How can a few people join a team and be immediately helpful and productive? Pair programming.
Bryan Dollery has described this and other aspects of how to reuse parts here.
I don't know anyone trying a "Reuse Team"; please let me know if you are. My small team works on several projects. We have code common to these projects in some separate directories, and it is mostly unit-tested. (Yes, we're not 100% pure XP.) We make minor changes to this common code, and avoid breaking the clients by continuing to pass the unit tests, and by building and testing the client projects.
Top 100 April Fool Hoaxes Hoax number eight reveals the source of story about a state legislature passing a law to make pi = three.
How to write Unmaintainable Code: Roedy Green's version on multiple pages, and another version on a single page.
I used to have April fool issues of Byte magazine that described the black hole diode (useful for reading disks still in the shrinkwrap), the sliderule emulator, and my favorite: the solution to the vexing problem of how many buttons should be on a computer mouse - the 'Megamaus', which had a full ascii keyboard on the back of what appeared to be a large rubber rat.
2003.Apr.01 TueSpeaking of Objects as cooperating, independent, agents...
Chris Uppal, on the Dolphin Smalltalk newsgroup, writes about his "Great Leap Forward from Java to Dolphin Smalltalk." He describes the big difference...
If you are like me, then you are currently thinking of a big difference between Smalltalk and Java being that Java stores code in files, whereas Smalltalk keeps it in the image. That's sort of true, and I'll get back to it, but, for a minute, just forget about code, it's not important (really!). What matters is objects.
The image is the place where the objects live. Technically, the image is a garbage-collected heap that can be saved to file, and later restored, thus saving and resuming the state of a computation. Technically that's true, but it isn't at all a helpful way to think about it. A more organic metaphor works much better. I think of the image as a deep, murky, pond where objects move around in the depths like fish. It's an important part of the metaphor that the objects are independent of me. Even if I designed and wrote the classes, once an object has been created it has an independent existence. I can talk to it, I can ask it to perform operations, but it is separate from me. In a sense it is "my" object, but it is only "mine" in the same way that a pet, or a rose bush, or a table, could be "mine".
The image is where the objects live. Not the code, the objects. We'll get back to the code in due course, but not yet. The Smalltalk environment is just a place where you can talk to objects; no more, no less. Oh, sure its got class browsers, debuggers, editors, etc, but that's all tinsel. What matters is that it is a place where you can interact with the objects.
I'll get back to the "tinsel" later too, but for now, I want to talk about the one part of the environment that isn't just a productivity aid: the workspaces. Workspaces are the medium through which you talk to objects. You can describe workspaces as "containing snippets of code" which you execute, but IMO that's exactly the wrong way to think of it. A better picture (slightly tongue-in-cheek) is as a kind of singles bar, where you can meet objects, talk to them, check them out, get to know them. Each workspace has a number of objects that are (temporarily) living there; they are the values of the variables in the workspace. In most cases they'll die when you close the workspace, but until you do they'll survive and you can talk to them. I keep some workspaces hanging around for days if they contain objects that are important for what I'm doing. The way you "talk" is by sending messages written in the Smalltalk programming language, but that's almost incidental. The important thing is that you are communicating with them using an interactive text-based medium, like using an IRC [chat] channel.
[...]
Another way of interacting with objects is to use the Inspector(s). They give you a much more nuts-and-bolts, low-level, view of the object -- a more intimate view, if you like. I, personally, don't think that the Smalltalk world has yet woken up to what inspectors could be, but the current implementations (like "flipper" in Dolphin) do at least allow you to see inside the objects.
I wish C++ had inspectors... but C++ throws away a lot of information when you compile the code. In the crappy debuggers that C++ programmers have to live with, we often can't view the run-time contents of an object properly and easily. The VC++ debugger for example, should be able to know that the object pointed to by a base-class pointer is actually an instance of a derived class, but it doesn't display the member variables that belong to the derived class -- just the member variables of the base class.
An image will contain many objects, some long lived (living, perhaps, for decades), most very short lived indeed. Some will be simple or trivial, like Strings and Points. Others will have complicated internal structures, and/or complicated behaviour. But they are all objects, and they all live in the image, and you talk to them in workspaces.
A Squeak Smalltalk image contains objects that have been "alive" since 1984 or earlier, because Squeak was derived from a Xerox/Apple implementation of Smalltalk-80. The objects in the image "sleep" when saved to disk, and awaken when restored from disk. This means that some objects in Squeak have been alive longer than some programmers have been.
Classes are one particularly interesting kind of object. Remember I'm still not talking about code (that comes later), I'm talking about the objects called classes. Just like any other objects, you can invite them to join you in a workspace:
[I modified the code slightly here...]
aclass := String.and then you can use the magical Smalltalk object-oriented IRC to talk to them:
aclass name. "--> #String"
aclass allSubclasses size. "--> 3"and so on. So classes are objects, and they live in the image.
[...]
Code is how we tell objects how to behave. It's text in the Smalltalk programming language. We're programmers so we care about code; when we wrote the tools for looking at objects, we naturally designed the tools so that we could also see the associated source code. For instance our special tool for looking at classes (the class hierarchy browser) allows us to see the source of the methods, to change the source and recompile, etc. That's natural for us as programmers. If we weren't programmers then we'd want different tools, and we'd be interested in talking to different objects. Such systems, built for non-programmers, are called "applications", but they are still just Smalltalk -- tools for talking to objects that live in an image. (A big difference is that the "image" of an application is typically not persistent, unlike the image of the IDE).
Back to code. Granted that the most important thing is the objects and how they behave, we still do care about the code. We want to organise it, back it up, put it under source code control, etc. A class is an object that lives in the image, but the source code for that class is something else. For all sorts of reasons, we want to keep that outside the image. The way that Dolphin organises source-code is via Packages. A package is a collection of the source code for classes and methods (and a few other things too, which don't matter here) that is kept outside the image in one or more files. You can load the package into the image, which will create an actual Package object, and class objects corresponding to the source-code. Or you can "uninstall" the package, which really means killing the Package object and the Class objects.
So a package is just a way of collecting related source-code together. [...] The package mechanism is relatively simple; it could be improved, but I find it adequate for my needs. Package files are text files, you can edit them with vi, or notepad, or whatever. Occasionally I do that if I want to make particularly sweeping changes to the source. Of course, if you do that then you have to install the changed version into the image before it'll do anything useful.
Notice how very different this way of thinking is from the way that even the best Java IDEs encourage you to think. When I started out in Smalltalk I was thinking of the IDE as if it was a Java IDE. I though of it as a tool that allowed me to write code, and had features to allow me to browse and test the code. After a year or so I realised that I'd turned the picture upside down completely, and in the process had revised my conception of what Object-Oriented programming is all about. As a Java (or C++) programmer I had pretty much thought my .java (and .cpp) files were the classes, and I thought that creating classes was what programming was about. I now think of the objects as being the important thing, and the classes as very secondary, hardly more than an implementation detail.
I feel that that has made me a better programmer. Of course it's not possible to know for sure, but if it has, then it all comes down to Smalltalk's workspaces...
The original message can be found here
Hal Macomber reminds us that Lean Development not only gets things done on time, but on budget and safer. Check it out
2003.Mar.31 MonChaos, Order, and Software Development
Kevin Kelly published Out Of Control: The New Biology of Machines, Social Systems, and the Economic World in 1994.
Jim Highsmith (James A. Highsmith III), wrote Adaptive Software Development: A Collaborative Approach to Managing Complex Systems in published in 2000, before he read Extreme Programming Explained: Embrace Change by Kent Beck, published in 1999.
The book on Scrum was published in 2001, and Highsmith's Agile Software Development Ecosystems was published in 2002.
What do these books have in common? Order (or the semblance of purpose) emerging from independent agents in situation that one might expected to be purely chaotic.
Kevin Kelly's book covers the most ground, of course, from bee hives and ant colonies, boot-strapping ecosystems, competing/cooperating agents within our brains, distributed control within robots, evolution and genetics, genetic algorithms, and so on. The theme is that order can arise "by itself" (it emerges, rather being designed up-front.)
The agile software authors are saying that a good software product can arise with a minimum of planning up-front. However, creating this order is not 'random', it arises from the constant thinking and re-thinking of the people involved through the life of the project.
Some people have objected to a lack of up-front planning or designing, making the analogy to a "hill-climbing" algorithm that gets stuck on a local maxima (getting stuck on a small hill, when the goal is a larger hill, a valley away). The difference of course, is that the hill-climbing algorithm is stupid, whereas many people are smart. People can see the big picture, and can do a little planning of refactoring to get from the current design to the desired design, even though the refactorings may temporarily go through a poor design along the way (but with all tests still passing!)
In a recent web-search, I came upon a paper written by a member of a group researching independent software agents. The paper was about how Extreme Programming is helping them write software successfully, allowing new programmers to become productive members of the team quickly, and how the process allows them to increase software re-use. That paper is here: Using Extreme Programming for Knowledge Transfer. Using XP to continue research like those described by Kevin Kelly... I don't have my copy of XPExplained or Adaptive Software Development handy, but I would expect Kelly's book, or books he references, to be referenced in the bibliography of Kent's and Highsmith's books. I enjoy feedback loops like that.
2003.Mar.29 SatLaurent Bossavit writes: I regularly hear from people who have experimented with daily meetings such as Scrum ("Daily Scrum") or XP "Stand-up meeting") recommend. With no exceptions, everyone says that such meetings are incredibly effective in getting issues solved quickly, gathering momentum within the team, etc. My experience is about the same, although I prefer brief, informal "huddles" to formal meetings.
With a three person team, I tried daily standup meetings, but didn't find them that useful, since we were co-located anyway. An attempt at doing combined standup meetings with two unrelated teams was even less useful. It seems that having a common goal and related work makes daily meetings productive. That's probably why weekly "status meetings" of unrelated teams reporting to one manager are very unproductive. Like Laurent, I prefer brief huddles whenever information needs to be shared.
2003.Mar.28 FriWatching the TV show Boston Public, Principal Harper was a telling a student something like "when you have a choice, pick the hard choice. Nine times out of ten, it will be the right one." That rings true.
In developing software, writing tests is hard, so XP does it all the time. Design is hard to get right, so XP does it all the time. Communicating requirements is hard, so XP does it all the time (by talking to the person playing the role of Customer.) By doing these things all the time, we make them easy.
In corporations, telling someone the truth can be hard; standing up against peer and management pressure to avoid reality can be hard. Doing the right thing, tactfully, is hard, but more rewarding than living in the hell caused by ignoring reality.
2003.Mar.27 ThuPlease Ignore the Elephant in Your Living Room
The subject of legacy code appears on the Test Driven Development mailing list periodically. Our advice is to use test driven development for writing new code or bug fixes, and leave the rest of the legacy code alone, writing tests for old code only when you need the support for refactoring.
David Brady, on that mailing list, writes: "being behind schedule with 250,000 lines of monolithic, untested, difficult-to-test code is the PERFECT time to start learning how to test. You just have to ease into it one step at a time and be prepared for a long journey." My emphasis on the "one step at a time".
2003.Mar.26 WedTechnical Reviews, More on Test Driven Design
Scott's essay on TDD is up here and Ron Jeffries's critique is there. I'll have more to say about it after reading it a couple of times.
That recent issue of STQE Magazine also has a great short essay by Jerry Weinberg on technical reviews being a learning accelerator. One thing I want to point out is that junior programmers should be reviewing the work of master programmers, not necessarily to find errors, but to learn from the master - and of course, master programmers can make mistakes too, which are often visible to junior programmers as well as other master programmers. If the master programmer is humble, he/she can learn from a junior programmer, too.
I've been reading Weinberg and Freedman's book on the subject of technical reviews Handbook of Walkthroughs, Inspections and Technical Reviews (which was written in FAQ style - question/answer), and was a bit surprised by their recommendation that when people are being trained on how to technical (code) reviews, they should have some practice at conducting a review in the presence of hidden agendas.
Some examples of hidden agendas in code reviews: person A wants to impress person B. Person B wants to make person C look bad. Person C needs to go to the restroom, but doesn't want to say so. Person D is distracted by illness of his/her spouse.
Only Weinberg would write about hidden agendas in code reviews - too many writers and books on software development practices seem to assume that people act like machines.
On Weinberg's SHAPE forum, Charlie Adams wrote: "When people are getting tense about their software being reviewed, use Jerry's phrase, 'Yes, I trust your honesty, but I don't trust your infallibility. I don't trust anyone's infallibility.' (QSM 4: page 220) In my experience this has always calmed the atmosphere and allowed us to examine the code rather than the developer."
While I have done code reviews, both informal and formal, I prefer pair programming. It combines reviews with collaborative design, testing, and coding. Rather than go into all the reasons why pair programming is good, I'll point you to www.pairprogramming.com and Pair Programming Illuminated.
2003.Mar.25 TueTest Driven Development is about Designing, Not Testing
In a recent issue of STQE Magazine, Joel Spolsky wrote that Test Driven Development (TDD) doesn't substitute for "normal" testing. It seems like he doesn't understand that test driven development is about low-level design, not testing. Programmer Tests are a happy (and intentional) side-effect of the design and refactoring process. It is to avoid this misunderstanding that I prefer to call TDD "Test Driven Design".
Ron Jeffries and Scott Ambler had a little spat on the Agile Modeling Mailing List about TDD, not about whether it constitutes "design", but on how much design "up-front" it entails. Scott started it by writing here "An important observation is that both TDD and AMDD [Agile Model-Driven Development] are based on the idea that you should think through your design before you code. With TDD you do so by writing tests whereas with AMDD you do so by creating diagrams or other types of models such as Class Responsibility Collaborator (CRC) cards."
Ron replied "Does TDD suggest that you "think through your design before you code"? I see no such thing in TDD. In TDD we write ONE test, then make it work, then write another." [He's leaving out the refactoring step here, which is another area of design in TDD.]
Maybe Ron doesn't think writing each test is "thinking" or "designing", but I do. At the risk of being snide, I assert that each test represents more thinking than a lot of programmers do when they write code without tests. Perhaps Ron's extensive experience has made his designing unconscious.
When writing the test, you think about the API, the goal of the API, and how to verify the goal is met. That's design. Before you start writing the tests, you think about whether to extend an existing class (and its tests) or to start a new class and new tests. That's higher level design. After you write a test and make it pass, then you look to see if there is duplication or other design smells to be refactored away. Still more design. Perhaps Ron thinks this refactoring step is the only design step in TDD.
Check out Kent Beck's book: Test-Driven Development: By Example for an introduction to TDD. Unfortunately, only a very experienced (zen-master-level) programmer like Kent Beck can take the refactoring step of TDD (remove duplication) and derive all the other good design principles from that. So read Robert Martin's book Agile Software Development: Principles, Patterns, and Practices, which not only uses TDD extensively in its copious examples, but also documents design principles that every programmer should know.
2003.Mar.24 MonWhat a Tangled Web We Weave, When Don't Have Effective Tracking
Steve Norrie points to a recent online article by Jerry Weinberg published on CrossTalk, the Journal of Defense Software Engineering here: Destroying Communication and Control in Software Development.
I'd like to mention a few of the Extreme Programming (XP) solutions to some of the problems mentioned in this article. Be aware that XP is a light-weight process, relying on people rather than technology to do the right thing. (A savvy person can undermine technology anyway.)
Requirements. One of the first areas that communication can be destroyed is by not doing requirements well... not involving the customer, thinking that requirements are a waste of time, and so on. Extreme Programming recommends involving the customer, or a qualified representative of the customer, throughout the entire project. And in addition to talking about the requirements often and in detail, XP requires writing the requirements down in an executable form - automated acceptance tests.
The configuration management system (CMS). The CMS tracks requirements, design, code, test data, test results, user documentation, etc. Jerry notes that information in the CMS can be undermined by failing to keep it up to date, restricting read-access from people who should have access, removing data or failing to put data into it. Extreme Programming doesn't require any specific CMS system (software or otherwise) for this tracking, but does suggest using the simplest thing that actually works. For code, tests, and test data, I recommend using CVS or some other source-code-management system. For acceptance test results, many XP teams record those on a white board or poster board, updated them weekly or daily, charting them over the course of the project. XP does strongly recommend using index cards (story cards) for tracking requirements during initial and weekly planning, and also strongly recommends documenting the relationship between the automatic acceptance tests and the story cards. The person playing the Customer role, as well as the whole team, is responsible for keeping track of the stories. Many XP teams also record the stories in web-accessible ways, such as a wiki.
In regards to a bug-tracking database, some XP teams track bugs the same way as stories - index cards and acceptance tests. You might think that this won't work, but consider that several XP teams have reported that their bug rate after implementing XP dropped from hundreds per six months to around one bug per month. It helps that bugs are not usually recorded until after a story is finished, and in XP, a story is not finished until it is passing its acceptance tests - this requires conversation between the tester and coder as soon as problems are noticed during the implementation and testing of the story. When a bug is recorded, it probably indicates that an acceptance test that was passing has started failing.
Weinberg recommends that you "set and enforce a policy of complete and open information at all times." Agile processes like XP need accurate information daily. Many projects keep tracking information on poster boards and white boards, visible not only to all team members, but anyone in management who walks by. This is the "Project Progress Poster" concept that Weinberg recommends in Quality Software Management: Anticipating Change. Since in my company, few people in management walk by, we keep tracking information in our wiki web pages.
Quality Assurance. Weinberg recommends "Prevent these abuses by having quality assurance report to the highest levels of management, and not to project management." In XP, QA testers are delegates of the person playing the Customer role -- the same person who defines the requirements. QA testers should not only be implementing and running the automated acceptance tests, but also running stress tests and manual testing of the product's user interface.
Weinberg reports that testing often comes too late in the project to be useful in effective risk management. (There's a phrase in XP circles: "Doctor, it hurts when I do this..."). Don't wait until it is too late. XP requires testing to start in the first iteration of the project -- the first week. This and other XP practices enables effective risk management.
The XP solutions noted here require that project management be willing to face reality at all times. One of the quickest routes to failing with XP is to not do XP. If project management destroys information, hides it, degrades it, or inserts misleading inforation, intentionally or not, it is going to be very difficult to have a successful project, no matter what the methodology used.
2003.Mar.23 SunWriter Ferrets by Richard Bach
The Ferret Chronicles : Writer Ferrets: Chasing The Muse, by Richard Bach. Quite a delightful little book about writing, publishing and writer's block. Satisfying while reading, but less satisfying in retrospect. I'll be reading the other Ferrets books as I find them. It makes me wonder how much of the book is autobiographical... but not enough to ask the author and possibly get a disappointing answer. Does anyone have a translation of the runes before and after the title page?
2003.Mar.22 SatI read some years ago in the book A Great Place to Work that a good company allows people to not "be a part of the family" -- it's ok if the workplace is "just a job" -- though I hope it's a job done well.
I don't play pool or foosball at my office; I'm not in the company softball team; and while I wanted to get together with the radio-control-car guys, I was too busy and didn't want my rc-car to get muddy. My wife is still recovering from playing in a softball game for her office a week or more ago.
My preference for "fun" at the office would be people getting together to learn about software methods, design, and so on (topics I plan to write about in this blog). Getting a group of coworkers together to ride go-carts is not my idea of fun.
It appears that some people in Germany are blaming their dot-com collapse on "fun" in the workplace. Noted in Laurent Bossavit's blog pointing to article on Mair's End The Fun.
Mair's office rules seem to have gone too far in the other direction - uniforms required, no calendars or pictures on the walls, half-hour lunches. And this is in an advertising agency?!
I fail to see how uniforms get creative work done better or faster. Doesn't anyone blame to dot-com crash on bad business plans, profit-mongering brokers, and credulous investors?
A Great Place to Work says that the core truth of what makes a company successful, is mutual trust between employees and management. Quoting the preface: [a great place to work] requires the direct involvement of senior management who must insist that fostering an exceptional work environment is an explicit goal of the organization - one that is on a par with other explicit goals like making a profit or providing high-quality products or services. Being a great place to work cannot be a fad of the month or it will rightfully appear to employees as another version of management by manipulation.Nothing like a little bit of insomnia for allowing me some time to write.
This is a test of the Blosxom blogging system. If this had been an actual emergency, I would not be sitting here at a computer running a test.
A few weeks ago, around 5 am, I wrote a short essay on the "secret of agile software development", mentioned it on the AYE wiki, got feedback on it from two people and did two more drafts within the next few hours. By the time a third person reviewed it, it was pretty much in final form. It's great having "instant collaboration" like that.
I sent that to a magazine to hopefully be published, but so far I haven't heard back on it. In this day of email, I would hope to get an semi-automatic acknowledgement, if not an actual response, within a few days... but I guess computer-related magazines are not necessarily all that automated. If month goes by, and I still don't hear back, I'll accept Esther Derby's invitaton to publish the essay on the AYE wiki.