If you've got a great idea for a prestige class, but you're not sure how to make it balanced, the following strategy might help.
1) Come up with an idea for a prestige class. Don't think about mechanics yet. Get the concepts straight in your head.
2) Look at the normal classes to understand how their power increases as they gain levels. Fighters get one new ability (a feat) every two levels, a great BAB progression, great hp progression, average saves, and few skills. Wizards, and Sorcerers get a couple new spells every level from a pool of very versatile spells, few or no additional special abilities, poor hp, poor BAB, average saves, and few skills. Clerics get a couple new spells from a less versatile pool of spells, no additional special abilities, good HP, average BAB, great saves, and few skills. The other classes can be measured the same way.
3) Find the normal class which is most like the concept for your prestige class. Using the normal class as a template, start removing/reducing features that aren't important to your concept. Then start adding features that are important to the concept. Add and remove restrictions if you need to. Do this until the prestige class's abilities look like what you want. Keep track of how many things you added and removed.
4) Figure out the value of each feature and restriction you added or removed. This is tricky. Some features have varying importance in different circumstances. A fighter BAB is much more valuable to a combat class than a spellcasting class; the combat class will get more use out of it. Hit die size and saves are valuable to everybody. Skill points become more or less valuable depending on how good the class skill selection is. Other special abilities need to be examined carefully. Also, roleplaying restrictions (such as a Paladin's code of honor) aren't as valuable as you'd think.
5) Add up the values of the added features and the removed restrictions. Call this the "cool score" (for lack of a better term).
6) Add up the values of the removed features and the added restrictions. Call this the "lame score."
7) Compare the results from step 5 and 6. If the cool score is less than the lame score, your prestige class may be too weak; go back to step 3 and add some more abilities. If the cool score is much higher than the lame score, your prestige class may be too powerful; go back to step 3 and remove abilities. If the cool score is equal to or a bit higher than the lame score, it's almost balanced; move on to the next step.
8) Now is the time to come up with the prestige class prerequsites. This is where you put the final balance into the class. The greater the difference between the cool score and the lame score, the nastier the prerequisites need to be. "Nastier" means "more difficult for a player to accept." Nasty prerequisites are the price you have to pay to gain any extra power in a prestige class. "Wasted" or "useless" feats, skill ranks, and spell slots are the best nasty prerequisites. Roleplaying prerequisites (racial requirements, alignment restrictions, mandatory ceremonies) are generally poor prerequisites. If the cool score is equal to the lame score, don't make the prerequisites nasty; don't make the player pay a heavy price unless they get something in return -- just make the player pay for what they get for their PC.
9) Make some PCs with the new prestige class. Compare them to PCs that haven't taken the new prestige class. A PC with your (or any) prestige class should be no more powerful/useful in general situations than if they hadn't taken a prestige class. For example, a 7th-level Fighter/10th-level Duellist really isn't any better than a 17th-level Fighter. They have different abilities, sure, but in the long run they're pretty darn equal. If your prestige class meets that same standard, you're done! If not, go back a few steps and alter stuff until it does meet that standard.
Big Caveat: I've never actually created a prestige class, so take this with a grain of salt. I do, however, use these techniques to analyze many of the prestige classes I've seen, and they work quite well.
If my technique doesn't help, the following might: Dragon #274 had an article called "Class Acts" which discussed the techniques for designing prestige classes. It was written by Monte Cook, one of the core 3e designers. See if you can find a copy someplace. I've seen several libraries that stock Dragon in the periodicals section.