Usability for Nerds/Predictability
Imagine that you are writing a complicated text. Your screen looks like this:
Now you want to write something more. What happens when you start writing, let's say by pressing the A key? The highlighted text is deleted and replaced with the new text! I am sure all beginners have made this mistake. A novice user would certainly expect the A key to write an A, not to delete text. Even the most experienced users sometimes make this mistake when they are not looking at the screen and have forgotten that there was something highlighted. The problem here is that the behavior of the A key is not always the same. If there is no highlighted text then it just writes an A. If there is highlighted text then it deletes the text and replaces it with an A. You cannot expect the user to look at the screen all the time, so everybody can make mistakes when the same key doesn't always do the same thing.
Of course, there is an idea behind the principle that you can make corrections by highlighting the unwanted text and then just writing something else. But it only saves you one keystroke, the Delete key. And the consequence is that you may inadvertently lose a big piece of text that you have problems recalling. (There may be an undo function which can restore the deleted text, but if you don't discover the mistake immediately, it may be too late).
Avoid modes!
editThere is a similar problem with the CapsLock key. A letter key produces a capital letter when the CapsLock is on, and a small letter when the CapsLock is off. Technically speaking, the keyboard can be in two different modes: CapsLock on mode and CapsLock off mode. The meaning of the keys depend on the mode. Older systems may also use NumLock, ScrollLock, and Insert keys to produce different modes, but the use of these is avoided in most modern systems because modes always cause usability problems. The CapsLock key still lives because there is no good alternative. When modes are unavoidable then the mode should be displayed in a very conspicuous way on both keyboard and screen, for example by giving the cursor a different shape or color.