Robust key message handling in Windows

Handling text input correctly when developing a Win32 application is not as simple as it could be.  The different types of messages don’t seem to be all that well understood…  You need to manage quite a number of different, potentially conflicting types of input.

Flaws and legacy cruft in Windows’ keyboard input design create some painful limitations — attempts by application authors to work around these flaws have introduced a number of problems, some of which I discuss in the Pitfalls section at the end of this post.

I want to talk about 4 types of input, and how to create a robust (or at least consistent) way of differentiating and prioritising them:

• Character input — "abc" typed into a EDIT control
• Shortcuts or accelerators — up-arrow, Ctrl+X.  A shortcut is defined as modifier[s]+keystroke.
• Access keys — Alt+F for File menu.  An access key is defined as Alt+character, not Alt+keystroke.
• Control keys — backspace, enter, escape, etc.  These keys map to ASCII range 1-31 (and 127).

How does Keyman come into this?

You may know that I am the author of Keyman – a tool for creating much more sophisticated keyboard layouts than Windows provides.  However, nothing in this post relates to Keyman.  All of these issues are pure Windows – they don’t involve Keyman at all.  But if you get this model right for Windows, then Keyman keyboards will also just work without any further issues.

What’s the Problem?

As an example, I will use the English (United Kingdom) keyboard layout, the combination AltGr+O (AltGr is the same as the right alt key) and Microsoft Word.  I use Microsoft Word as it is probably the most widely used major application in the world — and it’s written by Microsoft.

Note: the AltGr modifier is regarded as identical to Ctrl+Alt by most Windows keyboards.  This is a holdover from the times when many keyboards did not include a right Alt key.  Most keyboards today do include the right Alt key, even many laptop keyboards.

So what does AltGr+O do?

• The English United Kingdom keyboard layout defines that this combination is character input that will insert the ó character into your document.
• Word thinks that Alt+O is an access key (either alt key) that opens the Format menu.
• Word also thinks that Ctrl+Alt+O is a shortcut that will switch to Outline view.

Which one takes precedence?  And which one should take precedence?  Read on to find out the answer to both questions.

Shortcuts and Characters and Accelerators … oh my

Unfortunately, there is no clear guidance on precedence between these three types of input.  Word does the following, which is fairly typical:

• Shortcuts take precedence over characters – Ctrl+Alt+O switches to outline view
• Characters take precedence over access keys – AltGr+O generates the ó character
• Left Alt+O will open the Format menu.

That sounds almost like the best possible compromise.  But is it?  What if your computer does not have a right Alt key?  Should Ctrl+Alt+O insert ó or should it switch to Outline view?  I believe that Word’s current precedence is wrong:

• There is an alternative method for accessing Outline view through the menu system;
• It is easy to redefine a particular shortcut if you really want to use it;
• It is not easy to redefine your keyboard layout, and accessing the ó character through Insert|Symbol is painful.

Unfortunately, Windows does not make changing this precedence easy for application authors — in fact, it is basically impossible to do it robustly.  So, to make the best of a bad situation, let’s stick with what Word does – at least it is familiar to end users!

Clean input is simple

1. Use the Windows message loop model of TranslateAccelerator/IsDialogMessage, TranslateMessage, DispatchMessage.  If you have your own shortcut management, use that in place of TranslateAccelerator.  Read up on message loops (but please, please, use threads rather than PeekMessage polling for lengthy operations – unlike "Examining a Message Queue").
2. Understand what the different character messages are for:

• WM_CHAR is a character.  Apart from a few special cases discussed below, these should always be inserted into the currently focused text area.
• WM_SYSCHAR is a system character.  Pass it to the system to allow access keys to operate.  Never insert these characters into the currently focused text area.
• WM_DEADCHAR is a notification message.  You can probably ignore it — as most apps do.
• WM_UNICHAR is a legacy compatibility message.  It allows you to receive Unicode input from certain utilities (notably Keyman) even on non-Unicode systems such as Win9x.  You do not need to use this message for modern, Unicode applications.

Special cases in character input

When a WM_CHAR message is received, there are a few character codes (in wParam) that should be handled as special cases:

8: backspace
9: tab
13: enter
27: escape
127: ctrl+backspace
all other codes < 32: typically, ignore these.

You may (quite sensibly) choose to ignore these at WM_CHAR time and instead process them in your WM_KEYDOWN handler.

Pitfalls to Avoid

I have seen examples of each of these errors in many projects, both open and closed source.  I’m not going to name any names — many of the projects have since corrected these issues … but not all.

I discovered most of these issues while debugging compatibility issues with Keyman for some projects, but I found that in all cases, these issues also impacted standard Windows keyboards.

• Don’t reinvent the wheel: avoid the temptation to use the keyboard information functions (MapVirtualKeyEx and related functions) to reimplement the WM_KEYDOWN -> WM_CHAR translation.  There are several reasons why:
  • The APIs don’t give enough information to do a complete implementation.  Some keyboards just will not work correctly.
  • You limit yourself to mimicking the existing infrastructure and guarantee that future enhancements to Windows keyboard input will not work on your application.
  • And finally, it’s a whole lot of work and maintenance for no real gain.
• Don’t try and correlate WM_CHAR and WM_KEYDOWN:
  • Windows messages are essentially stateless.  Any assumption of state is simply a case of relying on undocumented internal behaviour.
  • Don’t PeekMessage for WM_CHAR when a WM_KEYDOWN message is received to see if it has a corresponding WM_CHAR message.
  • Don’t assume that each WM_CHAR has been generated by a WM_KEYDOWN message – or vice versa.
  • This can break some voice input and pen applications, as well as Keyman!
• Don’t try and match specific message sequences.  If you find yourself doing this, it’s a good hint that you could probably rework your message handlers to avoid the problem.
• Don’t pay any attention to modifiers when processing WM_CHAR:
  • Don’t use GetKey[board]State or local flags for managing modifiers or Caps Lock – it’s a character that should not be modified before insertion into your text store.  It’s not a keystroke.
  • You’ll break any Windows keyboard layouts that use modifiers (hint: AltGr is only one of  the modifiers available for Windows keyboards).
  • Don’t do a special-case for AltGr – other modifiers are also legal.