Introduction

Keyman 8.0 improved mnemonic keyboards by remapping the On Screen Keyboard to the underlying hardware layout. Previous versions of Keyman did not do this well. This has been a big improvement and makes mnemonic keyboards much more useful with European hardware layouts.

However, apart from this one change, we had to postpone some of the design features that we wanted to add to the On Screen Keyboard. A couple of weeks ago we had a great discussion on how we could improve the On Screen Keyboard by making it more dynamic. While dynamic updates to the keyboard have been on the future feature list for some time, details of how this could work had never been properly explored.

We also linked this discussion to some tweaks to the keyboard options feature introduced in Keyman Desktop 8.0. These were tweaks primarily around testing for base layouts and providing feedback to users when options are selected. The extensions to keyboard options have some relationship to a dynamic On Screen Keyboard but will be expanded on further in a separate blog.

Dynamic On Screen Keyboard

The big breakthrough we made in our discussion was in how we could instruct the On Screen Keyboard to dynamically update. Initially we just thought we could use the existing rules in the keyboard source code to tell the keyboard how to dynamically update. This seemed logical, but the more we looked at it, the more we realised that this would end up being confusing for the end user. There were a few reasons:

1. Keyman Desktop keyboard rules have to deal not only with basic mapping but also with character reordering and normalisation. These rules would probably be confusing when displayed on the keys.
2. Some rules make changes to multiple characters in the context but displaying the full output would not be helpful to the end user.
3. Some rules may input an invisible character but the On Screen Keyboard should instead show a hint for that letter

So we came up with the idea of a new begin statement to cater for the On Screen Keyboard (OSK). This could point to the same group as the normal begin unicode statement, or to a separate group designed to update the On Screen Keyboard. This group could use all the same context matching and finesse that the normal groups could, but the output would only be going to the On Screen Keyboard. So one trick to remember is that the output of the On Screen Keyboard rules will not affect the context for the next keystroke. The new OSK begin rule would be written as follows:

begin OSK > use(osk)

group(osk) using keys

The processing of any groups fired by the begin OSK statement have an important nuance: every matching context rule will be processed in order to render the On Screen Keyboard, for each keystroke. This means that effectively the On Screen Keyboard would run the group 47 or 48 times when updating. This shouldn't have a significant performance cost on computers today. If no rule matches for a given key, then Keyman would render the default output based on either the underlying layout or the US base key (depending on whether the keyboard is mnemonic or positional).

Keeping this in mind, we realised that colour, highlighting, graphics and font hints could be added to the output with a few additional simple statements. These additional formatting hints would be ignored by the keystroke processor, and would just be used by the On Screen Keyboard engine.

• bg(colour | default), or background(colour | default)
  
  This would set the background colour for the whole key. The last colour set in the execution of the statements would be the one that would take effect. The colour reference will probably be either a #rrggbb triplet or a standard web colour name.
• bg(colour | default), or background(colour | default)
  
  This would set the foreground colour for the subsequent letters to be printed on the key cap. In some scenarios, for example with combining diacritics, this may not be possible, and needs to be investigated.
• font(name, size, style)
  
  There are some issues with specifying the size due to the resizeability of the on screen keyboard. It may be specified as a percentage of default size.
• image(source-image)
  
  Source-image would refer to a store name which would then cause the image to be embedded in the keyboard file for use in the On Screen Keyboard. Suggested file format would be PNG to allow for alpha transparency. There are issues about sizing and mixing images and text that would need to be resolved.
• hint(text)

  If the user hovered the mouse cursor over the key, the hint text would be displayed in a balloon. There may be some visual indication on the key of the availability of a hint.

As there may no longer be a static On Screen Keyboard, we may need to specify font and other preferences such as 101 or 102 key layout as well. This is also a topic for further discussion.

Example Dynamic On Screen Keyboard Source File

The following straightforward keyboard will be used as an example of how we envisage dynamic on screen keyboards working.

store(&VERSION) '8.0'
store(&NAME) 'Dynamic OSK Example'

begin Unicode > use(main)
begin OSK > use(main)

store(vowel) 'aeiouAEIOU'
store(diaK) "`" "[" "]" ";" "'" "."
store(diaO) U+0300 U+030F U+030B U+0308 U+0301 U+0307
store(ZWNJ) U+200C

group(main) using keys

+ any(diaK) > $ZWNJ index(diaO, 1)

$ZWNJ any(diaO) + any(vowel) > index(vowel, 3) context(2)

c nomatch -> use default output

For simplicity in this example I have not tried to use any pre-composed letters but just left the combining diacritics in place. You'll note that we are using the same rule set for the Unicode rules as for the OSK rules, despite the discussion above. This is because this example is demonstrating the concepts rather than being a comprehensive and complete keyboard layout.

Given the keyboard source above, the Dynamic On Screen Keyboard would display initially as the image below. There isn't anything extraordinary there. The key difference is that there is no static definition for the On Screen Keyboard; rather, Keyman uses the begin OSK statement to define the display of the On Screen Keyboard. Deadkeys (as per the dk(diac) statements) would not be highlighted in any way on the On Screen Keyboard by default. This is because the deadkey statement in Keyman keyboards tends to be used for state management and not purely as a deadkey flags.

Let's look at how the keyboard could be modified to highlight deadkeys. The formatting hints given the example above would be ignored by Keyman Engine for keystroke processing, so let's keep the rules together for now.

store(&VERSION) '8.0'
store(&NAME) 'Dynamic OSK Example'

begin Unicode > use(main)
begin OSK > use(main)

store(vowel) 'aeiouAEIOU'
store(diaK) "`" "[" "]" ";" "'" "."
store(diaO) U+0300 U+030F U+030B U+0308 U+0301 U+0307
store(ZWNJ) U+200C

group(main) using keys

+ any(diaK) > $ZWNJ index(diaO, 1) bg(yellow)

$ZWNJ any(diaO) + any(vowel) > index(vowel, 3) context(2)

c nomatch -> use default output

Now with an empty context the keyboard would display something like the following mockup:

Finally, let's tweak the keyboard to make the dynamic keys with context highlighted so they are more visible. I'm not going to make the example comprehensive – it'll only cover the grave accent key.

store(&VERSION) '8.0'
store(&NAME) 'Dynamic OSK Example'

begin Unicode > use(main)
begin OSK > use(main)

store(vowel) 'aeiouAEIOU'

store(diaK) "`" "[" "]" ";" "'" "."
store(diaO) U+0300 U+030F U+030B U+0308 U+0301 U+0307
store(ZWNJ) U+200C

group(main) using keys

+ any(diaK) > $ZWNJ index(diaO, 1) bg(yellow)

$ZWNJ any(diaO) + any(vowel) > index(vowel, 3) context(2) bg(yellow)

$ZWNJ U+0300 + '`' > '`' bg(green)
$ZWNJ U+0300 + any(diaK) > context(1) index(diaO, 3) bg(grey)

c nomatch -> use default output

This would render as per the following example.

Conclusion

The concepts outlined here present a simple model of how on screen keyboards could give feedback to the end user and help make keyboards more self-documenting. I'd be very interested in any feedback or suggestions you may have – just put them in a comment on the blog.

In the next blog post, I'll discuss how the tweaks to options we discussed can also be used for integration of the On Screen Keyboard.

Updated 23 Aug 2011: In examples, replaced any(diaK) with any(diaO) in context, and replaced deadkey usage with ZWNJ to avoid side-effects with typing diacritic after another letter.