Mouse meets touch

Touch is the latest input interaction weapon of choice for technology innovators.  Everyone in the core is aware that Apple has been supporting multi-touch tack-pads for a couple years now. Not to diminish their credit for making it mainstream, but the track-pad was an obvious candidate. A device where touch input was already mapped to traditional mouse cursor manipulation and its implementation was already that of a capacitive touch surface! No brainer! Thus, without any surprise, over the past few years we have seen the hardware industry starting to layer on touch surfaces (touch digitizers) onto our traditional mice.

Touch Mice

apple_mouse  touch_mouse  arc_touch

For the last couple years I was a touch interaction program manager for the Windows team, and I had the opportunity to partner alongside the Microsoft Hardware team in their research and development of the newly released Microsoft Touch Mouse. The goal was to create a touch optimized mouse, tailored for Windows7, which pushed industrial design (ID) and did not compromise on core mouse functionality and ergonomics.


Engineering a Touch Mouse UX

To be clear, I wasn’t on the hardware or research teams that build this mouse, but as a partner / consultant of sorts on the project there are a few design and software challenges that were interesting… worth sharing here at least.

As a UX program manager, the name of the game is to step back, analyze the requirements and figure out what the experience landscape of the project truly is. What I learnt quickly was that creating a touch mouse is not simple feat. When executing against any program like this there are many gears in motion to get it out the door. From a UX point of view, there are a handful of experience factors that need to be measured and balanced to deliver a desirable user experience. Each experience factor is tightly related with the next, and unfortunately in most cases, a direction in one factor will lead to a a large challenge or tradeoff from another… in the end you can only hope the customer agrees with the decisions you made.


Experience Factors



Industrial Design

It’s always nice to look good. The goal for the design team is to find the balance of brand, industrial design, and technology. When looking at the Microsoft Touch Mouse the goal seemed to be to drive the newly emerging ID brand family while pushing the boundaries on material and technology.




The early versions of Microsoft Touch Mouse included three types of digitizer explorations: camera-based, capacitive-sensing, and articulated (which used multiple existing mice linked together) // read more of the story here



Differentiated Design (Artist Edition)


Comfort and Ergonomics

Comfort and ergonomics has been the name of the game for great mice over the decades. When looking at a touch mouse there are some factors in particular that came into play. 

  • The more ergonomic you make a mouse the less flat it becomes. As you start to curve the surface, the interaction language (in this case finger gestures) can change drastically. 
  • The mouse will have a desired form which should compliment the desired primary target posture.
  • The gesture language needs to be supported in such a way that does not interfere with users comfort in the target posture.
  • It still needs to be a kick ass mouse at the end of it all.


When comparing the touch mouse to the magic mouse through the ergonomic lens, I feel that Microsoft succeeded over Apple in delivering a comfortable experience in the target posture.  Apple has been credited for a desirable design, however there have been many users complaining of developing wrist pains and finger fatigue.

The Magic Mouse does not contour to the users hand” – MMFixed

The folks over a MMFixed have done a great analysis of the magic mouse and have gone so far to offer a add on solution to making it more confortable for everyday use. 




When I think UX performance I am thinking: end user performance, software reliability, power, accuracy, and more generally “this thing is worth having”.

As the form factor starts to take direction (through the balancing act between ID and ergonomics), the software and hardware engineers start prototyping everything from digitizer tech, to touch gesture engines. 

For the Microsoft Touch Mouse having the drive for a more shapely and thus ergonomic form, the challenge now comes from the fact that the users hand will be gripping, resting and constantly touching more of the surface area on the mouse. Which also means that it’s the same area ideal to perform comfortable finger gestures.

Performance UX Challenges 

Interactive Language

    • Primary goal is to ensure gestures are comfortable to perform.
    • User must perceive the gesture to be quick and avoids re-posturing to perform (a low initial cognitive tax to quickly get to habitual / instinctive is the goal).

Gesture Recognition on a Curved Surface

    • Creating touch interaction and gestures engine against a flat surface is a relatively easy challenge today. (Phones, tablets, tables, track-pads all have established gesture recognition algorithms and techniques we can utilize).
    • The challenge comes from creating a reliable gesture engine on a curved surface, where the differences in the users hand size, shape and posture will lead to more drastic changes to that of touch screen.
    • Gesture like “pinch and stretch” became more fatigued and harder to recognized reliably due to the curved surface .
    • Interactions like a “thumb-swipe” were gained thanks to the distinct curves of the mouse (try it, its awesome!).

Posture and Constant Touching

    • Noise! Touch is a two state model (touching or not touching), how do you best recognize the users intent of gesturing rather than just a finger moved when the the mouse is just being used to manipulate the cursor?
    • With an active capacitive digitizer, constant touches will draw power and the gesture recognizer will always be on to constantly deterring intent. (defiantly don’t want to sell a mouse you need to change batteries for every month).
    • Below was my development hardware device the team used to work through all the above problems.



Tailored for Windows 7

The goal is to make a kick ass touch mouse, but more importantly a kick ass touch mouse for Windows7. A device created to amplify the Windows 7 experience. This means not just creating cool gestures that have no substantial mapping to the software it controls. Everyone one loves to pinch their fingers together to zoom. But is zooming really an essential Windows7 experience? Maybe new features like desktop window arranging and task switching should come to life. I don’t want to give it away, but they did a pretty good job not going completely overboard and over engineering the experience.

As the team from MS Hardware put it

  • Enhances Windows 7 navigation
  • Allows easy switching between tasks
  • Easy to learn and fun to use
  • Uses gestures to quickly scroll and pan, navigate, and manipulate content
  • Helps you get more done in less time


You can find the gesture set and experience demo here (in this ridiculous video).