https://docs.google.com/document/d/1iN9 ... ejolA/edit
1 **Consumer-level, developer eye trackers, and SDKs available**
2 **Eye tracking can remove physical limits**
3 **First round of eye trackers intended for developers**
4 **Eye tracking companies – Tobii and Eye Tribe**
5 **Eye-tracking (for initial warping/teleporting of cursor, and large cursor movements) + game controller (for precise cursor movements, finishing with an accurate cursor placement, and clicking)**
6 **Eye-tracking pointer motion teleport + mouse: use eye-tracking for initial warping/teleporting of cursor, and then use precision of mouse to finish selection – research paper on benefits of initially warping cursor**
6.1 **Mouse-cursor-teleport user setting: time that mouse controlled cursor must be in rest before eye control is involved again (mouse precision still in use)**
6.2 **Mouse-cursor-teleport user setting: point-of-gaze must be a certain distance from the mouse controlled cursor before eye control is involved again (eye-tracking is activated for larger cursor jumps)**
6.3 **Research paper: “Mouse and Keyboard Cursor Warping to Accelerate and Reduce the Effort of Routine HCI Input Tasks – competition between mouse vs. an eye tracker + mouse to click randomly generated buttons as fast as possible”**
7 **Eye tracking working with speech recognition**
7.1 **Adding context, and limiting the number of selection choices that a “select-what-I-say" speech command will match**
7.2 **Fast speech corrections with “correct-what-I’m-looking-at” button**
7.3 **Google Earth - eye tracker to get location-based information, and speech recognition to initiate an action according to location e.g. what city is this?**
7.4 **Commands that use both speech and eye input – e.g. select range of text lines**
7.4.1 **Detecting interface elements – Vimium, VimFx, LabelControl, et al.**
7.4.2 **Select range of text lines**
7.4.3 **Advantages of eye tracking and speech in range selection command**
8 **Make eye tracking work with existing, smaller-element applications**
8.1 **Magnification/zoom for clarification of smaller, close-together elements**
8.1.1 **Magnification in eye-tracking interfaces**
8.1.2 **Zoom in touch interfaces e.g. Chrome on Android**
8.1.3 **E.g. of zoom in PCEye interface**
8.1.4 **Multiple magnifications for commands that require multiple user input steps**
8.2 **Make interface elements of existing applications responsive to gaze e.g. detecting words on a webpage**
8.2.1 **e.g. research paper: detecting words on a webpage**
8.3 **Cursor snapping to interface elements**
8.3.1 **Dwell Clicker 2**
8.3.2 **EyeX for Windows**
8.3.3 **MyGaze tracker – snapping to desktop interface elements, buttons of WizKeys on-screen keyboard, and Peggle game**
8.4 **Generate/project large-button, touch/eye-tracking UI from existing non-touch/non-eye-tracking UI**
8.4.1 **Tag elements near point-of-gaze w/ colors, IDs, and lines – project to large elements (screenshot and mock-up)**
8.4.2 **Only pop out alternate elements that are near point-of-gaze, instead of considering all the elements, like in Vimium**
8.4.3 **Can have less organization, and use flow layout for projected larger elements: easier to set up, and quick eye movements can still work with disorganization (eye-controlled cursor covers distances between scattered objects quickly)**
8.4.4 **Possibly keep generated letters/numbers/IDs of Vimium-like, element-detection programs for labeling large elements**
8.4.5 **Generate colors to match smaller elements with larger elements – similar to semantic color highlighting in programming editors**
8.4.6 **Displaying lines to connect smaller elements to larger elements**
8.4.7 **Conclusion – two-step projection process can still be fast**
8.4.8 **E.g. DesktopEye: open-source prototype for generating eye-tracking compatible versions of window processes**
8.4.9 **E.g. generating speech-compatible versions of open windows**
8.5 **Conclusion – give people a sample of a potentially speedier touch/eye-tracking interface**
9 **Current interfaces, and open-source interfaces for eye-tracking**
9.1 **Interfaces by eye tracking organizations: PCEye, EyeX, GazeMouse, GazeTalk**
9.1.1 **Floating window, or detachable quick-access toolbar for quicker access?**
9.1.2 **GazeTalk – free, predictive text entry system**
9.1.2.1 **Notable feature of GazeTalk: accumulate dwell time: resume dwell time after looking away**
9.2 **Action upon slowing down, or stoppage of the cursor = eye is fixating**
9.2.1 **E.g. Activating widgets in GazeMouse, PCEye, and bkb**
9.2.2 **E.g. video demonstration: slow-down upon fixation to increase cursor steadiness in a game**
9.2.3 **E.g. Reverse cursor upon slowing down, or stoppage of the cursor (minimize overshooting) – found in video for paper, “Mouse and Keyboard Cursor Warping to Accelerate and Reduce the Effort of Routine HCI Input Tasks”**
9.2.3.1 **Conclusion**
9.3 **Other software interfaces – free head tracking software: Camera Mouse, eViacam, and Mousetrap – head-tracking interfaces similar to eye-tracking interfaces**
9.4 **Open-source interfaces**
9.4.1 **Eye-tracking Python SDK for buttons + another scripting language (AutoHotkey, AutoIt, AutoKey, PYAHK, Sikuli, etc.) for macros**
9.4.1.1 **AutoIt example – “left-click-with-magnification”**
9.4.1.2 **Automatic generation of touch/eye alternatives for smaller elements, vs. manual creation of on-screen buttons for more complex macros**
9.4.2 **Open-Source keyboards**
9.4.2.1 **e.g. web-based virtual keyboard, and JavaScript framework**
9.4.2.2 **e.g. Virtual Keyboard using jQuery UI**
9.4.3 **Alt Controller – cursor going over designed regions can launch actions – e.g. page-down or scroll-down when gaze reachs a region at the bottom**
9.4.4 **DesktopEye: open-source prototype for switching between window processes**
9.4.5 **Open source, eye-tracking, predictive-typing software program by Washington State University student group, Team Gleason**
9.4.6 **bkb: open source application to control keyboard and mouse with the Tobii EyeX, The Eye Tribe gaze tracker, or an Airmouse (e.g. Leap Motion, Haptix) – has mouse actions, virtual keyboard, and automatic scrolling**
9.4.6.1 **Repeating click mode**
9.4.6.2 **Automatically scroll down when eyes reach bottom of window**
9.4.7 **Gazespeaker – design your own grids that have cells that launch actions, predictive keyboard, automatic scrolling, shareable grids, desktop or tablet**
9.4.7.1 **Work from within the program with built-in interfaces for eye-tracking e.g. custom email interface, and web browser – similar to The Grid 2, Maestro, and Vmax+ software**
9.4.7.2 **Customized grids with customized cells made with a visual editor (cells launch AutoHotkey, AutoKey, Autoit, PYAHK, Sikuli etc.?)**
9.4.7.3 **Sharing custom grids and cells – visual grids are easier to share, as opposed to sharing something like an AutoHotkey or Autoit script/text file**
9.4.8 **Eye-Tracking Universal Driver (ETU-Driver) – independence from particular eye tracking device**
9.5 **Conclusion**
10 **Google speech and accessibility**
10.1 **Growth of speech**
10.1.1 **Taking more and more context into account**
10.1.1.1 **Google Research: - “we are releasing scripts that convert a set of public data into a language model consisting of over a billion words”**
10.1.2 **Valuing speech recognition**
10.1.2.1 **E.g. “Spell Up - a new word game and Chrome Experiment that helps you improve your English”**
10.1.2.2 **Speech recognition in Android Wear, Android Auto, and Android TV**
10.1.3 **E.g. speech-to-text dictation is coming to the desktop version of Google Docs**
10.2 **Google accessibility**
11 **Eye tracking + Google speech recognition**
12 **Eye tracking potential depends on software – e.g. eye-typing**
12.1 **Video examples of eye-typing: PCEye, JavaScript web-based virtual keyboard, bkb, and Gazespeaker**
12.2 **E.g. of fast eye-typing: Eyegaze Edge**
12.3 **Thesis – review of the research conducted in the area of gaze-based text entry**
12.4 **Advanced software e.g. Swype, Fleksy, SwiftKey – eye-typing with Minuum on Google Glass**
12.4.1 **BBC video interview: Gal Sont, a programmer with ALS, creates Click2Speak, an on-screen keyboard that is powered by Swiftkey**
12.4.2 **Eye-typing with Minuum on Google Glass**
12.4.3 **Google software**
12.5 **Touch software and UIs (and thus close-to-eye-tracking interfaces) coming to desktops, laptops, notebooks – e.g. touchscreen Windows 8 laptops, foldable Chromebook, Project Athena in Chromium, Material Design, Android apps going to Chrome OS, HTC Volantis (Flounder/Nexus 9) 8.9" tablet**
12.6 **Auto-complete and auto-correct for eye-typing code**
12.6.1 **Mirror the drop-down list of auto-complete suggestions to the on-screen keyboard**
12.6.2 **Virtual keyboard indicator for number of drop-down list suggestions**
13 **Building eye-tracking applications by using web tools**
13.1 **Research paper: Text 2.0 Framework: Writing Web-Based Gaze-Controlled Realtime Applications Quickly and Easily e.g. interactive reading: words disappear when you skim them**
13.2 **Text 2.0 framework (create eye tracking apps using HTML, CSS and JavaScript) now known as gaze.io**
13.3 **Advantages of using web technology**
13.4 **Tobii EyeX Chrome extension, and JavaScript API**
13.5 **Pupil: web-based virtual keyboard, and JavaScript framework**
13.6 **Conclusion**
14 **Future of eye tracking**
14.1 **Eye tracking patents, and possible, future support from larger companies**
14.2 **OpenShades – Google Glass eye tracking – WearScript: JavaScript on Glass**
14.3 **Augmented reality – future AR: manipulating virtual objects, disability profiles, overlay forearm with buttons (Minuum video), current AR: labeling objects (OpenShades)**
14.3.1 **Manipulating virtual objects, invisible disability profiles**
14.3.2 **Augmented reality Minuum keyboard on forearm**
14.3.3 **OpenShades augmented reality**
15 **Advantages of eye tracking:**
15.1 **Already using your eyes**
15.2 **Comfort and ergonomics**
15.2.1 **Vertical touchscreen pain**
15.3 **Augmentation, not replacement – “click-where-I’m-looking-at” keyboard button, or cursor teleport before using mouse**
15.4 **Bringing speed, recognition, and malleability of virtual buttons in a touch UI to desktop users and vertical touchscreen users that are using a non-touch UI**
15.4.1 **e.g. Control + <whatever> = action vs. visual shortcut: button that is labeled with action**
15.4.2 **Sharing easily learnable scripts and visual interfaces e.g. sharing Gazespeaker grids as XML files**
15.5 **Using only a keyboard for maximum speed**
15.5.1 **Elements that are hard to access, or access quickly by keyboard**
15.5.2 **E.g. editing code without using a mouse: “fold-the-block-of-code-that-I’m-looking-at”**
15.6 **Mobile touch: eye-highlighting + only needing a few buttons (e.g. “single-tap-where-I’m-looking”, “double-tap-where-I’m-looking”) – hands-free scrolling – vertical mobile touchscreen – two-step process for selecting smaller elements, like text links, on non-touch-optimized websites while using mobile**
15.6.1 **Touch gestures + “touch-where-I’m-looking” buttons vs. touch gestures alone vs. mouse-clicking on a desktop**
15.6.1.1 **Advantages of eye tracking + few function buttons: speed, comfort, and less finger and hand movement – single-taps and tablets**
15.6.1.2 **Example apps on mobile**
15.6.1.2.1 **Customizing the Android Navigation Bar (easy-to-reach buttons)**
15.6.1.2.2 **Eye Tribe’s Android demo: tap anywhere on the screen**
15.6.1.2.3 **Launchers that require more than just taps i.e. swiping, double taps – replace with eye tracking + single-taps**
15.6.1.2.4 **Eye Tribe’s corner thumb buttons**
15.6.2 **Vertical touchscreen + “tap-where-I’m-looking” button**
15.6.3 **Hands-free interaction: while eating, while using another computer, etc.**
15.6.4 **Two-step process for selecting harder-to-press links and characters on non-touch-optimized (or touch-optimized) websites while using mobile**
15.6.4.1 **Eye-tracking two-step process for selecting a range of characters and words**
15.7 **Future: head-mounted display with eye tracking + armband, watch, ring, computer vision system, augmented reality system, etc. for input (helps work with limited space and limited gestures)**
15.7.1 **Watches – small screen real estate works with eye-tracking**
15.7.2 **Clothes: Makey Makey and OpenShades**
15.7.3 **Computer vision recognition of forearm (Minuum video) – overlay augmented reality keys on forearm**
15.7.4 **Gestures**
15.7.4.1 **While mobile: armbands, rings**
15.7.4.2 **While stationary: Tango, Haptix, Leap Motion Table mode**
15.7.4.2.1 **Haptix (transform any flat surface into a 3-D multitouch surface)**
15.7.4.2.2 **Leap Motion Table mode (interact with surfaces) – if using air gestures without eye tracking, many more-physically-demanding gestures to memorize**
15.7.4.2.3 **Project Tango and Movidius chip**
15.7.5 **Conclusion**
15.7.6 **Oculus Rift + eye tracking**
15.7.6.1 **Oculus Rift with the Haytham gaze tracker**
15.7.6.2 **Selecting and manipulating virtual objects with more speed and comfort, and less hand positions – e.g. Oculus Rift with camera for augmented reality**
15.7.6.3 **Navigating 20 virtual stock trading screens in Oculus Rift**
15.7.6.4 **Oculus Rift + eye tracking for traversal - non-gamers – “go-to-where-I’m-looking-at” e.g. eye-tracking wheelchair by researchers at Imperial College London **
16 **Conclusion**
17 **Communities**
13 **Building eye-tracking applications by using web tools**
13.1 **Research paper: Text 2.0 Framework: Writing Web-Based Gaze-Controlled Realtime Applications Quickly and Easily e.g. interactive reading: words disappear when you skim them**
There is an eye tracking research paper called “The Text 2.0 Framework
Writing Web-Based Gaze-Controlled Realtime Applications Quickly and Easily” (http://gbuscher.com/publications/Bieder ... mework.pdf), which I briefly mentioned above.
Abstract: “A plugin enables any compatible browser to interpret a new set of gaze handlers that behave similar to existing HTML and JavaScript mouse and keyboard event facilities.
Keywords like onFixation, onGazeOver, and onRead can be attached to parts of the DOM tree and are triggered on the corresponding viewing behavior.
Using this frame-work we implemented a number of applications providing help on comprehension difficulties.”.
From what I understand, the researchers built some applications that use eye tracking to make reading interactive.
One of the features involves the program detecting that you’re skimming past words, so those words will fade out.
In order to make the various reading comprehension programs, they created a “simple-to-use [, now open-source,] framework to construct gaze-responsive applications using web technology”.
https://code.google.com/p/text20/.
13.2 **Text 2.0 framework (create eye tracking apps using HTML, CSS and JavaScript) now known as gaze.io**
The Text 2.0 framework is now known as gaze.io.
http://gaze.io/
https://github.com/gazeio
“Create stunning cross platform eye tracking apps, and it supports both interaction and analysis.
In runs on Chrome, Firefox, Internet Explorer 10+ and Safari, on Windows, Mac OS and Linux, given the eye tracker runs on said platform and has an open API”.
13.3 **Advantages of using web technology**
(From the research paper:)
“HTML, JavaScript and CSS are simple to use and there is a plethora of existing literature covering almost any aspect of these technologies.
That way it is also easier to properly polish an application to get an even more realistic use case than with only partially written prototypes.
Also, the usage of common languages and concepts significantly lowers the barrier for beginners to write eye tracking applications.
Masses of web designers are now easily capable of writing gaze-responsive applications.
Most parts of the application’s interface and reactions have been written by students familiar with web design, but not with eye tracking.
We think this fact nicely illustrates the potential of our easy-to-use eye tracking framework.
The API can be understood by any web designer with a bit of JavaScript experience intuitively.
Simple gaze-responsive applications can be implemented with almost no programming while complex projects benefit from a clear separation of structure (HTML), logic (JavaScript) and design (CSS).
We think the principal choice of using a browser plugin offers fundamental benefits in the long term.
The speed of the development progress in the web community is high and new exciting features are constantly integrated into browsers, enhancing the possibilities of gaze-responsive applications almost for free, as we showed with the implementation of a gaze enhanced tab exposé which we developed in only a few hours.”.
13.4 **Tobii EyeX Chrome extension, and JavaScript API**
Tobii has an extension that enables you to use Tobii EyeX in Chrome: https://chrome.google.com/webstore/deta ... ails?hl=en.
They say that the Chrome extension accesses the EyeX Engine via JavaScript, but is, at the moment, using an internal protocol/API on top of EyeX for Windows.
They plan to add official JavaScript support, and a JavaScript API quite soon.
13.5 **Pupil: web-based virtual keyboard, and JavaScript framework**
As mentioned above, Ignacio Freiberg from the eye-tracking subreddit (http://www.reddit.com/r/EyeTracking) put up a video that demonstrates typing on a web-based virtual keyboard that is based on an open source JavaScript framework.
“Hello everyone!
Imagine you can create web apps using HTML+CSS+Javascript, that can be controlled with Pupil.
Well, I've just built a first prototype (a virtual keyboard).
My short term plan is to build some more apps to show the potential of this, which is huge.
For example, with a few improvements the virtual keyboard could be extremely more effective (predictive text, error correction mechanisms, machine learning).
Many other apps could be *easily* created on this context, including speed reading, social network interfaces, marketing analysis apps, and others.
My long term plan is to build a nice, clean Javascript framework to let people build their own apps.
Just imagine you could add this to your web apps:
$(".myButton").sightOver(function() { });
Comments? Ideas?”.
https://groups.google.com/forum/#!msg/p ... Jn_7p1_OsJ
13.6 **Conclusion**
Programs and tools built with web technologies are improving at a very fast pace.
E.g. Chrome Apps (uses Cordova), which can be written in HTML, JavaScript and CSS, are coming to Android and iOS.
By using possibly easier programming languages, and needing just one code base for a simpler porting process, web tools are becoming more popular and enticing to use.
The Text 2.0 framework/gaze.io demonstrates that eye tracking is already ready to work with web technologies, which should help eye tracking expand.
14 **Future of eye tracking**
14.1 **Eye tracking patents, and possible, future support from larger companies**
Notably, Google has an eye tracking patent that involves recording advertisement impressions through eye focus with pay-per-gaze, and another patent that demonstrates a method to unlock a device by having a sensor in a head-mounted accessory (probably something like Google Glass) track the patterns of the pupil.
(http://www.engadget.com/2012/08/07/goog ... ck-system/)( http://www.wired.com/2013/09/how-google ... e-ad-tech/)
Apple filed an eye tracking patent that deals with something called Troxler fading: “One result of this phenomenon is the perceived fading of a fixated stimulus when its retinal image is made stationary on the retina, which is otherwise known as a stabilized retinal image.”.
“Apple provides systems, methods, and devices that provide a user of a GUI with one or more measures to counteract a perceptual fading of a cursor with respect to the GUI.” (http://www.patentlyapple.com/patently-a ... trols.html)
Also, Nokia has a patent for a future head-mounted device: “The patent explains that there will be two cameras on the wearable device, one which will track the gaze of an eye to help position the cursor on the screen.
The other camera will focus on hand gestures, which will allow users to select an item on a menu.
According to the patent, these gestures could be the shaking of a palm or the movement of a fist.
The first camera could be an infrared camera focused on one eye of the user of the NED.
The second camera would probably be a "so called side down-looking camera observing gestures of the hand."”.
(http://www.phonearena.com/news/Nokia-re ... cs_id53956).
In indicates that eye tracking should have even more backing in the near future.
14.2 **OpenShades – Google Glass eye tracking – WearScript: JavaScript on Glass**
People in the community, OpenShades, have already added eye-tracking to Google glass by adapting from Pupil, an open source eye tracking hardware and software platform that started as a thesis project at MIT.
OpenShades’ objectives include computer vision and recognition, augmented reality, and accessibility.
http://www.openshades.com/.
Just like the people behind the “Text 2.0 Framework: Writing Web-Based Gaze-Controlled Realtime Applications Quickly and Easily” paper saw the benefits of utilizing web technology with eye tracking, the people of OpenShades have also acknowledged the advantages of web tools, and developed WearScript: “WearScript is a library that allows you to execute Javascript on Glass that can interact with the underlying device.
WearScript combines the power of Android development on Glass with the learning curve of a website.
Go from concept to demo in a fraction of the time.”.
14.3 **Augmented reality – future AR: manipulating virtual objects, disability profiles, overlay forearm with buttons (Minuum video), current AR: labeling objects (OpenShades)**
14.3.1 **Manipulating virtual objects, invisible disability profiles**
At 13:30 of a video of Eye Tribe’s presentation at Techcrunch’s Hardware Battlefield (http://techcrunch.com/video/hardware-ba ... 518078469/), cofounder Sune Johansen pitches a future application of augmented reality in glasses to Martha Stewart and the rest of the judges.
You could use your eyes to manipulate the augmented reality projections.
Another example is where a sensor detects that you’re looking at a person, and an online profile pops up.
This could help people with invisible disabilities broadcast their status.
14.3.2 **Augmented reality Minuum keyboard on forearm**
The Minuum video demonstrates computer vision on Glass recognizing a user’s forearm, and an augmented reality system overlaying the forearm with a Minuum keyboard.
The virtual buttons don’t have to show up until the eye tracking system recognizes that the user is looking at the forearms.
(http://youtu.be/AjcHzO3-QEg?t=46s).
14.3.3 **OpenShades augmented reality**
The people of OpenShades showcase a budding example of augmented reality at 2:45 of a video: http://youtu.be/pHCwaaactyY?t=2m45s – “Interactive Augmented Reality using Google Glass”.
A person walks around a lunar landing display in a museum.
When he looks at the Buzz Aldrin and Neil Armstrong manikins, their names are rectified on the screen of the device.
Another augmented reality example from OpenShades can be found at 7:55 of their video, “WearScript: Myo/Pebble/Magnets, Improv, and Selfies”: http://youtu.be/vqdcejLAuU0?t=7m55s.
Here is a transcript of the commentary about augmented reality that begins at 7:55: “To give you a first-person view, I’ll attach Glass through a camera.
When the user taps Glass, a picture is taken, and AR tags are detected.
The detected tag IDs are rendered in an overlay the size of the original picture.
The overlay can be modified directly in JavaScript, opening up a wide range of augmented reality applications on Glass.
Subsequent pictures are aligned to the original using image registration.
We can then warp the overlay to a picture from the user’s current viewpoint, and then, using our prism-to-camera calibration, which we’ve described in previous videos, warp the overlay to align with the user’s view”.
15 **Advantages of eye tracking:**
15.1 **Already using your eyes**
Most people are already using their eyes in computer interaction anyway.
For example, before you move your mouse to select something, it is very likely that you’ve already seen the target.
The same thing goes for touch user interfaces.
Most of the time, a person will see a widget that they want to touch before they actually reach out, and physically touch it.
15.2 **Comfort and ergonomics**
I know of people that experimented with a touchscreen in a desktop environment.
The problem was that it was too un-ergonomical to keep reaching outwards, as the shoulders tire out fast.
I’ve seen plenty of pictures of environments with 3+ monitors, and I don't think that repeatedly lifting the arms to touch the screens would be comfortable over time.
Gorilla arm syndrome: "failure to understand the ergonomics of vertically mounted touchscreens for prolonged use.
By this proposition the human arm held in an unsupported horizontal position rapidly becomes fatigued and painful".
http://en.wikipedia.org/wiki/Touchscree ... lla_arm.22
15.2.1 **Vertical touchscreen pain**
Windows 8 touchscreen pain
“In fact, according to many experts, using a touchscreen in a vertical position is very risky ergonomically.
With touchscreen notebooks comparatively new, most research relates to desktops.
But while the problems with notebooks may not be as extreme, they still apply.
Indeed, Apple's Steve Jobs – not usually one to dismiss a pretty gadget on the grounds of uselessness – once said he'd never launch a touchscreen laptop because of what he called "gorilla arm".
"We've done tons of user testing on this," he said back in 2010, "and it turns out it doesn't work.
Touch surfaces don't want to be vertical.
It gives great demo, but after a short period of time you start to fatigue, and after an extended period of time, your arm wants to fall off."
It is possible, of course, to position a laptop so that the screen is reachable without lifting the elbows from the desk; but this means bringing it much closer than most people find comfortable visually.
Microsoft's online advice on using a PC safely doesn't mention touchscreens at all – and, ironically, instructs users to avoid just those movements that a touchscreen notebook demands.
A company spokesperson said Microsoft had no advice for users about safe use of the Surface touchscreen, and no comment about possible health issues.”
http://www.theguardian.com/technology/2 ... ptops-pain
“Although we reached out to Microsoft for this story, the company did not respond to our request to comment.
In order to touch the display on a notebook with that capability, users either have to fully extend their arm (bad and uncomfortable), lean forward (bad and awkward) or move the display closer (bad for your vision).”
http://blog.laptopmag.com/are-windows-8 ... our-health
Eye control can help you avoid these problems.
15.3 **Augmentation, not replacement – “click-where-I’m-looking-at” keyboard button, or cursor teleport before using mouse**
Eye control can be an additional input that works together with your hands.
e.g. you can use a macro program like Autohotkey or AutoIt to remap a keyboard button to click.
Appskey::Click
Look at any interface element to highlight it, and then touch the application key on the keyboard to left-click, and select it.
If you instead need the precision of the mouse, you can use eye tracking to first teleport the pointer to your target, and then let the mouse override at the end.
15.4 **Bringing speed, recognition, and malleability of virtual buttons in a touch UI to desktop users and vertical touchscreen users that are using a non-touch UI**
When using Autohotkey for remapping buttons, I sometimes didn't have enough unused keyboard buttons to attach macros and lines of code to them, so I'd have to make new scripts that use the same button.
After more buttons and scripts, it can be easy to forget which button does what.
With your eyes being able to cover large distances on a screen quickly, fast activation of on-screen interface elements without a touchscreen is now more feasible.
It virtually turns a non-touch screen into a touchscreen.
You can instantly invoke custom, virtual, on-screen buttons that have your macros and commands that are attached to them.
With on-screen buttons and controls, you can design them to look however you want.
Button labels can be descriptive and self-documenting.
Attaching macros to a growing list of keyboard shortcuts that use Control + Alt + Shift + <whatever> can be more difficult to memorize and maintain.
It would be much easier to build, support, and recognize a collection of customizable and virtual buttons, than rely only on limited and static physical keys.
15.4.1 **e.g. Control + <whatever> = action vs. visual shortcut: button that is labeled with action**
e.g. I remapped Control + F1 to launch a google search on whatever is on the clipboard:
^F1::Run google/com/search?hl=en&safe=off&q=%Clipboard% .
With another script, Control +F1could execute something completely different.
Within a single script, depending on the context, such as what program is currently running, or what window or dialog is in focus, the use of Control +F1could change again; it can get confusing.
It would be more intuitive to look at a virtual button that is actually labeled, "Google Search the Clipboard", and then tap my activation key.
15.4.2 **Sharing easily learnable scripts and visual interfaces e.g. sharing Gazespeaker grids as XML files**
As users of speech command programs, you already know this advantage, as you can create almost any series of natural language terms to activate your speech commands.
It’s probably easier to learn a new set of speech commands than a set of Control + Alt + Shift + <whatever> keyboard shortcuts because what you say can be the same as the action.
Although, visual interfaces, like on-screen buttons, would require even less learning and memorization as the labels are constantly in view.
Therefore, it will be very easy for people to understand the sample scripts and interfaces that will be shared with one another.
If they’re edited, tweaked, and re-shared, people will be able to see the changes more immediately, and learn them much faster
An example was mentioned above in the heading, “Sharing custom grids and cells”, where Gazespeaker grids that are stored in XML files can be shared with other people.
Interfaces that involve activating visual elements are more likely to become widespread than interfaces that require one to recall large lists of keyboard shortcuts.
15.5 **Using only a keyboard for maximum speed**
15.5.1 **Elements that are hard to access, or access quickly by keyboard**
You can try to rely on just a keyboard, and have all the shortcut mnemonics memorized, but not every element has an underlined letter e.g. items in a list box, or bookmarks in a bookmark toolbar.
Also, if you are using something like Vimium to generate keyboard shortcuts for web links, you can’t memorize them because they’re randomly generated every time.
A “touch/click-where-I’m-looking” keyboard button can help you quickly access the rest.
15.5.2 **E.g. editing code without using a mouse: “fold-the-block-of-code-that-I’m-looking-at”**
“Ctrl+NumPad- (Minus) = Fold method, class or any other foldable block of code at caret”.
Just look at the block of code, and press Ctrl+NumPad-.
Not only can an eye-controlled cursor be more instant then a mouse-controlled cursor, but replacing the mouse in certain instances means that you don’t have to keep lifting a hand away from the keyboard to use a mouse.
15.6 **Mobile touch: eye-highlighting + only needing a few buttons (e.g. “single-tap-where-I’m-looking”, “double-tap-where-I’m-looking”) – hands-free scrolling – vertical mobile touchscreen – two-step process for selecting smaller elements, like text links, on non-touch-optimized websites while using mobile**
15.6.1 **Touch gestures + “touch-where-I’m-looking” buttons vs. touch gestures alone vs. mouse-clicking on a desktop**
Eye-highlighting + few "tap-where-I’m-looking” function buttons
If you had eye control on a touch device, you could have a few go-to, base, function buttons (could be two or three) that you can press after you highlight something with your eyes.
One of the buttons could be a “tap-where-I’m-looking” button.
Look, touch an easy-to-reach “tap-where-I’m-looking” button, look, and then touch the same button again.
You don’t have to keep changing your hand and finger positions between each tap.
Touch alone: multiple touch gestures for different actions
Currently, if I take something like the Chrome app icon on the home screen of Android, I can tap it to open it, or long-press/hold it to move it.
(There's also double tap, triple tap, and swiping that are available for use).
Desktop: different types of mouse clicking for different actions
For the desktop, left and right single-click, left and right double-click, left and right mouse-drag, and the middle mouse click are some examples of mouse clicking that achieve different actions on a target once a mouse cursor is on it.
15.6.1.1 **Advantages of eye tracking + few function buttons: speed, comfort, and less finger and hand movement – single-taps and tablets**
Single-tapping keys would probably be faster and more comfortable than repeatedly doing double-clicks, double-taps, long-presses/holds, swipes, or multi-finger gestures, such as pinching and zooming.
On top of only needing single taps, since you may only require a few buttons, your thumbs or fingers reach out for much fewer things.
If you’re working with a larger, tablet-sized screen which requires more hand movement to reach all the buttons and widgets, then assigning yourself to merely a few buttons and hand positions would give you even more of a speed and comfort advantage over those that don’t incorporate eye input.
15.6.1.2 **Example apps on mobile**
15.6.1.2.1 **Customizing the Android Navigation Bar (easy-to-reach buttons)**
E.g. There are applications like Ultimate Dynamic Navbar which allow you to customize the easily reachable Android Navigation Bar.
Besides the three default buttons, you could add “single-tap-where-I’m-looking”, and “swipe-up-where-I’m-looking” (to perhaps simulate a Page Down for reading) buttons, and a couple other buttons (although just a “single-tap-where-I’m-looking” button should allow you to do a lot of things).
15.6.1.2.2 **Eye Tribe’s Android demo: tap anywhere on the screen**
“Looking at icons on a desktop instantly highlights them, and you can then tap anywhere on the screen to open up the selected app.” (http://www.stuff.tv/mwc-2014/eyes-eye-t ... ls/feature).
I guess that in one of Eye Tribe’s demos, they temporarily made the entire screen a “tap-where-I’m-looking” button.
15.6.1.2.3 **Launchers that require more than just taps i.e. swiping, double taps – replace with eye tracking + single-taps**
E.g. In addition to the normal opening of a Home screen Android icon with a single tap, the Action Launcher 2.0 app allows you to swipe-up/down or double-tap on an icon after an icon modification to open a branch of related icons (i.e. an Android folder), such as other shortcuts, or small Android widgets.
You can also swipe-up/down or double tap on an icon after a different kind of icon modification to open a full screen Android widget of the program that corresponds to that icon.
Since this application can demand more swiping and double-tapping, quick, more-easier-to-reach taps of a “swipe-where-I’m-looking”, or “double-tap-where-I’m-looking” button can help here.
15.6.1.2.4 **Eye Tribe’s corner thumb buttons**
In a presentation at NEXT Berlin, Eye Tribe demonstrates a tablet interface (http://youtu.be/rAlclfa-9uY?t=8m12s) where there are two corner buttons for each thumb.
This shows that combining eye tracking makes an interface require as little as two buttons to do many things.
15.6.2 **Vertical touchscreen + “tap-where-I’m-looking” button**
If you have a vertically propped up tablet with an external keyboard, you could remap a keyboard button to be the “tap-where-I’m-looking” button, and avoid the need to keep reaching out to touch the screen.
15.6.3 **Hands-free interaction: while eating, while using another computer, etc.**
Eye-tracking gives you the ability to have a page automatically scroll down when your eyes reach the bottom of the window (http://youtu.be/2q9DarPET0o?t=22s), or have an e-book automatically turn the page when the gaze reaches the corner of the text (http://youtu.be/2q9DarPET0o?t=1m19s).
These features would be especially handy for computer interaction while cooking and eating.
They could also be for interacting with a vertically set up touch device, or laptop that is more than an arms-length away while doing other stuff on a desktop computer.
15.6.4 **Two-step process for selecting harder-to-press links and characters on non-touch-optimized (or touch-optimized) websites while using mobile**
As mentioned above, touch interfaces have a two-step magnification feature for selecting elements that are smaller and/or too close together.
Here’s a picture of magnification in Chrome on Android: (http://i.stack.imgur.com/95nHA.png).
It occasionally appears when interacting on non-mobile, desktop versions of a website that are crowded with smaller elements.
However, even mobile-friendly and touch-optimized websites can still have small elements.
For example, if you look at Reddit’s mobile site, some buttons may be larger, but text links can still be quite skinny and narrow.
When links are tightly and vertically stacked, a user may have to be more meticulous and tense with their motions.
If you pinch-zoom-out to get a larger view of a website, small elements and links that are close together will get even smaller.
Since a two-step magnification process already exists for touch, the projected elements could be made to be just a bit larger for eye-tracking.
Look at a target element, and then quickly activate a “simulate-a-cautious-placement-of-the-finger-and-long-press” button that could be in the Navigation Bar.
It zooms in on the target element, and projects some of the smaller elements into large elements.
A user could then quickly re-touch the “cautious-placement-and-long-press” button to select it, or briefly dwell on the choice large element to select it.
15.6.4.1 **Eye-tracking two-step process for selecting a range of characters and words**
Just like selecting text links, a task like selecting a range of characters and words can be more arduous than selecting possibly wider website buttons.
For text selection, there could be a process that is similar to the drag-and-drop of the PCEye software that was mentioned above, where a zoom-in is used twice: a magnification is used on what’s picked up, and a magnification is used on where the picked-up item will go.
Look at the first character, and then explicitly activate a text-selection widget that could be in the Navigation Bar.
It zooms in to where you're looking at, and then projects some of the smaller characters that are near the point-of-gaze into large elements.
Look at the larger element that corresponds to the first character, and then you could quickly re-tap the text-selection widget to select the character, or dwell on the larger element to select the character.
It zooms out to let you find the last character, and then you repeat the magnification and selection on the last character to complete the range selection.
This could be a better way of selecting text on a smart phone or tablet.
It could be an alternative to putting possibly thick thumbs on a tiny target character or word, long pressing/holding to begin the range, and then possibly dragging the two selection indicators over long distances (possibly larger than the current viewport) on a small screen with small font.
At the very least, it can be more comfortable.
15.7 **Future: head-mounted display with eye tracking + armband, watch, ring, computer vision system, augmented reality system, etc. for input (helps work with limited space and limited gestures)**
Select your target element with eye tracking on Glass, and then tap one of the few function buttons that could be on another device, or then use a few touchless gestures.
15.7.1 **Watches – small screen real estate works with eye-tracking**
From pictures of smart watches like Galaxy Gear and WiMM, it looks like it has a display with room for 4 large-sized widgets, and I have read that the Pebble watch has 4 buttons.
The Samsung Gear Live, LG G, and Moto 360 watches also have smaller screen sizes.
These smaller screen sizes are acceptable because as mentioned above, incorporating eye tracking means that you can get away with only needing to touch a few function buttons.
15.7.2 **Clothes: Makey Makey and OpenShades**
In Brandyn White’s OpenShades Google Glass eye tracking video, he wires a Makey Makey (turns everyday objects into touchpads) to his shirt, and taps his clothes whenever he wants to do an action.
“New Glass Input Methods: Eye-Tracking, Web Control, Touch-Sensitive Clothing, and Bananas”: https://www.youtube.com/watch?v=QSn6s3DPTSg.
Tapping the left shoulder is one action, and tapping the right shoulder is another.
That’s only two spots.
15.7.3 **Computer vision recognition of forearm (Minuum video) – overlay augmented reality keys on forearm**
The Minuum video demonstrates computer vision on Glass recognizing a user’s forearm, and an augmented reality system overlaying the forearm with keys.
Compared to a physical keyboard, a forearm doesn’t have a lot of space.
15.7.4 **Gestures**
15.7.4.1 **While mobile: armbands, rings**
The amount of gestures that you can make with the Myo armband, Fin (thumb ring), Mycestro (index ring), Nod (ring), and Ring (another ring) might be limited by the number of physically-easy motions that you can do, and having to memorize all of them.
15.7.4.2 **While stationary: Tango, Haptix, Leap Motion Table mode**
15.7.4.2.1 **Haptix (transform any flat surface into a 3-D multitouch surface)**
There is a device called Haptix (https://www.kickstarter.com/projects/ha ... reinvented) that transforms any flat surface into a 3-D multitouch surface.
It’s kind of like the Leap Motion, but it doesn’t focus on air gestures.
Haptix faces downwards to a surface, and doesn’t require infrared, as opposed to the Leap Motion that faces upwards, and requires infrared.
A disadvantage of Haptix compared to a Laser Keyboard is that it doesn’t project anything on the surface to see where to press (overlaying augmented reality buttons on a surface by using computer vision, as shown in the Minuum video, is not available yet.
Although, a company called Ostendo is looking to bring its 2-D and 3-D miniature projectors, and holograms to smart phones and other consumer devices in 2015).
Since you only need a few buttons with eye tracking (e.g. “single-click-where-I’m-looking”, “double-click-where-I’m-looking”, etc.), perhaps you could divide your surface into a few large regions.
It could be 3 x 3, and nine cells for nine actions would still be enough to do a lot.
The areas would be large, so you wouldn’t need a projection to tell you exactly where to press.
(If you’re wondering about losing the physical keyboard letters in this set up, Haptix partnered with the ASETNIOP virtual keyboard.
ASETNIOP is "QWERTY with only 10 keys...
this virtual keyboard enlarges eight common letters -- a, s, e, t, n, i, o, p.
The rest of the alphabet is input by simple "chord" combos of fingers.
Good typists can clock 80 wpm.".
Like Fleksy, ASETNIOP allows you to type without looking.
ASETNIOP + Haptix = fast typing on any surface: https://www.youtube.com/watch?v=hyQo8uCVOXs.
Here is a video of a user typing at over 100 wpm on an iPad with ASETNIOP: https://www.youtube.com/watch?v=fP2TH7TQC8w).
15.7.4.2.2 **Leap Motion Table mode (interact with surfaces) – if using air gestures without eye tracking, many more-physically-demanding gestures to memorize**
A much coveted feature for Leap Motion is “Table mode”, which would allow users of Leap Motion to interact with surfaces, just like Haptix can.
A user was able to get Table mode with Leap Motion by hacking together some transparent glass so that the Leap Motion could see through the surface, and distinguish the hands.
Here’s a video of the user playing Reflex.te, a reaction game that’s usually meant for the mouse: https://www.youtube.com/watch?v=cgGkwGJcB1c.
GameWAVE is an app in Leap Motion’s app store.
“Uwyn’s GameWAVE, available for Mac and Windows, recently debuted in Airspace.
It allows you to completely control keyboard and mouse-based video games, as well as your OS”.
“20 distinct swipe and circle gestures, 6 continuous movement”.
Instead of 20 unique gestures for 20 actions, you could just have one Leap Motion gesture to bring up a menu of 20 on-screen buttons, and then use a “select-what-am-looking-at” gesture to activate one of the virtual buttons.
15.7.4.2.3 **Project Tango and Movidius chip**
Google’s Project Tango device (like a portable Kinect) has the Movidius chip: “New applications include post-capture refocusing, high quality zoom, augmented reality simulation, gaze or gesture-based user interfaces”.
If you were to sit, and plant down Project Tango, it might similarly record your gestures.
(It’s supposedly coming to consumers in 2015).
15.7.5 **Conclusion**
With the limited number of easily-memorizable gestures, or limited space for taps and gestures, these devices would benefit with eye tracking, which could be found in a head-mounted device.
Eye-tracking would reduce the number of inputs needed.
At the same time, the head-mounted device with eye-tracking could use the devices mentioned above for inputting.
It would provide a better way to do lengthy and rapid interactions with devices like Google Glass.
You wouldn’t have to constantly reach at your head, and lift your arms for long periods of time.
15.7.6 **Oculus Rift + eye tracking**
15.7.6.1 **Oculus Rift with the Haytham gaze tracker**
Researchers from ITU (where I think Eye Tribe comes from) added eye-tracking to the Oculus Rift with the Haytham gaze tracker, and OpenCV.
(Haytham is an open source gaze tracker from the IT University of Copenhagenlike, just like the ITU Gaze Tracker).
http://www.itu.dk/research/eye/?page_id=671
15.7.6.2 **Selecting and manipulating virtual objects with more speed and comfort, and less hand positions – e.g. Oculus Rift with camera for augmented reality**
AR-Rift: Stereo camera rig and augmented reality showcase: Virtual screens:
http://youtu.be/Bc_TCLoH2CA?t=12m25s.
In this video, a person attaches cameras to the Oculus Rift to see the real world, and augments reality with virtual objects.
At 12:25, the person moves virtual screens around by repeatedly lifting his arms, and activates menu items by making virtual objects touch each other.
With eye-tracking, selecting items in a floating virtual menu, and other objects in the world could be quick, just like with regular 2-D on-screen buttons.
If air gestures are used, like in the video, then eye tracking could also help replace some of the gestures, and thus mitigate horizontal-arm ergonomic problems.
He’s also able to switch from augmented reality to full virtual reality.
In this mode, there is no easy way to see your physical arms interacting with a physical keyboard.
Eye-tracking can minimize the amount of buttons and hand positions that are needed, so not being able to see and use all the buttons of a physical keyboard are less of a problem.
15.7.6.3 **Navigating 20 virtual stock trading screens in Oculus Rift**
Traders can have 12 or more monitors for prices, news, charts, analytics, financial data, alerts, messages, etc..
Bloomberg LP (makes financial software) built a virtual prototype of their data terminal for the Oculus Rift.
Here is the image of their prototype with 20 virtual screens: http://i.imgur.com/Z9atPdh.png
You may have no problem with moving a mouse-controlled cursor across 1 screen, but what happens if you want to one day have 4+ virtual screens? Moving a cursor with a mouse across 4+ screens would probably be more difficult, so eye-tracking teleportation would really help here.
15.7.6.4 **Oculus Rift + eye tracking for traversal - non-gamers – “go-to-where-I’m-looking-at” e.g. eye-tracking wheelchair by researchers at Imperial College London **
Also, integrating an eye tracker into the Oculus Rift could really help with traversing around.
Look at where you want to go, and then press a “go-to-where-I’m-looking-at” button to auto-move to that location.
Eye-tracking wheelchair – "move around simply by looking to where they wish to travel" - also for robots and drones
“Scientists in London have developed an algorithm-based decoder system that enables wheelchair users to move around simply by looking to where they wish to travel.
The researchers at Imperial College London say the system is inexpensive and easy to use and could transform the lives of people who are unable to use their limbs”.
“You could use it maybe one day to drive your car, you could use it to operate a robot, you may be able to use it to fly planes or drones or spaceships with this type of technology." https://www.youtube.com/watch?v=ZckE7FMWrw8.
The roller coaster demo on the Oculus Rift shows that people of all ages can enjoy the device.
And there was that video of the 90-year-old grandmother (https://www.youtube.com/watch?v=pAC5SeNH8jw) being absolutely in love with the Tuscany demo.
A family member was controlling her character in the virtual world.
If she wanted to move by herself, the Virtuix Omni or Cyberith Virtualizer treadmills, or a game controller would probably be too difficult to use.
16 **Conclusion**
Eye-tracking can clearly help with accessibility, but almost everyone would see their productivity increase by integrating eye tracking.
17 **Communities**
http://www.tendinosis.org/forum
http://www.reddit.com/r/EyeTracking
Disability & Assistive Technology Today: https://plus.google.com/u/0/communities ... 2302632679
http://www.reddit.com/r/RSI
https://www.facebook.com/RSISupportGroup
http://www.sorehand.org/
Tendinosis Google+ Community: https://plus.google.com/u/0/communities ... 5972411030
TalkingEyes: Eye-tracking and ALS Google+ Community: https://plus.google.com/u/0/communities ... 5695270907
http://theeyetribe.com/forum/
(There is a computer control section, and an accessibility section)
http://developer.tobii.com/community-forums/
Voice Coder: http://groups.yahoo.com/neo/groups/VoiceCoder/info
Palaver speech recognition: https://plus.google.com/communities/117 ... 2112738135
Eye Tracking: https://plus.google.com/u/0/communities ... 3443831724
Speech recognition community: https://plus.google.com/u/0/communities ... 4072731857
Computer Accessibility: https://plus.google.com/u/0/communities ... 7935418793
Web Accessibility: https://plus.google.com/u/0/communities ... 5224690148
LiSpeak - Linux Voice Command System: https://plus.google.com/communities/115 ... 9044979120
#ux #design #ui #usability #tendinitis #tendonitis #repetitivestraininjury #rsi #repetitivestrain #a11y #Google #palaver #AAC #stroke #strokerecovery #spinalcordinjury #quadriplegia #googleglass #cerebralpalsy #hci #humancomputerinteraction
