Hand Gesture Recognition and Virtual Game Control Based on 3D Accelerometer and EMG Sensors

Comments

Jill

Summary

In Hand Gesture Recognition and Virtual Game Control Based on 3D Accelerometer and EMG Sensors, the researchers propose a new form of input for virtual games and other applications using the motion of the hand. While previous forms of input using the hand would use only the accelerometer or the EMG sensors, this project takes both forms of input and combines them together for a more accurate and reproducible input for games.

The accelerometer is used to determine the direction of large scale gestures, and is not accustomed to use with small gestures. It can also sense rotation as well as large movement. The EMG sensor, or electromagnetic sensor, senses the electrical currents that muscles make as they move within the body, thus giving the system the knowledge of when a user has moved. The problem with EMG is that it is hard to determine the exact movement that was made, but it can sense when even the tiniest of movements occurs. The accelerometer (ACC) and the EMG inputs are processed in real-time and converted to recognized gestures and inputs to control the game.

To test this new innovation, the researchers built a virtual Rubik's Cube game, and allowed those in the study to manipulate the cube in real time in 3D space using the gesture input system. After being outfitted with EMG sensors along their wrist, and an accelerometer on the top of their hand, they were able to manipulate the cube, and the system was able to recognize their gestures, 91.7% of the time. Future work would include making the system more robust, extending applications to mobile device control, and the design of smaller sensors for inclusion in a glove, for example.

Discussion

I thought this seemed pretty interesting because unlike newer game control input systems (like Project Natal, by Microsoft, which uses a camera to detect user movement, gestures, and motions, without a controller), they are getting the exact movement from the user. It seemed similar to the accelerometer that Nintendo uses in their Wii controllers, but it would have a higher sensitivity as it actually senses the electrical impulses to the muscles in the body. My question would be if this system was actually more useful than the Wii motion controller system, or was just a novel way to control input using the whole hand and fingers instead of using an external controller.

Parakeet: A Continuous Speech Recognition System for Mobile Touch-Screen Devices

Comments

Nic

Summary

In Parakeet: A Continuous Speech Recognition System for Mobile Touch-Screen Devices, the authors present a new voice-recognition application for mobile touch-screen devices to allow the user to input text into the device. Using the voice recognition, the user can read the text they want to enter, along with using a simple touch-based interface for making corrections and changing words. The features are as follows:

• Voice recognition software converts the spoken word into text.
• When the recognition is done, a beep lets the user know.
• The user can then scan the sentence and easily make changes to words. The words in the sentence are ordered at the top of the screen, and below in columns are examples of other possible words that sound like the word spoken, along with a delete button at the bottom. The user can either replace the word that was recognized, or delete the word altogether.
• If the software is completely unable to recognize the word, it will default to a keyboard entry with predictive text as seen in current mobile devices.

After doing a user study, they found that users were able to enter up to 18 words per minute while sitting still, and 13 words per minute while walking around. While it may seem small, it is easily comparable to the 10 - 20 words per minute that each user had while entering text using the traditional T9 text prediction system (using a keyboard).

Discussion

If implemented correctly and included on all new cell phones, this entry system would revolutionize the text messaging system. I know lots of people that are older than me that don't see the point of text messaging...but if they could just speak what they want to send, just like they're having a conversation over the phone, and send that as a text message, then they could be more comfortable with the system and send messages to users that use traditional typing methods. But, the words per minute would have to be a lot higher before I would use it because I already text almost as fast as I type on the computer. There would be a lot of idiots walking around talking into their phone like the people with bluetooth headsets though....

Emotional Design

Comments

David

Summary

In Norman's first book, The Design of Everyday Things, he stressed the necessity of usability in products of all forms. With a first glance at a device or product, the user should instantly know how the product is supposed to be used, without question; they should not have to read a lengthy manual to understand the inner workings of the product. In Emotional Design, Norman departs from this notion partially and instead focuses on the emotional appeal of objects. His main point (as quoted by William Morris) is as follows:

Have nothing in your houses that you do not know to be useful, or do not believe to be beautiful.

Norman's description of emotional design includes the following points:

• Users report that they are more happy with objects that are aesthetically pleasing, even if they do nothing or do not work as they are supposed to.
• Appeal to the three levels of emotional response:

Visceral: the basic appearance and visual / aural / haptic reponses to the object

Behavioral: the pleasure and effectiveness of use (does it work correctly?)

Reflective: how the object appeals to the user in a high-level, like self-image, personal

reflection, or past memories

• Make the object have its own unique personality.
• The object should be not only beautiful but 'fun' as well. Beauty, fun, and pleasure all work together to form a sense of enjoyment of the product or object.
• Products should go beyond the initial expectations, produce an instinctive reactionary response, and deliver surprising novelty.
• Communication between product and user should make the user feel important and respected.
• Machines or robots that are created should have emotional responses so as to react with humans appropriately.
• Allow the user to have a personal relationship with the product, either through customization, years of use, or personal touches.

In all, Norman wants us to take his past argument (that all products should be usable, even if they're ugly) and change the weights he had given before. He now believes products should be beautiful and emotionally pleasing as well as be usable and work correctly, for successful products have both of these qualities.

Discussion

I thought this book was a nice calm change from the style of The Design of Everyday Things. In that previous book, he seemed to be angry at everyone, continually telling everyone how their products are wrong because they are hard to understand or use, and that they focus too much on aesthetics instead of usability. What was nice about Emotional Design was that he instead focuses on instructing designers on great ways to make their objects and products more emotionally appealing, and gave examples of great emotional designs, instead of criticizing everyone and giving examples of poorly designed products. It was a much more satisfying read than the past.

I completely agreed with Norman on the necessity of designing with emotional and aesthetic appeal. Apple laptops and PCs are not successful because they are more powerful than any other PC (because they're not), but because they are beautiful to look at and to use. I have always wanted an Apple computer (though I can't afford one) even though I know that as a computer programmer, I would not have as many resources for developing applications and doing my daily work as I would on a PC. This shows that the emotional design of the computers can far surpass the functional requirements. I know that I definitely buy products based on aesthetic qualities as much as functional qualities.

Data Driven Exploration of Musical Chord Sequences

Comments

Chris

Summary

In Data Driven Exploration of Musical Chord Sequences, Basu, Nichols, and Morris present a new way of classifying different genres of music based on inherent chord progression styles, and an interface for both music novices and experts alike to modify styles of chord progressions to form their own unique genre of music, or blend two artists or styles together.

The polygon slider component of the interface consists of a polygon in which each point represents a style of music or artist (and all of them are user-defined). The current point in the polygon (inside the polygon) can be defined by the user as well, and can be slid around inside the widget in real-time. Using a distance algorithm, the system finds the current distance of the current point from each point of the polygon (each style), and assigns a percentage to each quality being compared. The system will then take properties of the chord progressions from each of the styles (based on the percentage given) and create a new chord progression style that the user has defined. In this way, users can create their own style of music (as chord progressions are inherent to the differences in musical styles) and put the progression behind a melody to create music. The possible choices for inputs include styles such as Country, Rock, Classical, or Pop, and artists such as AC/DC, the Beatles, and Lenny Kravitz. Chord progression styles for each genre of music are obtained by averaging chord values (and sound frequencies) from a large database of songs in that genre. Progressions for artists are averaged in a similar way from a database of songs by that artist.

Discussion

Being a musician, I thought it was really cool that genres of music could be exactly specified by types of chord progressions. I mean, everyone knows that some types of music have chord styles that are always seen, for example the Blues scale and chords in Jazz and Blues, and minor chords in Alternative Rock. But to think that a specific chord progression such as C->F->G constitutes a Country song is pretty interesting. Now I can tell all my family and friends that I hate Country music because I was born to hate their exact chord progressions! Not my fault, I was just born that way. And the interface to be able to create your own chords (even as a novice) is great. Many people wish they knew how to play instruments, and seeing how it is an extremely hard process, this interface would give many people a way to be creative without taking the time and effort to learn an instrument (or compose music by writing scores of notes).

TrailBlazer: Enabling Blind Users to Blaze Trails Through the Web

Comments

Patrick

Summary

TrailBlazer: Enabling Blind Users to Blaze Trails Through the Web, by Bigham, Lau, and Nichols, details a new way for blind users of computers to navigate the web more quickly. Current accessibility programs for the blind consist of only a linear listing of all the text and options on the screen, whether it is listed through voice or through a refreshable Braille display. Trailblazer allows the user to access scripts written for the website (a script consists of a possible path in the website, and websites with scripts written will include the most important and popular actions on the website) and easily jump to the options they need, significantly reducing the time it takes to navigate a website. Scripts that are already written for websites are listed in a large database called "CoScriptor", but Trailblazer also allows users to create their own scripts. These scripts are a kind of instruction for the system, detailing actions the user wants to make on the website in a pseudocode language. Each time a new user creates a script, that script is added to the CoScriptor database, so as users use TrailBlazer, the database continues to grow and more websites are accessible by the blind.

Additionally, the system saves past user's actions and uses them to predict the next most likely action (picks the script most likely to be used). In this way, the system continues to improve with each time the system is used. After doing an extensive user study with blind users, all of the users said that TrailBlazer significantly reduced the amount of time that it took to navigate a website, but were not as happy with the steep learning curve that they had to overcome to use the software. Another problem with TrailBlazer was its inability to navigate dynamic web content such as Flash. As most of that content is not textual, and is contained in its own application, then it cannot navigate buttons, etc. But in general, most of the users were very happy with the interface.

Discussion

I thought that if this technology made it into all of the current web browsers, and it was accessible to all blind users, then it would be super successful. I can't imagine how long and frustrating it is to do even the most simple task as checking your email when you're blind. So, having a system that records the actions you do most often and predicts (with 75% accuracy) the action that you want to do, then you could check email, etc. much faster. The main problem would be whenever you want to go to a new website, or navigate a flash interface, and the system works much more slowly (or not at all, in the case of the flash interface).

The Inmates are Running the Asylum (Chapters 1 - 7)

Comments

Kerry

John

Summary

The Inmates are Running the Asylum, by Alan Cooper, is a book that promotes "interaction design" for technological devices and software so that the user is not confused or made to feel stupid by the complexity of the technology. He makes a few important points (just Chapters 1 - 7):

1. Any device combined with a computer is still a computer; the complexity and the anti-human behavior of the computer will dominate.
2. Because of this, we need to promote the "partnering of interaction design with programming". Interaction design is different than interface design; it is not just the design of an interface that is used to communicate with the computer, but the way that the user and computer interact with each other.
3. Cognitive Friction is the resistance of humans to understand the complex systems of rules that are always changing that computers and technology follows. Interaction design aims to reduce the cognitive friction so that everyday users feel more comfortable.
4. A clear distinction is made between user and programmer. The programmer is more apt to design the system to make their job of programming easier and in a way that they feel is easy to interact with. What they fail to see is that users do not think the way that they do, and the programmers are incapable of putting themselves in the shoes of the user.
5. Interaction design and programming has to be done by different people; it is nearly impossible to both design for the irrational and emotion world of humans and simultaneously design for the deterministic and rational world of computers.

In these first seven chapters, Cooper presented the problem of software that is too hard to use, and started to explain the solution of interaction design. His main point was that interaction design is necessary before programming and will save lots of money and time in software that is accepted by users and is successful in the real world. Programmers need to give up control so that their ideas and efforts in products will not be wasted.

Discussion

I thought Cooper presented some interesting ideas in the first seven chapters of this book. I agree with his ideas that interaction design should be done before programming; without a clear vision of how the user will react to and interact with the software, its hard to keep low-level implementation details from filtering up into the interface and confusing the user. But, I don't feel that it is mostly the programmer's fault, and that all programmers have their heads stuck staring at their monitors so that they can't understand the interaction of everyday users. I think the problem sits with management, marketing, and economics more. The managers demand a certain functionality as soon as possible, and as the biggest time expense is the base programming, the engineers have to hit the ground running. If given time (and if more management realized that design is not a waste of time or budget but instead a way to make sure your product is received well by the masses), then programmers could easily help make design decisions and produce a program that users like. I just feel that the time constraints put functionality first instead of usability. Interaction designers are important though, even after the main design phase, because they can work concurrently with the programmers to make sure that the project is going along at the correct rate and it is still usable. Now that Cooper has laid down the problem, I am looking forward to the rest of the book to see what his solution is.

Autism Online: A Comparison of Word Usage in Bloggers With and Without Autism Spectrum Disorders

Comments

Subodh

Summary

In Autism Online: A Comparison of Word Usage in Bloggers With and Without Autism Spectrum Disorders, the researchers present a study in which they analyze the language in blogs written by those with Autism Spectrum Disorders and the difference between blogs written by people without mental disorders. People with Autism and other related disorders primarily have problems in social interaction, specifically with communication issues and in face-to-face interaction. The web is a place where those with disorders can feel comfortable and free of the pressures of real-time personal interaction, and spend time thinking how they want to communicate. It gives them a place to be themselves and interact without fear of being judged. The researchers wanted to see if there was a significant communication difference in text of the internet between those with Autism and those without disorders, similar to the physical communication difference in personal interaction. After analyzing 50 blogs from those who have Autism (along with other requirements such as age, etc.) and 50 from those that did not have the disorder, the researchers found that there was not a large difference between the blogs in any categories (social, melancholy, ranty, work, metaphysical). The only difference was that the variability between blogs in the "social" category was much higher, which can probably be explained by the lack of social interest that is common in those with Autism Spectrum Disorders. The conclusion was that because of the lack of difference in communication patterns, blogs and the web in general are an appropriate place for those with disorders to interact socially and feel comfortable, mostly because the requirement for fast mental processing of social cues is taken away.

Discussion

It seemed really great to me that those who suffer with social and communication disorders can have a place to interact with others without feeling pressure of failure. It definitely means that those have a place to vent their feelings of anger and frustration that usually come with autism, and can possibly be a way for those with the disorder to interact with the public and possibly live a normal life, having jobs at home that required internet communication, and working through email and blogging. Hopefully the interaction online that they have can directly help them in their physical social interactions as well.

(Perceived) Interactivity: Does Interactivity Increase Enjoyment and Creative Identity in Artistic Spaces?

Comments
William

Summary
In (Perceived) Interactivity: Does Interactivity Increase Enjoyment and Creative Identity in Artistic Spaces?, by researchers at Cornell University, a study was presented to see the how the interactivity in an art exhibit and the relative enjoyment of the viewer are related, and if the interactivity makes a person see themselves as more creative. Interactive art, in general, is an exhibit in which the viewer's actions can affect the way things are displayed, played audibly, or moved around, and can give the user the sense that they are in control of the art, and creating things for themselves. The researchers performed an experiment that compared enjoyment of an artistic experience between groups that interacted with the system and groups that did not. The study took place in a small music studio with a system that was set up to receive interactions from viewers using a Wii remote. One group (the no interaction group) only listened to a pre-recorded session of music, and the other group (the interaction group) used the Wii remotes to change the way the music and sound effects were played. After the study, the researchers asked each participant: Did you enjoy the exhibit? Was the exhibit interactive? Did you feel more creative after interacting with this exhibit?

Their basic findings were that interactive art is more enjoyable for the viewer than art in which you have no control. But, the enjoyment of the exhibit depended on the perceived interactivity of the user, as many of the users in the no-interaction group rated the exhibit as being interactive (they were not told any information about how the music was produced). In this way, their first hypothesis was confirmed: interactive exhibits increase the enjoyment of the user. But, their second hypothesis was not confirmed: users that interact with an art exhibit do not have any changes in the perceived creativity of themselves.


Discussion
I thought this was an interesting study because the results seemed a little obvious. The exhibits at art museums, or museums in general, are always more fun if there are controls and things you can do to alter what has been shown. People in general are very hands on. But, I feel like interaction in a pure "art" exhibit is a little useless. Art museums are usually meant to showcase the creativity and vision of the artist, and if the audience can interact and change this art, then who is to say that it is the artist's creation anymore? I guess the idea of the interactive design is original to the artist, but I don't see this interactivity replacing traditional sculpting, painting, or music because those forms of expression are meant to let the artist show their feelings and ideas.

Learning from IKEA Hacking: “Iʼm Not One to Decoupage a Tabletop and Call It a Day.”

Comments

Nic

Tenen

Summary

Learning from IKEA Hacking:

“Iʼm Not One to Decoupage a Tabletop and Call It a Day” is about a culture of Do-It-Yourself (DIY) supporters, specifically focused on "IKEA Hacking", where people take existing IKEA products and furniture and modify them to fit their needs. The process is all about creativity and expression: a way for the user to change the product to be more personal for them instead of a mass-manufactured piece of furniture that half the nation already owns. Many liken the process to code hacking: you change a piece of code or exploit its weaknesses to achieve some goal that you may have. The IKEA hacker takes the weaknesses in these products and forms their own product that is theirs alone.

The article was specifically about how online websites and services strengthen the activities of the DIY community and more specifically, IKEA hackers. These people receive recognition online for their work, and feel better about themselves and their creativity. Many of the creations are personal directly to the creator; for example, one man says his creations inspire and remind him of his children. The online DIY videos on websites such as Instructables.com continue to strengthen the community and allow access points for those new to the practice. The researchers concluded that the internet will get more closely integrated with the material world as things go by, and the online activities of the IKEA hackers supports this claim.

Discussion

I thought this was an interesting idea: take something that has been mass-manufactured and represents the loss of individuality and exploit it's weaknesses (and easily-constructed parts) and form something new that is not only functional for you but also an expression of your creativity. I thought it was cool when they made the analogy to "an anarchic event"; that in some way hacking this IKEA furniture made the hacker feel like they were "sticking it to the man" or something. It was definitely interesting, and I'll give props to anyone that would take a normal chair and turn it into a chair at a gynecologist's office (below, the GYNEA chair).

Team Analytics: Understanding Teams in the Global Workplace

Comments

Kerry

William

Summary

In Team Analytics: Understanding Teams in the Global Workspace, the authors address the issue of communication problems between members of a distributed team. When team members may work at other companies, or other locations for the same company, it is often hard to form of visual image of who you are talking to, and keep time zones, etiquette, contact information, and calendars all in your head. Existing systems to look up directory information only allow for the viewing of one directory entry at a time, and so you cannot look at your whole team at once. It is also impossible to find the team's structure from this information. Team Analytics is an online web application that allows all members of a distributed team, no matter what location, to connect and view information about the team. The application has a few important parts:

1. A picture gallery of all team members, so each member can know who they are talking to.
2. An organization chart that shows visually the relationships between each of the members. This information can help a member determine what the communication style is within the group, whether formal or informal.
3. An attribute pie chart that shows the relative numbers of people in each division within a group.
4. A "timezone pain" chart that shows the location of each person in the group, and what times are good times to call another group member based on the current time in their location.
5. A "bizcard" section, which shows more personal information about each member, along with contact information and their picture.

After doing user studies with the product, which was tested in a large global corporation, users said it really helped them visualize who they were speaking to. It helped them coordinate conference calls between members in different countries, and the addition of the pictures allowed group members to keep track of each person they are talking to. In all, the program was a success.

Discussion

I thought this seemed like a pretty good program that hopefully will be on the web soon for companies to use. I know that I have a visual memory, and I remember faces very well. If I had to be in constant contact with many group members overseas, I would have a hard time telling all the voices apart. Having the pictures and the information about each person would definitely hep me form mental pictures of each member, and so this would help me out a lot! And the timezone feature is really important. I had never thought about how hard it would be to have a conference call between members in many different countries without one person being on the phone when they're usually asleep! Hopefully this tool would help them coordinate the call so everyone does it at a comfortable time.

An Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music Display and a Haptic Chair

Comments

coming soon...

Summary

In An Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music Display and a Haptic Chair, the researchers explained a new way for the deaf to have a fulfilling musical experience. Music is not only established through the audible sounds; it also manifests itself through vibration of the floor and walls, visual effects, and the movements of the artists (in live settings). While the deaf cannot hear the actual sounds of the music, they look to other ways to enjoy music. The researchers looked to develop a system in which the deaf can enjoy the vibration and visual phenomenon of music and get as close to actually hearing the music as they can. They found that about half of deaf people have never had any sort of musical experience before, and so they thought to introduce these types of people to a new form of entertainment and musical fulfillment.

The system consists of two parts: a visual system that translates pitch, tone, tempo, and other qualities into visual pictures, and a haptic chair that vibrates with the beat and intensity of the music. The visual system took the MIDI representation of the music, and converted it to visual form using XML and Flash (ActionScript 3.0). Different instruments were different colors, and as they played notes, different visual cues appeared on the screen. The haptic chair was a chair from IKEA with 2 different contact speakers attached to it. The user would sit in the chair with almost all of their body in contact with it, and as the music played, they would feel the vibrations throughout the chair.

After doing user studies with different types of music, most of the deaf participants said they felt no difference than actually listening to the music (how they imagined it). Many said that if they could hook their own music up to it, change the visual styles of the display, and use a hearing aid in conjunction, they would very much like to have the system for their home.

Discussion

I thought this was really interesting cause I have always felt there is so much emotion in music. Lots of emotion comes from the performer: their facial expressions, their body movements, the reaction of the audience....all of that is visual. And having a visualizer that shows the frequencies and notes, along with a chair that vibrates in place? That seems pretty close, or as least as close as you can get, to actually listening to music. This technology is really promising if they can develop it and manufacture it cheaply, and I wouldn't be surprised if it was on the market soon, targeted not only towards the deaf, but also towards those that can hear but want to experience their music in a different way.

PenLight: Combining a Mobile Projector and a Digital Pen for Dynamic Visual Overlay

Comments

coming soon...

Summary

PenLight: Combining a Mobile Projector and a Digital Pen for Dynamic Visual Overlay, a collaboration between Autodesk Research, Cornell University, and the University of Maryland, describes a digital pen with lots of extra functionality. The researchers were looking to resolve one of the important design points made by Norman in The Design of Everyday Things: the importance of feedback. Current digital pens can capture your writing using motion and tiny cameras when written on special paper, and later converted to text or saved on the computer for easy use. But, besides the physical ink that is produced by the pen, there is little useful visual feedback about what you have written. There is almost no feedback at all (except for some audio) for navigating through the menus of the pen, which can be frustrating when you're trying to make sure your work gets saved. The researchers came up with a prototype idea for the "PenLight", which would use a mobile projector (currently being added to cell phones and small devices) to project useful images about the menu and about your drawings and writings on the paper. The small projector would give great feedback so you always knew what you were doing.

The main ideas for the PenLight are:

1. Three different layers in 3D space that can be written on: the surface, hovering right above the page, and higher (the spatial layer). Different menus can be shown for the different layers, and the distance from the pen to the paper is used to determine the current layer that is being viewed.
2. The user can immediately see the menu options and choose them with appropriate visual feedback.
3. Moving the pen around vertically and horizontally in 2D space parallel with the plane of the paper will show you different parts of the image stored in the pen (as if viewing through a movable window).
4. Easy input with lots of applications.

The prototype they made consisted of an existing digital pen, a magnetic 3D tracking device attached to the pen, and (as mobile projectors are yet to be completely developed) an overhead projector simulating the projection from the pen. This system was able to completely emulate the PenLight's features as explained above.

The main application that the researchers gave was to help the architectural design field, as they still primarily work with physical schematics and drawings. Features for this program included:

1. Virtual Ink: you can write on layers within the pen's memory (directly overlaid on the paper) virtually without writing on the paper physically.
2. You can trace over an object on the paper and then move the object to other locations.
3. The pen can project a virtual guide for tracing to help with drawings.
4. The pen can overlay different content depending on the object on the paper; for example, it could overlay the electrical wires and water pipes over the drawing of a building.
5. You can overlay computations of distance, etc. on top of the schematics.
6. You can initiate a 3D walkthrough of the building by drawing a 2D path.
7. Copy/Paste from the physical drawing.
8. Searching through the physical drawing for an object or number.
9. Displaying the 3D building and cutting out 2D cross sections.

The current prototype that they creating can already do all of the above and more.

Discussion

I thought this was one of the coolest papers I have ever read. The ability to see the menus projected on the paper in front of you is already pretty cool, but also being able to draw a walkthrough on the paper and have it go through it in 3D? That's pretty amazing. Just the ability to leave extra notes about the content in separate layers without marking all over the paper is an amazing feature, and being able to wirelessly sync your new annotation to another architect's desk is great. I think this device (if they can develop it soon, and the projector, etc. can be developed and mounted on a pen) will reinvent the modeling and computational architecture field, and make designing buildings a much easier task.