|Name||Write one paragraph with your thoughts regarding the possible rewards and perils of sensory substitution and augmentation.||Write one paragraph with some ideas for sensory substitution and augmentation technologies for some target user group (e.g. average Joe, children, elderly, disabled, or other).|
|Sensory augmentation or substitution has the capacity to impact not only the disabled population, but people in general. By substituting particular sensor modalities with alternative ones, people with disabilities can access forms of sensory input that they would otherwise be unable to. Many systems that explore different variations of transforming visual, tactical and auditory signals have been developed with promising results. While these technologies may never reach the ability to be complete substitutions of the original sensory ability, they can provide a new mechanism for interaction with environments. Moving beyond the disabled population, sensory augmentation can play an interesting role in changing our interaction with our surroundings. Technologies—like the vibration belt to help with direction intuition—can be developed to enable people to develop new “senses” to assist in our goals. Attachable sensory technologies could be used to sense different forms of information depending on the goal, job, or context. In fact, human beings could access all sorts of otherwise “hidden” information in their surroundings for a more immersive experience. However, as with all technologies, there are tradeoffs. The form that sensory augmentation takes can range in its level of invasiveness—ranging from technologies that are easily wearable to implants requiring surgery. Depending on the manifestation of the technologies, the regulations surrounding their access are important to consider. Can people choose to get implants on will? Or do they need proof of disability? Who controls access? Unequal access to technologies that enable people to be more successful at particular jobs could create unequal access to employment and generate divides in the population. In addition, sensory augmentation, if used excessively, could change the development of particular life skills in people. For example, with examples like the belt that uses vibration to show true-north, people may not be able to navigate without the existence of these subtle nudges. Similarly, this could apply to other technologies wherein people are unable to interact with their environment effectively in the absence of these augmentations.||Vibration provides an interesting mechanism for sharing tactical feelings and emotions. When we converse with someone digitally—phone, computer etc.—we rely primarily on visual and auditory sensory inputs and outputs. It would be interesting to facilitate a new form of communication in these situations where visual and auditory information from one party and their surroundings is transmitted as vibrations to the other to create a “shared experience” and transmit these traditional sensory inputs in a new form. Smell could also be used as a mechanism for transmitting the shared experience. Another area of more obvious development for sensory substitution is for those lacking the ability to hear or see. Vibrations on the body could be used as a mechanism to transmit information about what is visible or being heard in the environment and indicators like proximity to objects. Additionally, music could be used to transmit information about visual cues.|
|I personally think that the biggest problem with sensory substitution as it stands today is the fact that many of the existing solutions claim the establishment of a new perceptual channel. The reading, “Reading the world through the skin and ears: a new perspective on sensory substitution”, addresses this misconception neatly by framing sensory substitution as an act of learning similar to learning how to read. The “wishful thinking” that current solutions actually create a new perceptual experience that is decoupled from existing perceptual channels is naive. Nevertheless, I believe that moving forward, sensory augmentation/substitution would allow us to experience the world in different way. I think one of the most powerful ideas of this concept is the notion of empathy. Sensory substitution/augmentation/alteration can allow us to experience reality from the perspective of another being. For instance, it would allow us to see the world from the perspective of a blind person. Conversely, a blind person could experience concepts that are beyond their physical constraints, such as colors or shapes. One of the biggest dangers of sensory substitution, however, is the possible exploitation or alteration of perception. We live in an era where technology can and is often easily exploited. Because devices that create new perceptual experiences are often by nature, really close to our body (perhaps even in the future connected to it), it’s easy to imagine scenarios where these could be exploited or have undesired side effects. For instance, in the Black Mirror episode, “Playtest”, an unsuspecting tourist is subjected to a device that connects with his senses and enables him to see, hear, and feel virtual entities in his environments. Some of these new perceptual experiences can be too shocking for the person, or even make him/her disconnected from reality. This is particularly dangerous when you think of someone altering another person’s perception without their knowledge.||As I mentioned in my previous response, I think it would be interesting to develop sensory substitution devices that would allow us to perceive the world from the perspective of others. I propose a set of sensory augmentation devices targeted towards the development of empathy towards patients of psychological illnesses. For instance, using mixed reality headsets, people might experience the world as a schizophrenic, hearing voices or seeing visions throughout their day, and experiencing a disjoint reality to understand the fragility of their minds. Another example would be to experience the world as a person with eating disorders, experiencing the perception of altered self. Looking in the mirror would alter the user’s physique, so to speak allowing the user to understand the ordeal people with these disabilities go through. More interestingly, these applications that alter perception/senses, can be used in the inverse fashion to help patients deal with their disabilities. For example, the application meant to allow students to understand the ordeal of schizophrenics, can be used as a tool to train them to distinguish between reality and hallucination. Similarly, for people with eating disorders, it could be used to alter their senses radically to balance out the erroneous perception of their physique.|
|The possible rewards of sensory substitution and augmentation are quite high, I think. People might be limited by lack of one or more of their senses: sensory substitution allows for people to experience the world accurately despite lacking something. The examples that come to mind are technologies that allow for people to “see” through “audio” or “touch”. Technologies that substitute one of the main five senses for lack of another one of the main five senses. Letting people “touch” through “seeing”, letting people “hear” through “touching”. Examples such as Bach-y-Rita's TVSS which was referenced in the readings quite a bit. These technologies are positive in the fact that they are helping people experience the world in many more ways than if they did not have the use of a technology. They are giving people the power to sense the world. At the same time, I think a possible peril of such systems is that they are just “substitutions”. They are not “the real thing”. It might be dangerous for people to think that a certain sensory substitution system is perfect or an exact representation of another sense. If people think that their technology is a perfect re-creation of another sense, then they might think they are aware clearly of the world around them when the resolution of their device might actually be low. People might hurt themselves by believing themselves to be getting the whole picture of the world around them when they’re actually just getting a fuzzy image. To explain it in another way, let me use the concept of glasses. I can see the world without my glasses, but it’s a fuzzy view and I may hurt myself by misjudging an object or distance if I think that my fuzzy image is a perfect representation of the world around me. Like that, I think people can be in danger if they believe that sensory substitution systems are perfect representations of the world around them. Also, for people who have been wearing a certain device for a while and find themselves without it, I feel like a possible peril is that they will have periods of trying to adapt back. Actually, in general, people will have to spend time in order to adjust to the system and understand it and also to wean off it. I think this adjustment time can be dangerous in situations where people might need an immediate usage of a sensory substitution device. ||For people who have trouble breathing: This is rather whimsical, but I was drinking a Snapple Lemon Tea and saw a fact on the bottom of the bottle cap that said “Adult humans are the only mammal that can’t breathe and swallow at the same time”. That made me think of the times I have been sick in the past. Usually, if my nose is super clogged, I’ll breathe through my mouth, as I’m sure many other people do. Sometimes, if I’m eating a lot and I’m sick and can’t breathe, I’ll feel panicked or worried. The lack of being able to use my nose and breathing and “tasting the air” through my nose is what causes me to worry. This got me thinking, I wonder if there’s some sort of sensory substitution to allow a person feel calm in such a situation where they can’t breathe. Not even when they are sick, but just in situations where they feel like they can’t breathe. I wonder if having your body feel the sensation of wind or having your body simulate the experience of inhaling/exhaling would be enough to make a person calm and take a breath and then continue eating or chewing or doing whatever they were. I think this is curious because it’s using your body as a sensory system that’s reacting to the feeling of a lack of a sense (the feeling of lack of your sense of smell) and then doing an action such as mimicking your body when it is inhaling/exhaling to help you feel better. Going a little more extreme, I wonder if we can augment our senses in such a way that we can have flaps on our skin that open and allow us to breathe (inhale/exhale) and taste the air around us. If we are choking or sick or cannot breathe because of some reason, then the flaps on our skin can open and act almost like the gills of fish. Like respiratory organs. That would be quite interesting, I think, to reroute our internal systems to allow us to breathe through our skin. And, of course, it would be interesting if we could also “taste” and “smell” through our skin. Then, in cases where we cannot breathe or use our nose or mouth because we are sick or not feeling well, our skin can help us understand the world around us.|
|Inconsistent signals of the augmented sensory leeds to variable performance. Machinery glitches maybe devastating when people already adapted to the new sensory experience. |
How many information do we need, how much should be augmented/substituted. How to leverage between too much information that becomes distractive and the minimal information we actually need.
|How sensory augmentation could be used for females' perception on safety. Walking alone at night in a quite street, a third eye looking to the back and corner (like a 360 camera) may help. A sense of potential danger or the augmented ability transcending physical body limitation to confront the danger may help.|
|Sensory substitutions like visual-to-auditory or visual-to-tactile are not really creating a new modality of sensory, their use are, in general, more focused to replace o compensate a deficiency. For those technologies I can’t imagine a negative scenario of its application. However when it comes to technologies that really create a new sensory modality, and if this new sensory became too relevant in our lives that it became often between certain communities, we may have a problem of environments designed to have features only perceptible by particular individuals with an additional technology on their bodies.||Currently most of our devices are wifi enabled, probably right now you’re reading it in one, so how do we perceive it? Nowadays we can just look in the top right corner of our smart devices screen and see the wifi Icon. However how can we really feel it? Feel all the electromagnetic waves floating and bouncing around (and through) us can be, at least, interesting. A simple skin mounted device with an wifi antenna and a vibration motor would give us some sensitivity about wifi waves, i.e. the device would vibrate based on the intensity of the wifi signal(s) around the user, something similar to that: (https://drive.google.com/open?id=0B7qSiATNAoevaDZxMGI0UUJPQms)|
|One risk of sensory substitution or augmentation is that it might diminish existing senses; if you're constantly using a device that augments your senses by delivering information visually to you, you might depend on your eyes less. Possibly, while learning the take advantage of new senses, one's brain might unlear less used ones, such as smell or taste. Clearly the benefits involve being able to directly recieve data you would not normally be able to.||Okay, how about a headset with a camera pointed behind you that generates highly stereophonic, high frequency sounds in response to object proximity behind you? That way you could develop another sense of things behind you, without interfering with the frequencies that you would encounter in everyday life. Or how about a device that you swallow/implant inside your stomach that heats up to notify you when someone you love is in danger? Or how about a device that squishes the traditional light frequency band so that it can deliver a wider range of light frequencies to you by converting them to colors your brain can interpret?|
|Sensory substitution and augmentation empowers certain population to complete tasks that they would not be able to given the impairment of their senses. However, the current sensory substitution technologies are not mature enough for commercial use yet. Sensory substitution is not the ultimate replacement of ‘natural senses’ - there exist a performance gap between the two and substitution sensors are not capable of accommodating people in complex tasks such as playing a music instrument or driving an automobile. Sensory substitution is also a hardware-centric technology that might be inaccessible to certain demographics because of high cost and low mobility. The long-term effects of the sensory substitution technologies are still unknown and by rewiring the brain, we might be putting excessive cognitive load on one sensor since we are making it doing work for another.||Sensory substitution technologies could help people with sensing impairments by stimulating specific areas in their brain and assist them in completing simple tasks. For example, it can help blind people perceive seeing using the vOICe vision technology by feeding visual information into their auditory senses. Besides helping the disabled, the technology could also be displayed at a museum for average joes to experience and experiment. For example, there could be a video game where participants wear headsets, blindfolds and play the game with information processed by their hearing sensors. A silent music concert becomes possible too where music is seen and perceived through vision senses. With sensory substitution technology, artists and composers are empowered with tools that could bring their creativity onto a different platform. Relating back to the skin augmentation technologies we covered last week, future cyborgs will be presented with opportunities both to design body sensors and then pick the sensors they would like to use for a specific task.|
|Sensory substitution augmentation is a very interesting concept and could improve human life in many ways. With sensory augmentation, humans are not limited by the receptors they have but could instead wear additional devices that enhance their perception of the world around them or make them perceive events (perhaps dangers) that they would otherwise not perceive through their sensory regular sensory inputs. Sensory substitution could help people who have lost one or more sensory organs. Like any technology, excessive reliance on such a system could produce unwanted after effects in absence of it. Since humans would get so used to such enhanced perception, the absence of sensory augmentation might itself lead to a more diminished sense of perception when compared to a human who has never tried a sensory augmentation device. Such devices might even cause a sensory overload.||Sensory substitution devices can help the visually impaired by helping them find objects in their vicinity. For instance, a camera mounted on the head could produce a vibration or a sound when a cane is in its field of view. Taking this idea further, different people could cause different sound signals. For the average joe, sensory augmentation systems could be used to alert in case of dangerous situations. For instance, if a car is speeding towards a person when the person is not looking, the device to cause certain vibrations to occur. In fact, sensory augmentation devices could enable humans to have a 360 degree vision by mounting a cameras facing in all four direction and providing different haptic and auditory signals based on the surroundings.|
|The possible rewards of sensory substitution and augmentation is that the technologies could potentially change and reshape our negative perceptions and memories. For example, some people with PTSD may not be able to react normally when receiving certain sensory inputs because the inputs remind them of their past dramatic experiences. In this sense, the sensory inputs trigger certain negative perceptions and memories. However, sensory substitution/augmentation could potentially reshape their perceptions of the sensory inputs and help them dissociate the sensory inputs with the negative perceptions. Another possible reward is that the sensory augmentation technologies could be used for long-term memory consolidation and recall. Our olfactory system has intimate anatomical associates and fiber connections with hippocampus, so olfactory system could potentially help us register, consolidate and recall memories via sense of smell. Thus, the olfactory augmentation and the substitution of other sensory inputs for sense of smell may potentially help humans who are visually or linguistically impaired remember things. However, one potential peril is that humans may use the sensory substitution/augmentation abusively. If we rely excessively on artificial sensory system, our natural sensory systems may degrade. For example, since I start wearing glasses, my eyes become more and more dependent on my glasses. In the beginning, I wore glasses just in class, and now I have to wear it almost 24 hours a day except sleep time.||One idea for sensory substitution technology is that we can understand smells visually through our eyes. Humans have a decent olfactory sensory system on average, but the olfactory information (e.g., smell of a rainy day) cannot be presented to other people as easily as visual objects (e.g., wine bottles). For example, we can smell the difference between different perfumes, but it is harder for us to describe smells and remember the differences between smells. Smells are hard to be described precisely in our everyday conversation. Thus, olfactory information could seem quite “subjective” to humans. However, if we could use our eyes to see the representations of different smells and use our ears to hear different smells, we would become much better at “remembering” and “recalling” smells. For example, different elements in smells (e.g., humidity, sweetness, density) can correspond to different colors and shapes. In this way, we would be able to precisely recognize different smells, record and remember different smells, and describe olfactory information more precisely and easily to other people. |
Adam Haar Horowitz
|It is so exciting when technology offers ways to create entirely new kinds of cognition, rather than simply enhance or recreate existing thinking. The Deroy paper strikes me as correct--that sensory substitution is not a horizontal cognitive addition, not emergent as the organic senses are but instead rising from inter-dependence of existing systems. It is a key question in philosophy of mind for the 21st century, then, of what semi-perceptual and semi-cognitive qualia are. What is the augmented or substituted experience of the world? Why would we expect sensory substitution to have full affective qualities when our emotional response to objects and colors is tied to our shared experience of them with other humans who perceive them similarly? And if we find ways of intaking information that are easier or more efficient than our natural systems, what do we lose by beginning to rely on those systems? |
We gain the possibility that memory chunking is not an issue with multimodal learning, that attentional and affect control are far superior in substituted sensory systems with less integration and thus less distractions, that PTSD triggers can be overcome with visual/auditory substitution...but an initial peril, to me, seems similar to our discussion of offloading memory; i.e., a loss of associative properties amongst our knowledge groupings internally, loss of metaphor and of impromptu AHA moments regarding previously unseen linkages, because the information does not exist 'horizontally' internally, as it otherwise would. We understand our sensory systems are plastic following lesions, and braille reading does activate the occipital cortex, but we know so very little about how the binding problem and the higher order associative properties of consciousness respond to such rewiring, control them from above, and make them fully, not semi-cognitive. This peril is the peril of the expanding semi or unconscious experience of the world.
|The transition of cognitive to subcognitive perception, and subsequently subcognitive to affective perception, needs far more work. What is the relationship between actively tracking ourselves perceiving something and the qualia created afterwards, how does this transition tie into emotion, what opportunities emerge during these transitions? Many ideas emerge for exploration: for instance, patients recovering sight from cataract removal report no emotional qualities of faces or colors (Gregory, 2003). As a new modality comes online, it apparently takes time for associative properties to follow. This opens up the door for PTSD modality-switching exposure therapies. PTSD is essentially an attention and affect disorder, so new cognitive relationships emerging between attention and affect in sensory substitution open up new possibilities for PTSD therapeutics. Experimenters could bring in a patient who cannot bear seeing scissors, for instance, and train them on a vibrotactile sensory substitution system. As their perception becomes effective but their affect lags, train them then on perceiving without reacting emotionally. The process will be far smoother because their immediate associative PTSD properties are eliminated in a new sensory modality, but 'objective' semantic<->object links remain intact. This is an extended version of attentional bias variability based incremental exposure training for PTSD. Once semantic links are tested and solidly set for a patient and vibrotactile perception moves from cognitive to subcognitive without affective traumatic associations, we may have a very effective PTSD alleviation. |
Other possibilities include expanding memory chunking limitations on working memory with multimodal or subcognitive processing and learning; creating learning paradigms in substituted senses without associative distractions for children; the recreation of affective touch and its equivalents, versus simply perceptive touch, for the populations who suffer comorbid limb loss and depression (highly correlated); the extension of Montessori learning styles (no walls in classrooms for impromptu associative learning) by giving subjects steady streams of complimentary (not identical) information in separate modalities, to bolster associative learning and creativity. (Carson, 2002 + 2003).
|Sensory substitution and augmentation are pretty mysterious types of augmentation. It is through our senses that we accept the world, and form our perceptions of it. Our senses are essentially the hardware by which we construct our entire reality. So sensory augmentation is a powerful form of prosthesis. Most of the benefits that come from its inception are first and foremost, the ability to perceive the world differently, and gain a different understanding of patterns and relationships between phenomena. The translation of visual information to auditory can provide completely different understandings of space that the visual sense, because of its material, either cancels out or makes insignificant (like being able to hear brightness or gradient changes, something that is harder to differentiate on a finer grain level through sight). Also with sensory augmentation, in the ideal world, humans can train themselves to become more aware of things like the presence of stress in others, or the breathing habits of a sleeping baby, by translating those stimuli into more perceivable senses, like hearing breathing rhythms, or seeing heat maps of mood. In this future, we can train our senses to be triggered by these things, the same way we train ourselves to read, or to learn an instrument. One of the perils of sensory augmentation though, is the question of unlearning a sensory switch once it’s been ingrained, and the possible distortion of reality it could have on the user. Since sensory augmentation is so personal because it deals directly with perception, it could start to distort the way in which people naturally perceive the world, and handicap them if they’ve removed or replaces an important association (for example, if you smelled every color you saw, it could possibly prevent you from the ability to smell things normally). It could also make certain experiences unpleasant or overstimulating (for example, touching taste could lead you to not want to eat anymore since the frequency of touch is much greater than taste, and over saturating the taste sense could be detrimental to nutrition). Finally, one other peril that a user of sensory augmentation could face is the societal struggle of not being able to relate to how other people sense the world. Because sensory augmentation leads to new perceptions of self and body and world, the user is almost creating an entire new world for themselves, which could eventually in long term, be isolating.||A big opportunity for helping a user group with sensory augmentation could be the reconfiguration of senses for people with prosthetic limbs. Because the loss of a limb also results in the loss of nerve endings that previously connected those appendages to touch in the brain, many amputees cannot feel the relationship their prosthetics have with the world around them. With sensory augmentation, one could essentially simulate “feeling” in the leg, either of touch or ground contact, by stimulating other nerves, such as a sensation in the shoulder or chest. This would allow the amputees to start to associate the relocated sense of touch, with a stimulus in their prosthetic, allowing them to “feel” their biomechanical limb as though it was their original biological one.|
|Having a “sixth sense” could enable people with sensory deficiencies to see, smell or feel things through direct brain stimulation, bypassing their dysfunctional physical senses. The danger of this is that, for instance, a blind person could be shown something via phosphenes generated by transcranial magnetic stimulation that is false or misleading but they would have no way of confirming the signal's veracity via their own sense of sight (seeing is believing).||I’m mildly color blind. In my career as a digital artist I am constantly tasked with making color-sensitive decisions. Over time I’ve learned to make accommodation for my lack of color discernment. In particular, I make a point of using software (e.g. Photoshop, Maya) that display a pixel's RGB values as part of the UI. Other software I like to use (e.g. Nuke compositing software ) will allow you to sample arbitrary groups of pixels and can output data visualizations of the color via histograms and other methods. This has been invaluable to me in my work. Over time I’ve developed an increased intuition of what color an RGB value represents without having to think much about it. It would be interesting to have these color data visualization tools available at the operating system level so that users with color sensing deficiencies can sample the colors of anything on their screens or as a smartphone or AR app to sample colors in the environment.|
|Sensory substitution and augmentation has advantages for user groups with sensory impairment. We can give sight to a blind person using touch or teach a deaf person how to hear with his eyes or hands. All our senses - sight, hearing, smell, touch, taste, spatial awareness - are our own inputs to have a perception of the world. Thus, they define the understanding and definition of our reality. As pointed out in most of the readings, training and experience are key to integrate a new sensory modality in our perception system. However, as O. Deroy and M. Auvray mention, most of the systems developed have failed to live up to their goals as they assume a that a sensory device can create a new sensory modality. I think their point of view is very interesting as they propose a new model where new sensory modalities should be created on top of underlying existing modalities. Also, minimizing perceptual learning becomes an essential goal for the creation of this new sensory modalities.|
Thinking about perils of sensory substitution and augmentation, in the same way that a a sensory disability can affect how a person gathers information from the world around them, sensory augmentation and substitution can cause us disturbances in the way we perceive the world. The brain is an unknown. And playing with our senses can be dangerous considering the brain-body connections that still need to be unveiled. Brain cognitive disorders could be the cause of a bad integration of new sensory modalities. Moreover, it is important to consider how a person will relate to others if the sensory modalities they user are different.
|A tongue system that allows you to taste what others are eating. It could be good for cooking/tasting lessons or for weight control / diets.|
A sound system that lets you know where others are.
Difference in sensing for different cultures. How can we develop a system that lets us feel/perceive what people from other cultures perceive. Is there a bias for different cultures in perceiving through our senses?
|I believe the greatest improvement for sensory augmentation would be for the safety of people in public places. Sensory improvement could give people’s reaction times a boost by warning them ahead of time (I.E. if a car that you haven’t seen, but heard slightly is coming at you in a crosswalk, sensory augmentation could make you detect it and give you a chance to evade it). Sensory augmentation could also be used to possibly detect a person’s behavior and give you a warning if you’re stepping out of line or if the person is behaving in a strange manner. Sensory substitution could also be explored to see if there could be any way to free the brain from having to process a certain type of sensory input (I.E. removing sight makes you smarter/you learn how to see with other senses or something similar). The biggest peril of sensory substitution would be if some sort of substitution were made and the brain could not handle the input (I.E. sensory overload, similar to Netflix’s Daredevil).||An interesting application would be the sensory augmentation of audio with a device that creates sounds based on if a fast-moving object is heading towards you. This could be done with radio signals and considering the Doppler effect. Another interesting application would be, similar to the vibrating belt, some wearable that would vibrate in the direction of an object you lost so you could find it or if it vibrates in the direction of a person that may be interesting to a user.|
|Although most of the current perils in sensory substitution are related to lack of congruency or too high a learning curve, there are other aspects to consider. One participant in Nagel Et Al.’s study who wore the sensory belt stated, "After removing the belt, my living space shrank quickly: the world appeared smaller and more chaotic because relative positions to places beyond the visual horizon were rather unorderly.” It is my hope that sensory substitution could help to expand our senses, minds and abilities, rather than making us fully dependent upon them. In terms of the rewards, Sensory substitution is possibly one of the most exciting yet difficult aspects of human-machine symbiosis. In recent years, there are notable developments such as the experiment by the Computer Engineering Department at the University of Washington. In the experiment, they explored whether it is "possible to send information from artificial sensors or computer-created worlds directly into the brains so that the brain can start to understand that information and make use of it to solve a task?” and found ways to communicate instructions through TMS. Despite initial steps forwards, the team still recognizes that they are 20 years short of developing their design for everyday use. Other researchers have experimented with tactile information; Nagel Et Al. examines this in the paper "Beyond sensory substitution—learning the sixth sense.” While specific applications were difficult to determine due to high variability across the small participant pool, other fascinating explorations such as cognitive-motor training and sensory learning arose. This aspect is one of the most rewarding areas of sensory substitution as it allows us to learn more about the range and limits of our cognitive plasticity. Furthermore, as Deroy et.al explores, those who lack certain senses can potentially gain them through sensory innovations. Although technologies such as tongue or tactile information for people with visual impairments have experienced a low adoption rate, early results reveal it is an area wherein the field can expand to help many. In the article, “Distal attribution: how we can distinguish between a sensation and the perception of an object that causes this sensation” the visual-to-auditory substitution analyzes features that could benefit individuals with gate challenges such as in Parkinson's Disease or blindness through sensory-motor coupling.||Sensory Substitution Technology: Tactile Driving Navigation|
Target User Group:Drivers
I am interested in designing a sensory substitution device for drivers. According to the National Safety Council, “at least 28 percent of vehicle crashes are caused by texting and cell phone use alone.” A majority of "cell-phone use” during driving is due to drivers relying upon navigation systems such as google maps, or waze which use a visual interface. Although we are quite comfortable with our smartphone visual interfaces, these mapping tools take the driver’s eyes off the road. Birds and other animals do not rely upon maps as they have a keen sense of directionality. It would be fascinating to mimic this sense by employing tactile sensory substitution for mapping directions. A device which conducts vibrations through the steering wheel and to a driver’s hands could greatly assist the navigation process as the driver would know whether to turn left or right without looking at a screen and endangering fellow drivers and pedestrians.
|While the concept of sensory substitution sounds really significant, I have sincere concerns about this technology. As human is evolved to gain sensations about the surroundings, does the sensory manipulation damage the current ecosystem for human biology? Should we restrain the technology for medical use first and take it slowly in terms of mass implementations? Should we keep the technology/ research open to public or let it controlled by certain divisions?||Idea 1: In building virtual reality environment, maybe we could use this technology to enrich the user experience. For example, let users gain realistic sensations about scent of flowers, tactile experience, Or subtle feelings when your eyes are hit by sunlight, etc., etc.; Idea 2: We can use this technology to help disabled people to regain full perceptions about the world, like enabling blind men to "see" things; Idea 3: it might be crazy but, by using sensory substitution, maybe we can help ordinary people to foster mindfulness. Because we as normal people are used to have complete sensations so sometimes we are oblivious of the importance of a certain sensor like taste.|
Juan Pablo Ugarte
|The possible rewards of sensorial augmentation and substitution are diverse and cover a vast range of possible applications. Nonetheless, they all share the promise of expanding the human’s perceptual space and channels to different degrees. However, more relevant in my opinion are the cognitive implications of such expansion; most importantly, whether they operate at a conscious, or subconscious level, and whether they provide information via existing sensor modalities or provide new ones. Although it is difficult to exhaustively sample the entire space of cognitive implications of sensory augmentation/substitution in search of its possible rewards, some of them, on a purely intuitive basis include:|
(i) The possibility of interfacing with digital machines and data at a sub-cognitive level (e.g. “feel” a physics engine simulation such as the one presented in AfterMath, or have distal awareness of a robotic arm in bilateral robotic tele-operation using one’s bare arms as controller). This would provide real-time information at a conscious level to support decision making in analytical and creative tasks;
(ii) the possibility of sensing other people’s emotional state, by means of profiling them using biochemical and electrical markers. This emotional-sensing would operate at a subconscious level and could help improve empathy and compassion. It could also work at the individual level or collective level (feeling the collective mood of large groups of people);
(iii) the possibility of feeling changes in the weather, such as the increase in electrostatic charge differential between the ground and the air that precedes thunderstorms or changes in air pressure. This would operate at a conscious level, and would serve as a counterpart of the subconscious effect that weather already has on human emotions and behavior. In that sense, this sensory augmentation would explicitly complement an existing similar “sense”.
All these examples expand the repertoire of human perception but also directly show the promise of expanding the scope of human action. Still, they mostly work as real-time information to support decision making at unconscious and/or conscious levels, and although the expand our perceptual space, do not necessarily expand our perceptual channels.
A possible peril of sensory augmentation is the transferal of relevant decision making to sub-cognitive processes. Just as in Nagel et al. experiment, where navigational decision making could be processed at a non-conscious level, one could think of sensory substitution techniques where the body receives data through one or more senses (e.g. tactile sense) and uses the brain’s remarkable pattern-recognition abilities to process it. David Eagleman performed a simple experiment where the test subjects received stock market information through a haptic feedback device, and made agnostic buy/sell decisions based on the information they received. In the context of the stock market speculation, buy/sell decisions have big, concrete repercussions in people’s life, as the 2008 subprime mortgage crisis dramatically demonstrated. Therefore, although they are logical-analytical problems, there certainly is a moral dimension to them.
The fact that a human brain could be successfully trained to receive and process data (e.g. sotck market information), and make informed-decisions based on it at a subconscious level, brings important ethical questions. Is the user morally responsible for the decisions she or he is making without real conscious real, moral judgment? What are the implications of decisions made without regard for moral consequences, assuming moral judgment cannot be delegated into the haptic feedback device? Interestingly, moral decisions many times are regarded as “visceral”, hinting that they might in fact not be entirely processed rationally/consciously. However, the rationally, analytic aspect of moral decision-making is undeniable, and thus bypassing it through a sub-cognitive decision-making process should rise some concerns, to say the least.
|Note: In this response, a robotic system is understood as a distributed system including the actual robotic mechanical elements and actuators, and its intrinsic and extrinsic networks of computers and sensors.|
In the context of human-robot interaction and human-robot collaboration, several strategies are employed to achieve a fluid communication between human and robotic agents. These strategies contemplate both explicit and implicit modes of communication through multiple channels and mediums. While prosthetics has explored these issues and made significant progress in bilateral communication using the human nervous system —EEG and EMG— other HRI and HRC applications have exploited the opportunities of this coupling rather unilaterally, focusing in human>robot control and not so much on robot>human sensorial feedback.
HRC in creative practices naturally lends itself to challenge this trend by inputting information about the robotic system’s state in the human agent. Using sensory substitution and EMG stimulation, the user can acquire distal sensing capabilities via existing or new perceptual channels. The interesting aspect of this augmentation would be the effect that these new sensorial data would have on decision making and spontaneous creativity. A concrete use of this augmentation could be robotic-bilateral tele-operation.
By mapping a robotic arm’s sensory data (acceleration, force sensing, etc.) into the arm of a human operator, would allow him or her to bilaterally tele-operate a robot by simply using their own arm. That would mean using the human operator as the control and motion planning strategist, and using the robot as the task executor. In other words, it would make a human feel and act as if they had a robotic arm attached to their body. By means of keeping the rich sensorial data but having the actuation, size, and range of the robot, they would be able to embody the robot and use it as a natural extension of their physicality –a distributed body. If we add other sensory feedback from the robotic system (e.g. processing of all sorts of data, maybe related to digital models and physical simulations and analysis, or perhaps big data and social media information), then the user could also inform decision making with this data which would otherwise be ingested via traditional means (e.g. reading). At this point, it would be worth asking whether the human controls the robot or vice-versa; because even though it is true the human would command the robot to operate, the robot would also be part of the human’s body, and as such would take part on the decision making process.
|Our senses are crucial for is to understand the world around us. Most of our experiences are defined by our sensory experiences, e.g. a warm hug, a soothing sound, mouthwatering smell, etc. When we talk about senses, we often think of people who are deprived of one sense or another, e.g. deaf or blind people, or just average Joe, who has all of the senses, but can be given superhuman senses, e.g. the ability to see through walls. But the notion of sensing isn’t binary. People who have one sense can suffer from over- or under-stimulation of one or more of their senses. Our senses are in an intricate balance and the “right” amount of sensing varies from one situation to another, and from one person to another. E.g. some people may get really happy on sunny days, but sunny days can be a problem for those whose eyes are oversensitive to sunlight. Hearing the world around us is important for most of us, but being over-sensitive to sounds can make people irritable. In fact, under- or over-reacting to one’s own senses is a one of the symptoms of autism spectrum disorder (ASD). ADHD, anxiety disorder, OCD, and depression are also linked to over- and under-stimulation of senses. In fact, there is a term for under- or over-stimulation of senses, i.e. Sensory Processing Disorder (SPD). SPD happens when sensory signals are either not detected or don’t get organized into appropriate responses. People with SPD may also suffer from depression, underachievement, and social isolation. Therefore, it is crucial that sensory technologies are considerate of the different levels optimal sensory experiences to avoid over- or under-stimulating a person’s senses.|
I think there are 4 kinds of sensory technologies:
1. Sensory augmentation: technologies that amplify or dampen our existing, e.g. hearing, or even add new sensing, e.g. our ability to locate the magnetic north.
2. Sensory substitution
3. Sensory control: Technologies that try to directly influence our perception or control our brain, e.g. through TMS
4. Sensory deception technologies: Technologies that help us feel our senses, but from time to time, change the perception of those senses. E.g. a heart rate sensor that shows an artificially lower heart rate when when a person is anxious to avoid freaking out people even more.
Of the four types of sensory technologies mentioned above, I am most skeptical of sensory control. While sensory control may be useful in very selective cases, e.g. TMS is useful for treating depression, the notion of someone else controlling my brain is scary in general.
Sensory augmentation technologies can be good, but they at least have to include a knob that allows people to adjust the levels of augmentation they want to avoid over or under-stimulating the senses. However, this knob is not the complete solution as it gives too much choice to people who may not know what are the optimal sensory experience levels for them or even if they do know, self-control is a tricky area as we have seen with people who try to reduce their phone or Facebook usage. Nonetheless, sensory augmentation technologies open humans to new sensory experiences, e.g. making people more introceptive and self-aware so that people may take better care of themselves.
Sensory deception can be used to trick humans to think they there calmer when they get anxious (by dampening their vital signs) or to feel stronger positive emotions when they feel even slightly happy or excited (by amplifying their good vital signs). Sensory deception, though useful, is not without its doubts. Should a person just not “experience” anxiety or should they learn to experience it so that they can deal with it constructively? I think it’s a combination of both. Also, is it moral to make someone think that they are more excited to meet someone than they really are? Would we be creating a false perception of true feelings or just helping people understand their feelings?
Last, we have sensory substitution technologies. There are several concerns about how effective these technologies are and what is the learning curve for people for different sensory experiences. Still, I think it is a useful experience, especially for people who are missing out on a few senses. There have been reports of people feeling emotions as if they had regained their sight (Gregory 2003).
|I think of three different ideas for sensory augmentation:|
1. Sensory filtering devices that selectively amplify or dampen sensory experiences. E.g. silencing sounds from the road (noise pollution) or in shopping malls, and amplifying voices of people who were are conversing with in loud environments. This is even applicable to perception of light and color as visual filters can be used to increase or decrease the intensity of colors.
2. Visual to auditory sensory substation system that allows blind people to “read” painting. It’s like audio version of braille. As a user moves their hand over an image, they hear the sounds associated with each pixel color. We can start with 8-bit images to keep it simple. We’ll use spatial sound so that the user understands the direction of the sound.
3. Sensory deception can be used to dampen people’s heart rate when they are anxious to make them feel calmer.
|1) Interesting thing to think about the sensory augmentation is whether that interpretation and interaction part has its own processing or not, because they can subconsciously change many perceptions, thoughts and decisions. 2) From my perspective, sensory perception is the most instinct and subconscious process that should require no cognitive translation. However, many sensor augmentation application pairs sensory input and output that have no intuitive connection which needs long time training and continuous conscious interpretation.||1) In real or virtual world, use physical or haptic feedback to extend our touch perception through skin and hair, just like cats using whiskers and fur to perceive space and distance. This can be very intuitive way to help vision-deprived people to "feel" the world. 2) For autism kids, they are not sensitive to social interaction (e.g. subtle facial expression and mood) but really good at other stimuli (e.g. number or music). There could be ways to detect subtle social cues and express them to those kids in their favorite way, e.g. playing different mood music that suits the social scenario.|
|Sensory substitution and augmentation technology could help those with disabilities to regain mobility and independence, it could also provide able bodied people an expanded understanding of the world around them. Devices that map visual information to tactile feedback could enable those who’re visually impaired to navigate with ease, the same concept could be applied to devices that exposes wavelength beyond the visible light spectrum. We could potentially enable people to see the heat distribution of the environment around them, so they could move to cooler parts of the room if they’re feeling too warm. The downside of providing people with super sensory inputs lays in privacy issues. How would we ensure privacy for people to safely utilize x-ray visions?||For average tourists walking down the streets of a new city, we could consider providing them a compass that leads them to interesting destinations, and avoid high risk areas. I had the experience of exploring the city of Vancouver alone and accidentally walking into the East Hastings neighborhood which is known for its high crime rate. I was shocked to see mostly homeless people walking around and doing questionable activities. I felt extremely unsafe, but I didn’t want to draw attention to myself by turning around and sprinting away, or taking out a shiny electronic device to find my way around, so I continued walking and making random turns hoping to get out of the neighborhood. This experience could’ve been prevented if there’s a discrete wearable compass that provides people direction to safer city blocks through haptic feedback. The last thing a tourist wants to do when feeling like a target for robbery is to show any valuable possession. A discrete haptic based navigation device would ease their worries and provide them the fastest way out of a risky area.|
|I liked Lanier story about the Lobster limbs and his reflection about how our evolutionary pedigree can literally expand our minds along with our limbs in extraordinary ways. The most obvious peril of sensory augmentation, imagined from the current landscape, has to do with potential challenges to democracy - to this I would also add culture. Beyond this I think the challenges have to do with imagination. Perhaps imagination is indeed what bridges the gaps here. I've just been at a computational neuroscience conference (Cosyne) in Utah all week, intensively thinking about neural nets and biology. This difference, between neural nets models and biology, came up a lot. One of the things I keep thinking about and had discussions about pertained to how to leverage interfaces, like (specifically) wearable technology, to run experiments that cross into a threshold that has a potential to teach neural nets and create feedback between humans and computers. Indeed, there was a very interesting discussion that included a meditation on what kinds of experiments it would be useful to run in the coming years, and this led to a productive series of sequences of thought about what augmented reality could contribute to this. I'm also thinking about haptography, which I write about in my next answer about target groups (even though it's less about a target group and more about a target problem).||Katherine J. Kuchenbecker, keynote speaker at the Vision Sciences Society meeting this May, and director of the new Haptic Intelligence Department at Max Planck in Stuttgart - notes "First, we invented a novel way to map deformations and vibrations sensed by a robotic fingertip to the actuation of a fingertip tactile display in real time. We then demonstrated the striking utility of such cues in a simulated tissue palpation task through integration with a da Vinci surgical robot. Second, we created the world’s most realistic haptic virtual surfaces by recording and modeling what a user feels when touching real objects with an instrumented stylus. The perceptual effects of displaying the resulting data-driven friction forces, tapping transients, and texture vibrations were quantified by having users compare the original surfaces to their virtual versions." I wonder how the technology Katherine describes could be useful in mapping neural nets and deep learning that articulate biology with greater fidelity.|
|As with last week's reading find myself this week leaning my views in this topic towards the positive implications for those with a disabilities. Sensory substitution for the blind is something that I find many presidents when searching the word sensory substitution on the web. Some projects approach it through both touch and audition, with special emphasis on the importance of training for the use of such devices, while thinking on potential pitfalls in their design. I think for sensory substitution a potential pitfall could be that by conveying extra or too much information about say the environment one to risks sensory overload an already limited with bandwidth sense.||I would like to be able to augment temperature receptors for people working in factories where often get burned by touching equipment they didn’t know it was extremely hot. I would also like to expand on the work of echo location but rather than being limited by the sounds created by the mouth. You will have a device that generates series of frequencies that are better suited for echolocation. I would also like to augment the perception of time so things slow down in certain times|