yosun.me: yosun is me » UX http://yosun.me a diary. maybe. Tue, 30 Sep 2014 22:07:10 +0000 en hourly 1 http://wordpress.org/?v=3.1 Tap Shake Messenger from #MutherMobile http://yosun.me/2011/06/26/tap-shake-messenger-from-muthermobile/ http://yosun.me/2011/06/26/tap-shake-messenger-from-muthermobile/#comments Sun, 26 Jun 2011 22:12:33 +0000 yosun http://yosun.me/?p=101 So, it’s an Android smartphone app that opens up the usually highly-visual nature of these devices for blind and visually-impaired people using haptics. The one I demo’d has no visual UI – it’s UX via haptic braille and long/short-touch input.

I spent the last weekend at WIP’s “Muther of all Hackathon” creating what I’d originally hoped to be a “smartphone platform for the blind” called “Tap Shake Client,” but had to downgrade to “Tap Shake Messenger” due to the main feature being only messaging and reverse geocoding fetching. I generally create silly or quirky visualization 3d-graphics based app’s for hackathon’s, and it’s the first time I tried creating an app potentially useful and life-changing for an entire population. It was fun! I won a couple of prizes, including Immersion’s grand prize and AT&T’s accessibility prize. Also won prizes from deCarta and BlueVia, and I think one other, whom my sleep-deprived amnesia from the weekend has caused me to forget. (Sheepishly, I admit the unexpected deCarta prize was by far the best. Their evangelist – a fellow Southpark fan – really reached out to the inner Cartman in me with a 64 GB iPad 2!!!)

Here’s my prezi presentation, that briefly explains the need as well as the disruptive notion of an app with no visual UI – with user interaction all via haptics and temporal touchscreen (long touch, short touch, etc):

Here’s the outline of the main Braille components:

Legend:
Each braille character is composed of 6 dots. Spaces are just spaces.
Long click = filled dot
Short click = unfilled dot
Double tap = space.
Six short tap (blank) = send the SMS!

I composed a very hackish representation of the braille character set using this serialized array, with 1′s representing filled dots and 0′s representing unfilled dots (Braille is almost a case-insensitive language, as the upper and lowercases are exactly the same, but the uppercase is prefixed with an upper case symbol, denoted as C below.. and yes, numbers are not yet represented – it would have taken just about 30 secs more to type in the numbers, but tempus fugit!):

1
2
3
4
5
6
7
8
	public int totalCharSet=28;
	public char lookupChars[]={    'E' ,'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z','C'  ,'N'  ,'.'   };
	    public int lookupDot1[]={   0  , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 0 , 0 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 0 , 0 , 1 , 1 , 0 , 1 , 1 , 1 , 0   , 0   , 0    };
	    public int lookupDot2[]={   0  , 0 , 1 , 0 , 0 , 0 , 1 , 1 , 1 , 1 , 1 , 0 , 1 , 0 , 0 , 0 , 1 , 1 , 1 , 1 , 1 , 0 , 1 , 1 , 0 , 0 , 0 , 0   , 0   , 1    };
	    public int lookupDot3[]={   0  , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 0 , 1 , 1 , 1 , 0   , 1   , 0    };
	    public int lookupDot4[]={   0  , 0 , 0 , 1 , 1 , 0 , 0 , 1 , 0 , 1 , 1 , 0 , 0 , 1 , 1 , 0 , 1 , 1 , 0 , 1 , 1 , 0 , 0 , 1 , 1 , 1 , 0 , 0   , 1   , 0    };
	    public int lookupDot5[]={   0  , 0 , 0 , 0 , 1 , 1 , 0 , 1 , 1 , 0 , 1 , 0 , 0 , 0 , 1 , 1 , 0 , 1 , 1 , 0 , 1 , 0 , 0 , 1 , 0 , 1 , 1 , 0   , 1   , 1    };
	    public int lookupDot6[]={   0  , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 1 , 1 , 1 , 1 , 1 , 1 , 1   , 1   , 1    };

I then used a combination of different parsing functions to convert input to output and the other way around.

Dots to Character

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
    /* given braille dots, outputs index (int)
     * @ bc[] = dots[] (braille dots) 
     */
    public int dots2index(int bc[]){
    	// given braille dots, returns index
    	int index=-1; // returns index of char, -1 if otherwise
    	for(int a=0;a<totalCharSet;a++){
    		if(bc[0]==lookupDot1[a]
    		 &&bc[1]==lookupDot2[a]
    		 &&bc[2]==lookupDot3[a]
             &&bc[3]==lookupDot4[a]    
        	 &&bc[4]==lookupDot5[a]      
             &&bc[5]==lookupDot6[a]        	                     
               ){
        			index = a;
        			return index;
                } 		
    	}
    	return index;
    }
 
    public char index2char(int index){
    	// given an index, returns character
    	return lookupChars[index];
    }
 
    public int parseNewDot(){
    	currentDotCount++;
    	if(currentDotCount>=6){
    		int index=dots2index(dots);
    		if(index!=0){
	    		playCharDots(dots); // sound off currently stored
	    		storeChar(index); // store char
	    		newBrailleChar(dots,charLength*2,0); // prints out char
	    		resetDots(); 
    		}else {
    			sendMsg();
    		}
    	}
    	return currentDotCount;
    }
 
    public boolean storeDot(int b){
    	dots[currentDotCount]=b;
    	return false;
    }

Character to Dots

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
   public void string2vibes(String s){
    	Log.i(appName,Integer.toString(vibeagain));
    	if(vibeagain==1){
	    	char[] cArray = s.toCharArray();
	    	int indexCount=0;
	    	int[] indexStored=new int[1024];
	    	for(char c:cArray){
	    		int c2i=char2index(c);
	    		indexStored[indexCount] = c2i;
	    		int[] tempDot={lookupDot1[c2i],lookupDot2[c2i],lookupDot3[c2i],lookupDot4[c2i],lookupDot5[c2i],lookupDot6[c2i]};
	    		playCharDots(tempDot);
	    	}
 
	    	vibeagain=0;
	    	new CountDownTimer(30000, 10000) {
	    		public void onTick(long millisUntilFinished) {
	    	     }
	    	     public void onFinish() {
	    	         vibeagain=1;
	    	     }
	    	  }.start();
    	}
    }
 
    public int char2index(char c){ 
    	// given a string char, returns braille index (lookupChars)
 
    	int brailleIndex=-1;
    	for(int i=0;i<totalCharSet;i++){
    		if(lookupChars[i]==c){
    			brailleIndex=i;
    			return i;
    		}
    	}
    	return brailleIndex;
    }

Dots to Haptics (currently using hardcoded UHL FX, #27 and #30 for long and short vibes, respectively). They sound good on a LG Optimus 3D, but weird on an Evo. See below for serialization note.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
    public int[] dot2fx(int d[]){
    	int[] temp=new int[6];
    	for(int i=0;i<6;i++){
    		if(d[i]==1){
    			temp[i]=27;
    		}else if(d[i]==0){
    			temp[i]=30;
    		}
    	}
    	return temp;
    }
 
    // a serializer is actually needed to play the series of "dot haptics"
    public void playCharDots(int[] d){
    	Log.i(appName,"Playing...");
    	mLauncher.playSequence(dot2fx(d),COMPONENT_GAP);
    }

Here’s the flow of the app. On launch, a poll is made to my web server with the latlng, and reverse geocoding lookup is done on deCarta. The app vibrates the location (string2vibes()). Then, the user can input a series of long and short clicks to touchscreen input braille (parseNewDot()). Each touch currently gives you an instant feedback (long vibrate or short), and after 6 touches, the entire braille character is haptically-regurgitated via the combination of long and short vibes. Six short touches sends the SMS sends it off to the hardcoded phone number.

Due to time constraints, the app basically used “out of the box” UHL Effects, instead of custom-tuned haptics via Motiv Studio. There were some issues with having these UHL Effects play back to back simultaneously, so Jason from Immersion helped with a serialization function that got the above to play back simultaneously. Instead of using the default Launcher class, I used Jason’s LauncherEx.

A few caveats to note: be sure to set permissions for haptics and SMS in the manifest, or your app will crash (almost) immediately. (That took about an hour or two of frantic hair-pulling during the hackathon..) Also, you should disable default haptic feedback in your haptics view

main.setHapticFeedbackEnabled(false);

.

Procedural Braille Character Renderer

Someone also asked about how the braille text was generated. It was a quick hack since I thought displaying visual feedback of regular letters only would be lame. Here’s the outline:

The debug braille glyphs are generated using the

android.graphics.Canvas,.Paint

libraries – particularly, it’s a mix of un/filled circles bounded by unfilled rectangles denoting a braille character grid. I have a Ball and Rect class for easy constructor creation of the balls and rectangle that compose each glyph. The core functions of the RenderBraille class are below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
    // multichars view dots are generated 10 from each
    float charSizeX=10; // each braille character is 10x20
    float charSizeY=20;
    float dotDistance=10;
    int dotRadius=3; // radius of each braille character is 3
    float dotType=0; // raised=1 or lowered=0
    float xmin=10;
    float ymin=10;
 
    public void newBrailleChar(int[] d,int cminx,int cminy){
    	float xmin=cminx*charSizeX;
    	float ymin=cminy*charSizeY;
    	float xmax=xmin+charSizeX;
    	float ymax=ymin+charSizeY;
    	main.addView(new Rect(this,(xmin+5),(ymin+5),(xmax+15),ymax+15,0));
    	for(int i=0;i<6;i++){
    		renderDot(i,cminx,cminy,d[i]);
    	}
    }
 
    public void renderDot(int dotCurrentIndex,int charIndX,int charIndY,int dotType){
    	int dotIndX=-1;
    	int dotIndY=-1;
        if(dotCurrentIndex==0){
       		dotIndX=0;
       		dotIndY=0;
        }else if(dotCurrentIndex==1){
       		dotIndX=0;
       		dotIndY=1;
        }else if(dotCurrentIndex==2){
       		dotIndX=0;
       		dotIndY=2;
        }else if(dotCurrentIndex==3){
       		dotIndX=1;
       		dotIndY=0;
        }else if(dotCurrentIndex==4){
       		dotIndX=1;
       		dotIndY=1;
        }else if(dotCurrentIndex==5){
       		dotIndX=1;
       		dotIndY=2;
        }
        main.addView(new Ball(this,xmin+dotDistance*dotIndX+charIndX*charSizeX,ymin+dotDistance*dotIndY+charIndY*charSizeY,dotRadius,dotType));
 
     }

You can generate a line of braille glyph by calling

newBrailleChar(dots,charLength*2,0); /* dots is an 1x6 array that is 1 for raised and 0 for lowered, counting from left column down to right, as with convention above. */
]]>
http://yosun.me/2011/06/26/tap-shake-messenger-from-muthermobile/feed/ 0