Blog

Basics of the MirrorTouch System

With MirrorTouch (the new name for my mirror-based multitouch system). For those who don’t remember, it is a project to create a retrofittable cheap new technology for touch detection. It can be made of mostly off-the-counter or even household items. The software has the potential to be VERY fast, many orders of magnitude faster than the current technology. It is less seceptable to occlusion than many other technologies.

It began well over 2 months ago. It started out with IDEALISTIC paint sketches and then a VB.NET application to parse it. Then it was ported to Python and could handle the same sketches. After discovering that in real life the positioning of the points varies due to some very strange and illogical factor, the project had a several-month hiatus.

The issue is clearly demonstrated here:

noooo!! why doesn't it work?!?!?!?!?!

Last week, I considered the project a failure. I was playing around with a flashlight and tried looking into the strange behavior of the light. And something began to dawn on me. The shape as on diagram 1, can be flattened out as a visualization for what it behaves like. So from the pyramid shape, it looks more like a little 4-pointed star. Since the mirror is only on two sides, you can simplify it to half a star emerging from a square.

To the side is a geometicalified sketch of it from my notebook. Here you can see the relation between the point and where it shows up on the mirror.

From that, you can use the distance between m and the y point (y-m) and divide it all over the distance from the mirror to the webcam (l) and plug it into y=mx+c form. Repeat that over the x axis and you can use basic algebra to find the interesction.

From that is the new magical formula that powers the application:

Yay! Purtyful!

The new formula is so magical that it actually works. Yes, it’s amazing, it has survived the most strictest of tests of mathematical consistency. It works.…. At least in theory. Now what about scientific tests? Oh no! it actually has to work in the physical world? Oh no!

With these 2 magical equations. I have (theoretically) in an idealistic model of the system, solved the issue with distortion. It should theoretically resolve all issues with the system. It should work.

So i set up the model again, attaching my webcam to a ruler and taping it to a speaker. Taping mirrors down on a piece of paper, and this time, Scribbling down measurements on the side. I got it to work. Workign without resetting configuration every time it ran. It works. It truly actually works. Multi-Touch too.

Since I cant get the webcam to feed directly to the python script, I have to use Cheese (it’s a linux app for taking pics from a webcam) to save screenies of the webcam mounted percariously from a ruler using only a bit of transparent Scotch tape. I copy the images over to the mirrortouch directory and go in the commandline and type in python process.py and watch as lines of logging output fly past as the windows autoscrolls down filled with coordinates and color hashes.

I watch as it generates a .png file.

It works in the _real_ world!

Yes it works! AMAZING!

Note: The random scribbles in the back aren’t for any contstructive purpose. No, actually they just stop my stupid webcam from adjusting the contrast and making everything all ugly and ewwie. If my webcam sucked less than maybe it would work but my webcam really does really really really suck.

Now if it could get ported over to somethign like C++, and actually parse a live video feed from the webcam then it may become an actual working implementable multitouch technology. As it stands, it’s just a multitouch proof of concept, and I don’t know C++ so it probably won’t work.

Anyone dying for the source code can find it in the SVN repository at : http://code.google.com/p/mirrortouch/ Just beware that it may take lots of scary and tedious configuring in current stages (Configuring color range of background in the band, configuring color range of target, setting distances and middle length and other horrors, but from the SVN you can also do the insanely boring act of running various images that are already there through the script, and most of the images just wont work even with replacing huge blocks of code).

So i had this little project involving a quote repository and due to some trouble installing the superior Chirpy system, I used RQMS and did a few changes to add features like ajax ranking so you don’t have to reload to rank something.

It only involved changing less than 10 lines and pasting a snippet of my vX Ajax library.

Replace lines 151, 152 and 153 with the following lines (respectively) in rash_output.php (the template).

<a href="?ratingplus<?=$GET_SEPARATOR_HTML?>id=<?=$row['id']?>” onclick=”plus(<?=$row['id']?>);return false” class=”quote_ratingplus”>+</a>
(<span id=”rating<?=$row['id']?>”><?=$row['rating']?></span>)
<a href=”?ratingminus<?=$GET_SEPARATOR_HTML?>id=<?=$row['id']?>” onclick=”minus(<?=$row['id']?>);return false” class=”quote_ratingminus”>-</a>


Then add the following somewhere arbitrary in the <head> of rash_template.php
<script type="text/javascript">
var _=_?_:{}
_.ajax=_.X=function(u,f,d,x){x=window.ActiveXObject;x=new(x?x:XMLHttpRequest)('Microsoft.XMLHTTP');x.open(d?'POST':'GET',u,1);x.setRequestHeader('Content-type','application/x-www-form-urlencoded');x.onreadystatechange=function(){x.readyState>3&&f?f(x.responseText,x):0};x.send(d)}
_.id=_.G=function(e){return e.style?e:_.d.getElementById(e)}
_.d=document

function plus(ID){
_.ajax(”?ratingplus&id=”+ID, function(){
_.G(”rating”+ID).innerHTML = parseInt(_.G(”rating”+ID).innerHTML)+1
})
}
function minus(ID){
_.ajax(”?ratingminus&id=”+ID, function(){
_.G(”rating”+ID).innerHTML = parseInt(_.G(”rating”+ID).innerHTML)-1
})
}
</script>

I’ve packaged a sample rash template with the ajax functionality. You can get it here: http://drop.io/ajaxrash

And the purpose of this is of course for one of my new projects, BotBash a conversation repository between a little (really simple) bot i built and strangers on Omegle. You can see it in action on http://botbash.antimatter15.com

Though what I described earlier is totally capable of Multitouch, I think it would probably be better to use a somewhat simpler implementation that is single-touch only.

http://distributed-pi.appspot.com/static/beta.html

(Potential number of jobs is no longer theoretically finite)

I think it’s probably better for me to move to a new web host, so I’m taking suggestions for good ones. It should be close to the $6 price range.

http://www.antimatter15.com/misc/bf.htm

So I built this interesting brainf*ck interpreter which offloads most (not all) of the arithmetic operations (it concatenates tons of +1s into some more interesting operations). It could easily be made to offload all operations, but then it becomes a bit too prolonged and boring, so for +1s, and +0s, the computer handles the task.

It’s fitted with a Hello World app, which takes quite a bit to start, but then it quickly types out the message.

I’m porting the multitouch concept to python. Why? Because i’m on linux now, mono sucks, and PIL is pretty awesome.

I’ve set up a mock-rig where there are 2 mirrors on the side angled by erasers with a crappy Creative NX webcam mounted on a textbook over it. It seems like the only practical use of this is for a multitouch-table sort because gravity makes it otherwise pretty hard to calibrate.

So after I rewrite the software I need to try getting it to parse an actual image. Then I need to somehow hook it up to streaming webcam data, and somehow do something else.

okay, so recently something really awesome happened. Google App Engine will now support Java. The great thing about this, it that decent flash export may happen because of that (yay!) and possibly actionscript later on.

With new Tri-detection system. Area Scan, Perimeter Scan, and Point Scan.

Early on, I recognized one of the biggest issues with my idea for using mirrors was the computational power necessary to run the finger-position-detection algorithm. I recently thought, that that would be totally superfluous.
My new idea is to use software to search a 1-pixel wide band of the mirrors to create several points. Those points are all combined to a list of all possible permutations. Each point goes through a method of determining whether or not it’s a fingertip. The easiest way, (and likely quite wildly inaccurate in the real world), is to measure the perimeter of a square that has that point of the center and compare it to the percent of that perimeter which is different from the surroundings. So then, you find the ones which work at all, and then you have your points!

I actually made a rough proof-of-concept system for this. It uses a very crude method of determining the different blobs on the mirrors (contiguous same color). And it uses a very crude surrounding box perimeter-ratio system. It’s to serve as a proof-of-concept type thing, not necessarily the precursor to an actual program that does something along the lines of it.

As for how fast it is, i’m not sure. I don’t even know how things like touchlib do it. If they scan through every pixel, and do more processing, then this is easily 50x faster. The speed of this is very largely dependent on the number of fingers touching it. w+h+4bf^2 would be a rough approximation of how many pixels would be needed to processed to get the result (w = width resolution, h = height resolution, b = size of surrounding box, f = number of fingers). On the Proof-of-concept, the input data is 200×200, The box set to a width of 20px, and there are 3 fingers touching, meaning ~1120 pixels searched. And if you were to scan through all the pixels (as I originally thought the idea would require), it would be wh, or 200*200, or 40,000. So the speed increase is by a factor of 36x, which is totally awesome. But again, I don’t know how others do it, they may have already an even faster way. But last year, I made a sort of object-tracking thing, which worked by scanning every pixel, and it was able to work at quite decent speed. So this, being an order of magnitude faster should work better.

Of course this is still a concept. There are still huge flaws not yet covered for like the fact in the real-world, the software would have to soemhow distinguish between the contents on the monitor and the hand in front. There may be a chance that someone is in an awkward position which tricks the software, the fact the software is completely  useless on just about anything other than a fingertip, and many many more. I still find it interesting anyway