Wow! This is really creative. Gets my gears turning on all the other applications for using opencv as an interface!

There is some well-informed speculation about latency up-thread, but I&#x27;ve tinkered with OpenCV from time to time.  Specifically, I used ROS and OpenCV to do face detection on a netbook in about 2012.  That worked out pretty well - it would track multiple faces in realtime.  Edge detection is a different algorithm, but isn&#x27;t super likely to be the origin of the latency issues they&#x27;re having.  It&#x27;s just a threshold check over a window of a few (e.g. 3 or 5) pixels, multiplied by the dimensions of the image.

Curious what kind of CPU usage the OpenCV code requires on the backend...

Webcams don&#x27;t have memory, so they can&#x27;t have latency. It&#x27;s all in the software. In fact, I&#x27;ve done some apps on Linux that just did readout of the frame from the camera sensor to ram, and displayed the buffer on the next framebuffer flip with zero copies. There was no perceived latency.It just takes some work.

Webcams usually have somewhat poor latency, at least in my experience. And I think the performance critical code is all in OpenCV, not Python.

YMMV, but I&#x27;ve experienced slight webcam lag on Macbooks even with Apple-optimized applications such as Photo Booth and FaceTime.  Perhaps the biggest bottleneck is at the hardware &#x2F; driver level instead of at the CPU-cycles level?

&gt; Rewriting it in C&#x2F;C++&#x2F;Rust instead of Python would be one...Would it? It&#x27;s using libraries for the heavy lifting already.

Too bad you can&#x27;t get CUDA easily on a recent Macintosh.

It looks like it&#x27;s just applying a bunch of filters&#x2F;convolutions to the image. You could write it in C++ to be pretty quick, in CUDA you could get it down to microseconds I think.Though that does not take account of the latency of the webcam itself.

They are asking what kind of problems need to be solved in order to speed up the latency.They are not asking what kind of problems having a touch screen on a laptop would solve.

I had a laptop with a touchscreen and the only time I ever used it was when the linux mouse drivers were playing up. So I guess that would be one use.

Unfortunately no.We could have a better&#x2F;safer C but alas we don&#x27;t. It&#x27;s either C or all the way to Zig&#x2F;Rust&#x2F;etc territory (or using C++ like C).

Zig seems very close to C to me. What makes you think it&#x27;s different?

The gap between C&#x27;s model and a somewhat memory safe language is unfortunately very large. You can either limit semantics a lot, use GC (mostly-D, Go, ...) or have relatively complex ways to describe life times (Rust).

If you&#x27;re looking for &quot;slightly&quot;, there are libraries to use that expose macros and function calls, instead of directly accessing pointers, along with the &quot;n&quot; version of libc functions.

Slightly off topic, but C&#x2F;C++&#x2F;Rust are always the go-to languages when talking about high performance. People love the speed of C but it&#x27;s so bare-bones that they move to C++, or it&#x27;s too dangerous so they move to Rust. But those are much too complicated, so people came up with Zig and Nim and D instead, but those are very far from C, even though they&#x27;re not as far as C++ and Rust. So I&#x27;m wondering, are there any languages almost as close to C but slightly safer and slightly less bare-bones? Because I like C a lot, I think it&#x27;s a great language with an excellent simplicity, except for some slight rough edges.

I&#x27;d be interested in a version that used the Vision Framework. Back when I looked at the Apple frameworks for iOS, they utilized the Neural Engine hardware available on iPhones while OpenCV didn&#x27;t (that may have changed).The laptop hardware is different of course, but it would be an interesting performance comparison.Here&#x27;s an interesting overview of using Core frameworks with Python: <a href="https:&#x2F;&#x2F;developer.apple.com&#x2F;videos&#x2F;play&#x2F;wwdc2018&#x2F;719&#x2F;" rel="nofollow">https:&#x2F;&#x2F;developer.apple.com&#x2F;videos&#x2F;play&#x2F;wwdc2018&#x2F;719&#x2F;</a>

Wow, this is a great idea!
Looking at the video, looks like the latency isn&#x27;t great; what kind of problems should be solved for this to have a ~100ms latency?Rewriting it in C&#x2F;C++&#x2F;Rust instead of Python would be one... Any ideas?

I wonder if this has ever been tentatively tested by the laptop makers, perhaps even by Apple. Surely thought about -- but tested?

The “Filter for skin colors” step would probably need to be fixed. That’s the kind of kludge that pops up all the time in demo programs but doesn’t generalize very well.

Not to mention the constant image processing going on in the background. I wonder how it&#x27;d affect battery life &#x2F; fan noise.

&gt;I really wouldn&#x27;t have expected this to work but once you read about it.... of course it would.It&#x27;s a fun DIY project and you can get it almost working. I suspect edge cases (stemming from different lighting conditions and imperfect finger detection) would render this method unworkable in general.

That is amazing.I really wouldn&#x27;t have expected this to work but once you read about it.... of course it would.This is one of those things that while there maybe wasn&#x27;t a &#x27;problem&#x27; shows how knowing a few things critical, you can really solve problems &#x2F; come up with solutions that nobody would have ever thought of (well I wouldn&#x27;t have...).

Resistive touchscreens which were dominant in 1990 typically have higher accuracy than the capacitive touchscreens of today. But they have terrible usability because you have to apply pressure to the screen to use them.

Have you tried the Dell XPS? Even with shitty linux touchscreen drivers&#x2F;experience I end up using it almost every day when scrolling. On Windows it felt very natural.

What hardware do you have? All the touchscreen windows laptops I&#x27;ve tried have the responsiveness and accuracy of a touchscreen from 1990.

I think Apple is trying to push folks to iPad Pros since they&#x27;re touch capable and now support a mouse.

how does the battery life compare to the touch enabled windows laptops?

Yes! Takes me back to the mid-90s when my then 12-year-old daughter would point out things on my computer screen BY TOUCHING THEM drove me NUTS!

I don&#x27;t miss it. While working I hate finger grease on my MacBook screen much more than I do on my iPhone or iPad. Also, my wife has a Surface laptop with touch screen and she never uses it.

Are you saying that because you don&#x27;t want it I can&#x27;t have it?

Touchscreens, like RGB lights or a numpad, are one of those things I specifically want my laptop to not have, regardless of if it can be disabled. I don&#x27;t want to have to pay extra for it, I don&#x27;t want the battery hit, and I don&#x27;t want something designed by someone who thinks it&#x27;s useful. All the touchscreen laptops I&#x27;ve seen have poor picture quality and the actual screen looks recessed from the glass, making it jarring to look at. It&#x27;s not a good workflow for me to have to take my hands off of home row to navigate either - I do all my mousing with a thumb on the trackpad.

Apple sells products, not hacks. Apple doesn&#x27;t clutter their SKUS with niche features that work poorly are almost never useful. Adding touch to a laptop or desktop just so you can occasionally touch scroll is deeply overkill.They sell iPads that are touch-enabled, that work well with touch.

For some reason I really like hitting the &#x27;submit&#x27; button via the touchscreen after filling out a web form.

That is great!On the topic of touchscreen Macbooks, I&#x27;ve been using my Mac a lot less since I&#x27;ve spent a lot of time on a touchscreen Windows laptop. It is super handy! A lot of times when using the Mac I&#x27;ll absentmindedly drag something on the screen only to be disappointed. Apple&#x27;s resistance to the idea reminds me of their stubbornness on the single button mouse. Admitting you&#x27;re wrong is not a weakness.

I don’t think it was intended for mass production, and that’s okay. It’s a really cool and impressive hack.

I think you may be approaching it differently than the creators are.They hacked something cool together, I suspect that is the point, and it is awesome.This isn&#x27;t a product.

But if the latency and accuracy isn&#x27;t great, did they solve anything?

The authors didn&#x27;t publish thinking &quot;this is going to rock the world, it&#x27;s ready for mass production&quot;... More along the lines  &quot;hey fellow hackers, we found a way to emulate a touch screen at 1% of the price&quot;.

Prototypes are cool, but I think this is dead.Anything that requires tons of visual calibration is serious work. Both for the programmer and the computer.It worked for a planned demonstration, but I imagine this would be impossible for Mass production. And I wonder if the users found it useful due to latency and accuracy problems.

Long time ago there was an ad PC vs Mac. PC guy had usb camera tapped on top of his head, claiming how PC are entering new modern age. Mac had camera build in. This kind of feels similar.

Although it was indeed posted over a year ago, 239 points is certainly traction.

&gt;Are reposts ok?&gt;If a story has had significant attention in the last year or so, we kill reposts as duplicates. If not, a small number of reposts is ok.That post is from over a year ago, and didn&#x27;t get much traction.

DUPE
<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=16745998" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=16745998</a>

Yes, but it would have a possibly annoying limitation of occlusion making the multi-touch fail at some positions. Fingers very close to the camera would appear huge and occlude everything else.

Sort of.  Actual optical multitouch systems use multiple cameras, so fingers blocked by other fingers can be seen from other angles.

This could be extended for multitouch too, yeah?

Sure. Ideally the keyboard wouldn’t have to look exactly like a standard keyboard. It could be split, and perhaps tented.Found this Kickstarter project that’s now a company. 
<a href="http:&#x2F;&#x2F;serafim-tech.com&#x2F;products&#x2F;serafim-keybo-world-s-most-advanced-projection-keyboard.html" rel="nofollow">http:&#x2F;&#x2F;serafim-tech.com&#x2F;products&#x2F;serafim-keybo-world-s-most-...</a>

Ooh, how about this?1. Point a camera at a surface (be it a piece of paper or whatever).2. Write&#x2F;print a word on that paper and use CV&#x2F;OCR to parse it.3. Detect when a person has pressed on that word and exec the command specified by the text.Instant macro pad?

Do you mean similar to a projection keyboard[0] but without actually projecting a keyboard onto the surface and using the integrated camera?[0] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Projection_keyboard" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Projection_keyboard</a>

Then you need a physical keyboard with which to make noise.

Who needs a camera when you have a microphone? <a href="https:&#x2F;&#x2F;phys.org&#x2F;news&#x2F;2005-09-recover-text-audio-keystrokes.html" rel="nofollow">https:&#x2F;&#x2F;phys.org&#x2F;news&#x2F;2005-09-recover-text-audio-keystrokes....</a>

I did that a few years ago, although it used a Kinect: <a href="https:&#x2F;&#x2F;pdfs.semanticscholar.org&#x2F;b07f&#x2F;4732e5d631cf182a59b1f28a9b458ddda785.pdf" rel="nofollow">https:&#x2F;&#x2F;pdfs.semanticscholar.org&#x2F;b07f&#x2F;4732e5d631cf182a59b1f2...</a> . It works with gestures, without a projected keyboard and has some audio and visual feedback on screen.

Maybe next someone will build a “camera keyboard“ for my iPad. Turn any surface into a keyboard with a camera and machine vision.

Very neat! I am wondering whether it could also recognize multiple fingers and thus sort of emulate multitouch gestures, although my guess is latency is not great with one finger already, so this might not work well.Als interested to see how it works with different lighting and how dirty the screen can get before recognition fails because the reflection is too „muddled“ ;)

I think they asked in a respectful manner, so I don’t see anything wrong with their comment really.Python 2 will be EOLed soon.<a href="https:&#x2F;&#x2F;pythonclock.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pythonclock.org&#x2F;</a>4 months 25 days 5 hours left until Python 2 is retired. After that point it will no longer be maintained by the Python project.Of course Linux distros and companies will keep Python 2 running for years to come because of all of the tools that are Python 2 only.New projects ought to be written in Python 3. However, in this case my guess would be that OP probably used Python 2.7 because macOS ships with Python 2.7 preinstalled.So if we could, Apple are the ones that we ought to speak to. Convince Apple to ship macOS with Python 3 pre-installed.

The switch from Python 2 to 3 was slowed down tremendously because major libraries did not fully support Python 3 immediately. The question regarding the dependencies was legitimate.

I agree with your general point, but your example if Ruby is very different. Python 2 is expected to be EOL Jan 1, as far as I know, Ruby is not.

With respect, if nobody finds the issues in the upstream dependencies and works to make them compatible with Python 3, they won&#x27;t be supported. That would be a shame for this project and others that use the same upstreams.I&#x27;d have hard time finding support for some Ruby 1.8 upstream dependencies for recent projects, so I don&#x27;t agree with you.

Put Python 2 on the dunzo list!!!
<a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=acLp0S3K6mk" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=acLp0S3K6mk</a>

While I understand the desire for a unified Python ecosystem as much as anyone - this kind of pettifogging is not going to convince anyone to switch to Py3.Consider if the authors of the article had used (for example) Ruby - you probably wouldn&#x27;t ask why they didn&#x27;t do it in Python3, so why do you do it if they write it in Python2?

I did a similar exercise late last year, using opencv v4 beta and python.  I defaulted to py3.  And immediately had problems, even with example-ish code.  Camera misconfiguration and  intermittent latency excursions, IIRC.  So I backtracked to the &quot;road well traveled&quot; of py2.  Which just worked.

This is great. I notice it&#x27;s Python 2, could this be ported to Python 3 easily or some inherent issue in upstream deps?

<a href="https:&#x2F;&#x2F;www.playosmo.com&#x2F;en&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.playosmo.com&#x2F;en&#x2F;</a> &lt;-- this also uses a mirror to refocus the built-in camera but on another surface, not the screen.

The software is a &quot;one time cost&quot; in quotes due to maintenance etc but talking in broad strokes which wouldn&#x27;t impact the claim of $1 touchOr to put it another way you wouldnt invent a new way to desalinate seawater and then say the cost is $10&#x2F;litre instead of 2 cent because of r&amp;d on the device you made..

The programmer has donated their time for free. The code has an MIT license.

What they did not factor in was the cost of the software.You can make junkyard bound stuff do amazing things when you program them right, but you pay for that programming or time spent.

I remember making a smart whiteboard with his software back in the day! It worked surprisingly well, even in a teacher&#x27;s classroom, with an online store bought ir led pen.

I remember following the work of this guy, Johnny Chung Lee, who did a ton of really neat human-interface stuff with the Wii remote&#x27;s IR sensor: <a href="http:&#x2F;&#x2F;johnnylee.net&#x2F;projects&#x2F;wii&#x2F;" rel="nofollow">http:&#x2F;&#x2F;johnnylee.net&#x2F;projects&#x2F;wii&#x2F;</a>

So I was expecting one of those sensors that can turn a TV into a touchscreen with like an IR sensor or something so I checked this wondering how they were so cheap.I was delighted to see this is so much better. Figuring out touch events from the reflection is really cool. Well done.

From the conclusion:&gt; With some simple modifications such as a higher resolution webcam (ours was 480p) and a curved mirror that allows the webcam to capture the entire screen, Sistine could become a practical low-cost touchscreen system.

I wonder what the field of view is like. The demos only show it being used towards the middle of the lower half of the screen - can the camera really see the top left or right corners?

Stunning out of the box thinking. Fantastic idea and implementation. Color me impressed.

A touch screen is definitely a more approachable method to interact with computers for people who are not as tech literate. My grandma had a computer for years and only kind of used it. Then she got an ipad from a relative and she uses everyday, from casting netflix to her chromecast to reading ebooks she rented from the library.I may not prefer touch screens but they can be a complete game changer for some users.

Those users need a touch OS like iOS or Android (or maybe Windows RT&#x2F;Metro), not a touchscreen on their desktop OS.

It helps older users who have trouble with the mouse&#x2F;touchpad.

I have a coworker who uses a touchscreen and changed my view on this because she does not use just the touchpad or just the touch screen. She uses both at the same time with one hand on the touchpad and one pointing at the screen.I agree it&#x27;s less productive on its own, but it works well when used as a hybrid where you have two types of pointers on one screen.

I use touch screens on all my devices: desktops, laptops, tablets, phones. I spend a lot of time writing code, and I need and want those touch screens, and working without them is often slow and clunky. Your feeling is irrelevant, and in general just wrong. Developers are end users.

It&#x27;s a good idea and great implementation.However I do strongly feel that touchscreen is the opposite to productivity, maybe usability to the end users, but definitely not what developers need&#x2F;want.

Use a pinball, not a flat mirror, and map back to something like a flat plane before processing. The registration is a bit harder, but not by an intractable amount.

One issue I imagine is coverage. How do you get decent horizontal resolution on top and on bottom of the screen?Either you have a mirror that converts your webcam into ultra wide angle so you cover the top edge of the screen (which you pay for by poor resolution on the bottom) or you decide not to cover the whole top area (which is where most window control elements are).The video action all happens on the lower half of the screen for a reason.A solution would be some sort of orthographic lense, that basically splits the sensor into multiple distributed lenses to avoid the FOV problems. This however would be not trivial to get right without serious tools.

The basic principle behind Sistine is simple. Surfaces viewed from an angle tend to look shiny, and you can tell if a finger is touching the surface by checking if it’s touching its own reflection.This is especially true of MacBook screens!

I had an idea years ago using a webcam and the &#x27;blood under fingernail&#x27; to have a trackpadless-trackpad (i.e. same as this project but on your table).Maybe some time-rich person will fork it and do so.

Next step is to gauge the amount of blood underneath the fingernail to tell the amount of pressure being applied.

Yep, this is truly awesome. These types of hacks are what got me into computing as a kid. Reminds me of early Hack-A-Day content before most of their posts became Arduino&#x2F;Raspberry Pi centric.

The difference is that when this is abandoned, it is complete

&gt;that end up abandoned the moment the event concludesWhat makes this one different in this regard?(I&#x27;m not trying to say it&#x27;s a meaningless project.)

&gt; The last hackathon I attended, the winners....(Not a comment on you, but you poked the button)While my hackathon experience is limited, I think any attempt to crown a &quot;winner&quot; reduces the experience. Of course, I am too sensitive because everyone can just enjoy it...but as soon as there&#x27;s a winner there starts to be rules that people try to game, the are added restrictions and limits that reduce what is being created, etc. And creating is the point.

The last hackathon I attended, the winners just built something from an Instructable that existed well before the hackathon.My friends and I built a time machine radio (i.e. you tune it to a decade, with a knob and everything, and radio news from the decade was included if we could find it) and won best hardware hack, so it wasn&#x27;t a total loss. Had a blast either way.

I especially liked the way it was presented on the website, with short videos for each step - made it really easy to follow!

This is exactly what a hackathon project should be. Delightful, innovative and scoped small enough that they produce results without too much bullshit. Not the usual &quot;world changing&quot; hacks that end up abandoned the moment the event concludes. If only events encouraged more hacks like these.

This is really f&#x27;ing cool. Nice work!!

As opposed to a driver that may have greater permissions?

No it can&#x27;t. Look at the photos. The glare is part of what makes this method work in the first place.

Do you think this MIT grad student  wants to steal your data?

Yay now their software can see everything on your screen.

Even 5$ webcam include a IR filter, the image quality without is pretty awful, so I’m pretty sure that apple webcams have it.

Maybe you could add an IR LED that could be observed by the webcam, but not in the visible spectrum. I’m not sure if the MacBook webcam includes an IR filter or not. It wouldn&#x27;t help the precision, but might make the camera less susceptible to changes in ambient light.

Coul use a mirror “broken” in half to get stereo vision maybe.

Hmm, given camera is on top, computing homography and rectifying the perspective image would give fairly good precision around the camera, falling off quickly with distance, likely not being very useful at the bottom of the screen. Also, it&#x27;s likely sensitive to both ambient light and the dominant light on the screen, in dark rooms with dark themes it might be pretty off, as most computer vision algorithms need constant tuning for changing conditions.

I love this. When I was young I saw a hack for an Amstrad which added light pen support. If I recall it effectively synced with the scans on the crt which tells you your coordinates. I wanted to build it so much as a kid, but never could. It’s the sort of thing that really fired my imagination and helped me fall in love with computers.

I hope this hack get’s so popular that the folks at Apple finally makes a MacBook with a proper touch screen.

This project makes laptop vendors selling touchscreen laptop with premium price cry.

The hovering aspect of this makes it something quite special, a feature normal touchscreens don&#x27;t have.  I imagine css ::hover elements would feel nice with this.

And they called it Sistine, which flabbergastingly shows that coming up with a good name for something is indeed possible.

What sorts of improvements could you do if you used a stylus instead?

The main idea was probably patented decades ago.  There have been optical touch screens around for ages (that use multiple cameras for good multitouch, and edge IR illumination to make detection more reliable and precise).

Is this something he should have patented?

This is a normal laptop screen.  What do you mean?

Could we just add some shiny window tint foil to normal laptop screens to achieve this effect?

This is super interesting, I&#x27;m about to undergo a small DIY smart mirror project and wanted touchscreen capabilities but all the hardware solutions seemed impractical and expensive. This might be a super easy way to make that work.

I normally hate touchscreen computers, but this project and its implementation are really cool.

Incredible. And here I thought a good assortment of lollies at the local corner store was the best $1 could do

I bought a case in India for my Kindle Paperwhite which came with a stylus [1]. That stylus works, and works quite well with my 2015 MBP.[1] - <a href="http:&#x2F;&#x2F;www.proelite.co.in&#x2F;Amazon-Kindle&#x2F;All-Amazon-Kindle-10th-Generation-2019-Slim-Smart-Flip-case-Cover--Deep-Blue--with-Stylus-Pen--Not-Fit-Paperwhite--id-3984513.html" rel="nofollow">http:&#x2F;&#x2F;www.proelite.co.in&#x2F;Amazon-Kindle&#x2F;All-Amazon-Kindle-10...</a>

Some accompanying software would definitely be required so that it ignores&#x2F;reduces the velocity factor, and has a way to differentiate movement with pen-up vs movement with pen-down.

Is anyone found a stylus that works with the macbook&#x27;s trackpad? The usecase is to be able to draw some diagrams during video calls&#x2F;screencasst.Trying to avoid the wacom&#x2F;tablet path to keep it lightweight :)

Pure brilliance. While many commenters seem to focus on computer applications this could be very viable solution for industrial environments for touch controls.

Turning a MacBook into a Touchscreen with $1 of Hardware (2018)