Tag Archives: kinect

Yandex, Russia’s ‘Homegrown Google’, Looks At Gesture-Based Interfaces To Power Apps

Yandex gesture social TV interface

Russian search giant Yandex has collaborated on developing an experimental gesture-based interface to explore how similar technology could be incorporated into future social apps and mobile products. The company offers digital services beyond search already, launching and expanding mapping services and translation apps, for instance, in a bid to drive growth as its domestic search share (60.5% as of Q4 2012) has not grown significantly in recent quarters. Future business growth for Yandex looks likely to depend on its ability to produce a pipeline of innovative products and services — hence its dabbling with gestures.

Yandex Labs, the division that came up with its voice-powered social search app Wonder (an app that was quickly blocked by Facebook), has been working with Carnegie Mellon University on a research project to create a gesture-based social interface designed for an Internet-connected TV. The interface, demoed in the above video, pulls in data from Facebook, Instagram and Foursquare to display personalised content that is navigated by the TV viewer from the comfort of their armchair using a range of hand gestures.

Here’s how Yandex describes the app on its blog:

The application features videos, music, photos and news shared by the user’s friends on social networks in a silent ‘screen saver’ mode. As soon as the user notices something interesting on the TV screen, they can easily play, open or interact with the current media object using hand gestures. For example, they can swipe their hand horizontally to flip through featured content, push a “magnetic button” to play music or video, move hands apart to open a news story for reading and then swipe vertically to scroll through it.

The app, which was built on a Mac OS X platform using Microsoft’s Kinect peripheral for gesture recognition, remains a prototype/research project, with no plans to make it into a commercial product. But Yandex is clearly probing the potential of gestures to power future apps.

Asked what sort of applications it believes could be suitable for the tech, Grigory Bakunov, Director of Technologies at Yandex, said mobile apps are a key focus. “Almost any [Yandex services] that are available on mobiles now: search (to interact with search results, to switch between different search verticals, like search in pictures/video/music), probably maps apps and so forth [could incorporate a gesture-based interface],” he told TechCrunch when asked which of its applications might benefit from the research.

Bakunov stressed these suggestions are not concrete plans as yet — just “possible” developments as it figures out how gesture interfaces can be incorporated into its suite of services in future. ”We chose social newsfeeds to test the system [demoed in the video] as it can bring different types of content on TV screen like music listened by friends, photo they shared or just status updates. Good way to check all types in one app,” he added.

As well as researching the potential use-cases for gesture interfaces, Yandex also wanted to investigate alternatives to using Microsoft’s proprietary Kinect technology.

“Microsoft Kinect has its own gesture system and machine learning behind it. But the problem is that if you want to use it for other, non-Microsoft products you should license it (and it costs quite a lot), plus it has been controlling by Microsoft fully. So, one of the target was to find out more opened alternative with accessible APIs, better features and more cost-effective,” said Bakunov.

Yandex worked with Carnegie Mellon students and Professor Ian Lane to train gesture recognition and evaluate several machine learning techniques, including Neural Networks, Hidden Markov Models and Support Vector Machines — with the latter technique showing accuracy improvements of a fifth vs the other evaluated systems, according to Yandex.

The blog adds:

They [students] put a lot of effort in building a real training set – they collected 1,500 gesture recordings, each gesture sequenced into 90 frames, and manually labeled from 4,500 to 5,600 examples of each gesture. By limiting the number of gestures to be recognized at any given moment and taking into account the current type of content, the students were able to significantly improve the gesture recognition rate.

CamBoard Pico Demos What Kind Of Gesture Control Your Next Computer Could Have Built-In

Screen Shot 2013-03-26 at 7.55.35 AM

CamBoard Pico is German firm pmdtec’s next-generation gesture input reference device. We showed you before what it could potentially do to change the computer interface, and now there are a couple of new videos from the company showing how it’s working with middleware makers and what it can truly accomplish in practice in actual shipping products.

The gesture detection in these videos is impressive, and shows a solution that’s not only small enough to be incorporated into devices like notebooks, but also works at a sufficient distance that it’s actually usable, accurately, when you’re up close and working with said devices as you would normally.

Individual finger detection and the ability to use the CamBoard Pico tech to accomplish simple, practical things like switching between open apps. Unlike Kinect, it looks like you can use the CamBoard pico even from your standard typing position on a notebook computer, just by raising a finger while typing. That’s much, much more useful than gesture tech that requires a user to adjust themselves back from the screen, or even worse, stand up to interact with a computer, and much more likely to gain wide adoption, rather than acting as a sort of novelty.

The second demo video, which shows pmdtec working with Metrilus middleware, demonstrates more the general gesture sort of control we’ve come to expect from Kinect and similar technologies, but again, the distance and flexibility are impressive. I’m excited to see what the forthcoming Leap Motion controller can accomplish when it ships later this year (it seems to offer similar functionality and working distance), but pmdtec’s goals and sales strategy are very different.

It’s targeting original design manufacturers (ODMs), who in term will sell through to OEMs. That means that together with its middleware partners, pmdtec can sell these things directly to computer manufacturers, meaning when you buy a future Acer, Asus or Sony laptop, it’ll come with accurate gesture recognition tech onboard if this product catches on. With these new practical demonstrations of how that might be of use even with current operating systems and interfaces, that’s a pretty exciting prospect.

Leak points to “always on” ’Net connection, one-time installs for next Xbox

Newly leaked documentation accompanying the developer’s kit for the successor to the Xbox 360, codenamed Durango, is rekindling rumors that the new system will require disc-based games to be installed to a hard drive before being played.

The “Hardware Overview” included in the Xbox Developer Kit help files was obtained and published by VGLeaks, and it matches up closely with rumors leaked last month by SuperDaE, a shadowy source who once tried to sell a Durango development kit on eBay. The document discusses how Durango games will be distributed via Blu-ray disc (an upgrade from the Xbox 360′s DVDs) but says those games won’t be playable directly from that optical media. Instead “all games will be installed on the hard drive… disc media will be used for distribution, but during gameplay, games will not use content from the optical disc.”

This suggests that Microsoft would use some sort of online-activation code or other method to confirm that a single Blu-ray disc is not being installed to multiple systems (and thus place limits on the standard secondhand resale of used game discs). While the help file doesn’t address this directly, it does mention that the system will “always maintain a network connection so the console software and games are always current.” The document also suggests players will be able to start a game while installation to the hard drive is ongoing, a setup similar to the one announced recently for the PlayStation 4.

Read 3 remaining paragraphs | Comments

Hand Recognition, Gesture Support Landing On Microsoft Kinect For Windows Soon

kinect-for-windows-sensor

The Kinect is arguably Microsoft’s most important innovation of the past decade, and has done more for changing the nature of computer interaction than pretty much any other recently created input devices. Today, Microsoft Research has demoed how it’s about to get even better, with the addition of hand recognition complete with refined gesture support.

Microsoft is showing off the new features at its TechFest even this week at Redmond HQ, and The Verge has a video of the new capabilities in action. As you can see from the video, you can use pinch-to-zoom, as well as hand gripping motions that allow for panning and scrolling too. The technique employs machine learning to recognize the difference between an open and closed hand, which is then integrated into Windows games and apps.










The video shows navigating maps, as well as playing Jetpack Joyride in Windows 8, and shows how a simple close finger gesture can be recognized as a mouse click essentially, which would be a very handy tool to add to Kinect’s Windows-based powers. The upgraded features are destined for Windows soon (though no specific timeline was given), though sadly there’s nothing to announce yet for bringing the enhanced gesture detection powers to Xbox.

The Kinect will soon have its own fair share of competitors when it comes to gesture-based input for desktop computing. The Leap Motion Controller ships May 13 to pre-order customers, and promises to deliver extremely fine gesture detection on both Windows and OS X, to the point where it should work for virtual painting and drawing applications, with a module much smaller and without the same physical space requirements as the Kinect. The MYO armband from Thalmic labs is also attracting a lot of early attention for its own, body sensor-based gesture control powers.

Microsoft did some terrific ground breaking in this space, but now the company has keep pace with a number of hungry young startups eager to blow it wide open. These new feature additions for Kinect for Windows should help Microsoft keep the competition interesting.

Inside Microsoft’s Cauldron Of Ideas: From Kinect, Bing And Killing The Blue Screen Of Death, To Code That Can Learn, Pixels You Can Hold And Drugs Compiled From DNA

Screen Shot 2012-11-16 at 21.13.30

If Steve Wozniak is worried Microsoft is now more innovative than Apple, the root cause for that concern undoubtedly lies within Microsoft’s network of research labs. Dotted around the globe, from Redmond to India and Asia via the UK, these university-style research institutions are the quiet engines behind innovations such as the Kinect depth camera which translates human movements into computable gestures, and Xbox users’ movements into gameplay.

Another notable Microsoft product that its research arm has played a substantial role in developing is the Bing search engine — with researchers knuckling down to crack problems such as how to compute relevance and design the auction mechanisms underlying search advertising. Microsoft Research has also helped to improve the reliability of the Windows OS via the development of Microsoft’s Static Driver Verifier (which addresses the problem of trusting third-party software – and has made the Blue Screen Of Death a rarity, where once it was a running joke).

From the outside looking in, Microsoft’s research labs look like the jewel in the crown of a corporation founded ice ages ago, in technology terms, helping to ensure that, despite being the grand old daddy of tech — with a former sales chief for a CEO — Redmond continues to be a huge force to be reckoned with in many of the spheres in which it plays.

The labs are “the far seeing eyes of Microsoft,” says Andrew Blake, lab director of Microsoft Research Cambridge, giving the insider’s view. “Our job is to be a cauldron bubbling with ideas and the ideas are there to be plucked out at the right moment,” he tells TechCrunch.

“It’s sort of intrinsically difficult to predict what’s going to be important,  so that’s why you have the cauldron bubbling, because let a thousand flowers bloom, let’s just see what happens. You genuinely don’t know what the outcomes are going to be.”

Microsoft spent a whopping $9.8 billion on R&D in its 2012 fiscal year but Blake says the labs account for “a small fraction” of that. “We don’t publish our budget but it’s a small fraction of the total spending on research and development,” he says. “I wouldn’t know how to spend [$9.8 billion]!”

On a press visit to Microsoft’s Cambridge Research lab, we are shown a glimpse of the huge variety of research projects bubbling away underneath the quiet corporate facade however modest its budget: from projects using machine learning to harness the power of big data to make better predictions about the Earth’s climate; to research into new user interface mechanisms that blend the real and the virtual so you can ‘hold’ a 3D ball of pixels in your hand; to a PhD project recycling Kinect components to fashion a wrist-mounted glove-less finger-motion-capturing device (below); to multidisciplinary research looking at making biological cells programmable using computer software.

If there’s a unifying thread connecting all the diverse projects going on under the Microsoft Research umbrella, it’s the sheer variety of research work being undertaken. This is not a model of corporate research tightly tied to product teams and immediate business aims, as is the case with Research at Google – which has a stated goal to “bring significant, practical benefits to our users, and to do so rapidly within a few years at most.”

Microsoft Research is more akin to a university research institution, says Blake, a structure that he argues makes for a far healthier and more sustainable entity. ”It’s clear to us that for a healthy research lab you need to have a renewal mechanism,” he says. “If you simply take people who are used to doing research and being free thinkers and you put a yoke on them, like on the oxen, and have them driving the technology wagon, eventually they get tired and where are they going to get their refreshment from? Where are the new ideas going to come from? So that’s why we have this as an integral part of our structure — right in in our DNA is basic research, and publishing, and going to conferences, and free association with the academic community.”

Blake notes that he has recently finished organising an academic conference in his own area of expertise — computer vision — adding that: “We senior people in Microsoft research, we take our turn doing those things and we publish a lot in those conferences and we have researchers visiting us from other universities and we visit other universities. There’s a lot of that stuff going on which is not that different from what you’d see in a university.”

Of course there are important distinctions to a university. For one thing Microsoft Research is privy to vast quantities of business data — which it can use to its advantage as a research aid. Instead of having to build a mini datacenter, say, to test research into improving the efficiency of data centers, Microsoft Research staff can “go and talk to the people who run the Azure business any time they want and try their ideas out and see if they’re scratching the right itch,” as Blake puts it. (And yes, the lab is working on a research project aimed at improving datacenter efficiency.)

So researchers certainly have relationships with product teams at Microsoft — but products being developed by the business do not limit the research work being undertaken, according to Blake. Information and ideas flow both ways.

“We may get a product group saying look we have  got to develop this thing in a set time frame, are you going to help us? And mostly people are pretty keen to try and we find out whether we’ve got anything to help. The business goals come from the business; we are not business people here, we are researchers,” he says.

And then from the other direction: ”We go out there quite a lot and sort of sell our ideas [to the business] but it doesn’t bother us if the ideas aren’t taken up immediately because we kind of think maybe it’s not the right moment,” says Blake. “Business has its own cycles and  you can’t do everything in business; you have to focus on whatever is the issue of the day. So it doesn’t put us off if we’ve invented something that we think is great and the business is not quite what they need at that moment.”

In the case of Kinect, says Blake, the Cambridge lab responded to commercial pressure from the business to develop the product by drawing on relevant bits of (in some cases years-old) research to see if they could be made to, well, connect — and that research ultimately went on to form the technological foundation for the commercial product.

The Kinect people approached us and because we had ideas at our fingertips we were able to pluck one off the shelf.

“[Prior to the idea for Kinect] we were looking at all kinds of things speculatively, some of the things we never thought they would particularly make products,” says Blake. “But the Kinect people approached us and because we had ideas at our fingertips we were able to pluck one off the shelf – the one that we thought would fit – and it did. And the solution actually surprised us. We had these ideas at our fingertips. We didn’t think those ideas were good for this problem but then we were really under pressure, which we were because there was just a year to work with the Xbox team developing solutions, so we had to place a bet.

“We ended up putting some quite surprising things together but they were things that were in our background and that we had been playing with over years. It would have been no good if somebody had said play with those now. It has to be part of your research experience that you have all these things either at your fingertips or at least in the back of your mind.”

There is one clear influence the business has over the research labs: the type of researchers they choose to hire. “We’re probably not going to hire some analytical chemists because we can’t really see at the moment how that would really impinge on the business – not to say that it’s impossible — but we don’t go out to hire a lot of analytical chemists,” says Blake.

“We hire a lot of people around some of the core disciplines of computing and some of the fringe disciplines of computing and sometimes we go almost outside computing altogether — as with our computational science group, where the primary goal they’re doing is actually the science. But the link to the business is that they’re power users of computation tools, and often their users are stressing our systems so hard that new things get invented. So we have this cluster of areas where we hire expertise that is very broadly related to the business. But then we fire the starting gun and these guys go off and you don’t know what they’re going to come up with.”

Asked which of the current projects going on in the lab he considers most promising, Blake is unwilling to play favourites. “You’re asking me to choose between my favourite children – I cant possibly do that,” he jokes.

“A lot of the ability to do good research is not just deep analytical thinking, which is more how the public probably thinks of research, but with the exercise of good taste — it’s as much about what you choose not to look into, as what you choose to look into,” he says, echoing the Steve Jobs product mantra that ‘deciding what not to do is as important as deciding what to do’. “Opportunity costs, what looks promising, people use their gut instincts to choose things which they think are going to be exciting. That’s why it’s so critical that I hire the very best research staff because it’s that good taste that is one of the things that you’re bringing into the organisation — so I genuinely would find it very very hard to say what’s going to blossom.”

He is willing to touch on promising areas of research — machine learning being a discipline he believes will play an increasingly important role in building new generations of software systems. Machine learning techniques are already being used to build products — such as the Kinect gesture recogniser (which can determine whether you’re raising your elbow or your knee), and to power the Xbox’s recommendation engine for games, TV and movies (which crunches your viewing data to predict what else you might like). But in an age of big data and  increasing complexity, machine learning technology is becoming an imperative for more and more applications.

“One of the very early lessons from artificial intelligence is that programming intelligent behaviour is just too hard — you just can’t capture it,” says Blake. “What’s better is for the software to develop in the way that humans learn, the way animals learn: by example. You show them  things and those things get generalised and those generalisations become the software – you don’t actually write the software, not entirely. The critical bits get built automatically through these learning programs.

“We have a group here that does machine learning — it’s about one-fifth of the lab — and now those ideas are sort of spreading outside that group.”

In the future maybe what Microsoft will be in is software for generating biological structures, it’s too important for us to ignore.

Specifically, says Blake, machine learning researchers are collaborating with researchers who design programming languages — to explore how software can be developed that can learn and understand uncertainty. ”Now what we’re doing is writing programs which instead of just adding numbers together or dealing with strings actually reasons about probabilities and will estimate how likely things are,” he says. “That’s quite a fundamental capability that we’re pioneers in.”

Asked to look further afield, to consider what Microsoft might be in 10 or 20 years’ time, should it still be around by then, Blake is quick to point out there is no way to know exactly what lies ahead, however farseeing the lab’s eyes or deep and rich its cauldron of ideas. But he does point to the “interface between computing and biology” as a “fascinating area” — and one Microsoft Research is “very involved” with now.

The multidisciplinary nature of this work means researchers with computer science backgrounds are teaming up with biologists. Or, in the case of Microsoft Research principal researcher, Luca Cardelli, have switched their focus from designing programming languages to trying to use computational thinking as a way to unlock biological mechanisms like cell division.

“What Luca and his collaborators have done is they’ve opened up that mechanism a bit further to show a bit more of the detail. But the insight they’ve got has come from computational thinking, if you like, having computational processes and analogy available to express what the cell is doing. And extraordinarily they just published the theoretical paper and at the same time a practical paper. An experiemental paper came out which showed sort of exactly the same thing — but in an experimental setting — so that’s quite a landmark piece of work,” says Blake.

“In the future maybe what Microsoft will be in is software for generating biological structures; it’s too important for us to ignore. We have no idea at the moment whether it makes a business,” he adds. “Some of the things we’re investigating seem way off any kind of business, but who knows whether they might be part of Microsoft’s business in the future.

“I think it’s pretty clear that in 20 years time the intersection of biology and computing will be a big thing… It might be that people are designing drugs by writing programs. Designing them from the ground up and making them out of DNA. They’d just send the programs off to be compiled; the way they’ll do that is they’ll just send them across the web to someone who produces DNA.”

Designing fragments of DNA certainly feels about as far away from churning out the next iteration — or even the next generation — of consumer technology as you can imagine a technology company could be. But Microsoft Corporation is undoubtedly a far stronger, future-proofed business for having such a far-sighted, far-reaching focus.

Apple TV eat your heart out.

Kinect Fusion, Microsoft’s 3-D Modeling Technology, Is Coming To The Kinect For Windows SDK

6180.BUILD-kinct-demo.jpg-340x0

Kinect Fusion, the Microsoft Research project which uses the Kinect sensor to create 3D models of objects and environments, will be made available in the Kinect for Windows SDK, Microsoft announced today. No word yet on when exactly that integration will go live, however – Microsoft only confirmed it will come in “a future release” of the developer toolkit.

That same SDK was just updated in October, however, which brought a number of new features, including access to extended-range depth data beyond 4 meters – a necessary upgrade for something like Kinect Fusion to properly function, we should note.

Originally a research project from Microsoft’s lab in Cambridge, U.K., Kinect Fusion works by taking the incoming depth data in the Kinect for Windows sensor, and uses the sequence of frames to build a highly detailed 3-D map of either an object, or even an environment, like an entire room, for example. The tool averages the readings over hundreds or thousands of frames in order to achieve more detail than it would get in a single reading.

To capture the data, you can either move the Kinect sensor around the thing you’re capturing in 3-D, or you can move the object in front of the sensor. In a pretty nifty demo provided to The Verge back in December 2011, Microsoft’s Kevin Schofield explained the technology very simply, saying “we’re building a model of the world by taking lots of individual frames and putting them together.” Kinect Fusion also updates what it’s seeing and rendering on the fly, as new objects (or people, in the case of the demo), come into the frame. The technology, Schofield noted, involves a $150 Kinect sensor, while industrial versions of a similar system cost around $50,000.

In today’s blog post announcing the forthcoming SDK integration, Microsoft’s Senior Program Manager for Kinect for Windows, Chris White, points out that Kinect Fusion will make sense in a number of business scenarios where 3-D object model reconstruction, 3-D augmented reality, or 3-D measurements come into play. For example, 3-D printing, industrial design, body scanning, augmented reality, and gaming can take advantage of this type of functionality.

Microsoft gave a sneak peek of the Kinect Fusion technology to attendees at its recent BUILD 2012 conference, but wasn’t making promises as to SDK integration at the time.

The New Internet Explorer For Xbox May Be Clumsy, But Wait Til It Gets Kinected

miorityreport

Internet Explorer suddenly became one of the most widely available Internet TV browsers today as it starts rolling out to Xboxes. And early reports say its joystick-based interface is awful. Whatever. It’s v1. What matters is that Microsoft has united the world’s most popular motion sensor tech and one of the world’s most “popular” web browsers.

Wave hello to the future of the Internet.

Engadget’s Sean Buckley has a deep, hands on review of the Xbox’s build of Internet Explorer. A key flaw is that you have to use a free-floating, joystick-controlled cursor to type and click windows on screen. What would be better is the familiar “traditional locked-grid, hunt-and-peck” control, as Buckley calls it. That’s what we’re used to for console game keyboards and menu navigation.

You know what might be even better than that, and is well within reach? Motion navigation through Kinect — 20 million of which have been sold. It’s not available yet on Xbox’s Internet Explorer, but I bet it’s in the works. You can use voice controls over Kinect to navigate the “Web Hub” of recently visited and favorite sites. but motion navigation could be huge.

Imagine just waving or pointing at buttons to click them, swiping between photos, or pulling open a navigation menu. Depending on how accurate it gets, typing a few characters through Kinect’s motion sensor might not be so bad either. I’ve seen some impressive motion tech like this built on Kinect by Oblong Industries, who created the motion-controlled screens in the film Minority Report.

Kinect is one area where Microsoft seems firmly ahead of its competition. Niether Google nor Apple, both very interested in the Internet-connected TV market, have a popular motion sensing television input. With all the users, Kinect is getting data that’s surely making it better.

A truly laid-back, couch-ready web browsing experience is going to need motion detection. One day I imagine we’ll use our second-screens / phones for complex television controls, and everything else we’ll do with our bodies. Even if its not ready yet, and it starts out awkward, IE’s deployment to Kinect-equipped Xboxes means Microsoft is stumbling in the right direction.

[Image Credit: ripten]

Hacker Uses A Kinect To Help His Mom Email After A Stroke

kinecticons_dash-01

Here’s a heartwarming story for a Hackathon Saturday: Chad Ruble’s mother suffers from aphasia due to a stroke. She hasn’t been able to use a keyboard for years because she is simply unable to recognize text. In order to help her, he built a Kinect-enabled interface that lets her move her hand around a series of simple icons – happy, sad, upset, etc. – and other icons that signify degree.

After swiping around the screen a bit, she was finally able to send an email using a few simple hand motions. She was overjoyed.

The first step was coming up with a visual “dashboard” to help her compose simple messages. Each icon is associated with a specific emotion, which can then be qualified by an amount. I used a Kinect with the SimpleOpenNI library for Processing along with some gesture recognition code from Matt Richardson to track the position of my mom’s hand. I then used a sample Processing sketch from Daniel Shiffman to generate and send the email by using the green arrow button. The red “X” resets the screen.

It’s great to see hacks like these actually helping folks who haven’t been able to do certain things due to physical handicaps. The best technology is technology that changes people’s lives and this is one of the purest examples of that ideal I’ve seen in a while.

via Make