We recently talked with Heapsylon’s CTO Mario Esposito about impressive gear that can interface among the physical world and video. Esposito also co-founded Heapsylon in 2009, which concentrates on wearable body sensing technology. Heapsylon, a Washington-based start-up, has launched Sensoria Fitness, which consists of mobile applications, the first e-textile pressure sensing smart sock, and Bluetooth anklet.
The mobile application monitor and guide you through real-time audio cues. The smart sock, completely made of fabric, can be washed or dried like a regular sock. The anklet magnetically snaps on the cuff of the sock and sends commands. Sensoria Fitness recently completed two successful campaign on IndieGoGo.
Esposito, a.k.a. “Product Delivery Ninja”, specializes in release to market (RTM) product releases and has founded other start-up companies including Kywix. Sensoria has expanded as a human interface device (HID) in the video game world, which, through a USB port, acts as an extended experience that does not require any drivers to install. Sensoria also empowers game developers to enhance their existing game experience as well as create new ones. Here is the information we discussed with Mr. Esposito:
Gear Diary: How do you develop the Sensoria Fitness system components (smart sock, anklet, and mobile application) for gaming?
Esposito: In general we try to build a platform…inventing something that can be adopted in multiple scenarios. We have a high range of opportunities. When I came to the idea for Sensoria Gaming, it was a no brainer for me. Instrumentation that records activity with games becomes very important, especially when shifting weight. Some can be healthy and many are informational.
Gear Diary: How do you approach these products?
Esposito: When we first look at a product we ask how use this thing. We look for a positive response and look at more scenarios. For example, we currently working on…are you familiar with the game Portal?
Gear Diary: Yes.
Esposito: …we’re putting dash and walking directly into game. Now you cannot really know if you are walking or not. We provide the action. We start by mapping all the (computer) keys with the sock – our human interface device or HID for short. Essentially when you support that standard, then you can communicate from anything. We are mapping each sensor to one key on the keyboard. We are also working within a game we can replicate snowboarding with our smart socks.
Gear Diary: Please describe your collaboration in-game development using the Sensoria Fitness products.
Esposito: I have been in the Xbox arena for some time. We love to spend time playing these games. Sometimes we take a break from development work or playing Xbox and go into hallway to discuss and try ideas. We started playing and had idea with my friend Mack integrating games with Sensoria. I play more arcade games while he plays more first person shooter (FPS) games. He’s more of a Halo guy.
Gear Diary: Do you have any personal stories about this detailed game development?
Esposito: I was looking at Dance Dance Revolution (DDR). My daughter was about 8. I remember she was coming to office often. During that time I was working to make this game work with the sock even two socks where we’re making a custom anchor for both. We were able to control the moves, which advances the game development to a level of control that they have wanted for a long time. On that development side, they need to know what’s going on; what’s the action – up all the way or slightly. We enable developers to create a more insightful experience, which has not changed much since I played it in a bar a long time ago. Reality gaming techniques have become so evolved. Sensoria is able to add more physicality into games. With the Nintendo Wii console, as you play DDR, the camera tells you if your posture is correct, but with our Sensoria smart socks, players can know if they are really standing the right way.
Gear Diary: What are other ideas are you developing and how did they arise?
Esposito: We are always looking into ways to change some software and enhance the experience from the garment point-of-view. All we do is make changes in functionality like how to receive signals –up, down, left, or right.
Another interesting thing we came across was the robotic ball Sphero, which I made a function when I walk the ball follows me. I also looked at running. When the heel went on ground too hard it changes color. Every time you hit ground in wrong way the color red appears, then green when it is correct. I would also run on treadmill with the Sphero ball. I would have it a cup container and keep running then watch what colors appeared.
I enjoy several video game scenarios. You are still looking at a 2D screen. You get different shades and experiences, but it’s still 2D. With remote-controlled (RC) technology like a toy car, we would put our foot on the ground to simulate stepping on the gas pedal. Now imagine brake and accelerator. The phone still talks (communicates) and can be used to hand steer car. Users get a dimensional experience and objects that moves around in a more sophisticated way.
We can develop an extension to console games like Xbox with integration with the phone. There are several options. Players can use sock to push the gas, which is a much better feeling than pushing the controller buttons or joystick. When you do that action you don’t have tactical response. You are moving the thumb. We use the foot, which gives us much more understanding of body movements.
Gear Diary: How do you feel Sensoria fitness compares with other fitness systems and video game systems?
Esposito: I will use an analogy. The Nike Fuel Band has few pitfalls. There is more technology employed by that device. It depends on how much movement you make. Every time you make a move you get fuel points and/or calories burned, even if you release fork from plate to eat where you are actually gaining weight or maybe the user is smoking while talking to buddy by the window. The user is not really getting healthy. The tech is meant for something else.
With joystick and other accessories, the control is integrated with the game. We try to get usable information to developer better than an accelerometer or counting steps. We base this information in reality with our technology. Like in DDR when you move your foot around, we want to know from where you start and how you land and if it is done correctly or not. With our added information, users get information from game system camera and our system, which can give them tips on how to actually move their body. They can move more towards the left and get that real-time feedback.
Gear Diary: What are some of your activities in your favorite game genres?
Esposito: I enjoy development in golf, which is one of my favorite ones because of the weight shift. Every time I move where and how standing is everything. Imagine Tiger Woods. If you have information that shows what Tiger Woods is really playing with the smart sock for a couple of hours and a game developer can record everything they need to know every time they do it. I can compare myself with Tiger Woods and get recommendations/feedback. Electronic Arts did something similar for basketball. They injected the real simulation of playing basketball. It’s expensive. We can empower many companies. Motion capture is a typical method, but instead of going that route games can be developed by the smart sock and one anchor where developers can build their own golf game with real-time feedback from users as they are playing and adopting the feedback.
Gear Diary: Do you have future plans for other specialized garments like body suits for motion capture in video games?
Esposito: Yes, we are working on other garments even body suits. The garment is the computer (as a concept). As an inventor, I envision what we should do. The sock and T-shirt talk to each other. The technology must work for you. It’s fortunate that most inventors are innovators. I try to think in terms of how this fits well in my lifestyle. I build something where people do not have to think about it. Every morning I wake up, get dressed then put on my socks and shoes. We inject something that is natural. Since you are putting smart sock on, whatever you do is collected. You get a little pain in back, then you see how to improve performance using data about what you have done. My grandma (bless her heart) tells me she gets hurt. Now she is 97 and says I have this pain…something is wrong. In reality it may have happened three weeks ago, but the memory of what happened leading up to the injury may be inaccurate. We also have to consider so many things that may have contributed to the injury in the past. This system provides people a way to travel back in time and track motions, pressure, acceleration, and force. The anklet gives information using an altimeter for variables like altitude and outside temperature. The GPS from the phone and other biometric data – all together users get a pretty good idea of what happened.
Gear Diary: How do you envision using the Sensoria Fitness components for first person shooter (FPS) games?
Esposito: Players could tap the smart sock to reload so you will always be shooting. We look at skiing and skateboards when simulate weight. A 3D camera gives skeletal information, but no one gives you the weight on ground like our system.
Gear Diary: How did your past experience factor into your current development projects?
Esposito: My experience with the Xbox One team was outstanding. I was fortunate work with many smart people who I deeply respect. I remember challenging ourselves with how precise we can be. In shifting weight, we rely on optical information with marginal error. With sensor fitness we can know exactly how players put the weight down while the camera tells you about posture towards screen. Sensoria can give you that information with the sensation for real. For surfing or anything where you stand, walk or do sports like bowling. The way you strike the ball – we start with the feet. The posture. The mechanics of the upper body. How you walk and put your foot down. The Sensoria has a strategic role in next wave of accessories. We augment information to develop a rich set of information, which gives users a better experience. I was just talking last week to one of my former Xbox One colleagues about it. To have the controller in hand, the camera looking at you, the speech recognition, and now feet. Users can really feel fully immersed.
Gear Diary: How do you address special situations and hardware issues like battery life?
Esposito: We have managed to address hardware batteries with something like a switch system. Why force people to “on is up” and “off is down”. Our idea for switch design is different. Once for on and twice for off. To switch off all together, the user does not have to think about it. If users are running then there is no pressure under feet or their feet are up in air, the system recognizes that. The event could be phone call, etc. It understands when people are using the device or not then powers off – not stand by mode. Given the device shape we have a few other “tricks” as of right now, which makes us pretty happy. I’m never happy to stop trying and always continue. I feel it can never be good enough.
The user does not like to charge anything. I want to reach the point where you harvest energy as you move. We also consider elderly and handicapped people. We considers if they normally have the device with them. What if they forget them? The device becomes useless. As they are walking, Sensoria Fitness components are attached on their body and become like an extension of their skin. There are two methods of understanding here: 1) prediction of their actions to learn over time how to arrange the experience in better way and 2) react to what’s happening in a much faster than any other system.
Many thanks to Mr. Esposito and check out Sensoria Fitness on YouTube.