SAP d-shop’s Virtual House – A journey from Physical to Virtual

This post was originally posted on SAP d-shop’s Virtual House – A journey from Physical to Virtual.

Some time ago…our good friends from SAP d-shop Newtown Square (Namely John Astill et all) built a IoT House for SAP Insurance. This little house (hand made by the way) used an Arduino Nano, a bunch of sensors and LED lights…and…which is pretty cool by the way…a 3D Printed washing machine with a water sensor…and of course…it was and it is…IoT enabled.

We thought it was pretty cool…so we have one at our own SAP d-shop at Silicon Valley and it had become a key part in all our d-shop tours.

Then…some time later, our friends from HCP Marketing (Namely Joe Binkley et all) and Intel build a Smart Building. A really nice building…controlled by Amazon Alexa that used an Intel Galileo, some Arduinos as well as servos, lights, a solar panel and even a fan…everything again…IoT enabled…but also as you may have guessed…voice controlled…so you can send the elevator up and down…open or closed the doors and even send the whole building on emergency mode…gladly…we had keep it on the d-shop for quite some time and it’s another of our “wow” factor demos every time someone comes to visit…

Having these two available for us…slowly sparked the fire of innovation and creativity…why don’t we build a Virtual House that can be used on the Oculus Rift and it’s controlled by Alexa?

Not an easy thing…but we for sure love challenges…and thanks to our previous experience Unity3D and Alexa working together we already knew how to start…

The architecture is pretty simple… The Heroku server is just an echo server, so it will repeat everything we pass to it as a JSON response. Our Unity app is constantly checking the Heroku server to see if there’s a message to respond to. Of course, for this to work as intended, we need to setup a skill on Amazon Alexa just to update the server. So, when we say “open door”, then Alexa will send a command to the Heroku server and this server then will produce an “open door” message in JSON. Our Unity app will read the Heroku Server, and act accordingly by opening the door…of course, we don’t want this to happened all over…so after Unity executes the action it sent a null message to the Heroku server, so next time the JSON response is going to be null as well and Unity will simply wait for the next valid command.

If you want to take a sneak peak of how the Virtual House looks like…here are a couple of screenshots…but don’t forget to watch the video, J You will get the full experience -;)

Now…this project started as a “Project in a Box” (for internal only…sorry about that) which means…all the source code and explanations on how to build it from the scratch should be provided…but…for obvious reasons…that didn’t happened L So instead…we turned this into a “Product in a Box” meaning that (Sorry again…internals only) you can download the compiled application and simply edit the configuration file to have it running on your own J No source code is provided by obviously a nice email can get you that -;)

Grab it from here

Now…that I got your full attention…please watch the video J It’s a nice journey from the IoT House to the Virtual House passing by the Smart Building…

Now…you may wonder about the 3D Models used for this Virtual House…as you can see on one of the images, most of them were downloaded but some of them were developed in house J using Blender…Like the Amazon Echo, the 3D Printed Robot and name tags, the Amazon Echo and obviously the house itself J For some other things like the plants and tables…those were imported into Blender and “hand painted” as the textures were not available.

Now…something that we believe it’s pretty important…is to list all the Pain Points and lessons learned while developing this application…

Pain Points and lesson learned:

• As this is a Product in a Box and not a Project in a Box, we’re not going to include the source code for this application, but what we’re going to do instead is let you know the pain points and lessons learned that came from this project.
• Unity uses the .NET Framework 3.5, which is already deprecated by .NET 4.0 so many things are not going to work simply because they haven’t been implemented…and why is that? Well…Unity uses Mono (which is .NET for Linux) and I guess they do it to maintain uniformity in all platforms. While Mono remains on .NET 3.5, Unity will not likely upgrade either.
• When loading scenes, the lighting gets all messed up…so you start in level one…more to level two and suddenly it looks like nighttime…the solution to that is simple…choose “Window à Lighting à Lightmaps”, uncheck the “Auto” checkbox and press “Build” to bake the light again.
• Coroutines are simply awesome. Normally, you can’t make your application wait or sleep…but by using Coroutines you certainly can…Coroutines are like threads.
• When using a light, make sure it’s turn off while the character is not in the room, because this will save some graphic processing and because even virtually…we need to be environment aware…
• Unity doesn’t have a wrap function or property for 3D Text…which is kind of problematic especially if you want to do a Twitter Wall…so your only chance is to build you own…although that’s not that hard…simply grab the incoming text, split it by space into an array…concatenate each word by checking first if the length of the string is lower than our threshold (which should be the maximum number of characters that fit where our 3D text is), is the string is bigger than the threshold, we simply add a carriage return (“\n”) before doing the concatenation.
• As your application grows you might feel the need to duplicate some assets, which is perfectly fine and doesn’t add too much processing (Especially if you create a Prefab and use that prefab), but don’t forget to assign them unique names, otherwise you’re going to have a headache if you application needs to interact with those assets.
• Sometimes you will download some 3D models from the web…other times you will create them using Blender…but don’t forget that sometimes just a simple sphere, cube or any other Unity primitive can work just fine by just using an image attached to it as its texture.
• When creating your Alexa skill…make sure not to make any spelling mistake…otherwise you will hit you head thinking why Alexa is doing what you’re asking her to do…
• When testing our your application both Debug.log() and Print() will become your best friends…nothing better than a printed value or message to realize what going wrong.
• When moving an object, always make sure to record its original position and then add the new value to that recorded position. Otherwise, something might provoke the values to go wrong…by having the original values recorded, you avoid having to recalculate the position but just call that variable and get things where they belong.
• When using 3D Text you will notice that even if you put another object in front of it…it will be always visible…which is not very likely…so we have two options…either create a shader to occlude it…or the easiest one…make the material that it’s in front of it transparent. That’s not perfect for all situations but at least will work.
• The biggest problems when making Unity and Alexa speak, is that when you ask Alexa to turn on the lights, she will respond “The lights are on” …but then if you ask a second time her response should be “The lights are already on” …to make this…we should need to use a Database or something to store state information…and when closing the application, we would need to clean up the states…while this might be doable…it’s a lot of work, and what happens if the application crashes? Would we need to go and reset the states manually? Not ideal…
• That leads me to the point of using the elevator…you can open the doors or sent it to any of the floors…for the main part…that’s easy…each floor is a scene, so you need to be on the first floor in order to make the elevator to floor two or three…but…what if you’re outside the elevator? You are on floor one…ask for floor three…and then you open the door…as your characters moves along with the elevator floor…when you open the door everything will look bad…solution? Simply using a cube without a mesh renderer, so it’s invisible…assign a collider with “is trigger” enabled…and validate that the player is colliding with the cube in order to make the elevator move…that way, even you ask for floor three and Alexa confirms that the elevator is going up…nothing will happen…when you open the door…we can assume that the elevator went down or up to your floor in order for you to hop in…just an illusion…but it works…
• Alexa doesn’t have an option to delay the re-prompt, so when exploring the Virtual House she will ask you “What else can I do for you?” and if we don’t respond the skill will just die…so we will need to wake her up again…that’s kind of sad due to the nature of the application…but nothing to be done unless Amazon releases a way on making the re-prompt to wait longer…
• As the whole Alexa-Unity3D relays on Heroku…expect some downtimes or responses from Alexa that are not actually replicated in the virtual world…might be an internet connection glitch or just Heroku glitch…

As I mentioned first…the environment gets affected by the weather…if it’s sunny…you will see a sunny clear sky…if it’s rainy you will see a dark and gloomy sky…and this involves using a Skybox…although not your regular Skybox…and what is a Skybox, anyway? Well…simply put…is a cube that covers your whole environment and has different images to simulate the environment…the problems is that the regular Skybox only allows you to assign six sides…which is of course not likely…you need to use a twelve side Skybox…then you can assign sunny image and also cloudy images…that way when checking the weather you can modify the luminosity and that will also affect the Skybox as it will use one or the other giving that nice effect on reflecting the outside weather…

Greetings,

Blag.

Development Culture.