Some
time ago…our good friends from SAP d-shop Newtown Square (Namely John Astill et
all) built a IoT House for SAP Insurance. This little house (hand made by the
way) used an Arduino Nano, a bunch of sensors and LED lights…and…which is
pretty cool by the way…a 3D Printed washing machine with a water sensor…and of
course…it was and it is…IoT enabled.
We thought it was pretty cool…so we have one at our own SAP
d-shop at Silicon Valley and it had become a key part in all our d-shop tours.
Then…some time later, our friends from HCP
Marketing (Namely Joe Binkley et all) and Intel build a Smart Building. A
really nice building…controlled by Amazon Alexa that used an Intel Galileo,
some Arduinos as well as servos, lights, a solar panel and even a
fan…everything again…IoT enabled…but also as you may have guessed…voice
controlled…so you can send the elevator up and down…open or closed the doors
and even send the whole building on emergency mode…gladly…we had keep it on the
d-shop for quite some time and it’s another of our “wow” factor demos every
time someone comes to visit…
Having these two available for us…slowly sparked the fire of
innovation and creativity…why don’t we build a Virtual House that can be used
on the Oculus Rift and it’s controlled by Alexa?
Not an easy thing…but we for sure love challenges…and thanks
to our previous experience Unity3D and Alexa working togetherwe already knew how to start…
The architecture is pretty simple… The Heroku server is just
an echo server, so it will repeat everything we pass to it as a JSON response.
Our Unity app is constantly checking the Heroku server to see if there’s a
message to respond to. Of course, for this to work as intended, we need to
setup a skill on Amazon Alexa just to update the server. So, when we say “open
door”, then Alexa will send a command to the Heroku server and this server then
will produce an “open door” message in JSON. Our Unity app will read the Heroku
Server, and act accordingly by opening the door…of course, we don’t want this
to happened all over…so after Unity executes the action it sent a null message
to the Heroku server, so next time the JSON response is going to be null as
well and Unity will simply wait for the next valid command.
If you want to take a sneak peak of how the
Virtual House looks like…here are a couple of screenshots…but don’t forget to
watch the video, J You will get the full experience -;)
Now…this project started as a “Project in a Box” (for
internal only…sorry about that) which means…all the source code and
explanations on how to build it from the scratch should be provided…but…for
obvious reasons…that didn’t happened L So instead…we turned this into a
“Product in a Box” meaning that (Sorry again…internals only) you can download
the compiled application and simply edit the configuration file to have it
running on your own J No source code is provided by obviously a nice email can get
you that -;)
Now…that I got your full attention…please watch the video J It’s a nice journey from the IoT
House to the Virtual House passing by the Smart Building…
Now…you may wonder about the 3D Models used for this Virtual
House…as you can see on one of the images, most of them were downloaded but
some of them were developed in house J using Blender…Like the Amazon Echo,
the 3D Printed Robot and name tags, the Amazon Echo and obviously the house
itself J For some other things like the
plants and tables…those were imported into Blender and “hand painted” as the
textures were not available.
Now…something that we believe it’s pretty
important…is to list all the Pain Points and lessons learned while developing
this application…
Pain Points and lesson
learned:
As this is a Product in a Box and not a Project in a Box,
we’re not going to include the source code for this application, but what we’re
going to do instead is let you know the pain points and lessons learned that
came from this project.
Unity
uses the .NET Framework 3.5, which is already deprecated by .NET 4.0 so many
things are not going to work simply because they haven’t been implemented…and
why is that? Well…Unity uses Mono (which is .NET for Linux) and I guess they do
it to maintain uniformity in all platforms. While Mono remains on .NET 3.5,
Unity will not likely upgrade either.
When
loading scenes, the lighting gets all messed up…so you start in level one…more
to level two and suddenly it looks like nighttime…the solution to that is
simple…choose “Window à Lighting à Lightmaps”, uncheck the “Auto” checkbox and press
“Build” to bake the light again.
Coroutines
are simply awesome. Normally, you can’t make your application wait or sleep…but
by using Coroutines you certainly can…Coroutines are like threads.
When
using a light, make sure it’s turn off while the character is not in the room,
because this will save some graphic processing and because even virtually…we
need to be environment aware…
Unity
doesn’t have a wrap function or property for 3D Text…which is kind of problematic
especially if you want to do a Twitter Wall…so your only chance is to build you
own…although that’s not that hard…simply grab the incoming text, split it by
space into an array…concatenate each word by checking first if the length of
the string is lower than our threshold (which should be the maximum number of
characters that fit where our 3D text is), is the string is bigger than the
threshold, we simply add a carriage return (“\n”) before doing the
concatenation.
As
your application grows you might feel the need to duplicate some assets, which
is perfectly fine and doesn’t add too much processing (Especially if you create
a Prefab and use that prefab), but don’t forget to assign them unique names,
otherwise you’re going to have a headache if you application needs to interact
with those assets.
Sometimes
you will download some 3D models from the web…other times you will create them
using Blender…but don’t forget that sometimes just a simple sphere, cube or any
other Unity primitive can work just fine by just using an image attached to it
as its texture.
When
creating your Alexa skill…make sure not to make any spelling mistake…otherwise
you will hit you head thinking why Alexa is doing what you’re asking her to do…
When
testing our your application both Debug.log() and Print() will become your best
friends…nothing better than a printed value or message to realize what going
wrong.
When
moving an object, always make sure to record its original position and then add
the new value to that recorded position. Otherwise, something might provoke the
values to go wrong…by having the original values recorded, you avoid having to
recalculate the position but just call that variable and get things where they
belong.
When
using 3D Text you will notice that even if you put another object in front of
it…it will be always visible…which is not very likely…so we have two
options…either create a shader to occlude it…or the easiest one…make the
material that it’s in front of it transparent. That’s not perfect for all
situations but at least will work.
The
biggest problems when making Unity and Alexa speak, is that when you ask Alexa
to turn on the lights, she will respond “The lights are on” …but then if you
ask a second time her response should be “The lights are already on” …to make
this…we should need to use a Database or something to store state
information…and when closing the application, we would need to clean up the
states…while this might be doable…it’s a lot of work, and what happens if the
application crashes? Would we need to go and reset the states manually? Not
ideal…
That
leads me to the point of using the elevator…you can open the doors or sent it
to any of the floors…for the main part…that’s easy…each floor is a scene, so
you need to be on the first floor in order to make the elevator to floor two or
three…but…what if you’re outside the elevator? You are on floor one…ask for
floor three…and then you open the door…as your characters moves along with the
elevator floor…when you open the door everything will look bad…solution? Simply
using a cube without a mesh renderer, so it’s invisible…assign a collider with
“is trigger” enabled…and validate that the player is colliding with the cube in
order to make the elevator move…that way, even you ask for floor three and
Alexa confirms that the elevator is going up…nothing will happen…when you open
the door…we can assume that the elevator went down or up to your floor in order
for you to hop in…just an illusion…but it works…
Alexa
doesn’t have an option to delay the re-prompt, so when exploring the Virtual
House she will ask you “What else can I do for you?” and if we don’t respond
the skill will just die…so we will need to wake her up again…that’s kind of sad
due to the nature of the application…but nothing to be done unless Amazon
releases a way on making the re-prompt to wait longer…
As
the whole Alexa-Unity3D relays on Heroku…expect some downtimes or responses
from Alexa that are not actually replicated in the virtual world…might be an
internet connection glitch or just Heroku glitch…
As I mentioned first…the environment gets affected by the
weather…if it’s sunny…you will see a sunny clear sky…if it’s rainy you will see
a dark and gloomy sky…and this involves using a Skybox…although not your
regular Skybox…and what is a Skybox, anyway? Well…simply put…is a cube that
covers your whole environment and has different images to simulate the
environment…the problems is that the regular Skybox only allows you to assign
six sides…which is of course not likely…you need to use a twelve side Skybox…then
you can assign sunny image and also cloudy images…that way when checking the
weather you can modify the luminosity and that will also affect the Skybox as
it will use one or the other giving that nice effect on reflecting the outside
weather…
Since a long time...I had the idea of making Unity3D and Alexa work together...however...other project kept me away for actually doing it...so...a couple of days ago...a conversation with a friend made me remember that I actually really wanted to do this...so I did :)
At first...I wasn't exactly sure how to do it...but then slowly the main idea came into my mind...what if Unity read a webservice that gets updated by Alexa? When the right command is parsed, then Unity will create the object and problems is solved...seems easy? Well...it actually is...
First things first...we need to create a small NodeJS webserver on Heroku...then...we need to install the Heroku Toolbelt...
Now...create a folder called node_alexa and inside create the following files...
Once you have that...log into your Heroku Toolbelt and write the following...
Heroku Toolbelt
cd node_alexa
git init .
git add .
git commit -m "Init"
heroku apps:create "yourappname"
git push heroku master
heroku ps:scale = web 0
heroku ps:scale = web 1
Your webservice is ready to rock :) You should be able to find by going to "http://yourappname.herokuapp.com/"
Now...this simple NodeJS powered webservice will serve as a simple Echo server...meaning...whatever you type will be returned as a json response...of course...if you type "empty" then the response will be a empty json...so the main idea here is that we can keep the last entered value...if you pass a command it will be called again when you don't pass any commands at all...so by calling it once...we can cal it multiple times without disrupting its value...
Next in line...will be to create our Unity app...
Create a new app and call it "WebService" or something like that...project name doesn't matter too much...
If the Hierarchy window select "Main Camera" and change the "Tranform" details like this...
Now, create a new "3D Object" -> "Cube" and name it "Platform" with the following "Transform" details...
After that, we might need to create four wall that will go around the platform...so create 4 "3D Object" -> "Cube" and name them "Wall 1", "Wall 2", "Wall 3" and "Wall 4"...
When everything is ready, your workspace should look like this...
Go to the project tab and create a new folder called "plugins" and then create a new C# file called "SimpleJSON"...inside copy the source code from here...this will allow us to use SimpleJSON to parse the JSON...
Now...create another folder called "Script" and inside create a new C# file called "MetaCoding"...or whatever you like...
MetaCoding.cs
using UnityEngine;
using System.Collections;
using System.Net;
using System.IO;
using SimpleJSON;
public class MetaCoding : MonoBehaviour {
int counter = 1;
IEnumerator DownloadWebService()
{
while (true) {
WWW w = new WWW("http://yourapp.herokuapp.com/?command");
yield return w;
print("Waiting for webservice\n");
yield return new WaitForSeconds(1f);
print("Received webservice\n");
ExtractCommand(w.text);
print("Extracted information");
WWW y = new WWW("http://yourapp.herokuapp.com/?command=empty");
yield return y;
print("Cleaned webservice");
yield return new WaitForSeconds(5);
}
}
void ExtractCommand(string json)
{
var jsonstring = JSON.Parse(json);
string command = jsonstring["command"];
print(command);
if (command == null) { return; }
string[] commands_array = command.Split(" "[0]);
if(commands_array.Length < 3)
{
return;
}
if (commands_array[0] == "create")
{
CreateObject(commands_array[1], commands_array[2]);
}
}
void CreateObject(string color, string shape)
{
string name = "NewObject_" + counter;
counter += 1;
GameObject NewObject = new GameObject(name);
switch (shape)
{
case "cube":
NewObject = GameObject.CreatePrimitive(PrimitiveType.Cube);
break;
case "sphere":
NewObject = GameObject.CreatePrimitive(PrimitiveType.Sphere);
break;
case "cylinder":
NewObject = GameObject.CreatePrimitive(PrimitiveType.Cylinder);
break;
case "capsule":
NewObject = GameObject.CreatePrimitive(PrimitiveType.Capsule);
break;
}
NewObject.transform.position = new Vector3(0, 5, 0);
NewObject.AddComponent();
switch (color)
{
case "red":
NewObject.GetComponent().material.color = Color.red;
break;
case "yellow":
NewObject.GetComponent().material.color = Color.yellow;
break;
case "green":
NewObject.GetComponent().material.color = Color.green;
break;
case "blue":
NewObject.GetComponent().material.color = Color.blue;
break;
case "black":
NewObject.GetComponent().material.color = Color.black;
break;
case "white":
NewObject.GetComponent().material.color = Color.white;
break;
}
}
// Use this for initialization
void Start () {
print("Started webservice import...\n");
StartCoroutine(DownloadWebService());
}
// Update is called once per frame
void Update () {
}
}
Once you have the code...simply attach the script to the Main Camera...
The basic concept for this script is pretty simple...We're creating "DownloadWebService" as an IEnumerator method so we can call it as a Coroutine...and that allow us to have a sleep as we want to give some time between calls...
This method will fetch our Heroku WebService looking for a "create" command...once it has it...it will parse the JSON response and split in 3...so we can have..."create", "blue" and "sphere"...this will call CreateObject which will then create a blue sphere...after we have done that...the coroutine will continue as simply send a new command to our WebService to clean the output...to make this work nicely...we want to give 5 seconds after we clean the webservice before trying to see if there's another "create" call...
And this call be will be done by our Alexa skill...so basically when saying "create blue sphere" on Alexa...she will be send the command to the WebService...update the message and our Unity app will grab it...do its work...and clean up the Webservice...the wait for Alexa to provide the next command...
So...to kind of wrap up...we need to create our Alexa skill...
First, we're going to create a Lambda function...so log in here...
Of course...I have everything already setup...so I'm going to create a dummy function just to show the steps...
Click on "Create Lambda Function" and you will be presented with this...
There's a bunch of course...so type in "Color" in the filter box...
Choose "alexa-skills-kit-color-expert"
Leave this as it is and press "Next"
Choose a name and a description...
Choose an existing role if you have it already...otherwise just create a lambda_basic_execution...then raise up Timeout to 10 seconds and leave everything else as it is...press "Next"...a confirmation window will appear...so just press "Create function"...
You will be presented with a screen where you can upload your source code (which will be doing later on) and an ARN number...which we need for the next step...
The following part deals with create the Alexa skill...so please follow along...and log in here...
Choose "Alexa Skills Kit"...and create a new skill...
Choose a name for you skill and the most important...choose an "Invocation Name"...which is what you're going to use tell Alexa to open you application...something like..."Alexa, open Sandbox"...click next...
On the Interaction Model tab we have two windows...fill this on "Intent Schema"...
This are all the commands that Alexa can understand...and yes...we could have used "Custom Slot Types" to make the code shorter...but...I have had the problems of not working pretty well with more than one slot...simply hit next...
Here, choose AWS Lambda ARN...and pick either North America or Europe depending on your physical location...the on the text box...simply copy and paste the ARN that you received from your Lambda function...
This will send you to the "Test" tab...but we don't want to and actually we can't use that yet...so go back to the "Skill Information" tab and you will find that a new field has appeared...
And that should be "Application Id"...copy this number and let's move on to the final step...
Create a folder called "Unity" and inside a folder called "src"...inside that folder copy this file "AlexaSkills.js"
We're going to use the "request" module of NodeJS...so install it locally on the Unity folder like this...
sudo npm install --prefix=~/Unity/src request
This will create a node_module folder with the request module on it...
Then, create a new file called "index.js"
index.js
var request = require("request")
, AlexaSkill = require('./AlexaSkill')
, APP_ID = 'yourappid';
var error = function (err, response, body) {
console.log('ERROR [%s]', err);
};
var getJsonFromUnity = function(color, shape, callback){
var command = "create " + color + " " + shape;
if(color == "thank you"){
callback("thank you");
}
else{
var options = { method: 'GET',
url: 'http://yourapp.herokuapp.com/',
qs: { command: command },
headers:
{ 'postman-token': '230914f7-c478-4f13-32fd-e6593d8db4d1',
'cache-control': 'no-cache' } };
var error_log = "";
request(options, function (error, response, body) {
if (!error) {
error_log = color + " " + shape;
}else{
error_log = "There was a mistake";
}
callback(error_log);
});
}
}
var handleUnityRequest = function(intent, session, response){
getJsonFromUnity(intent.slots.color.value,intent.slots.shape.value, function(data){
if(data != "thank you"){
var text = 'The ' + data + ' has been created';
var reprompt = 'Which shape would you like?';
response.ask(text, reprompt);
}else{
response.tell("You're welcome");
}
});
};
var Unity = function(){
AlexaSkill.call(this, APP_ID);
};
Unity.prototype = Object.create(AlexaSkill.prototype);
Unity.prototype.constructor = Unity;
Unity.prototype.eventHandlers.onSessionStarted = function(sessionStartedRequest, session){
console.log("onSessionStarted requestId: " + sessionStartedRequest.requestId
+ ", sessionId: " + session.sessionId);
};
Unity.prototype.eventHandlers.onLaunch = function(launchRequest, session, response){
// This is when they launch the skill but don't specify what they want.
var output = 'Welcome to Unity. Create any color shape by saying create and providing a color and a shape';
var reprompt = 'Which shape would you like?';
response.ask(output, reprompt);
console.log("onLaunch requestId: " + launchRequest.requestId
+ ", sessionId: " + session.sessionId);
};
Unity.prototype.intentHandlers = {
GetUnityIntent: function(intent, session, response){
handleUnityRequest(intent, session, response);
},
HelpIntent: function(intent, session, response){
var speechOutput = 'Create a new colored shape. Which shape would you like?';
response.ask(speechOutput);
}
};
exports.handler = function(event, context) {
var skill = new Unity();
skill.execute(event, context);
};
This code is very simple...because it mostly a template...you simply copy it...change a couple of things and you're ready to go...
Basically when you say "Alexa, open Unity"...she will listen for your requests...so you can say "create green cube"...so will call our Heroku WebService and the wait for another command...if you doesn't speak to her again...she will prompt you to say something...if you say "Thank you" she will politely deactivate herself...
And that's pretty much it...once Alexa send the command to the WebServer...our Unity App will read and act accordingly...creating whatever shape and color you requested...nice, huh?
But of course...you don't believe, don't you? It can't be that simple...well...yes and no...it's simple...but I took all the pain point and provide you with the nice and clean set of instructions...
So...here's how it looks like when you run the Unity app...
And here the action video...
Hope you like it...and stay tuned...because for me this was only a proof of concept...the real thing will become my next full time project...
This is my first blog of the year…so I want it to be something really nice and huge -:) You know how much I love the R Programming Language…but I also love other technologies as well…so taking a bunch of them and hooking them up together is what really brings me joy.
Now…you may be wondering about the blog title…”There’s a party at Alexa’s place”…well…I wanted it to describe the blog in a funny way…so let’s see what we’re going to build -;)
Got any idea? Basically…we’re going to use Amazon Alexa as our UI…when we ask a command…we’re going to call a NodeJS Server on Heroku (Which BTW has a PhantomJS client installed)…this NodeJS will call an R Server on Heroku (Using the Rook Server)…and this R Server is going to call HANA Cloud Platform to get some Flights information and generate nice graphics that are going to be returned to the NodeJS Server which is going to call our web browser to display the graphic generated by the R Server…of course…by using PhantomJS were going to read the generated web page on the browser and this will be sent back to Amazon Alexa so she can read out the response…interesting enough for you? I hope -:) I took me more than two weeks to get all this up and running…so you better like it -:P
So…let’s go in some simple steps…
GET A HANA CLOUD PLATFORM ACCOUNT
You should have one already…if not…just go here to create one…
Then…we need to download the HANA Cloud Platform SDK extract it and modify the file tools/neo.sh on line 57…
Instead of this…
javaExe="$JAVA_HOME/bin/$javaCommand"
Use this…
javaExe="$JAVA_HOME"
Why? Well…it will make sense later on…or maybe it will make sense now If you have the SAP HANA Client installed…otherwise download it from here take a note that will need to copy the ngdbc.jar file…
GETTING THE DATA THAT WE'RE GOING TO USE
As always…in almost all my blogs…we’re going to use tables from the Flight model…which of course…doesn’t exist on HANA Cloud Platform…
The easiest way (at least for me) was to access an R/3 server…and simply download the tables as XLS files…convert them into CSV files and upload them into HCP…
And BTW…for some weird reason my R/3 didn’t have American Airlines listen on the SCARR table…so I just added it -;)
Now…if you don’t have access to an R/3 system…then you can download the tables in CSV format from here
CREATE THE R SERVER ON HEROKU
If you don’t have the Heroku Tool Belt installed…then go and grab it…
Steps to install R on Heroku with Graphic Capabilities
mkdkir myproject && cd myproject
mkdir bin
echo “puts ‘OK’ > config.ru
echo “source ‘http://rubygems.org’\n gem ‘rack’” > Gemfile
#Open your project folder and modify the Gemfile to replace the “\n” with an actual break line…
bundle install
git init . && git add . && git commit –m “Init”
heroku apps:create myproject –stack=cedar
git push heroku master
#Copy and paste the content of my installR.sh into the /bin folder of your project
git add . && git commit –am “message” && git push heroku master
heroku ps:scale web=0
installR.sh
#!/bin/bash
function download() {
if [ ! -f "$2" ]; then
echo Downloading $2...
curl $1 -o $2
else
echo Got $2...
fi
}
set -e
r_version="${1:-3.2.3}"
r_version_major=${r_version:0:1}
if [ -z "$r_version" ]; then
echo "USAGE: $0 VERSION"
exit 1
fi
basedir="$( cd -P "$( dirname "$0" )" && pwd )"
# create output directory
vendordir=/app/vendor
mkdir -p $vendordir
# R
download http://cran.r-project.org/src/base/R-$r_version_major/R-$r_version.tar.gz R-$r_version.tar.gz
tar xzf R-$r_version.tar.gz
# build R
echo ============================================================
echo Building R
echo ============================================================
cd $basedir/R-$r_version/
./configure --prefix=$vendordir/R --with-blas --with-lapack --enable-R-shlib --with-readline=no --with-x=yes
make
cd /app/bin
ln -s R-$r_version/bin/R
rm R-3.2.3.tar.gz
rm -rf erb gem irb rake rdoc ri ruby testrb
rm ruby.exe
cd /app/bin/R-$r_version
rm -rf src
rm Make*
rm -rf doc
rm -rf tests
rm README ChangeLog COPYING INSTALL SVN-REVISION VERSION
Now…we need to do a very important step -:) We need to install the totally awesome heroku-buildpack-multi from ddollar.
With this done…we will have all the missing libraries needed to compile R on the new Cedar Stack on Heroku and also…we will have a nicely installed R instance with Graphic capabilities…but of course…we’re not done yet…
Installing the R Libraries
#This will open R on Heroku…
R
#This will install the libraries with their corresponding dependencies
install.packages("Rook",dependencies=TRUE)
install.packages("Cairo",dependencies=TRUE)
install.packages("maps",dependencies=TRUE)
install.packages("forecast",dependencies=TRUE)
install.packages("plotrix",dependencies=TRUE)
install.packages("ggplot2",dependencies=TRUE)
install.packages("ggmap",dependencies=TRUE)
install.packages("rJava",dependencies=TRUE)
install.packages("RJDBC",dependencies=TRUE)
q()
All right…we’re almost there -;) The problem with Heroku is that is not writable…meaning that once you get disconnected…you will lost all your work -:(
So…we need to back it up and sent it somewhere else…I used my R Server on Amazon WebServices for this…
First…we need to compress the bin folder like this…
tar -cvzf bin.tar.gz bin
and then we need to save this file in our external server…
and of course after that we need it on our project folder…so we need to send it from our external server to our project folder, where will simply would need to uncompressed it…
So…let’s take some time to understand what’s going on with this code…we’re going to create a Rook server…which will allow us to host webpages from R…then, we’re going to use our hcp.sh script to get the password for our HANA Cloud Platform bridge…so we can get an JDBC connection to the database…from there we want to get a list of all the airports and also read the airports from a file detailed later (this airports file contains the geolocation of the airports). With this…we want to filter out the airports from HANA with the airports from the flight…so we don’t have any extra data…now…we have three choices…airports, US airports or carriers…the first one will generate a map of the world with all the airports as little red dots…the second one will generate a map of the US with the airports as little red dots but also showing the name of the cities…the last one will generate a geometric histogram with the details of the flights distance according to their carriers…later on…we’re going to read the information of the generated graphic to create a hexadecimal string of the graphic along with some information that Alexa should spell out…easy as cake, huh?
Procfile
web: bundle exec rackup config.ru
We want this R Server to be able to access HANA Cloud Platform…so let’s do that before we keep going…
With the location of Java…apply this command…
heroku config:set JAVA_HOME='/usr/bin/java'
Now…Copy the following files into your project folder…
“tools” folder from the HANA Cloud Platform SDK ngdbc.jar from SAP HANA Client
Also…create this little script which is going to allow us to connect to HCP…
var WebSocketServer = require("ws").Server
, http = require("http")
, express = require("express")
, request = require('request')
, fs = require('fs')
, app = express()
, arr = []
, msg = ""
, port = process.env.PORT || 5000
, childProcess = require('child_process')
, phantomjs = require('phantomjs-prebuilt')
, path = require('path')
, binPath = phantomjs.path;
app.use(express.static(__dirname + "/"))
var server = http.createServer(app)
server.listen(port)
var wss = new WebSocketServer({server: server})
var childArgs = [path.join(__dirname, 'phantom.js')]
var childStats = [path.join(__dirname, 'readphantom.js')]
app.get('/path', function (req, res) {
if(req.query.command == 'map'){
URL = "http://blagrookheroku.herokuapp.com/custom/summarize?airports=xyz&us_airports=&carriers=";
request(URL, function (error, response, body) {
if (!error) {
arr = body.split("/");
msg = "There are " + arr[1] + " airports around the world";
var bitmap = new Buffer(arr[0], 'hex');
var jpeg = new Buffer(bitmap,'base64');
fs.writeFileSync('Graph.jpg', jpeg);
res.redirect('/');
};
});
}else if(req.query.command == 'usmap'){
URL = "http://blagrookheroku.herokuapp.com/custom/summarize?airports=&us_airports=xyz&carriers=";
request(URL, function (error, response, body) {
if (!error) {
arr = body.split("/");
msg = "There are " + arr[1] + " airports in the US";
var bitmap = new Buffer(arr[0], 'hex');
var jpeg = new Buffer(bitmap,'base64');
fs.writeFileSync('Graph.jpg', jpeg);
res.redirect('/');
};
});
}else if(req.query.command == 'carriers'){
URL = "http://blagrookheroku.herokuapp.com/custom/summarize?airports=&us_airports=&carriers=xyz";
request(URL, function (error, response, body) {
if (!error) {
arr = body.split("/");
msg = "" + arr[1];
var bitmap = new Buffer(arr[0], 'hex');
var jpeg = new Buffer(bitmap,'base64');
fs.writeFileSync('Graph.jpg', jpeg);
res.redirect('/');
};
});
}else if(req.query.command == 'stat') {
childProcess.execFile(binPath, childArgs, function(err, stdout, stderr){
if(!err){
res.redirect('/');
};
});
}else if(req.query.command == 'readstat') {
childProcess.execFile(binPath, childStats, function(err, stdout, stderr){
if(!err){
res.write(stdout);
res.end();
};
});
}else if(req.query.command == 'bye'){
if(fs.existsSync('Graph.jpg')){
fs.unlink('Graph.jpg');
}
res.redirect('/');
}
});
wss.on("connection", function(ws) {
var id = setInterval(function() {
fs.readFile('Graph.jpg', function(err, data) {
if(!err){
ws.send(JSON.stringify("Graph.jpg/" + msg), function() { })
}else{
ws.send(JSON.stringify("Gandalf.jpg/No problem...I'm crunching your data..."), function() { })
}
});
}, 3000)
ws.on("close", function() {
clearInterval(id)
})
})
Let’s explain the code for a little bit and believe me…I’m far from being a NodeJS expert…this is really the first time I develop something this complex…and it took me a really long time and tons of research…so please try not to criticize me too much -:(
We’re going to create a express application that uses Web Sockets in order to refresh the browser in order to show the graphics generated by our R Server…it will also call PhantomJS to both create and read the generated web page so we can send it back to Alexa…
Here…we have six choices…map, usmap and carriers…the first three are going to call our R Server passing all parameters but leaving the ones that we don’t need empty…and just passing “xyz” as parameter…
When we got the response from R it’s going to be a long string separated by an “/”…which is going to be the hexadecimal string for the graphic along with the text intended for Alexa…Node will read the graphic…generated it and then refresh the browser in order to show it on the screen…
The stats option will call our PhantomJS script to simply read the page and create a new file with the Javascript part already executed…the readstat will read this information and extract the text that we need for Alexa…finally…bye will delete the graphic and the web socket will call the main graphic to be displayed on the screen.
Finally…the web socket is going to constantly check…every 3 seconds to see if there’s a graphic or not…and the display the related image…
index.html
<html>
<head>
<title>I'm a NodeJS Page!</title>
<div id="container" align="center"/>
<script>
var host = location.origin.replace(/^http/, 'ws')
var ws = new WebSocket(host);
ws.onmessage = function (event) {
var container = document.getElementById('container');
var data = JSON.parse(event.data);
data = data.split("/");
var url = data[0];
var msg = data[1];
container.innerHTML = '<img src="' + url + '"></br><p><b>' + msg + '</b></p>';
};
</script>
</head>
<body>
</body>
</html>
This one is going to be called by our express application and it will simply call the web socket to determine what it needs to display…it has some Javascript…that’s why we need PhantomJS to interact with it…
phantom.js
var page = require('webpage').create();
var fs = require('fs');
page.open('http://blagnodeheroku.herokuapp.com/', function () {
window.setTimeout(function () {
page.evaluate(function(){
});
fs.write('stats.html', page.content, 'w');
phantom.exit();
},4000);
});
Not the best and most describing name…but who cares -:P Anyway…this script will load the page…that is the express application…wait for 4 seconds for the Javascript to get generated and then create a web page called stats.html
readphantom.js
var page = require('webpage').create(),
address = "stats.html";
page.open(address, function (status) {
if (status == 'success') {
var results = page.evaluate(function() {
return document.querySelector('p').innerText.trim();
});
console.log(results);
phantom.exit();
}
});
This script will simply read the stats.html page and return the text that it’s located inside the “p” tag…dead simple…
SETTING UP ALEXA Creating the Lambda function
Now…we need to setup Alexa…so we can control everything via voice commands -:)
First…we need to go to Amazon Lambda and log in if you have an account…otherwise…please create one…and make sure you’re on the West Virginia region…
In the list of functions…look for color…
Choose the NodeJS one…Python has been included as well…but wasn’t when I started to work on this blog -:)
Here, just click next....
I already create the function…but you shouldn’t have a problem...
Basic execution role is more than enough…
This will provide a pop up window…simply press the “Allow” button and then “Create function”…we will include the source code later on…but notice the ARN generated number…because we’re going to need it on the next step…
Creating the Skill
Go to http://developer.amazon.com and log in…then choose Apps & Services --> Alexa --> Alexa Skills Set
Choose Alexa Skills Kit and fill the blanks...
As soon as we hit next an application number will be generated on a new field called Application ID. Grab this number as we’re going to need it for our application code.
The Interaction Model section is very important as here we’re going to define the “Intent Schema” and “Sample Utterances”…the first will define the parameters that we’re going to send to Alexa and the second is how we are going to call our application.
Our variable is going to be called “command” and it’s going to be a LITERAL…other types are NUMBER, DATE, TIME and DURATION. The intent is the method that we’re going to call in our code…
Sample Utterances
GetFlightsIntent airports from {around the world|command}
GetFlightsIntent airports from {united states|command}
GetFlightsIntent flight distance from {carriers|command}
GetFlightsIntent {thank you|command}
The test section help us to say commands and see how Alexa responds…but we’re not going to do that here…we’re going to test it using a real Alexa device -;)
Forget about the Publishing Information section unless you really want to publish your application…
Create a folder call Flights…Alexa_Party…or whatever you fancy…then create a folder call src and copy this file in there…calling it AlexaSkills.js
We’re going to need to install only one library….”request”…
sudo npm install --prefix=~/Flights/src request
This will create a folder called “node_modules” with the package in our project folder…then create a file called “index.js” and copy and paste the following code…
index.js
var request = require("request")
, AlexaSkill = require('./AlexaSkill')
, APP_ID = 'amzn1.echo-sdk-ams.app.8c0bd993-723f-4ab2-80b5-84402a7a59ce';
var error = function (err, response, body) {
console.log('ERROR [%s]', err);
};
var getJsonFromFlights = function(command, callback){
var msg = "";
if(command == "thank you"){
request("http://blagnodeheroku.herokuapp.com/path/?command=bye", function (error, response, body) {
if (!error) {
console.log("Done");
};
});
setTimeout(function() {
callback("thank you");
},2000);
}else if (command == "around the world"){
request("http://blagnodeheroku.herokuapp.com/path/?command=bye", function (error, response, body) {
if (!error) {
request("http://blagnodeheroku.herokuapp.com/path/?command=map", function (error, response, body) {
if (!error) {
request("http://blagnodeheroku.herokuapp.com/path/?command=stat", function (error, response, body) {
if (!error) {
request("http://blagnodeheroku.herokuapp.com/path/?command=readstat", function (error, response, body) {
if (!error) {
msg = body;
};
});
};
});
};
});
}
});
setTimeout(function() {
callback(msg.trim());
},15000);
}else if (command == "united states"){
request("http://blagnodeheroku.herokuapp.com/path/?command=bye", function (error, response, body) {
if (!error) {
request("http://blagnodeheroku.herokuapp.com/path/?command=usmap", function (error, response, body) {
if (!error) {
request("http://blagnodeheroku.herokuapp.com/path/?command=stat", function (error, response, body) {
if (!error) {
request("http://blagnodeheroku.herokuapp.com/path/?command=readstat", function (error, response, body) {
if (!error) {
msg = body;
};
});
};
});
};
});
}
});
setTimeout(function() {
callback(msg.trim());
},15000);
}else if (command == "carriers"){
request("http://blagnodeheroku.herokuapp.com/path/?command=bye", function (error, response, body) {
if (!error) {
request("http://blagnodeheroku.herokuapp.com/path/?command=carriers", function (error, response, body) {
if (!error) {
request("http://blagnodeheroku.herokuapp.com/path/?command=stat", function (error, response, body) {
if (!error) {
request("http://blagnodeheroku.herokuapp.com/path/?command=readstat", function (error, response, body) {
if (!error) {
msg = body;
};
});
};
});
};
});
}
});
setTimeout(function() {
callback(msg.trim());
},15000);
}
};
var handleFlightsRequest = function(intent, session, response){
getJsonFromFlights(intent.slots.command.value, function(data){
if(data != "thank you"){
var text = data;
var reprompt = 'Please say a command?';
response.ask(text, reprompt);
}else{
response.tell("You're welcome");
}
});
};
var Flights = function(){
AlexaSkill.call(this, APP_ID);
};
Flights.prototype = Object.create(AlexaSkill.prototype);
Flights.prototype.constructor = Flights;
Flights.prototype.eventHandlers.onSessionStarted = function(sessionStartedRequest, session){
console.log("onSessionStarted requestId: " + sessionStartedRequest.requestId
+ ", sessionId: " + session.sessionId);
};
Flights.prototype.eventHandlers.onLaunch = function(launchRequest, session, response){
// This is when they launch the skill but don't specify what they want.
var output = 'Welcome to Flights. ' +
'Please, say a command.';
var reprompt = 'Please, say a command?';
response.ask(output, reprompt);
console.log("onLaunch requestId: " + launchRequest.requestId
+ ", sessionId: " + session.sessionId);
};
Flights.prototype.intentHandlers = {
GetFlightsIntent: function(intent, session, response){
handleFlightsRequest(intent, session, response);
},
HelpIntent: function(intent, session, response){
var speechOutput = 'Get the information for airports and flights. ' +
'Please say a command?';
response.ask(speechOutput);
}
};
exports.handler = function(event, context) {
var skill = new Flights();
skill.execute(event, context);
};
Time to explain what I was trying to do here -:P
The handleFlightsRequest method will manage the response that Alexa will spell out for us…and inside this method we can find getJsonFromFlights which will take the command defined in the our Intent Schema. This function will call our NodeJS server for the following commands…”thank you” will simply call the bye command….”around the world” will call the bye, map, stat and readstat commands…”united states” will call the bye, usmap, stat and readstat commands…finally carriers will call the bye, carriers, stat and readstat commands…
After 15 seconds (Yep…I know it’s too much but there are a lot of processes going on) Alexa will get the response message and simply speak it to us -;)
That’s pretty much it…now…I can show some images before we jump into the video…
