Cloud Foundry provides developers with several choices of frameworks, application infrastructure services and deployment clouds. This approach to providing a platform as a service enables the fast creation of useful cloud applications that take advantage of polyglot programming and multiple data services. As an example of how this enables us to be more productive, I was able to use Node.js, Redis, and MongoDB to create an Activity Streams application in a short time, despite the fact that I mainly develop Web Applications in Ruby. Based on the developer interest after a demo of this application at NodeSummit 2012 and MongoSF 2012, I was inspired to do a 3 part blog series that fully details the creation, deployment, and scaling of this application on CloudFoundry.com.
The application is based on an interesting project I came across recently called “node-express-boilerplate” developed by @mape which is a full-featured but simple boilerplate Node.js app. Given my previous experience working on Social Networking software and Open Web Standards, I decided to morph this app into an Activity Streams sample application.
Initial Boilerplate Architecture without MongoDB or Redis PubSub
“Node-express-boilerplate” is a starter application written in Node.js which showcases how users can log into a website using Facebook, Twitter or GitHub, display basic profile info from those sites, and have real-time communication between server and clients. In this first blog post, we are going to review the components of the boilerplate application so that developers can learn how to use them for other applications and deploy locally as well as on Cloud Foundry.
A tour of the Boilerplate Components
Before we move to the setup, here are some code highlights explaining the various parts of the application.
Express MVC Framework
Working with the Express framework is very easy to do because its creator, @tjholowaychuk, has made it simple yet flexible to use while providing good documentation and screencasts.
You can use any web templating engine. For my project, I switched from using Embedded Javascript (EJS) to jade which is even terser than HTML Abstraction Markup Language (Haml).
// Settings
app.configure(function() {
app.set('view engine', 'jade');
app.set('views', __dirname+'/views');
});
Route Middleware
This middleware allows us to chain functions to pre-process the request. In this example I am showing how to invoke loadUser to read the current user, getDistinctStreams to get all the different topics and getMetaData to get the list of verbs and object types.
app.get('/streams/:streamName', loadUser, getDistinctStreams, getMetaData, function(req, res) {
asmsDB.getActivityStream(req.params.streamName, 20, function (err, docs) {
var activities = [];
if (!err && docs) {
activities = docs;
}
req.streams[req.params.streamName].items = activities;
res.render('index', {
currentUser: req.user,
providerFavicon: req.providerFavicon,
streams : req.streams,
desiredStream : req.session.desiredStream,
objectTypes : req.objectTypes,
verbs: req.verbs
});
});
});
Session Management
In the boilerplate application, sessions are stored in Redis, allowing us to scale to more than one instance. Not only does this application properly manage user sessions, it also supports single sign-on with GitHub, Facebook and Twitter via module everyauth.
Packaging Assets
In real world applications today, users expect immediate feedback and this cannot be done if your client is executing multiple http requests to render the app. This boilerplate sample uses @mape‘s module connect-assetmanager which minifies and bundles CSS and Javascript assets so the responses are very fast. This module is very flexible and allows pre and post manipulation of assets.
Here is an example which packages the js and the css files
var assetManager = require('connect-assetmanager');
var assetHandler = require('connect-assetmanager-handlers');
...
var assetsSettings = {
'js': {
'route': /\/static\/js\/[a-z0-9]+\/.*\.js/
, 'path': './public/js/'
, 'dataType': 'javascript'
, 'files': [
'http://code.jquery.com/jquery-latest.js'
, siteConf.uri+'/socket.io/socket.io.js' // special case since the socket.io module serves its own js
, 'jquery.client.js'
]
, 'debug': true
, 'postManipulate': {
'^': [
assetHandler.uglifyJsOptimize
, function insertSocketIoPort(file, path, index, isLast, callback) {
callback(file.replace(/.#socketIoPort#./, siteConf.port));
}
]
}
}
, 'css': {
'route': /\/static\/css\/[a-z0-9]+\/.*\.css/
, 'path': './public/css/'
, 'dataType': 'css'
, 'files': [
'reset.css'
, 'client.css'
]
, 'debug': true
, 'postManipulate': {
'^': [
assetHandler.fixVendorPrefixes
, assetHandler.fixGradients
, assetHandler.replaceImageRefToBase64(__dirname+'/public')
, assetHandler.yuiCssOptimize
]
}
}
};
var assetsMiddleware = assetManager(assetsSettings);
Client-Server Real Time Communication
Socket.io is used to send messages back and forth between the user-agent and the server. This saves us from having to do entire page refreshes to show new content.
In the example below you can see how we subscribe to different events and change the page content accordingly.
var socketIoClient = io.connect(null, {
'port': '#socketIoPort#'
, 'rememberTransport': true
, 'transports': ['xhr-polling']
});
socketIoClient.on('connect', function () {
$$('#connected').addClass('on').find('strong').text('Online');
});
var image = $.trim($('#image').val());
var service = $.trim($('#service').val());
var $ul = $('#main_stream');
socketIoClient.on('message', function(json) {
var doc = JSON.parse(json);
if (doc) {
var $li = $(jade.templates["activity"]({activities: [doc]}));
$ul.prepend($li);
}
if ($ul.children.count > 20) {
$ul.children.last.remove();
}
});
Here are the steps we performed to run this application as-is on Cloud Foundry
Setup
- Get a CloudFoundry.com account if you don’t have one yet.
- Install vmc command line tool to deploy to Cloud Foundry.
- Get Node.js running locally on your machine.
- Install version 0.6.8 or later. NPM is the package manager which will be included.
- Get Redis running locally.
- Create Apps on Facebook, Twitter and GitHub for prod and local environments and note the keys.
Configure
Clone @mape‘s repo or fork it and clone your fork and install the dependencies
$ git clone git://github.com/mape/node-express-boilerplate.git $ cd node-express-boilerplate
Edit package.json to include module cloudfoundry
{
"name" : "node-express-boilerplate",
"description" : "A boilerplate used to quickly get projects going.",
"version" : "0.0.2",
"author" : "Mathias Pettersson ",
"engines" : ["node"],
"repository" : { "type":"git", "url":"http://github.com/mape/node-express-boilerplate" },
"dependencies" : {
"cloudfoundry": ">=0.1.0",
"connect" : ">=1.6.0",
"connect-assetmanager" : ">=0.0.21",
"connect-assetmanager-handlers" : ">=0.0.17",
"ejs" : ">=0.4.3",
"express" : ">=2.4.3",
"socket.io" : ">=0.7.8",
"connect-redis" : ">=1.0.7",
"connect-notifo" : ">=0.0.1",
"airbrake" : ">=0.2.0",
"everyauth" : ">=0.2.18"
}
}
Install the dependencies locally.
$ npm install
Copy to siteConfig.js
$ cp siteConfig.sample.js siteConfig.js
Edit siteConfig.js to use environment variables and the cloudfoundry module. Changes are detailed here.
Update server.js to use the new siteConfig.js settings as seen here.
Set environment variables for all services. Example in bash:
$ export twitter_consumer_key=2SXwj3HcMHsdsdsL4uuUBdjShw $ export twitter_consumer_secret=UFamzEOAEhLUwewewDwwEoCI72hN0fl8 $ export facebook_app_id=5925695687264066 $ export facebook_app_secret=cce6f5edefa89f4686e5e036e3ea $ export airbrake_api_key=63340934f6b376a001eacfc660d06205 $ export github_client_id='92df9d93813ab234e1' $ export github_client_secret='fa64d10d3a02eee08d00cda3c2965caea2a4ce22'
Run locally
$ node server.js
Run on CloudFoundry
Install vmc if you have not already done so
$ sudo gem install vmc --pre
Deploy the app to Cloud Foundry
Specify you want Redis “redis-asms” bound to your app
$ vmc push --runtime=node06 --nostart
Would you like to deploy from the current directory? [Yn]:
Application Name: node-express-start
Detected a Node.js Application, is this correct? [Yn]:
Application Deployed URL [node-express-start.cloudfoundry.com]:
Memory reservation (128M, 256M, 512M, 1G, 2G) [64M]: 128M
How many instances? [1]:
Bind existing services to 'node-express-start'? [yN]:
Create services to bind to 'node-express-start'? [yN]: Y
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
What kind of service?: 5
Specify the name of the service [redis-9bea7]: redis-asms
Create another? [yN]: N
Would you like to save this configuration? [yN]: Y
Manifest written to manifest.yml.
Creating Application: OK
Creating Service [redis-asms]: OK
Binding Service [redis-asms]: OK
Uploading Application:
Checking for available resources: OK
Processing resources: OK
Packing application: OK
Uploading (305K): OK
Push Status: OK
Note that here I responded Yes to saving the configuration which will create a file called manifest.yml. A manifest.yml helps you quickly push and update apps. You can read more about it here.
Now you can run this command to add the keys from the services.
$ export APP_NAME=your_app_name $ vmc env-add $APP_NAME airbrake_api_key=your_key $ vmc env-add $APP_NAME github_client_id=github_id $ vmc env-add $APP_NAME github_client_secret=github_secret $ vmc env-add $APP_NAME facebook_app_id=fb_id $ vmc env-add $APP_NAME facebook_app_secret=fb_secret $ vmc env-add $APP_NAME NODE_ENV=production $ vmc env-add $APP_NAME twitter_consumer_key=twitter_key $ vmc env-add $APP_NAME twitter_consumer_secret=twitter_secret
To finish, run:
vmc start
And that’s it! You are up and running with the boilerplate app as seen here. Please note that the app may not work to spec on IE, but works on Firefox, Safari and Chrome.
Observations so far
node-express-boilerplate is a great starting point on which to build an activity stream engine given that it addresses:
- Robust real-time messaging between browser and server. Socket.io adapts to the protocols supported by the server and client
- Performance via Asset Bundling and Minification
- Provides SSO Support to major Social Networks
- Handles scalable session management via Redis
- Built on a great MVC framework
Also, it was good to see that @mape had abstracted the infrastructure details via the creation of a siteConfig.js. We enhanced this even further by using environment variables.
As you saw on the walkthrough all this was possible thanks to the open source community and using a robust platform as a service like CloudFoundry which had everything I needed. I was able to use @igo‘s “cloudfoundry” module to assist in parsing environment details in my app and thus made siteConfig.js even more straightforward.
In the next blog post, I will cover how to start the modification of this app into an Activity Stream engine. The tutorial will include how to create a Node.js module like activity-streams-mongoose and how to manage the persistance of the Activity Streams data on MongoDB as well as real time syndication across multiple app instances with Redis PubSub.
Signup for Cloud Foundry today, and start building your own cool app!
-Monica Wilkinson, Cloud Foundry Team
