What a difference a new server can make

Every now and then, it’s good to have a clean out huh.. 2-3 seconds response time (!) down to 200ms. Not quite sure what happened with that old box, but hey. Fresh installs FTW.

VPS Hosting revisited

So two years ago I started up a linode – yep, I’m still with them for my LAMP stack stuff. I really have no complaints, whatsoever. Since being with them they’ve bumped up my storage to 48GB from 40GB (for free), I’ve had one network problem (which affected everyone in my datacentre) about 18 months ago, but apart from that – nada. zip. Not had to contact support, performance from the VPS itself has been excellent. Awesome.

So now it’s time for me to re-evaluate my other box which is running Railo (and everything else under the sun). This one, running Centos5 is feeling fairly long in the tooth. One of the reasons I’m looking round for a new VPS provider for Railo is that for some reason, Railo has decided to start being slow – like, instead of the usual 500ms response time average, that average has slipped to 2-3 seconds. However, as a) it’s an unmanaged VPS and b) even if I contacted my VPS support they wouldn’t really have a clue about JVM tuning, I’m a little out of luck, bar reinstalling Railo/Java and hoping for the best.

As hosting Java is a smidge more complex than your average PHP hosting, (well, different anyway) ‘Googling’ it doesn’t alway help much.

So, I’ve decided to try Viviotech. Several things have lead me here: 1) very, very good reports from the CF community generally 2) my pre-sales questions were answered so stupidly comprehensively by Jordan Michaels, including appropriate linux distros and future plans for load balancing 3) the price is about right – not worryingly cheap, but not stupidly expensive either and 4) Jordan actually wrote the Railo installer, plus he’s actually answered my questions in the past on the Railo Google Group, which gives me some faith in the support structure.

I ordered it at 9am UK time: within 10 minutes I got a call from Viviotech (from the US, to my UK mobile!) checking/confirming the order. It must have been 1am their time (!). Once all confirmed, off they went. It did take them a day to setup, but I wasn’t in much of a rush, so no problem there. Once all the details arrived, it was just a matter of logging in.

Some nice things they do out the box – move SSH to a non-standard port, add an anti-brute force hack script, setup backups, configure some basic firewall stuff and install vsftp (which personally I don’t use, as sftp should be a minimum). I  asked for Railo to be installed as I was feeling lazy: the only thing which caught me by surprise was Railo running as root – on querying this, they do it for usability, and pointed me to a simple script to change the user Railo runs as. All the Tomcat/Apache connectors were setup and working too.

So so far all very good – there’s something nice in not being *completely* on your own like with fully unmanaged solutions such as Linode. The only thing I miss is those lovely graphs you get in the Linode dashboard (oh and that iPhone app which is amazing) – still I guess there’s always Munin.

You can find out more about Viviotech here

cfWheels Relationships (Part 5)

HasManycheckbox() is obviously only one way of updating these relationships. The other main way is via cunning uses of includePartial().

Let’s return to our contact’s email addresses – remember the contact ‘hasMany’ of those. We need a way to a) list their existing email addresses, b) add new ones (preferably within the same update form as the main contact details and c) edit or delete existing addresses.

When you need to display/update more information that what a checkbox could hold, includePartial() comes to the rescue. Whilst I’m demonstrating this with the email address field, which just has a single value, there’s nothing stopping you adding additional fields for this example.

In our Contact’s edit form, we can simply do this:

<fieldset>
<legend>Email</legend>
    #includePartial(contact.emailaddresses)#
    #includePartial("emailaddressNew")#
</fieldset>

And then create a partial called _emailaddress.cfm in the /views/contacts/ folder, containing:

<cfoutput>
#btextField(
    objectName="contact",
    association="emailaddresses",
    position=arguments.current,
    property="email",
    label="Email Address",
    class="span4"
)#
</cfoutput>

This should look pretty familiar – after all, it’s *almost* the same as a bog standard textField on a model, except we’re listing the association, and crucially, the postition.

So if we’ve got multiple email addresses, what does the generated HTML look like?

<label class="control-label" for="contact-emailaddresses-2-email">Email Address</label>
<input class="span4" id="contact-emailaddresses-2-email" maxlength="500" name="contact[emailaddresses][2][email]" value="joe3@bloggers.com" type="text">
<label class="control-label" for="contact-emailaddresses-3-email">Email Address</label>
<input class="span4" id="contact-emailaddresses-3-email" maxlength="500" name="contact[emailaddresses][3][email]" value="joe2@bloggers.com" type="text">

The [emailaddresses][3][email] part of the naming convention translates to “object[association][id][property]”.

And when submitted, with params.contact, we can see a nested struct:

If we’ve included the association when we do the contact updating, i.e (contact.update(params.contact);) these properties will automatically update, with no extra code.

That’s all good for existing entries, but what about adding new ones? Wouldn’t it be nice if we could just “reinclude” that partial to add a new form? If you try that, you’ll hit the ‘arguments.position’ error – as that partial is expected a numeric value for the position in the array that the property is in. Even if you try and set a tickCount as a unique identifier, you’ll hit snags too. This is probably the most annoying thing (currently, i.e wheels 1.1.8) – if someone’s got an elegant solution to this, please let me know 🙂 At the moment I fall back to either adding via a separate params struct, and testing for it’s existence, or doing it via Javascript.

Right, that’s all for now – thoughts on improving this greatly recieved.

cfWheels Relationships (part 4)

So far we’ve setup the database tables and the models, so now we need to start thinking about getting and updating this contacts data, including the associated nested properties.

Outputting a simple add/edit form should hopefully be familiar to you by now (if not I recommend checking out the cfwheels screencasts: http://cfwheels.org/screencasts ). When updating our nested properties, there are several methods, each with pros & cons.

Firstly, let’s have a look at HasManyCheckBox();

The hasManyCheckbox() helper is really, really useful. I’m going to use our technologies relationship, as this is just the sort of data which is designed to be displayed/updated in this fashion.

#hasManyCheckBox(
    objectName="Contact",
    association="contactTechnologies",
        keys="#contact.key()#,#id#",
        label=name)#

Now what does this actually do? The loop itself should be obvious – we’re looping over *all* the potential values for technologies – after all, we want to be able to list those entries which aren’t ticked too: but we need to be able to make sure that the entries which are present in the join table are properly ticked. ObjectName=”Contact” is our parent object – the one which holds the nested properties. Association is the name of the nested property itself. Next, you have the most important part, the composite key. Remember when I mentioned about the column key order in the database construction for our join table? *This* is why it’s important: the order of the keys here *MUST* reflect the order in the database schema. I’ve used contact.key() which allows me to do this on a new model (which won’t actually have a primary key per se – yet).

So what does this wonderful piece of code actually output?

<label class="checkbox" for="Contact-contactTechnologies-1-1-_delete">ColdFusion<input id="Contact-contactTechnologies-1-1-_delete" type="checkbox" name="Contact[contactTechnologies][1,1][_delete]" value="0" checked="checked" /><input id="Contact-contactTechnologies-1-1-_delete-checkbox" type="hidden" name="Contact[contactTechnologies][1,1][_delete]($checkbox)" value="1" /></label>

<label class="checkbox" for="Contact-contactTechnologies-1-4-_delete">CSS<input id="Contact-contactTechnologies-1-4-_delete" type="checkbox" name="Contact[contactTechnologies][1,4][_delete]" value="0" /><input id="Contact-contactTechnologies-1-4-_delete-checkbox" type="hidden" name="Contact[contactTechnologies][1,4][_delete]($checkbox)" value="1" /></label>

<label class="checkbox" for="Contact-contactTechnologies-1-5-_delete">HTML5<input id="Contact-contactTechnologies-1-5-_delete" type="checkbox" name="Contact[contactTechnologies][1,5][_delete]" value="0" /><input id="Contact-contactTechnologies-1-5-_delete-checkbox" type="hidden" name="Contact[contactTechnologies][1,5][_delete]($checkbox)" value="1" /></label>

<label class="checkbox" for="Contact-contactTechnologies-1-6-_delete">jQuery<input id="Contact-contactTechnologies-1-6-_delete" type="checkbox" name="Contact[contactTechnologies][1,6][_delete]" value="0" /><input id="Contact-contactTechnologies-1-6-_delete-checkbox" type="hidden" name="Contact[contactTechnologies][1,6][_delete]($checkbox)" value="1" /></label>

etc. As you should be able to see from the above, there’s actually some fairly complex stuff going on: imagine it as cfWheels by default adding all the associations & relationships, then deleting all the relationships, then by checking a checkbox, you save that association from being deleted. Looking at the dumped params struct makes this more obvious:

The ‘1,1’, ‘1,2’ struct keys are the composite keys: 1 representing the contactid, and the second representing the technology (in this case). If the checkbox is checked, then the _DELETE sub struct returns false.

What’s nice about this approach is that you don’t have to do anything extra; when submitting and creating the final contact object, you just have to make sure you include the associations.

‘Why that’s awesome!’ I hear you say. Yes, yes it is. BUT. It doesn’t help us in a few areas.

1) We can’t add new values to the list easily – i.e I can’t add a new technology
2) We can’t edit the associated properties easily – i.e I can’t edit an existing technology

Of those two points, I would argue editing existing properties from within the contact update/add form is perhaps not the place for it – these properties could apply to other contacts, so pushing them out into their own form is probably better from a user perspective – there’s then no chance of the user thinking ‘oh I’ll just change ColdFusion to Railo’ and then the horror when all other contacts then list Railo too.

Adding new values to the technology list is however something I see happening. To do this, I’ve found the easiest solution is to just create a new form field tag, and then test for the values when updating the contact object.

This means when we submit our update or create request, we need to test for the existence of the field, return the created key, and then add the association manually.

I’ve found the easiest way to do this (so far) is to add it to a new private function: So our update function calls our new checking function ‘checkForNewTechnologies()’:

contact.update(params.contact);
	if(contact.hasErrors()){
         renderPage(action="edit");
	}
        else {
	checkForNewTechnologies();
	 flashInsert(success="The contact was updated successfully.");
	 redirectTo(action="view", key=contact.id);
}

Which looks like:

<cffunction name="checkForNewTechnologies" access="private">
<cfscript>
  if(structkeyexists(params, "newtechnology") AND len(params.newtechnology) GT 2) {
		nTechnology=model("technology").create(
			name=params.newtechnology
		  );
		nTechnologyAssociation=model("contacttechnology").create(
			contactid=contact.key(), technologyid=nTechnology.key()
		 );
  }
</cfscript>
</cffunction>

Obviously, this is just one way of many to do it. You could add the new technology via Ajax, return the generated ID, then replicate the form HTML via JS to pass the new value in along with the other checkboxes: this is quite a cumbersome technique as you rely on recreating the wheels generated HTML manually, but might have it’s uses.

cfWheels Relationships (part 2)

Now it’s just a matter of representing these relationships in cfWheels so we can create/update/delete some data.

models/Contact.cfc

<cfcomponent extends="Model" output="false">
    <cffunction name="init">
		<cfscript>
                // Model init functions
		property(name="firstname", label="First Name");
		property(name="lastname", label="Last Name");

		//Calculated Properties
		property(name="fullname", sql="CONCAT(firstname, ' ', lastname)");

		// Relationships
		hasMany(name="emailaddresses", dependent="deleteAll");
		hasMany(name="contacttechnologies", dependent="deleteAll");
		hasMany(name="contactcompanies", dependent="deleteAll");

		// Nested properties
		nestedProperties(associations="
                      emailaddresses,
                      contacttechnologies,
                      contactcompanies", allowDelete=true);

        </cfscript>
    </cffunction>
</cfcomponent>

So I’ve done a couple of things here worthy of note: I’ve set some default values for the firstname and lastname properties – namely the label name: I want this to display as ‘First Name’ rather than the default ‘firstname’ which is what the label=”” attribute is for. I’ve also added a very simple calculated property, of fullname, which takes the first and last names and puts them together at a database level. You can imagine how this can become very useful indeed – if I had prefix, suffix, middlename columns, I could build up various additional properties using those too.

Lastly, you’ll see the relationships & nested properties; emailaddresses has the ‘dependent=”deleteAll” attribute, which means when I delete a contact, the dependent email addresses will also be deleted. As the other relationships are also dependent to that contact, they’ll be deleted too – but bear in mind, this won’t delete the ‘source’ data, i.e the technologies or companies tables which is crucial as these are being used by other entities in the application. The nested properties call allows us to literally ‘nest’ models *within* the contacts model. More on this later.

models/Emailaddress.cfc

<cfcomponent extends="Model" output="false">
    <cffunction name="init">
		<cfscript>
        // Model init functions
		 belongsTo("contact");
        </cfscript>
    </cffunction>
</cfcomponent>

As this is our simplest relationship, an Email address is very easily represented with a simple ‘belongsTo’ call.

models/Technology.cfc

<cfcomponent extends="Model" output="false">
    <cffunction name="init">
		<cfscript>
		// Relationships
		hasMany(name="contacttechnologies");
        </cfscript>
    </cffunction>
</cfcomponent>

As this is ‘on the other end’ of the join for contact-technologies, we need to refer the technology model back to the join model.

models/Technology.cfc

<cfcomponent extends="Model" output="false">
    <cffunction name="init">
		<cfscript>
       	// Relationships
		belongsTo(name="contact");
		belongsTo(name="technology");
        </cfscript>
    </cffunction>
</cfcomponent>

And here is the join model – it ‘belongsTo’ both contact.cfc and technology.cfc and allows us to join the two.

models/Company.cfc

<cfcomponent extends="Model" output="false">
    <cffunction name="init">
		<cfscript>
		// Relationships
		hasMany(name="contactcompanies");
        </cfscript>
    </cffunction>
</cfcomponent>

models/Contactcompany.cfc

<cfcomponent extends="Model" output="false">
    <cffunction name="init">
		<cfscript>
       	// Relationships
		belongsTo(name="contact");
		belongsTo(name="company");
        </cfscript>
    </cffunction>
</cfcomponent>

Lastly, our company and contactcompany models.

Visualising the Models

I tend to find being able to visualise the data structure more useful than relying solely on code. What do these models look like if we just create a new one?

Let’s take the Contacts model, and simply call:

contact=model("contact").new();

The result?

As expected, but… what about our email addresses?  How do we get them into our nice new contact call?

We need to create a new instance of the emailaddress model, and pass it to the .new() call as a named argument. As there may be more than one email address, we need to put our email address model within an array, so cfWheels can keep track of multiple nested properties.

If we try:

newEmailaddress[1]=model("emailaddress").new();
contact=model("contact").new(emailaddresses=newEmailaddress);

We get:

So we now have an (empty) nested email address model within the new contact. Handy.

We can also extend this concept for the other relationships:

newEmailaddress[1]=model("emailaddress").new();
newContactCompany[1]=model("contactcompany").new();
newContactTechnology[1]=model("contacttechnology").new();

contact=model("contact").new(
	emailaddresses=newEmailaddress,
	contactCompanies=newContactCompany,
	contactTechnologies=newContactTechnology
);

Well, this is all very well, but what do these models look like with some data in them?

cfWheels Relationships (part 3)

Let’s look at an example call to a contact, but without any of it’s relationships.

contact=model("contact").findOne(where="id=1");

So, the simplest possible call – find me one contact row where id = 1.

Well, it’s good to see our createdAt field got autofilled, and our calculated property of fullname made it through too. deletedAt is null, which is also good. The above call would return false if deletedAt contained a value.

Let’s do the same call, but get those email addresses:

contact=model("contact").findOne(where="id=1", include="emailaddresses");

That’s better! Incidentally, if there were no email addresses, we’d get an empty array like so:

Let’s extend this to the other relationships:

contact=model("contact").findOne(where="id=1",
		include="emailaddresses,contactCompanies,contactTechnologies");

So here we can see the join table relationship nested properties too. Now, here’s the catch. How do we get the data from the otherside of the join?

We can’t do this, as we’re returning a single entity with objects:

contact=model("contact").findOne(where="id=1",
include="emailaddresses,contactCompanies(company),contactTechnologies(technology);

We *can* do it if we return this record as a query (which is the default for a ‘findAll’ call):

contact=model("contact").findAll(where="id=1",
include="emailaddresses,contactCompanies(company),contactTechnologies(technology),
returnAs="query");

However, whether you actually want to do this will be dependent on your application. Sometimes, this will make perfect sense, especially if you’re calling data in any sort of tabular fashion. But if you’re calling your model in order to populate a form (for example) then this complicates matters. More on ‘hasManyCheckBox’ and associated solutions later.

cfWheels Relationships (part 1)

No, this isn’t going to be a guide about getting on with your loved one (unless of course, your loved one is a certain open source coldFusion framework) but will specifically look at all the different types of relationships in cfWheels, and what they mean in terms of real world application design. I’m going to assume you have a working knowledge of cfWheels (objects/models/controllers/views/routes etc) as the documentation over at cfwheels.org is very good, and well worth digging into.

For this series, I’m going to take the old default example application of a ‘people’ database (or in this context, a contacts database).

Normalising Data

The old addage ‘DRY’ (or Don’t Repeat Yourself) applies just as much to your data, as it does to your controller & view code. I don’t consider myself a DBA by any stretch of the imagination, but it simply makes sense – don’t duplicate data (except in extreme circumstances – there I said it). With that in mind, let’s have a think about what this application should do, and how this might reflect our model conventions and model relationships.

The Contact model:

Well, at it’s core, a contacts directory needs, yes, you guessed it, a list of people. Let’s call them ‘Contacts’. At a mimimum, we’re going to need a contact.cfc model. This should contain any information which is unique to a contact, and has one value.

The table ‘contacts’ might look like this:

contacts
—————-
id [key, int, autoincrement]
firstname [varchar, 255]
lastname [varchar, 255]
createdAt [datetime]
updatedAt [datetime]
deletedAt [datetime]

Pretty simple – an id to reference with, firstname & lastname fields which we can add a calculated property to later to get a ‘fullname’ value, and three cfWheels specific fields, which will auto update – when it’s created, updated or deleted. Naturally, our id field will need to autoincrement, and be an integer.

So what about that contact’s data? Let’s assume we want to store the following data about this contact:

  • One or more email addresses, which are specific to that contact. This is a ‘one to many‘ relationship – one contact, multiple email addresses. It could be said a contact ‘hasMany‘ email addresses, and that those email addresses ‘belongTo‘ that contact.
  • A set of related technologies such as ‘mysql’, ‘php’, ‘coldFusion’, ‘cfWheels’. A contact might have many of these, and likewise, ‘mysql’ might need to refer to multiple contacts. So whilst a contact ‘hasMany’ technologies, they’re not exclusive to that contact.
  • A company relationship, such as ‘Adobe’, ‘Microsoft’, ‘Apple’ – we’d want to reuse these companies for other contacts too. Companies might also contain additional data such a postal address. We’d also want to store data about that relationship to that company – i.e a contacts postition, and be able to easily retrieve data about that company when looking at a contact.

Let’s take all these in turn, as each presents different challenges when referring back to our contact.

One or more email addresses: So a new table is called for to store our email addresses.

emailaddresses
——————-
id [key, int, autoincrement]
email [varchar, 255]
createdAt [datetime]
contactid [int]

As before, an id to reference with, an email column to store the actual value, and a createdAt so we know when it was added. To me, this data is less important to keep track of in terms of audit, so I’m ignoring the updatedAt/deletedAt fields. The contactid field here is the (excuse the pun) ‘key’ to the whole excercise. This ties in the email address to the contact, after we setup our model relationships later. This is probably the simplest relation type in this example.

A set of related technologies: We’ll need a table to store our ‘technologies’ – this will be a very simple table.

technologies
—————
id [key, int, autoincrement]
name [varchar, 255]

That’s all we need to store this, it’s essentially a glorified list of categories. But how do we store which contact has a technology, and which technologies refer to what contacts? We need a join table, which will simply be the referring id’s of the contact and the technology.

contacttechnologies
—————
contactid [key, int]
technologyid [key, int]

Note how I’ve named the table – and also the order of the columns. As “contact” has come first in the column name, I’ve made sure the column order reflects this – the reason why becomes obvious later on. As both columns are keys, this means we can’t have any duplicate rows – that’s a good thing – we don’t want random duplicate relationships popping up anywhere.

So this is what it looks like so far:

Company relationships

Our company relationships mean we need two things – one, a place to store our companies, and two, a join table to store the relationship in the same fashion as contacts – technologies as before.

companies (this is the same table structure as contacts for this example)
—————-
id [key, int, autoincrement]
name [varchar, 255]
createdAt [datetime]
updatedAt [datetime]
deletedAt [datetime]

contactcompanies
—————
contactid [key, int]
companyid [key, int]

This pattern should start to look a little familiar – it’s basically the same as the email address – contact relationship.

What does our db structure look like now?

Next, we’re going to look at representing these relationships within cfWheels.

Railo and Amazon S3

I don’t usually get the time to play around with Amazon’s cloud offerings, but I’ve started a project recently which has a need for CloudFront, Amazon’s Flash Media Server streaming solution.

I remembered that Railo 3.x and upwards has S3 support (which CloudFront uses as it’s storage) – I wasn’t really prepared for quite *how* easy this was to implement. As a test, all I wanted to do was build a list of files on a S3 bucket, and then construct some URLs for an MP3 playlist in flowplayer.

Firstly, you need to get your Amazon access keys (which I’ll assume you know how to do) via the AWS console.

Next, add this to your onApplicationStart() method (or on cfWheels, your events/onApplicationStart.cfm):

<cfscript>
application.s3.accessKeyId = "MYACCESSKEY";
application.s3.awsSecretKey = "MYSECRETKEY";
application.s3.host="s3.amazonaws.com";
</cfscript>

To get all the files in my S3 bucket, I can then simply do this:

<cfscript>
S3Key=application.s3.accessKeyID;
S3Sec=application.s3.awsSecretKey;
</cfscript>

<cfdirectory action="list"
directory="s3://#S3Key#:#S3Sec#@s3.amazonaws.com/BUCKETNAME/FOLDERNAME" name="s3Directory"
recurse="yes" type="file">

Then, for flowplayer, I needed that data to be looped out into a JS friendly format for the playlist (RTMP streaming uses the mp3: prefix, and I also need to strip out the secret/access key from the returned directory path)

<cfloop query="s3Directory">
<cfoutput>
{url: "mp3:#replace(directory, "s3://#S3Key#:#S3Sec#@/BUCKETNAME/", "", "all")#/#replace(name, '.mp3', '', 'all')#"}<cfif s3Directory.recordcount NEQ currentrow>,</cfif>
</cfoutput>
</cfloop>
</cfsavecontent>

Then in my flowplayer config, I can just add:

playlist: [#playlist#]

… as an option.

Nice and simple, and made v easy by Railo – thanks guys!

Build your own interactive map

The Oxford Internet Institute have been playing around with their own Visualization Library which is out there as open source. One of the things I’ve been doing recently is building the back end to a customiser, which takes your data and preferences and gives you a nicely compiled version.

The back end is written in bespoke ColdFusion, but am in the process of moving it over to cfWheels – having a framework with good documentation really helps when there’s more than one person working on it!

Anyway, it’s now in Beta, so give it a shot – http://api.oii.ox.ac.uk/InteractiveVisBuilder/ – as always, any feedback greatly appreciated!

Fun with isotope.js

Just released this – http://www.oxfordmartin.ox.ac.uk/labs/bigquestions – which uses the fab isotope.js library. A bit of fun / experiment – cfWheels powering the back end for that whole site; was stupidly easy to get wheels to return HTML via ajax to populate the grid.

Go have a play – some interesting videos there too.