Digitalistic

By their very nature Mashups are built on trust..

• You have to trust the API providers to keep that API up and running. Anyone that has done Twitter mashups know how much keeping an API up and running is worth.
• You have to trust the API providers to keep the API (somewhat) stable and to not remove features that you are depending on, or add features you really dont want. An example of this is all the mashups built on Google Maps, and we all know that there are way too many of those. Have the developers of those mashups really thought about what happens when Google puts ads on the Maps (and they will, they have already started in parts of the US)? I dont think so, the developers just trust Google to keep on going. If you have a commercial site using Google Maps you probably do not want ads for your competitors on those maps.
• If your mashup becomes a success you have to trust the API providers you are depending on not to change their terms of service so that your mashup suddenly becomes an illegal use of the API. Also, you have to trust the API provider not to use the fact that you are depending on their API for your business against you in a business negotiation.
• If you use web scraping you have to trust the site you scrape not to change to often (or at least trust your own ability to roll with the punches and update your scraping so it works with the new version of the site as well) etc. That is a lot of trust, and when mashups move from toys into real applications this becomes an issue.

All these things makes me a bit nervous, as I am a bit paranoid (note: this has not been clinically proven, I still think that THEY are out to get me). So how to mix a healthy bit of paranoia into my mashup building and get something good out of it all? What I do is that I always try to be aware of that I might have to switch API provider. Are you building a Twitter mashup? Why not also take a look at Jaiku’s API, or Pownce’s API (or Plurk or FriendFeed etc etc). You dont have to build your mashup so you can switch API provider in a matter of minutes, just be aware what else is out there so that you see the commonalities and don’t use to many features unique to one provider. This is the approach I am currently using when I am building mashups, at least I know that if shit hits the fan I can always go with somebody else. It will hurt a bit and take some effot, but I am not dead in the water. For the Google Maps example this would mean looking at Yahoo Maps and see what features the Yahoo Maps API have in common with the Google Maps API, and just use those common features. This can also come in handy if you hit the maximum number of requests on Google Maps, then it would be nice switch to Yahoo Maps automatically.

The risk with all this is of course to spend to much time preparing for something that won’t happen. It is the same situation as developers spending so much time making their code perfectly scalable and optimized that they acctually never ship anything in time. So dont go too far, but be aware of the situation. Trust is nice, but trusting several API providers to always do a good job and to not be evil in order for you to survive is quite risky.

When the big boys run websites (basically any brand name site you can think off) they use Content Delivery Networks (CDN’s) to host most of their content, especially images, stylesheets, files to download and other static content. The reason they do this is that the less they have to host themselves the less load they have on their servers, and the more content they can host closer to the end user the quicker the user can download it. The most famous CDN is probably Akamai, that almost run their own parallell internet. Akamai and other CDN providers cost big bucks though, so it is nothing for us mere mortals.

But thanks to Google anyone can now run their own CDN for free on Googles servers. It is really easy to set up and storing files for downloads, stylesheets etc on Google instead of on your own site takes the load of your servers (and consumes much less bandwidth of your hosting account) and speeds things up for the end users. It’s a win win situation, and it is also really really cool! If you are interested in more information about how to get the maximum performance from your web site then I recommend you read the excellent post Performance on a Shoe String on 24ways.

What is Google App Engine?
Using Google App Engine you can run web applications on Google’s servers. That means that you can benefit from Google’s huge world-wide server farms, it means that it is really easy to scale and to integrate with othe Google applications (for example using Google authentication in your applications). At the moment you have to write your applications in Python (don’t worry: no coding at all needed to use Google App Engine as a CDN, just keep reading), but hopefully they will expand it to other languages soon (personally I want to run PHP and CodeIgniter on Google App Engine!). The App Engine is Googles response to Amazons very successfull web services S3 (for storage) and EC2 (for computing). Amazons services are very powerful, but they do require a deeper level of technical knowledge to use than Google App Engine.

Currently Google App Engine is in a Preview Release (= beta), but it is free for anyone to join, all you need is a Google Account and a cell phone (more about this later). What you get is 500MB of free storage and around 5 million free pageviews a month, if you use more than that there is a small cost (see the Google App Engine blog for more details). The cost for these extra resources are almost the same as for Amazons Web Services, and with the freebies and ease of use thrown in Google App Engine is a bargin.

How to set up your own CDN
To use Google App Engine as your own personal CDN you need to install some things on your computer and edit a few configuration files. All this work is a one time thing though, after that all you need to do is run a simple program to upload new files to Google. Sorry to say that the scripts you download is for Windows only, if you are on a Mac or using Linux then you need to make your on script to do what deploy_digitalistic_cdn.bat does (if you do so please add this to the comments of this post for any one else to use).

1. Since Google App Engine only works with the programming language Python you need to download and install Python on your computer. If you have a Mac or run Linux you most probably already have Python installed, so you can skip this step. Download the correct installation file for your OS from Python 2.5.2 from http://www.python.org/download/ and install it. Use the default settings, except install it under “Program Files” instead of directly on the C: drive (or install it wherever you want, but in then you need to modify the scripts below.
2. Download the Google App Engine SDK from http://code.google.com/appengine/downloads.html and install it. During the SDK installation it will check if you have Python or not, so if you have a Python installation problem you will know it already here. The Google App Engine SDK is needed to be able to write and upload applications to Google. Just use the default settings when installing the SDK.
3. Sign up for Google App Engine at appengine.google.com. For this you need a Google account (your GMail address for example, if you dont have one it is free to create one).
4. Once you are signed up you need to create an application, so just click on the button “Create an Application” and give your application a name (called “application identifier”). This name needs to be unique among all users applications, so it might take a while to find a unique one. In my case I used “digitalisticcdn”. Save your new application. After you have created your first application you need to specify your cell phone number. Google will then send you a SMS with a code that you enter into their site. This confirms that you are the owner of this Google App Engine account (so don’t use it for spamming ;).
5. Download the file http://digitalisticcdn.appspot.com/files/digitalisticcdn.zip (hosted on my private CDN!) and unzip it to your harddrive. If you want you can rename the unzipped directory from “digitalisticcdn” to whatevery you want, for example the name of your own application. It doesnt really matter, it just makes easier for you to keep track of things in the future.
6. Use a text editor to edit the app.yaml file in the digitalisticcdn directory. Change “application: digitalisticcdn” to “application: <your application identifier” and save the file. This will tell Google App Engine what application to upload your files to.
7. Now it is time to add all the images, stylesheets, files, videos etc you want to upload to Google to the folders in the digitalisticcdn directory. Put all images into the /images folder etc. You can create any number of subfolders inside the images, files, stylesheets etc folders (for example /images/webhostninja.com/ninja.gif). You can always add more files at a later time, so if you just want to set things up to work you can skip this step for now. There is already an image in the /images folder for you to test that all is working as it should be.
8. Download http://digitalisticcdn.appspot.com/files/deploy_digitalistic_cdn.bat and edit it in a text editor. This file needs to point out your Python installation, your Google App Engine installation and your digitalisticcdn directory. If you installed the Google App Engine SDK in the default directory and Python in C:/Program Files/ then you don’t have to worry about those settings. Just change the last part of the file to point to your digitalisticcdn directory. Keep in mind that all paths with spaces in needs to be surrounded by quotes.
9. Double click on the newly changed deploy_digitalistic_cdn.bat file to upload all the files in the digitalisticcdn directory to Google. The first time you do this you need to specify your Google username and password.
10. You now have your own private CDN! Go to <your-application-identifier>.appspot.com/images/ninja.gif (in my case digitalisticcdn.appspot.com/images/ninja.gif) to see that it works.

How to use your private CDN
To use the files you upload to your Google App Engine CDN you just need to use the URL to the file on your site. If you want to show an image of a cool ninja from WebHostNinja.com you would just use digitalisticcdn.appspot.com/images/ninja.gif as your image source in your HTML. The same goes for stylesheets, files to download or whatever else you want to share on your CDN.

At any time you can add new files to your digitalisticcdn subdirectories (/images, /stylesheets etc) and run the deploy_digitalistic_cdn.bat file to upload them to Google. If you remove files from your digitalisticcdn directory and then run the bat file they will be deleted from your Google application.

You can check the statistics of your Google Application at appengine.google.com. For example you can see how much bandwidth and disk space you are using. It will take quite a lot of files and usage for you to use up the resources you get for free, but if you have a super popular site then it is worth taking a look here every now and then.

If you have read so far and found all this usefull then please Digg it. Thanks!

A few days ago I was finally ready to go live with one of the projects that have taken up my time since leaving Kapow - a price comparison search engine for web hosting called WebHostNinja.com. The site is all written using CodeIgniter and thanks to Kemie at monolinea.com it also looks great. What I am trying to do is a usefull site where you can compare different hosting providers and their plans so that you can find exactly what you need as cheaply as possible. There are already quite a few such sites out there, but they are all static, hard to use, ugly as hell and more focused on selling hostig than on providing a good service for the user. I think and hope that there is a market for WebHostNinja.com as well, if nothing else there is a chance of doing some cool images of ninjas.

There are still some bugs to work out, I want to add a lot more web hosting providers to the index and of course there are also loads of features I want to add. In my database I do have quite a lot of information about each hosting provider and their plans, so it would be cool to mash that up with some other data, but at this point in time my mind is blank what to do. If you have any ideas about this please let me know!

Another project that I am involved with is the Mexican TV-listing site enlatele.com.mx (”en la tele” = “on TV” in spanish). We have just finished an iteration in our continous development, and vastly improved the search and added other nifty things like channel logos and filtering channels based on your cable provider. I have several entires about writing widgets/gadgets for enlatele.com.mx for sites like MySpace etc, so that will give me a good excuse to soon dig deeper into the OpenSocial specs.

I have more things up my sleeve, projects that have much more to do with mashups. As soon as there is something I can tell you about I will, so watch this space.

As anyone that has surfed the web during the last years have noticed comment spam is wide spread, and highly annoying. Check the comments on blogs or web apps and you will find plenty of links left there by helpful people that really would like you to enlarge certain organs, sell you rolexxxes or need your help to get some dough out of Nigeria (I am the former prime ministers first bastard son, promise!). Fighting spam in your mail inbox is pretty simple these days, thanks to Thunderbird’s and GMails excellent anti-spam filters. Doing it in your Wordpress blogs have also been a walk in the park for a long time, simply use the Akismet plugin you get with any Wordpress installation and they will take care of it. Akismet is developed by Automattic (the company behind Wordpress) and describes itself as “a big machine that sucks up all the data it possibly can, looks for patterns, and learns from its mistakes”. Send your comment data to them and they will compare it to their database and tell you if it is spam or not.

Now when I am writing some of my own web apps I would like to avoid spam to overwhelm me, both via contact forms and comments. Lo and behold, Akismet is available as a web service, and via some ready made libraries it is very easy to use from outside Wordpress as well. I am using Elliot Haughin’s CodeIgniter Library, but there are also libraries for PHP, .NET, Ruby on Rails, Drupal etc etc. All you need to do is to get a Akismet API Key from Wordpress.com (register, then go to My Account and Edit Profile and you will find the key), download a library (or start writing your own from Akismets API) and start sending all comments to Akismet. The API Key is free for personal use, cost $5/month if you make more than $500/month from your site and cost a bit more for Enterprises. Well worth it I say, it will save you time and frustration all around, and make your website so much cleaner. There is no longer any excuse to suffer comment spam.

Thanks Akismet for one of the best and most usefull web services around!

Since leaving Kapow a week and a half ago I have been coding more than I have for the last 3 years combined, and I have done it all in PHP using the framework CodeIgniter. I looked at quite a few frameworks, CakePHP and Ruby on Rails for example, and quite a few applications/blogging platforms/Content Management Systems that can be hacked and adapted, wordpress and Drupal for example. In the end CodeIgniter won the day, and so far I am extremenly happy with my choice. There are a handfull reasons why I prefer CodeIgniter:

• It is PHP, this might be a dealbreaker for some, but for me that is a huge plus. Primarily the advantages of this is that I can easily host my creations on basically any cheap web host, that there are plenty of libraries and resources out there to make my life easier and also that I know PHP. For the last reason I could have gone with Ruby on Rails or Java as well, but it put all Python frameworks out of the competition.
• CodeIgniter is very easy to install and as easy to deploy. All you need for things to play nicely is an Apache server, a MySQL database and a copy of the CodeIgniter files. Deploying and setting up things are the most boring thing when it comes to writing your own apps, so it is a must for me that it is a breeze, I just dont have the patience to deal with deployment problems.
• Great documentation, the CodeIgniter user guide is excellent. This is a huge difference from many other frameworks and platforms, especially the ones developed by an open source community (CodeIgniter is developed by Ellis Labs, the guys behind the Expression Engine blogging platform). The developer community is also very active and knowledgable, so what isn’t in the user guide is in the CodeIgniter forums.
• Finally, a framework that improves my productivity. Most frameworks tries to do to much and are so huge and rigid that there is a huge learning curve if you want to doing anything but a “Hello World” app. CodeIgniter helps me with the stuff I need help with and doesn’t meddle in the rest. There is no need to hack 10 plugins of different qualities together to get what I need (like in Drupal), and that just makes developing fun as it should be. It is also the first MVC (Model-View-Controller) framework that helps me organise my code in a good way, something that I usually suck at otherwise.

In short I recommend that anybody that knows PHP and want a light weight, good framework checks out CodeIgniter. A good place to start are the CodeIgniter video tutorials, and if you get a bit deeper into things Elliot Haughin has a great blog that often covers CodeIgniter and he also have some great libraries that are well worth looking at (CodeIgniter libs for Twitter, Flickr and Akismet for example). Another great resource is the blog of Derek Allard, Technology Architect at Ellis Labs.