When the big boys run websites (basically any brand name site you can think off) they use Content Delivery Networks (CDN’s) to host most of their content, especially images, stylesheets, files to download and other static content. The reason they do this is that the less they have to host themselves the less load they have on their servers, and the more content they can host closer to the end user the quicker the user can download it. The most famous CDN is probably Akamai, that almost run their own parallell internet. Akamai and other CDN providers cost big bucks though, so it is nothing for us mere mortals.
But thanks to Google anyone can now run their own CDN for free on Googles servers. It is really easy to set up and storing files for downloads, stylesheets etc on Google instead of on your own site takes the load of your servers (and consumes much less bandwidth of your hosting account) and speeds things up for the end users. It’s a win win situation, and it is also really really cool! If you are interested in more information about how to get the maximum performance from your web site then I recommend you read the excellent post Performance on a Shoe String on 24ways.
What is Google App Engine?
Using Google App Engine you can run web applications on Google’s servers. That means that you can benefit from Google’s huge world-wide server farms, it means that it is really easy to scale and to integrate with othe Google applications (for example using Google authentication in your applications). At the moment you have to write your applications in Python (don’t worry: no coding at all needed to use Google App Engine as a CDN, just keep reading), but hopefully they will expand it to other languages soon (personally I want to run PHP and CodeIgniter on Google App Engine!). The App Engine is Googles response to Amazons very successfull web services S3 (for storage) and EC2 (for computing). Amazons services are very powerful, but they do require a deeper level of technical knowledge to use than Google App Engine.
Currently Google App Engine is in a Preview Release (= beta), but it is free for anyone to join, all you need is a Google Account and a cell phone (more about this later). What you get is 500MB of free storage and around 5 million free pageviews a month, if you use more than that there is a small cost (see the Google App Engine blog for more details). The cost for these extra resources are almost the same as for Amazons Web Services, and with the freebies and ease of use thrown in Google App Engine is a bargin.
How to set up your own CDN
To use Google App Engine as your own personal CDN you need to install some things on your computer and edit a few configuration files. All this work is a one time thing though, after that all you need to do is run a simple program to upload new files to Google. Sorry to say that the scripts you download is for Windows only, if you are on a Mac or using Linux then you need to make your on script to do what deploy_digitalistic_cdn.bat does (if you do so please add this to the comments of this post for any one else to use).
- Since Google App Engine only works with the programming language Python you need to download and install Python on your computer. If you have a Mac or run Linux you most probably already have Python installed, so you can skip this step. Download the correct installation file for your OS from Python 2.5.2 from http://www.python.org/download/ and install it. Use the default settings, except install it under “Program Files” instead of directly on the C: drive (or install it wherever you want, but in then you need to modify the scripts below.
Download the Google App Engine SDK from http://code.google.com/appengine/downloads.html and install it. During the SDK installation it will check if you have Python or not, so if you have a Python installation problem you will know it already here. The Google App Engine SDK is needed to be able to write and upload applications to Google. Just use the default settings when installing the SDK.- Sign up for Google App Engine at appengine.google.com. For this you need a Google account (your GMail address for example, if you dont have one it is free to create one).
Once you are signed up you need to create an application, so just click on the button “Create an Application” and give your application a name (called “application identifier”). This name needs to be unique among all users applications, so it might take a while to find a unique one. In my case I used “digitalisticcdn”. Save your new application. After you have created your first application you need to specify your cell phone number. Google will then send you a SMS with a code that you enter into their site. This confirms that you are the owner of this Google App Engine account (so don’t use it for spamming ;).- Download the file http://digitalisticcdn.appspot.com/files/digitalisticcdn.zip (hosted on my private CDN!) and unzip it to your harddrive. If you want you can rename the unzipped directory from “digitalisticcdn” to whatevery you want, for example the name of your own application. It doesnt really matter, it just makes easier for you to keep track of things in the future.
- Use a text editor to edit the app.yaml file in the digitalisticcdn directory. Change “application: digitalisticcdn” to “application: <your application identifier” and save the file. This will tell Google App Engine what application to upload your files to.
- Now it is time to add all the images, stylesheets, files, videos etc you want to upload to Google to the folders in the digitalisticcdn directory. Put all images into the /images folder etc. You can create any number of subfolders inside the images, files, stylesheets etc folders (for example /images/webhostninja.com/ninja.gif). You can always add more files at a later time, so if you just want to set things up to work you can skip this step for now. There is already an image in the /images folder for you to test that all is working as it should be.
- Download http://digitalisticcdn.appspot.com/files/deploy_digitalistic_cdn.bat and edit it in a text editor. This file needs to point out your Python installation, your Google App Engine installation and your digitalisticcdn directory. If you installed the Google App Engine SDK in the default directory and Python in C:/Program Files/ then you don’t have to worry about those settings. Just change the last part of the file to point to your digitalisticcdn directory. Keep in mind that all paths with spaces in needs to be surrounded by quotes.
- Double click on the newly changed deploy_digitalistic_cdn.bat file to upload all the files in the digitalisticcdn directory to Google. The first time you do this you need to specify your Google username and password.
You now have your own private CDN! Go to <your-application-identifier>.appspot.com/images/ninja.gif (in my case digitalisticcdn.appspot.com/images/ninja.gif) to see that it works.
How to use your private CDN
To use the files you upload to your Google App Engine CDN you just need to use the URL to the file on your site. If you want to show an image of a cool ninja from WebHostNinja.com you would just use digitalisticcdn.appspot.com/images/ninja.gif as your image source in your HTML. The same goes for stylesheets, files to download or whatever else you want to share on your CDN.
At any time you can add new files to your digitalisticcdn subdirectories (/images, /stylesheets etc) and run the deploy_digitalistic_cdn.bat file to upload them to Google. If you remove files from your digitalisticcdn directory and then run the bat file they will be deleted from your Google application.
You can check the statistics of your Google Application at appengine.google.com. For example you can see how much bandwidth and disk space you are using. It will take quite a lot of files and usage for you to use up the resources you get for free, but if you have a super popular site then it is worth taking a look here every now and then.
If you have read so far and found all this usefull then please Digg it. Thanks!
Andreas says:
Monday, June 9th, 2008at 5:52
php i all ära men codeigniter?
I alla fall, prova på lite python! Det är riktigt trevligt språk, många likheter med php och django är ett grymt ramverk för att göra webb.
Niek says:
Monday, June 9th, 2008at 6:05
Please change the url behind appengine.google.com (dellete http:).
Andreas says:
Monday, June 9th, 2008at 6:33
Andreas: Håller på med ett större projekt med Python och Django på Google App Engine just nu, så jag provar Python för fullt. Föredrar fortfarande PHP och CodeIgniter, men det är ju en smaksak.
Niek: thanks for catching my typo, it’s fixed
Daniella Newmark says:
Monday, June 9th, 2008at 9:17
Andreas,
This is a great article for those wanting to use the Google App Engine as a quasi CDN. I would not recommend using App Engine as a CDN as it is not an edge based solution providing downloads from servers closer to the user. The analytical reporting on objects stored in App Engine is also very poor and leaves much to be desired.
Anything free always has its drawbacks but for content that is not crucial to you business it will work fine. I would recommend many of the month-to-month CDN’s that are out on the market for content that is critical to your business.
Daniella Newmark
castmetrix.com
ContentWorkspace.com says:
Monday, June 9th, 2008at 13:21
This solution is definitely gaining momentum, it will be very interesting to see if global companies start to use storage clouds for faster distribution. The performance of enterprise content systems like Documentum is very poor, they need to cache content all over the web. Google, Amazon, Nervanix are all close to what people are looking for.
http://www.ContentWorkspace.com/solutions/google.htm
Joel says:
Monday, June 9th, 2008at 21:11
I think its a good way to get banned by google from every using any of their services. Did you read the eula? your probably not allowed to do this. Also could you change your font. I can baely read anything on your pages.
Andreas says:
Tuesday, June 10th, 2008at 0:10
Joel: What in Googles EULA states that I can not store non-pornographic content that I own the copyrights to and do not use for spamming on Google App Engine? I have read the TOS, Privacy Notice and Program Policy and as far as I can tell this should be OK. I would think that right now Google should be happy about anybody using and promoting Google App Engine, they are in for a hard fight with Amazon, Microsoft etc about this space.
eric casteleijn says:
Tuesday, June 10th, 2008at 3:49
I don’t think it’s a good candidate for CDN, not so much because of the EULA, but because it’s really meant for applications, and not for storage, (like S3 *is*, I’m betting google has something under their hat that *will* do that) and as such has a hard limit of 1000 separate files of maximum 1MB size. (The 1000 files is actually already a problem when you want to make use of exisiting python libraries.)
Andreas says:
Tuesday, June 10th, 2008at 4:01
Eric & ContentWorkSpace: Agreed, Google App Engine is not the _perfect_ CDN. Amazon S3 is a much more flexible and powerful solution when it comes to storage, and dedicated CDNs are of course much more powerful. But Google App Engine is free, easy to use even for people that are not programmers and a quick fix.
I wouldn’t suggest using Google App Engine as a CDN for a big enterprise project, but for any webmaster/blogger sharing big files for downloads etc I think Google App Engine is a good solution. Using it as a CDN is also an easy way to get started with Google App Engine.
maeghith says:
Tuesday, June 10th, 2008at 11:05
Nice idea.
But I would suggest, a “deploy_digitalistic_cdn.py” instead of the “deploy_digitalistic_cdn.bat” on the article, just for cross-platformness’ sake.
nerdd.net | news and opinion says:
Tuesday, June 10th, 2008at 12:53
10 Easy Steps to use Google App Engine as your own CDN | Digitalistic - Mashup or die trying | nerdd.net…
\r\nIt is easy to use Google App Engine as your own private Content Delivery Network (CDN), just lik…
IntelliJ IDEA中文爱好者博客 » Blog Archive » use Google App Engine as your own CDN says:
Tuesday, June 10th, 2008at 17:50
[…] http://www.digitalistic.com/2008/06/09/10-easy-steps-to-use-google-app-engine-as-your-own-cdn/ […]
Google App Engine over lunch « I was just thinking… says:
Wednesday, June 11th, 2008at 10:40
[…] up, is a solution I found very interesting. Andreas explains how GAE can be used as a Content Delivery Network (CDN). I won’t bother going into details of what a CDN is, it is explained well on his website. […]
Die Google App Engine als CDN benutzen - Peter Kröner - Die Kunst des Machbaren says:
Thursday, June 12th, 2008at 2:09
[…] den interessieren Nerd. Wenn man allerdings mit den 500 MB nichts besseres anzufangen weiß, kann man sie auch als Quasi-CDN benutzen. Ob diese Lösung jetzt gut ist oder nicht vermag ich nicht zu beurteilen, aber wenn jemand ein CDN […]
Janos says:
Friday, June 13th, 2008at 3:20
Newbie question maybe, but how do I remove files I no longer need?
Edmar says:
Saturday, June 14th, 2008at 17:24
Hello,
Nice blog ! This is a great article on AppEngine. I wanna see more articles like this in the future.
Andreas says:
Monday, June 16th, 2008at 1:10
Janos: to remove files form your CDN just remove the files in question from your local digitalisticcdn directory (or whatever you named it) and do a new upload. That deletes the files from your Google App Engine account as well.
Edmar: Thanks
Hoping to soon follow this post with a real example of how to use Google App Engine. Keep your eyes open, especially if you are a Ning fan 
IPSOJOBS Blog » How to create a simple but powerful CDN with Google App Engine (GAE) says:
Tuesday, June 17th, 2008at 2:19
[…] first approach, of course, was to look on Google for some help, the post of Andreas Krohn helped a lot to […]
Janos says:
Tuesday, June 17th, 2008at 3:20
Oh. That was kind of lame. Thanks. Both for the excellent article and the clarification.
Hareem Haque says:
Wednesday, June 25th, 2008at 14:03
A quick question. I am using the cdn app to store my css bat bash-script etc on app engine. Now could you tell me how i could retrieve the all the files that i uploaded to my app and store them onto a new pc.
Sort of syncing data via google app engine.
Regards
Hareem.
PS: If this sounds totally stupid. Please do not get angry. I am just a newbie.
philip andrew says:
Thursday, June 26th, 2008at 20:11
Hareem I feel very very angry with you.
links for 2008-07-11 « Bloggitation says:
Thursday, July 10th, 2008at 17:37
[…] 10 Easy Steps to use Google App Engine as your own CDN (tags: google gae web hosting sysadmin) Possibly related posts: (automatically generated)links for 2008-05-09A Google DLR? […]
Optimizar la carga de una web (y II) | El Blog de Topilloman says:
Sunday, July 20th, 2008at 16:11
[…] de trampa y usar el servicio de Google App Engine para montarnos una CDN. Con la información de esta entrada monté lo básico y pasé todo lo que pude a la página de Google Apps, pero me mosqueaba que […]
10 pasos fáciles para usar Google App Engine como tu propia red de distribución de contenido | Andrebills says:
Tuesday, August 5th, 2008at 7:12
[…] artículo fue escrito originalmente en inglés y traducido con permiso del autor al español por Christian Van Der Henst […]
Rishi says:
Tuesday, August 5th, 2008at 12:53
Fantastic walk through! Looks like I have something to keep me occupied this rainy evening.
Google CDN | RK-home says:
Tuesday, August 5th, 2008at 17:18
[…] Andreas Krohn at Digitalistic wrote a fantastic article regarding how to do just this. Using Google’s AppEngine utility (currently in beta testing), users are able to publish files to Google. Since these files can be referenced via a static link, this service can be used to distribute website content (cascading style sheets, images, etc.) to users. In principle, content distribution networks have many outlets throughout the world. Therefore, users from virtually any country will always be “near” the host site, whether they live in Asia, Europe, or America (in this case, wherever Google servers operate). […]
Jam says:
Wednesday, August 6th, 2008at 5:59
I came up with this post from a suggestion given in http://vagabundia.blogspot.com about Google App, and found the explanations clear and easy to understand, even for people that do not speak English as a mother language. Really, thank you very much. I will visit this site often from now on.
solrac says:
Tuesday, August 12th, 2008at 9:18
Same as Jam, I came from Vagabundia’s post and agree is a good explanation and a very good idea, I just started to use it.
Many thanks Andreas
PD: how can i change (only a little) the start page of my new “appspot” site? (of course thinkout remove your credits)
solrac says:
Tuesday, August 12th, 2008at 10:21
opps XD
.. think is the ‘index.html’ file included on your digitalisticcdn.zip file, didn’t it?
Andreas says:
Tuesday, August 12th, 2008at 10:27
Yes Solrac your right, the startpage is all in the index.html file (and the CSS file it is importing). Just change what you need and reupload it to your Google App Engine site and your should be all done.
Len says:
Friday, September 19th, 2008at 13:57
I setup the app engine on google and see the url in Manage Start Page but if I click it or enter it I get a page cannot be displayed. My Bat file seems to work pretty fast so I’m not sure I’m transfering anything either.
Andreas says:
Sunday, September 21st, 2008at 9:49
@Len If you try running the command in the bat file from a command promt you can see if it uploads something or not, and if there is a problem you will (hopefully) also see a helpfull error message.