Using a CDN for the filemanager?

Permalink
I'm building a project where my client is basically going to need unlimited storage space... so storing files on the web server probably ain't gonna cut it. I'm going to use RackSpace's CloudFiles CDN as the api is pretty straight forward.

My question is, does anyone have any experience with this? Forking the core to use a CDN instead of a local system path? I know C5 has the "alternative storage location" feature, but I'm assuming this is going to take heavier revision than just that to make it play nice.

Better yet, instead of forking the core, does anyone know of a way to override core files w/in a package?

If anyone else is interested in this, I'll release the code when I'm done with it.

Thanks,
Jon

focus43
 
olliephillips replied on at Permalink Reply
olliephillips
Interesting post, will be watching this thread (where are the popcorn smilies @c5?)

Dont think you'll need to fork the core. In almost every case I've found, deploying the core files into their equivalent root directories will let you modify them to do what you need and leave the core intact.

I've had less success with packages to be honest, but the application flow is C5 -> Package -> root. So, in theory as long as your own root directories aren't overriding your package it should work.
12345j replied on at Permalink Reply
12345j
I think its possible but may be very slow.
focus43 replied on at Permalink Reply
focus43
12345j,

Isn't the point of a CDN the performance boost? Can I ask what the reason is you think it would be slow?
Mnkras replied on at Permalink Reply
Mnkras
i think he means that adding of files may be slower, not sure about that
focus43 replied on at Permalink Reply
focus43
Glad this thread is getting a decent amount of attention.

Just for clarification - when I'm referring to CDN's here, I'm focusing on those that charge based on space used (like Amazon and Rackspace), not ones like LimeLight w/ minimum monthly contracts. I'm more interested in increased storage space than blazing performance... though speed should be an added benefit if this is done right.

So here's the status: Aside from increased storage space, the other reason I want to use a CDN is for security purposes. As it stands now, the only way to make files truly secure in C5 is to store them in an alternate storage directory - and that actually works well if your server has the space. Mine doesn't.

No surprise - after lots of searching - Amazon has the best solution. Rackspace has a good CDN, but there is no token based security. Files are either public or private. If you want a file to be secure and visible only through C5, your server has to initialize a CDN connection, download the raw data, parse it in php, then send it to the client. Thats slow as hell, and would rock the server if it had to send lots of files. Your server basically stands as a proxy between your visitor and the cdn, and that won't fly.

With Amazon however, you can set files to always be private... except for someone who tries to access them from a URL with an attached time-based token. Amazon's SDK makes it easy to spit out a url that is good for x amount of time, based on your security credentials. This way, all C5 would have to do is return the appropriate URL, not download then parse etc.

I think I'm on the right track, but if any of you guys have any advice I'm all ears.
focus43 replied on at Permalink Reply
focus43
For anyone interested: I have it working on Amazon's S3 service. Everything works well in the file manager - even thumbnails. The directory structure is exactly mirrored in Amazon as it would be on a server's file system. Also, you can control your content's Time to Live before it is inaccessible... So if someone wanted to hotlink to your content, or send a file's URL w/out your permission, it'll die after a certain time.

Mnkras, you're right about uploading - it is a little slower. Files go to your server first (where C5 is running), C5 gets all the info it needs from it, then sends it on to Amazon. So your application server acts as a proxy of sorts. When downloading though, a link is generated directly to the file on Amazon's CDN - so it flies.

Its actually pretty awesome. If others are willing to experiment with it, some feedback would be great.

Only thing is that not everything works from within a package. You have to override a few core models/libraries/helpers - but that can be done in the root directory, no core overwriting/hacking is needed.

If anyone wants the code let me know and I'll post it.
jbx replied on at Permalink Reply
jbx
Hey, yes, that would be great. Playing with this has been on my list for a while, but if you can save me the hassle then I'm thankful!

Looking forward to seeing the code!

Jon
focus43 replied on at Permalink Reply 1 Attachment
focus43
Jon,
All the code is attached. Because some of the core files have to be overridden, its more than just a package, so copy all these folders to your c5 root directory.

Try this out in a sandbox before you do anything else with it... of course.

Couple notes: once you start storing your site files on AS3, dont count on being able to convert back to storing them locally. Its one or the other.

Uploading and deleting is a little slower, but once files are up on AS3, if they're set to public (you'll see in the AS3 config options), they fly. You get all the speed benefits of the CDN.

I haven't found any problems with it yet - let me know if you do so I can improve it.
jaredquinn replied on at Permalink Reply
jaredquinn
I personally would look at using an attribute to control this, you would then upload and manage all your files in your file manager as usual, if the "Serve from CDN" attribute is turned on you would sync the file up to your CDN at that point.

If the attribute is turned on you would then direct to the CDN for serving up the file. This could be particularly useful for larger videos etc.

The only other issue I can see with it is that if you use something like the ImageHelper to generate thumbnails on the fly from files in your media library, it will expect to create and serve those cached versions up from the files/cache directory - you may need to look at extending ImageHelper also to sync cached versions of thumbnails of the images with the CDN attribute enabled up to the CDN also and serve them from there too.
focus43 replied on at Permalink Reply
focus43
We're on the same page - that's exactly what I'd like to see eventually as well... so that storage could be determined on a per-file basis instead of all or nothing.

Part of the problem currently is how much of the core has to be modified to make this work. Andrew got in touch with me and said he's considering refactoring some of the file manager functionality so that this add-on could be developed fully w/in a package, instead of placing model/helper/library overrides in the root. I'm going to hold off developing this further pending his decision on that.
pjhagel replied on at Permalink Reply
pjhagel
Has there been any further development on this system? It seems that everything has been removed from GitHub, and non of the system seem to be running.

Is this still a system that you/Andrew are looking into building? I would be very interested in where this stands. I work with a lot of clients who host large files (video and audio) and we would really like to start storing all that on a CDN.
PerryGovier replied on at Permalink Reply
PerryGovier
What's the URL? Can we see it in action?
focus43 replied on at Permalink Reply
focus43
Yup, if you want to check outhttp://www.coopergreenteam.com.... Its in full production there. (Not a very aesthetically pleasing site, but I was only tasked with rebuilding it on C5). If you look at the urls of the images, about 2/3 of them are pointing to AS3 cdn urls. Any files that are part of the theme are on the local server, or any image files that were resized and cached. Currently the only thing the package doesn't do is save thumbnails generated by C5 up to AS3... but it should be a relatively easy fix down the road.

Unfortunately, the only downside right now is that enabling the AS3 package causes a conflict w/ C5's new caching mechanisms... So page caching doesn't work at the moment. But I'm working on a fix, and will re-post the code above once its done.
Enlive replied on at Permalink Reply
Enlive
What's the progress on this? I have tried implementing the code attached above but doesn't seem to work for me. I turned off cache and it seems like it wants to upload files to AS3, but doesn't.

What permissions do I need to give the bucket?

And with the account credentials, does the AWS Account ID need the hyphens?

Any help would be appreciated.
pixelhandler replied on at Permalink Reply
@bolder - I am very interested in trying out this code. We have a cache server in front of our sites so I don't need the concrete5 caching I can rely on the cache server instead. So I would really like to give our clients a bulk uploader (c5 already has multiple upload featrue) and the storage on Amazon S3. Please let me know if you have any updates to the code.

Best regards,

-Bill
focus43 replied on at Permalink Reply
focus43
Hey Bill,
If you can wait a little while until the next release of C5 (coming this summer I think), the core will have installable custom storage locations built in. I've been working w/ Andrew on rewriting the underlying file storage functionality, and as soon as its officially supported, there is an AS3 package ready to go (that will be for free).

All my work is available here:https://github.com/jhartman86/concrete5...

You should be able to download and install it as a fresh copy of C5, and then install the Amazon Web Services package as an extension. If you try it out, please let me know.

If you want to specifically see the AS3 integration stuff, its located here:https://github.com/jhartman86/concrete5/blob/master/web/packages/ama...

Its been working and stable for me so far, but I'm not actually going to put anything into production myself until its officially integrated into the core (just for maintaining future upgradability and such).
hereNT replied on at Permalink Reply
hereNT
I see the ability to define additional storage locations in the new version of c5, but I don't see your package in the marketplace. How would someone go about implementing this with the current version of c5?
focus43 replied on at Permalink Reply
focus43
Are you referring to 5.4.2.2 or are you looking at the upcoming 5.5 release in GitHub? If you've seen the installable storage location stuff integrated into the core, its news to me.

I'm waiting to hear from the core team as to what release they plan on building it into, and will then release the S3 package to the marketplace. Currently, you have to override some core files to get it working (if you get the code I put up in a post above), which means the upgrade path of Concrete5 could get broken down the road.

Then again, if you know something I don't know about the progress on this, please let me know.
hereNT replied on at Permalink Reply
hereNT
I was talking about 5.4.2.2, it lets you define a secondary storage location, I thought that's what you needed. It looks like it will work with s3fs, we can check if a file is in the secondary location and change the url for videos so I think we can do everything we need to do for the couple of projects we're working on, but it seems like there could be a lot more you could do, wasn't sure how different your solution was.
focus43 replied on at Permalink Reply
focus43
Gotcha. So they've actually had the secondary storage location thing in there for a while. Not sure since when, but its been in the past few versions. The major "but" with it right now though, is that it only lets you define a secondary storage location on the local disk. So it works off the assumption that everything is stored on the server hosting your site.

Since AS3 works at a totally different location, I actually had to go and fork most of the C5 core's file handling code (models, libraries, helpers, etc) to get it working. Because of that, and because the ability to store files at a CDN or AS3 should be valuable to a lot of folks looking for more power out of C5, Andrew and I have been in contact to build the code into the core. If they didn't, then anytime you download an upgrade, you could be overriding some new required functionality by using the forked code to make AS3 work... which could break stuff. The plan is, just like you can download/install different attribute types - you should be able to do so with file storage location types. So i've been told at least.
ginslo replied on at Permalink Reply
ginslo
First, thanks for putting this together and attaching the code in the previous thread. It's exactly what I need for saving space on my main server (which is on Amazon EC2). I have AS3 up and working on a test site. It works great on standard images but I'm running into problems when using blocks such as the default slideshow block which comes with C5 and two photo gallery add-ons with lightbox popups (Simple Image Gallery - 1.01 and Amiant Image Gallery - 0.1.3).

The images are uploading perfectly into my S3 bucket. I can browse the file manager and choose images, just like before. However when using the slideshow block type, the images are not displaying at all. It's just showing an empty region where the images should be. The photo galleries have the thumbnails rendering normally (looks like the thumbnails are stored locally), but the popup lightbox viewer is showing ascii code instead of rendering jpeg images.

See my tests here:
http://www.rvcoder.com/

Have you seen this in your tests?
joelhansen replied on at Permalink Reply
joelhansen
I'm working on a C5 forum that is expecting a large amount of attachments/ community file-sharing. Unfortunately for me modifying the core and breaking the ability to update is out of the questions. Still, this is super interesting...

Curious... Do you know what kind of control you're looking to implement in any future plugin - such as what aspects of the local install would stay and which would be pulled externally? Theme/ files relating to the site layout would want to be local, but are you planning any kind of control of this in the backend?

Thank you Jon for the great work. This is exciting.
novologic replied on at Permalink Reply
novologic
I am working to do something similar with a new CDN service called cloudinary

it provides image transformation and manipulation along with CDN running on top of amazon.

best of all they offer 500 MB of storage, 50,000 images, and 1 GB of data transfer a month all for free.

could really help small biz sites.


Very easy implementatio by simply prepending existing full path "src" locations with

http://res.cloudinary.com/<your... cloudinary cloud name>/image/fetch/


this would allow for the CDN repository to stay on the local server and use the native C5 File Manager, but allow the files to be served up by the CDN.

Where in C5 can i override the output of src locations for files in the file manager?
novologic replied on at Permalink Reply
novologic
After a couple of weeks carefully dissecting the file manager in concrete5, in order to fully understand how it works and is used within the system. we have come to the conclusion that you would need to really FORK C5 in order to make this work as well as FORK any add on that interfaces with C5.

this is a shame as otherwise it would be fairly easy to integrate into C5 if there were standards around how block called and worked with files. hopefully in the near future C5 will be updated to make it easier for 3rd parties to integrate their CDN offerings into C5.

would really help make C5 a true contender in the CMS arena.

Particularly with the very easy and affordable CND options appearing likehttp://cloudinary.com
Imperative replied on at Permalink Reply
Imperative
I realize I'm reviving a necro-thread for my first post, but as a seasoned WP theme builder looking into Concrete5 for the first time, CDN capability was one of the first things I searched and this was the top hit.

While I haven't seen support for Amazon's CDN, I'd like to voice my support for the direction that C5 has gone - which is full support for Cloudflare:

http://www.concrete5.org/documentation/how-tos/developers/using-clo...

The result is fast. If you are coming over from WordPress, you're probably looking for something like W3 Total Cache, which is a dangerous amount of overkill even on your WP sites.

Cloudflare works a treat with C5 and basically automates with your WP plugin-free as well (run P3 Plugin Profiler some time and you'll see the performance hit a caching plugin costs you anyway).

Long post short... hit the Cloudflare link and test it before deciding you aren't going to use C5 due to issues with CloudFront.