Using a CDN for the filemanager?
My question is, does anyone have any experience with this? Forking the core to use a CDN instead of a local system path? I know C5 has the "alternative storage location" feature, but I'm assuming this is going to take heavier revision than just that to make it play nice.
Better yet, instead of forking the core, does anyone know of a way to override core files w/in a package?
If anyone else is interested in this, I'll release the code when I'm done with it.
Dont think you'll need to fork the core. In almost every case I've found, deploying the core files into their equivalent root directories will let you modify them to do what you need and leave the core intact.
I've had less success with packages to be honest, but the application flow is C5 -> Package -> root. So, in theory as long as your own root directories aren't overriding your package it should work.
Isn't the point of a CDN the performance boost? Can I ask what the reason is you think it would be slow?
Just for clarification - when I'm referring to CDN's here, I'm focusing on those that charge based on space used (like Amazon and Rackspace), not ones like LimeLight w/ minimum monthly contracts. I'm more interested in increased storage space than blazing performance... though speed should be an added benefit if this is done right.
So here's the status: Aside from increased storage space, the other reason I want to use a CDN is for security purposes. As it stands now, the only way to make files truly secure in C5 is to store them in an alternate storage directory - and that actually works well if your server has the space. Mine doesn't.
No surprise - after lots of searching - Amazon has the best solution. Rackspace has a good CDN, but there is no token based security. Files are either public or private. If you want a file to be secure and visible only through C5, your server has to initialize a CDN connection, download the raw data, parse it in php, then send it to the client. Thats slow as hell, and would rock the server if it had to send lots of files. Your server basically stands as a proxy between your visitor and the cdn, and that won't fly.
With Amazon however, you can set files to always be private... except for someone who tries to access them from a URL with an attached time-based token. Amazon's SDK makes it easy to spit out a url that is good for x amount of time, based on your security credentials. This way, all C5 would have to do is return the appropriate URL, not download then parse etc.
I think I'm on the right track, but if any of you guys have any advice I'm all ears.
Mnkras, you're right about uploading - it is a little slower. Files go to your server first (where C5 is running), C5 gets all the info it needs from it, then sends it on to Amazon. So your application server acts as a proxy of sorts. When downloading though, a link is generated directly to the file on Amazon's CDN - so it flies.
Its actually pretty awesome. If others are willing to experiment with it, some feedback would be great.
Only thing is that not everything works from within a package. You have to override a few core models/libraries/helpers - but that can be done in the root directory, no core overwriting/hacking is needed.
If anyone wants the code let me know and I'll post it.
Looking forward to seeing the code!
All the code is attached. Because some of the core files have to be overridden, its more than just a package, so copy all these folders to your c5 root directory.
Try this out in a sandbox before you do anything else with it... of course.
Couple notes: once you start storing your site files on AS3, dont count on being able to convert back to storing them locally. Its one or the other.
Uploading and deleting is a little slower, but once files are up on AS3, if they're set to public (you'll see in the AS3 config options), they fly. You get all the speed benefits of the CDN.
I haven't found any problems with it yet - let me know if you do so I can improve it.
If the attribute is turned on you would then direct to the CDN for serving up the file. This could be particularly useful for larger videos etc.
The only other issue I can see with it is that if you use something like the ImageHelper to generate thumbnails on the fly from files in your media library, it will expect to create and serve those cached versions up from the files/cache directory - you may need to look at extending ImageHelper also to sync cached versions of thumbnails of the images with the CDN attribute enabled up to the CDN also and serve them from there too.
Part of the problem currently is how much of the core has to be modified to make this work. Andrew got in touch with me and said he's considering refactoring some of the file manager functionality so that this add-on could be developed fully w/in a package, instead of placing model/helper/library overrides in the root. I'm going to hold off developing this further pending his decision on that.
Is this still a system that you/Andrew are looking into building? I would be very interested in where this stands. I work with a lot of clients who host large files (video and audio) and we would really like to start storing all that on a CDN.
Unfortunately, the only downside right now is that enabling the AS3 package causes a conflict w/ C5's new caching mechanisms... So page caching doesn't work at the moment. But I'm working on a fix, and will re-post the code above once its done.
What permissions do I need to give the bucket?
And with the account credentials, does the AWS Account ID need the hyphens?
Any help would be appreciated.
If you can wait a little while until the next release of C5 (coming this summer I think), the core will have installable custom storage locations built in. I've been working w/ Andrew on rewriting the underlying file storage functionality, and as soon as its officially supported, there is an AS3 package ready to go (that will be for free).
All my work is available here:https://github.com/jhartman86/concrete5...
You should be able to download and install it as a fresh copy of C5, and then install the Amazon Web Services package as an extension. If you try it out, please let me know.
If you want to specifically see the AS3 integration stuff, its located here:https://github.com/jhartman86/concrete5/blob/master/web/packages/ama...
Its been working and stable for me so far, but I'm not actually going to put anything into production myself until its officially integrated into the core (just for maintaining future upgradability and such).
I'm waiting to hear from the core team as to what release they plan on building it into, and will then release the S3 package to the marketplace. Currently, you have to override some core files to get it working (if you get the code I put up in a post above), which means the upgrade path of Concrete5 could get broken down the road.
Then again, if you know something I don't know about the progress on this, please let me know.
Since AS3 works at a totally different location, I actually had to go and fork most of the C5 core's file handling code (models, libraries, helpers, etc) to get it working. Because of that, and because the ability to store files at a CDN or AS3 should be valuable to a lot of folks looking for more power out of C5, Andrew and I have been in contact to build the code into the core. If they didn't, then anytime you download an upgrade, you could be overriding some new required functionality by using the forked code to make AS3 work... which could break stuff. The plan is, just like you can download/install different attribute types - you should be able to do so with file storage location types. So i've been told at least.
The images are uploading perfectly into my S3 bucket. I can browse the file manager and choose images, just like before. However when using the slideshow block type, the images are not displaying at all. It's just showing an empty region where the images should be. The photo galleries have the thumbnails rendering normally (looks like the thumbnails are stored locally), but the popup lightbox viewer is showing ascii code instead of rendering jpeg images.
See my tests here:
Have you seen this in your tests?
Curious... Do you know what kind of control you're looking to implement in any future plugin - such as what aspects of the local install would stay and which would be pulled externally? Theme/ files relating to the site layout would want to be local, but are you planning any kind of control of this in the backend?
Thank you Jon for the great work. This is exciting.
it provides image transformation and manipulation along with CDN running on top of amazon.
best of all they offer 500 MB of storage, 50,000 images, and 1 GB of data transfer a month all for free.
could really help small biz sites.
Very easy implementatio by simply prepending existing full path "src" locations with
http://res.cloudinary.com/<your... cloudinary cloud name>/image/fetch/
this would allow for the CDN repository to stay on the local server and use the native C5 File Manager, but allow the files to be served up by the CDN.
Where in C5 can i override the output of src locations for files in the file manager?
this is a shame as otherwise it would be fairly easy to integrate into C5 if there were standards around how block called and worked with files. hopefully in the near future C5 will be updated to make it easier for 3rd parties to integrate their CDN offerings into C5.
would really help make C5 a true contender in the CMS arena.
Particularly with the very easy and affordable CND options appearing likehttp://cloudinary.com
While I haven't seen support for Amazon's CDN, I'd like to voice my support for the direction that C5 has gone - which is full support for Cloudflare:
The result is fast. If you are coming over from WordPress, you're probably looking for something like W3 Total Cache, which is a dangerous amount of overkill even on your WP sites.
Cloudflare works a treat with C5 and basically automates with your WP plugin-free as well (run P3 Plugin Profiler some time and you'll see the performance hit a caching plugin costs you anyway).
Long post short... hit the Cloudflare link and test it before deciding you aren't going to use C5 due to issues with CloudFront.