Unfortunately, somewhere along the line, performance problems started to become more than just a nuisance and threatened to become a meme. We had to do something about them.

The Past

It All Started with Files

In concrete5 5.2.0, we introduced a completely new file manager. Files were now full-fledged objects with extensible metadata, permissions and sets. Retrieving information about a particular file object was much more expensive (in terms of database queries) than it used to be.

All of a sudden, certain add-ons like galleries – which gathered large amounts of files at once – were quite slow. They had to run hundreds of queries just to render a page.

Caching to the Rescue

In web applications, caching typically means retrieving an object from a faster place – be it RAM, or a serialized object in a hard disk – than reconstituting it entirely from the database. This lessens the load on the database and can dramatically improve performance. We added our own custom caching layer, which stored objects in the file system, and this problem went away. We liked this solution so much that we added more objects to this cache.

Enter Zend Cache

When we started working on localizing our add-ons (so they could be included in translation) we realized that it would make a lot of sense to use Zend Translate, which is a library contained with the Zend Framework. Zend Translate makes extensive use of Zend Cache. We took a look at Zend Cache, and realized that without a lot of work, we could remove our custom caching layer, replace it with Zend Cache, and automatically get access to different cache backends, like APC, Xcache, memcached and more. It was a no-brainer.

Not entirely. Zend Cache has always had its share of quirks. The biggest one? The default cache, which stores objects in the filesystem, gets slower and slower the more objects you place in it, due to some garbage collection routines that Zend Cache uses. If you disable garbage collection, the cache is fast, but it fills up with millions of files. This was exacerbated by the fact that, as we added features like extensible attributes, layouts and full page caching, we found ourselves storing more and more objects in that cache.

And the worst part? While we were trying to lessen database access, we just shuttled the problem over to the file system, which now had to deal with thousands of small files on every request. Yes, this problem goes away if you configure an alternate cache backend, but most concrete5 sites don't do that. On web hosts with slow disk I/O this was a killer: hitting a site with no cache entries or out of date cache entries could take a really long time. Oftentimes the next access would be faster, but not always; the filesystem could be a significant bottleneck on shared hosts, and it was clear that caching was hurting as often as it was helping.

Cache Bigger Objects: Enter 5.6

We recognized this, so we tried to cache larger and larger objects in fewer and fewer cache entries. The idea was that if we could still use the cache but access it less often we'd get the benefits of the cache without as much of the lookup slowness.

This reached a head in concrete5 5.6.0. Our completely rebuilt permissions system was designed from scratch to be extensible, powerful and able to handle anything. Unfortunately, it requires quite a bit more database access to work with. Rather than cache each permission lookup, we ran all permission lookups every time a page was requested, and cached that in the page object. This worked – but performance was significantly degraded for sites that had caching off. And when I say significantly degraded, I mean significantly degraded. I think at one point I benchmarked around 700-800 queries running on a simple page when caching was disabled.

This was where we were in the Fall of 2012: concrete5 was more extensible than ever, but also slower than ever. Sites would fail due to slow disk I/O and the response was either "disable your cache" or "enable your cache." Disabling the cache would bloat the database queries by the hundreds, many of them duplicates. It was clear that caching was causing more problems than it was solving.

At this point, we had several layers of performance mechanisms:

1. The in-memory cache: powered by a simple array, the in-memory cache (aka "CacheLocal", if you're curious) is a way of storing a requested object in memory, so that it wasn't retrieved from the file system if it had already been retrieved once in a given request.
2. The basic cache: powered by Zend Cache, this was the object-based caching, usually performed by the file system, that I've described above.
3. Block caching: introduced in 5.4.0, block caching is a powerful way of preserving all that is great about blocks while allowing developers to mark ways that their blocks can be cached, in order to preserve their performance on render.
4. Full Page caching: also introduced in 5.4.0, full page caching is a way to store the entire contents of a page and render it ideally without accessing the database.
5. Overrides Caching: introduced in 5.6.0, overrides caching was our way of saving the state of your overridden file system to a Config variable, so that we don't have to do as many file_exists() checks throughout our code.

While these were all noble attempts at solving the performance problems of a complex and flexible application, they were almost all failing:

#2 failed because too many items were being placed in the cache, the file system is slow on many web hosts, the item still has to be requested from the database and then written to the file system, and it just wasn't much faster to request from the file system than it was to get out of the database. Furthermore, by forcing more data to be written into the cache in order to create fewer entries, we actually did many more database queries to make #2 viable than we normally did. This killed performance especially when cache was disabled.

#3 was great in theory, but it used #2 as its backend, which meant all the problems that plagued #2 filtered down to #3. This was the same problem with #4 – but in addition #4 suffered from firing far too late in the loading process. Practically, #4 never ran without still connecting to the database, which rendered its benefits minimal. Finally, #5 was a good idea in theory, but still required a database connection, and was confusing, because it couldn't be cleared by deleting files/cache/ directory in the filesystem – its data was stored in a different way.

The Present: 5.6.1

It's no secret: we have great plans for concrete5 5.7.0. But these plans are somewhat contingent on concrete5 running everywhere, and running everywherewell. We need to work well on good servers, and work acceptably on even the slowest ones. We simply cannot afford and will not accept performance continuing to be such a problem.

So I set out to fix it. To do this, I decided to look at what worked, and throw out what didn't. The override cache worked and was a good idea, but storing its information in the database to get around the slowness of the cache was stupid. Full page caching was a great idea, and the options that triggered it were sensible. But relying on Zend Cache and firing late in the loading process? This was a bad idea. We had tried to make it so that controller actions and certain interactive items could still be used with full page caching. While this was neat in theory, it made full page caching much less effective, and – most importantly – just plain slower.

Block caching is a great idea. It occurred to me if we had had block output caching first, when the gallery problems had cropped up with the file block, we may never have implemented anything else.

What didn't work? The one-size-fits-all cache, using Zend Cache as a backend. At best, it still added uncertainty to concrete5 ("Something not working right? Clear the cache, I guess."). At worst, it degraded performance on first run and, for many sites, in perpetuity.

Let's be clear: Zend Cache is a fine piece of software. We will still continue to support its usage in our API and actively use it when interactive with Zend Framework components that use it (like Zend Translate). But shoving every object in there because we make too many database queries? That's like applying a bandaid to a cut with a pair of scissors.

What's New

You can currently download a beta version of concrete5 5.6.1 to see these changes in action. I would encourage you to do so, and to take it for a test drive on any web host. The cheaper the host the better.

What's been done? A lot.

1. We don't use the Cache anymore. It's still there and used for Zend Framework libraries, but rarely accessed anymore. I think the information about the dashboard picture of the day might use it – but that's it.

2. We do use the in-memory cache. A lot. This is CacheLocal. The first time in a given request that a page, file, permissions record or other objects are requested, they are added to the local cache, and then subsequent requests for these items are retrieved from an array. This dramatically reduces database queries without incurring any disk I/O problems. Yes, you do occasionally run into odd cache-related bugs, and it could cause some problems for hosts with very restrictive memory settings, but these should both be rare, since CacheLocal doesn't persist across subsequent requests.

3. We do use block-level caching, for records and output, but we store this information in the database. We're already requesting data from these database tables, so adding another column to the mix and displaying its contents is trivial (and much faster than looking in the file system for Zend Cache entries.) All the same block caching settings that were valid pre-5.6.1 are still honored. The best part about block-level caching: we save block output and information when a block is saved, which means the data is already there for subsequent requests, leading to much less slowness on initial request for a given page.

4. We do use the Environment cache, but it's stored as a serialized object in files/cache/environment.cache. Clearing the cache will remove this file, as well as just deleting files/cache/ yourself.

5. We have updated our CSS cache to store files in files/cache/css/. This is used whenever a theme uses customizable stylesheets. Files are generated and referenced directly without going through concrete5.

6. We have completely rewritten the full page caching library. Full Page caching is completely modular and extensible. It fires very early in the request. If a site enables full page caching, and the pages are in the cache, a site can completely lose its database connection and it will still render as though nothing is wrong.

7. We have optimized and removed unnecessary queries. So many queries were structured in certain ways to appease the mighty Gods of cached objects, that when this was no longer a concern they can be rewritten. Page and File custom attributes are an example of this: we used to retrieve all attributes about a particular file or page when that object was retrieved, in order to cache as much in the object as possible. Now we only retrieve this information on demand.

The results speak for themselves. concrete5 5.6.1 is much more responsive on many web hosts. Yes, it still makes robust use of a database (non-logged-in users will see around 50-70 database queries rendering a standard starting point page that's not cached in the full page cache) but this is a massive decrease from 5.6.0.2. And its all being done without the cache adding extra disk I/O.

The Future

We are still actively exploring how we might speed up some core block types like Auto-Nav and the PageList class. This may require some creative, not-necessarily-object-oriented problem solving, but we're up to the challenge, armed with some of the lessons we've learned.

One Size Doesn't Fit All

The features we'd added for the sake of performance weren't all hindrances – but their impact was lessened because we tried to funnel them all through the same approach (a file-based cache.) Now, although we have some slightly more idiosyncratic approaches to solving these problems, the results are much, much better.

Keep the Good – Ditch the Bad

We've never been afraid to rebuild something in a better way.

Remember the Problem You're Trying to Solve

We should have realized we had a problem the moment we started trying to fix the cache. The cache isn't a feature of the software: the cache is means to an end (faster software!) Focusing on actually reducing database usage throughout the application in sensible ways – rather than adding them and shuttling them into a cache once they've run – is going to serve us far better.

Let us know what you think.

(Reposted from Andy's Blog)

Did you know that a major 4g wireless provider uses concrete5 across their web presence, which serves 66m+ pageviews a month?

How about the largest publisher and distributor of children’s books in the world? 

The UK division of a well known american car company? 

Matt Mullenweg, the co-founder and leader of Wordpress is calling him out on it; “when you violate someone’s license it is breaking the law. It’s a definition of breaking the law.”

Chris Pearson says he doesn’t have to release his theme under the GPL “They are not the highest authority node up on the tree that gets to decide everything that happens underneath them.”

You can watch them get all sassy with each other for an hour, if you want. Plenty of people have and some of them are asking for our view on it.

concrete5 is a competitor to Wordpress in ways, and we had to choose a license when we started as well. We specifically did NOT choose the GPL for exactly this reason. Here’s how we see it…

1) Much of this is about distribution. Our understanding of this is pretty simple:

If Thesis is distributed as a stand alone download that /includes/ a copy of Wordpress, then Chris is legally wrong. If you distribute GPL software, everything you add to it has to be GPL compatible. Clear and simple, period.

If Thesis is distributed on its own, I think Chris might have an argument to make. It’s easy to think “hey this does nothing without Wordpress, it is dependent on it, it needs to honor whatever legal requirements Wordpress comes up with” but I don’t think that’s technically true. The GPL is about the copying and distribution of software and it doesn’t really cover this with a tremendous amount of clarity. There’s plenty of examples that would have been a lot more interesting to discuss than what they did in the interview. For example, just because you’ve written software that runs on Linux doesn’t mean it has to be GPL. If you distribute Linux WITH your software as a single solution it does however. How’s that for weird?

Regardless, Chris would have a much better high ground to stand on if Thesis actually worked with different CMS backends – much the way that C# application you wrote for linux could also run on a variety of other operating systems.

2) Matt seems like an awful nice guy, and Pearson comes off like a total douche in this interview. I’ve never met either, I’m sure they’re both awesome, I’m just saying after losing an hour to listening to this crap there’s a pretty clear answer for who I’d like to have a beer with. That’s a tremendous shame because frankly Chris is the underdog here trying to build and maintain a nice small business and Wordpress is the big player trying to squash entrepreneurialism. Regardless, Matt comes off as the hero cause he’s a nice guy and Chris comes off poorly because of the way he makes his arguments. Important lessons there, it’s probably time for us to do a better job stripping drupal references from our customer testimonials. 

3) Wow you can tell the difference that some funding makes. Let me just be clear about what I believe to be the real motivators here, please correct me if I’m mis-informed: Wordpress  Automattic has raised over $40m in venture capital. They have over 25 million blogs out there, and fundamentally they are in the content business. They don’t make their real money by selling wordpress, or taking a cut of marketplace add-ons, or offering paid hosting, or any of the stuff we do, they make their money on content. The advertising value on wordpress.com is huge. You have 25 million individuals using your platform to create content, you can monetize that in big ways. That’s why wordpress may be frequently used as a CMS to build some corporate site, but you’ll never see their core team drop features that help my wife (who has an active wordpress blog about DIY sewing), in favor of features that make some corporate extranet easier to run. Matt doesn’t have to worry about making payroll in two weeks, he has to worry about balancing ads and content on Wordpress.com so my wife keeps going there to find other cool sewing blogs she wants to cross link to. Wordpress’s real competitors are Twitter, Facebook, Google – they’re in that big business of re-inventing media. That’s why the GPL makes sense to them. The more wordpress is out there, the better for wordpress, as long as it’s called Wordpress.

Chris on the other hand is selling a Theme that helps turn Wordpress into a application that does something more. Again I’m just guessing here, but it wouldn’t shock me at all to hear Chris’s company is self funded, profitable, and it hasn’t been easy to get there. The idea of having a product that you sell at $50 a pop being distributed for free or even worse sold for $49 somewhere else has to make him physically sick. The carrot of “but people will want you for support” is a pretty grim answer.

I’m not arguing that Matt has an easy life and Chris doesn’t. Certainly the stress of looking Phil Black in the eye and saying “yes your $40m will turn into $800 million, sir” can’t be fun. I’m just saying the two challenges are very different and you can read the distinction in motivation from just the tone of their voices alone.

4) The GPL is stupid, and O’Reilly did us all a tremendous disservice when he came up with “open source”. Yeah I said it, so blah! When I was a developer growing up in the 80’s, we had licenses that actually meant what they said. If you wanted to just give something away, you called it Freeware. If you wanted to save some money on sales but still own your software, you called it Shareware or Crippleware depending on if you offered a fully functional copy with additional features or if you did something like disable save. These labels came from the DIY software world where entrepreneurs could start successful businesses cheap by distributing stuff on BBS’s. (go look up Apogee Games). Meanwhile there were any number of “big” projects that were being distributed under licenses that made sense for schools and huge corporate problems. NASA develops some standard and wants to share it with the world, how do they do that? Several big software vendors see value in a piece of software existing, but not being “owned” by any commercial entity, how do they do that? Everyone wrote their own license and while it was confusing, it worked. Then in the late 90’s the successful technical book publisher O’Reilly came along and dubbed everything I’ve listed as “Open Source” for the benefit of the media which was having a hard time understanding how Linux could compete with Windows. Well that’s cool and all, certainly having concepts that everyone can understand in a word is great, but clearly we aren’t really there. Confusion abounds. People talk about “free beer vs. free speech” all the time, it sounds like a broken record. Any one with half a brain knows that nothing worth having in life is truly free (in cost), yet we also agree that the idea buying a car with the hood welded shut sounds like getting screwed. The goal to provide some clarity across all the different types of licenses that software was released under by calling half “open source” and the other half “commercial” has utterly failed.

5) You say you want freedom? Then the GPL isn’t for you. It is not “freedom” to force people who extend your software to honor ANYTHING you say. I’m not saying it isn’t a good business idea, I imagine it may frequently be a great business idea, but it’s not “freedom” so don’t try to take the moral high ground. You’re limiting people and it doesn’t matter that the perceived motivation of your limit is to enforce further freedom. Freedom doesn’t work that way, but proponents of the GPL seem to think it needs protection. Here’s how we see it:

If you’re for the GPL, you believe freedom is a fragile flower that has to be protected. “This started as free, we’re going to make sure it says free with all our impressive powers.”

If you’re against the GPL, you believe freedom is a force of nature. It may not look that powerful at a glance, but it’s gonna win in the end. It’s like entropy. It exists, it will win. It doesn’t need your help, all it needs is your awareness and faith, and sooner or later it’ll come out on top.

Freedom is the MIT license which to paraphrase in three words says : “Don’t sue us”. If your goal really is to give something away for no cost and have the world be “free” to do whatever it wants with it, that’s all you need. Limit the creators exposure to liability, which would limit their own freedom, and you’ve made it “free.” Of course if you do that you run the risk of someone taking your software packing it up and screwing you over in any number of ways, but no one said freedom was easy.

These issues with the GPL are not new, and it’s sad to see this play out yet again. Frankly I like to think that any legal document’s job is to create clarity, and whatever your view may be, its clear the GPL is pretty gray in spots. In some ways, I hope this does go to court so we can all get a clear answer on how this thing is supposed to work.

Meanwhile if you want to be part of something that is free, and is eager to be free in a simple understandable way, you should be developing stuff for concrete5.

UPDATE : Orrrrr I’m completely wrong.

As more debate continues in IRC and other forums a point has come up that we didn’t address in the original post. Thesis uses wordpress’s theme engine and that includes any number of lines of code that wordpress wrote. Clearly that is their work, covered by their license, and Thesis is a derivative of it. THAT being the case, he very well may be violating the GPL. What gets interesting there is where is the line for that not being derivative? If he just goes through and renames all the functions and variables but it functions the same way, is that new work? What if he changes some logic too, for loops become while loops, etc. Where is the line where something is no longer derivative but a new thing?

What if Thesis makes an abstraction layer from scratch that does nothing but give them some differently named hooks to the same stuff, and then releases THAT abstraction layer LGPL and continues to sell their theme? That sounds legal, annoyingly stupid but legal.

Regardless the fact that everyone’s so confused about this does bring serious questions to the fore on the worth of GPL and what ‘freedom’ means. I hope we find out.

Going open source has provided me the opportunity to run into any number of people who are very very bright at one thing, don’t foster that fundamental curiosity about topics they have no expertise in, and yet seem to think their impressive intellect makes their opinion right on all topics regardless. Just because you’re amazing at the 100 yard dash doesn’t mean you know dick all about water polo.

I actually had one person complain about their in-ability to do something in concrete5 with the line “look, I /am/ a rocket scientist at such and such university.. I should be able to XYZ.” Strangely I am not a rocket scientist and I dropped out of college, but I can do XYZ quite easily. Go figure.

I was reminded of this just now when a thread on slashdot came up. Some fellow is chastising Apple for calling their new iPhone screen technology “Retina Display.” Apparently this “marketing drivel” has so offended this guy he feels the need to rant about it on slashdot:

“Again though, why the use of meaningless words? Couldn’t he have just said “the resolution/DPI is so dense that your eyes won’t be able to distinguish individual pixels”? What, does the average Apple customer really seek the need of some special word to wrap up the device’s capabilities in? And if they do, what does that say about their average customer?

I think it’s insulting to the people that buy Apple’s products, regardless of whether people seek it out or not.”

from slashdot.org

Holy crap dude! For real?? Yes. This is what we have language for. You use words and phrases to sum up larger concepts so a conversation can happen at an acceptable pace. Words do create some ambiguity as definitions tend to be subjective, but without some consolidation it’s difficult for anyone to get beyond facts and into useful concepts in a real world situation. I mean clearly this individual is smart enough to understand what a pixel is, how DPI might work, etc… But the basic purpose of language (were not even talking about marketing yet!) seems to not only escape them, but insult them as well.

I take two lessons from this:

1) I am a moron. This one shows up a lot in my life lessons from open source. So far I find the dumber I assume myself to be, the more pleasantly surprised I am when I get something right. If I am hearing about something that others think is awesome and my geek-gland says “that’s stupid marketing drivel,” chances are they’re right and I’m wrong.

2) Just because you’re talking to someone “smart,” doesn’t mean they have a clue what they’re talking about.

We also didn’t see a tremendous amount of intentional abuse of our systems. While we did build some large, active, and successful sites, the code behind them was only open to people who had worked with us in the past. To break one of our sites in the commercial days, you’d simply have to guess at vulnerabilities instead of being able to scour code for them first.

Now things have changed. Every feature idea we have is more and more tempered by “what will this do to existing sites or old versions.” We’ve learned about (and quickly addressed) any number of vulnerabilities that some guy in his basement found for free – when paid consultants had found none for five years before. You quickly learn that even the most well intentioned work can cause havoc.

For example, there are two blocks in concrete5 that are commonly used to build navigations: the Auto-Nav block and the Page List block. The Auto-Nav block had been built to honor a certain variable you can set at a page level to hide that page from the navigation. The Page List did not honor the same attribute, and someone from the community pointed out that it really would make more sense if it did. We agreed and “fixed” it as part of some other version changes. Weeks later, people started complaining that their sites we’re missing pages. After some frustration we realized “duh” of course people had built sites that worked around the way the blocks behaved in the past and the simple “fix” to the Page List block actually broke their sites.

We spend a lot of our time trying to manage issues that could mature like this now. I tend to take much longer to release something than I used to. I tend to be more thoughtful about the reasons and needs behind any feature changes. The temptation to “fix” something that could have been better implemented in the first place is very strong. Learning to first resist and then very delicately architect not necessarily the perfect, but rather the lowest impact solution, has been a new adventure for me. Once you go open source it is safe to assume that someone somewhere who is smarter than you and has all the time in the world, is finding the mistakes, finding the holes, and making up their own weird work-arounds which will impact you later. Be careful what you touch because no good deed goes unpunished.

concrete5 core is free and open source. When we say free, we mean “free beer.” Our belief is that content management is a human right, and we are committed to making it easy for everyone in the world to run a website.

However, not every add-on in our marketplace is free. All of them are open source – meaning once you buy it you are “free” to do what you want to it for that site, and you can get “under the hood” completely.

Why do we do this?
A lot of time and money has gone into concrete5. Anyone who doesn’t think we’re generous has vastly underestimated the amount of energy that goes into making a powerful CMS that makes sense and doesn’t require an expert to configure. We are people too however and not only do we need to provide for our families, we also need to continue to put the level of coherent leadership into the project that it benefited from when we were strictly commercial software for 5 years.

A huge part (IMHO – most) of what makes concrete5 so compelling over other projects like Drupal and Joomlais the fact that it does take the risk of saying “this is the right solution” to many problems. No, we don’t believe you really need 300 results for “permissions” when searching for add-ons to a project. How about manning up and just getting the core solution right? That’s our philosophy. We don’t always hit the mark because we’re human, but we’ve done pretty well so far, and that’s the goal for the future as well.

Same deal with the marketplace. Other projects seem to have a pretty low barrier of entry for add-ons. I’m not entirely sure if there even is any barrier, but if it exists, surely the question is “does this add-on work with a clean install of our app? Didn’t break? Okay give them a project area – goooo Open Source!” Well bravo for Free Love and everyone being Super Duper, but I see that as unfair to the next schmoe who is trying to figure out how to solve their own problem. If I download a weather widget and my whole site breaks because it uses the same table as some image gallery I already had running, everyone fails. The weather widget developer looks like an ass, and so does the image gallery developer. Both end up doing way more individualized support than they should to keep their customers happy. Moreover, the overall project fails because now no one can trust any add-on to do anything easily. This is where Drupal certainly is today, just look at Acquia’s business model. This is unacceptable to us.

At concrete5.org you will only find things that work. If they don’t work, we’re gonna make them work for you. $15, $55, or even a couple hundred bucks is a small price to pay for something that solves a real business need for you and is going to work in a seamless, happy, healthy way. When we evaluate a new add-on, the question will be “does it work on an install with EVERYTHING added?” This is a huge challenge, but we think it’s going to be critical to the success of our project in the big picture.

How do we decide what is in core?
Anything that will make a fundamental change to the way concrete5 works which would impact all add-ons and benefits the community/project is free. So recent additions that fall in that category include things like:

• A My Profile section that various add-ons like forums or ecommerce would depend on.
• An advanced file system that all add-ons can use.
• A better way to create shared central blocks.

The reality is that with other projects like Drupal, once you’ve installed one modification to the way core permissions work you’ve effectively rendered their massive marketplace to you. How can the huge community really even help when everyone’s configuration is a unique one off? We’re going to do everything we can to keep this project from splintering into core pieces that don’t work with one another and render all subsequent add-ons a crap shoot.

Anything that we think is a basic building block to 80% of the websites out there in the world, we’ll make either part of the core or free in the marketplace. So things around embedding most types of content, some interaction like guestbooks and forums..etc.. We’re not asking “would everyone benefit” – because who doesn’t want free stuff, but rather “do you /need/ this to get your point across with the software.” Some examples:

• You can place banner ads using the Content Block, the HTML block or the Image block and your site visitors will never know the difference. Want to track click-throughs, impressions, and pull from centralized ad groups that randomize choices? You can do that, you can have it TONIGHT! That costs $55.
• You can assign a date to any page in concrete5, so it’s possible to make many different type of chronilogical interfaces with the core. You can also just include a Google Calendar with the HTML block. Want a month view, list view, and ajax driven agenda view to a multiple calendar system that makes event pages spread across your site? Want that working NOW? You need to pay something too.

How do we price things?
We make it up. We don’t frankly care how long it took us to write it, we don’t care how much the competition sells it for, we’re guessing how much you’re willing to pay to have it “just work.” No lie. This is business 101.

But wait, what about the Community??!?
Here’s some promises to our community we’ve always kept and always will:

• Your stuff can go in our marketplace. We don’t care if you’re selling it or giving it away, if your able to give us a stable, solid, correctly packaged add-on for concrete5 and we don’t think it’s malware, we’ll stick it in the marketplace. I can’t promise you we’ll feature it above our own in every interface view, but we’ll gladly post your free image gallery block right up there next to our own $15 per site one. If yours is better and you can make the community happy using it, so much the better for everyone.
• We will help you sell your own stuff. Software is about support as much as creation. If you’re making something and giving it away, you might consider selling it and making a buck. Getting out of the purely hourly revenue model is the dream of almost every developer out there, and we think we’re gonna make a lot of dreams come true with this awesome marketplace. If you’re making stuff that people want, you should want to help them install and use it safely. You should want to add to it over time. You should be compenstated for your efforts. If you’d like to sell something you’ve built in our marketplace, all we ask is 25% cut of the price. This is less than Apple’s iPhone App store and from what I can tell comperable toDot Net Nuke’s system if not better.
• This stuff is not going to be super expensive. I’ve seen libraries that take this approach where solutions cost many thousands of dollars. Crap, I remember buying a Digital Asset Management system from the Cold Fusion marketplace back in the day for 8k and still shoveling out 30k to the developers to customize it to our client’s needs. This isn’t the model here. The highest price we’ve even debated setting a product at so far is $255. That’s what most targeted consumer software is priced at today. There’s no five figure recurring yearly license fees here.

What does this mean for the project?
It means a lot of great stuff. It means we’re going to end up building a community that is not only passionate, but is actually making a better life for themselves and their families out of their contributions. It means when you go shopping for add-ons for your site, you’ll be able to do it with a smile on your face and try stuff out without fear.

Is it open source? Absolutely yes. Open source as a term is really just a catch all for any number of different license types from the 80’s and early 90’s when we were cutting our teeth in the BBS scene, and this idea honors them all quite well.

If you’re still not sold, ponder this:
THE MAN is actually just A Man.

Thanks for making it though this rant, hope you agree – I’m sure you all won’t. If you don’t I’m all ears on constructive suggestions. If the reply is “it should be free because I want it to be, and it’s not my problem how you or the project succeeed in the big picture…”… the door is that-a way. <grin>

Usagi Project will attend Tokyo OSC with concrete5 Japanese version.

Tokyo Open Source Conference 2009/Spring
Japan Electronics College Building No.7
1-25-4 Hyakunin-cho
Shinjuky, Tokyo, 169-8522
Japan

東京オープンソースコンファレンス2009/Spring
日本電子専門学校 7号館
東京都新宿区百人町1-25-4

RSVP the seminar at
http://www.ospn.jp/osc2009-spring/modules/eguide/event.php?eid=76

Also you can just show up at our demo booth during the event.
———-
Feb 20 (Fri) 10:00am -5:30pm
Feb 21 (Sat) 10:00am -4:30pm

Date & time for first meeting
Feb 20 (Fri) – 21 (Sat) (Seminar starts on Feb 21 11am)

In the meantime, here’s an email thread I had recently with a new c5 fan where I lay down some of the ideas and plans we’ve been putting together as we watch our baby take off here:

> General feedback was submitted to ConcreteCMS.com. Here is the information.
>
> Name: Dennis
> <stripped>
>
> congrats on your CMS! we are impressed and actually thinking about using it for our clients.
>
> we are a design agency based sydney australia and have a variety of clients, from local hairdressers to government departments. we are looking to use an open source cms to use for our small to medium sites. your system so far seems to be the most user friendly one.
>
> I have worked with other systems before, like joomla and wordpress, etc. now all of them have a massive following and ten thousand different modules, while your system seems to have a limited amount of modules (which is actually quite nice). but what if we need a certain module? could you develop it for us and can we re-use it on projects?
>
> let’s say a blog – writing an article, getting comments (display upon approval), etc – is something like this already developed? if not how much would it be – just an estimate as the specs are a bit vague obviously?
>
> and last but not least, would you feature us on your page as design partner if we return the favor?
>
> that’s it for now from my side.
> looking forward to hearing from you.
>
> cheers
> dennis
> www.digitalgarden.com.au
>

Hey Dennis,

It’s nice to hear from more folk as excited about c5 as we are. Thanks!

In short – yes.

In detail:
1) I think c5 would be perfect for your shop. We too have been frustrated by the lack of coherent control, or scalable architecture in many of the CMS solutions available. We plan to always keep the core of c5 simple and approachable. It’s our belief that most website development challenges can be solved with a dozen or two well architected blocks, and that’s what we’ll be shooting for as we continue to get to the “perfect core” in 2008.

2) We do have a guestbook and blog structure in a previous version of concrete that we’ll be migrating as part of that “core tools” library, I’m sure. Also slated for that: more easily customized navigation controls, multi-lingual interface, a cleaned up advanced permission model (which right now has to be turned on and isn’t quite as elegant as the rest of c5.)… all sorts of more useful goodies..

3) We will be launching a marketplace for blocks and themes shortly as well. If you as a third party developer are interested in making and reselling either, you’ll be able to do it there easily. We’ll also have a job board and something similar to etsy.com’s alchemy where you can request or pledge for developments to c5 that other developers could build for. We’re thinking there will be a 10% commission for the c5 core team, and there may be paid placement opportunities on that site as it matures as well.

4) There will also be a hosting site. It won’t be the cheapest place in town, but it will have a very stable centralized version of c5 running with some nice backup/redundancy options.

5) I would love to feature your email and this reply on our blog at concreteTheStudio.com if that’s amenable to you. Show me some sites built out of c5 and we’ll talk about the Support page of concrete5.org, if you’re interested in that as well.

if you’re ever in pdx, beers on me.

thx again
-frz
ps: i love your site. nicely done.

However, someone from your end must of been at osCon because we appeared on your site a few days ago. Here’s a snapshot of our google analytics for the last month:

Gotta say, we were gonna wait till we had a release we were calling final before posting to you,OpenSourceCMS.com. The fact that we just magically showed up is great! We’ll just take that as a pat on the back that what we consider Beta is pretty damn stable, and we’d like to say thanks.

(ps: hey reader, wanna help? vote for us on their site. when they first added us they linked to our demo in such a way that it wouldn’t work so we got some low votes that are messing up our average.)

I get that you want to provide for your family, sooo what are you thinking?
Are we going to offer a “freemeium” model where you get crippleware for free and the useful parts in expensive add-ons?
Nope.

Are we going to have a different license depending who you are?
Nope.

Are we going have a donation button or something?
Yes, but it will point to our favorite charity, which can do more good with the cash than us.

So I give up, why do you think destroying your perfectly viable license revenue is going to provide stability and creative freedom?

Here’s what I see. The biggest challenge my crew has is bizdev. We’re not perfect at everything, but we sure can deliver sweet stuff and we improve every day, execept for sales.

We’ve only really won good gigs through word of mouth. We’ve tried just about everything, and without marrying yourself to a particular vertical, it’s very difficult to define a meaningful marketing strategy for a web/IT services company that wants to do “cool stuff.” From my experience you do your best, and try to cultivate as many life long associates and friends who will recommend you as you go.

As the network slowly grows, things get easier over time, but it doesn’t really deliver with security and creative freedom if it ties you to a limited local gossipy scene. (yeah i said it).

So while completely giving away something we have and can charge a lot for, we’re actually doing ourselves a practical favor. Sure, we’ll be giving up a revenue stream, but we’re dropping a expensive business development challenge that we’ve never been good at or interested in solving. We certainly will still spend some real resources to make Concrete5 known – but a lot of that can be our time instead of cash. Moreover, if what we’ve been working on all these years is really as good as we think it is, we stand to jump-start a process that would traditionally take much longer. I’m interesting in seeing what a larger open source developer community might contribute to the project from a code standpoint, but I’m hungry for their evangelism about concrete5 to their clients. I don’t need (or want) to own every dollar that is made off of concrete5. Why not just get out of the way and respond to opportunities as they arise as thousands of people deliver concrete5 powered solutions to their clients?

That’s the practical reason to go “free beer.” The real one is better:

Content management is a basic human right.

It costs next to nothing to write your thoughts on a piece of paper and nail it to a door, it should cost about the same to make a basic website without it having to be a blog. If we can do that, we’ll win one way or another.

We’ve come to recognize it’s the brand. We will trademark our name as Concrete5™ – and make money by being the official host, trainer, documenter, and support provider. Conversely we may look at any of those roles and tap a better suited partner as an “Official Concrete5 Solution” in return for some license or revenue model.

The Ruby on Rails guy looks to have similar ideas around his brand and license model, which is also MIT.