Cache Causing Speed Problems

Permalink 4 users found helpful
I'm having a problem with the cache settings on one of my concrete5 sites. Upon launch, I moved the site from my testing server to its live server, and turned on all the cache options. Within an hour, the site was having major issues. It was taking anywhere from 11 seconds to several minutes for each page to load, and sometimes it was timing out and failing to connect to the database at all.

I contacted the host (Bluehost) and they couldn't find any problems on their end. My client upgraded to a Pro Account in hopes that would help. I also went in and cleared the cache. At first, the new hosting account seemed to do the trick. The site was loading better, though still it was slower than it seemed it should have been.

I set up a check with Pingdom and have been monitoring the site for the last couple of weeks. After the initial drop-off in page load times, response times steadily began to creep back up. After a few days, the site was almost as slow as it had been before the upgrade.

I went in and cleared the cache again, and turned off full-page caching. Then I watched again, and the same pattern started. Page-load times increased by the hour. I went back in, cleared the cache and turned off the Overrides Cache. Page load time starting crawling up again. Finally I turned off all caching, cleared the cache one last time and since then the site has been loading in under two seconds for many days in a row. I've attached a graph that shows the period of time the site was slow.

I mainly wanted to bring this to the community to see if anyone else is having this problem, and if so, perhaps gain some insight as to why this would be happening. This site gets a decent amount of traffic (600-900 visitors per day). When I ran tests on the page load times, the lag was in the initial connection to the sever. The page would "wait" for up to 30 seconds sometimes before the first page elements would start to load.

The site is here:
http://clubpilatesstudio.com/

Site is running on concrete5 5.6.0.1.

1 Attachment

brownwingstudio
 
theone85ca replied on at Permalink Reply
You're not the only one.

I've had exactly the same issue with exactly the same fix. VERY strange indeed.

Thanks to Phallanx for turning me on to this thread.
severas replied on at Permalink Reply
severas
I am having the same problem, too, VERY frustrating.
andrew replied on at Permalink Reply
andrew
I wouldn't turn off the overrides cache. There's really no reason not to have it on and it doesn't use the other caching system that is slow on some servers.
Phallanx replied on at Permalink Reply
Phallanx
I wouldn't turn off the overrides cache. There's really no reason not to have it on and it doesn't use the other caching system that is slow on some servers.

How does it work?
severas replied on at Permalink Reply
severas
Okay, I found a temporary fix that worked for me at least.

I had to make ALL of the following changes. Doing them one by one, had no effect. Within a few hours of clearing the cache, my website (http://www.dmr.fm ) would be taking up to 15 seconds to load

Turn 'Basic Cache' Off.
Turn 'Full Page Caching' Off.
Set Full Page Rebuild to Automatic.
Clear the Cache.

After applying all of these settings, my website is loading in about 2-4 seconds instead of 8-15. We've been running it on these settings for about 72 hours so far, with no drop off in load times yet.

Of course, these are NOT ideal settings to run a website in. Hopefully Concrete5 issues an update to fix these new problems ASAP.
brownwingstudio replied on at Permalink Reply
brownwingstudio
Sounds like exactly what I experienced. And I agree, a real fix for this would be nice.
andrew replied on at Permalink Reply
andrew
AGAIN: I wouldn't turn overrides cache off. I will by shocked if you turned it On and then saw any kind of speed degradation.
brownwingstudio replied on at Permalink Reply
brownwingstudio
You are right. I turned my overrides cache back on yesterday, and the site is still loading at the same rate. Thanks.
Phallanx replied on at Permalink Reply
Phallanx
@brownwingstudio
Meaning it doesn't do anything?
brownwingstudio replied on at Permalink Reply
brownwingstudio
Yeah, no change at all as far as page load times are concerned.
WHW1dotCOM replied on at Permalink Reply
Is this issue only on 5.6.x, or does it also exist on 5.5.x?

The "Overrides Cache" setting does not even exist on 5.5.x, right?
brownwingstudio replied on at Permalink Reply
brownwingstudio
It's version 5.6.0.1
thebigeventpage replied on at Permalink Reply
thebigeventpage
I am so glad you posted this step by step way of speeding up the load time on my site. I am a newb, technically I know nothing and I have been tearing my hair out for weeks and working with Dreamhost and tweaking my site to try and figure out why my site loads so slowly, up to 35 seconds at time.

After I changed the cache settings as suggesting it loads a lot faster.

Thank you for posting this!
WHW1dotCOM replied on at Permalink Reply
Well, whatever your current settings are, last I checked, it took about 10 or 11 seconds to load on a fresh new visit. That's not so bad, and fine, I think.
brownwingstudio replied on at Permalink Reply
brownwingstudio
Yeah, the site is fine with the cache off. I still just don't understand why turning on the cache would have caused such a massive problem and was hoping for some insight.
Phallanx replied on at Permalink Reply
Phallanx
@brownwingstudio
I've just been playing with the dispatcher.php. I stuck in a load of microtimes after every "section" to see what was taking how long.

It might be a worthwhile thing to do on on your set-up.
Spletnapot replied on at Permalink Reply
Spletnapot
The only real solution is a host that has enough resources.
If site runs ok at certain hours and than like a snail... just means that some other site on hosting is using most resources at that time.
Cache on or off.... does not matter at that point.
brownwingstudio replied on at Permalink Reply
brownwingstudio
I don't think the host is the issue here. The pattern is very clear. When the cache is off, the site runs fine, always. When the cache is on, it runs slower and slower and slower until I clear the cache. Then the site runs better for a while, then gradually gets slower and slower again. If I leave the cache off, the problem is entirely absent.
grosbedo replied on at Permalink Reply
Same problem here: website was VERYYYY slow, and then I've stumbled upon this thread and tried to disable all cache (and clear the cache just after).

Result: blazingly-fast website!

In the end, I still turned on the Overrides Cache, thank's to Andrew, and indeed it speeds up a bit the website.

Another thing I also did: change the Tracking Code from header to footer, which also helps a bit (quite important if you have Piwik as I do, since it's kind of a memory hog).

I am also sure that it's not related to the hosting, I've done several tests with pingdom.com and they all come down to this conclusion: the cache system of the latest Concrete5 is faulty (except Overrides Cache which seems to not share the same mechanism).

Thus, meanwhile we get an update: if you encounter some slowing issues with your Concrete5 website, disable all caches but Overrides Cache, then clear the cache. Also you can set the tracker code to footer if you didn't.
mkly replied on at Permalink Reply
mkly
The caching system is not faulty as much as once you get a large number of cache entries you can run into several issues which can hurt performance instead of help.

Here are some resources

http://andrewembler.com/posts/improving-the-performance-of-zend-cac...
http://www.concrete5.org/marketplace/addons/clear-cache/...
http://www.concrete5.org/documentation/how-tos/developers/configure...
http://www.concrete5.org/documentation/developers/system/caching...

Hope that helps a bit,
Mike
Phallanx replied on at Permalink Reply
Phallanx
@mkly
If it is not broken, then perhaps you can explain this to me since I am struggling to understand exactly how this works

This is an output from a benchmarking script when caching is turned on. The script is a modified dispatcher.php which has logging calls inserted throughout the script and just logs the time between calls.
0->start: 1350147643.5064
1->Requires1: 0.01619
2->autoloadCore: 2.0E-5
3->Cache::startup: 0.04849
4->app.php: 0.00304
------------ repeated----------
3->Cache::startup: 0.03305
4->app.php: 0.00356
------------         ----------
5->Requires2: 0.06851
6->Config: 0.00016
7->Includes: 0.00125
8->Requires3: 0.01111
9->site events: 9.0E-5
------------ repeated----------

The number on the left hand side is an incremental number every time the log function gets called. I expect this to increment by one for each entry in the result. I have manually added the "---repeated---" to highlight identical calls.

Question:
Why are there repeated calls to the same section of the dispatcher.php (e.g Cache::startup) and how is it possible to jump straight to an individual section of the php file without first calling previous logging functions (i.e recall 5,6,7,8,9 without first calling 1,2,3,4)?

Are we looking at function caching and, if so, why are the functions called multiple times with the same arguments?

Here is one with caching turned off for comparison
0->start: 1350149018.8083
1->Requires1: 0.01725
2->autoloadCore: 2.0E-5
3->Cache::startup: 0.02627
4->app.php: 0.00352
5->Requires2: 0.06597
6->Config: 0.00036
7->Includes: 0.00154
8->Requires3: 0.01208
9->site events: 8.0E-5
<snip>

Note there is no replication and the numbers are contiguous.
mkly replied on at Permalink Reply
mkly
That's some great investigating. Do you have a github account? It seems like you have a solid understanding of concrete5 internals but I don't remember any pull requests from you solving the things you discover are incorrect.

Maybe it was before I was around? Anyhow, feel free to contribute as I feel like you have some great ideas. I'd love to see it.

As for those details, I can't be sure.

Could you attach some line numbers to the current dispatcher.php on github here?
https://github.com/concrete5/concrete5/blob/master/web/concrete/disp...
What do you think is causing these issues?

And better yet, attach you profiling script?

Thanks,
Mike
Phallanx replied on at Permalink Reply
Phallanx
@mkly
No. I don't use git and as I said, I don't know what is going on (only that it smells like a kipper behind a radiator). The mod (at present) replaces the core dispatcher so can't really be integrated with the production version. That's not to say you couldn't put a debug switch in if you are so inclined. However, if the cache is broken (and we all suspect it is), then it is up to the core team to sort out such a fundamental issue (or at least respond to my post with an explanation)

I've already posted a version of the benchmarking mod on this thread
http://www.concrete5.org/community/forums/customizing_c5/slow-site1... (although it doesn't have the numbering) to try and figure out what is going on with that poor devils site.
mkly replied on at Permalink Reply
mkly
If your worried that there is a conspiracy going on, I'm pretty sure there isn't, and I'm sure any help would be very much appreciated. I can't see why anyone would want something to remain "broken" as you say.

And if you ever do get a github account it's at

http://github.com/concrete5

Best,
Mike
TardyOne replied on at Permalink Reply
@mkly
Conspiracy? No. Apathy?. Yes.
andrew replied on at Permalink Best Answer Reply
andrew
It's a matter of priority. Fortunately, we're making time on this right now.

Unfortunately, the file-based caching of the Zend Framework really doesn't work well for the number of objects that we're trying to stick in it. So we're going to take a different approach for the next major release of concrete5 (one that people probably would say we should have taken all along.)

1. Remove the usage of Zend Cache for all but a few small things (e.g. caching remote information about the picture of the day.) The library remains for applications and sites that need it.

2. Use in-memory once-per-request caching frequently, to ensure that the same database queries aren't repeated during a given request.

3. Try and actually reduce the number of queries that concrete5 runs. This is part of the "First make it work, then make it fast" philosophy: we've spent a lot of time on the "make it work" part - now comes the second part.

4. For block caching, store the fragments in the database instead of in Zend Cache. (Note: this might slightly reduce performance for systems that use APC Zend Caching, but it will improve performance for the 99.9% of sites that stick to the defaults.)

5. Make our full page caching fire much, much earlier in the process (e.g. if a page is in the full page cache, no database connection is made, and the page is retrieved before loading 99% of concrete5's code.)

6. Minor optimizations throughout.

If this sounds good to you, you might consider checking out the 5.7-performance branch. There are a lot of things happening in it. It's not production stable nor is there really that much to see in it. But under the hood things are happening.
Phallanx replied on at Permalink Reply
Phallanx
@andrew
That's great. But what about the ones that are suffering now? (and additionally, the many like me that cannot upgrade if/when there is one because the upgrade is also broken).

Are you saying that you acknowledge there is or has always been a problem with the cache and now understand what the issues are? What is your comment on the questions I raised? What is the "technical" explanation of why the cache can cause a site not to respond for 10s of seconds and deteriorates with time?

I really don't want to have to reverse engineer your whole caching system just so that I can help others when you know it inside out already and aren't! Throw us a bone here!
andrew replied on at Permalink Reply
andrew
If people are suffering now they should probably just turn it off – or better yet, if APC is installed on their server, keep it on but then add

define('CACHE_LIBRARY', 'apc');


into their config/site.php.

In general, the main problem boils down to the Zend file caching being slow – or rather, just ill-suited to the sheer number of cache entries that concrete5 throws into it. Additionally, Zend Cache with the file backend has an issue where its garbage collection just seems to thrash a site and toss it into oblivion, if the site gets larger than some kind of nebulous point.

http://andrewembler.com/posts/improving-the-performance-of-zend-cac...

However, this isn't a permanent solution. In the past, we've tried to use the cache minimize the sheer number of database queries that concrete5 performs. Unfortunately, in some cases that meant actually adding to the number of queries, and then caching the resulting object. This actually leads to more queries, if the cache is off.

So instead, going forward, we're going to be removing reliance on Zend Cache for most things. Block-level record and output caching will be moved to the database, and in-memory caching should minimize most duplicate queries. Full page caching will be improved.

Finally, I believe the 10 seconds you're referring to is quite likely the problem with Zend Cache file garbage collection. Implement the suggestion at my link above (and then manually remove cache files periodically), go with APC, or turn off the cache.
Phallanx replied on at Permalink Reply
Phallanx
@andrew
That is only really an option for VPS/kvm since you need a console to install APC.

I've tracked down some of the problems to the "local cache" and the curious behaviour of the zend cache when multiple items with the same ID are inserted. Analysis of the "repeated calls" seems to emanate from the switching and reloading of the dispatcher when you have upgraded (I'd be interested to see if some of the people that are suffering are all using upgrades?).

By the way. I have successfully replaced the zend cache by using SQlite as a cache. This would be much more desirable than storing in a MySQL or other type of database especially for those that have their databases residing on a separate server. However, not all web service providers have SQLite installed.
Iandesign replied on at Permalink Reply
Great.....

I have just spent the last week working out Concrete 5 only to realise the cache does not work. No point using it then is there.


Thank god for Phallanx for outing the truth of the matter.
Iandesign replied on at Permalink Reply
I'll wait for 5.7. See if it works then.
apc123 replied on at Permalink Reply
apc123
I have APC installed. So should my site.php follow contain both
define('CACHE_LIBRARY', 'apc');
define('CACHE_FRONTEND_OPTIONS', serialize(array('automatic_cleaning_factor' => 0)));

thanks,
Anthony
1234321 replied on at Permalink Reply
All the above suggestions did nothing for me, I read somewhere that the answer to the problem is adding the line to site.php apc123 already posted

1.define('CACHE_FRONTEND_OPTIONS', serialize(array('automatic_cleaning_factor' => 0)));

Question is, "where in the world is" site.php located.

I dabbled in cpanel in the past and I believe had access through ftp to the entire site, but so far I can't figure this out in Concrete5. help!
roketto replied on at Permalink Reply
roketto
site.php is in /config/
roketto replied on at Permalink Reply
roketto
Well, safe to same I'm very angry about this problem, and extremely happy that I've finally found the solution.

I've nearly lost one of my best clients thanks to this issue. This is no minor problem. For those who are unlucky enough to have experienced this, I feel your pain and I'm disappointed there hasn't been a formal notification from C5 on the matter.

This issue has been haunting me for the last two months, and it was incredibly difficult to investigate. I used tons of speed optimization tools, optimized all the resources on my site, all to no avail.

I'm just glad I've finally found the answer. Things are no loading much faster. Thanks so much Browningstudio!
Elstud replied on at Permalink Reply
Elstud
I put On everywhere in the cache, especially here abnd it's perfect now :

Put the full page ..
Off- manually for some pages.
On - If block are ready.
>>>>> On - everytime. <<<<<<<<<<<<<<<<<<<<<<