Caching Issue: Single Website Running Behind NLB with 4 Identical Servers
Permalink 2 users found helpful
Hello,
I have setup Concrete5 to run 4 identical servers all behind a NLB virtual IP. Servers are automatically selected for each visitor based on load and that visitor will remain on their chosen server until their session expires. All servers share the same DB.
Each server has the following configuration/features:
- Windows Server 2008 R2
- IIS 7.5
- PHP 5.XX
- WinCache 1.1
- Identical Concrete5 5.4.1 web root.
Each servers config and concrete5 files are synced using DFS replication. Syncing is almost instantaneous.
Additionally Concrete5 is configured to use WinCache instead on the default file-based caching from Zend.
Performance is wonderful. Without WinCache not so much.
What's the problem? Well if a user makes a change on Server1 the Server2 and the others won't refresh their cache and display the change until it manually cleared on that server.
For example:
- A user is connected with Server1
- The user logs in and changes text in a block on the home page
- The save/publish their changes
- The user logs out
- Another user visits the site
- NLB connects them with Server2
- Thy are taken to the home page
- A cached page is rendered without the first users changes
- As many times as the second user refreshes the page the first user's changes are never seen.
- If an admin logs into the dashboard on Server2 and clears the cache then the first user's changes will display
How can changes made one server clear the caches of the other servers?
I have setup Concrete5 to run 4 identical servers all behind a NLB virtual IP. Servers are automatically selected for each visitor based on load and that visitor will remain on their chosen server until their session expires. All servers share the same DB.
Each server has the following configuration/features:
- Windows Server 2008 R2
- IIS 7.5
- PHP 5.XX
- WinCache 1.1
- Identical Concrete5 5.4.1 web root.
Each servers config and concrete5 files are synced using DFS replication. Syncing is almost instantaneous.
Additionally Concrete5 is configured to use WinCache instead on the default file-based caching from Zend.
Performance is wonderful. Without WinCache not so much.
What's the problem? Well if a user makes a change on Server1 the Server2 and the others won't refresh their cache and display the change until it manually cleared on that server.
For example:
- A user is connected with Server1
- The user logs in and changes text in a block on the home page
- The save/publish their changes
- The user logs out
- Another user visits the site
- NLB connects them with Server2
- Thy are taken to the home page
- A cached page is rendered without the first users changes
- As many times as the second user refreshes the page the first user's changes are never seen.
- If an admin logs into the dashboard on Server2 and clears the cache then the first user's changes will display
How can changes made one server clear the caches of the other servers?
I completely agree with you that forcing all the cache to be cleared is ugly and bad for performance. However that is the only way to get all the servers on the same page.
I also thought the cache should be the same across all servers because DFS was replicating. Unfortunately that is not the case. WinCache is so much more efficient because it stores cache in memory not the filesystem like the default Zend.
I did do a test with file based caching and that replicated across servers just fine and servers were kept in sync.
So my real question here is how can I get the good performance of WinCache without the headaches of manually clearing it on other servers?
I also thought the cache should be the same across all servers because DFS was replicating. Unfortunately that is not the case. WinCache is so much more efficient because it stores cache in memory not the filesystem like the default Zend.
I did do a test with file based caching and that replicated across servers just fine and servers were kept in sync.
So my real question here is how can I get the good performance of WinCache without the headaches of manually clearing it on other servers?
I see. Unfortunately I don't have a lot of experience with WinCache, never used it for a production site.
But as far as I know it hasn't been built for such a set up and I don't think you'll be able to use it, unless I'm wrong about that.
Sharing resources across servers is going to generate some overhead which you quite likely can't avoid.
Personally, I'd work with memcache in such a situation. It's supported by Concrete5 (Zend Cache) as well and has been built for a configuration like you're describing..
But as far as I know it hasn't been built for such a set up and I don't think you'll be able to use it, unless I'm wrong about that.
Sharing resources across servers is going to generate some overhead which you quite likely can't avoid.
Personally, I'd work with memcache in such a situation. It's supported by Concrete5 (Zend Cache) as well and has been built for a configuration like you're describing..
I have not heard of memcache before. Does this offer the same full page caching as the other adapters?
Also I wonder what the process of getting this installed is.
Anyone have experience configuring it for win32/64?
Also I wonder what the process of getting this installed is.
Anyone have experience configuring it for win32/64?
Alright I almost have installed memcached as a service using MemCacheD Manager and pointed to the appropriate php_memcache.dll extension.
php_info() tells us that memcache is installed.
I can run a test script to try and store something in the cache and it works.
In the \config\site.php file I have CACHE_LIBRARY set to 'memcached'
But no joy nothing is caching in Concrete5.
Am I missing something obvious?
php_info() tells us that memcache is installed.
I can run a test script to try and store something in the cache and it works.
In the \config\site.php file I have CACHE_LIBRARY set to 'memcached'
But no joy nothing is caching in Concrete5.
Am I missing something obvious?
Yes, you have to specify some options. Not all cache backends need this but memcache runs on a server, the client has to know where to look for this server.
Look here for all the options:
http://framework.zend.com/manual/en/zend.cache.backends.html#zend.c...
You can then serialize an array and pass it to CACHE_LIBRARY_OPTIONS like this:
Look here for all the options:
http://framework.zend.com/manual/en/zend.cache.backends.html#zend.c...
You can then serialize an array and pass it to CACHE_LIBRARY_OPTIONS like this:
Thank you so much for you help.
I am trying to get this working on the first server without success. Below is my configuration:
/config/site.php:
php.ini:
Things seem to be installed correctly, however nothing is getting cached.
I am trying to get this working on the first server without success. Below is my configuration:
/config/site.php:
define('CACHE_LIBRARY', 'memcached'); $cacheBackendOptions = array ('servers' => array('host' => '10.176.60.10') define('CACHE_BACKEND_OPTIONS', serialize($cacheBackendOptions)); $cacheFrontendOptions = array ('servers' => array('host' => '10.176.60.10' define('CACHE_FRONTEND_OPTIONS', serialize($cacheFrontendOptions));
php.ini:
[PHP_MEMCACHE] extension=php_memcache.dll memcache.allow_failover = 1 memcache.max_failover_attempts=20 memcache.chunk_size =8192 memcache.default_port = 11211
Things seem to be installed correctly, however nothing is getting cached.
in the code you've posted are some missing brackets? Don't you get a PHP error?
Somehow the text was truncated when I pasted it.
define('CACHE_LIBRARY', 'memcached'); $cacheBackendOptions = array ('servers' => array('host' => '10.176.60.10')); define('CACHE_BACKEND_OPTIONS', serialize($cacheBackendOptions)); $cacheFrontendOptions = array ('servers' => array('host' => '10.176.60.10')); define('CACHE_FRONTEND_OPTIONS', serialize($cacheFrontendOptions));
When I try to clear the cache from the dash board I get this PHP error:
PHP Fatal error: Call to a member function setOption() on a non-object in F:\inetpub\concrete5\concrete\libraries\cache.php on line 71
Maybe this will shed some light on why things aren't working?
PHP Fatal error: Call to a member function setOption() on a non-object in F:\inetpub\concrete5\concrete\libraries\cache.php on line 71
Maybe this will shed some light on why things aren't working?
that might be because tags aren't supported with the memcached backup. Not sure if Concrete5 depends on it.
Would have to dig around the code for a bit but can't help you right now, got to go to Germany. I might have some spare time to look into this later in the evening.
Would have to dig around the code for a bit but can't help you right now, got to go to Germany. I might have some spare time to look into this later in the evening.
Okay, one more before I leave. Seems to work fine for me.
Used thishttp://www.splinedancer.com/memcached-win32/memcached-1.2.4-Win32-P... (haven't worked with memcache on Windows before)
and used your setting with two changes - no frontend and set 127.0.0.1
I also can't reproduce the the error you get, seems like getLibrary isn't returning an object.
are you sure the code above is executed properly? Directory writable etc.? Sometimes had the problem that my unzip tool didn't extract empty folders...
Used thishttp://www.splinedancer.com/memcached-win32/memcached-1.2.4-Win32-P... (haven't worked with memcache on Windows before)
and used your setting with two changes - no frontend and set 127.0.0.1
I also can't reproduce the the error you get, seems like getLibrary isn't returning an object.
public function getLibrary() { static $cache; if (!isset($cache) && defined('DIR_FILES_CACHE')) { if (is_dir(DIR_FILES_CACHE) && is_writable(DIR_FILES_CACHE)) { Loader::library('3rdparty/Zend/Cache'); $frontendOptions = array( 'lifetime' => 7200, 'automatic_serialization' => true ); $backendOptions = array( 'cache_dir' => DIR_FILES_CACHE ); if (defined('CACHE_BACKEND_OPTIONS')) { $opts = unserialize(CACHE_BACKEND_OPTIONS); foreach($opts as $k => $v) {
Viewing 15 lines of 32 lines. View entire code block.
are you sure the code above is executed properly? Directory writable etc.? Sometimes had the problem that my unzip tool didn't extract empty folders...
We had a similar problem using APC - changes made to a block from server 1 wouldn't be
reflected on server 2 as long as the block was being served from cache. To fix, we
set a cache timeout in getByID() in Block.php:
Note - you'll need to define SECONDS_TO_CACHE_BLOCKS - we use:
Hope it helps -
mb
reflected on server 2 as long as the block was being served from cache. To fix, we
set a cache timeout in getByID() in Block.php:
if ($c != null || $a != null) { $ca = new Cache(); //MB: - added timeout for caching, otherwise block changes aren't reflected across load balanced webs $ca->set('block', $bID . ':' . $cID . ':' . $cvID . ':' . $arHandle, $b, SECONDS_TO_CACHE_BLOCKS); } else { $ca = new Cache(); //MB: - here also - $ca->set('block', $bID, $b, SECONDS_TO_CACHE_BLOCKS); } return $b;
Note - you'll need to define SECONDS_TO_CACHE_BLOCKS - we use:
Hope it helps -
mb
That seems to be rather dangerous imho.
1. I don't see the point of having a cache which invalidates after 120 seconds. Rendering an object for the first time is going to take longer if you use a cache but if only use that cached object for 120 it basically means that you write the cache more often than you actually use it.
2. When you use AJAX it can still happen that a call ends up on another server where the cache hasn't been cleared. You might get mixed objects in one page view. Probably not going to cause a lot of problems but if it does you'll have a lot of time to analyse it.
3. It might still be useful when you use 5.4.1 where you can use a full cache where you're able to specify the pages with the most hits. This would help to cache the pages where you actually have several hits within 2 minutes, but caching every object, even the deepest one for only 2 minutes seem to be odd imho..
1. I don't see the point of having a cache which invalidates after 120 seconds. Rendering an object for the first time is going to take longer if you use a cache but if only use that cached object for 120 it basically means that you write the cache more often than you actually use it.
2. When you use AJAX it can still happen that a call ends up on another server where the cache hasn't been cleared. You might get mixed objects in one page view. Probably not going to cause a lot of problems but if it does you'll have a lot of time to analyse it.
3. It might still be useful when you use 5.4.1 where you can use a full cache where you're able to specify the pages with the most hits. This would help to cache the pages where you actually have several hits within 2 minutes, but caching every object, even the deepest one for only 2 minutes seem to be odd imho..
Remo is absolutely right - you'll lose the benefit of caching blocks if you set the timeout too short (although I don't think this is as dangerous as say, running with scissors).
But because we run APC locally on each server instead of a single shared memcache setup (fewer moving parts and no network performance hits) we need a way to periodically sync the caches. If you use a shared cache, then there is no reason to automatically expire cached blocks.
Your mileage may vary, but for us, hitting the database at most every 2 minutes gives us great backend performance and has eliminated stale content problems on load balanced servers.
Also, we did run into the ajax issue Remo mentioned & solved it by using sticky sessions.
mb
But because we run APC locally on each server instead of a single shared memcache setup (fewer moving parts and no network performance hits) we need a way to periodically sync the caches. If you use a shared cache, then there is no reason to automatically expire cached blocks.
Your mileage may vary, but for us, hitting the database at most every 2 minutes gives us great backend performance and has eliminated stale content problems on load balanced servers.
Also, we did run into the ajax issue Remo mentioned & solved it by using sticky sessions.
mb
I see, thanks for following up!
Remo, thank you so much for your help.
Memcache proved to be the solution we were looking for. Configuring it definitely took some trial and error though.
Long story short we were running into compatibility problems with PHP extensions.
What we ended up doing was the following:
Install memcache as a service on each of the servers that would provide memory storage for the cache.
One server was configured to manage memcache across servers using MemCached Manager.http://allegiance.chi-town.com/MemCacheDManager.aspx...
In order for the php_memcache.dll to work it needed to be a specific build. In our case it was the PHP 5.3.3 Win32 non thread safe VC9 build. Once we had the correct build it ran fine.
The last step was to configure /config/site.php for memcache. Making sure to reference all memcache servers to the the cache config.
Also it was important to ensure that each server was using the same CACHE_ID.
Once all this was setup and the cache was primed we had steller performance. on a 1,500 page site we have consistent load times of ~150ms and is able to handle a huge number of requests/second. We used a custom load testing tool that simulates user actions and includes the load time of images and additional http requests such as javascript and css.
Additionally all servers are sharing the same cache information. If a block is updated on one server all the other servers stay in sync.
We have complete control of cache expiration times and we can feel confident in making far future. Essentially Concrete5 is designed in a way that makes changes appear instantly without invalidating the entire cache. So we have the best of both worlds; great performance and fast change replication across servers.
I hope this can help others trying do something similar.
Concrete5 runs awesome behind NLB with IIS7.5 and FastCGI on the Windows Server 2008 platform. I know it states on this site that the developers don't recommend it but I would like to say that Concrete5 is perfectly suited for this environment.
We have been in a 6 month long search for a replacement CMS and have conducted many performance and user experience tests comparing many popular CMS's. Concrete5 has proven to be a winner time and time again. We are very excited use it in production and are very confident about its abilities.
Memcache proved to be the solution we were looking for. Configuring it definitely took some trial and error though.
Long story short we were running into compatibility problems with PHP extensions.
What we ended up doing was the following:
Install memcache as a service on each of the servers that would provide memory storage for the cache.
One server was configured to manage memcache across servers using MemCached Manager.http://allegiance.chi-town.com/MemCacheDManager.aspx...
In order for the php_memcache.dll to work it needed to be a specific build. In our case it was the PHP 5.3.3 Win32 non thread safe VC9 build. Once we had the correct build it ran fine.
The last step was to configure /config/site.php for memcache. Making sure to reference all memcache servers to the the cache config.
Also it was important to ensure that each server was using the same CACHE_ID.
Once all this was setup and the cache was primed we had steller performance. on a 1,500 page site we have consistent load times of ~150ms and is able to handle a huge number of requests/second. We used a custom load testing tool that simulates user actions and includes the load time of images and additional http requests such as javascript and css.
Additionally all servers are sharing the same cache information. If a block is updated on one server all the other servers stay in sync.
We have complete control of cache expiration times and we can feel confident in making far future. Essentially Concrete5 is designed in a way that makes changes appear instantly without invalidating the entire cache. So we have the best of both worlds; great performance and fast change replication across servers.
I hope this can help others trying do something similar.
Concrete5 runs awesome behind NLB with IIS7.5 and FastCGI on the Windows Server 2008 platform. I know it states on this site that the developers don't recommend it but I would like to say that Concrete5 is perfectly suited for this environment.
We have been in a 6 month long search for a replacement CMS and have conducted many performance and user experience tests comparing many popular CMS's. Concrete5 has proven to be a winner time and time again. We are very excited use it in production and are very confident about its abilities.
Glad I could help, at the end it was just a single word "memcache" I mentioned, but it's great to hear that it's working for you too!
Running Concrete5 on Windows isn't a problem and wasn't a problem, it's just because most dev's don't work on Windows. I have to admit, I mostly work on Windows and I'm quite happy with it ;-)
Running Concrete5 on Windows isn't a problem and wasn't a problem, it's just because most dev's don't work on Windows. I have to admit, I mostly work on Windows and I'm quite happy with it ;-)
If you have a lot of changes on your page you probably end up having the same result as if there was no cache.. Okay, not exactly but you'd still slow down the site.
I can't see why there's a problem - if DFS replicates immediately, the cache is the same on all servers, isn't it?
Why not use a shared file system like CXFS, OCFS2, VMFS? If you'd have the files directory on a single storage system, you wouldn't have such problems..?