Backup Voodoo (c5.6)

Documentation

Support

Support extends to installation and setup of this addon and correcting bugs within this addon. Support does not include free consultancy or assistance in restoring from backups or recovering failed sites. For help with such, please see the concrete5 forums.

DANGER

  • A faulty or incomplete backup can break your site, but you won't know about it until you try to recover from it. If you are not 100% sure that a table is non-critical, then don't mess with it!
  • Full site backups can be very big. Leaving backups on the server uses space, can comprise a security risk and is not a safe place to leave a backup if your server suffers a disc failue. It is strongly recommended that you download backups soon after they are made and then delete them from the server.

PROBLEMS MAKING BACKUPS

Backup Voodoo reoprts to the dashboard and logs any problems encountered making backups. At the end of each backup run, Backup Voddo will make suggestions for improving backup performance. Please see the Problem Solving section for advice on solving problems.

Dashboard pages

The Backup Voodoo dashboard pages can be found under Dashboard > System & Settings > Backup & Restore

Configure Backup Voodoo

Configure how Backup Voodoo makes backups of the database and file system.

Do Backup Voodoo

Run Backup Voodoo to create database and file system backups.

Review Backup Voodoo backups

Review and examine Backup Voodoo backups. Download, delete and database recovery.

Setup Backup Voodoo Jobs

Setup jobs for Backup Voodoo, matching jobs to the configuration.

General Principles

When the standard concrete5 Backup runs out of server time, it is often the result of a few large tables. In some cases such large tables are not critical to the site. For example, while the Page Statistics table is used for some statistics presentation and by some analysis addons, a functional backup can be made without that table, or with a version of that table that is slightly out of synchronisation with the rest of the site.

Other tables, such as eCommerce order records, can become bloated with null entries and can be reduced to just the real orders. The same usually applies to growing log tables, both from the core and from various addons. With a log, maybe only the last few entries are of importance to a backup.

Often overlooked, in addition to a copy of the database you also need all relevant files to recover a damaged site.

Backup Voodoo squeezes a site backup of database and files into the server resources available by spreading backups of the database and files across a number of passes and reducing non-critical tables. As a site administrator you configure how this is achieved from the dashboard. None of this is preferable to making an exact backup all in one go, but for sites on hosting plans that do not provide shell access, or for site administrators who do not have shell expertise, or for sites that are just too big, Backup Voodoo provides a viable means of making a backup.

Getting Started

Configure Backup Voodoo

Backup Voodoo provides Easy Start options that will preset the configuration for typical backup requirements. For most sites, simply select the Easy Start option that is closest to your requirements. If your site has any addons installed that keep a lot of data or particularly generous or restrictive server resources, you may then want to select the Custom Configuration option to adjust the Backup Voodoo settings to suit your specific requirements.

Every time you make a backup, Backup Voodoo will analyse the outcome and make suggestions for configuration settings to improve the next backup run.

Do Backup Voodoo

With the configuration settings saved, now go to the Do Backup Voodoo page and execute a backup. You will see each pass of the database and files progress down the dashboard.

Sites change. Databases and files grow. Every time you make a backup, Backup Voodoo will analyse the outcome and make suggestions for configuration settings to improve the next backup run. Many of these adjustments will be automatic. All you need to do is review the suggestions, click the one that you think most appropriate, and your next backup run will be that little bit smoother.

Review Backup Voodoo backups

The backup you have just made will be listed in the Review Backup Voodoo backups dashboard page. Check the details and download it for safety.

Learn how to recover now, before it is too late!

Having installed Backup Voodoo and found how easy it can be to make a full backup, don't wait for your site to fail before learning about the more difficult art of recovery. Download a Backup Voodoo backup and use it to make a clone on an offline development system. Install a WAMP, LAMP or MAMP bundle, then make a clone of your site as described below. That will give you confidence that the backup settings you have chosen are what you need to recover your site when you really have to.

The following information about restoring and cloning from a backup is a guide to general tools that are provided by most web hosting services and are not part of the Backup Voodoo addon.

Restoring and cloning a site from backups

The guidance here is not exact

So much of restoring a site depends on exactly what was in the site and what was actually backed up that the guidance below cannot be precise. Overall, you need a database, you need all the files, and you need to populate the database, in that order. Filling in the details where they diverge from the below is completely up to you!

Restoring a complete site

Restoring a complete site cannot be achieved from within the site. You will have to use FTP and phpMyAdmin or similar. As a prerequisite, you will need to have downloaded a set of Backup Voodoo database and files backup files.

  1. Review the backup files and their content. Once you start, there is no going back!
  2. Use phpMyAdmin to confirm that the database still exists.
  3. Load the files backup to the server and unzip into the relevant folders.
  4. Load to the server and unzip any additional files such as concrete5 core files and addon packages for the same version that was backed up. Do not visit the site.
  5. Use phpMyAdmin to import the database backup files in the sequence they were created.
  6. Visit the site. Go to the home page and then the dashboard to check the restoration is complete.
  7. Clear all caches. Exit maintenance mode.

Cloning a complete site

Cloning a complete site cannot be achieved from within the cloned site. You will have to use FTP and phpMyAdmin or similar. As a prerequisite, you will need to have downloaded a set of Backup Voodoo database and files backup files.

  1. Review the backup files and their content. Once you start, there is no going back!
  2. Use phpMyAdmin to create a new database, noting the database name, password and server.
  3. Use phpMyAdmin to confirm that the database still exists.
  4. Load the files backup to the server and unzip into the relevant folders.
  5. Load to the server and unzip any additional files such as concrete5 core files and addon packages for the same version that was backed up. Do not visit the site.
  6. Edit config/site.php to have the details for the new database, but keep the password salt of the cloned site. You may also need to remove DIRNAME_APP_UPDATED.
  7. You may also need to edit any site specific parts of .htaccess
  8. You may also need to adjust permissions on the 'files/', 'packages/', 'updates/' and 'config/' folders
  9. Use phpMyAdmin to import the database backup files in the sequence they were created.
  10. Visit the site. Go to the home page and then the dashboard to check the restoration is complete.
  11. Clear all caches. Exit maintenance mode.

If cloning a site from Windows to Linux hosting, you will usually also need to fix the case of database table names using an addon such as Database Case Sensitivity Migration

Reverting a site database

Reverting a site database may be possible from within the site, depending on whether the restoration can be achieved within the server limits. However, be prepared that a restore from within concrete5 may exceed server limits and fail, after all, you are probably using Backup Voodoo because you have encountered such failures in the past. In this case, you will need to import the database backup using phpMyAdmin.

From within concrete5

  1. Download the backup files, just in case the restore from within concrete5 does not work.
  2. Review the backup files and their content. Once you start, there is no going back.
  3. Use the Backup Voodoo Restore Database option for the respective backup.
  4. Clear all caches. Exit maintenance mode.
  5. Logout and go to the home page and then the dashboard to check the restoration is complete.

From phpMyAdmin

  1. Review the backup files and their content. Once you start, there is no going back.
  2. Use phpMyAdmin to import the database backup files in the sequence they were created.
  3. Clear all caches. Exit maintenance mode.
  4. Logout and go to the home page and then the dashboard to check the restoration is complete.

Recovering selected files

It is not recommended that you attempt to restore all files using FTP unless you also restore the corresponding database. concrete5 uses the database to associate files with storage locations, so unless the database and files storage locations are consistent, such a restoration of files will likely fail.

However, if you have inadvertently removed one or more files using the file manager, you may be able to replace then from a backup. As a prerequisite, you will need to have downloaded a set of Backup Voodoo files backup files.

  1. Unzip the files backup on your development system.
  2. On your site, use the concrete5 File Manager to find and upload selected files.
  3. Add the files to relevant sets.
  4. Visit and edit the relevant pages and blocks to add the uploaded files.

Further issues and assistance

Most Backup Voodoo installations will only need to consider the information above.

Everything below is advanced configuration.

Advanced Configuration with Configure Backup Voodoo

Files

Files are backed up to a zip file through a number of passes. Configure what directoreies are backed up and the number of passes that a files backup is split into on the Settings tab.

Database Filter Categories

To configure Backup Voodoo, select a few big core and addon database tables, then decide how you would like to filter them from the main backup according to:

Reduce

Backup a table in the first backup pass, but only backup the last (or first) X entries.

Skip

Do not backup a table at all.

Postpone

Leave a table out of the first backup pass, but make a backup in a subsequent pass.

Postpone and Reduce

Leave a table out of the first backup pass, but make a backup in a subsequent pass and only backup the last (or first) X entries.

No Action

You dont need to use this category, but may find it a useful parking place for tables you are thinking of doing something with later or just want to make a note of.

Postponed Tables

The first pass of Backup Voodoo will make the big backup, similar to the standard concrete5 backup, but with some tables reduced or postponed. The next pass of Backup Voodoo will process the postponed tables, which may themselves be reduced. Postponed tables may be marked to "Begin next postponed backup pass", which means they will be postponed to a further pass of Backup Voodoo. You can select the sequence in which postponed tables are processed by dragging them up and down the list. In general, place the most critical tables near the top of the list.

Reduced Tables

With some tables, you want a backup, but only care about the last few entries. For example, maybe you only ever need to keep enough Page Statistics for the last month, or the last 10 entries in the Jobs Log. Rather than backup the entire table, you can specify a maximum number of entries to be actually backed up.

Preventing Faulty Backups

During the time it takes to make a number of passes to complete a backup, there is a risk that a site has changed sufficiently that tables backed up in the second or subsequent passes are no longer consistent with the first pass. To eliminate such risk, avoid making changes to a site between Backup Voodoo passes. If visitors can make changes or place eCommerce orders, you may want to place the site in maintenance mode while making a backup. By deafult, Backup Voodoo will do this for you.

The really bad risks are that you leave out a critical table, reduce it in a way that looses important data, or postpone it and it becomes inconsistent with other tables. If unsure about whether a table is critical to site or addon functionality, please contact the respective apon developer for advice.

Backup Voodoo Jobs

Backup Voodoo can also install jobs so you can automate backups.

Backup Voodoo does not schedule the jobs

Backup Voodoo will install (and remove) jobs appropriate to the settings you select on the Setup Backup Voodoo Jobs dashboard page and the overall Backup Voodoo configuration. However, Backup Voodoo does not include job scheduling. Site administrators then need to set up scheduling for these Backup Voodoo jobs using cron. The indicated jobs must be run the number of times shown in the sequence shown. Failure to do so will result in a faulty backup.

Activities performed by Backup Voodoo jobs

Backup Voodoo completes a backup as a sequence of the following activities:

Begin a backup.

Starts a new backup, setting the file names to be used and possibly placing the site in maintenance mode.

Database backup.

Makes one pass of the database backup. Multiple passes may be necessary, depending on how Backup Voodoo is configured.

Files backup.

Makes one pass of the files backup. Multiple passes may be necessary, depending on how Backup Voodoo is configured.

Report.

Ends a backup, checking everything has completed, clearing up and removing maintenance mode.

These activities can be jobs in their own right, or automatically combined into compound jobs according to the selected settings. Individual jobs give a site administrator greatest control. Compound jobs may be more convenient in some circumstances. In all cases the above sequence must be maintained.

 

Multiple pass jobs

Some Backup Voodoo jobs need to be run a multiple number of times. There are two basic strategies for doing so:

Fixed repetition

When you configure Backup Voodoo, the Setup jobs tab of the Setup Backup Voodoo Jobs dashboard page will tell you which jobs need to be run and how many times they each need to be run.

Variable repetition

Any multi-pass Backup Voodoo job will return a message containing text like '[-N]' after each run, indicating that further runs are required. The job needs to be repeated until the returned message no longer contains such text.

In general, if you run a multi-pass Backup Voodoo job too many times, excess passes will do nothing.

Queueable Job

For sites using concrete5.6.2+, Backup Voodoo provides a queueable job that makes it much easier to schedule backups using cron. The Backup Voodoo queueable job combines all necessary jobs into a concrete5 queueable job where the slices of the job are managed by the core concrete5 queueable job system.

Automating the queueable job requires:

  1. A cron job to be run as a clock tick, continuously at very regular intervals. This triggers concrete5 to check for and run queued steps of all queueable jobs as and when necessary.
  2. A second cron job to trigger the start of the Backup Voodoo Queueable Job. This will subsequently be processed in steps according to the clock tick. 

For details, see the dashboard Automated Jobs page and click the icon for 'Automation Instructions'.

Maintenance Mode

Placing a site in maintenance mode while a Backup Voodoo multi-pass backup is running will prevent visitors from doing anything that could create inconsistencies in the backup. However, you have to be confident that the site will be taken out of maintenance mode when the Backup Voodoo processing has completed. If you use maintenance mode with automated jobs, you have to consider that the site could be left in maintenance mode if a job fails.

If maintenance mode is used in conjunction with the piggy-backed scheduling of Flexjob Scheduler, you could actually prevent the traffic that such a scheduling mechanism requires to complete the jobs! In such cases, it is best to restrict the database backup to a single pass, or limit postponed tables to logs and statistics that don't matter. A files backup is less critical, as generally files are added rather than removed and change less frequently.

Maximum execution time

All jobs must be scheduled and completed within the Maximum Execution Time set in the Settings tab, by default set as 10 minutes. Failure to achieve this will result in a backup being judged faulty and, for the Combined repeated job, once the time has elapsed since the backup was begun, a new backup will be started.

Problem Solving

The Easy Start options provide preset configurations for typical backup requirements that are a good starting point for most sites. However, if your site has any addons installed that keep a lot of data or particularly generous or restrictive server resources, you may then want to select the Custom Configuration option to adjust the Backup Voodoo settings to suit your specific requirements.

Every time you make a backup, Backup Voodoo will optionally analyse the outcome and make suggestions for configuration settings to improve the next backup run.

A backup database pass runs out of server time

If a database pass runs out of time, you will need to look for further tables that can be reduced or postponed. 

A backup files pass runs out of server time

If a files pass runs out of time, you can simply adjust the slider on the settings tab to break it into more passes that are each smaller. You may also want to make sure that 'interleave' is checked, as this will distribute the files backup more evenly between passes.

Dashboard timeout exceeded

"AJAX Error when communicating with server"

Either of the above can cause this error message to appear. It can also be caused by a sever that has not run out of resources, but is simply taking so long that the browser resident part of Backup Voodoo has grown tired of waiting. In this case, increasing the AJAX timeout on the settings tab may be all you need to do to solve the proble,

While making a backup, each pass is scheduled via a series of ajax transactions between your web browser and the site server. If your connection is slow, there is a possibility that this could timeout before the server completes a backup pass. The timeout can be adjusted using the slider on the Settings tab.

Maximum execution time exceeded

While making a backup, the cumulative time from start to finish is used as a heuristic to judge if something has gone wrong. If your connection is slow or site is very big, there is a possibility that this could timeout before all backup passes are copmplete. The maximum execution time can be adjusted using the slider on the Settings tab.

A backup becomes too big to download

Both files and database backups are by default concatenated into a single file each. If these files are too big to download, you can adjust the limit at which Backup voodoo will start another zip file, or uncheck the concatenation options on the Settings tab and each pass will be saved into a separate file. 

Configuration constant definitions

Much of the deep inside configuration of Backup Voodoo is controlled by constants that can be overridden by setting definitions in config/site.php. Internally, these constants are set up and described in models/jl_backup_voodoo_general_config.php. Experts and developers should refer to the comments and descriptions within that file as a prelude to redefining such constants in config/site.php for details of all such constants.

BACKUP_VOODOO_ADD_<uppercase of package_handle>

Where a package has one or more tables that is not specified in the package db.xml and that you need Bachup Voodoo to make available for assigning actions, a constant of this form can be used to specify the table names as a comma separated list. For example:

 define('BACKUP_VOODOO_ADD_JFRITSCH_SEARCHRANK', 'SearchRankKeyword,SearchRankRank');

BACKUP_VOODOO_EXCLUDE_<uppercase of package_handle>

Where a package has one or more tables that can be found in the package db.xml and that you need Bachup Voodoo to specifically exclude, so actions cannot be assigned, a constant of this form can be used to specify the table names as a comma separated list. For example:

 define('BACKUP_VOODOO_EXCLUDE_CORE_COMMERCE', 'CoreCommerceProductAttributeValues,CoreCom...

BACKUP_VOODOO_ELIGIBLE_CORE_TABLES

Backup Voodoo only makes a limited list of concrete5 core tables available for actions, as specified by the constant BACKUP_VOODOO_ELIGIBLE_CORE_TABLES.

 define('BACKUP_VOODOO_ELIGIBLE_CORE_TABLES', 'Logs,JobsLog,PageStatistics');

BACKUP_VOODOO_EXCLUDE_FILEPATHS_

Backup Voodoo will exclude files from the backup if they contain or end in strings set by these constants.

define('BACKUP_VOODOO_EXCLUDE_FILEPATHS_CONTAINING', implode('|', array('.php_sucuribackup.'))); 
define('BACKUP_VOODOO_EXCLUDE_FILEPATHS_ENDING', .......); // currently no default value

Support

Support extends to installation and setup of this addon and correcting bugs within this addon. Support does not include free consultancy or assistance in restoring from backups or recovering failed sites. For help with such, please see the concrete5 forums.