Exporting forms as CSV needs too much memory

Permalink 2 2 Browser Info Environment
When exporting submitted form results as CSV, the larger the number of entries to export, the greater you have to set PHP memory limit.

We have a site collecting support requests related to COVID-19 for a region in the UK. The site was constructed normally with a standard 128Mb PHP memory limit. After a while the report export for the support form started to fail silently. I turned out that the PHP memory limit needed increasing so I doubled it to 256Mb. Two days later it started failing again, once more the memory limit was being hit so I upped it to 1Gb.

This would appear to be due to the entore CSV result set being collected in memory prior to being output as the CSV. Obviously this is a very bad thing.

It's not even a huge result set, I'm only talking about 3000 rows of 23 fields yet it needs over 256Mb of memory and growing. At some point it will kill the site to try outputting the CSV data. The memory requirement also seems to be growing more quickly than in a linear fashion as there were around 2200 records when 128Mb wasn't enough and only 3200 when 256Mb wasn't enough which seems roughly exponential.

These results should be streamed properly row-by-row, not hogged in memory causing failures.

It's also worth noting that as the memory limit failure is a silent failure then people will unwittingly be getting incomplete CSVs unless they notice that there are more recent entries in the form results page than in the CSV itself, so they'll have bad data.

Related forum thread:https://www.concrete5.org/community/forums/customizing_c5/exporting-...


Status: New
surefyre

concrete5 Environment Information

# concrete5 Version
Core Version - 8.5.0
Version Installed - 8.5.0
Database Version - 20190301133300

# concrete5 Packages
Everyone Health PDHME Theme (1.0.6), Priority Digital Health (0.9.10), Provide PDHME Theme (1.0.6)

# concrete5 Overrides
src/Concrete/Captcha/SecurimageController.php, src/Concrete/Captcha, src/Concrete

# concrete5 Cache Settings
Block Cache - Off
Overrides Cache - Off
Full Page Caching - Off
Full Page Cache Lifetime - Every 6 hours (default setting).

# Server Software
Apache/2.4.29 (Ubuntu)

# Server API
apache2handler

# PHP Version
7.2.24-0ubuntu0.18.04.4

# PHP Extensions
apache2handler, calendar, Core, ctype, date, exif, fileinfo, filter, ftp, gettext, hash, iconv, json, libxml, mbstring, mysqli, mysqlnd, openssl, pcre, PDO, pdo_mysql, Phar, posix, readline, Reflection, session, shmop, sockets, sodium, SPL, standard, sysvmsg, sysvsem, sysvshm, tokenizer, Zend OPcache, zlib

# PHP Settings
max_execution_time - 300
log_errors_max_len - 1024
max_file_uploads - 20
max_input_nesting_level - 64
max_input_time - 60
max_input_vars - 1000
memory_limit - 1024M
post_max_size - 8M
upload_max_filesize - 2M
mysqli.max_links - Unlimited
mysqli.max_persistent - Unlimited
pcre.backtrack_limit - 1000000
pcre.recursion_limit - 100000
session.cache_limiter - <i>no value</i>
session.gc_maxlifetime - 7200
opcache.max_accelerated_files - 10000
opcache.max_file_size - 0
opcache.max_wasted_percentage - 5

Browser User-Agent String

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36