When exporting submitted form results as CSV, the larger the number of entries to export, the greater you have to set PHP memory limit.

We have a site collecting support requests related to COVID-19 for a region in the UK. The site was constructed normally with a standard 128Mb PHP memory limit. After a while the report export for the support form started to fail silently. I turned out that the PHP memory limit needed increasing so I doubled it to 256Mb. Two days later it started failing again, once more the memory limit was being hit so I upped it to 1Gb.

This would appear to be due to the entore CSV result set being collected in memory prior to being output as the CSV. Obviously this is a very bad thing.

It's not even a huge result set, I'm only talking about 3000 rows of 23 fields yet it needs over 256Mb of memory and growing. At some point it will kill the site to try outputting the CSV data. The memory requirement also seems to be growing more quickly than in a linear fashion as there were around 2200 records when 128Mb wasn't enough and only 3200 when 256Mb wasn't enough which seems roughly exponential.

These results should be streamed properly row-by-row, not hogged in memory causing failures.

It's also worth noting that as the memory limit failure is a silent failure then people will unwittingly be getting incomplete CSVs unless they notice that there are more recent entries in the form results page than in the CSV itself, so they'll have bad data.

Related forum thread:https://www.concrete5.org/community/forums/customizing_c5/exporting-...