Host complaining about number of files and directories
Permalink 1 user found helpful
My host is complaining about the number of files and number of directories and warning that my site may be disabled if it affects the performance of the shared server (Linux 2.6.32-46.1.BHsmp).
I know that this has been discussed before, but do you have any advice on what I should tell the host people or what I can do to protect my site?
I have 3244 directories and 1614 files accommodating a total of 940 files and an additional 8424 directories and 2856 files for thumbnails.
The reason for this is that for each file, the File Manager sets up three directories and for some files, two additional files. For example:
files/9913/
files/9913/0721
files/9913/0721/3068
files/9913/0721/3068/MyPic.jpg
or
files/9913/
files/9913/0721
files/9913/0721/index.html
files/9913/0721/3068
files/9913/0721/3068/index.html
files/9913/0721/3068/Mydoc.pdf
I'm not sure when the index.html files (0 bytes) are created; I've seen it for both image and non-image files. (Maybe these files are created for recent versions of C5.)
In addition, for each image file, 9 directories are set up to handle three thumbnail files:
files/thumbnails/9913/0721/3068/
files/thumbnails/level2/9913/0721/3068/
files/thumbnails/level3/9913/0721/3068/
For some images, two index.html files are also created for each of the three thumbnails. And the directories and index.html files are created even if all three thumbnails are not needed (evidently on a small image).
And when I delete a file, the directories and index.html files remain behind.
This means that for each file I add in File Manager, C5 adds a minimum of 3 directories and one file and possibly 12 directories and 12 files to the server. For my specific case, I am averaging 12.4 directories and 4.8 files for each file entry in File Manager. Many of the directories are empty (from deleted files?).
This seems like a lot of overhead and I wonder why Concrete5 does things this way, and whether it will be changed in the future.
I know that this has been discussed before, but do you have any advice on what I should tell the host people or what I can do to protect my site?
I have 3244 directories and 1614 files accommodating a total of 940 files and an additional 8424 directories and 2856 files for thumbnails.
The reason for this is that for each file, the File Manager sets up three directories and for some files, two additional files. For example:
files/9913/
files/9913/0721
files/9913/0721/3068
files/9913/0721/3068/MyPic.jpg
or
files/9913/
files/9913/0721
files/9913/0721/index.html
files/9913/0721/3068
files/9913/0721/3068/index.html
files/9913/0721/3068/Mydoc.pdf
I'm not sure when the index.html files (0 bytes) are created; I've seen it for both image and non-image files. (Maybe these files are created for recent versions of C5.)
In addition, for each image file, 9 directories are set up to handle three thumbnail files:
files/thumbnails/9913/0721/3068/
files/thumbnails/level2/9913/0721/3068/
files/thumbnails/level3/9913/0721/3068/
For some images, two index.html files are also created for each of the three thumbnails. And the directories and index.html files are created even if all three thumbnails are not needed (evidently on a small image).
And when I delete a file, the directories and index.html files remain behind.
This means that for each file I add in File Manager, C5 adds a minimum of 3 directories and one file and possibly 12 directories and 12 files to the server. For my specific case, I am averaging 12.4 directories and 4.8 files for each file entry in File Manager. Many of the directories are empty (from deleted files?).
This seems like a lot of overhead and I wonder why Concrete5 does things this way, and whether it will be changed in the future.
2.3 years
225 pages; 1636 page versions
The 3244 directories is just the files in File Manager:
813 images
118 documents
10 files
for a total of 941 files.
The thumbnails add 8424 directories for a total of 11668 for the 941 files.
Some files do have multiple versions; I'm guessing about 50 or so.
The total website has 18536 directories for 37810 files.
The host's TOS states: "Accounts with a large number of files (inode count in excess of 200,000) can have an adverse affect on server performance." I guess at my present 18,536 + 37,810 = 56,346 inodes I am OK. I will call customer service and see if they agree.
But I still think that the C5 overhead, particularly the failure to clean-up after file deletion, cound use some improvement.
225 pages; 1636 page versions
The 3244 directories is just the files in File Manager:
813 images
118 documents
10 files
for a total of 941 files.
The thumbnails add 8424 directories for a total of 11668 for the 941 files.
Some files do have multiple versions; I'm guessing about 50 or so.
The total website has 18536 directories for 37810 files.
The host's TOS states: "Accounts with a large number of files (inode count in excess of 200,000) can have an adverse affect on server performance." I guess at my present 18,536 + 37,810 = 56,346 inodes I am OK. I will call customer service and see if they agree.
But I still think that the C5 overhead, particularly the failure to clean-up after file deletion, cound use some improvement.
you are right feller, i do some experimen with 5.4.2.2
when i delete the image, the image deleted, but the folder still there.
there is 4 folder remain with index.html inside
maybe you can make php script to delete zero size files, it will clean your folder.
but ask another budy, maybe i'm wrong :)
when i delete the image, the image deleted, but the folder still there.
there is 4 folder remain with index.html inside
maybe you can make php script to delete zero size files, it will clean your folder.
but ask another budy, maybe i'm wrong :)
I think this is a good idea. Has anybody tried it yet?
The following from my hosting service (BlueHost):
"We suggest that you keep your file count below 50,000 files. We have found that when you go over 50,000 files then it can cause performance issues. We will not deactivate your account unless you go over 200,000 files."
"We suggest that you keep your file count below 50,000 files. We have found that when you go over 50,000 files then it can cause performance issues. We will not deactivate your account unless you go over 200,000 files."
I had more than 51,000 files, too, before I cleaned them out tonight. My host is BlueHost also.
I've been with concrete5 for a couple of years and have updated to new versions at least 4 or 5 times. The files from the old versions (both zipped and unzipped) were in my "public_html/updates" folder. I deleted them (I kept the latest version) and some old mail. Then I called Bluehost tech support to ask them to update the file count (they give a phone number on the file count page and it only took a couple of minutes.) It reduced down to 29,700 files.
Then I noticed there were still thousands of files in the "public_html/files/tmp" folder. I read a post here
http://www.concrete5.org/community/forums/chat/about-some-files-in-...
that says it's safe to delete that entire folder, so I did. It also said I could delete the contents of the "public_html/files/trash" folder, even though not recommended, so I deleted half those files which I will never use again. I called Bluehost again, this time it took 10-15 minutes and they verbally told me that it reduced down to 18,000 files (it hasn't shown up in my CPanel yet).
Woot! That's a huge reduction in useless files and wasted space.
Hope that helps.
I've been with concrete5 for a couple of years and have updated to new versions at least 4 or 5 times. The files from the old versions (both zipped and unzipped) were in my "public_html/updates" folder. I deleted them (I kept the latest version) and some old mail. Then I called Bluehost tech support to ask them to update the file count (they give a phone number on the file count page and it only took a couple of minutes.) It reduced down to 29,700 files.
Then I noticed there were still thousands of files in the "public_html/files/tmp" folder. I read a post here
http://www.concrete5.org/community/forums/chat/about-some-files-in-...
that says it's safe to delete that entire folder, so I did. It also said I could delete the contents of the "public_html/files/trash" folder, even though not recommended, so I deleted half those files which I will never use again. I called Bluehost again, this time it took 10-15 minutes and they verbally told me that it reduced down to 18,000 files (it hasn't shown up in my CPanel yet).
Woot! That's a huge reduction in useless files and wasted space.
Hope that helps.
This sounds like a great addition to the system maintenance tab - to empty the directories where it is safe to do so. Only wish I knew where to start on something like that!
I've known about the BH files/directory limits for a few months now, and have had to spread out my C5 installs across several accounts due to the fact that an C5 install seem to average about 10k files per install for an average sized website (for my business - small, local websites), mostly due to the fact that each file uploaded to the site generates multiple files and directories for some odd reason.
So you can imagine my dismay to log in to one of the BH accounts yesterday and see that *over 285k* files/directories were in use on the account. !!! I had received no suspension warnings or otherwise from BH, but have been trying to keep an eye on file usage anyway, so this was a shocker, and I am glad that they had not shut me down...
After looking around the file structure on the server, I narrowed it down to one particularly crazy c5 install. It is a photographers website, and she has about 500 pics there. However, for those 500 pics, there were *in excess of 33k files/folders in use*. The /files/thumbnails directory (and subs) alone accounted for nearly 22k of that number. Doing some research, and ***after downloading a backup*** just in case, today I deleted the entire /files/thumbnails directory, all in one fell swoop. So far, I see no difference in how the site is working or how the images are being displayed, neither when logged in, or when viewed simply as a website visitor.
I also deleted everything from the /files/tmp and /files/trash directories, and ran the "Remove Old Versions" option in the Dashboard over and again until it displayed 0 Old Versions Removed consistently.
So, with caution highly suggested, and of course YMMV, there is a tip for other BH'ers to perhaps use in order to slim down resources used on a shared hosting account.
So you can imagine my dismay to log in to one of the BH accounts yesterday and see that *over 285k* files/directories were in use on the account. !!! I had received no suspension warnings or otherwise from BH, but have been trying to keep an eye on file usage anyway, so this was a shocker, and I am glad that they had not shut me down...
After looking around the file structure on the server, I narrowed it down to one particularly crazy c5 install. It is a photographers website, and she has about 500 pics there. However, for those 500 pics, there were *in excess of 33k files/folders in use*. The /files/thumbnails directory (and subs) alone accounted for nearly 22k of that number. Doing some research, and ***after downloading a backup*** just in case, today I deleted the entire /files/thumbnails directory, all in one fell swoop. So far, I see no difference in how the site is working or how the images are being displayed, neither when logged in, or when viewed simply as a website visitor.
I also deleted everything from the /files/tmp and /files/trash directories, and ran the "Remove Old Versions" option in the Dashboard over and again until it displayed 0 Old Versions Removed consistently.
So, with caution highly suggested, and of course YMMV, there is a tip for other BH'ers to perhaps use in order to slim down resources used on a shared hosting account.
I've written a package that may helps removing unuseful files.
Unzip it under the /packages folder and install with Dashboard > Extend concrete5.
You'll find a new page as Dashboard > System & Settings > Optimization > Clear Files
Unzip it under the /packages folder and install with Dashboard > Extend concrete5.
You'll find a new page as Dashboard > System & Settings > Optimization > Clear Files
@mlocati
Nice.
What is the licensing on this?
Nice.
What is the licensing on this?
MIT license: you can do whatever you want with it, except claiming it's your software and complaining with me for problems;)
@mlocati
Sweet.
I especially like the deleting of files older than a certain age. It's something Miser needs for it's cache files but cannot implement because of performance.
Sweet.
I especially like the deleting of files older than a certain age. It's something Miser needs for it's cache files but cannot implement because of performance.
Thank you for sharing that!
Some time ago I submitted that package to the PRB.
Once it gets approved, you'll be able to find it in the standard marketplace of concrete5 (for free ;) )
Once it gets approved, you'll be able to find it in the standard marketplace of concrete5 (for free ;) )
@mlocati
Exactly what I was looking for.
Hopefully the package submission is approved soon.
Exactly what I was looking for.
Hopefully the package submission is approved soon.
Hey, anybody experienced a large number of session files being kept in the files/tmp folder.
I don't know after which update it started, but I saw it first on a relatively small site with not that much traffic on it. I'm now cleaning the directory "by hand" - haven't done it in three months I think - and it contains nearly 7800 files. Some are file size 0 some 26 byte.
I don't think that they were cleared by any in system job.
THX,
SAP
I don't know after which update it started, but I saw it first on a relatively small site with not that much traffic on it. I'm now cleaning the directory "by hand" - haven't done it in three months I think - and it contains nearly 7800 files. Some are file size 0 some 26 byte.
I don't think that they were cleared by any in system job.
THX,
SAP
I don't know the reason of this, but the solution I posted athttp://www.concrete5.org/community/forums/usage/host-complaining-ab... may help you
Here it is four years later and I'm still running into this problem... did the add on ever get updated? I just tried installing it and it won't work with 5.7. I also don't see it in the marketplace. Did that ever happen, and if so, what is it called? Any help would be great. thanks!
I didn't create a version for 5.7+...
One thing you could manually do is to check that updates folder only contains the latest version - I think that all the previous versions can be safely removed
One thing you could manually do is to check that updates folder only contains the latest version - I think that all the previous versions can be safely removed
how long has this site live?
like mike said, it's ridicolius, you pay space for host and yet they complain