100 000 images in a single directory
Clash Royale CLAN TAG#URR8PPP
up vote
2
down vote
favorite
I am taking over the hosting of a poorly written application (I am not paid for this) and I am wondering which filesystem would serve best a hundred thousand images totaling some 900GB in a single directory. As this server is chiefly my development server, there is plenty of spare CPU power (it's an E3-1275V2 idle most of the time) and RAM (about 20-27 GB free), these are not considerations.
The files never change. Maybe they are deleted rarely but most definitely they are not changed.
filesystems
asked Nov 23 at 15:06
chx
531415
add a comment |
up vote
2
down vote
favorite
I am taking over the hosting of a poorly written application (I am not paid for this) and I am wondering which filesystem would serve best a hundred thousand images totaling some 900GB in a single directory. As this server is chiefly my development server, there is plenty of spare CPU power (it's an E3-1275V2 idle most of the time) and RAM (about 20-27 GB free), these are not considerations.
The files never change. Maybe they are deleted rarely but most definitely they are not changed.
filesystems
asked Nov 23 at 15:06
chx
531415
2
This is just my opinion, but I'd go withxfsas it was originally designed for this workload. Second choice iszfsbut I've never done any production work with this filesystem.
– Doug O'Neal
Nov 23 at 15:17
1
I don't think the fs will matter much in this case. What matters is how the application handles the directory.
– Henrik
Nov 23 at 15:31
1
I would divide up the image files into one or two layers of subdirectories, make the application access them at the new locations and then use whatever filesystem was default one.
– Kusalananda
Nov 23 at 15:39
1
And note that applications that try something to do with the file names (likels, which will attempt to sort them) still present problems, even if the file system handles it well.
– dirkt
Nov 23 at 16:13
add a comment |
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I am taking over the hosting of a poorly written application (I am not paid for this) and I am wondering which filesystem would serve best a hundred thousand images totaling some 900GB in a single directory. As this server is chiefly my development server, there is plenty of spare CPU power (it's an E3-1275V2 idle most of the time) and RAM (about 20-27 GB free), these are not considerations.
The files never change. Maybe they are deleted rarely but most definitely they are not changed.
filesystems
asked Nov 23 at 15:06
chx
531415
I am taking over the hosting of a poorly written application (I am not paid for this) and I am wondering which filesystem would serve best a hundred thousand images totaling some 900GB in a single directory. As this server is chiefly my development server, there is plenty of spare CPU power (it's an E3-1275V2 idle most of the time) and RAM (about 20-27 GB free), these are not considerations.
The files never change. Maybe they are deleted rarely but most definitely they are not changed.
filesystems
filesystems
asked Nov 23 at 15:06
chx
531415
asked Nov 23 at 15:06
chx
531415
asked Nov 23 at 15:06
chx
531415
asked Nov 23 at 15:06
chx
531415
asked Nov 23 at 15:06
chx
531415
531415
2
This is just my opinion, but I'd go withxfsas it was originally designed for this workload. Second choice iszfsbut I've never done any production work with this filesystem.
– Doug O'Neal
Nov 23 at 15:17
1
I don't think the fs will matter much in this case. What matters is how the application handles the directory.
– Henrik
Nov 23 at 15:31
1
I would divide up the image files into one or two layers of subdirectories, make the application access them at the new locations and then use whatever filesystem was default one.
– Kusalananda
Nov 23 at 15:39
1
And note that applications that try something to do with the file names (likels, which will attempt to sort them) still present problems, even if the file system handles it well.
– dirkt
Nov 23 at 16:13
add a comment |
2
This is just my opinion, but I'd go withxfsas it was originally designed for this workload. Second choice iszfsbut I've never done any production work with this filesystem.
– Doug O'Neal
Nov 23 at 15:17
1
I don't think the fs will matter much in this case. What matters is how the application handles the directory.
– Henrik
Nov 23 at 15:31
1
I would divide up the image files into one or two layers of subdirectories, make the application access them at the new locations and then use whatever filesystem was default one.
– Kusalananda
Nov 23 at 15:39
1
And note that applications that try something to do with the file names (likels, which will attempt to sort them) still present problems, even if the file system handles it well.
– dirkt
Nov 23 at 16:13
2
2
This is just my opinion, but I'd go with
xfs as it was originally designed for this workload. Second choice is zfs but I've never done any production work with this filesystem.– Doug O'Neal
Nov 23 at 15:17
This is just my opinion, but I'd go with
xfs as it was originally designed for this workload. Second choice is zfs but I've never done any production work with this filesystem.– Doug O'Neal
Nov 23 at 15:17
1
1
I don't think the fs will matter much in this case. What matters is how the application handles the directory.
– Henrik
Nov 23 at 15:31
I don't think the fs will matter much in this case. What matters is how the application handles the directory.
– Henrik
Nov 23 at 15:31
1
1
I would divide up the image files into one or two layers of subdirectories, make the application access them at the new locations and then use whatever filesystem was default one.
– Kusalananda
Nov 23 at 15:39
I would divide up the image files into one or two layers of subdirectories, make the application access them at the new locations and then use whatever filesystem was default one.
– Kusalananda
Nov 23 at 15:39
1
1
And note that applications that try something to do with the file names (like
ls, which will attempt to sort them) still present problems, even if the file system handles it well.– dirkt
Nov 23 at 16:13
And note that applications that try something to do with the file names (like
ls, which will attempt to sort them) still present problems, even if the file system handles it well.– dirkt
Nov 23 at 16:13
add a comment |
1 Answer
1
active
oldest
votes
up vote
3
down vote
accepted
In generally xfs is developed to handle very large files with random access and many (millions) of files in single directory.
answered Nov 23 at 15:40
Romeo Ninov
5,01431627
The speed usually does not depend on the filesystem but rather on the efficience of the DNLC.
– schily
Nov 23 at 17:47
@schily I did not test such solution on really large scale setup, but IMHO this filesystem provide one of the best balances and speed.
– Romeo Ninov
Nov 23 at 17:59
1
Well let's say it claims this feature, but since it is not available on Solaris or FreeBSD, you cannot compare it with ZFS and for this reason such a claim does not really help.
– schily
Nov 23 at 18:01
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
accepted
In generally xfs is developed to handle very large files with random access and many (millions) of files in single directory.
answered Nov 23 at 15:40
Romeo Ninov
5,01431627
The speed usually does not depend on the filesystem but rather on the efficience of the DNLC.
– schily
Nov 23 at 17:47
@schily I did not test such solution on really large scale setup, but IMHO this filesystem provide one of the best balances and speed.
– Romeo Ninov
Nov 23 at 17:59
1
Well let's say it claims this feature, but since it is not available on Solaris or FreeBSD, you cannot compare it with ZFS and for this reason such a claim does not really help.
– schily
Nov 23 at 18:01
add a comment |
up vote
3
down vote
accepted
In generally xfs is developed to handle very large files with random access and many (millions) of files in single directory.
answered Nov 23 at 15:40
Romeo Ninov
5,01431627
The speed usually does not depend on the filesystem but rather on the efficience of the DNLC.
– schily
Nov 23 at 17:47
@schily I did not test such solution on really large scale setup, but IMHO this filesystem provide one of the best balances and speed.
– Romeo Ninov
Nov 23 at 17:59
1
Well let's say it claims this feature, but since it is not available on Solaris or FreeBSD, you cannot compare it with ZFS and for this reason such a claim does not really help.
– schily
Nov 23 at 18:01
add a comment |
up vote
3
down vote
accepted
up vote
3
down vote
accepted
In generally xfs is developed to handle very large files with random access and many (millions) of files in single directory.
answered Nov 23 at 15:40
Romeo Ninov
5,01431627
In generally xfs is developed to handle very large files with random access and many (millions) of files in single directory.
answered Nov 23 at 15:40
Romeo Ninov
5,01431627
answered Nov 23 at 15:40
Romeo Ninov
5,01431627
answered Nov 23 at 15:40
Romeo Ninov
5,01431627
answered Nov 23 at 15:40
Romeo Ninov
5,01431627
5,01431627
The speed usually does not depend on the filesystem but rather on the efficience of the DNLC.
– schily
Nov 23 at 17:47
@schily I did not test such solution on really large scale setup, but IMHO this filesystem provide one of the best balances and speed.
– Romeo Ninov
Nov 23 at 17:59
1
Well let's say it claims this feature, but since it is not available on Solaris or FreeBSD, you cannot compare it with ZFS and for this reason such a claim does not really help.
– schily
Nov 23 at 18:01
add a comment |
The speed usually does not depend on the filesystem but rather on the efficience of the DNLC.
– schily
Nov 23 at 17:47
@schily I did not test such solution on really large scale setup, but IMHO this filesystem provide one of the best balances and speed.
– Romeo Ninov
Nov 23 at 17:59
1
Well let's say it claims this feature, but since it is not available on Solaris or FreeBSD, you cannot compare it with ZFS and for this reason such a claim does not really help.
– schily
Nov 23 at 18:01
The speed usually does not depend on the filesystem but rather on the efficience of the DNLC.
– schily
Nov 23 at 17:47
The speed usually does not depend on the filesystem but rather on the efficience of the DNLC.
– schily
Nov 23 at 17:47
@schily I did not test such solution on really large scale setup, but IMHO this filesystem provide one of the best balances and speed.
– Romeo Ninov
Nov 23 at 17:59
@schily I did not test such solution on really large scale setup, but IMHO this filesystem provide one of the best balances and speed.
– Romeo Ninov
Nov 23 at 17:59
1
1
Well let's say it claims this feature, but since it is not available on Solaris or FreeBSD, you cannot compare it with ZFS and for this reason such a claim does not really help.
– schily
Nov 23 at 18:01
Well let's say it claims this feature, but since it is not available on Solaris or FreeBSD, you cannot compare it with ZFS and for this reason such a claim does not really help.
– schily
Nov 23 at 18:01
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f483700%2f100-000-images-in-a-single-directory%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
This is just my opinion, but I'd go with
xfsas it was originally designed for this workload. Second choice iszfsbut I've never done any production work with this filesystem.– Doug O'Neal
Nov 23 at 15:17
1
I don't think the fs will matter much in this case. What matters is how the application handles the directory.
– Henrik
Nov 23 at 15:31
1
I would divide up the image files into one or two layers of subdirectories, make the application access them at the new locations and then use whatever filesystem was default one.
– Kusalananda
Nov 23 at 15:39
1
And note that applications that try something to do with the file names (like
ls, which will attempt to sort them) still present problems, even if the file system handles it well.– dirkt
Nov 23 at 16:13