Which filesystem will give me the best performance for both writing and reading small files?
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
Here's the context -
My company developed an application that picks up (watches for) xml files of <10kb in size from a directory, reads it in sends the body as an api call to an external service and then moves the file into a processed directory.
Due to the volume of files - roughly 2000/min we were getting dreadful performance out of NTFS. We were no where near able to keep up with the processing.
I'm a Linux guy through and through and from experience Linux would handle this situation a lot better especially with things like inotify which are leaps and bounds ahead of the ntfs api, that's why I've ported the code to .NET Core to give it a shot.
At home I use XFS on my Workstations and ZFS on my servers, so aside from ext4 - I have no real experience with any other filesystem.
So my question is - which filesystem (preferably in-tree) would be the most performant for this kind of workload.
linux filesystems
asked 2 mins ago
dcrdev
811
add a comment |Â
up vote
0
down vote
favorite
Here's the context -
My company developed an application that picks up (watches for) xml files of <10kb in size from a directory, reads it in sends the body as an api call to an external service and then moves the file into a processed directory.
Due to the volume of files - roughly 2000/min we were getting dreadful performance out of NTFS. We were no where near able to keep up with the processing.
I'm a Linux guy through and through and from experience Linux would handle this situation a lot better especially with things like inotify which are leaps and bounds ahead of the ntfs api, that's why I've ported the code to .NET Core to give it a shot.
At home I use XFS on my Workstations and ZFS on my servers, so aside from ext4 - I have no real experience with any other filesystem.
So my question is - which filesystem (preferably in-tree) would be the most performant for this kind of workload.
linux filesystems
asked 2 mins ago
dcrdev
811
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Here's the context -
My company developed an application that picks up (watches for) xml files of <10kb in size from a directory, reads it in sends the body as an api call to an external service and then moves the file into a processed directory.
Due to the volume of files - roughly 2000/min we were getting dreadful performance out of NTFS. We were no where near able to keep up with the processing.
I'm a Linux guy through and through and from experience Linux would handle this situation a lot better especially with things like inotify which are leaps and bounds ahead of the ntfs api, that's why I've ported the code to .NET Core to give it a shot.
At home I use XFS on my Workstations and ZFS on my servers, so aside from ext4 - I have no real experience with any other filesystem.
So my question is - which filesystem (preferably in-tree) would be the most performant for this kind of workload.
linux filesystems
asked 2 mins ago
dcrdev
811
Here's the context -
My company developed an application that picks up (watches for) xml files of <10kb in size from a directory, reads it in sends the body as an api call to an external service and then moves the file into a processed directory.
Due to the volume of files - roughly 2000/min we were getting dreadful performance out of NTFS. We were no where near able to keep up with the processing.
I'm a Linux guy through and through and from experience Linux would handle this situation a lot better especially with things like inotify which are leaps and bounds ahead of the ntfs api, that's why I've ported the code to .NET Core to give it a shot.
At home I use XFS on my Workstations and ZFS on my servers, so aside from ext4 - I have no real experience with any other filesystem.
So my question is - which filesystem (preferably in-tree) would be the most performant for this kind of workload.
linux filesystems
linux filesystems
asked 2 mins ago
dcrdev
811
asked 2 mins ago
dcrdev
811
asked 2 mins ago
dcrdev
811
asked 2 mins ago
dcrdev
811
asked 2 mins ago
dcrdev
811
811
add a comment |Â
add a comment |Â
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f479782%2fwhich-filesystem-will-give-me-the-best-performance-for-both-writing-and-reading%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
