Filesystem in RAM that swaps to disk after a specified size
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I have a process on a Linux machine that fetches some chunk of data from a host and sends it to another host. I do not control the size of the chunks I fetch, but I have a general idea of their median/average size.
I don't want to pay the cost of writing the data to the disk, as most of the time the chunks fit in RAM, so it would be a waste to write them on disk to reread them and delete them just after that.
What I would want to do is have a filesystem like tmpfs, with a limit on the in-RAM size, that swaps to disk when the limit is reached.
I tried to use a tmpfs with a lot of swap, but Linux tends to swap my programs instead of swapping the content of the tmpfs, and it deadlocks the machine.
What could I use to avoid paying the cost of disk writes for files I'm going to delete soon?
filesystems ram tmpfs
asked 10 hours ago
Antoine Pietri
1011
New contributor
Antoine Pietri is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
up vote
0
down vote
favorite
I have a process on a Linux machine that fetches some chunk of data from a host and sends it to another host. I do not control the size of the chunks I fetch, but I have a general idea of their median/average size.
I don't want to pay the cost of writing the data to the disk, as most of the time the chunks fit in RAM, so it would be a waste to write them on disk to reread them and delete them just after that.
What I would want to do is have a filesystem like tmpfs, with a limit on the in-RAM size, that swaps to disk when the limit is reached.
I tried to use a tmpfs with a lot of swap, but Linux tends to swap my programs instead of swapping the content of the tmpfs, and it deadlocks the machine.
What could I use to avoid paying the cost of disk writes for files I'm going to delete soon?
filesystems ram tmpfs
asked 10 hours ago
Antoine Pietri
1011
New contributor
Antoine Pietri is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Raisecommit=mount option.
– Ipor Sircer
10 hours ago
@IporSircer ... if you are using ext3/ext4. But it sounds like you can still get unlucky and be hit by a commit anyway, it just decreases the frequency? Maybe XFS is better. And if you want more than 30 seconds for any FS, you must also raise the sysctlvm.dirty_expire_centiseconds.
– sourcejedi
9 hours ago
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have a process on a Linux machine that fetches some chunk of data from a host and sends it to another host. I do not control the size of the chunks I fetch, but I have a general idea of their median/average size.
I don't want to pay the cost of writing the data to the disk, as most of the time the chunks fit in RAM, so it would be a waste to write them on disk to reread them and delete them just after that.
What I would want to do is have a filesystem like tmpfs, with a limit on the in-RAM size, that swaps to disk when the limit is reached.
I tried to use a tmpfs with a lot of swap, but Linux tends to swap my programs instead of swapping the content of the tmpfs, and it deadlocks the machine.
What could I use to avoid paying the cost of disk writes for files I'm going to delete soon?
filesystems ram tmpfs
asked 10 hours ago
Antoine Pietri
1011
New contributor
Antoine Pietri is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I have a process on a Linux machine that fetches some chunk of data from a host and sends it to another host. I do not control the size of the chunks I fetch, but I have a general idea of their median/average size.
I don't want to pay the cost of writing the data to the disk, as most of the time the chunks fit in RAM, so it would be a waste to write them on disk to reread them and delete them just after that.
What I would want to do is have a filesystem like tmpfs, with a limit on the in-RAM size, that swaps to disk when the limit is reached.
I tried to use a tmpfs with a lot of swap, but Linux tends to swap my programs instead of swapping the content of the tmpfs, and it deadlocks the machine.
What could I use to avoid paying the cost of disk writes for files I'm going to delete soon?
filesystems ram tmpfs
filesystems ram tmpfs
asked 10 hours ago
Antoine Pietri
1011
New contributor
Antoine Pietri is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 10 hours ago
Antoine Pietri
1011
New contributor
Antoine Pietri is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 10 hours ago
Antoine Pietri
1011
New contributor
Antoine Pietri is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 10 hours ago
Antoine Pietri
1011
asked 10 hours ago
Antoine Pietri
1011
1011
New contributor
Antoine Pietri is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Antoine Pietri is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Antoine Pietri is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Raisecommit=mount option.
– Ipor Sircer
10 hours ago
@IporSircer ... if you are using ext3/ext4. But it sounds like you can still get unlucky and be hit by a commit anyway, it just decreases the frequency? Maybe XFS is better. And if you want more than 30 seconds for any FS, you must also raise the sysctlvm.dirty_expire_centiseconds.
– sourcejedi
9 hours ago
add a comment |
Raisecommit=mount option.
– Ipor Sircer
10 hours ago
@IporSircer ... if you are using ext3/ext4. But it sounds like you can still get unlucky and be hit by a commit anyway, it just decreases the frequency? Maybe XFS is better. And if you want more than 30 seconds for any FS, you must also raise the sysctlvm.dirty_expire_centiseconds.
– sourcejedi
9 hours ago
Raise
commit= mount option.– Ipor Sircer
10 hours ago
Raise
commit= mount option.– Ipor Sircer
10 hours ago
@IporSircer ... if you are using ext3/ext4. But it sounds like you can still get unlucky and be hit by a commit anyway, it just decreases the frequency? Maybe XFS is better. And if you want more than 30 seconds for any FS, you must also raise the sysctl
vm.dirty_expire_centiseconds.– sourcejedi
9 hours ago
@IporSircer ... if you are using ext3/ext4. But it sounds like you can still get unlucky and be hit by a commit anyway, it just decreases the frequency? Maybe XFS is better. And if you want more than 30 seconds for any FS, you must also raise the sysctl
vm.dirty_expire_centiseconds.– sourcejedi
9 hours ago
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
Normal file cashing should do this for you without any extra work.
Or use the commit mount option.
Or use lvmcache.
Or use a pipe.
answered 10 hours ago
user1133275
2,487414
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
Normal file cashing should do this for you without any extra work.
Or use the commit mount option.
Or use lvmcache.
Or use a pipe.
answered 10 hours ago
user1133275
2,487414
add a comment |
up vote
1
down vote
Normal file cashing should do this for you without any extra work.
Or use the commit mount option.
Or use lvmcache.
Or use a pipe.
answered 10 hours ago
user1133275
2,487414
add a comment |
up vote
1
down vote
up vote
1
down vote
Normal file cashing should do this for you without any extra work.
Or use the commit mount option.
Or use lvmcache.
Or use a pipe.
answered 10 hours ago
user1133275
2,487414
Normal file cashing should do this for you without any extra work.
Or use the commit mount option.
Or use lvmcache.
Or use a pipe.
answered 10 hours ago
user1133275
2,487414
answered 10 hours ago
user1133275
2,487414
answered 10 hours ago
user1133275
2,487414
answered 10 hours ago
user1133275
2,487414
2,487414
add a comment |
add a comment |
Antoine Pietri is a new contributor. Be nice, and check out our Code of Conduct.
Antoine Pietri is a new contributor. Be nice, and check out our Code of Conduct.
Antoine Pietri is a new contributor. Be nice, and check out our Code of Conduct.
Antoine Pietri is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f481251%2ffilesystem-in-ram-that-swaps-to-disk-after-a-specified-size%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Raise
commit=mount option.– Ipor Sircer
10 hours ago
@IporSircer ... if you are using ext3/ext4. But it sounds like you can still get unlucky and be hit by a commit anyway, it just decreases the frequency? Maybe XFS is better. And if you want more than 30 seconds for any FS, you must also raise the sysctl
vm.dirty_expire_centiseconds.– sourcejedi
9 hours ago