mjhjmtu

I have some large files and i am ok with them being read at disk I/O capacity. I wish to have file-system cache free for other files. Is it possible to turn off file-system caching for specific files, on Linux?
I wish to do this programmatically via native lib + java.

You're looking for your Java equivalent of the O_DIRECT flag for open(2). See http://man7.org/linux/man-pages/man2/open.2.html

You can do so for an opened instance of the file, but not persistently for the file itself. You do so per instance of the opened file by using direct IO. I'm not sure how to do this in Java, but in C and C++, you pass the O_DIRECT flag to the open() call.

Note however that this has a couple of potentially problematic implications, namely:

• It's downright dangerous on certain filesystems. Most notably, current versions of BTRFS have serious issues with direct IO when you're writing to the file.


• You can't mix direct IO with regular cached I/O unless you use some form of synchronization. Cached writes won't show up for certain to direct IO reads until you call fsync() or fdatasync(), and direct IO writes may not show up for cached IO reads ever.

There is however an alternative method if you can tolerate having the data temporarily in cache. You can use the POSIX fadvise interface (through the posix_fadvise system call on Linux) to tell the kernel you don't need data from the file when you're done reading it. By using the POSIX_FADV_DONTNEED flag, you can tell the kernel to drop a specific region of a particular file from cache. You can actually do this as you are processing the file too (by reading a chunk, and then immediately after reading calling posix_fadvise on that region of the file), though the regions you call this on have to be aligned to the system's page size. This is generally the preferred portable method of handling things, as it works on any POSIX compliant system with the real-time extensions (which is pretty much any POSIX compliant system).