Use direct I/O with disk cache

[ivanyu]

The disk cache (obviously) reads and writes files. Currently, the files are open normally so they get into the OS page cache. Kafka heavily depends on the page cache for its consumer performance. A big read from the remote storage via the disk cache may wipe the hot local data from the page cache, thus affecting negatively non-remote consumer performance.

We should consider opening disk cache files in the direct I/O mode to not touch the page cache and make this behavior configurable.

[funky-eyes]

when is this feature expected to be available back?