*A few months ago, I wrote a [summary of a summarized blog post](Summary%20on%20User-land%20Disk%20IO.md). It was more like personal note taking that I decided to share with others. This time, I've decided to take it a step further by explaining everything a bit more and organizing it into a single document to make it easier for myself (and hopefully for you) to understand, reducing the need for scattered searches. I hope you enjoy it!* When building high-performance storage systems (databases, key-value stores, etc.), understanding how the OS, filesystem, and storage hardware interact is crucial. How you do file I/O (synchronous vs buffered, direct I/O, async I/O) affects latency and throughput. How you ensure durability (fsync, write-through flags, filesystem journaling) affects data safety on crashes. And the choice of filesystem (ext4, XFS, raw block, etc.) and kernel tuning (I/O schedulers, memory dirty-ratio, disk monitoring) can make or break your system's reliability and performance. This deep dive unpacks the technical mechanisms and trade-offs behind these layers, with examples from real systems (PostgreSQL, RocksDB, Windows file APIs, etc.) and practical tips for engineers. ```mermaid flowchart TD A[Application] --> B{I/O Strategy} B --> C[Buffered I/O] B --> D[Direct I/O O_DIRECT] B --> E[Synchronous O_SYNC] B --> F[Asynchronous I/O] C --> G[Page Cache] D --> H[Raw Storage] E --> I[Immediate Flush] F --> J[Event Loop] G --> K[Filesystem Layer] H --> K I --> K J --> K K --> L{Filesystem Type} L --> M[ext4] L --> N[XFS] L --> O[Raw Block Device] M --> P[Storage Hardware] N --> P O --> P P --> Q[HDD/SSD/NVMe] style A fill:#e1f5fe style P fill:#fff3e0 style Q fill:#f3e5f5 ``` # File I/O Strategies Modern OSes offer multiple ways to write data to disk. Choosing the right strategy (direct I/O, synchronous writes, buffered I/O, async I/O) depends on your workload and durability needs. ## O_DIRECT (Bypass Page Cache) The `O_DIRECT` flag (Linux-specific) tells the kernel to bypass the page cache and do direct I/O to the storage device. This avoids _double caching_ (once in userspace buffers, once in OS cache) and gives more predictable I/O timing, which databases often use. For example, when writing a huge WAL (Write-Ahead Log), using `O_DIRECT` means the data goes straight to disk buffers. ```mermaid flowchart LR subgraph "Normal Buffered I/O" A1[Application Buffer] --> B1[Page Cache] --> C1[Storage Device] end subgraph "O_DIRECT I/O" A2[Application Buffer] --> C2[Storage Device] end style A1 fill:#e3f2fd style A2 fill:#e3f2fd style B1 fill:#fff3e0 style C1 fill:#f3e5f5 style C2 fill:#f3e5f5 ``` In Linux you can use: ```c int fd = open("data.log", O_CREAT|O_WRONLY|O_DIRECT, 0644); if (fd < 0) { perror("open O_DIRECT"); /* handle error */ } ``` This ensures writes go directly to disk (bypassing RAM cache). However, O_DIRECT imposes strict alignment requirements on the buffer address, file offset, and length. If these aren’t aligned to the filesystem block (often 4KB or the device’s block size), writes will fail with `EINVAL`. In practice you might use `posix_memalign` to allocate an aligned buffer, or ensure your I/O size is a multiple of the block size. **When to use O_DIRECT:** It shines for large, sequential I/O where the OS cache gives little benefit. Many databases (e.g. MySQL’s InnoDB, RocksDB, PostgreSQL WAL) default to direct I/O or write-through to avoid OS-level caching surprises. Bypassing the cache means you manage buffering entirely in your application, which can improve consistency and throughput for SSDs (no needless copying). **Caveats:** O_DIRECT I/O is generally slower for small writes (due to no caching, immediate device I/O) and cannot be combined easily with normal buffered reads/writes on the same file. Also, some filesystems or OSes may still have subtle buffering or metadata updates (depending on mount options). Always test on your hardware. ## O_SYNC / O_DSYNC (Synchronous I/O) The flags `O_SYNC` and `O_DSYNC` (Linux, POSIX) make each write call block until the data is on persistent media. With `O_SYNC`, every `write()` acts like a mini `fsync()` – the data _and_ required metadata (file size, allocation bitmap, etc.) are flushed to disk before `write()` returns. `O_DSYNC` is weaker: it waits for the data blocks to hit disk but may defer some metadata updates (except file size and modification time) For example: ```c int fd = open("file.dat", O_CREAT|O_WRONLY|O_SYNC, 0644); write(fd, buf, len); // returns only after data+metadata are on disk ``` This is often used for durability: each write is effectively **synchronous write-through**. However, the downside is performance: every write blocks on disk I/O, so throughput and latency suffer. For high write rates, the overhead can be huge. Some systems offer `fdatasync()` (POSIX) to explicitly flush data (no metadata) when needed, as an alternative to opening with `O_SYNC`. In practice, many apps do normal writes (buffered) and then call `fsync()` or `fdatasync()` at logical commit points. **Pitfalls:** Not all FSes treat metadata the same. For instance, with `O_SYNC`, you might expect full on-disk consistency after each write, but some filesystems (or storage controllers) may defer journaling. Also, network filesystems often ignore these flags. As a general rule, use `O_SYNC`/`O_DSYNC` only when you truly need per-write durability. ## Buffered I/O and fallocate By default, file I/O is _buffered_. When you `write()`, data typically goes into the kernel’s page cache and is flushed to disk later. This gives high throughput (writes return quickly) but means a crash before a flush can lose data. For durability, you then use `fsync()` to force the dirty pages out. The model is: write-many and sync-occasionally. **Buffered I/O** is good for throughput and small writes (the OS can merge and reorder writes efficiently). For example, PostgreSQL does buffered writes to heap files and uses `fsync()` at transaction commit (asynchronously grouping many writes). **Preallocation (`fallocate`)**: To avoid fragmentation or unexpected `ENOSPC`, databases often preallocate file space. POSIX provides `posix_fallocate(fd, offset, len)`, which blocks until space is reserved (on ext4 it also zeroes blocks). Linux-specific `fallocate(fd, mode, offset, len)` has more features (e.g. punch holes). For example: ```c int fd = open("data.dat", O_CREAT|O_WRONLY, 0644); posix_fallocate(fd, 0, 1024*1024*1024); // allocate 1GB ``` Preallocation can improve I/O consistency (space guaranteed) and performance (less fragmentation). Note: `fallocate` doesn’t write actual data, just reserves space, so if power fails, the space may appear zeroed in the file. ## Asynchronous I/O Options For high concurrency, asynchronous I/O lets a thread issue multiple IOs and be notified on completion. On Linux we have: ```mermaid sequenceDiagram participant App as Application participant Kernel as Kernel participant Disk as Storage Device Note over App,Disk: Synchronous I/O (blocking) App->>Kernel: write() call Kernel->>Disk: Write data Disk-->>Kernel: Write complete Kernel-->>App: write() returns Note over App,Disk: Asynchronous I/O (non-blocking) App->>Kernel: io_submit() - queue multiple requests App->>App: Continue processing Kernel->>Disk: Write data 1 Kernel->>Disk: Write data 2 Kernel->>Disk: Write data N App->>Kernel: io_getevents() - check completion Disk-->>Kernel: Multiple completions Kernel-->>App: Completion events ``` - **POSIX AIO (aio_read/write)**: poor performance historically (kernel spawns threads to emulate async I/O). - **Linux native AIO (`libaio`)**: works only on direct I/O by default (no page cache), better throughput. Good for databases doing parallel writes to raw disks. - **io_uring**: modern async I/O API (since Linux 5.x) that supports both buffered and direct I/O with minimal syscalls. It's increasingly used in high-performance applications. RocksDB, for instance, can use Linux AIO for background flush and compaction, enabling overlap of computation and disk I/O. Code example for Linux AIO (libaio) would involve `io_setup()`, `io_submit()`, etc., but that’s too low-level for this post. The key tip: if you have multiple disks or a fast NVMe, asynchronous queues (via io_uring or threaded aio) can maximize IOPS and throughput. ## Practical Tips for File I/O - **Batch your I/O**: Write in large blocks (e.g. 1MB or more) to amortize per-call cost. - **Align buffers**: When using `O_DIRECT`, allocate your buffers with `posix_memalign` to disk block boundaries - **Use write barriers / flushes carefully**: If using `O_DSYNC`, know it may not flush metadata. Consider `fdatasync()` to flush data after writes. - **Fallocate early**: After creating a database file, `fallocate` the expected size. This avoids unexpected delays later and enforces available space. - **Try RocksDB’s suggestions**: For example, RocksDB’s wiki suggests using `O_DIRECT | O_TRUNC` for leveldb files, `O_SYNC` for write-ahead logs, and using AIO for background flushes. # Durability Mechanisms Ensuring data is truly on stable storage is tricky. Here we cover `fsync`, `O_SYNC`/`O_DSYNC`, common pitfalls, and platform quirks. ## fsync, O_SYNC vs O_DSYNC explained ### `fsync`, The Chameleon - The Unix `fsync(fd)` system call tells the OS to flush all buffered data of the given file descriptor to the disk hardware. - On POSIX systems, `fsync()` guarantees that data is transferred to the device; it may also flush associated metadata (modification time, etc.). - `fdatasync(fd)` is similar but only flushes file data (not all metadata). ```mermaid flowchart TD A["write() calls"] --> B[Data in Page Cache] B --> C["fsync() called"] C --> D{What gets flushed?} D --> E[Data Blocks] D --> F[Metadata] D --> G[Journal Commits] E --> H[Storage Device] F --> H G --> H I["fdatasync() called"] --> J[Data Blocks Only] B --> I J --> H style C fill:#ffeb3b style I fill:#4caf50 style H fill:#f3e5f5 ``` ### O_SYNC, O_DSYNC - Strange siblings - When you open a file with `O_SYNC` on Linux, each `write()` call will behave like an `fdatasync()` after the data write (and also updates metadata). - With `O_DSYNC`, each write is like calling `fsync()` on the _data only_, not necessarily the metadata (except file size/time). In practice, if you open with `O_SYNC`, you don’t need extra `fsync()`, since writes block until durability. With buffered I/O (no flags), you must call `fsync()` at commit. **Key point:** An `fsync()` can trigger more work than just flushing your data – it may force the filesystem journal to commit transactions, and if multiple files share a journal, you could see all pending metadata flush at once. This makes `fsync()` expensive and somewhat unpredictable in latency. ## Real-world fsync issues (e.g., “fsyncgate”) In real deployments, many have observed that `fsync()` can become a bottleneck. On ext4 (the default Linux FS), each `fsync()` typically commits the filesystem journal (metadata changes) to stable storage. This can stall all writes until the journal commit completes. In some kernels, ext4 batches fsync calls up to a timeout (controlled by `commit=` mount option or journaling batch time) to improve throughput, but this adds latency. One infamous anecdote is the so-called _fsyncgate_ in certain workloads, where heavy fsync usage (such as many small sync calls) leads to severe performance cliffs and lock-ups. The exact name “fsyncgate” refers to news in early 2010s when Chrome (WebKit) noticed filesystem fsync behaviors causing delays. The lesson is simple but priceless: if you rely heavily on `fsync()`, tune or choose your filesystem and kernel wisely. Some DBs mitigate this by: - batching fsyncs, - disabling journaling for certain writes, - using hardware with a battery-backed cache. In contrast, some filesystems (like XFS or ZFS) handle metadata differently (XFS also journals metadata but in a different way, ZFS uses copy-on-write) which can yield different `fsync` behavior. But any filesystem needs to ensure data on disk eventually. ## Platform-specific notes - **Linux**: Use `fsync()` or `fdatasync()` on file descriptors to force data to disk. You can also mount filesystems with options like `barrier=1` (default) to ensure write barriers, or disable journal (`data=writeback` is fastest but unsafe on crash). By default ext4 is very safe (ordered journaling). - **macOS (APFS/HFS+)**: macOS historically has had strange behaviors . HFS+ with journaling used to _not_ write data out on `fsync()` but only metadata, so applications often use `fcntl(f, F_FULLFSYNC)` on macOS to do a full flush. APFS (newer Apple FS) changed many behaviors but still, using `O_SYNC` is slower on Mac and it’s often better to rely on the higher-level CoreData/SQLite settings or use `F_FULLFSYNC`. - One tip: to bypass the buffer cache on macOS, use `fcntl(fd, F_NOCACHE, 1)` which is roughly analogous to Linux’s `O_DIRECT`, though it still goes through some caching. - **Windows**: The Win32 APIs use flags like `FILE_FLAG_WRITE_THROUGH` or `FILE_FLAG_NO_BUFFERING` to control caching. If you create a file handle with `FILE_FLAG_WRITE_THROUGH`, writes will go to disk (or RAID controller) immediately without being held in the OS cache. In code, you can do something like: ```c HANDLE h = CreateFile(L"dbfile.dat", GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_FLAG_WRITE_THROUGH, NULL); ``` - This ensures multi-sector writes don’t get batched in volatile cache. To explicitly flush, Windows has `FlushFileBuffers(h)`, similar to `fsync()`. - Keep in mind: Windows disk caching and drivers may still reorder or delay writes, so enterprise apps often use both write-through flags and flush calls. - **RAID/NVMe**: Many storage controllers and NVMe SSDs have large volatile caches. Some respect the FUA (Force Unit Access) bit on writes to skip the cache, but not all. On NVMe, the “AWUPF” (Atomic Write Unit Power Fail) feature indicates whether sequential multi-block writes can be guaranteed atomic on power failure. Most consumer SSDs don’t guarantee FUA for every write (they rely on a battery or supercap). In short, on Linux you normally rely on `fsync()`/flush commands rather than trying to use FUA flags directly. # Force Unit Access (FUA) Force Unit Access is a low-level flag in SCSI/ATA protocols that means “write this data through all caches to media”. It was intended to let the OS tell the disk to bypass its volatile write cache for that operation. In practice, many drives respect it (especially if not battery-backed), but it can hurt performance. For e.g., ATA WRITE commands have a bit for FUA, and SCSI WRITE commands have a similar flag. If a block device use FUA, every write with FUA=1 will not return until the data is on non-volatile media. This is similar in effect to a write followed by a flush. **Hardware limitations:** Many modern disks and RAID controllers keep data in DRAM or a volatile cache. They often guarantee data on power-loss only if they have a battery-backed cache, and typically only flush on explicit commands or barriers. NVMe introduced a related concept: the AWUPF (Atomic Write Unit Power Fail) field in the Identify Namespace data, which tells how many sectors can be written atomically under power failure. It’s complicated and rare that hosts use this directly. In Linux, the block layer can use FUA if the device says it supports it, but often the driver translates it to a cache flush command instead. In short, as a software engineer you usually don’t handle FUA bits directly. You rely on higher-level guarantees (`fsync()`, write-through flags) and trust the kernel/driver to issue the necessary flushes or FUA commands. So as the bottom line: **FUA is a thing, but on modern NVMe/SSD it’s typically abstracted by the storage stack.** # Filesystems Deep Dive Which filesystem you choose can affect different attributes like performance, scalability, and durability. We’ll focus on ext4 and XFS (common for Linux servers) and some general points about metadata and raw devices. ## Why XFS and Ext4 are often chosen ```mermaid flowchart LR subgraph "ext4 Architecture" A1[Data Blocks] --> B1[Block Groups] C1[Metadata Journal] --> B1 D1[extern tree Inodes] --> B1 B1 --> E1[Ordered Mode:<br/>Data before Metadata] end subgraph "XFS Architecture" A2[Data Blocks] --> B2[Allocation Groups] C2[Metadata Journal] --> B2 D2[B+Tree Inodes] --> B2 B2 --> E2[Parallel I/O:<br/>No Global Locks] end style E1 fill:#e3f2fd style E2 fill:#fff3e0 ``` - **ext4:** The default for many Linux distros. There are many reasons for it to being default. - It uses journaling for metadata (and optionally data). - It's mature and generally stable. - Ext4's default "ordered" mode ensures that data blocks are written to disk before the journal entry is committed, preventing metadata from pointing to garbage. - Ext4 also supports extents (big contiguous blocks) and fast fsck for large files. - Because it was the successor to ext3, it's often the safe, conservative choice. For many workloads, ext4's performance is excellent, and it handles mixed reads/writes well. - **XFS:** Known for high performance on large files and parallel I/O. - XFS is also a journaling filesystem (metadata only) but is built for scalability. - It preallocates a lot of metadata space and can allocate extents very quickly even under heavy concurrency. - XFS tends to excel in throughput, especially on RAID or multi-disk. It has better online defragmentation (`xfs_fsr`) and tends to avoid global locks. - Many big data companies (e.g. Facebook) favor XFS for workloads like MySQL or data warehouses. ScyllaDB also suggest using XFS as the filesystem. In practice, both ext4 and XFS are popular for databases. Some database vendors test on both. For example, PostgreSQL has recommendations for ext4 and XFS, and on newer Linux kernels XFS might even outperform ext4 slightly for large WAL writes. As a rule of thumb that I've learned about databases, it's always worth benchmarking all options on your specific workload. ## Metadata optimizations (journaling, extents, etc.) Filesystems optimize how they update metadata (like directory trees, allocation maps) to minimize disk latency. - **Journaling:** Both ext4 and XFS journal metadata by default. This means changes to inodes or directory entries are first written to a log. On a crash, the journal is replayed to make metadata consistent. Journaling adds overhead on `fsync()` of metadata or file close. Ext4 additionally offers _data journaling_ modes (`data=journal`), but that’s slow and rarely used outside of unusual crash-safety needs. Typically ext4 uses `data=ordered` (default) or `data=writeback`. - **Extent allocation:** Ext4 and XFS use extents (contiguous block runs) to reduce fragmentation. This speeds up large file scans and makes preallocation efficient. XFS in particular uses large extent groups, which helps when writing big sequential files (like database dumps or backups). - **Batching and latencies:** As I've mentioned earlier, some kernels have features like “delayed fsync” or “group commit” in the fs layer. For e.g., ext4 can batch multiple fsync calls that happen within a short time window (controlled by `commit=sec` mount option). This improves throughput at the cost of a bit of latency, but default settings try to balance it. ### Raw device access Sometimes databases are given a raw block device (no filesystem) to manage themselves. For example, Oracle’s Direct NFS or some enterprise setups use raw partitions. The advantage is zero filesystem overhead – the DB can control exactly how data is placed. There are no journals or caches in between. On the other hand, you lose the flexibility and safety of a filesystem (no easy human tools for listing files, backing up with `tar`, etc.). In Linux, raw access usually means just using a block device path (like `/dev/sdb1`) or a device-mapper target, and doing block I/O on it. If you do use raw, you still must ensure alignment (partition offsets, I/O sizes) and flush caches (using `fsync` on a file descriptor bound to the raw device or `blockdev --flushbufs`) for durability. Raw mode is less common now that filesystems and SSDs are so fast, but it can still be found in legacy or specialized systems. # Kernel-Level Optimization Tips Beyond choosing the I/O API and filesystem, the Linux kernel provides options to tune I/O behavior and monitor what’s happening. We will go through some of them, these are the ones that I found interesting and useful. ## I/O Schedulers In your early days, if you liked OS stuff like I did and had a chance to thumb through _Modern Operating Systems_ by Tanenbaum, you definitely knew about I/O schedulers or at least had heard of them. In that book, I/O schedulers are defined this way (which I found easy to understand): *On top of the disk drivers is the I/O scheduler, which is responsible for ordering and issuing disk-operation requests in a way that tries to conserve wasteful disk head movement or to meet some other system policy* Linux historically offered different block schedulers: ```mermaid flowchart TD A[I/O Requests from Applications] --> B{I/O Scheduler} B --> C[CFQ - Complete Fair Queuing] B --> D[Deadline Scheduler] B --> E[Noop/None Scheduler] B --> F[BFQ - Budget Fair Queuing] B --> G[MQ-Deadline] C --> H["Fair sharing<br/>Multiple process queues<br/>Good for: Desktop/Mixed"] D --> I["Deadline-based<br/>Prevents starvation<br/>Good for: Databases/RT"] E --> J["Simple FIFO<br/>No reordering<br/>Good for: SSDs/RAID"] F --> K["Fair + High throughput<br/>Budget-based<br/>Good for: Desktop/HDDs"] G --> L["Multi-queue aware<br/>Scalable deadlines<br/>Good for: NVMe/Modern"] H --> M[Block Device] I --> M J --> M K --> M L --> M style B fill:#ffeb3b style M fill:#f3e5f5 ``` - **CFQ (Completely Fair Queuing):** Tries to fairly share the disk among processes, good for desktop/mixed workloads. Can be suboptimal for single-threaded sequential I/O. As of kernel 5.x, CFQ is often replaced by the newer `mq-deadline` by default. - **Deadline:** Aims to minimize I/O latency by imposing deadlines on requests (prevents starvation). Good for database or real-time systems where latency predictability matters. - **Noop:** Very simple FIFO queue, essentially no reordering. Best for SSDs or RAID controllers that do their own scheduling. Minimal overhead. - **BFQ (Budget Fair Queuing):** Focuses on I/O fairness and high throughput, good for desktop and HDDs. - **Kyber/MQ-deadline:** Newer multi-queue aware schedulers (the default in modern kernels) designed for NVMe and multi-queue storage, combining deadline-like latency guarantees with scalability. You can see/set the scheduler for a device via sysfs. For example: ```bash $ cat /sys/block/sda/queue/scheduler noop [deadline] cfq $ echo noop > /sys/block/sda/queue/scheduler ``` This example shows that `deadline` is active (in brackets), and you can switch to `noop`. **Tips:** - For SSDs/NVMe, `noop` or `mq-deadline` (the new multi-queue deadline) is usually recommended, since the hardware handles reordering well. - For spinning disks or mixed loads, `deadline` often outperforms CFQ. BFQ can improve throughput on HDDs, but isn’t in all kernels by default. - Again remember, Always benchmark your workload: e.g. some Postgres users find `deadline` or `noop` gives better latency for WAL writes, while others stick with the default on newer kernels. ## Tuning `vm.dirty_ratio` The Linux VM has parameters controlling how much dirty (modified) data can accumulate in RAM before being written out. The key sysctls are: ```mermaid flowchart TD A["Application writes<br/>(buffered I/O)"] --> B["Dirty Pages<br/>in RAM"] B --> C{"Dirty page<br/>threshold check"} C --> D["< vm.dirty_background_ratio<br/>(~10% default)"] C --> E["< vm.dirty_ratio<br/>(~20% default)"] C --> F["> vm.dirty_ratio<br/>CRITICAL!"] D --> G["Normal operation<br/>No immediate flush"] E --> H["Background flusher<br/>threads activate"] F --> I["All writes BLOCK<br/>until flush completes"] H --> J["Gradual write-out<br/>to storage"] I --> K["Forced immediate<br/>write-out"] style F fill:#f44336,color:#fff style I fill:#f44336,color:#fff style H fill:#ff9800,color:#fff style G fill:#4caf50,color:#fff ``` - `vm.dirty_ratio`: percentage of memory that can be filled with dirty pages before forcing processes to write them (default ~20%). - `vm.dirty_background_ratio`: a lower percentage at which background kernel flush threads start writing dirty pages (default ~10%). - `vm.dirty_bytes`/`vm.dirty_background_bytes`: absolute versions to override the above. As you can expect, If `vm.dirty_ratio` is too high, a process can buffer a lot of data in RAM, then suddenly stall when hitting the limit (all writes block until flush). If it’s too low, you may flush very often in small chunks, hurting throughput. For a database server, it often makes sense to lower these values so that writes go out steadily in the background rather than in huge bursts. For e.g.: ```bash sysctl -w vm.dirty_ratio=10 sysctl -w vm.dirty_background_ratio=5 ``` This would limit dirty to 10% of RAM (active tasks block at that point) and start background flush at 5%. On a machine with 64GB RAM, 10% is 6.4GB of dirty data – still a lot, but you can tune down further if you have very critical latency needs. Some administrators even switch to byte-based (`dirty_bytes`) to set an absolute cap (e.g. 1GB) regardless of memory size. ## Disk Monitoring with `/proc/diskstats` To diagnose I/O issues, monitor `/proc/diskstats` or use tools like `iostat`/`dstat`. The `/proc/diskstats` file has one line per block device, with fields such as reads completed, sectors read, writes completed, sectors written, and I/O time. For e.g.: ``` $ grep sda /proc/diskstats 8 0 sda 10240 0 409600 5000 51200 0 256000 3000 0 3000 8000 ``` Here, the fields after the device name are counts of operations and timing. In practice, `iostat -x 1` is more user-friendly, showing IOPS, MB/s, and latency per device every second. Watching for high await times or saturation helps decide if you need more disks/RAID, a different scheduler, or other tuning. Another useful interface is `blktrace` (with `blkparse`) or `bcc`/`bpf` tools (like `biolatency.py`) for deep analysis, but that is beyond scope (maybe It could be the next wasticle(wasted article)) # Conclusion File I/O and durability are complex layers spanning application, OS, and hardware. Here's the complete picture: ```mermaid flowchart TD subgraph "Application Layer" A1["Database/App<br/>Write Requests"] A2["I/O Strategy Choice"] A1 --> A2 end subgraph "OS/Kernel Layer" B1["Page Cache<br/>(Buffered)"] B2["Direct I/O<br/>(O_DIRECT)"] B3["Sync Flags<br/>(O_SYNC/DSYNC)"] B4["I/O Scheduler"] B5["VFS Layer"] A2 --> B1 A2 --> B2 A2 --> B3 B1 --> B4 B2 --> B4 B3 --> B4 B4 --> B5 end subgraph "Filesystem Layer" C1["ext4/XFS<br/>Journaling"] C2["Metadata<br/>Management"] C3["Block<br/>Allocation"] B5 --> C1 C1 --> C2 C1 --> C3 end subgraph "Hardware Layer" D1["Storage Device<br/>Cache"] D2["Persistent<br/>Storage"] C2 --> D1 C3 --> D1 D1 --> D2 end subgraph "Monitoring & Tuning" E1["iostat, /proc/diskstats"] E2["vm.dirty_ratio tuning"] E3["I/O scheduler selection"] end style A1 fill:#e3f2fd style D2 fill:#f3e5f5 style E1 fill:#fff3e0 ``` Key takeaways for engineers: - **Know your I/O flags:** Use `O_DIRECT` when you want raw device behavior and can manage caching yourself. Use `O_SYNC/O_DSYNC` or explicit `fsync()` only when you need guaranteed persistence, but expect latency costs. - **Understand fsync:** It may flush more than just your file (journals, etc.), so batch commits when possible. Keep journaling overhead in mind. - **Choose the right filesystem:** ext4 and XFS are both solid for high-performance workloads, but test your specific pattern. Use RAID or SSDs as needed. - **Tune the kernel:** Set the I/O scheduler appropriate for your device (e.g. `deadline` or `noop` for SSDs). Adjust `vm.dirty_ratio` to prevent big write stalls. Monitor with `iostat` or `/proc/diskstats`. **Ultimately, the fastest system is the one you measure**. Benchmark with realistic data, simulate crashes to test durability, and iterate on these settings. As long as you understand what each layer is doing (and occasionally quote the docs), you can build a reliable, high-throughput storage subsystem. #disk #linux #filesystem #database