== sanity test 398b: DIO and buffer IO race ============== 21:00:08 (1746147608) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.31462 s, 21.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32328: Thu May 1 21:00:45 2025 read: IOPS=47, BW=191KiB/s (196kB/s)(5992KiB/31364msec) clat (usec): min=54, max=69137, avg=9163.71, stdev=8332.60 lat (usec): min=54, max=69139, avg=9167.70, stdev=8332.11 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 169], | 30.00th=[ 5080], 40.00th=[ 6390], 50.00th=[ 7570], 60.00th=[ 8979], | 70.00th=[11207], 80.00th=[14877], 90.00th=[19792], 95.00th=[24773], | 99.00th=[36439], 99.50th=[45876], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 23, max= 295, per=25.26%, avg=190.48, stdev=52.22, samples=62 iops : min= 5, max= 73, avg=47.23, stdev=13.16, samples=62 write: IOPS=50, BW=201KiB/s (206kB/s)(6296KiB/31364msec) clat (usec): min=439, max=315157, avg=11077.32, stdev=13973.44 lat (usec): min=439, max=315160, avg=11080.03, stdev=13973.40 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 717], 10.00th=[ 775], 20.00th=[ 996], | 30.00th=[ 2114], 40.00th=[ 5473], 50.00th=[ 8356], 60.00th=[ 10814], | 70.00th=[ 14353], 80.00th=[ 18482], 90.00th=[ 25560], 95.00th=[ 31065], | 99.00th=[ 50594], 99.50th=[ 56361], 99.90th=[193987], 99.95th=[316670], | 99.99th=[316670] bw ( KiB/s): min= 7, max= 312, per=25.25%, avg=199.50, stdev=52.05, samples=62 iops : min= 1, max= 78, avg=49.48, stdev=13.11, samples=62 lat (usec) : 100=3.58%, 250=6.54%, 500=0.55%, 750=4.23%, 1000=6.25% lat (msec) : 2=5.05%, 4=4.98%, 10=29.30%, 20=25.65%, 50=13.15% lat (msec) : 100=0.65%, 250=0.03%, 500=0.03% cpu : usr=0.18%, sys=13.67%, ctx=4858, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32329: Thu May 1 21:00:45 2025 read: IOPS=46, BW=187KiB/s (191kB/s)(5940KiB/31821msec) clat (usec): min=50, max=271276, avg=9095.17, stdev=10903.50 lat (usec): min=51, max=271278, avg=9096.26, stdev=10903.54 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 133], | 30.00th=[ 4555], 40.00th=[ 6259], 50.00th=[ 7635], 60.00th=[ 8979], | 70.00th=[ 11207], 80.00th=[ 14746], 90.00th=[ 19268], 95.00th=[ 23725], | 99.00th=[ 33817], 99.50th=[ 45351], 99.90th=[ 94897], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 7, max= 335, per=24.42%, avg=184.10, stdev=66.39, samples=63 iops : min= 1, max= 83, avg=45.70, stdev=16.68, samples=63 write: IOPS=49, BW=199KiB/s (204kB/s)(6348KiB/31821msec) clat (usec): min=409, max=127771, avg=11374.26, stdev=12676.99 lat (usec): min=410, max=127773, avg=11376.17, stdev=12677.47 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 725], 10.00th=[ 791], 20.00th=[ 1139], | 30.00th=[ 2180], 40.00th=[ 5080], 50.00th=[ 8291], 60.00th=[ 10945], | 70.00th=[ 14353], 80.00th=[ 18744], 90.00th=[ 26608], 95.00th=[ 33424], | 99.00th=[ 54264], 99.50th=[ 71828], 99.90th=[122160], 99.95th=[127402], | 99.99th=[127402] bw ( KiB/s): min= 88, max= 328, per=25.05%, avg=197.92, stdev=49.57, samples=62 iops : min= 22, max= 82, avg=49.16, stdev=12.46, samples=62 lat (usec) : 100=4.13%, 250=7.49%, 500=0.55%, 750=3.55%, 1000=5.47% lat (msec) : 2=6.09%, 4=5.57%, 10=27.38%, 20=26.24%, 50=12.60% lat (msec) : 100=0.81%, 250=0.10%, 500=0.03% cpu : usr=0.09%, sys=13.56%, ctx=4856, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32330: Thu May 1 21:00:45 2025 read: IOPS=47, BW=190KiB/s (194kB/s)(6028KiB/31761msec) clat (usec): min=72, max=66485, avg=9228.66, stdev=8273.86 lat (usec): min=73, max=66486, avg=9230.34, stdev=8273.84 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 139], | 30.00th=[ 4883], 40.00th=[ 6521], 50.00th=[ 7832], 60.00th=[ 9503], | 70.00th=[12125], 80.00th=[15008], 90.00th=[19792], 95.00th=[23725], | 99.00th=[33817], 99.50th=[39060], 99.90th=[64750], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 64, max= 312, per=25.39%, avg=191.44, stdev=54.43, samples=62 iops : min= 16, max= 78, avg=47.65, stdev=13.59, samples=62 write: IOPS=49, BW=197KiB/s (202kB/s)(6260KiB/31761msec) clat (usec): min=449, max=431865, avg=11246.18, stdev=15831.33 lat (usec): min=450, max=431868, avg=11247.78, stdev=15831.82 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 1029], | 30.00th=[ 2024], 40.00th=[ 4621], 50.00th=[ 8291], 60.00th=[ 10945], | 70.00th=[ 14484], 80.00th=[ 18482], 90.00th=[ 25297], 95.00th=[ 32637], | 99.00th=[ 52167], 99.50th=[ 65799], 99.90th=[113771], 99.95th=[429917], | 99.99th=[429917] bw ( KiB/s): min= 80, max= 319, per=25.10%, avg=198.31, stdev=47.89, samples=62 iops : min= 20, max= 79, avg=49.35, stdev=11.96, samples=62 lat (usec) : 100=4.04%, 250=6.67%, 500=0.52%, 750=4.00%, 1000=6.32% lat (msec) : 2=5.40%, 4=5.47%, 10=26.82%, 20=27.15%, 50=12.79% lat (msec) : 100=0.75%, 250=0.03%, 500=0.03% cpu : usr=0.20%, sys=13.61%, ctx=4859, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32331: Thu May 1 21:00:45 2025 read: IOPS=47, BW=191KiB/s (195kB/s)(6052KiB/31750msec) clat (usec): min=52, max=81279, avg=9182.62, stdev=8413.29 lat (usec): min=53, max=81280, avg=9184.53, stdev=8412.83 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 139], | 30.00th=[ 4883], 40.00th=[ 6325], 50.00th=[ 7832], 60.00th=[ 9372], | 70.00th=[11469], 80.00th=[14877], 90.00th=[20055], 95.00th=[24249], | 99.00th=[33817], 99.50th=[39584], 99.90th=[77071], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 7, max= 328, per=24.95%, avg=188.10, stdev=55.08, samples=63 iops : min= 1, max= 82, avg=46.65, stdev=13.83, samples=63 write: IOPS=49, BW=196KiB/s (201kB/s)(6236KiB/31750msec) clat (usec): min=463, max=332230, avg=11329.82, stdev=14299.45 lat (usec): min=464, max=332231, avg=11331.73, stdev=14299.60 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 996], | 30.00th=[ 1926], 40.00th=[ 4817], 50.00th=[ 8455], 60.00th=[ 11338], | 70.00th=[ 14746], 80.00th=[ 19006], 90.00th=[ 25822], 95.00th=[ 33424], | 99.00th=[ 50594], 99.50th=[ 63701], 99.90th=[106431], 99.95th=[333448], | 99.99th=[333448] bw ( KiB/s): min= 31, max= 367, per=24.61%, avg=194.44, stdev=54.97, samples=63 iops : min= 7, max= 91, avg=48.24, stdev=13.81, samples=63 lat (usec) : 100=3.97%, 250=7.00%, 500=0.39%, 750=3.94%, 1000=6.45% lat (msec) : 2=5.57%, 4=5.40%, 10=26.69%, 20=26.11%, 50=13.80% lat (msec) : 100=0.62%, 250=0.03%, 500=0.03% cpu : usr=0.16%, sys=13.57%, ctx=4860, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=755KiB/s (773kB/s), 187KiB/s-191KiB/s (191kB/s-196kB/s), io=23.4MiB (24.6MB), run=31364-31821msec WRITE: bw=790KiB/s (809kB/s), 196KiB/s-201KiB/s (201kB/s-206kB/s), io=24.6MiB (25.7MB), run=31364-31821msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32332: Thu May 1 21:01:42 2025 read: IOPS=17, BW=68.4KiB/s (70.0kB/s)(5992KiB/87611msec) clat (usec): min=1701, max=382149, avg=8011.13, stdev=12813.17 lat (usec): min=1702, max=382150, avg=8014.33, stdev=12812.89 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 19], 95.00th=[ 26], | 99.00th=[ 39], 99.50th=[ 48], 99.90th=[ 100], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 7, max= 231, per=25.37%, avg=68.49, stdev=43.69, samples=173 iops : min= 1, max= 57, avg=16.91, stdev=10.93, samples=173 write: IOPS=17, BW=71.9KiB/s (73.6kB/s)(6296KiB/87611msec) clat (msec): min=11, max=222, avg=47.92, stdev=26.33 lat (msec): min=11, max=222, avg=47.92, stdev=26.33 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 20], 10.00th=[ 23], 20.00th=[ 27], | 30.00th=[ 32], 40.00th=[ 36], 50.00th=[ 41], 60.00th=[ 46], | 70.00th=[ 55], 80.00th=[ 66], 90.00th=[ 87], 95.00th=[ 101], | 99.00th=[ 131], 99.50th=[ 144], 99.90th=[ 190], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 16, max= 184, per=25.44%, avg=72.00, stdev=34.22, samples=173 iops : min= 4, max= 46, avg=17.79, stdev= 8.61, samples=173 lat (msec) : 2=0.20%, 4=16.93%, 10=22.98%, 20=7.85%, 50=33.59% lat (msec) : 100=15.82%, 250=2.60%, 500=0.03% cpu : usr=0.05%, sys=3.35%, ctx=3387, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32333: Thu May 1 21:01:42 2025 read: IOPS=16, BW=66.9KiB/s (68.5kB/s)(5940KiB/88761msec) clat (usec): min=1864, max=337981, avg=7843.71, stdev=12282.38 lat (usec): min=1865, max=337982, avg=7845.39, stdev=12282.43 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 17], 95.00th=[ 24], | 99.00th=[ 43], 99.50th=[ 51], 99.90th=[ 159], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 7, max= 256, per=25.10%, avg=67.76, stdev=43.13, samples=174 iops : min= 1, max= 64, avg=16.78, stdev=10.86, samples=174 write: IOPS=17, BW=71.5KiB/s (73.2kB/s)(6348KiB/88761msec) clat (msec): min=11, max=188, avg=48.48, stdev=26.53 lat (msec): min=11, max=188, avg=48.48, stdev=26.53 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 21], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 32], 40.00th=[ 36], 50.00th=[ 41], 60.00th=[ 46], | 70.00th=[ 55], 80.00th=[ 67], 90.00th=[ 88], 95.00th=[ 104], | 99.00th=[ 134], 99.50th=[ 144], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 24, max= 168, per=25.25%, avg=71.47, stdev=33.12, samples=176 iops : min= 6, max= 42, avg=17.69, stdev= 8.36, samples=176 lat (msec) : 2=0.13%, 4=16.67%, 10=23.01%, 20=7.32%, 50=34.54% lat (msec) : 100=15.14%, 250=3.16%, 500=0.03% cpu : usr=0.06%, sys=3.31%, ctx=3413, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32334: Thu May 1 21:01:42 2025 read: IOPS=17, BW=68.2KiB/s (69.8kB/s)(6028KiB/88447msec) clat (usec): min=2081, max=99375, avg=7747.55, stdev=7960.02 lat (usec): min=2082, max=99377, avg=7749.21, stdev=7959.86 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2671], 10.00th=[ 2966], 20.00th=[ 3392], | 30.00th=[ 3851], 40.00th=[ 4293], 50.00th=[ 4817], 60.00th=[ 5473], | 70.00th=[ 6718], 80.00th=[ 9241], 90.00th=[17957], 95.00th=[24249], | 99.00th=[38536], 99.50th=[42730], 99.90th=[77071], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 7, max= 207, per=25.28%, avg=68.26, stdev=42.97, samples=175 iops : min= 1, max= 51, avg=16.81, stdev=10.77, samples=175 write: IOPS=17, BW=70.8KiB/s (72.5kB/s)(6260KiB/88447msec) clat (msec): min=11, max=493, avg=48.95, stdev=29.15 lat (msec): min=11, max=493, avg=48.95, stdev=29.15 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 21], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 33], 40.00th=[ 36], 50.00th=[ 42], 60.00th=[ 47], | 70.00th=[ 55], 80.00th=[ 66], 90.00th=[ 86], 95.00th=[ 102], | 99.00th=[ 140], 99.50th=[ 157], 99.90th=[ 268], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 15, max= 184, per=25.00%, avg=70.76, stdev=32.87, samples=175 iops : min= 3, max= 46, avg=17.44, stdev= 8.27, samples=175 lat (msec) : 4=16.54%, 10=23.47%, 20=7.52%, 50=34.18%, 100=15.56% lat (msec) : 250=2.67%, 500=0.07% cpu : usr=0.13%, sys=3.26%, ctx=3408, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32335: Thu May 1 21:01:42 2025 read: IOPS=17, BW=69.8KiB/s (71.5kB/s)(6052KiB/86646msec) clat (usec): min=1837, max=99399, avg=7489.97, stdev=8255.65 lat (usec): min=1838, max=99400, avg=7492.04, stdev=8255.48 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2638], 10.00th=[ 2966], 20.00th=[ 3326], | 30.00th=[ 3720], 40.00th=[ 4178], 50.00th=[ 4752], 60.00th=[ 5342], | 70.00th=[ 6259], 80.00th=[ 8717], 90.00th=[16057], 95.00th=[23462], | 99.00th=[39584], 99.50th=[49021], 99.90th=[93848], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 7, max= 232, per=26.23%, avg=70.82, stdev=43.67, samples=170 iops : min= 1, max= 58, avg=17.49, stdev=10.91, samples=170 write: IOPS=17, BW=71.0KiB/s (73.7kB/s)(6236KiB/86646msec) clat (msec): min=11, max=336, avg=48.15, stdev=27.62 lat (msec): min=11, max=336, avg=48.15, stdev=27.62 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 20], 10.00th=[ 23], 20.00th=[ 28], | 30.00th=[ 32], 40.00th=[ 36], 50.00th=[ 41], 60.00th=[ 46], | 70.00th=[ 54], 80.00th=[ 66], 90.00th=[ 88], 95.00th=[ 106], | 99.00th=[ 136], 99.50th=[ 146], 99.90th=[ 207], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 16, max= 175, per=25.46%, avg=72.04, stdev=33.73, samples=172 iops : min= 4, max= 43, avg=17.80, stdev= 8.43, samples=172 lat (msec) : 2=0.16%, 4=17.48%, 10=22.85%, 20=7.91%, 50=34.05% lat (msec) : 100=14.52%, 250=2.99%, 500=0.03% cpu : usr=0.06%, sys=3.37%, ctx=3404, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=271KiB/s (277kB/s), 66.9KiB/s-69.8KiB/s (68.5kB/s-71.5kB/s), io=23.4MiB (24.6MB), run=86646-88761msec WRITE: bw=283KiB/s (290kB/s), 70.8KiB/s-71.0KiB/s (72.5kB/s-73.7kB/s), io=24.6MiB (25.7MB), run=86646-88761msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32421: Thu May 1 21:01:50 2025 read: IOPS=55, BW=891KiB/s (913kB/s)(5920KiB/6641msec) clat (usec): min=78, max=67095, avg=7751.58, stdev=8834.90 lat (usec): min=79, max=67096, avg=7752.72, stdev=8834.91 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 92], 10.00th=[ 98], 20.00th=[ 109], | 30.00th=[ 123], 40.00th=[ 1270], 50.00th=[ 6194], 60.00th=[ 8717], | 70.00th=[11469], 80.00th=[14091], 90.00th=[17957], 95.00th=[22414], | 99.00th=[42206], 99.50th=[56886], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 512, max= 1440, per=25.56%, avg=887.77, stdev=346.05, samples=13 iops : min= 32, max= 90, avg=55.38, stdev=21.54, samples=13 write: IOPS=59, BW=959KiB/s (982kB/s)(6368KiB/6641msec) clat (usec): min=649, max=98280, avg=9239.12, stdev=11545.69 lat (usec): min=651, max=98283, avg=9240.69, stdev=11545.84 clat percentiles (usec): | 1.00th=[ 807], 5.00th=[ 971], 10.00th=[ 1045], 20.00th=[ 1188], | 30.00th=[ 1418], 40.00th=[ 2311], 50.00th=[ 3556], 60.00th=[ 7570], | 70.00th=[11338], 80.00th=[16712], 90.00th=[23725], 95.00th=[30802], | 99.00th=[47973], 99.50th=[72877], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 512, max= 1728, per=26.15%, avg=955.54, stdev=361.08, samples=13 iops : min= 32, max= 108, avg=59.62, stdev=22.54, samples=13 lat (usec) : 100=5.86%, 250=12.11%, 500=0.52%, 750=0.65%, 1000=3.26% lat (msec) : 2=15.89%, 4=8.72%, 10=17.84%, 20=24.35%, 50=10.16% lat (msec) : 100=0.65% cpu : usr=0.14%, sys=15.38%, ctx=1016, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32422: Thu May 1 21:01:50 2025 read: IOPS=55, BW=888KiB/s (909kB/s)(5744KiB/6471msec) clat (usec): min=51, max=56601, avg=7400.61, stdev=8840.93 lat (usec): min=52, max=56602, avg=7401.71, stdev=8841.07 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 88], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 120], 40.00th=[ 192], 50.00th=[ 5997], 60.00th=[ 8455], | 70.00th=[ 9896], 80.00th=[13698], 90.00th=[17957], 95.00th=[23200], | 99.00th=[43779], 99.50th=[52167], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 288, max= 1341, per=25.53%, avg=886.58, stdev=304.09, samples=12 iops : min= 18, max= 83, avg=55.00, stdev=18.89, samples=12 write: IOPS=63, BW=1011KiB/s (1036kB/s)(6544KiB/6471msec) clat (usec): min=705, max=120039, avg=9245.39, stdev=11956.41 lat (usec): min=706, max=120040, avg=9248.56, stdev=11956.60 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 955], 10.00th=[ 1029], 20.00th=[ 1156], | 30.00th=[ 1418], 40.00th=[ 2376], 50.00th=[ 4621], 60.00th=[ 7832], | 70.00th=[ 11076], 80.00th=[ 16450], 90.00th=[ 25035], 95.00th=[ 30540], | 99.00th=[ 53216], 99.50th=[ 66847], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 541, max= 1469, per=27.11%, avg=990.50, stdev=302.99, samples=12 iops : min= 33, max= 91, avg=61.50, stdev=18.97, samples=12 lat (usec) : 100=6.90%, 250=12.11%, 500=0.65%, 750=0.39%, 1000=3.91% lat (msec) : 2=16.28%, 4=6.12%, 10=22.27%, 20=20.83%, 50=9.51% lat (msec) : 100=0.91%, 250=0.13% cpu : usr=0.19%, sys=15.26%, ctx=1014, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32423: Thu May 1 21:01:50 2025 read: IOPS=56, BW=907KiB/s (929kB/s)(6256KiB/6895msec) clat (usec): min=50, max=86002, avg=8031.82, stdev=9723.07 lat (usec): min=50, max=86004, avg=8032.93, stdev=9723.08 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 90], 10.00th=[ 99], 20.00th=[ 114], | 30.00th=[ 130], 40.00th=[ 537], 50.00th=[ 6587], 60.00th=[ 9241], | 70.00th=[11338], 80.00th=[13566], 90.00th=[18744], 95.00th=[21890], | 99.00th=[35914], 99.50th=[78119], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 447, max= 1568, per=25.71%, avg=893.08, stdev=330.73, samples=13 iops : min= 27, max= 98, avg=55.31, stdev=20.77, samples=13 write: IOPS=54, BW=875KiB/s (896kB/s)(6032KiB/6895msec) clat (usec): min=663, max=150588, avg=9577.66, stdev=13638.09 lat (usec): min=664, max=150590, avg=9579.30, stdev=13638.11 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 938], 10.00th=[ 1057], 20.00th=[ 1205], | 30.00th=[ 1483], 40.00th=[ 2311], 50.00th=[ 3490], 60.00th=[ 7504], | 70.00th=[ 11076], 80.00th=[ 15926], 90.00th=[ 23462], 95.00th=[ 33424], | 99.00th=[ 57410], 99.50th=[ 86508], 99.90th=[149947], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 480, max= 1408, per=23.12%, avg=844.69, stdev=257.97, samples=13 iops : min= 30, max= 88, avg=52.31, stdev=16.14, samples=13 lat (usec) : 100=5.47%, 250=14.32%, 500=0.52%, 750=0.65%, 1000=2.86% lat (msec) : 2=15.36%, 4=7.16%, 10=18.49%, 20=23.31%, 50=10.94% lat (msec) : 100=0.78%, 250=0.13% cpu : usr=0.19%, sys=14.43%, ctx=1039, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32424: Thu May 1 21:01:50 2025 read: IOPS=59, BW=948KiB/s (971kB/s)(6032KiB/6363msec) clat (usec): min=65, max=115362, avg=7957.86, stdev=11818.88 lat (usec): min=65, max=115363, avg=7958.88, stdev=11818.90 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 87], 10.00th=[ 95], 20.00th=[ 103], | 30.00th=[ 120], 40.00th=[ 176], 50.00th=[ 5669], 60.00th=[ 8160], | 70.00th=[ 10945], 80.00th=[ 13698], 90.00th=[ 18482], 95.00th=[ 23987], | 99.00th=[ 74974], 99.50th=[ 86508], 99.90th=[115868], 99.95th=[115868], | 99.99th=[115868] bw ( KiB/s): min= 352, max= 1824, per=27.28%, avg=947.42, stdev=443.45, samples=12 iops : min= 22, max= 114, avg=58.58, stdev=27.72, samples=12 write: IOPS=61, BW=983KiB/s (1007kB/s)(6256KiB/6363msec) clat (usec): min=627, max=108183, avg=8437.85, stdev=12870.74 lat (usec): min=629, max=108186, avg=8439.75, stdev=12870.59 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 963], 10.00th=[ 1045], 20.00th=[ 1205], | 30.00th=[ 1401], 40.00th=[ 2114], 50.00th=[ 3195], 60.00th=[ 5145], | 70.00th=[ 8717], 80.00th=[ 12911], 90.00th=[ 21890], 95.00th=[ 29492], | 99.00th=[ 82314], 99.50th=[ 96994], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 383, max= 1632, per=27.17%, avg=992.83, stdev=429.28, samples=12 iops : min= 23, max= 102, avg=61.42, stdev=26.81, samples=12 lat (usec) : 100=7.68%, 250=12.37%, 500=0.91%, 750=0.78%, 1000=3.39% lat (msec) : 2=16.28%, 4=8.46%, 10=20.18%, 20=20.44%, 50=8.20% lat (msec) : 100=1.04%, 250=0.26% cpu : usr=0.17%, sys=15.26%, ctx=982, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3474KiB/s (3557kB/s), 888KiB/s-948KiB/s (909kB/s-971kB/s), io=23.4MiB (24.5MB), run=6363-6895msec WRITE: bw=3655KiB/s (3743kB/s), 875KiB/s-1011KiB/s (896kB/s-1036kB/s), io=24.6MiB (25.8MB), run=6363-6895msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32416: Thu May 1 21:02:11 2025 read: IOPS=13, BW=216KiB/s (221kB/s)(5920KiB/27451msec) clat (usec): min=2768, max=79511, avg=8881.46, stdev=8643.70 lat (usec): min=2769, max=79512, avg=8883.24, stdev=8643.57 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3785], 10.00th=[ 4113], 20.00th=[ 4555], | 30.00th=[ 5080], 40.00th=[ 5538], 50.00th=[ 6128], 60.00th=[ 6980], | 70.00th=[ 8094], 80.00th=[ 9634], 90.00th=[15139], 95.00th=[28967], | 99.00th=[48497], 99.50th=[60556], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 31, max= 672, per=25.20%, avg=215.68, stdev=122.60, samples=53 iops : min= 1, max= 42, avg=13.08, stdev= 7.75, samples=53 write: IOPS=14, BW=232KiB/s (238kB/s)(6368KiB/27451msec) clat (msec): min=25, max=244, avg=60.27, stdev=26.99 lat (msec): min=25, max=244, avg=60.27, stdev=26.99 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 33], 10.00th=[ 36], 20.00th=[ 40], | 30.00th=[ 45], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 61], | 70.00th=[ 67], 80.00th=[ 75], 90.00th=[ 90], 95.00th=[ 117], | 99.00th=[ 163], 99.50th=[ 167], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 32, max= 352, per=25.56%, avg=230.02, stdev=72.35, samples=54 iops : min= 2, max= 22, avg=13.96, stdev= 4.54, samples=54 lat (msec) : 4=3.78%, 10=35.81%, 20=4.95%, 50=23.96%, 100=28.39% lat (msec) : 250=3.12% cpu : usr=0.03%, sys=3.11%, ctx=864, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32417: Thu May 1 21:02:11 2025 read: IOPS=12, BW=205KiB/s (210kB/s)(5744KiB/27980msec) clat (usec): min=2857, max=70272, avg=9109.02, stdev=9909.52 lat (usec): min=2859, max=70273, avg=9110.81, stdev=9909.32 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3785], 10.00th=[ 4047], 20.00th=[ 4555], | 30.00th=[ 4948], 40.00th=[ 5342], 50.00th=[ 5866], 60.00th=[ 6456], | 70.00th=[ 7439], 80.00th=[ 9372], 90.00th=[16319], 95.00th=[30278], | 99.00th=[60556], 99.50th=[62129], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 31, max= 512, per=23.81%, avg=203.80, stdev=123.07, samples=55 iops : min= 1, max= 32, avg=12.51, stdev= 7.68, samples=55 write: IOPS=14, BW=234KiB/s (239kB/s)(6544KiB/27980msec) clat (msec): min=24, max=209, avg=59.67, stdev=27.06 lat (msec): min=24, max=209, avg=59.67, stdev=27.06 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 32], 10.00th=[ 34], 20.00th=[ 40], | 30.00th=[ 44], 40.00th=[ 49], 50.00th=[ 55], 60.00th=[ 59], | 70.00th=[ 66], 80.00th=[ 77], 90.00th=[ 87], 95.00th=[ 110], | 99.00th=[ 163], 99.50th=[ 169], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 62, max= 352, per=25.57%, avg=230.09, stdev=77.60, samples=55 iops : min= 3, max= 22, avg=14.15, stdev= 4.95, samples=55 lat (msec) : 4=4.17%, 10=34.64%, 20=4.04%, 50=25.00%, 100=28.78% lat (msec) : 250=3.39% cpu : usr=0.08%, sys=2.95%, ctx=857, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32419: Thu May 1 21:02:11 2025 read: IOPS=14, BW=233KiB/s (238kB/s)(6256KiB/26895msec) clat (usec): min=3127, max=61413, avg=8735.22, stdev=8617.35 lat (usec): min=3128, max=61415, avg=8736.90, stdev=8617.24 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3523], 10.00th=[ 3884], 20.00th=[ 4490], | 30.00th=[ 4883], 40.00th=[ 5342], 50.00th=[ 5932], 60.00th=[ 6652], | 70.00th=[ 7701], 80.00th=[ 8979], 90.00th=[14746], 95.00th=[28967], | 99.00th=[46924], 99.50th=[58459], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 31, max= 512, per=27.47%, avg=235.17, stdev=133.36, samples=53 iops : min= 1, max= 32, avg=14.43, stdev= 8.41, samples=53 write: IOPS=14, BW=224KiB/s (230kB/s)(6032KiB/26895msec) clat (msec): min=19, max=238, avg=61.55, stdev=26.55 lat (msec): min=19, max=238, avg=61.55, stdev=26.55 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 33], 10.00th=[ 36], 20.00th=[ 42], | 30.00th=[ 47], 40.00th=[ 52], 50.00th=[ 57], 60.00th=[ 63], | 70.00th=[ 68], 80.00th=[ 78], 90.00th=[ 89], 95.00th=[ 104], | 99.00th=[ 163], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 62, max= 383, per=24.78%, avg=223.06, stdev=72.28, samples=53 iops : min= 3, max= 23, avg=13.68, stdev= 4.54, samples=53 lat (msec) : 4=5.99%, 10=35.94%, 20=5.08%, 50=21.22%, 100=28.65% lat (msec) : 250=3.12% cpu : usr=0.03%, sys=3.04%, ctx=859, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32420: Thu May 1 21:02:11 2025 read: IOPS=13, BW=223KiB/s (228kB/s)(6032KiB/27069msec) clat (usec): min=2837, max=93197, avg=8537.35, stdev=9490.67 lat (usec): min=2839, max=93199, avg=8539.15, stdev=9490.55 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3589], 10.00th=[ 3949], 20.00th=[ 4621], | 30.00th=[ 5080], 40.00th=[ 5473], 50.00th=[ 5932], 60.00th=[ 6456], | 70.00th=[ 7308], 80.00th=[ 8979], 90.00th=[15664], 95.00th=[22676], | 99.00th=[60031], 99.50th=[84411], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 31, max= 512, per=26.28%, avg=224.96, stdev=118.51, samples=53 iops : min= 1, max= 32, avg=13.85, stdev= 7.51, samples=53 write: IOPS=14, BW=231KiB/s (237kB/s)(6256KiB/27069msec) clat (msec): min=23, max=346, avg=60.93, stdev=28.95 lat (msec): min=23, max=346, avg=60.93, stdev=28.95 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 42], | 30.00th=[ 46], 40.00th=[ 51], 50.00th=[ 56], 60.00th=[ 61], | 70.00th=[ 67], 80.00th=[ 73], 90.00th=[ 92], 95.00th=[ 114], | 99.00th=[ 159], 99.50th=[ 209], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 62, max= 352, per=25.65%, avg=230.83, stdev=75.63, samples=54 iops : min= 3, max= 22, avg=14.22, stdev= 4.78, samples=54 lat (msec) : 4=5.34%, 10=36.07%, 20=4.56%, 50=22.79%, 100=27.21% lat (msec) : 250=3.91%, 500=0.13% cpu : usr=0.03%, sys=3.11%, ctx=849, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=856KiB/s (877kB/s), 205KiB/s-233KiB/s (210kB/s-238kB/s), io=23.4MiB (24.5MB), run=26895-27980msec WRITE: bw=901KiB/s (922kB/s), 224KiB/s-234KiB/s (230kB/s-239kB/s), io=24.6MiB (25.8MB), run=26895-27980msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32457: Thu May 1 21:02:14 2025 read: IOPS=9, BW=9873KiB/s (10.1MB/s)(7168KiB/726msec) clat (usec): min=568, max=96302, avg=57596.18, stdev=35737.35 lat (usec): min=570, max=96304, avg=57597.91, stdev=35737.37 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 570], 10.00th=[ 570], 20.00th=[16909], | 30.00th=[55837], 40.00th=[55837], 50.00th=[72877], 60.00th=[79168], | 70.00th=[79168], 80.00th=[81265], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min=10240, max=10240, per=85.81%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7052KiB/s (7222kB/s)(5120KiB/726msec) clat (usec): min=39190, max=84210, avg=59264.51, stdev=20660.14 lat (usec): min=39207, max=84361, avg=59330.97, stdev=20709.30 clat percentiles (usec): | 1.00th=[39060], 5.00th=[39060], 10.00th=[39060], 20.00th=[39060], | 30.00th=[41157], 40.00th=[41157], 50.00th=[54264], 60.00th=[54264], | 70.00th=[77071], 80.00th=[77071], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 2048, max= 2048, per=12.26%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=16.67%, 100=66.67% cpu : usr=0.00%, sys=27.17%, ctx=358, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32458: Thu May 1 21:02:14 2025 read: IOPS=1, BW=1193KiB/s (1222kB/s)(2048KiB/1716msec) clat (msec): min=106, max=192, avg=149.50, stdev=60.42 lat (msec): min=106, max=192, avg=149.50, stdev=60.42 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 192], | 70.00th=[ 192], 80.00th=[ 192], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 1753, max= 2048, per=15.93%, avg=1900.50, stdev=208.60, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5967KiB/s (6111kB/s)(10.0MiB/1716msec) clat (msec): min=43, max=288, avg=128.20, stdev=77.22 lat (msec): min=43, max=288, avg=128.27, stdev=77.22 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 53], | 30.00th=[ 62], 40.00th=[ 79], 50.00th=[ 96], 60.00th=[ 136], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 186], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 4096, max= 6144, per=30.92%, avg=5166.67, stdev=1027.19, samples=3 iops : min= 4, max= 6, avg= 5.00, stdev= 1.00, samples=3 lat (msec) : 50=8.33%, 100=33.33%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=15.86%, ctx=257, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32459: Thu May 1 21:02:14 2025 read: IOPS=4, BW=4261KiB/s (4363kB/s)(6144KiB/1442msec) clat (usec): min=552, max=215777, avg=79792.24, stdev=82187.72 lat (usec): min=554, max=215778, avg=79793.73, stdev=82187.21 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 807], | 30.00th=[ 807], 40.00th=[ 64226], 50.00th=[ 64226], 60.00th=[ 68682], | 70.00th=[128451], 80.00th=[128451], 90.00th=[214959], 95.00th=[214959], | 99.00th=[214959], 99.50th=[214959], 99.90th=[214959], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 6144, max= 6144, per=51.48%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4261KiB/s (4363kB/s)(6144KiB/1442msec) clat (msec): min=48, max=405, avg=136.65, stdev=144.10 lat (msec): min=48, max=405, avg=136.72, stdev=144.10 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 60], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 4096, max= 4096, per=24.52%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 50=8.33%, 100=41.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=15.75%, ctx=239, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32460: Thu May 1 21:02:14 2025 read: IOPS=3, BW=3172KiB/s (3248kB/s)(5120KiB/1614msec) clat (usec): min=367, max=118931, avg=38130.03, stdev=51511.27 lat (usec): min=368, max=118932, avg=38131.18, stdev=51511.50 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 367], | 30.00th=[ 816], 40.00th=[ 816], 50.00th=[ 10290], 60.00th=[ 10290], | 70.00th=[ 60031], 80.00th=[ 60031], 90.00th=[119014], 95.00th=[119014], | 99.00th=[119014], 99.50th=[119014], 99.90th=[119014], 99.95th=[119014], | 99.99th=[119014] bw ( KiB/s): min= 8192, max= 8192, per=68.64%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4441KiB/s (4548kB/s)(7168KiB/1614msec) clat (msec): min=91, max=398, avg=194.11, stdev=107.63 lat (msec): min=91, max=398, avg=194.17, stdev=107.62 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 115], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 148], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 264], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 2039, max= 6144, per=24.50%, avg=4093.00, stdev=2052.50, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 20=8.33%, 100=16.67%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=13.76%, ctx=462, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.7MiB/s (12.2MB/s), 1193KiB/s-9873KiB/s (1222kB/s-10.1MB/s), io=20.0MiB (20.0MB), run=726-1716msec WRITE: bw=16.3MiB/s (17.1MB/s), 4261KiB/s-7052KiB/s (4363kB/s-7222kB/s), io=28.0MiB (29.4MB), run=726-1716msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32453: Thu May 1 21:02:14 2025 read: IOPS=4, BW=5027KiB/s (5147kB/s)(7168KiB/1426msec) clat (msec): min=19, max=271, avg=80.69, stdev=87.35 lat (msec): min=19, max=271, avg=80.70, stdev=87.35 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 27], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 52], 60.00th=[ 71], | 70.00th=[ 71], 80.00th=[ 88], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 4079, max= 6119, per=44.47%, avg=5099.00, stdev=1442.50, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3590KiB/s (3677kB/s)(5120KiB/1426msec) clat (msec): min=90, max=224, avg=152.28, stdev=51.51 lat (msec): min=90, max=225, avg=152.57, stdev=51.85 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 4079, max= 4079, per=25.41%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=16.67%, 100=33.33%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=4.84%, ctx=34, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32454: Thu May 1 21:02:14 2025 read: IOPS=1, BW=1223KiB/s (1252kB/s)(2048KiB/1675msec) clat (usec): min=45990, max=78703, avg=62347.15, stdev=23131.11 lat (usec): min=45993, max=78705, avg=62349.43, stdev=23130.51 clat percentiles (usec): | 1.00th=[45876], 5.00th=[45876], 10.00th=[45876], 20.00th=[45876], | 30.00th=[45876], 40.00th=[45876], 50.00th=[45876], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 2031, max= 2048, per=17.79%, avg=2039.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=6113KiB/s (6260kB/s)(10.0MiB/1675msec) clat (msec): min=60, max=266, avg=143.17, stdev=71.60 lat (msec): min=60, max=266, avg=143.27, stdev=71.66 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 64], | 30.00th=[ 88], 40.00th=[ 90], 50.00th=[ 116], 60.00th=[ 150], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 4096, max= 6119, per=33.87%, avg=5436.67, stdev=1161.11, samples=3 iops : min= 4, max= 5, avg= 4.67, stdev= 0.58, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=3.41%, ctx=32, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32455: Thu May 1 21:02:14 2025 read: IOPS=3, BW=3440KiB/s (3523kB/s)(6144KiB/1786msec) clat (msec): min=43, max=296, avg=112.58, stdev=99.78 lat (msec): min=43, max=296, avg=112.58, stdev=99.78 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 72], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 4071, max= 4096, per=35.61%, avg=4083.50, stdev=17.68, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3440KiB/s (3523kB/s)(6144KiB/1786msec) clat (msec): min=73, max=348, avg=181.72, stdev=107.82 lat (msec): min=73, max=348, avg=181.81, stdev=107.82 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 232], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 2035, max= 4096, per=21.13%, avg=3392.67, stdev=1176.03, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=33.33%, 500=16.67% cpu : usr=0.00%, sys=2.86%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32456: Thu May 1 21:02:14 2025 read: IOPS=2, BW=2922KiB/s (2993kB/s)(5120KiB/1752msec) clat (msec): min=28, max=225, avg=84.53, stdev=80.79 lat (msec): min=28, max=225, avg=84.53, stdev=80.79 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 8159, max= 8159, per=71.16%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4091KiB/s (4190kB/s)(7168KiB/1752msec) clat (msec): min=86, max=265, avg=186.03, stdev=74.17 lat (msec): min=86, max=265, avg=186.10, stdev=74.18 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 123], | 30.00th=[ 128], 40.00th=[ 128], 50.00th=[ 192], 60.00th=[ 251], | 70.00th=[ 251], 80.00th=[ 257], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 4079, max= 4096, per=25.46%, avg=4087.33, stdev= 8.50, samples=3 iops : min= 3, max= 4, avg= 3.33, stdev= 0.58, samples=3 lat (msec) : 50=16.67%, 100=25.00%, 250=41.67%, 500=16.67% cpu : usr=0.17%, sys=4.00%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.2MiB/s (11.7MB/s), 1223KiB/s-5027KiB/s (1252kB/s-5147kB/s), io=20.0MiB (20.0MB), run=1426-1786msec WRITE: bw=15.7MiB/s (16.4MB/s), 3440KiB/s-6113KiB/s (3523kB/s-6260kB/s), io=28.0MiB (29.4MB), run=1426-1786msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32467: Thu May 1 21:02:17 2025 read: IOPS=3, BW=15.6MiB/s (16.3MB/s)(12.0MiB/771msec) clat (msec): min=62, max=383, avg=220.34, stdev=160.83 lat (msec): min=62, max=383, avg=220.34, stdev=160.83 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 8175, max= 8175, per=51.69%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.81%, ctx=479, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32469: Thu May 1 21:02:17 2025 read: IOPS=0, BW=3954KiB/s (4049kB/s)(4096KiB/1036msec) clat (nsec): min=179394k, max=179394k, avg=179394307.00, stdev= 0.00 lat (nsec): min=179397k, max=179397k, avg=179396650.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 8015, max= 8015, per=50.68%, avg=8015.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7907KiB/s (8097kB/s)(8192KiB/1036msec) clat (msec): min=372, max=389, avg=380.93, stdev=12.26 lat (msec): min=377, max=389, avg=383.87, stdev= 8.45 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 388], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8192, max= 8192, per=28.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=20.68%, ctx=292, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32471: Thu May 1 21:02:17 2025 write: IOPS=2, BW=11.6MiB/s (12.2MB/s)(12.0MiB/1033msec) clat (msec): min=182, max=372, avg=302.66, stdev=104.88 lat (msec): min=182, max=372, avg=303.00, stdev=105.05 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 8175, max= 8192, per=28.42%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=20.83%, ctx=84, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32472: Thu May 1 21:02:17 2025 write: IOPS=2, BW=10.5MiB/s (11.1MB/s)(12.0MiB/1138msec) clat (msec): min=289, max=388, avg=338.48, stdev=49.86 lat (msec): min=291, max=389, avg=339.34, stdev=49.05 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8192, max= 8192, per=28.45%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=25.07%, ctx=85, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.4MiB/s (16.2MB/s), 3954KiB/s-15.6MiB/s (4049kB/s-16.3MB/s), io=16.0MiB (16.8MB), run=771-1036msec WRITE: bw=28.1MiB/s (29.5MB/s), 7907KiB/s-11.6MiB/s (8097kB/s-12.2MB/s), io=32.0MiB (33.6MB), run=1033-1138msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32468: Thu May 1 21:02:17 2025 read: IOPS=2, BW=10.5MiB/s (11.0MB/s)(12.0MiB/1140msec) clat (msec): min=256, max=535, avg=354.27, stdev=156.82 lat (msec): min=256, max=535, avg=354.27, stdev=156.82 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.09%, sys=1.76%, ctx=21, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32470: Thu May 1 21:02:17 2025 read: IOPS=0, BW=2981KiB/s (3053kB/s)(4096KiB/1374msec) clat (nsec): min=678195k, max=678195k, avg=678195255.00, stdev= 0.00 lat (nsec): min=678198k, max=678198k, avg=678198326.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 6206, max= 6206, per=52.05%, avg=6206.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5962KiB/s (6105kB/s)(8192KiB/1374msec) clat (msec): min=188, max=373, avg=281.02, stdev=130.20 lat (msec): min=189, max=373, avg=281.32, stdev=130.22 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.29%, sys=3.79%, ctx=29, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32473: Thu May 1 21:02:17 2025 write: IOPS=2, BW=9274KiB/s (9497kB/s)(12.0MiB/1325msec) clat (msec): min=124, max=657, avg=415.35, stdev=269.81 lat (msec): min=124, max=658, avg=415.87, stdev=270.13 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 6627, max= 6627, per=27.79%, avg=6627.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.30%, sys=3.02%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32474: Thu May 1 21:02:17 2025 write: IOPS=3, BW=12.6MiB/s (13.2MB/s)(12.0MiB/955msec) clat (msec): min=125, max=545, avg=285.88, stdev=226.52 lat (msec): min=126, max=545, avg=286.17, stdev=226.51 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min=16125, max=16125, per=67.62%, avg=16125.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.42%, sys=3.98%, ctx=22, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.6MiB/s (12.2MB/s), 2981KiB/s-10.5MiB/s (3053kB/s-11.0MB/s), io=16.0MiB (16.8MB), run=1140-1374msec WRITE: bw=23.3MiB/s (24.4MB/s), 5962KiB/s-12.6MiB/s (6105kB/s-13.2MB/s), io=32.0MiB (33.6MB), run=955-1374msec