== sanity test 398b: DIO and buffer IO race ============== 07:33:01 (1749123181) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.41614 s, 35.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2619: Thu Jun 5 07:33:23 2025 read: IOPS=77, BW=308KiB/s (315kB/s)(5992KiB/19454msec) clat (usec): min=51, max=233830, avg=6164.62, stdev=7497.86 lat (usec): min=51, max=233835, avg=6166.71, stdev=7498.50 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 103], 10.00th=[ 119], 20.00th=[ 2999], | 30.00th=[ 3720], 40.00th=[ 4424], 50.00th=[ 5211], 60.00th=[ 6128], | 70.00th=[ 7439], 80.00th=[ 9110], 90.00th=[ 11600], 95.00th=[ 14222], | 99.00th=[ 20579], 99.50th=[ 25560], 99.90th=[ 51119], 99.95th=[233833], | 99.99th=[233833] bw ( KiB/s): min= 184, max= 463, per=24.38%, avg=300.89, stdev=72.14, samples=38 iops : min= 46, max= 115, avg=75.11, stdev=18.06, samples=38 write: IOPS=80, BW=324KiB/s (331kB/s)(6296KiB/19454msec) clat (usec): min=461, max=74863, avg=6411.74, stdev=6722.21 lat (usec): min=461, max=74863, avg=6412.93, stdev=6722.13 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 693], 10.00th=[ 750], 20.00th=[ 873], | 30.00th=[ 1188], 40.00th=[ 2343], 50.00th=[ 4817], 60.00th=[ 6652], | 70.00th=[ 8356], 80.00th=[10552], 90.00th=[14746], 95.00th=[18482], | 99.00th=[30016], 99.50th=[36439], 99.90th=[51643], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 223, max= 552, per=24.35%, avg=314.58, stdev=72.97, samples=38 iops : min= 55, max= 138, avg=78.53, stdev=18.26, samples=38 lat (usec) : 100=2.12%, 250=4.79%, 500=0.59%, 750=5.18%, 1000=8.14% lat (msec) : 2=6.45%, 4=12.63%, 10=41.24%, 20=16.18%, 50=2.57% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.27%, sys=19.73%, ctx=5655, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2620: Thu Jun 5 07:33:23 2025 read: IOPS=77, BW=310KiB/s (317kB/s)(5940KiB/19168msec) clat (usec): min=51, max=42431, avg=5556.54, stdev=4444.99 lat (usec): min=51, max=42431, avg=5557.81, stdev=4444.93 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 98], 10.00th=[ 115], 20.00th=[ 2638], | 30.00th=[ 3490], 40.00th=[ 4178], 50.00th=[ 4817], 60.00th=[ 5604], | 70.00th=[ 6849], 80.00th=[ 8586], 90.00th=[10945], 95.00th=[13304], | 99.00th=[19006], 99.50th=[20579], 99.90th=[35390], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 168, max= 432, per=25.08%, avg=309.53, stdev=65.65, samples=38 iops : min= 42, max= 108, avg=77.26, stdev=16.47, samples=38 write: IOPS=82, BW=331KiB/s (339kB/s)(6348KiB/19168msec) clat (usec): min=459, max=237544, avg=6808.07, stdev=8779.62 lat (usec): min=460, max=237546, avg=6809.88, stdev=8779.30 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 881], | 30.00th=[ 1319], 40.00th=[ 3195], 50.00th=[ 5276], 60.00th=[ 6915], | 70.00th=[ 8717], 80.00th=[ 11076], 90.00th=[ 15008], 95.00th=[ 18482], | 99.00th=[ 27395], 99.50th=[ 38011], 99.90th=[ 66847], 99.95th=[238027], | 99.99th=[238027] bw ( KiB/s): min= 224, max= 536, per=25.49%, avg=329.29, stdev=73.90, samples=38 iops : min= 56, max= 134, avg=82.21, stdev=18.48, samples=38 lat (usec) : 100=2.93%, 250=5.66%, 500=0.39%, 750=4.17%, 1000=8.66% lat (msec) : 2=5.37%, 4=12.60%, 10=41.15%, 20=16.63%, 50=2.31% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.25%, sys=20.01%, ctx=5257, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2621: Thu Jun 5 07:33:23 2025 read: IOPS=78, BW=313KiB/s (320kB/s)(6028KiB/19288msec) clat (usec): min=51, max=62335, avg=5717.86, stdev=4734.83 lat (usec): min=51, max=62336, avg=5718.94, stdev=4734.80 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 95], 10.00th=[ 111], 20.00th=[ 2606], | 30.00th=[ 3556], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 5800], | 70.00th=[ 6980], 80.00th=[ 8848], 90.00th=[11469], 95.00th=[13829], | 99.00th=[19006], 99.50th=[27395], 99.90th=[45876], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 104, max= 464, per=25.05%, avg=309.08, stdev=89.10, samples=38 iops : min= 26, max= 116, avg=77.13, stdev=22.26, samples=38 write: IOPS=81, BW=325KiB/s (332kB/s)(6260KiB/19288msec) clat (usec): min=471, max=245962, avg=6740.13, stdev=8944.47 lat (usec): min=472, max=245965, avg=6741.81, stdev=8944.66 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 693], 10.00th=[ 758], 20.00th=[ 873], | 30.00th=[ 1221], 40.00th=[ 3163], 50.00th=[ 5276], 60.00th=[ 6783], | 70.00th=[ 8717], 80.00th=[ 11076], 90.00th=[ 14746], 95.00th=[ 18744], | 99.00th=[ 28443], 99.50th=[ 31851], 99.90th=[ 66847], 99.95th=[246416], | 99.99th=[246416] bw ( KiB/s): min= 120, max= 518, per=24.78%, avg=320.21, stdev=77.59, samples=38 iops : min= 30, max= 129, avg=79.92, stdev=19.41, samples=38 lat (usec) : 100=3.19%, 250=5.37%, 500=0.39%, 750=4.62%, 1000=8.04% lat (msec) : 2=5.27%, 4=12.83%, 10=40.79%, 20=17.06%, 50=2.31% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.23%, sys=19.71%, ctx=5378, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2622: Thu Jun 5 07:33:23 2025 read: IOPS=79, BW=317KiB/s (324kB/s)(6052KiB/19113msec) clat (usec): min=50, max=47936, avg=5687.66, stdev=4569.84 lat (usec): min=50, max=47936, avg=5689.31, stdev=4571.25 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 97], 10.00th=[ 113], 20.00th=[ 2704], | 30.00th=[ 3556], 40.00th=[ 4228], 50.00th=[ 4883], 60.00th=[ 5800], | 70.00th=[ 6980], 80.00th=[ 8717], 90.00th=[11207], 95.00th=[13698], | 99.00th=[19006], 99.50th=[25035], 99.90th=[40633], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 56, max= 558, per=25.56%, avg=315.45, stdev=80.54, samples=38 iops : min= 14, max= 139, avg=78.63, stdev=20.05, samples=38 write: IOPS=81, BW=326KiB/s (334kB/s)(6236KiB/19113msec) clat (usec): min=455, max=240861, avg=6653.61, stdev=8996.83 lat (usec): min=456, max=240865, avg=6655.10, stdev=8996.62 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 693], 10.00th=[ 750], 20.00th=[ 865], | 30.00th=[ 1270], 40.00th=[ 2835], 50.00th=[ 4883], 60.00th=[ 6652], | 70.00th=[ 8455], 80.00th=[ 10945], 90.00th=[ 14877], 95.00th=[ 18482], | 99.00th=[ 31589], 99.50th=[ 36963], 99.90th=[ 71828], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 64, max= 502, per=25.18%, avg=325.34, stdev=79.72, samples=38 iops : min= 16, max= 125, avg=81.11, stdev=19.91, samples=38 lat (usec) : 100=3.32%, 250=5.08%, 500=0.39%, 750=5.08%, 1000=7.75% lat (msec) : 2=5.99%, 4=12.70%, 10=40.98%, 20=16.34%, 50=2.25% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.29%, sys=19.55%, ctx=5275, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1234KiB/s (1264kB/s), 308KiB/s-317KiB/s (315kB/s-324kB/s), io=23.4MiB (24.6MB), run=19113-19454msec WRITE: bw=1292KiB/s (1323kB/s), 324KiB/s-331KiB/s (331kB/s-339kB/s), io=24.6MiB (25.7MB), run=19113-19454msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2615: Thu Jun 5 07:33:54 2025 read: IOPS=29, BW=118KiB/s (121kB/s)(5992KiB/50824msec) clat (usec): min=1542, max=241810, avg=6019.36, stdev=7737.13 lat (usec): min=1543, max=241811, avg=6020.64, stdev=7737.11 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2343], 10.00th=[ 2606], 20.00th=[ 3032], | 30.00th=[ 3589], 40.00th=[ 4228], 50.00th=[ 4752], 60.00th=[ 5145], | 70.00th=[ 5866], 80.00th=[ 6849], 90.00th=[ 10683], 95.00th=[ 14877], | 99.00th=[ 24773], 99.50th=[ 30016], 99.90th=[ 65274], 99.95th=[242222], | 99.99th=[242222] bw ( KiB/s): min= 23, max= 231, per=24.98%, avg=117.41, stdev=56.79, samples=101 iops : min= 5, max= 57, avg=29.25, stdev=14.23, samples=101 write: IOPS=30, BW=124KiB/s (127kB/s)(6296KiB/50824msec) clat (msec): min=10, max=103, avg=26.48, stdev=13.43 lat (msec): min=10, max=103, avg=26.48, stdev=13.43 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 29], 80.00th=[ 35], 90.00th=[ 46], 95.00th=[ 55], | 99.00th=[ 74], 99.50th=[ 79], 99.90th=[ 97], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 32, max= 200, per=25.04%, avg=123.19, stdev=45.03, samples=101 iops : min= 8, max= 50, avg=30.69, stdev=11.31, samples=101 lat (msec) : 2=0.65%, 4=17.38%, 10=25.26%, 20=24.09%, 50=28.91% lat (msec) : 100=3.65%, 250=0.07% cpu : usr=0.16%, sys=4.99%, ctx=3903, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2616: Thu Jun 5 07:33:54 2025 read: IOPS=29, BW=116KiB/s (119kB/s)(5940KiB/51002msec) clat (usec): min=1651, max=254854, avg=6135.09, stdev=8161.66 lat (usec): min=1652, max=254855, avg=6136.42, stdev=8161.65 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2376], 10.00th=[ 2606], 20.00th=[ 3097], | 30.00th=[ 3720], 40.00th=[ 4293], 50.00th=[ 4883], 60.00th=[ 5407], | 70.00th=[ 5932], 80.00th=[ 6783], 90.00th=[ 10421], 95.00th=[ 14615], | 99.00th=[ 25560], 99.50th=[ 34341], 99.90th=[ 62653], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 8, max= 287, per=24.65%, avg=115.84, stdev=60.08, samples=101 iops : min= 2, max= 71, avg=28.87, stdev=15.02, samples=101 write: IOPS=31, BW=124KiB/s (127kB/s)(6348KiB/51002msec) clat (msec): min=9, max=106, avg=26.31, stdev=13.26 lat (msec): min=9, max=106, avg=26.32, stdev=13.26 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 29], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 54], | 99.00th=[ 72], 99.50th=[ 82], 99.90th=[ 100], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 32, max= 192, per=25.03%, avg=123.13, stdev=45.13, samples=101 iops : min= 8, max= 48, avg=30.69, stdev=11.28, samples=101 lat (msec) : 2=0.52%, 4=16.18%, 10=26.46%, 20=24.90%, 50=27.90% lat (msec) : 100=3.97%, 250=0.03%, 500=0.03% cpu : usr=0.14%, sys=5.07%, ctx=3932, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2617: Thu Jun 5 07:33:54 2025 read: IOPS=29, BW=119KiB/s (122kB/s)(6028KiB/50610msec) clat (usec): min=1582, max=74370, avg=6100.25, stdev=5420.33 lat (usec): min=1583, max=74371, avg=6101.54, stdev=5420.30 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2409], 10.00th=[ 2638], 20.00th=[ 3130], | 30.00th=[ 3720], 40.00th=[ 4228], 50.00th=[ 4817], 60.00th=[ 5342], | 70.00th=[ 5932], 80.00th=[ 6915], 90.00th=[10552], 95.00th=[15270], | 99.00th=[28705], 99.50th=[39584], 99.90th=[53740], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 8, max= 248, per=25.29%, avg=118.85, stdev=57.11, samples=101 iops : min= 2, max= 62, avg=29.62, stdev=14.26, samples=101 write: IOPS=30, BW=124KiB/s (127kB/s)(6260KiB/50610msec) clat (msec): min=9, max=257, avg=26.38, stdev=14.28 lat (msec): min=9, max=257, avg=26.38, stdev=14.28 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 29], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 55], | 99.00th=[ 71], 99.50th=[ 78], 99.90th=[ 104], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 32, max= 200, per=25.07%, avg=123.37, stdev=46.00, samples=101 iops : min= 8, max= 50, avg=30.75, stdev=11.51, samples=101 lat (msec) : 2=0.49%, 4=16.63%, 10=26.76%, 20=24.35%, 50=28.06% lat (msec) : 100=3.65%, 250=0.03%, 500=0.03% cpu : usr=0.15%, sys=5.10%, ctx=3941, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2618: Thu Jun 5 07:33:54 2025 read: IOPS=29, BW=119KiB/s (122kB/s)(6052KiB/50767msec) clat (usec): min=1714, max=74825, avg=5987.31, stdev=5179.11 lat (usec): min=1715, max=74827, avg=5989.88, stdev=5180.66 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2376], 10.00th=[ 2606], 20.00th=[ 3097], | 30.00th=[ 3589], 40.00th=[ 4228], 50.00th=[ 4817], 60.00th=[ 5342], | 70.00th=[ 5997], 80.00th=[ 6915], 90.00th=[10552], 95.00th=[14615], | 99.00th=[23987], 99.50th=[34866], 99.90th=[65274], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 16, max= 272, per=25.24%, avg=118.64, stdev=54.67, samples=101 iops : min= 4, max= 68, avg=29.59, stdev=13.65, samples=101 write: IOPS=30, BW=123KiB/s (126kB/s)(6236KiB/50767msec) clat (msec): min=9, max=257, avg=26.67, stdev=14.34 lat (msec): min=9, max=257, avg=26.67, stdev=14.34 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 55], | 99.00th=[ 75], 99.50th=[ 84], 99.90th=[ 97], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 32, max= 192, per=24.87%, avg=122.37, stdev=43.59, samples=101 iops : min= 8, max= 48, avg=30.52, stdev=10.93, samples=101 lat (msec) : 2=0.49%, 4=17.12%, 10=26.43%, 20=23.47%, 50=29.04% lat (msec) : 100=3.42%, 500=0.03% cpu : usr=0.16%, sys=5.03%, ctx=3918, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=471KiB/s (482kB/s), 116KiB/s-119KiB/s (119kB/s-122kB/s), io=23.4MiB (24.6MB), run=50610-51002msec WRITE: bw=493KiB/s (505kB/s), 123KiB/s-124KiB/s (126kB/s-127kB/s), io=24.6MiB (25.7MB), run=50610-51002msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2675: Thu Jun 5 07:33:59 2025 read: IOPS=112, BW=1807KiB/s (1850kB/s)(5920KiB/3277msec) clat (usec): min=61, max=23019, avg=4000.02, stdev=3537.44 lat (usec): min=62, max=23019, avg=4000.83, stdev=3537.48 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 88], 10.00th=[ 101], 20.00th=[ 125], | 30.00th=[ 1663], 40.00th=[ 3228], 50.00th=[ 3687], 60.00th=[ 4490], | 70.00th=[ 5276], 80.00th=[ 6587], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[16581], 99.50th=[21890], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 1088, max= 2272, per=24.37%, avg=1781.33, stdev=455.11, samples=6 iops : min= 68, max= 142, avg=111.33, stdev=28.44, samples=6 write: IOPS=121, BW=1943KiB/s (1990kB/s)(6368KiB/3277msec) clat (usec): min=597, max=30257, avg=4286.65, stdev=4533.16 lat (usec): min=598, max=30258, avg=4288.05, stdev=4533.07 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 848], 10.00th=[ 938], 20.00th=[ 1106], | 30.00th=[ 1270], 40.00th=[ 1450], 50.00th=[ 2180], 60.00th=[ 3884], | 70.00th=[ 5145], 80.00th=[ 7046], 90.00th=[10028], 95.00th=[13698], | 99.00th=[23462], 99.50th=[27132], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 1408, max= 2528, per=25.25%, avg=1941.33, stdev=471.90, samples=6 iops : min= 88, max= 158, avg=121.33, stdev=29.49, samples=6 lat (usec) : 100=4.43%, 250=8.98%, 500=0.65%, 750=1.04%, 1000=6.77% lat (msec) : 2=17.45%, 4=17.97%, 10=34.77%, 20=7.16%, 50=0.78% cpu : usr=0.37%, sys=26.53%, ctx=1599, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2677: Thu Jun 5 07:33:59 2025 read: IOPS=117, BW=1873KiB/s (1918kB/s)(5744KiB/3066msec) clat (usec): min=65, max=17870, avg=4067.46, stdev=2848.83 lat (usec): min=65, max=17871, avg=4068.73, stdev=2849.11 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 97], 10.00th=[ 108], 20.00th=[ 190], | 30.00th=[ 3163], 40.00th=[ 3589], 50.00th=[ 4015], 60.00th=[ 4555], | 70.00th=[ 5342], 80.00th=[ 6128], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[13042], 99.50th=[14091], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1564, max= 2432, per=25.68%, avg=1876.67, stdev=308.85, samples=6 iops : min= 97, max= 152, avg=117.17, stdev=19.46, samples=6 write: IOPS=133, BW=2134KiB/s (2186kB/s)(6544KiB/3066msec) clat (usec): min=641, max=24529, avg=3853.88, stdev=3882.75 lat (usec): min=641, max=24530, avg=3856.02, stdev=3884.41 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 848], 10.00th=[ 938], 20.00th=[ 1139], | 30.00th=[ 1254], 40.00th=[ 1418], 50.00th=[ 1729], 60.00th=[ 3195], | 70.00th=[ 4621], 80.00th=[ 6980], 90.00th=[ 9765], 95.00th=[11863], | 99.00th=[15795], 99.50th=[18482], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 1852, max= 2432, per=27.74%, avg=2132.67, stdev=214.91, samples=6 iops : min= 115, max= 152, avg=133.17, stdev=13.63, samples=6 lat (usec) : 100=2.99%, 250=6.90%, 500=0.52%, 750=1.30%, 1000=5.73% lat (msec) : 2=21.22%, 4=19.79%, 10=35.55%, 20=5.86%, 50=0.13% cpu : usr=0.16%, sys=28.74%, ctx=1631, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2679: Thu Jun 5 07:33:59 2025 read: IOPS=122, BW=1954KiB/s (2001kB/s)(6256KiB/3202msec) clat (usec): min=60, max=15408, avg=3864.12, stdev=3214.73 lat (usec): min=60, max=15410, avg=3865.02, stdev=3214.79 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 116], | 30.00th=[ 188], 40.00th=[ 3130], 50.00th=[ 3752], 60.00th=[ 4621], | 70.00th=[ 5604], 80.00th=[ 6783], 90.00th=[ 8225], 95.00th=[ 9503], | 99.00th=[11338], 99.50th=[13566], 99.90th=[15401], 99.95th=[15401], | 99.99th=[15401] bw ( KiB/s): min= 1280, max= 2459, per=26.48%, avg=1935.17, stdev=407.89, samples=6 iops : min= 80, max= 153, avg=120.83, stdev=25.32, samples=6 write: IOPS=117, BW=1884KiB/s (1929kB/s)(6032KiB/3202msec) clat (usec): min=582, max=38078, avg=4246.40, stdev=4240.98 lat (usec): min=583, max=38079, avg=4247.98, stdev=4240.73 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 857], 10.00th=[ 988], 20.00th=[ 1123], | 30.00th=[ 1319], 40.00th=[ 1631], 50.00th=[ 2507], 60.00th=[ 3982], | 70.00th=[ 5211], 80.00th=[ 7373], 90.00th=[ 9765], 95.00th=[11600], | 99.00th=[19530], 99.50th=[21365], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 1376, max= 2272, per=23.78%, avg=1828.67, stdev=314.07, samples=6 iops : min= 86, max= 142, avg=114.17, stdev=19.56, samples=6 lat (usec) : 100=5.21%, 250=10.68%, 500=0.26%, 750=0.91%, 1000=4.69% lat (msec) : 2=17.19%, 4=17.84%, 10=37.24%, 20=5.60%, 50=0.39% cpu : usr=0.44%, sys=26.65%, ctx=1577, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2681: Thu Jun 5 07:33:59 2025 read: IOPS=115, BW=1851KiB/s (1896kB/s)(6032KiB/3258msec) clat (usec): min=53, max=19146, avg=3888.49, stdev=3300.29 lat (usec): min=54, max=19146, avg=3889.76, stdev=3301.04 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 86], 10.00th=[ 100], 20.00th=[ 123], | 30.00th=[ 212], 40.00th=[ 3163], 50.00th=[ 3752], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 6783], 90.00th=[ 8455], 95.00th=[ 9372], | 99.00th=[12518], 99.50th=[12911], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 608, max= 2203, per=24.07%, avg=1759.17, stdev=596.89, samples=6 iops : min= 38, max= 137, avg=109.83, stdev=37.20, samples=6 write: IOPS=120, BW=1920KiB/s (1966kB/s)(6256KiB/3258msec) clat (usec): min=662, max=37597, avg=4390.56, stdev=5169.77 lat (usec): min=663, max=37601, avg=4392.00, stdev=5169.75 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 922], 10.00th=[ 996], 20.00th=[ 1090], | 30.00th=[ 1254], 40.00th=[ 1516], 50.00th=[ 2114], 60.00th=[ 3687], | 70.00th=[ 4948], 80.00th=[ 7308], 90.00th=[ 9765], 95.00th=[12518], | 99.00th=[28705], 99.50th=[36963], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 704, max= 2432, per=23.99%, avg=1844.50, stdev=672.45, samples=6 iops : min= 44, max= 152, avg=115.17, stdev=41.92, samples=6 lat (usec) : 100=5.08%, 250=9.90%, 500=0.52%, 750=0.65%, 1000=5.08% lat (msec) : 2=19.01%, 4=18.23%, 10=35.16%, 20=5.21%, 50=1.17% cpu : usr=0.15%, sys=26.25%, ctx=1549, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7309KiB/s (7485kB/s), 1807KiB/s-1954KiB/s (1850kB/s-2001kB/s), io=23.4MiB (24.5MB), run=3066-3277msec WRITE: bw=7690KiB/s (7875kB/s), 1884KiB/s-2134KiB/s (1929kB/s-2186kB/s), io=24.6MiB (25.8MB), run=3066-3277msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2674: Thu Jun 5 07:34:07 2025 read: IOPS=30, BW=494KiB/s (506kB/s)(5920KiB/11980msec) clat (usec): min=2346, max=44245, avg=5336.99, stdev=4610.24 lat (usec): min=2347, max=44245, avg=5338.33, stdev=4610.15 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2704], 10.00th=[ 2835], 20.00th=[ 3097], | 30.00th=[ 3261], 40.00th=[ 3556], 50.00th=[ 3851], 60.00th=[ 4113], | 70.00th=[ 4555], 80.00th=[ 5538], 90.00th=[ 9634], 95.00th=[14484], | 99.00th=[27132], 99.50th=[28967], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 256, max= 1024, per=24.49%, avg=489.65, stdev=174.29, samples=23 iops : min= 16, max= 64, avg=30.57, stdev=10.88, samples=23 write: IOPS=33, BW=532KiB/s (544kB/s)(6368KiB/11980msec) clat (usec): min=11804, max=64619, avg=25025.11, stdev=8471.26 lat (usec): min=11805, max=64622, avg=25027.68, stdev=8472.41 clat percentiles (usec): | 1.00th=[13304], 5.00th=[14353], 10.00th=[15401], 20.00th=[17957], | 30.00th=[20055], 40.00th=[21627], 50.00th=[23462], 60.00th=[25560], | 70.00th=[27657], 80.00th=[30540], 90.00th=[35914], 95.00th=[41681], | 99.00th=[56886], 99.50th=[62653], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 352, max= 736, per=25.40%, avg=534.22, stdev=111.92, samples=23 iops : min= 22, max= 46, avg=33.35, stdev= 7.04, samples=23 lat (msec) : 4=26.43%, 10=17.32%, 20=18.62%, 50=36.85%, 100=0.78% cpu : usr=0.19%, sys=6.00%, ctx=880, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2676: Thu Jun 5 07:34:07 2025 read: IOPS=30, BW=480KiB/s (492kB/s)(5744KiB/11959msec) clat (usec): min=2164, max=28580, avg=5232.57, stdev=4127.55 lat (usec): min=2165, max=28581, avg=5233.79, stdev=4127.48 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2638], 10.00th=[ 2835], 20.00th=[ 3064], | 30.00th=[ 3261], 40.00th=[ 3490], 50.00th=[ 3720], 60.00th=[ 4047], | 70.00th=[ 4686], 80.00th=[ 6456], 90.00th=[ 9110], 95.00th=[14091], | 99.00th=[25560], 99.50th=[27132], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 128, max= 992, per=24.26%, avg=485.04, stdev=203.06, samples=23 iops : min= 8, max= 62, avg=30.09, stdev=12.72, samples=23 write: IOPS=34, BW=547KiB/s (560kB/s)(6544KiB/11959msec) clat (usec): min=11911, max=58233, avg=24549.58, stdev=8479.12 lat (usec): min=11913, max=58236, avg=24551.89, stdev=8479.40 clat percentiles (usec): | 1.00th=[12649], 5.00th=[14222], 10.00th=[15270], 20.00th=[17171], | 30.00th=[19006], 40.00th=[21103], 50.00th=[22938], 60.00th=[25035], | 70.00th=[27395], 80.00th=[30802], 90.00th=[35914], 95.00th=[41681], | 99.00th=[51119], 99.50th=[52691], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 271, max= 736, per=26.03%, avg=547.48, stdev=118.53, samples=23 iops : min= 16, max= 46, avg=34.00, stdev= 7.50, samples=23 lat (msec) : 4=27.73%, 10=15.10%, 20=22.40%, 50=33.98%, 100=0.78% cpu : usr=0.10%, sys=5.86%, ctx=882, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2678: Thu Jun 5 07:34:07 2025 read: IOPS=35, BW=576KiB/s (590kB/s)(6256KiB/10866msec) clat (usec): min=2099, max=29396, avg=4885.18, stdev=3537.61 lat (usec): min=2100, max=29398, avg=4886.47, stdev=3537.60 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2737], 10.00th=[ 2933], 20.00th=[ 3130], | 30.00th=[ 3326], 40.00th=[ 3523], 50.00th=[ 3720], 60.00th=[ 4047], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 8356], 95.00th=[11469], | 99.00th=[24511], 99.50th=[26870], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 319, max= 1149, per=29.33%, avg=586.29, stdev=210.31, samples=21 iops : min= 19, max= 71, avg=36.48, stdev=13.10, samples=21 write: IOPS=34, BW=555KiB/s (568kB/s)(6032KiB/10866msec) clat (usec): min=11300, max=53255, avg=23704.53, stdev=7540.45 lat (usec): min=11302, max=53257, avg=23706.73, stdev=7540.48 clat percentiles (usec): | 1.00th=[12911], 5.00th=[14222], 10.00th=[15270], 20.00th=[16909], | 30.00th=[18744], 40.00th=[20841], 50.00th=[22414], 60.00th=[24249], | 70.00th=[26608], 80.00th=[29492], 90.00th=[33817], 95.00th=[37487], | 99.00th=[48497], 99.50th=[52167], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 384, max= 736, per=26.50%, avg=557.33, stdev=101.44, samples=21 iops : min= 24, max= 46, avg=34.67, stdev= 6.34, samples=21 lat (msec) : 4=29.95%, 10=17.58%, 20=20.44%, 50=31.64%, 100=0.39% cpu : usr=0.07%, sys=6.65%, ctx=887, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2680: Thu Jun 5 07:34:07 2025 read: IOPS=31, BW=511KiB/s (523kB/s)(6032KiB/11801msec) clat (usec): min=1971, max=43501, avg=5284.36, stdev=4687.66 lat (usec): min=1972, max=43502, avg=5285.62, stdev=4687.60 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2606], 10.00th=[ 2868], 20.00th=[ 3130], | 30.00th=[ 3326], 40.00th=[ 3589], 50.00th=[ 3851], 60.00th=[ 4146], | 70.00th=[ 4490], 80.00th=[ 5407], 90.00th=[ 9241], 95.00th=[15139], | 99.00th=[26870], 99.50th=[34866], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 223, max= 958, per=25.67%, avg=513.17, stdev=160.88, samples=23 iops : min= 13, max= 59, avg=31.91, stdev=10.06, samples=23 write: IOPS=33, BW=530KiB/s (543kB/s)(6256KiB/11801msec) clat (usec): min=12419, max=64715, avg=24956.42, stdev=8610.40 lat (usec): min=12421, max=64717, avg=24958.92, stdev=8610.53 clat percentiles (usec): | 1.00th=[12780], 5.00th=[14091], 10.00th=[15664], 20.00th=[17171], | 30.00th=[19530], 40.00th=[21365], 50.00th=[23725], 60.00th=[25822], | 70.00th=[27919], 80.00th=[31065], 90.00th=[35390], 95.00th=[40633], | 99.00th=[56361], 99.50th=[63701], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 288, max= 734, per=25.33%, avg=532.65, stdev=126.75, samples=23 iops : min= 18, max= 45, avg=33.13, stdev= 8.00, samples=23 lat (msec) : 2=0.26%, 4=26.95%, 10=17.32%, 20=20.31%, 50=34.37% lat (msec) : 100=0.78% cpu : usr=0.09%, sys=6.24%, ctx=879, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1999KiB/s (2047kB/s), 480KiB/s-576KiB/s (492kB/s-590kB/s), io=23.4MiB (24.5MB), run=10866-11980msec WRITE: bw=2104KiB/s (2154kB/s), 530KiB/s-555KiB/s (543kB/s-568kB/s), io=24.6MiB (25.8MB), run=10866-11980msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2702: Thu Jun 5 07:34:10 2025 read: IOPS=5, BW=5885KiB/s (6026kB/s)(7168KiB/1218msec) clat (msec): min=32, max=660, avg=125.67, stdev=235.66 lat (msec): min=32, max=660, avg=125.67, stdev=235.66 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 36], 60.00th=[ 40], | 70.00th=[ 40], 80.00th=[ 44], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min=10240, max=10240, per=69.90%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4204KiB/s (4304kB/s)(5120KiB/1218msec) clat (msec): min=31, max=115, avg=66.71, stdev=35.62 lat (msec): min=31, max=115, avg=66.76, stdev=35.62 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 6144, max= 6144, per=29.96%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=27.28%, ctx=850, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2703: Thu Jun 5 07:34:10 2025 read: IOPS=1, BW=1465KiB/s (1500kB/s)(2048KiB/1398msec) clat (usec): min=46918, max=75717, avg=61317.95, stdev=20364.21 lat (usec): min=46921, max=75719, avg=61320.50, stdev=20363.87 clat percentiles (usec): | 1.00th=[46924], 5.00th=[46924], 10.00th=[46924], 20.00th=[46924], | 30.00th=[46924], 40.00th=[46924], 50.00th=[46924], 60.00th=[76022], | 70.00th=[76022], 80.00th=[76022], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 2048, max= 2048, per=13.98%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7325KiB/s (7501kB/s)(10.0MiB/1398msec) clat (msec): min=20, max=695, avg=125.26, stdev=206.11 lat (msec): min=20, max=696, avg=125.35, stdev=206.12 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 23], | 30.00th=[ 25], 40.00th=[ 35], 50.00th=[ 43], 60.00th=[ 61], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 178], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min=10240, max=10240, per=49.93%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=19.47%, ctx=436, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2704: Thu Jun 5 07:34:10 2025 read: IOPS=4, BW=5061KiB/s (5182kB/s)(6144KiB/1214msec) clat (usec): min=570, max=59792, avg=23089.44, stdev=25919.58 lat (usec): min=571, max=59793, avg=23090.85, stdev=25919.60 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 570], 10.00th=[ 570], 20.00th=[ 742], | 30.00th=[ 742], 40.00th=[ 1713], 50.00th=[ 1713], 60.00th=[30278], | 70.00th=[45351], 80.00th=[45351], 90.00th=[60031], 95.00th=[60031], | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 4096, max= 4096, per=27.96%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5061KiB/s (5182kB/s)(6144KiB/1214msec) clat (msec): min=33, max=662, avg=173.98, stdev=241.44 lat (msec): min=33, max=662, avg=174.06, stdev=241.44 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 96], 50.00th=[ 96], 60.00th=[ 97], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 4096, max= 4096, per=19.97%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (usec) : 750=16.67% lat (msec) : 2=8.33%, 50=33.33%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=16.41%, ctx=518, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2705: Thu Jun 5 07:34:10 2025 read: IOPS=4, BW=4235KiB/s (4337kB/s)(5120KiB/1209msec) clat (usec): min=436, max=72365, avg=31473.88, stdev=31922.56 lat (usec): min=437, max=72366, avg=31475.13, stdev=31922.39 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 437], | 30.00th=[ 578], 40.00th=[ 578], 50.00th=[30540], 60.00th=[30540], | 70.00th=[53740], 80.00th=[53740], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 6144, max= 6144, per=41.94%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5929KiB/s (6071kB/s)(7168KiB/1209msec) clat (msec): min=18, max=642, avg=142.48, stdev=222.89 lat (msec): min=18, max=642, avg=143.48, stdev=222.32 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 23], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 80], 60.00th=[ 81], | 70.00th=[ 81], 80.00th=[ 106], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 6144, per=19.97%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.25%, sys=19.62%, ctx=207, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.3MiB/s (15.0MB/s), 1465KiB/s-5885KiB/s (1500kB/s-6026kB/s), io=20.0MiB (20.0MB), run=1209-1398msec WRITE: bw=20.0MiB/s (21.0MB/s), 4204KiB/s-7325KiB/s (4304kB/s-7501kB/s), io=28.0MiB (29.4MB), run=1209-1398msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2707: Thu Jun 5 07:34:10 2025 read: IOPS=4, BW=4707KiB/s (4819kB/s)(7168KiB/1523msec) clat (msec): min=19, max=581, avg=164.24, stdev=208.82 lat (msec): min=19, max=581, avg=164.25, stdev=208.82 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 32], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 65], 60.00th=[ 95], | 70.00th=[ 95], 80.00th=[ 309], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 4079, max= 4079, per=30.33%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3362KiB/s (3442kB/s)(5120KiB/1523msec) clat (usec): min=33463, max=95258, avg=65312.83, stdev=26865.91 lat (usec): min=33502, max=95338, avg=65410.44, stdev=26870.81 clat percentiles (usec): | 1.00th=[33424], 5.00th=[33424], 10.00th=[33424], 20.00th=[33424], | 30.00th=[42730], 40.00th=[42730], 50.00th=[68682], 60.00th=[68682], | 70.00th=[86508], 80.00th=[86508], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] lat (msec) : 20=8.33%, 50=33.33%, 100=41.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=6.11%, ctx=51, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2708: Thu Jun 5 07:34:10 2025 read: IOPS=1, BW=1362KiB/s (1394kB/s)(2048KiB/1504msec) clat (msec): min=22, max=107, avg=65.22, stdev=59.83 lat (msec): min=22, max=107, avg=65.23, stdev=59.83 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 108], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 2048, max= 2048, per=15.23%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6809KiB/s (6972kB/s)(10.0MiB/1504msec) clat (msec): min=36, max=640, avg=132.09, stdev=183.34 lat (msec): min=36, max=640, avg=132.19, stdev=183.34 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 39], | 30.00th=[ 48], 40.00th=[ 55], 50.00th=[ 56], 60.00th=[ 70], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 171], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 4087, max= 4087, per=21.71%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 250=25.00%, 750=8.33% cpu : usr=0.00%, sys=7.58%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2709: Thu Jun 5 07:34:10 2025 read: IOPS=4, BW=4151KiB/s (4251kB/s)(6144KiB/1480msec) clat (msec): min=22, max=145, avg=59.37, stdev=46.77 lat (msec): min=22, max=145, avg=59.37, stdev=46.77 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 44], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 2048, max= 2048, per=15.23%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4151KiB/s (4251kB/s)(6144KiB/1480msec) clat (msec): min=34, max=685, avg=181.27, stdev=250.63 lat (msec): min=34, max=685, avg=181.37, stdev=250.63 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 109], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 2048, max= 4087, per=16.29%, avg=3067.50, stdev=1441.79, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 50=41.67%, 100=25.00%, 250=25.00%, 750=8.33% cpu : usr=0.07%, sys=7.44%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2710: Thu Jun 5 07:34:10 2025 read: IOPS=3, BW=3407KiB/s (3488kB/s)(5120KiB/1503msec) clat (msec): min=18, max=149, avg=52.25, stdev=55.08 lat (msec): min=18, max=149, avg=52.25, stdev=55.08 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 43], 80.00th=[ 43], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 8192, max= 8192, per=60.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4769KiB/s (4884kB/s)(7168KiB/1503msec) clat (msec): min=40, max=652, avg=175.89, stdev=216.34 lat (msec): min=40, max=652, avg=175.98, stdev=216.33 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 87], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 194], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 2048, max= 8192, per=25.38%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 250=25.00%, 750=8.33% cpu : usr=0.00%, sys=7.72%, ctx=52, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.1MiB/s (13.8MB/s), 1362KiB/s-4707KiB/s (1394kB/s-4819kB/s), io=20.0MiB (20.0MB), run=1480-1523msec WRITE: bw=18.4MiB/s (19.3MB/s), 3362KiB/s-6809KiB/s (3442kB/s-6972kB/s), io=28.0MiB (29.4MB), run=1480-1523msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2718: Thu Jun 5 07:34:12 2025 read: IOPS=3, BW=14.9MiB/s (15.7MB/s)(12.0MiB/803msec) clat (msec): min=238, max=287, avg=258.27, stdev=25.41 lat (msec): min=238, max=287, avg=258.27, stdev=25.41 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 7953, max= 7953, per=38.98%, avg=7953.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.37%, sys=9.35%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2719: Thu Jun 5 07:34:12 2025 read: IOPS=2, BW=8943KiB/s (9158kB/s)(4096KiB/458msec) clat (nsec): min=77647k, max=77647k, avg=77647245.00, stdev= 0.00 lat (nsec): min=77651k, max=77651k, avg=77650612.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[78119], 5.00th=[78119], 10.00th=[78119], 20.00th=[78119], | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[78119], | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] write: IOPS=4, BW=17.5MiB/s (18.3MB/s)(8192KiB/458msec) clat (msec): min=134, max=234, avg=184.12, stdev=70.74 lat (msec): min=134, max=234, avg=184.58, stdev=70.58 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 234], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.22%, sys=16.63%, ctx=46, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2721: Thu Jun 5 07:34:12 2025 write: IOPS=3, BW=15.9MiB/s (16.7MB/s)(12.0MiB/754msec) clat (msec): min=123, max=460, avg=245.46, stdev=186.81 lat (msec): min=123, max=460, avg=245.98, stdev=186.69 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min=16384, max=16384, per=37.70%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.27%, sys=13.00%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2723: Thu Jun 5 07:34:12 2025 write: IOPS=4, BW=16.7MiB/s (17.5MB/s)(12.0MiB/720msec) clat (msec): min=90, max=496, avg=229.85, stdev=230.99 lat (msec): min=90, max=496, avg=230.26, stdev=230.92 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min=16384, max=16384, per=37.70%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.42%, sys=15.99%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.9MiB/s (20.9MB/s), 8943KiB/s-14.9MiB/s (9158kB/s-15.7MB/s), io=16.0MiB (16.8MB), run=458-803msec WRITE: bw=42.4MiB/s (44.5MB/s), 15.9MiB/s-17.5MiB/s (16.7MB/s-18.3MB/s), io=32.0MiB (33.6MB), run=458-754msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2717: Thu Jun 5 07:34:12 2025 read: IOPS=4, BW=19.7MiB/s (20.7MB/s)(12.0MiB/608msec) clat (msec): min=95, max=240, avg=167.90, stdev=72.86 lat (msec): min=95, max=240, avg=167.90, stdev=72.86 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 241], 95.00th=[ 241], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 8192, max= 8192, per=49.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=28.01%, ctx=1531, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2720: Thu Jun 5 07:34:12 2025 read: IOPS=1, BW=4133KiB/s (4232kB/s)(4096KiB/991msec) clat (nsec): min=162089k, max=162089k, avg=162089361.00, stdev= 0.00 lat (nsec): min=162091k, max=162091k, avg=162090735.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 163], 50.00th=[ 163], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 8192, max= 8192, per=49.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8266KiB/s (8465kB/s)(8192KiB/991msec) clat (msec): min=319, max=395, avg=357.71, stdev=54.11 lat (msec): min=319, max=396, avg=358.13, stdev=54.11 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=19.90%, ctx=800, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2722: Thu Jun 5 07:34:12 2025 write: IOPS=3, BW=12.1MiB/s (12.7MB/s)(12.0MiB/990msec) clat (msec): min=116, max=506, avg=324.24, stdev=196.24 lat (msec): min=116, max=507, avg=324.73, stdev=196.56 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=28.01%, ctx=65, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2724: Thu Jun 5 07:34:12 2025 write: IOPS=3, BW=13.3MiB/s (13.9MB/s)(12.0MiB/905msec) clat (msec): min=149, max=450, avg=272.57, stdev=157.68 lat (msec): min=149, max=451, avg=273.10, stdev=157.97 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 8192, max= 8192, per=24.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=33.85%, ctx=95, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.1MiB/s (16.9MB/s), 4133KiB/s-19.7MiB/s (4232kB/s-20.7MB/s), io=16.0MiB (16.8MB), run=608-991msec WRITE: bw=32.3MiB/s (33.9MB/s), 8266KiB/s-13.3MiB/s (8465kB/s-13.9MB/s), io=32.0MiB (33.6MB), run=905-991msec