== sanity test 398b: DIO and buffer IO race ============== 15:21:48 (1741634508) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.35237 s, 37.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1728: Mon Mar 10 15:22:09 2025 read: IOPS=82, BW=331KiB/s (339kB/s)(5992KiB/18126msec) clat (usec): min=48, max=175075, avg=4990.18, stdev=6308.65 lat (usec): min=48, max=175076, avg=4990.99, stdev=6308.71 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 88], 10.00th=[ 102], 20.00th=[ 127], | 30.00th=[ 2638], 40.00th=[ 3294], 50.00th=[ 3916], 60.00th=[ 4817], | 70.00th=[ 6063], 80.00th=[ 8094], 90.00th=[ 10814], 95.00th=[ 13304], | 99.00th=[ 20055], 99.50th=[ 22152], 99.90th=[ 39584], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 160, max= 528, per=24.97%, avg=330.58, stdev=79.64, samples=36 iops : min= 40, max= 132, avg=82.53, stdev=19.94, samples=36 write: IOPS=86, BW=347KiB/s (356kB/s)(6296KiB/18126msec) clat (usec): min=490, max=68517, avg=6691.85, stdev=7024.31 lat (usec): min=490, max=68518, avg=6693.89, stdev=7024.24 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 668], 10.00th=[ 758], 20.00th=[ 938], | 30.00th=[ 1598], 40.00th=[ 3294], 50.00th=[ 4948], 60.00th=[ 6456], | 70.00th=[ 8094], 80.00th=[10945], 90.00th=[15401], 95.00th=[20579], | 99.00th=[30278], 99.50th=[37487], 99.90th=[64750], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 256, max= 520, per=25.03%, avg=346.92, stdev=67.02, samples=36 iops : min= 64, max= 130, avg=86.61, stdev=16.74, samples=36 lat (usec) : 50=0.07%, 100=4.26%, 250=7.32%, 500=0.78%, 750=4.88% lat (usec) : 1000=6.28% lat (msec) : 2=5.66%, 4=18.49%, 10=34.11%, 20=14.81%, 50=3.19% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.20%, sys=20.89%, ctx=6095, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1729: Mon Mar 10 15:22:09 2025 read: IOPS=82, BW=331KiB/s (339kB/s)(5940KiB/17942msec) clat (usec): min=46, max=55237, avg=4841.64, stdev=4414.47 lat (usec): min=47, max=55237, avg=4842.51, stdev=4414.58 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 90], 10.00th=[ 104], 20.00th=[ 147], | 30.00th=[ 2769], 40.00th=[ 3326], 50.00th=[ 3982], 60.00th=[ 4686], | 70.00th=[ 5800], 80.00th=[ 7701], 90.00th=[10814], 95.00th=[12649], | 99.00th=[18744], 99.50th=[21890], 99.90th=[29754], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 192, max= 552, per=25.10%, avg=332.37, stdev=79.22, samples=35 iops : min= 48, max= 138, avg=82.97, stdev=19.82, samples=35 write: IOPS=88, BW=354KiB/s (362kB/s)(6348KiB/17942msec) clat (usec): min=473, max=174155, avg=6702.58, stdev=7978.81 lat (usec): min=474, max=174156, avg=6703.71, stdev=7978.81 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 652], 10.00th=[ 758], 20.00th=[ 938], | 30.00th=[ 1352], 40.00th=[ 3294], 50.00th=[ 5145], 60.00th=[ 6652], | 70.00th=[ 8586], 80.00th=[ 10814], 90.00th=[ 14877], 95.00th=[ 19006], | 99.00th=[ 27657], 99.50th=[ 34341], 99.90th=[ 76022], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 248, max= 600, per=25.35%, avg=351.34, stdev=78.26, samples=35 iops : min= 62, max= 150, avg=87.71, stdev=19.59, samples=35 lat (usec) : 50=0.07%, 100=3.84%, 250=6.58%, 500=0.59%, 750=4.79% lat (usec) : 1000=7.03% lat (msec) : 2=6.02%, 4=17.64%, 10=35.97%, 20=14.88%, 50=2.44% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.19%, sys=21.87%, ctx=6005, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1730: Mon Mar 10 15:22:09 2025 read: IOPS=83, BW=333KiB/s (341kB/s)(6028KiB/18119msec) clat (usec): min=47, max=44369, avg=4948.31, stdev=4807.54 lat (usec): min=47, max=44370, avg=4950.66, stdev=4806.78 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 82], 10.00th=[ 100], 20.00th=[ 135], | 30.00th=[ 2704], 40.00th=[ 3261], 50.00th=[ 3884], 60.00th=[ 4752], | 70.00th=[ 6128], 80.00th=[ 7963], 90.00th=[10945], 95.00th=[13698], | 99.00th=[20055], 99.50th=[25560], 99.90th=[39060], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 160, max= 495, per=25.06%, avg=331.81, stdev=79.65, samples=36 iops : min= 40, max= 123, avg=82.81, stdev=19.82, samples=36 write: IOPS=86, BW=345KiB/s (354kB/s)(6260KiB/18119msec) clat (usec): min=499, max=211302, avg=6736.26, stdev=8523.25 lat (usec): min=500, max=211304, avg=6737.72, stdev=8523.31 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 685], 10.00th=[ 775], 20.00th=[ 938], | 30.00th=[ 1434], 40.00th=[ 3195], 50.00th=[ 5080], 60.00th=[ 6456], | 70.00th=[ 8356], 80.00th=[ 10814], 90.00th=[ 14877], 95.00th=[ 19006], | 99.00th=[ 33817], 99.50th=[ 39060], 99.90th=[ 58459], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 184, max= 568, per=24.81%, avg=343.81, stdev=71.19, samples=36 iops : min= 46, max= 142, avg=85.81, stdev=17.82, samples=36 lat (usec) : 50=0.07%, 100=4.82%, 250=6.38%, 500=0.68%, 750=4.69% lat (usec) : 1000=6.84% lat (msec) : 2=6.25%, 4=17.51%, 10=34.73%, 20=15.40%, 50=2.54% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.23%, sys=21.72%, ctx=6033, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1731: Mon Mar 10 15:22:09 2025 read: IOPS=83, BW=334KiB/s (342kB/s)(6052KiB/18128msec) clat (usec): min=53, max=48300, avg=4728.12, stdev=4582.94 lat (usec): min=54, max=48301, avg=4729.28, stdev=4583.01 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 83], 10.00th=[ 101], 20.00th=[ 128], | 30.00th=[ 2507], 40.00th=[ 3228], 50.00th=[ 3818], 60.00th=[ 4621], | 70.00th=[ 5866], 80.00th=[ 7701], 90.00th=[10552], 95.00th=[12911], | 99.00th=[18744], 99.50th=[22676], 99.90th=[35390], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 55, max= 504, per=25.04%, avg=331.53, stdev=87.96, samples=36 iops : min= 13, max= 126, avg=82.72, stdev=22.01, samples=36 write: IOPS=85, BW=344KiB/s (352kB/s)(6236KiB/18128msec) clat (usec): min=477, max=209497, avg=6963.87, stdev=8763.95 lat (usec): min=477, max=209499, avg=6965.07, stdev=8763.86 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 660], 10.00th=[ 742], 20.00th=[ 898], | 30.00th=[ 1352], 40.00th=[ 3359], 50.00th=[ 5211], 60.00th=[ 6718], | 70.00th=[ 8586], 80.00th=[ 11469], 90.00th=[ 15533], 95.00th=[ 19530], | 99.00th=[ 33162], 99.50th=[ 41157], 99.90th=[ 65274], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 55, max= 544, per=24.72%, avg=342.67, stdev=91.31, samples=36 iops : min= 13, max= 136, avg=85.50, stdev=22.89, samples=36 lat (usec) : 100=4.82%, 250=7.36%, 500=0.91%, 750=5.47%, 1000=6.61% lat (msec) : 2=5.31%, 4=17.25%, 10=34.05%, 20=15.40%, 50=2.67% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.20%, sys=21.13%, ctx=5904, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1325KiB/s (1356kB/s), 331KiB/s-334KiB/s (339kB/s-342kB/s), io=23.4MiB (24.6MB), run=17942-18128msec WRITE: bw=1387KiB/s (1420kB/s), 344KiB/s-354KiB/s (352kB/s-362kB/s), io=24.6MiB (25.7MB), run=17942-18128msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1732: Mon Mar 10 15:22:44 2025 read: IOPS=28, BW=114KiB/s (116kB/s)(5992KiB/52745msec) clat (usec): min=1708, max=177521, avg=5992.37, stdev=6991.43 lat (usec): min=1709, max=177524, avg=5993.51, stdev=6991.40 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 15], | 99.00th=[ 28], 99.50th=[ 36], 99.90th=[ 87], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 24, max= 240, per=25.14%, avg=113.64, stdev=50.04, samples=105 iops : min= 6, max= 60, avg=28.32, stdev=12.52, samples=105 write: IOPS=29, BW=119KiB/s (122kB/s)(6296KiB/52745msec) clat (usec): min=11763, max=95544, avg=27735.41, stdev=11487.22 lat (usec): min=11764, max=95545, avg=27736.87, stdev=11486.99 clat percentiles (usec): | 1.00th=[13829], 5.00th=[15533], 10.00th=[16712], 20.00th=[18744], | 30.00th=[20841], 40.00th=[22676], 50.00th=[25035], 60.00th=[27132], | 70.00th=[29754], 80.00th=[34341], 90.00th=[43779], 95.00th=[52167], | 99.00th=[68682], 99.50th=[72877], 99.90th=[90702], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 48, max= 184, per=25.20%, avg=119.20, stdev=37.41, samples=105 iops : min= 12, max= 46, avg=29.71, stdev= 9.40, samples=105 lat (msec) : 2=0.46%, 4=20.41%, 10=22.14%, 20=18.16%, 50=35.77% lat (msec) : 100=3.03%, 250=0.03% cpu : usr=0.13%, sys=5.00%, ctx=3888, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1733: Mon Mar 10 15:22:44 2025 read: IOPS=27, BW=112KiB/s (115kB/s)(5940KiB/53093msec) clat (usec): min=1710, max=189365, avg=5807.39, stdev=6599.46 lat (usec): min=1710, max=189367, avg=5808.49, stdev=6599.44 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2245], 10.00th=[ 2507], 20.00th=[ 2868], | 30.00th=[ 3195], 40.00th=[ 3752], 50.00th=[ 4424], 60.00th=[ 5145], | 70.00th=[ 5735], 80.00th=[ 6783], 90.00th=[ 10683], 95.00th=[ 14353], | 99.00th=[ 25560], 99.50th=[ 28705], 99.90th=[ 60031], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 8, max= 248, per=24.77%, avg=111.96, stdev=53.44, samples=105 iops : min= 2, max= 62, avg=27.90, stdev=13.38, samples=105 write: IOPS=29, BW=120KiB/s (122kB/s)(6348KiB/53093msec) clat (usec): min=12559, max=98586, avg=27948.53, stdev=11724.42 lat (usec): min=12560, max=98588, avg=27949.90, stdev=11724.37 clat percentiles (usec): | 1.00th=[14091], 5.00th=[15401], 10.00th=[16712], 20.00th=[18744], | 30.00th=[20841], 40.00th=[22938], 50.00th=[25035], 60.00th=[27395], | 70.00th=[30016], 80.00th=[34866], 90.00th=[44303], 95.00th=[52167], | 99.00th=[69731], 99.50th=[73925], 99.90th=[91751], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 47, max= 176, per=25.17%, avg=119.05, stdev=37.57, samples=105 iops : min= 11, max= 44, avg=29.68, stdev= 9.45, samples=105 lat (msec) : 2=0.72%, 4=20.44%, 10=21.78%, 20=18.33%, 50=35.42% lat (msec) : 100=3.29%, 250=0.03% cpu : usr=0.12%, sys=4.95%, ctx=3880, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1734: Mon Mar 10 15:22:44 2025 read: IOPS=28, BW=114KiB/s (116kB/s)(6028KiB/53015msec) clat (usec): min=1607, max=89211, avg=6064.66, stdev=5720.22 lat (usec): min=1608, max=89212, avg=6065.67, stdev=5720.21 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2278], 10.00th=[ 2474], 20.00th=[ 2868], | 30.00th=[ 3294], 40.00th=[ 3916], 50.00th=[ 4621], 60.00th=[ 5276], | 70.00th=[ 6063], 80.00th=[ 7373], 90.00th=[11600], 95.00th=[15270], | 99.00th=[26084], 99.50th=[32900], 99.90th=[86508], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 16, max= 232, per=25.14%, avg=113.63, stdev=47.46, samples=105 iops : min= 4, max= 58, avg=28.31, stdev=11.88, samples=105 write: IOPS=29, BW=118KiB/s (121kB/s)(6260KiB/53015msec) clat (msec): min=11, max=207, avg=27.96, stdev=12.61 lat (msec): min=11, max=207, avg=27.96, stdev=12.61 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 26], 60.00th=[ 28], | 70.00th=[ 30], 80.00th=[ 34], 90.00th=[ 44], 95.00th=[ 54], | 99.00th=[ 71], 99.50th=[ 83], 99.90th=[ 94], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 40, max= 184, per=24.86%, avg=117.59, stdev=38.36, samples=105 iops : min= 10, max= 46, avg=29.30, stdev= 9.58, samples=105 lat (msec) : 2=0.75%, 4=19.73%, 10=22.40%, 20=17.06%, 50=37.04% lat (msec) : 100=2.99%, 250=0.03% cpu : usr=0.14%, sys=4.93%, ctx=3891, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1735: Mon Mar 10 15:22:44 2025 read: IOPS=28, BW=115KiB/s (118kB/s)(6052KiB/52539msec) clat (usec): min=1698, max=89407, avg=5819.80, stdev=5149.64 lat (usec): min=1698, max=89408, avg=5820.95, stdev=5149.58 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2245], 10.00th=[ 2474], 20.00th=[ 2802], | 30.00th=[ 3163], 40.00th=[ 3720], 50.00th=[ 4490], 60.00th=[ 5145], | 70.00th=[ 5932], 80.00th=[ 7046], 90.00th=[10945], 95.00th=[15664], | 99.00th=[24249], 99.50th=[31065], 99.90th=[67634], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 8, max= 248, per=25.46%, avg=115.08, stdev=52.47, samples=105 iops : min= 2, max= 62, avg=28.68, stdev=13.16, samples=105 write: IOPS=29, BW=119KiB/s (122kB/s)(6236KiB/52539msec) clat (msec): min=12, max=206, avg=27.98, stdev=12.49 lat (msec): min=12, max=206, avg=27.98, stdev=12.49 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 22], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 28], | 70.00th=[ 31], 80.00th=[ 35], 90.00th=[ 43], 95.00th=[ 53], | 99.00th=[ 72], 99.50th=[ 80], 99.90th=[ 95], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 40, max= 176, per=25.05%, avg=118.50, stdev=37.26, samples=105 iops : min= 10, max= 44, avg=29.53, stdev= 9.38, samples=105 lat (msec) : 2=0.62%, 4=20.96%, 10=21.97%, 20=17.58%, 50=36.00% lat (msec) : 100=2.83%, 250=0.03% cpu : usr=0.14%, sys=4.98%, ctx=3922, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=452KiB/s (463kB/s), 112KiB/s-115KiB/s (115kB/s-118kB/s), io=23.4MiB (24.6MB), run=52539-53093msec WRITE: bw=474KiB/s (485kB/s), 118KiB/s-120KiB/s (121kB/s-122kB/s), io=24.6MiB (25.7MB), run=52539-53093msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1785: Mon Mar 10 15:22:49 2025 read: IOPS=98, BW=1575KiB/s (1613kB/s)(5920KiB/3759msec) clat (usec): min=59, max=40287, avg=4407.01, stdev=4656.26 lat (usec): min=60, max=40288, avg=4409.97, stdev=4656.69 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 119], | 30.00th=[ 155], 40.00th=[ 3032], 50.00th=[ 3982], 60.00th=[ 4817], | 70.00th=[ 6128], 80.00th=[ 7635], 90.00th=[10290], 95.00th=[11600], | 99.00th=[17171], 99.50th=[27132], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 894, max= 2304, per=25.55%, avg=1594.71, stdev=455.24, samples=7 iops : min= 55, max= 144, avg=99.43, stdev=28.72, samples=7 write: IOPS=105, BW=1694KiB/s (1735kB/s)(6368KiB/3759msec) clat (usec): min=652, max=31415, avg=5235.91, stdev=5299.95 lat (usec): min=653, max=31416, avg=5237.63, stdev=5299.67 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 1004], 10.00th=[ 1139], 20.00th=[ 1352], | 30.00th=[ 1565], 40.00th=[ 1795], 50.00th=[ 2474], 60.00th=[ 4555], | 70.00th=[ 6456], 80.00th=[ 9241], 90.00th=[13042], 95.00th=[15795], | 99.00th=[23462], 99.50th=[30278], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 1277, max= 2368, per=26.09%, avg=1713.43, stdev=376.43, samples=7 iops : min= 79, max= 148, avg=106.86, stdev=23.81, samples=7 lat (usec) : 100=4.69%, 250=12.37%, 500=1.17%, 750=0.39%, 1000=2.34% lat (msec) : 2=21.22%, 4=11.85%, 10=30.73%, 20=13.80%, 50=1.43% cpu : usr=0.40%, sys=27.38%, ctx=1417, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1786: Mon Mar 10 15:22:49 2025 read: IOPS=95, BW=1523KiB/s (1559kB/s)(5744KiB/3772msec) clat (usec): min=56, max=18141, avg=4406.50, stdev=4091.40 lat (usec): min=57, max=18142, avg=4407.40, stdev=4091.42 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 81], 10.00th=[ 96], 20.00th=[ 119], | 30.00th=[ 157], 40.00th=[ 3458], 50.00th=[ 4228], 60.00th=[ 4948], | 70.00th=[ 6063], 80.00th=[ 7832], 90.00th=[10028], 95.00th=[12256], | 99.00th=[16188], 99.50th=[17433], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1216, max= 1984, per=24.75%, avg=1545.14, stdev=255.14, samples=7 iops : min= 76, max= 124, avg=96.57, stdev=15.95, samples=7 write: IOPS=108, BW=1735KiB/s (1777kB/s)(6544KiB/3772msec) clat (usec): min=698, max=33429, avg=5268.53, stdev=5292.62 lat (usec): min=699, max=33429, avg=5270.01, stdev=5292.52 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 979], 10.00th=[ 1139], 20.00th=[ 1319], | 30.00th=[ 1532], 40.00th=[ 1827], 50.00th=[ 2507], 60.00th=[ 5014], | 70.00th=[ 6980], 80.00th=[ 8586], 90.00th=[12518], 95.00th=[15270], | 99.00th=[23200], 99.50th=[26084], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 1376, max= 2112, per=26.59%, avg=1746.29, stdev=244.30, samples=7 iops : min= 86, max= 132, avg=109.14, stdev=15.27, samples=7 lat (usec) : 100=5.08%, 250=9.90%, 500=1.56%, 750=0.52%, 1000=3.12% lat (msec) : 2=20.83%, 4=9.11%, 10=37.11%, 20=11.46%, 50=1.30% cpu : usr=0.11%, sys=28.43%, ctx=1392, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1787: Mon Mar 10 15:22:49 2025 read: IOPS=101, BW=1630KiB/s (1670kB/s)(6256KiB/3837msec) clat (usec): min=56, max=39375, avg=4564.70, stdev=5355.23 lat (usec): min=56, max=39376, avg=4565.48, stdev=5355.21 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 91], 10.00th=[ 102], 20.00th=[ 121], | 30.00th=[ 139], 40.00th=[ 375], 50.00th=[ 3851], 60.00th=[ 5080], | 70.00th=[ 6390], 80.00th=[ 8455], 90.00th=[10683], 95.00th=[12387], | 99.00th=[29754], 99.50th=[38011], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 1248, max= 2272, per=26.44%, avg=1650.29, stdev=325.74, samples=7 iops : min= 78, max= 142, avg=103.14, stdev=20.36, samples=7 write: IOPS=98, BW=1572KiB/s (1610kB/s)(6032KiB/3837msec) clat (usec): min=669, max=29720, avg=5358.63, stdev=5126.69 lat (usec): min=670, max=29720, avg=5360.13, stdev=5127.14 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 963], 10.00th=[ 1123], 20.00th=[ 1369], | 30.00th=[ 1582], 40.00th=[ 2040], 50.00th=[ 2868], 60.00th=[ 4883], | 70.00th=[ 7111], 80.00th=[ 9765], 90.00th=[12649], 95.00th=[15664], | 99.00th=[22414], 99.50th=[27919], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 960, max= 1856, per=23.67%, avg=1554.29, stdev=320.99, samples=7 iops : min= 60, max= 116, avg=97.14, stdev=20.06, samples=7 lat (usec) : 100=4.30%, 250=14.71%, 500=2.08%, 750=0.26%, 1000=2.60% lat (msec) : 2=17.32%, 4=11.59%, 10=31.77%, 20=13.67%, 50=1.69% cpu : usr=0.23%, sys=27.14%, ctx=1387, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1788: Mon Mar 10 15:22:49 2025 read: IOPS=98, BW=1576KiB/s (1614kB/s)(6032KiB/3828msec) clat (usec): min=53, max=53010, avg=4518.03, stdev=5583.17 lat (usec): min=53, max=53011, avg=4518.86, stdev=5583.27 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 77], 10.00th=[ 96], 20.00th=[ 114], | 30.00th=[ 139], 40.00th=[ 1090], 50.00th=[ 4047], 60.00th=[ 5014], | 70.00th=[ 6063], 80.00th=[ 7832], 90.00th=[10159], 95.00th=[11731], | 99.00th=[26084], 99.50th=[45876], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 1181, max= 2304, per=25.11%, avg=1567.14, stdev=351.76, samples=7 iops : min= 73, max= 144, avg=97.71, stdev=22.19, samples=7 write: IOPS=102, BW=1634KiB/s (1673kB/s)(6256KiB/3828msec) clat (usec): min=771, max=30454, avg=5368.11, stdev=5221.96 lat (usec): min=772, max=30457, avg=5374.94, stdev=5227.37 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1352], | 30.00th=[ 1549], 40.00th=[ 1811], 50.00th=[ 2671], 60.00th=[ 5080], | 70.00th=[ 6849], 80.00th=[ 9372], 90.00th=[13435], 95.00th=[15795], | 99.00th=[22938], 99.50th=[24249], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 1312, max= 2016, per=25.04%, avg=1644.57, stdev=228.51, samples=7 iops : min= 82, max= 126, avg=102.57, stdev=14.30, samples=7 lat (usec) : 100=5.86%, 250=11.59%, 500=1.82%, 750=0.26%, 1000=2.21% lat (msec) : 2=19.66%, 4=11.33%, 10=32.81%, 20=12.76%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=27.15%, ctx=1400, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6242KiB/s (6392kB/s), 1523KiB/s-1630KiB/s (1559kB/s-1670kB/s), io=23.4MiB (24.5MB), run=3759-3837msec WRITE: bw=6568KiB/s (6725kB/s), 1572KiB/s-1735KiB/s (1610kB/s-1777kB/s), io=24.6MiB (25.8MB), run=3759-3837msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1790: Mon Mar 10 15:23:00 2025 read: IOPS=26, BW=419KiB/s (430kB/s)(5920KiB/14113msec) clat (usec): min=1931, max=45851, avg=5972.99, stdev=5247.71 lat (usec): min=1933, max=45853, avg=5974.68, stdev=5247.44 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2704], 10.00th=[ 2868], 20.00th=[ 3163], | 30.00th=[ 3458], 40.00th=[ 3818], 50.00th=[ 4178], 60.00th=[ 4621], | 70.00th=[ 5342], 80.00th=[ 6587], 90.00th=[12387], 95.00th=[17171], | 99.00th=[24249], 99.50th=[38536], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 128, max= 896, per=24.48%, avg=413.26, stdev=177.88, samples=27 iops : min= 8, max= 56, avg=25.59, stdev=11.13, samples=27 write: IOPS=28, BW=451KiB/s (462kB/s)(6368KiB/14113msec) clat (usec): min=12055, max=78100, avg=29855.84, stdev=12704.51 lat (usec): min=12057, max=78101, avg=29859.23, stdev=12706.83 clat percentiles (usec): | 1.00th=[12780], 5.00th=[15533], 10.00th=[17695], 20.00th=[19268], | 30.00th=[21890], 40.00th=[24249], 50.00th=[26608], 60.00th=[29230], | 70.00th=[32113], 80.00th=[36963], 90.00th=[48497], 95.00th=[59507], | 99.00th=[69731], 99.50th=[73925], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 191, max= 734, per=25.07%, avg=445.22, stdev=153.65, samples=27 iops : min= 11, max= 45, avg=27.59, stdev= 9.66, samples=27 lat (msec) : 2=0.13%, 4=21.48%, 10=20.31%, 20=16.93%, 50=36.33% lat (msec) : 100=4.82% cpu : usr=0.11%, sys=5.96%, ctx=934, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1791: Mon Mar 10 15:23:00 2025 read: IOPS=25, BW=405KiB/s (415kB/s)(5744KiB/14187msec) clat (usec): min=2200, max=36507, avg=5813.02, stdev=4861.43 lat (usec): min=2201, max=36508, avg=5814.89, stdev=4861.11 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2573], 10.00th=[ 2704], 20.00th=[ 3064], | 30.00th=[ 3326], 40.00th=[ 3654], 50.00th=[ 3982], 60.00th=[ 4490], | 70.00th=[ 5473], 80.00th=[ 7242], 90.00th=[11469], 95.00th=[14615], | 99.00th=[30016], 99.50th=[34866], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 64, max= 832, per=23.85%, avg=402.61, stdev=190.88, samples=28 iops : min= 4, max= 52, avg=25.04, stdev=12.02, samples=28 write: IOPS=28, BW=461KiB/s (472kB/s)(6544KiB/14187msec) clat (usec): min=12427, max=90354, avg=29408.67, stdev=12176.05 lat (usec): min=12430, max=90356, avg=29411.37, stdev=12176.15 clat percentiles (usec): | 1.00th=[13173], 5.00th=[15795], 10.00th=[17433], 20.00th=[19792], | 30.00th=[22152], 40.00th=[24511], 50.00th=[26870], 60.00th=[29230], | 70.00th=[32375], 80.00th=[36439], 90.00th=[44827], 95.00th=[54264], | 99.00th=[71828], 99.50th=[76022], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 191, max= 736, per=25.78%, avg=457.79, stdev=145.18, samples=28 iops : min= 11, max= 46, avg=28.50, stdev= 9.14, samples=28 lat (msec) : 4=23.96%, 10=16.54%, 20=16.02%, 50=39.97%, 100=3.52% cpu : usr=0.12%, sys=5.69%, ctx=924, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1792: Mon Mar 10 15:23:00 2025 read: IOPS=28, BW=455KiB/s (466kB/s)(6256KiB/13752msec) clat (usec): min=2153, max=33683, avg=5869.70, stdev=4950.83 lat (usec): min=2153, max=33683, avg=5871.07, stdev=4950.76 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2638], 10.00th=[ 2868], 20.00th=[ 3163], | 30.00th=[ 3425], 40.00th=[ 3720], 50.00th=[ 4113], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 7439], 90.00th=[10421], 95.00th=[15664], | 99.00th=[29754], 99.50th=[33162], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 160, max= 1088, per=27.23%, avg=459.67, stdev=251.04, samples=27 iops : min= 10, max= 68, avg=28.63, stdev=15.70, samples=27 write: IOPS=27, BW=439KiB/s (449kB/s)(6032KiB/13752msec) clat (usec): min=13010, max=93624, avg=30230.75, stdev=13882.34 lat (usec): min=13013, max=93626, avg=30236.96, stdev=13881.84 clat percentiles (usec): | 1.00th=[13829], 5.00th=[15926], 10.00th=[17433], 20.00th=[19530], | 30.00th=[22152], 40.00th=[24249], 50.00th=[26608], 60.00th=[29492], | 70.00th=[31851], 80.00th=[38011], 90.00th=[47973], 95.00th=[61080], | 99.00th=[81265], 99.50th=[91751], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 192, max= 736, per=24.48%, avg=434.85, stdev=145.70, samples=27 iops : min= 12, max= 46, avg=27.07, stdev= 9.17, samples=27 lat (msec) : 4=24.35%, 10=20.57%, 20=14.97%, 50=36.20%, 100=3.91% cpu : usr=0.21%, sys=5.91%, ctx=912, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1793: Mon Mar 10 15:23:00 2025 read: IOPS=27, BW=437KiB/s (448kB/s)(6032KiB/13793msec) clat (usec): min=2215, max=27955, avg=5626.85, stdev=4395.69 lat (usec): min=2215, max=27956, avg=5628.08, stdev=4395.63 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2573], 10.00th=[ 2835], 20.00th=[ 3097], | 30.00th=[ 3392], 40.00th=[ 3654], 50.00th=[ 3949], 60.00th=[ 4490], | 70.00th=[ 5080], 80.00th=[ 6390], 90.00th=[11076], 95.00th=[14746], | 99.00th=[25035], 99.50th=[27132], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 127, max= 736, per=25.68%, avg=433.41, stdev=179.98, samples=27 iops : min= 7, max= 46, avg=26.81, stdev=11.45, samples=27 write: IOPS=28, BW=454KiB/s (464kB/s)(6256KiB/13793msec) clat (usec): min=10745, max=92119, avg=29670.94, stdev=12493.32 lat (usec): min=10747, max=92120, avg=29673.32, stdev=12493.66 clat percentiles (usec): | 1.00th=[13435], 5.00th=[16319], 10.00th=[17957], 20.00th=[20579], | 30.00th=[22676], 40.00th=[24249], 50.00th=[26608], 60.00th=[28705], | 70.00th=[31589], 80.00th=[37487], 90.00th=[45876], 95.00th=[54264], | 99.00th=[77071], 99.50th=[90702], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 192, max= 704, per=25.40%, avg=451.04, stdev=144.06, samples=27 iops : min= 12, max= 44, avg=27.93, stdev= 9.09, samples=27 lat (msec) : 4=25.78%, 10=17.32%, 20=13.80%, 50=39.45%, 100=3.65% cpu : usr=0.19%, sys=5.95%, ctx=937, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1688KiB/s (1729kB/s), 405KiB/s-455KiB/s (415kB/s-466kB/s), io=23.4MiB (24.5MB), run=13752-14187msec WRITE: bw=1776KiB/s (1819kB/s), 439KiB/s-461KiB/s (449kB/s-472kB/s), io=24.6MiB (25.8MB), run=13752-14187msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1814: Mon Mar 10 15:23:02 2025 read: IOPS=11, BW=11.6MiB/s (12.2MB/s)(7168KiB/601msec) clat (usec): min=756, max=76150, avg=38485.16, stdev=23459.79 lat (usec): min=758, max=76152, avg=38486.80, stdev=23460.02 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 758], 10.00th=[ 758], 20.00th=[26084], | 30.00th=[31851], 40.00th=[31851], 50.00th=[38536], 60.00th=[41681], | 70.00th=[41681], 80.00th=[54264], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min=10199, max=10199, per=42.58%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8519KiB/s (8724kB/s)(5120KiB/601msec) clat (msec): min=34, max=110, avg=59.81, stdev=30.59 lat (msec): min=34, max=110, avg=59.87, stdev=30.57 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 59], 60.00th=[ 59], | 70.00th=[ 60], 80.00th=[ 60], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 8159, max= 8159, per=24.33%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 1000=8.33% lat (msec) : 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=32.17%, ctx=959, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1815: Mon Mar 10 15:23:02 2025 read: IOPS=2, BW=2395KiB/s (2453kB/s)(2048KiB/855msec) clat (usec): min=34421, max=63306, avg=48863.98, stdev=20424.21 lat (usec): min=34424, max=63307, avg=48865.93, stdev=20423.55 clat percentiles (usec): | 1.00th=[34341], 5.00th=[34341], 10.00th=[34341], 20.00th=[34341], | 30.00th=[34341], 40.00th=[34341], 50.00th=[34341], 60.00th=[63177], | 70.00th=[63177], 80.00th=[63177], 90.00th=[63177], 95.00th=[63177], | 99.00th=[63177], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 4096, max= 4096, per=17.10%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.7MiB/s (12.3MB/s)(10.0MiB/855msec) clat (msec): min=20, max=134, avg=74.86, stdev=40.75 lat (msec): min=20, max=134, avg=74.98, stdev=40.67 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 26], | 30.00th=[ 41], 40.00th=[ 50], 50.00th=[ 66], 60.00th=[ 86], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 123], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min=10240, max=10240, per=30.54%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=25.00%, 250=33.33% cpu : usr=0.00%, sys=27.87%, ctx=207, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1816: Mon Mar 10 15:23:02 2025 read: IOPS=9, BW=9585KiB/s (9815kB/s)(6144KiB/641msec) clat (usec): min=490, max=52068, avg=28403.88, stdev=22679.29 lat (usec): min=491, max=52070, avg=28405.16, stdev=22679.34 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 490], 10.00th=[ 490], 20.00th=[ 545], | 30.00th=[ 545], 40.00th=[31065], 50.00th=[31065], 60.00th=[40633], | 70.00th=[45876], 80.00th=[45876], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 8192, max= 8192, per=34.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9585KiB/s (9815kB/s)(6144KiB/641msec) clat (msec): min=33, max=144, avg=75.29, stdev=46.07 lat (msec): min=33, max=144, avg=75.35, stdev=46.06 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 69], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8192, max= 8192, per=24.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=41.67%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=31.72%, ctx=804, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1817: Mon Mar 10 15:23:02 2025 read: IOPS=6, BW=6632KiB/s (6791kB/s)(5120KiB/772msec) clat (usec): min=584, max=66017, avg=31306.09, stdev=29502.89 lat (usec): min=585, max=66019, avg=31307.77, stdev=29502.93 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 586], | 30.00th=[ 2024], 40.00th=[ 2024], 50.00th=[35390], 60.00th=[35390], | 70.00th=[52691], 80.00th=[52691], 90.00th=[65799], 95.00th=[65799], | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 8192, max= 8192, per=34.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9285KiB/s (9508kB/s)(7168KiB/772msec) clat (msec): min=32, max=155, avg=84.10, stdev=47.46 lat (msec): min=32, max=155, avg=84.17, stdev=47.47 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 92], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 115], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min=10240, max=10240, per=30.54%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 50=33.33%, 100=25.00%, 250=25.00% cpu : usr=0.26%, sys=26.26%, ctx=539, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.4MiB/s (24.5MB/s), 2395KiB/s-11.6MiB/s (2453kB/s-12.2MB/s), io=20.0MiB (20.0MB), run=601-855msec WRITE: bw=32.7MiB/s (34.3MB/s), 8519KiB/s-11.7MiB/s (8724kB/s-12.3MB/s), io=28.0MiB (29.4MB), run=601-855msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1809: Mon Mar 10 15:23:02 2025 read: IOPS=12, BW=12.2MiB/s (12.8MB/s)(7168KiB/572msec) clat (usec): min=16018, max=54221, avg=31632.74, stdev=14016.34 lat (usec): min=16020, max=54223, avg=31634.33, stdev=14016.32 clat percentiles (usec): | 1.00th=[16057], 5.00th=[16057], 10.00th=[16057], 20.00th=[18744], | 30.00th=[22414], 40.00th=[22414], 50.00th=[28967], 60.00th=[38536], | 70.00th=[38536], 80.00th=[42206], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min=12288, max=12288, per=55.50%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8951KiB/s (9166kB/s)(5120KiB/572msec) clat (msec): min=30, max=105, avg=66.22, stdev=26.57 lat (msec): min=30, max=105, avg=66.28, stdev=26.57 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 67], 60.00th=[ 67], | 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 8192, max= 8192, per=26.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=14.89%, ctx=78, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1810: Mon Mar 10 15:23:02 2025 read: IOPS=2, BW=2381KiB/s (2439kB/s)(2048KiB/860msec) clat (msec): min=18, max=130, avg=74.50, stdev=79.49 lat (msec): min=18, max=130, avg=74.50, stdev=79.49 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 19], 60.00th=[ 131], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 4096, max= 4096, per=18.50%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.6MiB/s (12.2MB/s)(10.0MiB/860msec) clat (msec): min=32, max=158, avg=68.68, stdev=40.59 lat (msec): min=32, max=158, avg=68.77, stdev=40.58 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 55], 60.00th=[ 59], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 121], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min=10240, max=10240, per=33.04%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=12.69%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1811: Mon Mar 10 15:23:02 2025 read: IOPS=6, BW=6642KiB/s (6802kB/s)(6144KiB/925msec) clat (msec): min=33, max=163, avg=81.05, stdev=45.19 lat (msec): min=33, max=163, avg=81.05, stdev=45.19 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 78], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 6144, max= 6144, per=27.75%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6642KiB/s (6802kB/s)(6144KiB/925msec) clat (msec): min=34, max=151, avg=70.93, stdev=43.69 lat (msec): min=35, max=151, avg=71.01, stdev=43.69 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 68], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 8192, max= 8192, per=26.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=50.00%, 250=16.67% cpu : usr=0.11%, sys=10.05%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1812: Mon Mar 10 15:23:02 2025 read: IOPS=7, BW=7541KiB/s (7721kB/s)(5120KiB/679msec) clat (usec): min=17980, max=46189, avg=30250.77, stdev=10983.43 lat (usec): min=17981, max=46189, avg=30252.10, stdev=10983.30 clat percentiles (usec): | 1.00th=[17957], 5.00th=[17957], 10.00th=[17957], 20.00th=[17957], | 30.00th=[22152], 40.00th=[22152], 50.00th=[31851], 60.00th=[31851], | 70.00th=[33162], 80.00th=[33162], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 8192, max= 8192, per=37.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.3MiB/s (10.8MB/s)(7168KiB/679msec) clat (msec): min=34, max=122, avg=74.30, stdev=31.59 lat (msec): min=34, max=122, avg=74.38, stdev=31.58 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 51], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 91], | 70.00th=[ 91], 80.00th=[ 103], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min=10240, max=10240, per=33.04%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=16.08%, ctx=61, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.6MiB/s (22.7MB/s), 2381KiB/s-12.2MiB/s (2439kB/s-12.8MB/s), io=20.0MiB (20.0MB), run=572-925msec WRITE: bw=30.3MiB/s (31.7MB/s), 6642KiB/s-11.6MiB/s (6802kB/s-12.2MB/s), io=28.0MiB (29.4MB), run=572-925msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1826: Mon Mar 10 15:23:03 2025 read: IOPS=9, BW=37.3MiB/s (39.1MB/s)(12.0MiB/322msec) clat (msec): min=33, max=186, avg=102.62, stdev=77.36 lat (msec): min=33, max=186, avg=102.62, stdev=77.36 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 89], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=33.96%, ctx=1070, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1828: Mon Mar 10 15:23:03 2025 read: IOPS=1, BW=7367KiB/s (7544kB/s)(4096KiB/556msec) clat (nsec): min=90320k, max=90320k, avg=90319669.00, stdev= 0.00 lat (nsec): min=90322k, max=90322k, avg=90322236.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[90702], 5.00th=[90702], 10.00th=[90702], 20.00th=[90702], | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702], | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 8192, max= 8192, per=27.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=14.4MiB/s (15.1MB/s)(8192KiB/556msec) clat (msec): min=147, max=299, avg=223.57, stdev=107.25 lat (msec): min=148, max=299, avg=223.80, stdev=107.18 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=16.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=33.51%, ctx=513, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1830: Mon Mar 10 15:23:03 2025 write: IOPS=4, BW=18.7MiB/s (19.6MB/s)(12.0MiB/642msec) clat (msec): min=203, max=218, avg=211.14, stdev= 7.62 lat (msec): min=203, max=219, avg=211.42, stdev= 7.71 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min=16384, max=16384, per=32.10%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=33.85%, ctx=70, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1831: Mon Mar 10 15:23:03 2025 write: IOPS=4, BW=19.4MiB/s (20.3MB/s)(12.0MiB/619msec) clat (msec): min=162, max=267, avg=202.60, stdev=56.82 lat (msec): min=162, max=267, avg=202.90, stdev=56.81 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min=16351, max=16351, per=32.04%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=44.50%, ctx=87, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.8MiB/s (30.2MB/s), 7367KiB/s-37.3MiB/s (7544kB/s-39.1MB/s), io=16.0MiB (16.8MB), run=322-556msec WRITE: bw=49.8MiB/s (52.3MB/s), 14.4MiB/s-19.4MiB/s (15.1MB/s-20.3MB/s), io=32.0MiB (33.6MB), run=556-642msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1824: Mon Mar 10 15:23:04 2025 read: IOPS=3, BW=15.3MiB/s (16.1MB/s)(12.0MiB/782msec) clat (msec): min=85, max=346, avg=243.17, stdev=138.34 lat (msec): min=85, max=346, avg=243.18, stdev=138.34 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=39.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=12.04%, ctx=38, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1825: Mon Mar 10 15:23:04 2025 read: IOPS=1, BW=5498KiB/s (5630kB/s)(4096KiB/745msec) clat (nsec): min=360877k, max=360877k, avg=360876671.00, stdev= 0.00 lat (nsec): min=360878k, max=360878k, avg=360877751.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 5851, max= 5851, per=27.93%, avg=5851.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.7MiB/s (11.3MB/s)(8192KiB/745msec) clat (msec): min=109, max=259, avg=184.31, stdev=106.40 lat (msec): min=109, max=259, avg=184.62, stdev=106.47 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 259], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 5851, max= 5851, per=13.98%, avg=5851.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.27%, sys=14.38%, ctx=73, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1827: Mon Mar 10 15:23:04 2025 write: IOPS=3, BW=15.3MiB/s (16.1MB/s)(12.0MiB/783msec) clat (msec): min=78, max=346, avg=255.73, stdev=153.16 lat (msec): min=79, max=346, avg=256.12, stdev=153.13 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min=11409, max=11409, per=27.26%, avg=11409.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=9.97%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1829: Mon Mar 10 15:23:04 2025 write: IOPS=6, BW=24.1MiB/s (25.3MB/s)(12.0MiB/498msec) clat (msec): min=46, max=319, avg=155.05, stdev=144.59 lat (msec): min=46, max=319, avg=155.33, stdev=144.63 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 100], 50.00th=[ 100], 60.00th=[ 100], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.20%, sys=14.66%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.5MiB/s (21.5MB/s), 5498KiB/s-15.3MiB/s (5630kB/s-16.1MB/s), io=16.0MiB (16.8MB), run=745-782msec WRITE: bw=40.9MiB/s (42.9MB/s), 10.7MiB/s-24.1MiB/s (11.3MB/s-25.3MB/s), io=32.0MiB (33.6MB), run=498-783msec