== sanity test 398b: DIO and buffer IO race ============== 23:33:28 (1746156808) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.9066 s, 26.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4823: Thu May 1 23:34:10 2025 read: IOPS=39, BW=158KiB/s (162kB/s)(5992KiB/37968msec) clat (usec): min=63, max=255647, avg=11344.07, stdev=12251.85 lat (usec): min=63, max=255649, avg=11345.67, stdev=12251.89 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 219], | 30.00th=[ 5866], 40.00th=[ 7635], 50.00th=[ 9372], 60.00th=[ 11731], | 70.00th=[ 14353], 80.00th=[ 17957], 90.00th=[ 23725], 95.00th=[ 28705], | 99.00th=[ 45351], 99.50th=[ 52691], 99.90th=[141558], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 71, max= 344, per=24.58%, avg=155.36, stdev=55.61, samples=75 iops : min= 17, max= 86, avg=38.48, stdev=14.03, samples=75 write: IOPS=41, BW=166KiB/s (170kB/s)(6296KiB/37968msec) clat (usec): min=473, max=223432, avg=13237.92, stdev=14573.76 lat (usec): min=474, max=223432, avg=13239.55, stdev=14573.69 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 816], 20.00th=[ 1172], | 30.00th=[ 2442], 40.00th=[ 6128], 50.00th=[ 9765], 60.00th=[ 13173], | 70.00th=[ 17433], 80.00th=[ 22676], 90.00th=[ 30802], 95.00th=[ 39060], | 99.00th=[ 64226], 99.50th=[ 74974], 99.90th=[ 88605], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 71, max= 352, per=24.66%, avg=163.23, stdev=61.90, samples=75 iops : min= 17, max= 88, avg=40.45, stdev=15.59, samples=75 lat (usec) : 100=3.19%, 250=6.74%, 500=0.29%, 750=2.80%, 1000=6.61% lat (msec) : 2=5.31%, 4=4.69%, 10=22.49%, 20=27.90%, 50=18.62% lat (msec) : 100=1.24%, 250=0.10%, 500=0.03% cpu : usr=0.22%, sys=12.04%, ctx=5093, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4824: Thu May 1 23:34:10 2025 read: IOPS=39, BW=157KiB/s (160kB/s)(5940KiB/37931msec) clat (usec): min=61, max=212162, avg=10940.59, stdev=11925.58 lat (usec): min=62, max=212164, avg=10945.75, stdev=11929.49 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 96], 10.00th=[ 110], 20.00th=[ 153], | 30.00th=[ 5735], 40.00th=[ 7373], 50.00th=[ 8979], 60.00th=[ 10683], | 70.00th=[ 13304], 80.00th=[ 16909], 90.00th=[ 23725], 95.00th=[ 29230], | 99.00th=[ 45876], 99.50th=[ 57410], 99.90th=[168821], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 8, max= 336, per=24.56%, avg=155.24, stdev=63.53, samples=75 iops : min= 2, max= 84, avg=38.53, stdev=16.00, samples=75 write: IOPS=41, BW=167KiB/s (171kB/s)(6348KiB/37931msec) clat (usec): min=537, max=210899, avg=13542.33, stdev=15565.90 lat (usec): min=538, max=210900, avg=13546.82, stdev=15565.53 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 1188], | 30.00th=[ 2474], 40.00th=[ 5866], 50.00th=[ 9634], 60.00th=[ 12780], | 70.00th=[ 17171], 80.00th=[ 22938], 90.00th=[ 31851], 95.00th=[ 40109], | 99.00th=[ 62129], 99.50th=[ 90702], 99.90th=[152044], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 24, max= 336, per=24.84%, avg=164.41, stdev=56.38, samples=75 iops : min= 6, max= 84, avg=40.83, stdev=14.15, samples=75 lat (usec) : 100=3.09%, 250=7.29%, 500=0.36%, 750=2.51%, 1000=6.58% lat (msec) : 2=5.24%, 4=4.52%, 10=23.80%, 20=26.99%, 50=17.84% lat (msec) : 100=1.53%, 250=0.26% cpu : usr=0.13%, sys=12.01%, ctx=4918, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4825: Thu May 1 23:34:10 2025 read: IOPS=39, BW=159KiB/s (163kB/s)(6028KiB/37901msec) clat (usec): min=67, max=90773, avg=11009.92, stdev=9499.75 lat (usec): min=68, max=90775, avg=11011.23, stdev=9499.70 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 98], 10.00th=[ 112], 20.00th=[ 586], | 30.00th=[ 6194], 40.00th=[ 7832], 50.00th=[ 9503], 60.00th=[11469], | 70.00th=[13960], 80.00th=[16909], 90.00th=[22414], 95.00th=[28443], | 99.00th=[39060], 99.50th=[51119], 99.90th=[68682], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 71, max= 303, per=25.42%, avg=160.64, stdev=53.68, samples=74 iops : min= 17, max= 75, avg=39.84, stdev=13.45, samples=74 write: IOPS=41, BW=165KiB/s (169kB/s)(6260KiB/37901msec) clat (usec): min=508, max=269466, avg=13504.88, stdev=16368.73 lat (usec): min=509, max=269469, avg=13507.62, stdev=16368.70 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 1090], | 30.00th=[ 2409], 40.00th=[ 5997], 50.00th=[ 9634], 60.00th=[ 13173], | 70.00th=[ 17433], 80.00th=[ 22938], 90.00th=[ 31589], 95.00th=[ 39060], | 99.00th=[ 61080], 99.50th=[ 66323], 99.90th=[225444], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 15, max= 320, per=24.81%, avg=164.24, stdev=54.19, samples=75 iops : min= 3, max= 80, avg=40.73, stdev=13.61, samples=75 lat (usec) : 100=2.96%, 250=6.51%, 500=0.23%, 750=3.12%, 1000=6.80% lat (msec) : 2=4.59%, 4=4.39%, 10=23.24%, 20=28.52%, 50=18.29% lat (msec) : 100=1.24%, 250=0.07%, 500=0.03% cpu : usr=0.21%, sys=11.99%, ctx=5055, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4826: Thu May 1 23:34:10 2025 read: IOPS=39, BW=159KiB/s (163kB/s)(6052KiB/37956msec) clat (usec): min=62, max=249693, avg=10916.72, stdev=11772.74 lat (usec): min=62, max=249694, avg=10917.92, stdev=11773.36 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 93], 10.00th=[ 108], 20.00th=[ 161], | 30.00th=[ 5669], 40.00th=[ 7439], 50.00th=[ 9110], 60.00th=[ 11076], | 70.00th=[ 13698], 80.00th=[ 17171], 90.00th=[ 22676], 95.00th=[ 28443], | 99.00th=[ 40633], 99.50th=[ 63177], 99.90th=[108528], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 88, max= 320, per=25.24%, avg=159.51, stdev=51.01, samples=74 iops : min= 22, max= 80, avg=39.59, stdev=12.81, samples=74 write: IOPS=41, BW=164KiB/s (168kB/s)(6236KiB/37956msec) clat (usec): min=482, max=285836, avg=13648.76, stdev=15607.27 lat (usec): min=482, max=285838, avg=13651.45, stdev=15606.78 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 1188], | 30.00th=[ 2409], 40.00th=[ 5735], 50.00th=[ 9896], 60.00th=[ 13698], | 70.00th=[ 17695], 80.00th=[ 23200], 90.00th=[ 32375], 95.00th=[ 40633], | 99.00th=[ 60556], 99.50th=[ 76022], 99.90th=[105382], 99.95th=[287310], | 99.99th=[287310] bw ( KiB/s): min= 24, max= 320, per=24.50%, avg=162.19, stdev=52.95, samples=75 iops : min= 6, max= 80, avg=40.27, stdev=13.28, samples=75 lat (usec) : 100=3.71%, 250=6.74%, 500=0.52%, 750=2.60%, 1000=6.45% lat (msec) : 2=5.37%, 4=4.82%, 10=22.10%, 20=27.86%, 50=18.26% lat (msec) : 100=1.43%, 250=0.10%, 500=0.03% cpu : usr=0.20%, sys=11.81%, ctx=5036, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=632KiB/s (648kB/s), 157KiB/s-159KiB/s (160kB/s-163kB/s), io=23.4MiB (24.6MB), run=37901-37968msec WRITE: bw=662KiB/s (678kB/s), 164KiB/s-167KiB/s (168kB/s-171kB/s), io=24.6MiB (25.7MB), run=37901-37968msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4818: Thu May 1 23:35:49 2025 read: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5992KiB/134391msec) clat (msec): min=3, max=265, avg=13.21, stdev=11.53 lat (msec): min=3, max=265, avg=13.21, stdev=11.53 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 33], | 99.00th=[ 46], 99.50th=[ 51], 99.90th=[ 100], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 7, max= 135, per=26.06%, avg=45.61, stdev=25.59, samples=260 iops : min= 1, max= 33, avg=11.11, stdev= 6.39, samples=260 write: IOPS=11, BW=46.8KiB/s (47.0kB/s)(6296KiB/134391msec) clat (msec): min=23, max=290, avg=72.69, stdev=28.49 lat (msec): min=23, max=290, avg=72.69, stdev=28.49 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 68], 60.00th=[ 74], | 70.00th=[ 82], 80.00th=[ 93], 90.00th=[ 108], 95.00th=[ 128], | 99.00th=[ 165], 99.50th=[ 184], 99.90th=[ 226], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 7, max= 80, per=25.24%, avg=46.44, stdev=13.59, samples=268 iops : min= 1, max= 20, avg=11.32, stdev= 3.45, samples=268 lat (msec) : 4=0.42%, 10=26.11%, 20=12.50%, 50=19.76%, 100=34.31% lat (msec) : 250=6.84%, 500=0.07% cpu : usr=0.09%, sys=2.70%, ctx=3789, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4819: Thu May 1 23:35:49 2025 read: IOPS=10, BW=43.7KiB/s (44.8kB/s)(5940KiB/135912msec) clat (msec): min=3, max=257, avg=13.37, stdev=11.49 lat (msec): min=3, max=257, avg=13.37, stdev=11.49 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 20], 90.00th=[ 27], 95.00th=[ 33], | 99.00th=[ 49], 99.50th=[ 55], 99.90th=[ 84], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 7, max= 144, per=25.31%, avg=44.29, stdev=25.64, samples=265 iops : min= 1, max= 36, avg=10.78, stdev= 6.47, samples=265 write: IOPS=11, BW=46.7KiB/s (47.8kB/s)(6348KiB/135912msec) clat (msec): min=24, max=250, avg=73.02, stdev=29.24 lat (msec): min=24, max=250, avg=73.02, stdev=29.24 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 37], 10.00th=[ 42], 20.00th=[ 50], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 68], 60.00th=[ 75], | 70.00th=[ 82], 80.00th=[ 93], 90.00th=[ 110], 95.00th=[ 129], | 99.00th=[ 174], 99.50th=[ 197], 99.90th=[ 230], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 15, max= 87, per=25.10%, avg=46.18, stdev=13.98, samples=271 iops : min= 3, max= 21, avg=11.25, stdev= 3.55, samples=271 lat (msec) : 4=0.65%, 10=25.68%, 20=12.43%, 50=19.89%, 100=33.50% lat (msec) : 250=7.78%, 500=0.07% cpu : usr=0.04%, sys=2.72%, ctx=3837, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4820: Thu May 1 23:35:49 2025 read: IOPS=11, BW=44.2KiB/s (45.2kB/s)(6028KiB/136482msec) clat (msec): min=3, max=116, avg=14.12, stdev=11.59 lat (msec): min=3, max=116, avg=14.13, stdev=11.59 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 36], | 99.00th=[ 63], 99.50th=[ 81], 99.90th=[ 94], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 7, max= 144, per=25.52%, avg=44.67, stdev=25.16, samples=266 iops : min= 1, max= 36, avg=10.86, stdev= 6.30, samples=266 write: IOPS=11, BW=45.9KiB/s (46.0kB/s)(6260KiB/136482msec) clat (msec): min=25, max=311, avg=73.49, stdev=29.20 lat (msec): min=25, max=311, avg=73.49, stdev=29.20 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 50], | 30.00th=[ 57], 40.00th=[ 62], 50.00th=[ 68], 60.00th=[ 75], | 70.00th=[ 84], 80.00th=[ 93], 90.00th=[ 110], 95.00th=[ 126], | 99.00th=[ 169], 99.50th=[ 186], 99.90th=[ 226], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 15, max= 80, per=24.60%, avg=45.27, stdev=14.17, samples=272 iops : min= 3, max= 20, avg=11.02, stdev= 3.62, samples=272 lat (msec) : 4=0.42%, 10=24.74%, 20=13.74%, 50=19.66%, 100=33.79% lat (msec) : 250=7.62%, 500=0.03% cpu : usr=0.09%, sys=2.72%, ctx=3847, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4821: Thu May 1 23:35:49 2025 read: IOPS=11, BW=44.9KiB/s (45.9kB/s)(6052KiB/134930msec) clat (usec): min=3501, max=95251, avg=13847.70, stdev=10871.42 lat (usec): min=3503, max=95252, avg=13849.35, stdev=10871.57 clat percentiles (usec): | 1.00th=[ 4015], 5.00th=[ 4948], 10.00th=[ 5538], 20.00th=[ 6456], | 30.00th=[ 7373], 40.00th=[ 8356], 50.00th=[ 9634], 60.00th=[11207], | 70.00th=[15008], 80.00th=[20579], 90.00th=[27919], 95.00th=[33817], | 99.00th=[55313], 99.50th=[62653], 99.90th=[91751], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 7, max= 151, per=26.41%, avg=46.22, stdev=25.18, samples=259 iops : min= 1, max= 37, avg=11.27, stdev= 6.32, samples=259 write: IOPS=11, BW=46.2KiB/s (47.3kB/s)(6236KiB/134930msec) clat (msec): min=23, max=310, avg=72.99, stdev=28.78 lat (msec): min=23, max=310, avg=73.00, stdev=28.78 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 44], 20.00th=[ 50], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 68], 60.00th=[ 74], | 70.00th=[ 82], 80.00th=[ 93], 90.00th=[ 111], 95.00th=[ 128], | 99.00th=[ 165], 99.50th=[ 178], 99.90th=[ 249], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8, max= 80, per=24.89%, avg=45.79, stdev=13.51, samples=269 iops : min= 2, max= 20, avg=11.15, stdev= 3.43, samples=269 lat (msec) : 4=0.49%, 10=25.33%, 20=12.79%, 50=20.51%, 100=33.27% lat (msec) : 250=7.58%, 500=0.03% cpu : usr=0.04%, sys=2.78%, ctx=3844, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=176KiB/s (180kB/s), 43.7KiB/s-44.9KiB/s (44.8kB/s-45.9kB/s), io=23.4MiB (24.6MB), run=134391-136482msec WRITE: bw=184KiB/s (189kB/s), 45.9KiB/s-46.8KiB/s (46.0kB/s-47.0kB/s), io=24.6MiB (25.7MB), run=134391-136482msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4948: Thu May 1 23:35:56 2025 read: IOPS=61, BW=989KiB/s (1012kB/s)(5920KiB/5988msec) clat (usec): min=52, max=62264, avg=6927.08, stdev=8291.48 lat (usec): min=52, max=62265, avg=6928.81, stdev=8291.87 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 91], 10.00th=[ 99], 20.00th=[ 112], | 30.00th=[ 137], 40.00th=[ 4047], 50.00th=[ 6128], 60.00th=[ 7308], | 70.00th=[ 9372], 80.00th=[11600], 90.00th=[15401], 95.00th=[19268], | 99.00th=[50594], 99.50th=[59507], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 223, max= 1568, per=26.18%, avg=985.27, stdev=386.46, samples=11 iops : min= 13, max= 98, avg=61.18, stdev=24.44, samples=11 write: IOPS=66, BW=1063KiB/s (1089kB/s)(6368KiB/5988msec) clat (usec): min=620, max=91087, avg=8367.25, stdev=10927.27 lat (usec): min=622, max=91089, avg=8370.30, stdev=10928.08 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 996], 10.00th=[ 1074], 20.00th=[ 1254], | 30.00th=[ 1565], 40.00th=[ 2073], 50.00th=[ 3097], 60.00th=[ 6325], | 70.00th=[10290], 80.00th=[15139], 90.00th=[21365], 95.00th=[27919], | 99.00th=[51119], 99.50th=[80217], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 159, max= 2048, per=26.94%, avg=1066.64, stdev=478.78, samples=11 iops : min= 9, max= 128, avg=66.27, stdev=30.21, samples=11 lat (usec) : 100=5.34%, 250=11.07%, 500=1.17%, 750=0.52%, 1000=2.60% lat (msec) : 2=18.49%, 4=7.55%, 10=25.00%, 20=19.92%, 50=7.29% lat (msec) : 100=1.04% cpu : usr=0.20%, sys=16.65%, ctx=1149, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4949: Thu May 1 23:35:56 2025 read: IOPS=56, BW=903KiB/s (924kB/s)(5744KiB/6363msec) clat (usec): min=54, max=70071, avg=7202.99, stdev=7759.03 lat (usec): min=54, max=70073, avg=7210.56, stdev=7758.69 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 90], 10.00th=[ 103], 20.00th=[ 121], | 30.00th=[ 202], 40.00th=[ 4555], 50.00th=[ 6521], 60.00th=[ 7898], | 70.00th=[10028], 80.00th=[12387], 90.00th=[15533], 95.00th=[19006], | 99.00th=[35914], 99.50th=[41681], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 64, max= 1500, per=24.26%, avg=913.25, stdev=390.53, samples=12 iops : min= 4, max= 93, avg=56.75, stdev=24.37, samples=12 write: IOPS=64, BW=1028KiB/s (1053kB/s)(6544KiB/6363msec) clat (usec): min=835, max=85248, avg=8925.19, stdev=12740.74 lat (usec): min=836, max=85249, avg=8926.78, stdev=12740.77 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1004], 10.00th=[ 1090], 20.00th=[ 1287], | 30.00th=[ 1647], 40.00th=[ 2343], 50.00th=[ 3851], 60.00th=[ 6456], | 70.00th=[ 9765], 80.00th=[14222], 90.00th=[20579], 95.00th=[29230], | 99.00th=[71828], 99.50th=[76022], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 224, max= 1717, per=25.74%, avg=1019.50, stdev=404.57, samples=12 iops : min= 14, max= 107, avg=63.42, stdev=25.29, samples=12 lat (usec) : 100=4.17%, 250=10.16%, 500=1.04%, 750=0.65%, 1000=2.86% lat (msec) : 2=17.45%, 4=8.33%, 10=25.65%, 20=22.01%, 50=6.12% lat (msec) : 100=1.56% cpu : usr=0.13%, sys=15.86%, ctx=1165, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4951: Thu May 1 23:35:56 2025 read: IOPS=72, BW=1165KiB/s (1193kB/s)(6256KiB/5369msec) clat (usec): min=52, max=107516, avg=7206.11, stdev=8974.59 lat (usec): min=53, max=107517, avg=7207.05, stdev=8974.60 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 96], 10.00th=[ 103], 20.00th=[ 119], | 30.00th=[ 178], 40.00th=[ 5211], 50.00th=[ 6390], 60.00th=[ 7635], | 70.00th=[ 9372], 80.00th=[ 11469], 90.00th=[ 15139], 95.00th=[ 19268], | 99.00th=[ 28705], 99.50th=[ 80217], 99.90th=[107480], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 768, max= 1884, per=31.57%, avg=1188.30, stdev=326.86, samples=10 iops : min= 48, max= 117, avg=73.70, stdev=20.23, samples=10 write: IOPS=70, BW=1123KiB/s (1150kB/s)(6032KiB/5369msec) clat (usec): min=782, max=39337, avg=6685.71, stdev=7586.52 lat (usec): min=783, max=39339, avg=6687.18, stdev=7586.57 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 996], 10.00th=[ 1090], 20.00th=[ 1221], | 30.00th=[ 1500], 40.00th=[ 1958], 50.00th=[ 3032], 60.00th=[ 5276], | 70.00th=[ 7504], 80.00th=[12125], 90.00th=[18744], 95.00th=[21365], | 99.00th=[36963], 99.50th=[38536], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 734, max= 1373, per=28.15%, avg=1114.90, stdev=186.03, samples=10 iops : min= 45, max= 85, avg=69.10, stdev=11.68, samples=10 lat (usec) : 100=3.65%, 250=12.24%, 500=0.78%, 750=0.13%, 1000=2.60% lat (msec) : 2=17.84%, 4=7.94%, 10=29.30%, 20=19.92%, 50=5.34% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.26%, sys=18.26%, ctx=1184, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4952: Thu May 1 23:35:56 2025 read: IOPS=62, BW=1006KiB/s (1030kB/s)(6032KiB/5996msec) clat (usec): min=52, max=71864, avg=7083.44, stdev=8524.07 lat (usec): min=52, max=71864, avg=7095.70, stdev=8517.62 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 114], | 30.00th=[ 133], 40.00th=[ 4228], 50.00th=[ 6325], 60.00th=[ 7832], | 70.00th=[ 8717], 80.00th=[11731], 90.00th=[16057], 95.00th=[20317], | 99.00th=[43779], 99.50th=[54264], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 288, max= 1408, per=26.80%, avg=1008.73, stdev=313.95, samples=11 iops : min= 18, max= 88, avg=62.82, stdev=19.53, samples=11 write: IOPS=65, BW=1043KiB/s (1068kB/s)(6256KiB/5996msec) clat (usec): min=623, max=66421, avg=8166.61, stdev=10015.07 lat (usec): min=625, max=66423, avg=8171.04, stdev=10013.31 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 996], 10.00th=[ 1106], 20.00th=[ 1270], | 30.00th=[ 1598], 40.00th=[ 2245], 50.00th=[ 3982], 60.00th=[ 6718], | 70.00th=[ 9634], 80.00th=[14222], 90.00th=[19792], 95.00th=[25035], | 99.00th=[55313], 99.50th=[64226], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 352, max= 1600, per=26.87%, avg=1064.09, stdev=353.91, samples=11 iops : min= 22, max= 100, avg=66.27, stdev=22.11, samples=11 lat (usec) : 100=4.69%, 250=12.37%, 500=0.52%, 750=0.91%, 1000=2.73% lat (msec) : 2=16.67%, 4=7.16%, 10=28.39%, 20=19.14%, 50=6.51% lat (msec) : 100=0.91% cpu : usr=0.08%, sys=16.81%, ctx=1127, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3764KiB/s (3855kB/s), 903KiB/s-1165KiB/s (924kB/s-1193kB/s), io=23.4MiB (24.5MB), run=5369-6363msec WRITE: bw=3960KiB/s (4055kB/s), 1028KiB/s-1123KiB/s (1053kB/s-1150kB/s), io=24.6MiB (25.8MB), run=5369-6363msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4950: Thu May 1 23:36:22 2025 read: IOPS=11, BW=186KiB/s (191kB/s)(5920KiB/31780msec) clat (usec): min=2826, max=59738, avg=10882.79, stdev=7798.89 lat (usec): min=2827, max=59739, avg=10886.48, stdev=7798.42 clat percentiles (usec): | 1.00th=[ 3818], 5.00th=[ 4621], 10.00th=[ 5342], 20.00th=[ 6390], | 30.00th=[ 6980], 40.00th=[ 7701], 50.00th=[ 8356], 60.00th=[ 9372], | 70.00th=[10814], 80.00th=[12911], 90.00th=[19268], 95.00th=[27919], | 99.00th=[43254], 99.50th=[54264], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 63, max= 574, per=24.67%, avg=184.06, stdev=98.50, samples=63 iops : min= 3, max= 35, avg=11.24, stdev= 6.08, samples=63 write: IOPS=12, BW=200KiB/s (205kB/s)(6368KiB/31780msec) clat (msec): min=23, max=195, avg=69.56, stdev=23.13 lat (msec): min=23, max=195, avg=69.57, stdev=23.13 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 38], 10.00th=[ 43], 20.00th=[ 50], | 30.00th=[ 56], 40.00th=[ 61], 50.00th=[ 67], 60.00th=[ 72], | 70.00th=[ 81], 80.00th=[ 90], 90.00th=[ 99], 95.00th=[ 108], | 99.00th=[ 138], 99.50th=[ 171], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 127, max= 351, per=25.24%, avg=198.11, stdev=40.55, samples=63 iops : min= 7, max= 21, avg=12.10, stdev= 2.54, samples=63 lat (msec) : 4=0.78%, 10=30.60%, 20=12.24%, 50=14.97%, 100=37.24% lat (msec) : 250=4.17% cpu : usr=0.08%, sys=2.83%, ctx=893, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4953: Thu May 1 23:36:22 2025 read: IOPS=11, BW=179KiB/s (183kB/s)(5744KiB/32085msec) clat (usec): min=3475, max=94216, avg=10969.31, stdev=9439.64 lat (usec): min=3476, max=94217, avg=10970.80, stdev=9439.58 clat percentiles (usec): | 1.00th=[ 3982], 5.00th=[ 4555], 10.00th=[ 5080], 20.00th=[ 6128], | 30.00th=[ 6980], 40.00th=[ 7570], 50.00th=[ 8291], 60.00th=[ 9110], | 70.00th=[10683], 80.00th=[12649], 90.00th=[19006], 95.00th=[26608], | 99.00th=[54264], 99.50th=[64750], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 32, max= 448, per=23.82%, avg=177.72, stdev=92.25, samples=64 iops : min= 2, max= 28, avg=10.88, stdev= 5.80, samples=64 write: IOPS=12, BW=204KiB/s (209kB/s)(6544KiB/32085msec) clat (msec): min=24, max=146, avg=68.69, stdev=23.54 lat (msec): min=24, max=146, avg=68.69, stdev=23.54 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 61], 50.00th=[ 67], 60.00th=[ 72], | 70.00th=[ 80], 80.00th=[ 89], 90.00th=[ 101], 95.00th=[ 112], | 99.00th=[ 136], 99.50th=[ 142], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 127, max= 383, per=25.89%, avg=203.23, stdev=51.75, samples=64 iops : min= 7, max= 23, avg=12.47, stdev= 3.25, samples=64 lat (msec) : 4=0.52%, 10=30.60%, 20=11.33%, 50=15.62%, 100=36.59% lat (msec) : 250=5.34% cpu : usr=0.07%, sys=2.73%, ctx=884, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4954: Thu May 1 23:36:22 2025 read: IOPS=12, BW=204KiB/s (209kB/s)(6256KiB/30723msec) clat (usec): min=3003, max=87406, avg=10855.31, stdev=8832.22 lat (usec): min=3004, max=87407, avg=10857.70, stdev=8832.31 clat percentiles (usec): | 1.00th=[ 3687], 5.00th=[ 4817], 10.00th=[ 5407], 20.00th=[ 6194], | 30.00th=[ 6980], 40.00th=[ 7635], 50.00th=[ 8586], 60.00th=[ 9372], | 70.00th=[10683], 80.00th=[12518], 90.00th=[17171], 95.00th=[24511], | 99.00th=[61080], 99.50th=[64226], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 31, max= 479, per=27.46%, avg=204.85, stdev=102.30, samples=61 iops : min= 1, max= 29, avg=12.56, stdev= 6.29, samples=61 write: IOPS=12, BW=196KiB/s (201kB/s)(6032KiB/30723msec) clat (msec): min=23, max=188, avg=69.88, stdev=24.32 lat (msec): min=23, max=188, avg=69.88, stdev=24.32 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 38], 10.00th=[ 44], 20.00th=[ 50], | 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 67], 60.00th=[ 72], | 70.00th=[ 81], 80.00th=[ 90], 90.00th=[ 102], 95.00th=[ 110], | 99.00th=[ 148], 99.50th=[ 176], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 96, max= 352, per=24.89%, avg=195.41, stdev=45.70, samples=61 iops : min= 6, max= 22, avg=11.97, stdev= 2.92, samples=61 lat (msec) : 4=0.91%, 10=32.03%, 20=14.06%, 50=13.80%, 100=34.11% lat (msec) : 250=5.08% cpu : usr=0.02%, sys=2.92%, ctx=893, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4955: Thu May 1 23:36:22 2025 read: IOPS=11, BW=190KiB/s (194kB/s)(6032KiB/31831msec) clat (usec): min=3258, max=87404, avg=10614.25, stdev=8366.33 lat (usec): min=3259, max=87406, avg=10615.88, stdev=8366.34 clat percentiles (usec): | 1.00th=[ 3621], 5.00th=[ 4621], 10.00th=[ 5276], 20.00th=[ 6194], | 30.00th=[ 6783], 40.00th=[ 7504], 50.00th=[ 8356], 60.00th=[ 9241], | 70.00th=[10290], 80.00th=[12387], 90.00th=[17433], 95.00th=[25560], | 99.00th=[52691], 99.50th=[69731], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 31, max= 479, per=25.12%, avg=187.43, stdev=94.83, samples=63 iops : min= 1, max= 29, avg=11.29, stdev= 5.93, samples=63 write: IOPS=12, BW=197KiB/s (201kB/s)(6256KiB/31831msec) clat (msec): min=28, max=155, avg=70.90, stdev=24.52 lat (msec): min=28, max=155, avg=70.90, stdev=24.52 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 37], 10.00th=[ 43], 20.00th=[ 50], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 75], | 70.00th=[ 82], 80.00th=[ 91], 90.00th=[ 106], 95.00th=[ 117], | 99.00th=[ 142], 99.50th=[ 153], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 95, max= 320, per=24.92%, avg=195.59, stdev=43.62, samples=63 iops : min= 5, max= 20, avg=11.79, stdev= 2.80, samples=63 lat (msec) : 4=0.65%, 10=31.90%, 20=12.63%, 50=14.06%, 100=33.98% lat (msec) : 250=6.77% cpu : usr=0.05%, sys=2.94%, ctx=899, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=747KiB/s (764kB/s), 179KiB/s-204KiB/s (183kB/s-209kB/s), io=23.4MiB (24.5MB), run=30723-32085msec WRITE: bw=785KiB/s (804kB/s), 196KiB/s-204KiB/s (201kB/s-209kB/s), io=24.6MiB (25.8MB), run=30723-32085msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4991: Thu May 1 23:36:25 2025 read: IOPS=11, BW=11.9MiB/s (12.5MB/s)(7168KiB/588msec) clat (usec): min=564, max=83920, avg=50125.29, stdev=34882.05 lat (usec): min=566, max=83924, avg=50126.97, stdev=34882.20 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[ 586], | 30.00th=[57410], 40.00th=[57410], 50.00th=[62653], 60.00th=[70779], | 70.00th=[70779], 80.00th=[74974], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=12239, max=12239, per=84.98%, avg=12239.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8707KiB/s (8916kB/s)(5120KiB/588msec) clat (usec): min=22305, max=78421, avg=42073.14, stdev=21646.96 lat (usec): min=22325, max=78644, avg=42171.88, stdev=21726.58 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[22414], | 30.00th=[30016], 40.00th=[30016], 50.00th=[37487], 60.00th=[37487], | 70.00th=[41681], 80.00th=[41681], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 8159, max= 8159, per=40.47%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=33.33%, 100=50.00% cpu : usr=0.00%, sys=35.26%, ctx=510, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4992: Thu May 1 23:36:25 2025 read: IOPS=1, BW=1440KiB/s (1475kB/s)(2048KiB/1422msec) clat (usec): min=55756, max=65511, avg=60634.12, stdev=6898.21 lat (usec): min=55758, max=65515, avg=60636.98, stdev=6899.75 clat percentiles (usec): | 1.00th=[55837], 5.00th=[55837], 10.00th=[55837], 20.00th=[55837], | 30.00th=[55837], 40.00th=[55837], 50.00th=[55837], 60.00th=[65274], | 70.00th=[65274], 80.00th=[65274], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 2043, max= 2048, per=14.20%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=7, BW=7201KiB/s (7374kB/s)(10.0MiB/1422msec) clat (msec): min=43, max=286, avg=117.70, stdev=70.32 lat (msec): min=43, max=286, avg=117.78, stdev=70.31 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 58], | 30.00th=[ 79], 40.00th=[ 94], 50.00th=[ 100], 60.00th=[ 102], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 182], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 4087, max= 6144, per=25.37%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=8.33%, 100=50.00%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=18.58%, ctx=138, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4993: Thu May 1 23:36:25 2025 read: IOPS=5, BW=5689KiB/s (5825kB/s)(6144KiB/1080msec) clat (usec): min=612, max=114115, avg=52712.50, stdev=45261.07 lat (usec): min=614, max=114116, avg=52713.88, stdev=45260.96 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 611], 10.00th=[ 611], 20.00th=[ 693], | 30.00th=[ 693], 40.00th=[ 50070], 50.00th=[ 50070], 60.00th=[ 71828], | 70.00th=[ 78119], 80.00th=[ 78119], 90.00th=[113771], 95.00th=[113771], | 99.00th=[113771], 99.50th=[113771], 99.90th=[113771], 99.95th=[113771], | 99.99th=[113771] bw ( KiB/s): min= 8192, max= 8192, per=56.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5689KiB/s (5825kB/s)(6144KiB/1080msec) clat (msec): min=34, max=194, avg=103.57, stdev=54.81 lat (msec): min=34, max=195, avg=103.84, stdev=54.89 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 85], 50.00th=[ 85], 60.00th=[ 117], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 194], 95.00th=[ 194], | 99.00th=[ 194], 99.50th=[ 194], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 4096, max= 6144, per=25.39%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=8.33%, 100=41.67%, 250=33.33% cpu : usr=0.00%, sys=17.15%, ctx=425, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4994: Thu May 1 23:36:25 2025 read: IOPS=3, BW=3636KiB/s (3724kB/s)(5120KiB/1408msec) clat (usec): min=619, max=126107, avg=42092.70, stdev=53381.82 lat (usec): min=621, max=126108, avg=42094.30, stdev=53381.81 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 619], 10.00th=[ 619], 20.00th=[ 619], | 30.00th=[ 3163], 40.00th=[ 3163], 50.00th=[ 16909], 60.00th=[ 16909], | 70.00th=[ 63701], 80.00th=[ 63701], 90.00th=[126354], 95.00th=[126354], | 99.00th=[126354], 99.50th=[126354], 99.90th=[126354], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 8175, max= 8175, per=56.76%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5091KiB/s (5213kB/s)(7168KiB/1408msec) clat (msec): min=50, max=272, avg=154.00, stdev=70.86 lat (msec): min=50, max=273, avg=154.10, stdev=70.86 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 92], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 2043, max= 6144, per=20.30%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 20=8.33%, 100=25.00%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=15.00%, ctx=234, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.1MiB/s (14.7MB/s), 1440KiB/s-11.9MiB/s (1475kB/s-12.5MB/s), io=20.0MiB (20.0MB), run=588-1422msec WRITE: bw=19.7MiB/s (20.6MB/s), 5091KiB/s-8707KiB/s (5213kB/s-8916kB/s), io=28.0MiB (29.4MB), run=588-1422msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4986: Thu May 1 23:36:25 2025 read: IOPS=5, BW=5120KiB/s (5243kB/s)(7168KiB/1400msec) clat (msec): min=42, max=163, avg=88.67, stdev=40.28 lat (msec): min=42, max=163, avg=88.67, stdev=40.28 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 53], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 92], 60.00th=[ 93], | 70.00th=[ 93], 80.00th=[ 107], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 4096, max= 6083, per=37.10%, avg=5089.50, stdev=1405.02, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3657KiB/s (3745kB/s)(5120KiB/1400msec) clat (msec): min=86, max=267, avg=139.34, stdev=74.79 lat (msec): min=86, max=267, avg=139.44, stdev=74.82 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 96], 40.00th=[ 96], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 4096, max= 4096, per=21.33%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=8.33%, 100=50.00%, 250=33.33%, 500=8.33% cpu : usr=0.07%, sys=3.72%, ctx=40, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4987: Thu May 1 23:36:25 2025 read: IOPS=1, BW=1508KiB/s (1544kB/s)(2048KiB/1358msec) clat (usec): min=51642, max=73729, avg=62686.08, stdev=15618.34 lat (usec): min=51644, max=73731, avg=62687.87, stdev=15617.72 clat percentiles (usec): | 1.00th=[51643], 5.00th=[51643], 10.00th=[51643], 20.00th=[51643], | 30.00th=[51643], 40.00th=[51643], 50.00th=[51643], 60.00th=[73925], | 70.00th=[73925], 80.00th=[73925], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 2043, max= 2048, per=14.91%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=7, BW=7541KiB/s (7721kB/s)(10.0MiB/1358msec) clat (msec): min=54, max=273, avg=116.47, stdev=63.71 lat (msec): min=54, max=273, avg=116.73, stdev=63.69 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 86], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 171], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 6131, max= 6144, per=31.96%, avg=6137.50, stdev= 9.19, samples=2 iops : min= 5, max= 6, avg= 5.50, stdev= 0.71, samples=2 lat (msec) : 100=75.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=4.86%, ctx=40, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4988: Thu May 1 23:36:25 2025 read: IOPS=4, BW=4115KiB/s (4214kB/s)(6144KiB/1493msec) clat (msec): min=17, max=364, avg=94.06, stdev=133.84 lat (msec): min=17, max=364, avg=94.06, stdev=133.84 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 56], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 4087, max= 4087, per=29.80%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4115KiB/s (4214kB/s)(6144KiB/1493msec) clat (msec): min=71, max=187, avg=139.39, stdev=48.66 lat (msec): min=71, max=187, avg=139.51, stdev=48.67 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 6107, max= 6107, per=31.80%, avg=6107.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=16.67%, 100=33.33%, 250=33.33%, 500=8.33% cpu : usr=0.13%, sys=3.62%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4989: Thu May 1 23:36:25 2025 read: IOPS=3, BW=3441KiB/s (3523kB/s)(5120KiB/1488msec) clat (msec): min=42, max=106, avg=71.87, stdev=31.05 lat (msec): min=42, max=106, avg=71.87, stdev=31.04 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 63], 60.00th=[ 63], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8192, max= 8192, per=59.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4817KiB/s (4933kB/s)(7168KiB/1488msec) clat (msec): min=78, max=203, avg=157.11, stdev=51.27 lat (msec): min=78, max=204, avg=157.21, stdev=51.27 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 92], | 30.00th=[ 165], 40.00th=[ 165], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 180], 80.00th=[ 203], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 2048, max= 6131, per=21.30%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=25.00%, 250=58.33% cpu : usr=0.34%, sys=4.51%, ctx=99, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.4MiB/s (14.0MB/s), 1508KiB/s-5120KiB/s (1544kB/s-5243kB/s), io=20.0MiB (20.0MB), run=1358-1493msec WRITE: bw=18.8MiB/s (19.7MB/s), 3657KiB/s-7541KiB/s (3745kB/s-7721kB/s), io=28.0MiB (29.4MB), run=1358-1493msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5001: Thu May 1 23:36:28 2025 read: IOPS=3, BW=12.6MiB/s (13.2MB/s)(12.0MiB/956msec) clat (msec): min=178, max=343, avg=251.40, stdev=84.36 lat (msec): min=178, max=343, avg=251.40, stdev=84.36 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8192, max= 8192, per=73.16%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=10.98%, ctx=551, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5003: Thu May 1 23:36:28 2025 read: IOPS=0, BW=2800KiB/s (2867kB/s)(4096KiB/1463msec) clat (nsec): min=331212k, max=331212k, avg=331211593.00, stdev= 0.00 lat (nsec): min=331214k, max=331214k, avg=331213807.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8192, max= 8192, per=73.16%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5599KiB/s (5734kB/s)(8192KiB/1463msec) clat (msec): min=413, max=635, avg=524.20, stdev=156.75 lat (msec): min=413, max=635, avg=524.56, stdev=156.96 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 6501, max= 6501, per=29.03%, avg=6501.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.14%, sys=17.85%, ctx=360, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5005: Thu May 1 23:36:28 2025 write: IOPS=2, BW=9116KiB/s (9335kB/s)(12.0MiB/1348msec) clat (msec): min=331, max=524, avg=420.02, stdev=97.05 lat (msec): min=332, max=524, avg=420.40, stdev=97.01 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 6585, max= 8159, per=32.92%, avg=7372.00, stdev=1112.99, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=20.86%, ctx=84, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5007: Thu May 1 23:36:28 2025 write: IOPS=3, BW=14.3MiB/s (14.0MB/s)(12.0MiB/840msec) clat (msec): min=164, max=389, avg=240.20, stdev=129.47 lat (msec): min=164, max=390, avg=240.55, stdev=129.52 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8175, max= 8175, per=36.50%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.12%, sys=32.30%, ctx=87, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.9MiB/s (11.5MB/s), 2800KiB/s-12.6MiB/s (2867kB/s-13.2MB/s), io=16.0MiB (16.8MB), run=956-1463msec WRITE: bw=21.9MiB/s (22.9MB/s), 5599KiB/s-14.3MiB/s (5734kB/s-14.0MB/s), io=32.0MiB (33.6MB), run=840-1463msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5002: Thu May 1 23:36:28 2025 read: IOPS=2, BW=8480KiB/s (8684kB/s)(12.0MiB/1449msec) clat (msec): min=349, max=494, avg=413.20, stdev=74.27 lat (msec): min=349, max=494, avg=413.21, stdev=74.27 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min=13791, max=13791, per=100.00%, avg=13791.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.35%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5004: Thu May 1 23:36:28 2025 read: IOPS=0, BW=2445KiB/s (2504kB/s)(4096KiB/1675msec) clat (nsec): min=432403k, max=432403k, avg=432402932.00, stdev= 0.00 lat (nsec): min=432405k, max=432405k, avg=432405402.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 8159, max= 8159, per=83.42%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4891KiB/s (5008kB/s)(8192KiB/1675msec) clat (msec): min=110, max=1048, avg=579.80, stdev=663.31 lat (msec): min=110, max=1049, avg=580.05, stdev=663.42 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 1053], | 70.00th=[ 1053], 80.00th=[ 1053], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.48%, sys=3.52%, ctx=37, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5006: Thu May 1 23:36:28 2025 write: IOPS=3, BW=14.0MiB/s (15.7MB/s)(12.0MiB/802msec) clat (msec): min=128, max=409, avg=241.52, stdev=148.26 lat (msec): min=129, max=409, avg=241.85, stdev=148.25 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min=16384, max=16384, per=83.75%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=4.00%, ctx=18, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5008: Thu May 1 23:36:28 2025 write: IOPS=1, BW=7959KiB/s (8150kB/s)(12.0MiB/1544msec) clat (msec): min=302, max=688, avg=478.97, stdev=194.77 lat (msec): min=303, max=688, avg=479.38, stdev=194.91 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8175, max= 8192, per=41.83%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.52%, sys=3.76%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9781KiB/s (10.0MB/s), 2445KiB/s-8480KiB/s (2504kB/s-8684kB/s), io=16.0MiB (16.8MB), run=1449-1675msec WRITE: bw=19.1MiB/s (20.0MB/s), 4891KiB/s-14.0MiB/s (5008kB/s-15.7MB/s), io=32.0MiB (33.6MB), run=802-1675msec