== sanity test 398b: DIO and buffer IO race ============== 20:07:07 (1746144427) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.04605 s, 24.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21947: Thu May 1 20:07:49 2025 read: IOPS=39, BW=159KiB/s (163kB/s)(5992KiB/37580msec) clat (usec): min=73, max=103751, avg=13664.43, stdev=9818.85 lat (usec): min=73, max=103752, avg=13666.01, stdev=9818.53 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 126], 10.00th=[ 4424], 20.00th=[ 6783], | 30.00th=[ 8455], 40.00th=[ 9765], 50.00th=[ 11600], 60.00th=[ 13566], | 70.00th=[ 16057], 80.00th=[ 20055], 90.00th=[ 25822], 95.00th=[ 31851], | 99.00th=[ 46400], 99.50th=[ 51643], 99.90th=[ 77071], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 24, max= 279, per=24.90%, avg=158.88, stdev=49.46, samples=75 iops : min= 6, max= 69, avg=39.45, stdev=12.37, samples=75 write: IOPS=41, BW=168KiB/s (172kB/s)(6296KiB/37580msec) clat (usec): min=552, max=253289, avg=10739.40, stdev=12886.42 lat (usec): min=553, max=253291, avg=10740.68, stdev=12886.48 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 1029], | 30.00th=[ 2114], 40.00th=[ 4146], 50.00th=[ 7242], 60.00th=[ 10028], | 70.00th=[ 13304], 80.00th=[ 18220], 90.00th=[ 26608], 95.00th=[ 33817], | 99.00th=[ 49546], 99.50th=[ 57934], 99.90th=[ 90702], 99.95th=[252707], | 99.99th=[252707] bw ( KiB/s): min= 8, max= 336, per=24.96%, avg=166.76, stdev=55.16, samples=75 iops : min= 2, max= 84, avg=41.43, stdev=13.80, samples=75 lat (usec) : 100=0.98%, 250=2.21%, 500=0.26%, 750=2.51%, 1000=7.49% lat (msec) : 2=5.31%, 4=5.96%, 10=26.40%, 20=30.34%, 50=17.77% lat (msec) : 100=0.72%, 250=0.03%, 500=0.03% cpu : usr=0.19%, sys=11.78%, ctx=5147, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21948: Thu May 1 20:07:49 2025 read: IOPS=39, BW=158KiB/s (162kB/s)(5940KiB/37575msec) clat (usec): min=55, max=107814, avg=14289.89, stdev=10375.48 lat (usec): min=56, max=107814, avg=14291.05, stdev=10375.48 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 141], 10.00th=[ 4883], 20.00th=[ 6980], | 30.00th=[ 8717], 40.00th=[ 10421], 50.00th=[ 12256], 60.00th=[ 14353], | 70.00th=[ 16581], 80.00th=[ 20317], 90.00th=[ 26084], 95.00th=[ 31851], | 99.00th=[ 50070], 99.50th=[ 61604], 99.90th=[ 92799], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 8, max= 304, per=24.71%, avg=157.63, stdev=50.46, samples=75 iops : min= 2, max= 76, avg=39.23, stdev=12.66, samples=75 write: IOPS=42, BW=169KiB/s (173kB/s)(6348KiB/37575msec) clat (usec): min=545, max=308159, avg=10195.17, stdev=13189.49 lat (usec): min=545, max=308160, avg=10203.34, stdev=13191.68 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 930], | 30.00th=[ 1582], 40.00th=[ 3228], 50.00th=[ 6652], 60.00th=[ 9765], | 70.00th=[ 12911], 80.00th=[ 17171], 90.00th=[ 25035], 95.00th=[ 32375], | 99.00th=[ 45351], 99.50th=[ 50594], 99.90th=[ 84411], 99.95th=[308282], | 99.99th=[308282] bw ( KiB/s): min= 40, max= 368, per=25.16%, avg=168.04, stdev=57.56, samples=75 iops : min= 10, max= 92, avg=41.84, stdev=14.43, samples=75 lat (usec) : 100=0.88%, 250=1.69%, 500=0.13%, 750=2.93%, 1000=8.46% lat (msec) : 2=6.25%, 4=5.27%, 10=23.63%, 20=32.49%, 50=17.51% lat (msec) : 100=0.68%, 250=0.03%, 500=0.03% cpu : usr=0.17%, sys=11.88%, ctx=5148, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21949: Thu May 1 20:07:49 2025 read: IOPS=40, BW=160KiB/s (164kB/s)(6028KiB/37620msec) clat (usec): min=66, max=100506, avg=13882.24, stdev=10074.41 lat (usec): min=66, max=100507, avg=13884.86, stdev=10075.85 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 125], 10.00th=[ 4686], 20.00th=[ 6718], | 30.00th=[ 8455], 40.00th=[ 10159], 50.00th=[ 11994], 60.00th=[ 14091], | 70.00th=[ 16581], 80.00th=[ 19792], 90.00th=[ 25035], 95.00th=[ 31327], | 99.00th=[ 50594], 99.50th=[ 60556], 99.90th=[ 92799], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 87, max= 288, per=25.30%, avg=161.39, stdev=45.03, samples=74 iops : min= 21, max= 72, avg=40.08, stdev=11.38, samples=74 write: IOPS=41, BW=166KiB/s (170kB/s)(6260KiB/37620msec) clat (usec): min=574, max=327506, avg=10559.75, stdev=13958.27 lat (usec): min=575, max=327507, avg=10564.81, stdev=13957.39 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 947], | 30.00th=[ 1663], 40.00th=[ 3589], 50.00th=[ 6915], 60.00th=[ 9765], | 70.00th=[ 12649], 80.00th=[ 17957], 90.00th=[ 26346], 95.00th=[ 33817], | 99.00th=[ 51119], 99.50th=[ 57410], 99.90th=[ 84411], 99.95th=[329253], | 99.99th=[329253] bw ( KiB/s): min= 16, max= 344, per=24.72%, avg=165.12, stdev=56.73, samples=75 iops : min= 4, max= 86, avg=41.01, stdev=14.21, samples=75 lat (usec) : 100=1.33%, 250=2.15%, 500=0.10%, 750=2.77%, 1000=8.27% lat (msec) : 2=5.73%, 4=4.92%, 10=24.97%, 20=31.15%, 50=17.58% lat (msec) : 100=0.98%, 250=0.03%, 500=0.03% cpu : usr=0.21%, sys=11.61%, ctx=5063, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21950: Thu May 1 20:07:49 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(6052KiB/37539msec) clat (usec): min=68, max=82754, avg=13976.76, stdev=9582.07 lat (usec): min=68, max=82756, avg=13978.29, stdev=9583.11 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 293], 10.00th=[ 5145], 20.00th=[ 6915], | 30.00th=[ 8717], 40.00th=[10159], 50.00th=[11863], 60.00th=[13960], | 70.00th=[16909], 80.00th=[20055], 90.00th=[25560], 95.00th=[31065], | 99.00th=[47449], 99.50th=[58459], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 56, max= 328, per=25.55%, avg=163.03, stdev=47.30, samples=74 iops : min= 14, max= 82, avg=40.54, stdev=11.87, samples=74 write: IOPS=41, BW=166KiB/s (170kB/s)(6236KiB/37539msec) clat (usec): min=465, max=305154, avg=10407.87, stdev=13642.76 lat (usec): min=466, max=305157, avg=10410.16, stdev=13642.56 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 906], | 30.00th=[ 1631], 40.00th=[ 3261], 50.00th=[ 6652], 60.00th=[ 9896], | 70.00th=[ 12911], 80.00th=[ 16909], 90.00th=[ 25822], 95.00th=[ 34341], | 99.00th=[ 50070], 99.50th=[ 54264], 99.90th=[102237], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 7, max= 280, per=24.77%, avg=165.44, stdev=49.53, samples=75 iops : min= 1, max= 70, avg=41.13, stdev=12.42, samples=75 lat (usec) : 100=0.81%, 250=1.63%, 500=0.13%, 750=3.42%, 1000=8.27% lat (msec) : 2=4.98%, 4=5.14%, 10=25.29%, 20=32.23%, 50=17.22% lat (msec) : 100=0.81%, 250=0.03%, 500=0.03% cpu : usr=0.17%, sys=11.70%, ctx=5054, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=638KiB/s (654kB/s), 158KiB/s-161KiB/s (162kB/s-165kB/s), io=23.4MiB (24.6MB), run=37539-37620msec WRITE: bw=668KiB/s (684kB/s), 166KiB/s-169KiB/s (170kB/s-173kB/s), io=24.6MiB (25.7MB), run=37539-37620msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21951: Thu May 1 20:08:06 2025 read: IOPS=27, BW=108KiB/s (111kB/s)(5992KiB/55233msec) clat (msec): min=3, max=348, avg=13.73, stdev=14.88 lat (msec): min=3, max=348, avg=13.73, stdev=14.88 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 40], | 99.00th=[ 64], 99.50th=[ 74], 99.90th=[ 106], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 16, max= 232, per=24.89%, avg=107.75, stdev=54.48, samples=110 iops : min= 4, max= 58, avg=26.61, stdev=13.65, samples=110 write: IOPS=28, BW=114KiB/s (117kB/s)(6296KiB/55233msec) clat (msec): min=5, max=104, avg=21.90, stdev=14.03 lat (msec): min=5, max=104, avg=21.90, stdev=14.03 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 25], 80.00th=[ 31], 90.00th=[ 42], 95.00th=[ 51], | 99.00th=[ 72], 99.50th=[ 85], 99.90th=[ 92], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 31, max= 247, per=25.22%, avg=114.48, stdev=55.57, samples=109 iops : min= 7, max= 61, avg=28.28, stdev=13.98, samples=109 lat (msec) : 4=1.07%, 10=30.76%, 20=40.85%, 50=23.24%, 100=3.94% lat (msec) : 250=0.10%, 500=0.03% cpu : usr=0.17%, sys=5.41%, ctx=3525, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21952: Thu May 1 20:08:06 2025 read: IOPS=26, BW=107KiB/s (110kB/s)(5940KiB/55338msec) clat (msec): min=3, max=283, avg=13.66, stdev=14.29 lat (msec): min=3, max=283, avg=13.67, stdev=14.28 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 41], | 99.00th=[ 63], 99.50th=[ 69], 99.90th=[ 140], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 7, max= 296, per=24.64%, avg=106.70, stdev=58.64, samples=110 iops : min= 1, max= 74, avg=26.35, stdev=14.69, samples=110 write: IOPS=28, BW=115KiB/s (117kB/s)(6348KiB/55338msec) clat (msec): min=4, max=138, avg=21.97, stdev=14.70 lat (msec): min=4, max=138, avg=21.98, stdev=14.70 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 24], 80.00th=[ 30], 90.00th=[ 41], 95.00th=[ 53], | 99.00th=[ 78], 99.50th=[ 90], 99.90th=[ 125], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 7, max= 231, per=25.04%, avg=113.66, stdev=55.66, samples=110 iops : min= 1, max= 57, avg=28.10, stdev=13.95, samples=110 lat (msec) : 4=0.88%, 10=31.38%, 20=39.84%, 50=23.40%, 100=4.23% lat (msec) : 250=0.23%, 500=0.03% cpu : usr=0.05%, sys=5.47%, ctx=3447, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21953: Thu May 1 20:08:06 2025 read: IOPS=27, BW=110KiB/s (113kB/s)(6028KiB/54611msec) clat (msec): min=3, max=102, avg=13.15, stdev=11.48 lat (msec): min=3, max=102, avg=13.15, stdev=11.48 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 28], 95.00th=[ 37], | 99.00th=[ 57], 99.50th=[ 71], 99.90th=[ 100], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 23, max= 240, per=25.60%, avg=110.83, stdev=52.76, samples=108 iops : min= 5, max= 60, avg=27.40, stdev=13.22, samples=108 write: IOPS=28, BW=115KiB/s (117kB/s)(6260KiB/54611msec) clat (msec): min=5, max=325, avg=22.15, stdev=16.07 lat (msec): min=5, max=325, avg=22.15, stdev=16.07 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 24], 80.00th=[ 31], 90.00th=[ 41], 95.00th=[ 52], | 99.00th=[ 73], 99.50th=[ 80], 99.90th=[ 117], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 24, max= 263, per=25.11%, avg=114.00, stdev=55.05, samples=109 iops : min= 6, max= 65, avg=28.19, stdev=13.82, samples=109 lat (msec) : 4=0.91%, 10=31.18%, 20=40.62%, 50=23.50%, 100=3.61% lat (msec) : 250=0.13%, 500=0.03% cpu : usr=0.09%, sys=5.51%, ctx=3490, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21954: Thu May 1 20:08:06 2025 read: IOPS=27, BW=109KiB/s (112kB/s)(6052KiB/55285msec) clat (msec): min=2, max=140, avg=13.90, stdev=12.88 lat (msec): min=2, max=140, avg=13.90, stdev=12.88 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 31], 95.00th=[ 42], | 99.00th=[ 62], 99.50th=[ 69], 99.90th=[ 103], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 23, max= 264, per=25.35%, avg=109.75, stdev=56.20, samples=109 iops : min= 5, max= 66, avg=27.07, stdev=14.11, samples=109 write: IOPS=28, BW=113KiB/s (116kB/s)(6236KiB/55285msec) clat (msec): min=4, max=317, avg=21.88, stdev=16.25 lat (msec): min=4, max=317, avg=21.88, stdev=16.25 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 23], 80.00th=[ 29], 90.00th=[ 42], 95.00th=[ 52], | 99.00th=[ 74], 99.50th=[ 89], 99.90th=[ 136], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 15, max= 255, per=24.69%, avg=112.09, stdev=55.03, samples=110 iops : min= 3, max= 63, avg=27.65, stdev=13.79, samples=110 lat (msec) : 4=0.78%, 10=30.47%, 20=41.47%, 50=23.11%, 100=3.94% lat (msec) : 250=0.20%, 500=0.03% cpu : usr=0.14%, sys=5.35%, ctx=3486, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=434KiB/s (444kB/s), 107KiB/s-110KiB/s (110kB/s-113kB/s), io=23.4MiB (24.6MB), run=54611-55338msec WRITE: bw=454KiB/s (465kB/s), 113KiB/s-115KiB/s (116kB/s-117kB/s), io=24.6MiB (25.7MB), run=54611-55338msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22016: Thu May 1 20:08:17 2025 read: IOPS=38, BW=620KiB/s (634kB/s)(5920KiB/9556msec) clat (usec): min=76, max=77592, avg=14382.31, stdev=10486.52 lat (usec): min=76, max=77594, avg=14383.44, stdev=10486.71 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 110], 10.00th=[ 375], 20.00th=[ 7177], | 30.00th=[ 8848], 40.00th=[10421], 50.00th=[12518], 60.00th=[14877], | 70.00th=[17171], 80.00th=[20579], 90.00th=[26346], 95.00th=[32637], | 99.00th=[54789], 99.50th=[59507], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 383, max= 896, per=24.02%, avg=594.78, stdev=123.07, samples=18 iops : min= 23, max= 56, avg=36.78, stdev= 7.95, samples=18 write: IOPS=41, BW=666KiB/s (682kB/s)(6368KiB/9556msec) clat (usec): min=635, max=65821, avg=10514.97, stdev=11107.79 lat (usec): min=636, max=65823, avg=10516.94, stdev=11108.20 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1074], 10.00th=[ 1139], 20.00th=[ 1352], | 30.00th=[ 2245], 40.00th=[ 4228], 50.00th=[ 7111], 60.00th=[ 9765], | 70.00th=[12780], 80.00th=[17957], 90.00th=[25560], 95.00th=[33162], | 99.00th=[50594], 99.50th=[61604], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 446, max= 1056, per=24.95%, avg=649.89, stdev=180.16, samples=18 iops : min= 27, max= 66, avg=40.22, stdev=11.39, samples=18 lat (usec) : 100=0.78%, 250=3.91%, 500=0.13%, 750=0.26%, 1000=0.78% lat (msec) : 2=13.67%, 4=5.99%, 10=23.44%, 20=31.25%, 50=18.49% lat (msec) : 100=1.30% cpu : usr=0.40%, sys=12.69%, ctx=1379, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22019: Thu May 1 20:08:17 2025 read: IOPS=37, BW=603KiB/s (618kB/s)(5744KiB/9518msec) clat (usec): min=86, max=86025, avg=13912.10, stdev=11137.53 lat (usec): min=86, max=86026, avg=13913.99, stdev=11138.12 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 111], 10.00th=[ 125], 20.00th=[ 6652], | 30.00th=[ 8455], 40.00th=[10421], 50.00th=[12387], 60.00th=[14222], | 70.00th=[16188], 80.00th=[19530], 90.00th=[26346], 95.00th=[34341], | 99.00th=[56361], 99.50th=[72877], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 384, max= 864, per=24.04%, avg=595.11, stdev=139.06, samples=18 iops : min= 24, max= 54, avg=37.00, stdev= 8.78, samples=18 write: IOPS=42, BW=688KiB/s (704kB/s)(6544KiB/9518msec) clat (usec): min=884, max=52734, avg=10804.11, stdev=10220.88 lat (usec): min=887, max=52735, avg=10805.78, stdev=10220.87 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1090], 10.00th=[ 1156], 20.00th=[ 1516], | 30.00th=[ 2507], 40.00th=[ 4817], 50.00th=[ 7701], 60.00th=[11076], | 70.00th=[14091], 80.00th=[20841], 90.00th=[26870], 95.00th=[28967], | 99.00th=[41681], 99.50th=[45351], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 480, max= 1024, per=25.71%, avg=669.72, stdev=155.06, samples=18 iops : min= 30, max= 64, avg=41.67, stdev= 9.72, samples=18 lat (usec) : 100=1.30%, 250=4.82%, 500=0.13%, 750=0.13%, 1000=0.39% lat (msec) : 2=13.02%, 4=6.77%, 10=22.66%, 20=31.25%, 50=18.49% lat (msec) : 100=1.04% cpu : usr=0.04%, sys=13.12%, ctx=1370, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22020: Thu May 1 20:08:17 2025 read: IOPS=40, BW=647KiB/s (662kB/s)(6256KiB/9671msec) clat (usec): min=65, max=82865, avg=13178.61, stdev=8998.17 lat (usec): min=66, max=82866, avg=13179.76, stdev=8998.23 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 111], 10.00th=[ 143], 20.00th=[ 7111], | 30.00th=[ 8717], 40.00th=[10814], 50.00th=[12256], 60.00th=[14484], | 70.00th=[16712], 80.00th=[19268], 90.00th=[22676], 95.00th=[27395], | 99.00th=[41681], 99.50th=[47449], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 192, max= 926, per=25.78%, avg=638.32, stdev=183.68, samples=19 iops : min= 12, max= 57, avg=39.58, stdev=11.38, samples=19 write: IOPS=38, BW=624KiB/s (639kB/s)(6032KiB/9671msec) clat (usec): min=780, max=77017, avg=11463.28, stdev=11318.04 lat (usec): min=782, max=77018, avg=11464.98, stdev=11318.07 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1090], 10.00th=[ 1188], 20.00th=[ 1418], | 30.00th=[ 2606], 40.00th=[ 4948], 50.00th=[ 8160], 60.00th=[11600], | 70.00th=[14615], 80.00th=[19530], 90.00th=[27395], 95.00th=[34341], | 99.00th=[44827], 99.50th=[53216], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 256, max= 928, per=23.32%, avg=607.37, stdev=161.72, samples=19 iops : min= 16, max= 58, avg=37.68, stdev=10.13, samples=19 lat (usec) : 100=1.30%, 250=4.82%, 500=0.13%, 750=0.26%, 1000=1.56% lat (msec) : 2=11.07%, 4=5.86%, 10=20.18%, 20=36.72%, 50=17.71% lat (msec) : 100=0.39% cpu : usr=0.17%, sys=13.07%, ctx=1367, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22021: Thu May 1 20:08:17 2025 read: IOPS=40, BW=648KiB/s (664kB/s)(6032KiB/9305msec) clat (usec): min=82, max=84947, avg=13681.17, stdev=10286.90 lat (usec): min=82, max=84948, avg=13682.22, stdev=10286.90 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 109], 10.00th=[ 126], 20.00th=[ 6587], | 30.00th=[ 8356], 40.00th=[10290], 50.00th=[12518], 60.00th=[14615], | 70.00th=[17433], 80.00th=[20055], 90.00th=[25035], 95.00th=[31851], | 99.00th=[52167], 99.50th=[55837], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 320, max= 990, per=24.94%, avg=617.56, stdev=154.92, samples=18 iops : min= 20, max= 61, avg=38.11, stdev= 9.58, samples=18 write: IOPS=42, BW=672KiB/s (688kB/s)(6256KiB/9305msec) clat (usec): min=791, max=65148, avg=10292.72, stdev=11265.21 lat (usec): min=792, max=65149, avg=10294.38, stdev=11265.29 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1090], 10.00th=[ 1188], 20.00th=[ 1385], | 30.00th=[ 2073], 40.00th=[ 3130], 50.00th=[ 5932], 60.00th=[ 9372], | 70.00th=[13435], 80.00th=[17957], 90.00th=[23987], 95.00th=[31327], | 99.00th=[54264], 99.50th=[60556], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 416, max= 892, per=25.00%, avg=651.17, stdev=156.01, samples=18 iops : min= 26, max= 55, avg=40.22, stdev= 9.78, samples=18 lat (usec) : 100=1.69%, 250=4.43%, 500=0.65%, 1000=1.30% lat (msec) : 2=14.19%, 4=7.68%, 10=19.01%, 20=32.16%, 50=17.84% lat (msec) : 100=1.04% cpu : usr=0.09%, sys=13.01%, ctx=1327, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2477KiB/s (2536kB/s), 603KiB/s-648KiB/s (618kB/s-664kB/s), io=23.4MiB (24.5MB), run=9305-9671msec WRITE: bw=2606KiB/s (2668kB/s), 624KiB/s-688KiB/s (639kB/s-704kB/s), io=24.6MiB (25.8MB), run=9305-9671msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22014: Thu May 1 20:08:22 2025 read: IOPS=25, BW=415KiB/s (425kB/s)(5920KiB/14268msec) clat (msec): min=3, max=105, avg=14.26, stdev=12.02 lat (msec): min=3, max=105, avg=14.26, stdev=12.02 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 19], 90.00th=[ 28], 95.00th=[ 41], | 99.00th=[ 61], 99.50th=[ 73], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 223, max= 736, per=24.61%, avg=408.57, stdev=168.52, samples=28 iops : min= 13, max= 46, avg=25.25, stdev=10.53, samples=28 write: IOPS=27, BW=446KiB/s (457kB/s)(6368KiB/14268msec) clat (usec): min=6482, max=77869, avg=22496.51, stdev=12923.78 lat (usec): min=6484, max=77871, avg=22498.70, stdev=12923.69 clat percentiles (usec): | 1.00th=[ 7635], 5.00th=[10159], 10.00th=[11076], 20.00th=[12649], | 30.00th=[14615], 40.00th=[16450], 50.00th=[17957], 60.00th=[19792], | 70.00th=[24249], 80.00th=[31327], 90.00th=[41157], 95.00th=[49021], | 99.00th=[70779], 99.50th=[74974], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 224, max= 800, per=25.48%, avg=445.18, stdev=182.98, samples=28 iops : min= 14, max= 50, avg=27.54, stdev=11.44, samples=28 lat (msec) : 4=0.52%, 10=24.87%, 20=45.18%, 50=25.78%, 100=3.52% lat (msec) : 250=0.13% cpu : usr=0.22%, sys=5.34%, ctx=888, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22015: Thu May 1 20:08:22 2025 read: IOPS=25, BW=402KiB/s (412kB/s)(5744KiB/14279msec) clat (usec): min=3311, max=88500, avg=14089.03, stdev=10623.03 lat (usec): min=3313, max=88501, avg=14090.22, stdev=10622.99 clat percentiles (usec): | 1.00th=[ 3851], 5.00th=[ 5145], 10.00th=[ 5735], 20.00th=[ 7046], | 30.00th=[ 8029], 40.00th=[ 8979], 50.00th=[ 9896], 60.00th=[11469], | 70.00th=[14746], 80.00th=[19006], 90.00th=[30278], 95.00th=[36439], | 99.00th=[52167], 99.50th=[52691], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 64, max= 800, per=24.06%, avg=399.43, stdev=189.55, samples=28 iops : min= 4, max= 50, avg=24.61, stdev=11.87, samples=28 write: IOPS=28, BW=458KiB/s (469kB/s)(6544KiB/14279msec) clat (msec): min=5, max=115, avg=22.10, stdev=13.37 lat (msec): min=5, max=115, avg=22.10, stdev=13.37 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 30], 90.00th=[ 39], 95.00th=[ 51], | 99.00th=[ 71], 99.50th=[ 77], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 255, max= 828, per=25.93%, avg=453.04, stdev=175.86, samples=28 iops : min= 15, max= 51, avg=27.96, stdev=11.03, samples=28 lat (msec) : 4=0.52%, 10=26.04%, 20=42.58%, 50=27.73%, 100=2.99% lat (msec) : 250=0.13% cpu : usr=0.08%, sys=5.38%, ctx=874, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22017: Thu May 1 20:08:22 2025 read: IOPS=27, BW=438KiB/s (449kB/s)(6256KiB/14271msec) clat (usec): min=2922, max=65831, avg=13709.00, stdev=10748.51 lat (usec): min=2923, max=65832, avg=13710.23, stdev=10748.43 clat percentiles (usec): | 1.00th=[ 3982], 5.00th=[ 5080], 10.00th=[ 5669], 20.00th=[ 6456], | 30.00th=[ 7439], 40.00th=[ 8225], 50.00th=[ 9241], 60.00th=[10945], | 70.00th=[13173], 80.00th=[20317], 90.00th=[29492], 95.00th=[39060], | 99.00th=[49546], 99.50th=[63701], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 160, max= 1120, per=26.39%, avg=438.07, stdev=236.87, samples=28 iops : min= 10, max= 70, avg=26.96, stdev=14.84, samples=28 write: IOPS=26, BW=423KiB/s (433kB/s)(6032KiB/14271msec) clat (usec): min=7127, max=95924, avg=23221.96, stdev=13361.89 lat (usec): min=7129, max=95924, avg=23224.30, stdev=13361.76 clat percentiles (usec): | 1.00th=[ 8160], 5.00th=[10159], 10.00th=[11338], 20.00th=[13435], | 30.00th=[15139], 40.00th=[16909], 50.00th=[18482], 60.00th=[21103], | 70.00th=[24511], 80.00th=[31327], 90.00th=[43254], 95.00th=[52691], | 99.00th=[67634], 99.50th=[78119], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 160, max= 798, per=23.83%, avg=416.32, stdev=173.06, samples=28 iops : min= 10, max= 49, avg=25.61, stdev=10.79, samples=28 lat (msec) : 4=0.52%, 10=29.69%, 20=37.76%, 50=29.04%, 100=2.99% cpu : usr=0.11%, sys=5.35%, ctx=874, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22018: Thu May 1 20:08:22 2025 read: IOPS=26, BW=418KiB/s (428kB/s)(6032KiB/14424msec) clat (usec): min=3404, max=86985, avg=14417.03, stdev=12018.48 lat (usec): min=3405, max=86986, avg=14419.71, stdev=12018.19 clat percentiles (usec): | 1.00th=[ 4359], 5.00th=[ 5145], 10.00th=[ 5669], 20.00th=[ 6718], | 30.00th=[ 7570], 40.00th=[ 8586], 50.00th=[ 9372], 60.00th=[11207], | 70.00th=[14091], 80.00th=[20055], 90.00th=[31851], 95.00th=[40109], | 99.00th=[58459], 99.50th=[63177], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 127, max= 800, per=24.53%, avg=407.25, stdev=178.94, samples=28 iops : min= 7, max= 50, avg=25.04, stdev=11.16, samples=28 write: IOPS=27, BW=434KiB/s (444kB/s)(6256KiB/14424msec) clat (msec): min=5, max=115, avg=22.55, stdev=13.22 lat (msec): min=5, max=115, avg=22.56, stdev=13.22 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 26], 80.00th=[ 31], 90.00th=[ 41], 95.00th=[ 50], | 99.00th=[ 68], 99.50th=[ 73], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 191, max= 768, per=24.42%, avg=426.68, stdev=188.29, samples=28 iops : min= 11, max= 48, avg=26.25, stdev=11.82, samples=28 lat (msec) : 4=0.26%, 10=28.78%, 20=39.06%, 50=28.26%, 100=3.52% lat (msec) : 250=0.13% cpu : usr=0.15%, sys=5.26%, ctx=890, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1661KiB/s (1700kB/s), 402KiB/s-438KiB/s (412kB/s-449kB/s), io=23.4MiB (24.5MB), run=14268-14424msec WRITE: bw=1747KiB/s (1789kB/s), 423KiB/s-458KiB/s (433kB/s-469kB/s), io=24.6MiB (25.8MB), run=14268-14424msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22042: Thu May 1 20:08:25 2025 read: IOPS=5, BW=5904KiB/s (6046kB/s)(7168KiB/1214msec) clat (usec): min=545, max=101607, avg=42338.08, stdev=41743.40 lat (usec): min=546, max=101609, avg=42339.62, stdev=41743.59 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 553], | 30.00th=[ 3130], 40.00th=[ 3130], 50.00th=[ 51119], 60.00th=[ 56886], | 70.00th=[ 56886], 80.00th=[ 82314], 90.00th=[101188], 95.00th=[101188], | 99.00th=[101188], 99.50th=[101188], 99.90th=[101188], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 4071, max= 6144, per=30.28%, avg=5107.50, stdev=1465.83, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=4, BW=4217KiB/s (4319kB/s)(5120KiB/1214msec) clat (msec): min=50, max=339, avg=139.41, stdev=117.23 lat (msec): min=50, max=339, avg=139.46, stdev=117.25 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 2048, max= 4071, per=12.95%, avg=3059.50, stdev=1430.48, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (usec) : 750=16.67% lat (msec) : 4=8.33%, 100=41.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=18.63%, ctx=249, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22044: Thu May 1 20:08:25 2025 read: IOPS=1, BW=1722KiB/s (1764kB/s)(2048KiB/1189msec) clat (usec): min=504, max=91803, avg=46153.64, stdev=64558.02 lat (usec): min=505, max=91805, avg=46155.58, stdev=64558.87 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[91751], | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 2048, max= 2048, per=12.14%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8612KiB/s (8819kB/s)(10.0MiB/1189msec) clat (msec): min=38, max=213, avg=98.00, stdev=52.44 lat (msec): min=38, max=213, avg=98.07, stdev=52.43 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 52], | 30.00th=[ 69], 40.00th=[ 71], 50.00th=[ 72], 60.00th=[ 91], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 150], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 6144, max=10240, per=34.69%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=8.33%, 100=50.00%, 250=33.33% cpu : usr=0.00%, sys=22.22%, ctx=138, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22045: Thu May 1 20:08:25 2025 read: IOPS=5, BW=5684KiB/s (5820kB/s)(6144KiB/1081msec) clat (usec): min=429, max=94382, avg=56398.65, stdev=44134.23 lat (usec): min=430, max=94383, avg=56400.08, stdev=44134.29 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 791], | 30.00th=[ 791], 40.00th=[69731], 50.00th=[69731], 60.00th=[80217], | 70.00th=[92799], 80.00th=[92799], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 8192, max= 8192, per=48.56%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5684KiB/s (5820kB/s)(6144KiB/1081msec) clat (msec): min=57, max=143, avg=98.44, stdev=35.37 lat (msec): min=57, max=143, avg=98.48, stdev=35.38 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 92], 50.00th=[ 92], 60.00th=[ 117], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 4096, max= 6144, per=21.68%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 100=58.33%, 250=25.00% cpu : usr=0.00%, sys=21.11%, ctx=194, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22046: Thu May 1 20:08:25 2025 read: IOPS=4, BW=4551KiB/s (4660kB/s)(5120KiB/1125msec) clat (usec): min=569, max=70384, avg=37150.94, stdev=33822.31 lat (usec): min=571, max=70386, avg=37152.76, stdev=33822.17 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 570], 10.00th=[ 570], 20.00th=[ 570], | 30.00th=[ 594], 40.00th=[ 594], 50.00th=[56886], 60.00th=[56886], | 70.00th=[57410], 80.00th=[57410], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 8192, max= 8192, per=48.56%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6372KiB/s (6524kB/s)(7168KiB/1125msec) clat (msec): min=51, max=156, avg=100.38, stdev=35.08 lat (msec): min=51, max=156, avg=100.44, stdev=35.08 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 72], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 110], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 118], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 4096, max= 6144, per=21.68%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (usec) : 750=16.67% lat (msec) : 100=50.00%, 250=33.33% cpu : usr=0.00%, sys=19.75%, ctx=237, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.5MiB/s (17.3MB/s), 1722KiB/s-5904KiB/s (1764kB/s-6046kB/s), io=20.0MiB (20.0MB), run=1081-1214msec WRITE: bw=23.1MiB/s (24.2MB/s), 4217KiB/s-8612KiB/s (4319kB/s-8819kB/s), io=28.0MiB (29.4MB), run=1081-1214msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22039: Thu May 1 20:08:25 2025 read: IOPS=5, BW=5434KiB/s (5565kB/s)(7168KiB/1319msec) clat (msec): min=35, max=149, avg=84.30, stdev=46.04 lat (msec): min=35, max=149, avg=84.30, stdev=46.04 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 44], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 65], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 129], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 4079, max= 8192, per=42.21%, avg=6135.50, stdev=2908.33, samples=2 iops : min= 3, max= 8, avg= 5.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3882KiB/s (3975kB/s)(5120KiB/1319msec) clat (msec): min=35, max=222, avg=136.78, stdev=79.75 lat (msec): min=36, max=222, avg=136.91, stdev=79.73 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 222], 95.00th=[ 222], | 99.00th=[ 222], 99.50th=[ 222], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 2048, max= 6119, per=20.07%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 50=25.00%, 100=16.67%, 250=58.33% cpu : usr=0.00%, sys=4.10%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22040: Thu May 1 20:08:25 2025 read: IOPS=1, BW=1547KiB/s (1584kB/s)(2048KiB/1324msec) clat (usec): min=30971, max=67133, avg=49052.52, stdev=25570.29 lat (usec): min=30972, max=67135, avg=49053.90, stdev=25570.62 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[31065], | 30.00th=[31065], 40.00th=[31065], 50.00th=[31065], 60.00th=[67634], | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 2048, max= 2048, per=14.09%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=7, BW=7734KiB/s (7920kB/s)(10.0MiB/1324msec) clat (msec): min=33, max=226, avg=99.66, stdev=57.86 lat (msec): min=33, max=226, avg=99.74, stdev=57.87 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 35], | 30.00th=[ 58], 40.00th=[ 69], 50.00th=[ 88], 60.00th=[ 113], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 136], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 2048, max=12288, per=35.23%, avg=7168.00, stdev=7240.77, samples=2 iops : min= 2, max= 12, avg= 7.00, stdev= 7.07, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=41.67% cpu : usr=0.15%, sys=6.80%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22041: Thu May 1 20:08:25 2025 read: IOPS=4, BW=4361KiB/s (4465kB/s)(6144KiB/1409msec) clat (msec): min=36, max=138, avg=94.00, stdev=39.53 lat (msec): min=36, max=138, avg=94.00, stdev=39.53 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 101], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 6144, max= 6144, per=42.27%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4361KiB/s (4465kB/s)(6144KiB/1409msec) clat (msec): min=33, max=233, avg=123.17, stdev=75.43 lat (msec): min=34, max=233, avg=123.25, stdev=75.42 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 121], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 2048, max= 6131, per=20.10%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=25.00%, 250=58.33% cpu : usr=0.14%, sys=6.11%, ctx=66, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22043: Thu May 1 20:08:25 2025 read: IOPS=4, BW=4122KiB/s (4221kB/s)(5120KiB/1242msec) clat (msec): min=29, max=246, avg=132.69, stdev=81.69 lat (msec): min=29, max=246, avg=132.69, stdev=81.69 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 85], 40.00th=[ 85], 50.00th=[ 142], 60.00th=[ 142], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 247], 95.00th=[ 247], | 99.00th=[ 247], 99.50th=[ 247], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 8175, max= 8175, per=56.24%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5771KiB/s (5910kB/s)(7168KiB/1242msec) clat (msec): min=32, max=129, avg=72.68, stdev=37.78 lat (msec): min=32, max=129, avg=72.76, stdev=37.77 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 63], 60.00th=[ 99], | 70.00th=[ 99], 80.00th=[ 104], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 6131, max= 8175, per=35.15%, avg=7153.00, stdev=1445.33, samples=2 iops : min= 5, max= 7, avg= 6.00, stdev= 1.41, samples=2 lat (msec) : 50=33.33%, 100=25.00%, 250=41.67% cpu : usr=0.00%, sys=3.79%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.2MiB/s (14.9MB/s), 1547KiB/s-5434KiB/s (1584kB/s-5565kB/s), io=20.0MiB (20.0MB), run=1242-1409msec WRITE: bw=19.9MiB/s (20.8MB/s), 3882KiB/s-7734KiB/s (3975kB/s-7920kB/s), io=28.0MiB (29.4MB), run=1242-1409msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22053: Thu May 1 20:08:28 2025 read: IOPS=4, BW=19.0MiB/s (19.0MB/s)(12.0MiB/630msec) clat (msec): min=56, max=302, avg=177.66, stdev=123.27 lat (msec): min=56, max=302, avg=177.66, stdev=123.27 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8175, max= 8175, per=64.82%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=23.37%, ctx=677, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22054: Thu May 1 20:08:28 2025 read: IOPS=0, BW=3153KiB/s (3229kB/s)(4096KiB/1299msec) clat (nsec): min=341662k, max=341662k, avg=341661980.00, stdev= 0.00 lat (nsec): min=341663k, max=341663k, avg=341663257.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8192, max= 8192, per=64.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6306KiB/s (6458kB/s)(8192KiB/1299msec) clat (msec): min=221, max=637, avg=429.64, stdev=294.11 lat (msec): min=222, max=637, avg=430.01, stdev=294.01 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=15.87%, ctx=173, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22055: Thu May 1 20:08:28 2025 write: IOPS=2, BW=9232KiB/s (9454kB/s)(12.0MiB/1331msec) clat (msec): min=180, max=697, avg=422.60, stdev=260.06 lat (msec): min=181, max=698, avg=424.12, stdev=260.29 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 8175, max= 8192, per=33.24%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=19.70%, ctx=67, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22056: Thu May 1 20:08:28 2025 write: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1023msec) clat (msec): min=222, max=391, avg=326.35, stdev=91.34 lat (msec): min=222, max=392, avg=326.65, stdev=91.34 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8175, max= 8192, per=33.24%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.10%, sys=28.67%, ctx=78, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.3MiB/s (12.9MB/s), 3153KiB/s-19.0MiB/s (3229kB/s-19.0MB/s), io=16.0MiB (16.8MB), run=630-1299msec WRITE: bw=24.0MiB/s (25.2MB/s), 6306KiB/s-11.7MiB/s (6458kB/s-12.3MB/s), io=32.0MiB (33.6MB), run=1023-1331msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22057: Thu May 1 20:08:28 2025 read: IOPS=1, BW=7719KiB/s (7904kB/s)(12.0MiB/1592msec) clat (msec): min=283, max=609, avg=498.47, stdev=186.53 lat (msec): min=283, max=609, avg=498.47, stdev=186.53 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 600], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.13%, sys=1.51%, ctx=23, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22058: Thu May 1 20:08:28 2025 read: IOPS=0, BW=2339KiB/s (2395kB/s)(4096KiB/1751msec) clat (nsec): min=584429k, max=584429k, avg=584429402.00, stdev= 0.00 lat (nsec): min=584432k, max=584432k, avg=584431697.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 8094, max= 8094, per=86.51%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4678KiB/s (4791kB/s)(8192KiB/1751msec) clat (msec): min=129, max=963, avg=546.37, stdev=589.80 lat (msec): min=129, max=963, avg=546.69, stdev=589.73 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 961], | 70.00th=[ 961], 80.00th=[ 961], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 5953, max= 5953, per=31.81%, avg=5953.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.34%, sys=4.97%, ctx=86, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22059: Thu May 1 20:08:28 2025 write: IOPS=1, BW=7995KiB/s (8187kB/s)(12.0MiB/1537msec) clat (msec): min=299, max=624, avg=490.92, stdev=169.69 lat (msec): min=300, max=624, avg=491.23, stdev=169.69 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min=16384, max=16384, per=87.55%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.07%, sys=7.49%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22060: Thu May 1 20:08:28 2025 write: IOPS=2, BW=9600KiB/s (9830kB/s)(12.0MiB/1280msec) clat (msec): min=238, max=691, avg=412.84, stdev=243.46 lat (msec): min=239, max=691, avg=413.25, stdev=243.77 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8175, max= 8175, per=43.69%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.15%, ctx=45, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9357KiB/s (9582kB/s), 2339KiB/s-7719KiB/s (2395kB/s-7904kB/s), io=16.0MiB (16.8MB), run=1592-1751msec WRITE: bw=18.3MiB/s (19.2MB/s), 4678KiB/s-9600KiB/s (4791kB/s-9830kB/s), io=32.0MiB (33.6MB), run=1280-1751msec