== sanity test 398b: DIO and buffer IO race ============== 01:11:25 (1746076285) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.47999 s, 20.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32290: Thu May 1 01:12:32 2025 read: IOPS=24, BW=99.1KiB/s (102kB/s)(5992KiB/60440msec) clat (usec): min=64, max=204560, avg=17731.89, stdev=17439.40 lat (usec): min=65, max=204561, avg=17733.68, stdev=17439.18 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 99], 10.00th=[ 112], 20.00th=[ 149], | 30.00th=[ 8979], 40.00th=[ 12649], 50.00th=[ 15270], 60.00th=[ 18482], | 70.00th=[ 22152], 80.00th=[ 27657], 90.00th=[ 38011], 95.00th=[ 47449], | 99.00th=[ 72877], 99.50th=[ 90702], 99.90th=[173016], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 23, max= 215, per=25.50%, avg=98.44, stdev=34.13, samples=120 iops : min= 5, max= 53, avg=24.24, stdev= 8.52, samples=120 write: IOPS=26, BW=104KiB/s (107kB/s)(6296KiB/60440msec) clat (usec): min=578, max=302249, avg=21338.81, stdev=24159.29 lat (usec): min=579, max=302251, avg=21341.04, stdev=24158.94 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 816], 20.00th=[ 1090], | 30.00th=[ 4178], 40.00th=[ 8979], 50.00th=[ 15139], 60.00th=[ 21103], | 70.00th=[ 27657], 80.00th=[ 35390], 90.00th=[ 50594], 95.00th=[ 65274], | 99.00th=[ 99091], 99.50th=[141558], 99.90th=[175113], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 24, max= 200, per=25.84%, avg=104.39, stdev=35.77, samples=119 iops : min= 6, max= 50, avg=25.73, stdev= 8.94, samples=119 lat (usec) : 100=2.47%, 250=8.43%, 500=0.20%, 750=2.57%, 1000=7.29% lat (msec) : 2=3.19%, 4=2.93%, 10=10.48%, 20=23.73%, 50=31.41% lat (msec) : 100=6.58%, 250=0.68%, 500=0.03% cpu : usr=0.14%, sys=8.10%, ctx=4677, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32291: Thu May 1 01:12:32 2025 read: IOPS=23, BW=95.9KiB/s (98.2kB/s)(5940KiB/61933msec) clat (usec): min=64, max=169981, avg=18045.46, stdev=17024.96 lat (usec): min=64, max=169983, avg=18047.12, stdev=17025.24 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 100], 10.00th=[ 117], 20.00th=[ 269], | 30.00th=[ 9765], 40.00th=[ 12649], 50.00th=[ 15008], 60.00th=[ 17957], | 70.00th=[ 22414], 80.00th=[ 27919], 90.00th=[ 37487], 95.00th=[ 48497], | 99.00th=[ 76022], 99.50th=[ 83362], 99.90th=[160433], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 8, max= 224, per=24.54%, avg=94.72, stdev=37.63, samples=123 iops : min= 2, max= 56, avg=23.34, stdev= 9.47, samples=123 write: IOPS=25, BW=102KiB/s (105kB/s)(6348KiB/61933msec) clat (usec): min=519, max=346655, avg=21951.36, stdev=26657.85 lat (usec): min=520, max=346657, avg=21956.47, stdev=26658.94 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 766], 10.00th=[ 807], 20.00th=[ 1045], | 30.00th=[ 3294], 40.00th=[ 7767], 50.00th=[ 14746], 60.00th=[ 21365], | 70.00th=[ 27919], 80.00th=[ 37487], 90.00th=[ 52691], 95.00th=[ 70779], | 99.00th=[102237], 99.50th=[149947], 99.90th=[250610], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 8, max= 240, per=24.90%, avg=100.61, stdev=41.43, samples=123 iops : min= 2, max= 60, avg=24.82, stdev=10.38, samples=123 lat (usec) : 100=2.54%, 250=7.10%, 500=0.20%, 750=2.15%, 1000=8.17% lat (msec) : 2=3.45%, 4=3.87%, 10=9.51%, 20=24.19%, 50=30.50% lat (msec) : 100=7.49%, 250=0.81%, 500=0.03% cpu : usr=0.19%, sys=8.04%, ctx=4764, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32292: Thu May 1 01:12:32 2025 read: IOPS=24, BW=97.1KiB/s (99.4kB/s)(6028KiB/62080msec) clat (usec): min=66, max=324425, avg=18291.37, stdev=19535.98 lat (usec): min=67, max=324427, avg=18293.31, stdev=19536.40 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 97], 10.00th=[ 116], 20.00th=[ 151], | 30.00th=[ 9372], 40.00th=[ 12387], 50.00th=[ 15008], 60.00th=[ 18220], | 70.00th=[ 22676], 80.00th=[ 28443], 90.00th=[ 37487], 95.00th=[ 51119], | 99.00th=[ 74974], 99.50th=[102237], 99.90th=[231736], 99.95th=[325059], | 99.99th=[325059] bw ( KiB/s): min= 15, max= 246, per=24.93%, avg=96.23, stdev=41.43, samples=124 iops : min= 3, max= 61, avg=23.61, stdev=10.34, samples=124 write: IOPS=25, BW=101KiB/s (103kB/s)(6260KiB/62080msec) clat (usec): min=622, max=335399, avg=21849.33, stdev=24946.74 lat (usec): min=623, max=335401, avg=21851.11, stdev=24946.79 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 1037], | 30.00th=[ 3294], 40.00th=[ 7111], 50.00th=[ 14353], 60.00th=[ 21890], | 70.00th=[ 28967], 80.00th=[ 39584], 90.00th=[ 54789], 95.00th=[ 68682], | 99.00th=[ 98042], 99.50th=[116917], 99.90th=[181404], 99.95th=[333448], | 99.99th=[333448] bw ( KiB/s): min= 31, max= 253, per=24.72%, avg=99.89, stdev=38.30, samples=124 iops : min= 7, max= 63, avg=24.53, stdev= 9.59, samples=124 lat (usec) : 100=2.70%, 250=7.81%, 500=0.26%, 750=2.54%, 1000=7.65% lat (msec) : 2=3.45%, 4=3.78%, 10=9.90%, 20=22.82%, 50=29.95% lat (msec) : 100=8.43%, 250=0.65%, 500=0.07% cpu : usr=0.11%, sys=8.02%, ctx=4713, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32294: Thu May 1 01:12:32 2025 read: IOPS=24, BW=98.1KiB/s (100kB/s)(6052KiB/61713msec) clat (usec): min=65, max=184472, avg=17834.88, stdev=16360.30 lat (usec): min=66, max=184472, avg=17836.30, stdev=16360.24 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 102], 10.00th=[ 116], 20.00th=[ 208], | 30.00th=[ 9503], 40.00th=[ 12518], 50.00th=[ 15401], 60.00th=[ 18482], | 70.00th=[ 21890], 80.00th=[ 27919], 90.00th=[ 38011], 95.00th=[ 47449], | 99.00th=[ 67634], 99.50th=[ 78119], 99.90th=[168821], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 23, max= 199, per=25.47%, avg=98.30, stdev=37.86, samples=122 iops : min= 5, max= 49, avg=24.19, stdev= 9.48, samples=122 write: IOPS=25, BW=101KiB/s (103kB/s)(6236KiB/61713msec) clat (usec): min=540, max=378509, avg=22097.53, stdev=27525.40 lat (usec): min=541, max=378512, avg=22100.08, stdev=27524.64 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 1123], | 30.00th=[ 3392], 40.00th=[ 8291], 50.00th=[ 14746], 60.00th=[ 21103], | 70.00th=[ 28443], 80.00th=[ 38536], 90.00th=[ 54264], 95.00th=[ 68682], | 99.00th=[ 99091], 99.50th=[115868], 99.90th=[337642], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 15, max= 240, per=24.78%, avg=100.12, stdev=36.50, samples=123 iops : min= 3, max= 60, avg=24.65, stdev= 9.14, samples=123 lat (usec) : 100=2.12%, 250=7.81%, 500=0.20%, 750=2.18%, 1000=7.26% lat (msec) : 2=3.74%, 4=3.74%, 10=9.80%, 20=24.87%, 50=30.08% lat (msec) : 100=7.68%, 250=0.42%, 500=0.10% cpu : usr=0.13%, sys=8.09%, ctx=4891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=387KiB/s (396kB/s), 95.9KiB/s-99.1KiB/s (98.2kB/s-102kB/s), io=23.4MiB (24.6MB), run=60440-62080msec WRITE: bw=405KiB/s (415kB/s), 101KiB/s-104KiB/s (103kB/s-107kB/s), io=24.6MiB (25.7MB), run=60440-62080msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32295: Thu May 1 01:14:36 2025 read: IOPS=8, BW=32.2KiB/s (32.0kB/s)(5992KiB/185990msec) clat (msec): min=3, max=373, avg=17.21, stdev=18.31 lat (msec): min=3, max=373, avg=17.21, stdev=18.31 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 34], 95.00th=[ 47], | 99.00th=[ 82], 99.50th=[ 90], 99.90th=[ 257], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 7, max= 151, per=26.59%, avg=34.31, stdev=22.81, samples=343 iops : min= 1, max= 37, avg= 8.19, stdev= 5.76, samples=343 write: IOPS=8, BW=33.9KiB/s (34.7kB/s)(6296KiB/185990msec) clat (msec): min=33, max=424, avg=101.55, stdev=53.95 lat (msec): min=33, max=424, avg=101.55, stdev=53.95 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 46], 10.00th=[ 53], 20.00th=[ 63], | 30.00th=[ 71], 40.00th=[ 80], 50.00th=[ 88], 60.00th=[ 96], | 70.00th=[ 109], 80.00th=[ 129], 90.00th=[ 171], 95.00th=[ 218], | 99.00th=[ 296], 99.50th=[ 342], 99.90th=[ 393], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 7, max= 72, per=24.71%, avg=33.36, stdev=14.36, samples=370 iops : min= 1, max= 18, avg= 7.95, stdev= 3.68, samples=370 lat (msec) : 4=0.16%, 10=19.14%, 20=17.61%, 50=13.90%, 100=30.44% lat (msec) : 250=17.15%, 500=1.60% cpu : usr=0.04%, sys=2.09%, ctx=3764, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32296: Thu May 1 01:14:36 2025 read: IOPS=7, BW=31.0KiB/s (32.7kB/s)(5940KiB/185884msec) clat (msec): min=3, max=352, avg=17.45, stdev=18.41 lat (msec): min=3, max=352, avg=17.45, stdev=18.41 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 34], 95.00th=[ 48], | 99.00th=[ 90], 99.50th=[ 106], 99.90th=[ 197], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 7, max= 120, per=26.72%, avg=34.46, stdev=22.61, samples=339 iops : min= 1, max= 30, avg= 8.26, stdev= 5.68, samples=339 write: IOPS=8, BW=34.1KiB/s (34.0kB/s)(6348KiB/185884msec) clat (msec): min=28, max=434, avg=100.66, stdev=55.04 lat (msec): min=28, max=434, avg=100.67, stdev=55.04 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 62], | 30.00th=[ 69], 40.00th=[ 78], 50.00th=[ 86], 60.00th=[ 95], | 70.00th=[ 108], 80.00th=[ 129], 90.00th=[ 169], 95.00th=[ 224], | 99.00th=[ 309], 99.50th=[ 342], 99.90th=[ 414], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 7, max= 71, per=24.98%, avg=33.72, stdev=14.72, samples=370 iops : min= 1, max= 17, avg= 8.08, stdev= 3.75, samples=370 lat (msec) : 4=0.16%, 10=18.91%, 20=17.55%, 50=13.74%, 100=31.05% lat (msec) : 250=17.15%, 500=1.43% cpu : usr=0.05%, sys=2.09%, ctx=3824, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32297: Thu May 1 01:14:36 2025 read: IOPS=8, BW=32.9KiB/s (33.7kB/s)(6028KiB/183316msec) clat (msec): min=2, max=124, avg=17.52, stdev=14.72 lat (msec): min=2, max=124, avg=17.53, stdev=14.72 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 20], 80.00th=[ 25], 90.00th=[ 37], 95.00th=[ 48], | 99.00th=[ 75], 99.50th=[ 88], 99.90th=[ 113], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 7, max= 143, per=26.20%, avg=33.80, stdev=23.10, samples=352 iops : min= 1, max= 35, avg= 8.15, stdev= 5.80, samples=352 write: IOPS=8, BW=34.1KiB/s (34.0kB/s)(6260KiB/183316msec) clat (msec): min=25, max=409, avg=100.10, stdev=54.20 lat (msec): min=25, max=409, avg=100.11, stdev=54.20 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 45], 10.00th=[ 52], 20.00th=[ 61], | 30.00th=[ 68], 40.00th=[ 75], 50.00th=[ 85], 60.00th=[ 95], | 70.00th=[ 109], 80.00th=[ 130], 90.00th=[ 169], 95.00th=[ 215], | 99.00th=[ 292], 99.50th=[ 338], 99.90th=[ 401], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 7, max= 72, per=25.06%, avg=33.83, stdev=14.94, samples=365 iops : min= 1, max= 18, avg= 8.16, stdev= 3.81, samples=365 lat (msec) : 4=0.33%, 10=18.88%, 20=16.44%, 50=16.08%, 100=29.49% lat (msec) : 250=17.51%, 500=1.27% cpu : usr=0.04%, sys=2.15%, ctx=3751, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32298: Thu May 1 01:14:36 2025 read: IOPS=8, BW=32.9KiB/s (33.7kB/s)(6052KiB/184015msec) clat (msec): min=3, max=260, avg=17.58, stdev=17.29 lat (msec): min=3, max=260, avg=17.59, stdev=17.29 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 24], 90.00th=[ 34], 95.00th=[ 48], | 99.00th=[ 80], 99.50th=[ 102], 99.90th=[ 194], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 7, max= 128, per=27.82%, avg=35.89, stdev=22.24, samples=334 iops : min= 1, max= 32, avg= 8.71, stdev= 5.61, samples=334 write: IOPS=8, BW=33.9KiB/s (34.7kB/s)(6236KiB/184015msec) clat (msec): min=25, max=394, avg=100.75, stdev=52.71 lat (msec): min=25, max=394, avg=100.76, stdev=52.71 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 62], | 30.00th=[ 70], 40.00th=[ 79], 50.00th=[ 88], 60.00th=[ 96], | 70.00th=[ 110], 80.00th=[ 129], 90.00th=[ 171], 95.00th=[ 209], | 99.00th=[ 284], 99.50th=[ 305], 99.90th=[ 363], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 7, max= 72, per=24.89%, avg=33.60, stdev=14.46, samples=367 iops : min= 1, max= 18, avg= 8.14, stdev= 3.71, samples=367 lat (msec) : 4=0.16%, 10=19.01%, 20=16.76%, 50=15.85%, 100=29.52% lat (msec) : 250=17.22%, 500=1.46% cpu : usr=0.03%, sys=2.17%, ctx=3828, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=129KiB/s (132kB/s), 31.0KiB/s-32.9KiB/s (32.7kB/s-33.7kB/s), io=23.4MiB (24.6MB), run=183316-185990msec WRITE: bw=135KiB/s (138kB/s), 33.9KiB/s-34.1KiB/s (34.7kB/s-34.0kB/s), io=24.6MiB (25.7MB), run=183316-185990msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32463: Thu May 1 01:14:48 2025 read: IOPS=36, BW=584KiB/s (598kB/s)(5920KiB/10135msec) clat (usec): min=72, max=148548, avg=12512.27, stdev=15977.66 lat (usec): min=72, max=148549, avg=12516.66, stdev=15979.18 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 92], 10.00th=[ 103], 20.00th=[ 121], | 30.00th=[ 141], 40.00th=[ 955], 50.00th=[ 9896], 60.00th=[ 12911], | 70.00th=[ 17957], 80.00th=[ 22414], 90.00th=[ 28967], 95.00th=[ 36963], | 99.00th=[ 56361], 99.50th=[115868], 99.90th=[147850], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 192, max= 1216, per=25.76%, avg=584.70, stdev=225.20, samples=20 iops : min= 12, max= 76, avg=36.30, stdev=14.05, samples=20 write: IOPS=39, BW=628KiB/s (643kB/s)(6368KiB/10135msec) clat (usec): min=791, max=164735, avg=13536.30, stdev=17004.95 lat (usec): min=792, max=164738, avg=13538.08, stdev=17004.99 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1057], 10.00th=[ 1123], 20.00th=[ 1237], | 30.00th=[ 1876], 40.00th=[ 3884], 50.00th=[ 5473], 60.00th=[ 11600], | 70.00th=[ 16712], 80.00th=[ 23462], 90.00th=[ 38011], 95.00th=[ 46924], | 99.00th=[ 61604], 99.50th=[ 64750], 99.90th=[164627], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 256, max= 1176, per=26.42%, avg=630.85, stdev=213.20, samples=20 iops : min= 16, max= 73, avg=39.20, stdev=13.30, samples=20 lat (usec) : 100=4.04%, 250=13.93%, 500=0.65%, 750=0.39%, 1000=1.17% lat (msec) : 2=15.62%, 4=6.38%, 10=12.76%, 20=19.01%, 50=22.79% lat (msec) : 100=2.86%, 250=0.39% cpu : usr=0.20%, sys=10.66%, ctx=1004, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32465: Thu May 1 01:14:48 2025 read: IOPS=34, BW=545KiB/s (558kB/s)(5744KiB/10549msec) clat (usec): min=79, max=150934, avg=13079.52, stdev=16847.32 lat (usec): min=79, max=150934, avg=13085.74, stdev=16844.12 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 95], 10.00th=[ 104], 20.00th=[ 121], | 30.00th=[ 137], 40.00th=[ 3556], 50.00th=[ 10683], 60.00th=[ 13042], | 70.00th=[ 17433], 80.00th=[ 22938], 90.00th=[ 31327], 95.00th=[ 40109], | 99.00th=[ 68682], 99.50th=[143655], 99.90th=[149947], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 128, max= 960, per=23.56%, avg=534.90, stdev=209.54, samples=20 iops : min= 8, max= 60, avg=33.10, stdev=13.08, samples=20 write: IOPS=38, BW=620KiB/s (635kB/s)(6544KiB/10549msec) clat (usec): min=876, max=163208, avg=13963.62, stdev=18103.81 lat (usec): min=877, max=163209, avg=13966.04, stdev=18103.84 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1020], 10.00th=[ 1090], 20.00th=[ 1254], | 30.00th=[ 1844], 40.00th=[ 3523], 50.00th=[ 5800], 60.00th=[ 11338], | 70.00th=[ 18744], 80.00th=[ 26346], 90.00th=[ 36439], 95.00th=[ 45876], | 99.00th=[ 64226], 99.50th=[ 67634], 99.90th=[162530], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 224, max= 958, per=25.27%, avg=603.50, stdev=188.69, samples=20 iops : min= 14, max= 59, avg=37.40, stdev=11.74, samples=20 lat (usec) : 100=3.78%, 250=13.41%, 500=0.26%, 750=0.39%, 1000=2.34% lat (msec) : 2=15.23%, 4=5.99%, 10=12.24%, 20=20.18%, 50=23.57% lat (msec) : 100=2.08%, 250=0.52% cpu : usr=0.11%, sys=10.41%, ctx=1009, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32466: Thu May 1 01:14:48 2025 read: IOPS=38, BW=615KiB/s (629kB/s)(6256KiB/10177msec) clat (usec): min=72, max=90275, avg=11555.97, stdev=14437.94 lat (usec): min=72, max=90276, avg=11557.25, stdev=14437.89 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 88], 10.00th=[ 98], 20.00th=[ 117], | 30.00th=[ 133], 40.00th=[ 161], 50.00th=[ 8586], 60.00th=[12256], | 70.00th=[16057], 80.00th=[22414], 90.00th=[29492], 95.00th=[35914], | 99.00th=[66847], 99.50th=[86508], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 317, max= 1120, per=27.44%, avg=622.80, stdev=258.90, samples=20 iops : min= 19, max= 70, avg=38.55, stdev=16.30, samples=20 write: IOPS=37, BW=593KiB/s (607kB/s)(6032KiB/10177msec) clat (usec): min=930, max=138561, avg=14878.68, stdev=18411.65 lat (usec): min=933, max=138563, avg=14880.80, stdev=18411.58 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1090], 10.00th=[ 1172], 20.00th=[ 1369], | 30.00th=[ 2540], 40.00th=[ 4424], 50.00th=[ 7308], 60.00th=[ 11863], | 70.00th=[ 19006], 80.00th=[ 26870], 90.00th=[ 37487], 95.00th=[ 44827], | 99.00th=[ 84411], 99.50th=[112722], 99.90th=[139461], 99.95th=[139461], | 99.99th=[139461] bw ( KiB/s): min= 384, max= 896, per=24.73%, avg=590.65, stdev=127.62, samples=20 iops : min= 24, max= 56, avg=36.55, stdev= 7.93, samples=20 lat (usec) : 100=5.34%, 250=16.28%, 500=0.26%, 750=0.39%, 1000=1.30% lat (msec) : 2=12.89%, 4=5.73%, 10=12.63%, 20=19.14%, 50=22.79% lat (msec) : 100=2.86%, 250=0.39% cpu : usr=0.21%, sys=10.38%, ctx=1055, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32469: Thu May 1 01:14:48 2025 read: IOPS=36, BW=589KiB/s (603kB/s)(6032KiB/10247msec) clat (usec): min=70, max=72530, avg=12939.99, stdev=14176.18 lat (usec): min=70, max=72531, avg=12952.35, stdev=14179.82 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 94], 10.00th=[ 109], 20.00th=[ 121], | 30.00th=[ 139], 40.00th=[ 2704], 50.00th=[11469], 60.00th=[15008], | 70.00th=[18482], 80.00th=[23200], 90.00th=[29754], 95.00th=[39584], | 99.00th=[64750], 99.50th=[71828], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 224, max= 926, per=25.88%, avg=587.45, stdev=168.88, samples=20 iops : min= 14, max= 57, avg=36.30, stdev=10.49, samples=20 write: IOPS=38, BW=611KiB/s (625kB/s)(6256KiB/10247msec) clat (usec): min=918, max=162048, avg=13273.53, stdev=18427.57 lat (usec): min=921, max=162050, avg=13275.46, stdev=18427.53 clat percentiles (usec): | 1.00th=[ 988], 5.00th=[ 1057], 10.00th=[ 1156], 20.00th=[ 1336], | 30.00th=[ 2089], 40.00th=[ 3458], 50.00th=[ 5604], 60.00th=[ 9372], | 70.00th=[ 14877], 80.00th=[ 23987], 90.00th=[ 34341], 95.00th=[ 44303], | 99.00th=[107480], 99.50th=[125305], 99.90th=[162530], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 288, max= 1085, per=25.53%, avg=609.65, stdev=219.57, samples=20 iops : min= 18, max= 67, avg=37.70, stdev=13.64, samples=20 lat (usec) : 100=3.26%, 250=14.32%, 500=0.78%, 750=0.52%, 1000=1.17% lat (msec) : 2=14.06%, 4=7.68%, 10=12.11%, 20=21.22%, 50=21.74% lat (msec) : 100=2.60%, 250=0.52% cpu : usr=0.20%, sys=10.83%, ctx=991, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2271KiB/s (2325kB/s), 545KiB/s-615KiB/s (558kB/s-629kB/s), io=23.4MiB (24.5MB), run=10135-10549msec WRITE: bw=2389KiB/s (2446kB/s), 593KiB/s-628KiB/s (607kB/s-643kB/s), io=24.6MiB (25.8MB), run=10135-10549msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32464: Thu May 1 01:15:28 2025 read: IOPS=7, BW=121KiB/s (124kB/s)(5920KiB/48892msec) clat (msec): min=4, max=136, avg=15.71, stdev=13.45 lat (msec): min=4, max=136, avg=15.71, stdev=13.45 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 19], 90.00th=[ 26], 95.00th=[ 42], | 99.00th=[ 85], 99.50th=[ 94], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 31, max= 416, per=25.72%, avg=123.44, stdev=73.30, samples=94 iops : min= 1, max= 26, avg= 7.31, stdev= 4.60, samples=94 write: IOPS=8, BW=130KiB/s (133kB/s)(6368KiB/48892msec) clat (msec): min=37, max=303, avg=108.03, stdev=35.07 lat (msec): min=37, max=303, avg=108.03, stdev=35.07 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 59], 10.00th=[ 70], 20.00th=[ 81], | 30.00th=[ 89], 40.00th=[ 96], 50.00th=[ 104], 60.00th=[ 113], | 70.00th=[ 123], 80.00th=[ 132], 90.00th=[ 150], 95.00th=[ 169], | 99.00th=[ 220], 99.50th=[ 228], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 63, max= 192, per=25.87%, avg=130.64, stdev=31.94, samples=96 iops : min= 3, max= 12, avg= 7.76, stdev= 2.08, samples=96 lat (msec) : 10=17.32%, 20=22.53%, 50=8.07%, 100=23.83%, 250=28.12% lat (msec) : 500=0.13% cpu : usr=0.02%, sys=2.18%, ctx=889, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32467: Thu May 1 01:15:28 2025 read: IOPS=7, BW=115KiB/s (118kB/s)(5744KiB/49848msec) clat (msec): min=3, max=121, avg=15.62, stdev=13.40 lat (msec): min=3, max=121, avg=15.62, stdev=13.40 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 28], 95.00th=[ 39], | 99.00th=[ 77], 99.50th=[ 121], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 31, max= 352, per=26.08%, avg=125.20, stdev=71.24, samples=90 iops : min= 1, max= 22, avg= 7.33, stdev= 4.57, samples=90 write: IOPS=8, BW=131KiB/s (134kB/s)(6544KiB/49848msec) clat (msec): min=38, max=314, avg=107.62, stdev=39.30 lat (msec): min=38, max=314, avg=107.62, stdev=39.30 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 58], 10.00th=[ 65], 20.00th=[ 78], | 30.00th=[ 86], 40.00th=[ 94], 50.00th=[ 103], 60.00th=[ 110], | 70.00th=[ 118], 80.00th=[ 133], 90.00th=[ 157], 95.00th=[ 182], | 99.00th=[ 247], 99.50th=[ 262], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 31, max= 234, per=25.95%, avg=131.06, stdev=38.76, samples=98 iops : min= 1, max= 14, avg= 7.71, stdev= 2.45, samples=98 lat (msec) : 4=0.26%, 10=15.23%, 20=22.66%, 50=8.59%, 100=25.00% lat (msec) : 250=27.86%, 500=0.39% cpu : usr=0.04%, sys=2.19%, ctx=877, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32468: Thu May 1 01:15:28 2025 read: IOPS=8, BW=132KiB/s (135kB/s)(6256KiB/47417msec) clat (usec): min=4829, max=87198, avg=15781.89, stdev=13092.24 lat (usec): min=4831, max=87199, avg=15783.72, stdev=13092.10 clat percentiles (usec): | 1.00th=[ 5145], 5.00th=[ 6259], 10.00th=[ 6980], 20.00th=[ 8356], | 30.00th=[ 9372], 40.00th=[10290], 50.00th=[11469], 60.00th=[13173], | 70.00th=[14746], 80.00th=[17957], 90.00th=[30016], 95.00th=[46924], | 99.00th=[76022], 99.50th=[80217], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 31, max= 352, per=28.84%, avg=138.43, stdev=84.71, samples=90 iops : min= 1, max= 22, avg= 8.18, stdev= 5.38, samples=90 write: IOPS=7, BW=127KiB/s (130kB/s)(6032KiB/47417msec) clat (msec): min=35, max=339, avg=108.91, stdev=38.07 lat (msec): min=35, max=339, avg=108.91, stdev=38.07 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 61], 10.00th=[ 68], 20.00th=[ 79], | 30.00th=[ 85], 40.00th=[ 94], 50.00th=[ 104], 60.00th=[ 114], | 70.00th=[ 126], 80.00th=[ 136], 90.00th=[ 157], 95.00th=[ 180], | 99.00th=[ 226], 99.50th=[ 232], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 32, max= 224, per=25.31%, avg=127.80, stdev=33.48, samples=93 iops : min= 2, max= 14, avg= 7.52, stdev= 2.14, samples=93 lat (msec) : 10=17.97%, 20=23.70%, 50=7.55%, 100=24.74%, 250=25.91% lat (msec) : 500=0.13% cpu : usr=0.04%, sys=2.24%, ctx=903, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32470: Thu May 1 01:15:28 2025 read: IOPS=7, BW=124KiB/s (127kB/s)(6032KiB/48772msec) clat (msec): min=4, max=113, avg=16.12, stdev=14.33 lat (msec): min=4, max=113, avg=16.12, stdev=14.33 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 29], 95.00th=[ 42], | 99.00th=[ 85], 99.50th=[ 109], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 31, max= 351, per=27.05%, avg=129.86, stdev=78.62, samples=91 iops : min= 1, max= 21, avg= 7.63, stdev= 4.92, samples=91 write: IOPS=8, BW=128KiB/s (131kB/s)(6256KiB/48772msec) clat (msec): min=38, max=278, avg=108.28, stdev=38.95 lat (msec): min=38, max=278, avg=108.28, stdev=38.95 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 65], 20.00th=[ 74], | 30.00th=[ 86], 40.00th=[ 95], 50.00th=[ 103], 60.00th=[ 112], | 70.00th=[ 123], 80.00th=[ 138], 90.00th=[ 157], 95.00th=[ 178], | 99.00th=[ 236], 99.50th=[ 271], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 63, max= 224, per=25.56%, avg=129.09, stdev=36.75, samples=96 iops : min= 3, max= 14, avg= 7.58, stdev= 2.42, samples=96 lat (msec) : 10=18.23%, 20=21.35%, 50=8.33%, 100=24.61%, 250=27.08% lat (msec) : 500=0.39% cpu : usr=0.03%, sys=2.17%, ctx=907, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=481KiB/s (492kB/s), 115KiB/s-132KiB/s (118kB/s-135kB/s), io=23.4MiB (24.5MB), run=47417-49848msec WRITE: bw=506KiB/s (518kB/s), 127KiB/s-131KiB/s (130kB/s-134kB/s), io=24.6MiB (25.8MB), run=47417-49848msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32515: Thu May 1 01:15:33 2025 read: IOPS=2, BW=2237KiB/s (2290kB/s)(7168KiB/3205msec) clat (usec): min=672, max=1574.6k, avg=321674.87, stdev=555995.57 lat (usec): min=673, max=1574.6k, avg=321676.64, stdev=555995.52 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], | 20.00th=[ 78119], 30.00th=[ 112722], 40.00th=[ 112722], | 50.00th=[ 130548], 60.00th=[ 156238], 70.00th=[ 156238], | 80.00th=[ 198181], 90.00th=[1568670], 95.00th=[1568670], | 99.00th=[1568670], 99.50th=[1568670], 99.90th=[1568670], | 99.95th=[1568670], 99.99th=[1568670] bw ( KiB/s): min= 2027, max= 4087, per=47.73%, avg=3050.00, stdev=1179.02, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 1.15, samples=4 write: IOPS=1, BW=1598KiB/s (1636kB/s)(5120KiB/3205msec) clat (msec): min=132, max=270, avg=176.29, stdev=55.95 lat (msec): min=133, max=270, avg=176.54, stdev=55.78 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 4055, max= 4055, per=45.33%, avg=4055.00, stdev= 0.00, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 lat (usec) : 750=8.33% lat (msec) : 100=8.33%, 250=66.67%, 500=8.33% cpu : usr=0.00%, sys=9.49%, ctx=247, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32516: Thu May 1 01:15:33 2025 read: IOPS=0, BW=900KiB/s (922kB/s)(2048KiB/2275msec) clat (msec): min=147, max=206, avg=177.46, stdev=41.68 lat (msec): min=147, max=206, avg=177.46, stdev=41.68 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 207], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 2043, max= 2048, per=32.01%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4501KiB/s (4609kB/s)(10.0MiB/2275msec) clat (msec): min=42, max=934, avg=189.08, stdev=283.48 lat (msec): min=42, max=934, avg=189.16, stdev=283.48 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 54], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 68], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 401], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 6131, max=10178, per=91.15%, avg=8154.50, stdev=2861.66, samples=2 iops : min= 5, max= 9, avg= 7.00, stdev= 2.83, samples=2 lat (msec) : 50=16.67%, 100=41.67%, 250=25.00%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=11.79%, ctx=224, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32517: Thu May 1 01:15:33 2025 read: IOPS=3, BW=3124KiB/s (3199kB/s)(6144KiB/1967msec) clat (usec): min=585, max=844406, avg=195863.51, stdev=323010.28 lat (usec): min=587, max=844408, avg=195864.99, stdev=323010.29 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 971], | 30.00th=[ 971], 40.00th=[ 86508], 50.00th=[ 86508], 60.00th=[ 90702], | 70.00th=[152044], 80.00th=[152044], 90.00th=[843056], 95.00th=[843056], | 99.00th=[843056], 99.50th=[843056], 99.90th=[843056], 99.95th=[843056], | 99.99th=[843056] bw ( KiB/s): min= 2043, max= 8175, per=79.95%, avg=5109.00, stdev=4335.98, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 write: IOPS=3, BW=3124KiB/s (3199kB/s)(6144KiB/1967msec) clat (msec): min=70, max=214, avg=101.21, stdev=56.18 lat (msec): min=70, max=215, avg=101.27, stdev=56.19 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 73], 40.00th=[ 75], 50.00th=[ 75], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 2043, max= 6131, per=45.69%, avg=4087.00, stdev=2044.00, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 100=58.33%, 250=16.67%, 1000=8.33% cpu : usr=0.10%, sys=13.33%, ctx=238, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32518: Thu May 1 01:15:33 2025 read: IOPS=1, BW=1772KiB/s (1814kB/s)(5120KiB/2890msec) clat (usec): min=559, max=146683, avg=86771.39, stdev=59219.29 lat (usec): min=561, max=146685, avg=86773.21, stdev=59219.28 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[ 562], | 30.00th=[ 68682], 40.00th=[ 68682], 50.00th=[ 80217], 60.00th=[ 80217], | 70.00th=[137364], 80.00th=[137364], 90.00th=[145753], 95.00th=[145753], | 99.00th=[145753], 99.50th=[145753], 99.90th=[145753], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 2048, max= 6144, per=64.10%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2480KiB/s (2540kB/s)(7168KiB/2890msec) clat (msec): min=63, max=689, avg=332.83, stdev=214.40 lat (msec): min=63, max=690, avg=333.10, stdev=214.43 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 209], | 30.00th=[ 239], 40.00th=[ 239], 50.00th=[ 251], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 542], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 2043, max= 6144, per=34.33%, avg=3070.75, stdev=2048.83, samples=4 iops : min= 1, max= 6, avg= 2.75, stdev= 2.22, samples=4 lat (usec) : 750=8.33% lat (msec) : 100=25.00%, 250=33.33%, 500=16.67%, 750=16.67% cpu : usr=0.00%, sys=8.65%, ctx=209, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6390KiB/s (6543kB/s), 900KiB/s-3124KiB/s (922kB/s-3199kB/s), io=20.0MiB (20.0MB), run=1967-3205msec WRITE: bw=8946KiB/s (9161kB/s), 1598KiB/s-4501KiB/s (1636kB/s-4609kB/s), io=28.0MiB (29.4MB), run=1967-3205msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32520: Thu May 1 01:15:33 2025 read: IOPS=1, BW=1900KiB/s (1946kB/s)(7168KiB/3772msec) clat (msec): min=64, max=1054, avg=364.05, stdev=402.73 lat (msec): min=64, max=1055, avg=364.05, stdev=402.73 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 66], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 176], 60.00th=[ 230], | 70.00th=[ 230], 80.00th=[ 827], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 1756, max= 4031, per=43.55%, avg=2364.60, stdev=938.39, samples=5 iops : min= 1, max= 3, avg= 1.60, stdev= 0.89, samples=5 write: IOPS=1, BW=1357KiB/s (1390kB/s)(5120KiB/3772msec) clat (msec): min=136, max=288, avg=202.65, stdev=57.77 lat (msec): min=136, max=288, avg=202.76, stdev=57.76 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 2015, max= 5269, per=47.91%, avg=3642.00, stdev=2300.93, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 100=16.67%, 250=58.33%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=3.50%, ctx=51, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32521: Thu May 1 01:15:33 2025 read: IOPS=0, BW=744KiB/s (762kB/s)(2048KiB/2752msec) clat (msec): min=53, max=182, avg=118.04, stdev=90.90 lat (msec): min=53, max=182, avg=118.05, stdev=90.90 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 182], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 2031, max= 2048, per=37.57%, avg=2039.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3721KiB/s (3810kB/s)(10.0MiB/2752msec) clat (msec): min=141, max=400, avg=228.53, stdev=82.05 lat (msec): min=141, max=400, avg=228.61, stdev=82.05 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 190], 60.00th=[ 226], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 317], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 2019, max= 6144, per=53.48%, avg=4064.75, stdev=1684.59, samples=4 iops : min= 1, max= 6, avg= 3.50, stdev= 2.08, samples=4 lat (msec) : 100=8.33%, 250=58.33%, 500=33.33% cpu : usr=0.00%, sys=4.69%, ctx=48, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32522: Thu May 1 01:15:33 2025 read: IOPS=1, BW=2016KiB/s (2064kB/s)(6144KiB/3048msec) clat (msec): min=56, max=416, avg=178.01, stdev=142.95 lat (msec): min=56, max=416, avg=178.01, stdev=142.95 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 125], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 2048, max= 4096, per=62.47%, avg=3391.67, stdev=1164.10, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=1, BW=2016KiB/s (2064kB/s)(6144KiB/3048msec) clat (msec): min=91, max=675, avg=265.09, stdev=213.70 lat (msec): min=91, max=676, avg=265.17, stdev=213.70 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 228], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 1838, max= 2048, per=26.35%, avg=2003.17, stdev=81.81, samples=6 iops : min= 1, max= 2, avg= 1.33, stdev= 0.52, samples=6 lat (msec) : 100=25.00%, 250=41.67%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=3.51%, ctx=47, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32523: Thu May 1 01:15:33 2025 read: IOPS=1, BW=1692KiB/s (1733kB/s)(5120KiB/3026msec) clat (msec): min=65, max=597, avg=191.76, stdev=227.07 lat (msec): min=65, max=597, avg=191.76, stdev=227.07 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 94], 40.00th=[ 94], 50.00th=[ 97], 60.00th=[ 97], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2048, max= 6107, per=75.11%, avg=4077.50, stdev=2870.15, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2369KiB/s (2426kB/s)(7168KiB/3026msec) clat (msec): min=162, max=546, avg=290.84, stdev=125.72 lat (msec): min=162, max=546, avg=290.97, stdev=125.72 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 188], | 30.00th=[ 262], 40.00th=[ 262], 50.00th=[ 266], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 317], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2035, max= 4096, per=37.31%, avg=2836.00, stdev=1086.14, samples=5 iops : min= 1, max= 4, avg= 2.40, stdev= 1.14, samples=5 lat (msec) : 100=25.00%, 250=25.00%, 500=33.33%, 750=16.67% cpu : usr=0.07%, sys=3.37%, ctx=49, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5429KiB/s (5560kB/s), 744KiB/s-2016KiB/s (762kB/s-2064kB/s), io=20.0MiB (20.0MB), run=2752-3772msec WRITE: bw=7601KiB/s (7784kB/s), 1357KiB/s-3721KiB/s (1390kB/s-3810kB/s), io=28.0MiB (29.4MB), run=2752-3772msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32532: Thu May 1 01:15:38 2025 read: IOPS=1, BW=7807KiB/s (7994kB/s)(12.0MiB/1574msec) clat (msec): min=466, max=568, avg=515.28, stdev=50.99 lat (msec): min=466, max=568, avg=515.28, stdev=50.99 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 468], 10.00th=[ 468], 20.00th=[ 468], | 30.00th=[ 468], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=14.05%, ctx=304, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32533: Thu May 1 01:15:38 2025 read: IOPS=0, BW=1591KiB/s (1629kB/s)(4096KiB/2575msec) clat (nsec): min=1066.8M, max=1066.8M, avg=1066813927.00, stdev= 0.00 lat (nsec): min=1066.8M, max=1066.8M, avg=1066816798.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1070], 5.00th=[ 1070], 10.00th=[ 1070], 20.00th=[ 1070], | 30.00th=[ 1070], 40.00th=[ 1070], 50.00th=[ 1070], 60.00th=[ 1070], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8094, max= 8094, per=100.00%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3181KiB/s (3258kB/s)(8192KiB/2575msec) clat (msec): min=669, max=775, avg=722.39, stdev=74.83 lat (msec): min=669, max=775, avg=722.72, stdev=74.72 clat percentiles (msec): | 1.00th=[ 667], 5.00th=[ 667], 10.00th=[ 667], 20.00th=[ 667], | 30.00th=[ 667], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 776], | 70.00th=[ 776], 80.00th=[ 776], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 8159, max= 8159, per=64.12%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=10.76%, ctx=104, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32535: Thu May 1 01:15:38 2025 write: IOPS=1, BW=4878KiB/s (4995kB/s)(12.0MiB/2519msec) clat (msec): min=631, max=940, avg=824.15, stdev=168.24 lat (msec): min=631, max=940, avg=826.50, stdev=169.48 clat percentiles (msec): | 1.00th=[ 634], 5.00th=[ 634], 10.00th=[ 634], 20.00th=[ 634], | 30.00th=[ 634], 40.00th=[ 902], 50.00th=[ 902], 60.00th=[ 902], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 8143, max= 8192, per=64.18%, avg=8167.50, stdev=34.65, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33%, 1000=66.67% cpu : usr=0.08%, sys=10.76%, ctx=106, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32536: Thu May 1 01:15:38 2025 write: IOPS=1, BW=5980KiB/s (6123kB/s)(12.0MiB/2055msec) clat (msec): min=369, max=1147, avg=656.32, stdev=427.78 lat (msec): min=369, max=1148, avg=656.81, stdev=427.74 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min=16031, max=16031, per=100.00%, avg=16031.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=11.93%, ctx=87, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6363KiB/s (6515kB/s), 1591KiB/s-7807KiB/s (1629kB/s-7994kB/s), io=16.0MiB (16.8MB), run=1574-2575msec WRITE: bw=12.4MiB/s (13.0MB/s), 3181KiB/s-5980KiB/s (3258kB/s-6123kB/s), io=32.0MiB (33.6MB), run=2055-2575msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32537: Thu May 1 01:15:39 2025 read: IOPS=1, BW=4475KiB/s (4582kB/s)(12.0MiB/2746msec) clat (msec): min=149, max=1430, avg=885.64, stdev=661.48 lat (msec): min=149, max=1430, avg=885.64, stdev=661.48 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 1083], 50.00th=[ 1083], 60.00th=[ 1083], | 70.00th=[ 1435], 80.00th=[ 1435], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=3.10%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32538: Thu May 1 01:15:39 2025 read: IOPS=0, BW=1402KiB/s (1435kB/s)(4096KiB/2922msec) clat (nsec): min=1320.8M, max=1320.8M, avg=1320830873.00, stdev= 0.00 lat (nsec): min=1320.8M, max=1320.8M, avg=1320832935.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1318], 5.00th=[ 1318], 10.00th=[ 1318], 20.00th=[ 1318], | 30.00th=[ 1318], 40.00th=[ 1318], 50.00th=[ 1318], 60.00th=[ 1318], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 8094, max= 8094, per=100.00%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2804KiB/s (2871kB/s)(8192KiB/2922msec) clat (msec): min=742, max=805, avg=773.71, stdev=44.65 lat (msec): min=742, max=805, avg=774.30, stdev=44.80 clat percentiles (msec): | 1.00th=[ 743], 5.00th=[ 743], 10.00th=[ 743], 20.00th=[ 743], | 30.00th=[ 743], 40.00th=[ 743], 50.00th=[ 743], 60.00th=[ 802], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 5769, max= 5769, per=51.74%, avg=5769.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.10%, sys=4.35%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32539: Thu May 1 01:15:39 2025 write: IOPS=1, BW=4181KiB/s (4281kB/s)(12.0MiB/2939msec) clat (msec): min=497, max=1576, avg=930.15, stdev=570.13 lat (msec): min=497, max=1581, avg=932.22, stdev=572.71 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 498], 40.00th=[ 718], 50.00th=[ 718], 60.00th=[ 718], | 70.00th=[ 1569], 80.00th=[ 1569], 90.00th=[ 1569], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 6224, max= 8192, per=64.65%, avg=7208.00, stdev=1391.59, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.03%, sys=2.55%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32540: Thu May 1 01:15:39 2025 write: IOPS=1, BW=5283KiB/s (5410kB/s)(12.0MiB/2326msec) clat (msec): min=264, max=1209, avg=625.39, stdev=510.36 lat (msec): min=267, max=1209, avg=626.81, stdev=509.35 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 8110, max= 8110, per=72.74%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.04%, sys=7.83%, ctx=64, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5607KiB/s (5742kB/s), 1402KiB/s-4475KiB/s (1435kB/s-4582kB/s), io=16.0MiB (16.8MB), run=2746-2922msec WRITE: bw=10.9MiB/s (11.4MB/s), 2804KiB/s-5283KiB/s (2871kB/s-5410kB/s), io=32.0MiB (33.6MB), run=2326-2939msec