== sanity test 398b: DIO and buffer IO race ============== 22:11:52 (1746151912) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.74545 s, 28.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=935: Thu May 1 22:12:30 2025 read: IOPS=44, BW=178KiB/s (183kB/s)(5992KiB/33571msec) clat (usec): min=60, max=268931, avg=9841.73, stdev=11064.00 lat (usec): min=61, max=268932, avg=9843.38, stdev=11063.84 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 95], 10.00th=[ 105], 20.00th=[ 137], | 30.00th=[ 5014], 40.00th=[ 6652], 50.00th=[ 8225], 60.00th=[ 10028], | 70.00th=[ 12780], 80.00th=[ 15795], 90.00th=[ 20579], 95.00th=[ 26608], | 99.00th=[ 38011], 99.50th=[ 43779], 99.90th=[ 79168], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 48, max= 320, per=25.22%, avg=178.83, stdev=51.83, samples=66 iops : min= 12, max= 80, avg=44.32, stdev=13.02, samples=66 write: IOPS=46, BW=188KiB/s (192kB/s)(6296KiB/33571msec) clat (usec): min=455, max=97666, avg=11827.67, stdev=12398.52 lat (usec): min=456, max=97667, avg=11832.14, stdev=12399.60 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 742], 10.00th=[ 807], 20.00th=[ 1090], | 30.00th=[ 2212], 40.00th=[ 5407], 50.00th=[ 8848], 60.00th=[11600], | 70.00th=[15270], 80.00th=[20055], 90.00th=[28443], 95.00th=[36439], | 99.00th=[55837], 99.50th=[67634], 99.90th=[72877], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 32, max= 310, per=25.21%, avg=187.08, stdev=54.36, samples=66 iops : min= 8, max= 77, avg=46.38, stdev=13.60, samples=66 lat (usec) : 100=3.81%, 250=7.26%, 500=0.52%, 750=3.03%, 1000=6.64% lat (msec) : 2=5.86%, 4=4.69%, 10=24.77%, 20=27.57%, 50=15.10% lat (msec) : 100=0.72%, 500=0.03% cpu : usr=0.19%, sys=13.33%, ctx=4908, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=936: Thu May 1 22:12:30 2025 read: IOPS=43, BW=176KiB/s (180kB/s)(5940KiB/33846msec) clat (usec): min=60, max=216934, avg=9301.61, stdev=10545.70 lat (usec): min=60, max=216936, avg=9302.90, stdev=10545.72 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 91], 10.00th=[ 102], 20.00th=[ 127], | 30.00th=[ 3163], 40.00th=[ 6325], 50.00th=[ 7701], 60.00th=[ 9503], | 70.00th=[ 12125], 80.00th=[ 15139], 90.00th=[ 20317], 95.00th=[ 25822], | 99.00th=[ 39060], 99.50th=[ 43779], 99.90th=[113771], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 7, max= 312, per=24.68%, avg=174.99, stdev=57.49, samples=67 iops : min= 1, max= 78, avg=43.54, stdev=14.41, samples=67 write: IOPS=46, BW=188KiB/s (192kB/s)(6348KiB/33846msec) clat (usec): min=467, max=108755, avg=12431.25, stdev=13956.74 lat (usec): min=467, max=108756, avg=12435.10, stdev=13954.80 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 1045], | 30.00th=[ 1958], 40.00th=[ 4359], 50.00th=[ 8848], 60.00th=[ 12256], | 70.00th=[ 15533], 80.00th=[ 20841], 90.00th=[ 28967], 95.00th=[ 40109], | 99.00th=[ 66847], 99.50th=[ 73925], 99.90th=[ 96994], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 8, max= 335, per=25.09%, avg=186.13, stdev=63.44, samples=67 iops : min= 2, max= 83, avg=46.33, stdev=15.87, samples=67 lat (usec) : 100=4.56%, 250=8.24%, 500=0.72%, 750=3.97%, 1000=6.35% lat (msec) : 2=6.15%, 4=4.95%, 10=22.20%, 20=26.56%, 50=15.04% lat (msec) : 100=1.17%, 250=0.10% cpu : usr=0.23%, sys=13.01%, ctx=4847, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=937: Thu May 1 22:12:30 2025 read: IOPS=45, BW=181KiB/s (185kB/s)(6028KiB/33350msec) clat (usec): min=66, max=67231, avg=9044.05, stdev=8349.18 lat (usec): min=67, max=67232, avg=9045.20, stdev=8349.16 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 95], 10.00th=[ 103], 20.00th=[ 135], | 30.00th=[ 4752], 40.00th=[ 6390], 50.00th=[ 7898], 60.00th=[ 9503], | 70.00th=[11338], 80.00th=[14746], 90.00th=[19268], 95.00th=[24511], | 99.00th=[35914], 99.50th=[42730], 99.90th=[65274], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 48, max= 375, per=25.50%, avg=180.79, stdev=55.06, samples=66 iops : min= 12, max= 93, avg=44.92, stdev=13.77, samples=66 write: IOPS=46, BW=188KiB/s (192kB/s)(6260KiB/33350msec) clat (usec): min=487, max=254107, avg=12468.64, stdev=14082.09 lat (usec): min=487, max=254108, avg=12473.16, stdev=14086.25 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 742], 10.00th=[ 816], 20.00th=[ 1139], | 30.00th=[ 2474], 40.00th=[ 5604], 50.00th=[ 9372], 60.00th=[ 12911], | 70.00th=[ 16319], 80.00th=[ 20317], 90.00th=[ 28967], 95.00th=[ 36439], | 99.00th=[ 62129], 99.50th=[ 70779], 99.90th=[102237], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 64, max= 311, per=25.28%, avg=187.59, stdev=48.35, samples=66 iops : min= 16, max= 77, avg=46.62, stdev=12.12, samples=66 lat (usec) : 100=4.17%, 250=7.65%, 500=0.39%, 750=3.12%, 1000=6.02% lat (msec) : 2=5.83%, 4=4.79%, 10=25.26%, 20=27.90%, 50=13.93% lat (msec) : 100=0.88%, 250=0.03%, 500=0.03% cpu : usr=0.17%, sys=13.45%, ctx=5068, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=938: Thu May 1 22:12:30 2025 read: IOPS=45, BW=182KiB/s (186kB/s)(6052KiB/33314msec) clat (usec): min=48, max=289156, avg=9343.23, stdev=11342.63 lat (usec): min=48, max=289157, avg=9345.10, stdev=11342.27 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 133], | 30.00th=[ 4686], 40.00th=[ 6325], 50.00th=[ 7767], 60.00th=[ 9241], | 70.00th=[ 11863], 80.00th=[ 15139], 90.00th=[ 20055], 95.00th=[ 25560], | 99.00th=[ 37487], 99.50th=[ 44303], 99.90th=[ 83362], 99.95th=[287310], | 99.99th=[287310] bw ( KiB/s): min= 32, max= 344, per=25.49%, avg=180.74, stdev=53.78, samples=66 iops : min= 8, max= 86, avg=44.95, stdev=13.51, samples=66 write: IOPS=46, BW=187KiB/s (192kB/s)(6236KiB/33314msec) clat (usec): min=405, max=143678, avg=12161.88, stdev=13153.92 lat (usec): min=406, max=143679, avg=12165.03, stdev=13153.08 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 807], 20.00th=[ 1106], | 30.00th=[ 2089], 40.00th=[ 4948], 50.00th=[ 8586], 60.00th=[ 12125], | 70.00th=[ 15926], 80.00th=[ 21103], 90.00th=[ 29230], 95.00th=[ 36963], | 99.00th=[ 55837], 99.50th=[ 63177], 99.90th=[ 90702], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 48, max= 319, per=25.16%, avg=186.68, stdev=53.76, samples=66 iops : min= 12, max= 79, avg=46.44, stdev=13.49, samples=66 lat (usec) : 50=0.03%, 100=4.17%, 250=8.14%, 500=0.42%, 750=3.39% lat (usec) : 1000=6.45% lat (msec) : 2=5.76%, 4=4.62%, 10=25.68%, 20=24.80%, 50=15.56% lat (msec) : 100=0.91%, 250=0.03%, 500=0.03% cpu : usr=0.18%, sys=13.26%, ctx=4919, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=709KiB/s (726kB/s), 176KiB/s-182KiB/s (180kB/s-186kB/s), io=23.4MiB (24.6MB), run=33314-33846msec WRITE: bw=743KiB/s (761kB/s), 187KiB/s-188KiB/s (192kB/s-192kB/s), io=24.6MiB (25.7MB), run=33314-33846msec rand-rw: (groupid=0, jobs=1): err= 0: pid=940: Thu May 1 22:13:46 2025 read: IOPS=13, BW=54.7KiB/s (56.0kB/s)(5992KiB/109500msec) clat (msec): min=2, max=271, avg=10.55, stdev=12.10 lat (msec): min=2, max=271, avg=10.55, stdev=12.10 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 21], 95.00th=[ 27], | 99.00th=[ 55], 99.50th=[ 75], 99.90th=[ 128], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 7, max= 160, per=25.22%, avg=54.98, stdev=32.89, samples=215 iops : min= 1, max= 40, avg=13.51, stdev= 8.29, samples=215 write: IOPS=14, BW=57.5KiB/s (58.9kB/s)(6296KiB/109500msec) clat (msec): min=14, max=258, avg=59.42, stdev=29.39 lat (msec): min=14, max=258, avg=59.42, stdev=29.39 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 24], 10.00th=[ 28], 20.00th=[ 34], | 30.00th=[ 41], 40.00th=[ 48], 50.00th=[ 55], 60.00th=[ 62], | 70.00th=[ 69], 80.00th=[ 81], 90.00th=[ 97], 95.00th=[ 114], | 99.00th=[ 155], 99.50th=[ 182], 99.90th=[ 218], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 8, max= 128, per=24.99%, avg=56.97, stdev=24.61, samples=218 iops : min= 2, max= 32, avg=14.00, stdev= 6.23, samples=218 lat (msec) : 4=4.07%, 10=30.86%, 20=9.80%, 50=25.68%, 100=25.00% lat (msec) : 250=4.52%, 500=0.07% cpu : usr=0.09%, sys=2.94%, ctx=3683, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=941: Thu May 1 22:13:46 2025 read: IOPS=13, BW=54.1KiB/s (55.4kB/s)(5940KiB/109833msec) clat (msec): min=2, max=293, avg=10.35, stdev=11.82 lat (msec): min=2, max=293, avg=10.35, stdev=11.82 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 20], 95.00th=[ 29], | 99.00th=[ 47], 99.50th=[ 56], 99.90th=[ 124], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 7, max= 184, per=24.83%, avg=54.13, stdev=34.97, samples=217 iops : min= 1, max= 46, avg=13.35, stdev= 8.79, samples=217 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(6348KiB/109833msec) clat (msec): min=14, max=219, avg=59.41, stdev=30.19 lat (msec): min=14, max=219, avg=59.42, stdev=30.19 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 23], 10.00th=[ 27], 20.00th=[ 34], | 30.00th=[ 41], 40.00th=[ 47], 50.00th=[ 55], 60.00th=[ 63], | 70.00th=[ 70], 80.00th=[ 81], 90.00th=[ 99], 95.00th=[ 117], | 99.00th=[ 161], 99.50th=[ 184], 99.90th=[ 213], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 7, max= 143, per=25.08%, avg=57.18, stdev=25.38, samples=219 iops : min= 1, max= 35, avg=14.11, stdev= 6.38, samples=219 lat (msec) : 4=3.97%, 10=30.99%, 20=10.45%, 50=25.68%, 100=24.02% lat (msec) : 250=4.85%, 500=0.03% cpu : usr=0.07%, sys=2.92%, ctx=3699, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=942: Thu May 1 22:13:46 2025 read: IOPS=13, BW=54.9KiB/s (56.2kB/s)(6028KiB/109770msec) clat (usec): min=2095, max=92810, avg=10219.41, stdev=8438.71 lat (usec): min=2097, max=92811, avg=10222.22, stdev=8440.51 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3621], 10.00th=[ 4490], 20.00th=[ 5407], | 30.00th=[ 6194], 40.00th=[ 6849], 50.00th=[ 7570], 60.00th=[ 8455], | 70.00th=[ 9896], 80.00th=[12649], 90.00th=[19792], 95.00th=[26870], | 99.00th=[42730], 99.50th=[52167], 99.90th=[82314], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 7, max= 168, per=25.08%, avg=54.67, stdev=34.97, samples=218 iops : min= 1, max= 42, avg=13.41, stdev= 8.80, samples=218 write: IOPS=14, BW=57.0KiB/s (58.4kB/s)(6260KiB/109770msec) clat (msec): min=15, max=331, avg=60.18, stdev=30.88 lat (msec): min=15, max=331, avg=60.19, stdev=30.88 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 24], 10.00th=[ 27], 20.00th=[ 34], | 30.00th=[ 41], 40.00th=[ 47], 50.00th=[ 55], 60.00th=[ 64], | 70.00th=[ 72], 80.00th=[ 83], 90.00th=[ 100], 95.00th=[ 117], | 99.00th=[ 159], 99.50th=[ 174], 99.90th=[ 226], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 7, max= 136, per=24.75%, avg=56.42, stdev=25.10, samples=219 iops : min= 1, max= 34, avg=13.85, stdev= 6.34, samples=219 lat (msec) : 4=3.39%, 10=31.38%, 20=10.48%, 50=25.81%, 100=24.02% lat (msec) : 250=4.88%, 500=0.03% cpu : usr=0.06%, sys=2.94%, ctx=3735, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=943: Thu May 1 22:13:46 2025 read: IOPS=13, BW=55.2KiB/s (56.5kB/s)(6052KiB/109638msec) clat (usec): min=1920, max=73915, avg=10073.65, stdev=8456.70 lat (usec): min=1920, max=73916, avg=10075.26, stdev=8456.64 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3425], 10.00th=[ 4113], 20.00th=[ 5211], | 30.00th=[ 5997], 40.00th=[ 6783], 50.00th=[ 7439], 60.00th=[ 8356], | 70.00th=[ 9634], 80.00th=[11994], 90.00th=[19530], 95.00th=[27132], | 99.00th=[47973], 99.50th=[52691], 99.90th=[62653], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 7, max= 176, per=25.88%, avg=56.42, stdev=34.04, samples=213 iops : min= 1, max= 44, avg=13.92, stdev= 8.54, samples=213 write: IOPS=14, BW=56.9KiB/s (58.2kB/s)(6236KiB/109638msec) clat (msec): min=14, max=248, avg=60.38, stdev=31.44 lat (msec): min=14, max=248, avg=60.38, stdev=31.44 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 23], 10.00th=[ 27], 20.00th=[ 34], | 30.00th=[ 40], 40.00th=[ 48], 50.00th=[ 55], 60.00th=[ 63], | 70.00th=[ 72], 80.00th=[ 84], 90.00th=[ 101], 95.00th=[ 122], | 99.00th=[ 165], 99.50th=[ 182], 99.90th=[ 197], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 16, max= 128, per=24.78%, avg=56.51, stdev=25.23, samples=219 iops : min= 4, max= 32, avg=13.93, stdev= 6.34, samples=219 lat (msec) : 2=0.03%, 4=4.46%, 10=31.28%, 20=9.93%, 50=25.33% lat (msec) : 100=23.86%, 250=5.11% cpu : usr=0.08%, sys=2.91%, ctx=3710, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=219KiB/s (224kB/s), 54.1KiB/s-55.2KiB/s (55.4kB/s-56.5kB/s), io=23.4MiB (24.6MB), run=109500-109833msec WRITE: bw=229KiB/s (234kB/s), 56.9KiB/s-57.8KiB/s (58.2kB/s-59.2kB/s), io=24.6MiB (25.7MB), run=109500-109833msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1046: Thu May 1 22:13:51 2025 read: IOPS=94, BW=1519KiB/s (1555kB/s)(5920KiB/3898msec) clat (usec): min=49, max=23935, avg=4560.75, stdev=4452.79 lat (usec): min=49, max=23936, avg=4565.65, stdev=4449.41 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 86], 10.00th=[ 97], 20.00th=[ 114], | 30.00th=[ 137], 40.00th=[ 3326], 50.00th=[ 4293], 60.00th=[ 5407], | 70.00th=[ 6456], 80.00th=[ 8029], 90.00th=[10159], 95.00th=[12256], | 99.00th=[19268], 99.50th=[22676], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 448, max= 2432, per=24.09%, avg=1480.00, stdev=584.69, samples=7 iops : min= 28, max= 152, avg=92.29, stdev=36.48, samples=7 write: IOPS=102, BW=1634KiB/s (1673kB/s)(6368KiB/3898msec) clat (usec): min=644, max=69714, avg=5514.73, stdev=6904.41 lat (usec): min=645, max=69716, avg=5516.80, stdev=6904.32 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 947], 10.00th=[ 1004], 20.00th=[ 1139], | 30.00th=[ 1336], 40.00th=[ 1631], 50.00th=[ 2606], 60.00th=[ 4883], | 70.00th=[ 6325], 80.00th=[ 8717], 90.00th=[12911], 95.00th=[17171], | 99.00th=[37487], 99.50th=[39060], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 512, max= 2464, per=24.88%, avg=1608.00, stdev=585.86, samples=7 iops : min= 32, max= 154, avg=100.29, stdev=36.55, samples=7 lat (usec) : 50=0.13%, 100=5.99%, 250=10.94%, 500=0.78%, 750=0.39% lat (usec) : 1000=4.69% lat (msec) : 2=18.75%, 4=9.37%, 10=34.90%, 20=11.72%, 50=2.21% lat (msec) : 100=0.13% cpu : usr=0.31%, sys=23.02%, ctx=1308, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1047: Thu May 1 22:13:51 2025 read: IOPS=93, BW=1488KiB/s (1524kB/s)(5744KiB/3859msec) clat (usec): min=68, max=32924, avg=4444.78, stdev=4562.00 lat (usec): min=68, max=32925, avg=4445.84, stdev=4562.23 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 90], 10.00th=[ 98], 20.00th=[ 109], | 30.00th=[ 130], 40.00th=[ 3163], 50.00th=[ 4178], 60.00th=[ 5014], | 70.00th=[ 6325], 80.00th=[ 7635], 90.00th=[10159], 95.00th=[11731], | 99.00th=[19530], 99.50th=[30278], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 320, max= 2016, per=23.80%, avg=1462.29, stdev=562.83, samples=7 iops : min= 20, max= 126, avg=91.29, stdev=35.08, samples=7 write: IOPS=105, BW=1696KiB/s (1736kB/s)(6544KiB/3859msec) clat (usec): min=655, max=63818, avg=5505.98, stdev=7951.64 lat (usec): min=656, max=63821, avg=5508.06, stdev=7951.96 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 938], 10.00th=[ 1020], 20.00th=[ 1139], | 30.00th=[ 1270], 40.00th=[ 1467], 50.00th=[ 1942], 60.00th=[ 4047], | 70.00th=[ 6128], 80.00th=[ 8848], 90.00th=[12256], 95.00th=[16188], | 99.00th=[43254], 99.50th=[50594], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 544, max= 2176, per=25.59%, avg=1654.29, stdev=568.49, samples=7 iops : min= 34, max= 136, avg=103.29, stdev=35.54, samples=7 lat (usec) : 100=5.47%, 250=10.94%, 500=0.91%, 750=0.26%, 1000=4.56% lat (msec) : 2=22.53%, 4=8.98%, 10=33.07%, 20=10.55%, 50=2.34% lat (msec) : 100=0.39% cpu : usr=0.10%, sys=22.24%, ctx=1306, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1048: Thu May 1 22:13:51 2025 read: IOPS=108, BW=1740KiB/s (1782kB/s)(6256KiB/3595msec) clat (usec): min=48, max=36352, avg=4557.98, stdev=4314.49 lat (usec): min=48, max=36353, avg=4558.86, stdev=4314.53 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 88], 10.00th=[ 100], 20.00th=[ 116], | 30.00th=[ 155], 40.00th=[ 3752], 50.00th=[ 4555], 60.00th=[ 5211], | 70.00th=[ 6390], 80.00th=[ 8029], 90.00th=[ 9503], 95.00th=[11207], | 99.00th=[17957], 99.50th=[20841], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 958, max= 2080, per=28.41%, avg=1745.43, stdev=396.24, samples=7 iops : min= 59, max= 130, avg=108.86, stdev=25.02, samples=7 write: IOPS=104, BW=1678KiB/s (1718kB/s)(6032KiB/3595msec) clat (usec): min=543, max=55802, avg=4780.55, stdev=5635.11 lat (usec): min=544, max=55803, avg=4782.92, stdev=5634.97 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 938], 10.00th=[ 1029], 20.00th=[ 1156], | 30.00th=[ 1287], 40.00th=[ 1450], 50.00th=[ 2040], 60.00th=[ 3884], | 70.00th=[ 5800], 80.00th=[ 8356], 90.00th=[11863], 95.00th=[15401], | 99.00th=[22414], 99.50th=[36963], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 1248, max= 2304, per=25.58%, avg=1653.71, stdev=393.18, samples=7 iops : min= 78, max= 144, avg=103.14, stdev=24.59, samples=7 lat (usec) : 50=0.13%, 100=5.60%, 250=11.07%, 500=0.65%, 750=0.52% lat (usec) : 1000=3.91% lat (msec) : 2=19.92%, 4=9.90%, 10=37.11%, 20=10.03%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.25%, sys=24.90%, ctx=1253, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1049: Thu May 1 22:13:51 2025 read: IOPS=98, BW=1579KiB/s (1617kB/s)(6032KiB/3821msec) clat (usec): min=54, max=53666, avg=4404.03, stdev=5450.65 lat (usec): min=54, max=53667, avg=4404.89, stdev=5450.63 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 79], 10.00th=[ 92], 20.00th=[ 105], | 30.00th=[ 118], 40.00th=[ 172], 50.00th=[ 3949], 60.00th=[ 4883], | 70.00th=[ 6063], 80.00th=[ 7832], 90.00th=[10290], 95.00th=[12387], | 99.00th=[26608], 99.50th=[35390], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 415, max= 1984, per=25.14%, avg=1544.43, stdev=529.61, samples=7 iops : min= 25, max= 124, avg=96.29, stdev=33.37, samples=7 write: IOPS=102, BW=1637KiB/s (1677kB/s)(6256KiB/3821msec) clat (usec): min=573, max=69554, avg=5497.74, stdev=8065.54 lat (usec): min=573, max=69555, avg=5499.12, stdev=8065.60 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 938], 10.00th=[ 1020], 20.00th=[ 1139], | 30.00th=[ 1303], 40.00th=[ 1631], 50.00th=[ 2376], 60.00th=[ 4490], | 70.00th=[ 6063], 80.00th=[ 8291], 90.00th=[11994], 95.00th=[15401], | 99.00th=[45876], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 479, max= 2528, per=25.10%, avg=1622.14, stdev=615.03, samples=7 iops : min= 29, max= 158, avg=101.14, stdev=38.75, samples=7 lat (usec) : 100=7.94%, 250=11.98%, 500=1.04%, 750=0.26%, 1000=4.04% lat (msec) : 2=19.79%, 4=9.11%, 10=33.33%, 20=10.42%, 50=1.56% lat (msec) : 100=0.52% cpu : usr=0.21%, sys=22.23%, ctx=1247, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6145KiB/s (6292kB/s), 1488KiB/s-1740KiB/s (1524kB/s-1782kB/s), io=23.4MiB (24.5MB), run=3595-3898msec WRITE: bw=6465KiB/s (6620kB/s), 1634KiB/s-1696KiB/s (1673kB/s-1736kB/s), io=24.6MiB (25.8MB), run=3595-3898msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1041: Thu May 1 22:14:09 2025 read: IOPS=16, BW=268KiB/s (274kB/s)(5920KiB/22085msec) clat (usec): min=2506, max=92775, avg=7793.21, stdev=8204.96 lat (usec): min=2507, max=92776, avg=7795.43, stdev=8204.58 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3425], 10.00th=[ 3720], 20.00th=[ 4178], | 30.00th=[ 4621], 40.00th=[ 5080], 50.00th=[ 5604], 60.00th=[ 6259], | 70.00th=[ 7308], 80.00th=[ 8586], 90.00th=[12387], 95.00th=[19792], | 99.00th=[45876], 99.50th=[67634], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 63, max= 704, per=24.44%, avg=261.74, stdev=139.05, samples=43 iops : min= 3, max= 44, avg=16.19, stdev= 8.75, samples=43 write: IOPS=18, BW=288KiB/s (295kB/s)(6368KiB/22085msec) clat (msec): min=16, max=149, avg=48.13, stdev=18.20 lat (msec): min=16, max=149, avg=48.13, stdev=18.20 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 25], 10.00th=[ 29], 20.00th=[ 34], | 30.00th=[ 37], 40.00th=[ 41], 50.00th=[ 45], 60.00th=[ 50], | 70.00th=[ 55], 80.00th=[ 61], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 107], 99.50th=[ 121], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 160, max= 480, per=25.43%, avg=286.30, stdev=71.91, samples=43 iops : min= 10, max= 30, avg=17.72, stdev= 4.57, samples=43 lat (msec) : 4=7.42%, 10=33.98%, 20=5.21%, 50=33.33%, 100=19.53% lat (msec) : 250=0.52% cpu : usr=0.02%, sys=3.70%, ctx=868, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1042: Thu May 1 22:14:09 2025 read: IOPS=16, BW=257KiB/s (263kB/s)(5744KiB/22363msec) clat (usec): min=2504, max=44653, avg=7312.48, stdev=5565.39 lat (usec): min=2505, max=44654, avg=7314.14, stdev=5565.28 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3326], 10.00th=[ 3720], 20.00th=[ 4228], | 30.00th=[ 4686], 40.00th=[ 5211], 50.00th=[ 5800], 60.00th=[ 6390], | 70.00th=[ 7242], 80.00th=[ 8455], 90.00th=[11338], 95.00th=[16188], | 99.00th=[34341], 99.50th=[38011], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 32, max= 576, per=23.78%, avg=254.68, stdev=121.97, samples=44 iops : min= 2, max= 36, avg=15.61, stdev= 7.74, samples=44 write: IOPS=18, BW=293KiB/s (300kB/s)(6544KiB/22363msec) clat (msec): min=16, max=148, avg=48.10, stdev=17.87 lat (msec): min=16, max=148, avg=48.10, stdev=17.87 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 25], 10.00th=[ 29], 20.00th=[ 33], | 30.00th=[ 37], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 51], | 70.00th=[ 56], 80.00th=[ 63], 90.00th=[ 72], 95.00th=[ 83], | 99.00th=[ 99], 99.50th=[ 100], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 191, max= 448, per=25.62%, avg=288.52, stdev=70.15, samples=44 iops : min= 11, max= 28, avg=17.73, stdev= 4.48, samples=44 lat (msec) : 4=7.55%, 10=32.94%, 20=5.21%, 50=32.68%, 100=21.35% lat (msec) : 250=0.26% cpu : usr=0.12%, sys=3.57%, ctx=851, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1043: Thu May 1 22:14:09 2025 read: IOPS=18, BW=300KiB/s (307kB/s)(6256KiB/20864msec) clat (usec): min=2820, max=37142, avg=6766.96, stdev=3740.62 lat (usec): min=2821, max=37143, avg=6768.60, stdev=3740.51 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3556], 10.00th=[ 3851], 20.00th=[ 4359], | 30.00th=[ 4752], 40.00th=[ 5211], 50.00th=[ 5735], 60.00th=[ 6259], | 70.00th=[ 6980], 80.00th=[ 8586], 90.00th=[11207], 95.00th=[13304], | 99.00th=[21365], 99.50th=[33424], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 96, max= 736, per=28.39%, avg=304.02, stdev=137.30, samples=41 iops : min= 6, max= 46, avg=18.68, stdev= 8.68, samples=41 write: IOPS=18, BW=289KiB/s (296kB/s)(6032KiB/20864msec) clat (msec): min=16, max=157, avg=48.24, stdev=18.52 lat (msec): min=16, max=157, avg=48.25, stdev=18.51 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 25], 10.00th=[ 28], 20.00th=[ 33], | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 50], | 70.00th=[ 55], 80.00th=[ 62], 90.00th=[ 74], 95.00th=[ 86], | 99.00th=[ 102], 99.50th=[ 107], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 159, max= 480, per=25.75%, avg=290.00, stdev=77.28, samples=41 iops : min= 9, max= 30, avg=17.80, stdev= 4.93, samples=41 lat (msec) : 4=7.29%, 10=36.98%, 20=6.64%, 50=30.34%, 100=18.23% lat (msec) : 250=0.52% cpu : usr=0.09%, sys=3.95%, ctx=867, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1044: Thu May 1 22:14:09 2025 read: IOPS=17, BW=277KiB/s (284kB/s)(6032KiB/21741msec) clat (usec): min=2829, max=37088, avg=7352.33, stdev=4436.61 lat (usec): min=2830, max=37090, avg=7353.99, stdev=4436.54 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3589], 10.00th=[ 3851], 20.00th=[ 4424], | 30.00th=[ 5145], 40.00th=[ 5538], 50.00th=[ 5997], 60.00th=[ 6652], | 70.00th=[ 7701], 80.00th=[ 9241], 90.00th=[11994], 95.00th=[15008], | 99.00th=[27657], 99.50th=[31851], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 127, max= 608, per=25.76%, avg=275.91, stdev=107.74, samples=43 iops : min= 7, max= 38, avg=17.07, stdev= 6.75, samples=43 write: IOPS=17, BW=288KiB/s (295kB/s)(6256KiB/21741msec) clat (msec): min=17, max=153, avg=48.38, stdev=18.33 lat (msec): min=17, max=153, avg=48.38, stdev=18.33 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 23], 10.00th=[ 27], 20.00th=[ 33], | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 51], | 70.00th=[ 56], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 81], | 99.00th=[ 101], 99.50th=[ 124], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 160, max= 544, per=25.63%, avg=288.56, stdev=78.81, samples=43 iops : min= 10, max= 34, avg=17.86, stdev= 4.91, samples=43 lat (msec) : 4=6.12%, 10=35.42%, 20=6.77%, 50=30.73%, 100=20.44% lat (msec) : 250=0.52% cpu : usr=0.05%, sys=3.76%, ctx=871, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1071KiB/s (1097kB/s), 257KiB/s-300KiB/s (263kB/s-307kB/s), io=23.4MiB (24.5MB), run=20864-22363msec WRITE: bw=1127KiB/s (1154kB/s), 288KiB/s-293KiB/s (295kB/s-300kB/s), io=24.6MiB (25.8MB), run=20864-22363msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1076: Thu May 1 22:14:11 2025 read: IOPS=13, BW=13.3MiB/s (13.9MB/s)(7168KiB/528msec) clat (usec): min=413, max=73316, avg=29147.16, stdev=29445.79 lat (usec): min=446, max=73316, avg=29152.90, stdev=29440.59 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 469], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[33817], 60.00th=[41681], | 70.00th=[41681], 80.00th=[53740], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min=12263, max=12263, per=46.65%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9697KiB/s (9930kB/s)(5120KiB/528msec) clat (usec): min=21732, max=78916, avg=44715.03, stdev=24980.24 lat (usec): min=21798, max=78934, avg=44757.71, stdev=24960.35 clat percentiles (usec): | 1.00th=[21627], 5.00th=[21627], 10.00th=[21627], 20.00th=[21627], | 30.00th=[28705], 40.00th=[28705], 50.00th=[30802], 60.00th=[30802], | 70.00th=[63177], 80.00th=[63177], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 8175, max= 8175, per=22.21%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=41.67%, 100=33.33% cpu : usr=0.00%, sys=35.04%, ctx=352, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1078: Thu May 1 22:14:11 2025 read: IOPS=2, BW=2629KiB/s (2692kB/s)(2048KiB/779msec) clat (usec): min=2438, max=43377, avg=22907.77, stdev=28947.96 lat (usec): min=2440, max=43379, avg=22909.93, stdev=28948.55 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2442], 10.00th=[ 2442], 20.00th=[ 2442], | 30.00th=[ 2442], 40.00th=[ 2442], 50.00th=[ 2442], 60.00th=[43254], | 70.00th=[43254], 80.00th=[43254], 90.00th=[43254], 95.00th=[43254], | 99.00th=[43254], 99.50th=[43254], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 2043, max= 2043, per=7.77%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.8MiB/s (13.5MB/s)(10.0MiB/779msec) clat (msec): min=26, max=190, avg=64.25, stdev=48.22 lat (msec): min=26, max=190, avg=64.34, stdev=48.23 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 33], | 30.00th=[ 35], 40.00th=[ 50], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 93], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 6131, max= 6131, per=16.66%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 4=8.33%, 50=41.67%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=30.85%, ctx=421, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1080: Thu May 1 22:14:11 2025 read: IOPS=11, BW=11.0MiB/s (11.6MB/s)(6144KiB/544msec) clat (usec): min=456, max=58213, avg=10126.41, stdev=23557.98 lat (usec): min=458, max=58215, avg=10127.64, stdev=23558.24 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 457], 10.00th=[ 457], 20.00th=[ 482], | 30.00th=[ 482], 40.00th=[ 482], 50.00th=[ 482], 60.00th=[ 519], | 70.00th=[ 611], 80.00th=[ 611], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] write: IOPS=11, BW=11.0MiB/s (11.6MB/s)(6144KiB/544msec) clat (msec): min=24, max=109, avg=56.22, stdev=30.53 lat (msec): min=24, max=109, avg=56.27, stdev=30.53 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 62], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] lat (usec) : 500=25.00%, 750=16.67% lat (msec) : 50=25.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=30.76%, ctx=58, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1082: Thu May 1 22:14:11 2025 read: IOPS=10, BW=10.9MiB/s (11.4MB/s)(5120KiB/459msec) clat (usec): min=471, max=61334, avg=20686.43, stdev=28512.02 lat (usec): min=472, max=61334, avg=20687.61, stdev=28511.43 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 474], | 30.00th=[ 578], 40.00th=[ 578], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[40109], 80.00th=[40109], 90.00th=[61080], 95.00th=[61080], | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] write: IOPS=15, BW=15.2MiB/s (15.0MB/s)(7168KiB/459msec) clat (usec): min=22312, max=86026, avg=48248.25, stdev=21437.87 lat (usec): min=22376, max=86136, avg=48324.25, stdev=21439.60 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[32375], | 30.00th=[40109], 40.00th=[40109], 50.00th=[44827], 60.00th=[46400], | 70.00th=[46400], 80.00th=[66323], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=50.00%, 100=25.00% cpu : usr=0.00%, sys=43.89%, ctx=121, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.7MiB/s (26.9MB/s), 2629KiB/s-13.3MiB/s (2692kB/s-13.9MB/s), io=20.0MiB (20.0MB), run=459-779msec WRITE: bw=35.9MiB/s (37.7MB/s), 9697KiB/s-15.2MiB/s (9930kB/s-15.0MB/s), io=28.0MiB (29.4MB), run=459-779msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1075: Thu May 1 22:14:11 2025 read: IOPS=6, BW=7090KiB/s (7260kB/s)(7168KiB/1011msec) clat (msec): min=11, max=155, avg=77.87, stdev=57.35 lat (msec): min=11, max=155, avg=77.87, stdev=57.35 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 26], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 90], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 133], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 6144, max= 6144, per=32.67%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5064KiB/s (5186kB/s)(5120KiB/1011msec) clat (msec): min=44, max=110, avg=77.93, stdev=29.48 lat (msec): min=44, max=110, avg=78.01, stdev=29.48 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 83], 60.00th=[ 83], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 2048, max= 2048, per=7.78%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=41.67% cpu : usr=0.30%, sys=3.47%, ctx=30, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1077: Thu May 1 22:14:11 2025 read: IOPS=1, BW=1881KiB/s (1926kB/s)(2048KiB/1089msec) clat (usec): min=38956, max=44626, avg=41791.52, stdev=4009.96 lat (usec): min=38958, max=44628, avg=41793.68, stdev=4009.44 clat percentiles (usec): | 1.00th=[39060], 5.00th=[39060], 10.00th=[39060], 20.00th=[39060], | 30.00th=[39060], 40.00th=[39060], 50.00th=[39060], 60.00th=[44827], | 70.00th=[44827], 80.00th=[44827], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 1774, max= 2043, per=10.15%, avg=1908.50, stdev=190.21, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=9, BW=9403KiB/s (9629kB/s)(10.0MiB/1089msec) clat (msec): min=29, max=173, avg=87.79, stdev=50.80 lat (msec): min=29, max=173, avg=87.87, stdev=50.78 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 32], | 30.00th=[ 48], 40.00th=[ 61], 50.00th=[ 65], 60.00th=[ 91], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 159], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 6131, max=10648, per=31.87%, avg=8389.50, stdev=3194.00, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (msec) : 50=41.67%, 100=25.00%, 250=33.33% cpu : usr=0.00%, sys=3.76%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1079: Thu May 1 22:14:11 2025 read: IOPS=6, BW=6865KiB/s (7030kB/s)(6144KiB/895msec) clat (msec): min=13, max=217, avg=69.78, stdev=76.29 lat (msec): min=13, max=217, avg=69.78, stdev=76.29 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 70], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 4096, max= 4096, per=21.78%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6865KiB/s (7030kB/s)(6144KiB/895msec) clat (msec): min=52, max=107, avg=76.26, stdev=18.93 lat (msec): min=52, max=109, avg=76.71, stdev=19.58 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 72], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 8192, max= 8192, per=31.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=8.33%, 100=58.33%, 250=16.67% cpu : usr=0.56%, sys=7.16%, ctx=124, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1081: Thu May 1 22:14:11 2025 read: IOPS=5, BW=5753KiB/s (5891kB/s)(5120KiB/890msec) clat (usec): min=13410, max=89395, avg=50274.52, stdev=32878.97 lat (usec): min=13412, max=89397, avg=50277.05, stdev=32878.13 clat percentiles (usec): | 1.00th=[13435], 5.00th=[13435], 10.00th=[13435], 20.00th=[13435], | 30.00th=[19006], 40.00th=[19006], 50.00th=[61080], 60.00th=[61080], | 70.00th=[68682], 80.00th=[68682], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 2048, max= 2048, per=10.89%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8054KiB/s (8247kB/s)(7168KiB/890msec) clat (msec): min=58, max=116, avg=83.24, stdev=20.11 lat (msec): min=58, max=116, avg=83.32, stdev=20.12 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 62], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 83], 60.00th=[ 91], | 70.00th=[ 91], 80.00th=[ 95], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 8192, max= 8192, per=31.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 100=75.00%, 250=8.33% cpu : usr=0.22%, sys=4.50%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.4MiB/s (19.3MB/s), 1881KiB/s-7090KiB/s (1926kB/s-7260kB/s), io=20.0MiB (20.0MB), run=890-1089msec WRITE: bw=25.7MiB/s (26.0MB/s), 5064KiB/s-9403KiB/s (5186kB/s-9629kB/s), io=28.0MiB (29.4MB), run=890-1089msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1091: Thu May 1 22:14:13 2025 read: IOPS=6, BW=26.8MiB/s (28.1MB/s)(12.0MiB/448msec) clat (usec): min=1816, max=251257, avg=141622.13, stdev=127428.17 lat (usec): min=1817, max=251259, avg=141624.26, stdev=127428.48 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1811], 10.00th=[ 1811], 20.00th=[ 1811], | 30.00th=[ 1811], 40.00th=[170918], 50.00th=[170918], 60.00th=[170918], | 70.00th=[250610], 80.00th=[250610], 90.00th=[250610], 95.00th=[250610], | 99.00th=[250610], 99.50th=[250610], 99.90th=[250610], 99.95th=[250610], | 99.99th=[250610] lat (msec) : 2=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.22%, sys=34.68%, ctx=786, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1093: Thu May 1 22:14:13 2025 read: IOPS=1, BW=5001KiB/s (5121kB/s)(4096KiB/819msec) clat (nsec): min=257823k, max=257823k, avg=257823409.00, stdev= 0.00 lat (nsec): min=257826k, max=257826k, avg=257825935.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 257], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 8192, max= 8192, per=40.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.77MiB/s (10.2MB/s)(8192KiB/819msec) clat (msec): min=116, max=342, avg=229.53, stdev=159.15 lat (msec): min=117, max=342, avg=229.83, stdev=159.25 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 342], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8192, max= 8192, per=24.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=28.36%, ctx=386, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1094: Thu May 1 22:14:13 2025 write: IOPS=3, BW=12.1MiB/s (12.7MB/s)(12.0MiB/988msec) clat (msec): min=249, max=430, avg=312.77, stdev=101.93 lat (msec): min=250, max=430, avg=313.19, stdev=101.95 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 251], 10.00th=[ 251], 20.00th=[ 251], | 30.00th=[ 251], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 8192, max= 8192, per=24.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.20%, sys=29.48%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1095: Thu May 1 22:14:13 2025 write: IOPS=3, BW=15.2MiB/s (15.9MB/s)(12.0MiB/791msec) clat (msec): min=155, max=301, avg=232.86, stdev=73.50 lat (msec): min=155, max=301, avg=233.15, stdev=73.48 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=24.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=39.24%, ctx=55, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.5MiB/s (20.5MB/s), 5001KiB/s-26.8MiB/s (5121kB/s-28.1MB/s), io=16.0MiB (16.8MB), run=448-819msec WRITE: bw=32.4MiB/s (33.0MB/s), 9.77MiB/s-15.2MiB/s (10.2MB/s-15.9MB/s), io=32.0MiB (33.6MB), run=791-988msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1088: Thu May 1 22:14:13 2025 read: IOPS=2, BW=11.3MiB/s (11.9MB/s)(12.0MiB/1060msec) clat (msec): min=169, max=455, avg=330.46, stdev=146.27 lat (msec): min=169, max=455, avg=330.47, stdev=146.27 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 5557, max= 5557, per=35.95%, avg=5557.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.19%, sys=1.51%, ctx=16, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1089: Thu May 1 22:14:13 2025 read: IOPS=2, BW=11.4MiB/s (11.0MB/s)(4096KiB/350msec) clat (nsec): min=46418k, max=46418k, avg=46417766.00, stdev= 0.00 lat (nsec): min=46423k, max=46423k, avg=46422786.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[46400], 5.00th=[46400], 10.00th=[46400], 20.00th=[46400], | 30.00th=[46400], 40.00th=[46400], 50.00th=[46400], 60.00th=[46400], | 70.00th=[46400], 80.00th=[46400], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] write: IOPS=5, BW=22.9MiB/s (23.0MB/s)(8192KiB/350msec) clat (msec): min=135, max=144, avg=140.07, stdev= 6.35 lat (msec): min=135, max=145, avg=140.71, stdev= 6.80 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.00%, sys=7.16%, ctx=17, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1090: Thu May 1 22:14:13 2025 write: IOPS=3, BW=13.7MiB/s (14.4MB/s)(12.0MiB/874msec) clat (msec): min=235, max=357, avg=280.29, stdev=67.03 lat (msec): min=236, max=357, avg=280.69, stdev=66.99 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8175, max= 8175, per=21.81%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.34%, sys=11.00%, ctx=82, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1092: Thu May 1 22:14:13 2025 write: IOPS=4, BW=16.9MiB/s (17.7MB/s)(12.0MiB/710msec) clat (msec): min=79, max=484, avg=229.26, stdev=222.00 lat (msec): min=79, max=484, avg=229.74, stdev=222.04 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 125], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min=16384, max=16384, per=43.70%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.14%, sys=2.54%, ctx=12, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.1MiB/s (15.8MB/s), 11.3MiB/s-11.4MiB/s (11.9MB/s-11.0MB/s), io=16.0MiB (16.8MB), run=350-1060msec WRITE: bw=36.6MiB/s (38.4MB/s), 13.7MiB/s-22.9MiB/s (14.4MB/s-23.0MB/s), io=32.0MiB (33.6MB), run=350-874msec