== sanity test 398b: DIO and buffer IO race ============== 00:13:18 (1746159198) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.34316 s, 21.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22064: Fri May 2 00:14:16 2025 read: IOPS=29, BW=118KiB/s (120kB/s)(5992KiB/50962msec) clat (usec): min=77, max=145448, avg=19805.40, stdev=13296.98 lat (usec): min=78, max=145449, avg=19806.68, stdev=13297.03 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 127], 10.00th=[ 6652], 20.00th=[ 10159], | 30.00th=[ 12649], 40.00th=[ 15008], 50.00th=[ 17433], 60.00th=[ 20579], | 70.00th=[ 24249], 80.00th=[ 28705], 90.00th=[ 35914], 95.00th=[ 42206], | 99.00th=[ 62129], 99.50th=[ 71828], 99.90th=[116917], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 55, max= 184, per=25.20%, avg=116.68, stdev=28.07, samples=101 iops : min= 13, max= 46, avg=28.73, stdev= 7.06, samples=101 write: IOPS=30, BW=124KiB/s (127kB/s)(6296KiB/50962msec) clat (usec): min=598, max=106593, avg=13471.51, stdev=14860.34 lat (usec): min=599, max=106594, avg=13474.69, stdev=14859.11 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 775], 10.00th=[ 816], 20.00th=[ 914], | 30.00th=[ 1516], 40.00th=[ 3490], 50.00th=[ 8717], 60.00th=[ 13304], | 70.00th=[ 17957], 80.00th=[ 24511], 90.00th=[ 35390], 95.00th=[ 42730], | 99.00th=[ 60031], 99.50th=[ 68682], 99.90th=[ 88605], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 47, max= 216, per=25.33%, avg=122.61, stdev=34.72, samples=101 iops : min= 11, max= 54, avg=30.22, stdev= 8.73, samples=101 lat (usec) : 100=0.72%, 250=2.77%, 500=0.16%, 750=1.76%, 1000=10.35% lat (msec) : 2=5.18%, 4=4.13%, 10=11.43%, 20=29.82%, 50=30.99% lat (msec) : 100=2.57%, 250=0.13% cpu : usr=0.16%, sys=9.38%, ctx=4894, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22065: Fri May 2 00:14:16 2025 read: IOPS=28, BW=116KiB/s (118kB/s)(5940KiB/51344msec) clat (usec): min=84, max=148106, avg=19066.20, stdev=13007.93 lat (usec): min=85, max=148107, avg=19068.56, stdev=13008.48 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 126], 10.00th=[ 6783], 20.00th=[ 10159], | 30.00th=[ 12256], 40.00th=[ 14615], 50.00th=[ 16909], 60.00th=[ 19792], | 70.00th=[ 22938], 80.00th=[ 27395], 90.00th=[ 33817], 95.00th=[ 40109], | 99.00th=[ 62653], 99.50th=[ 72877], 99.90th=[137364], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 47, max= 168, per=24.74%, avg=114.57, stdev=30.07, samples=102 iops : min= 11, max= 42, avg=28.25, stdev= 7.56, samples=102 write: IOPS=30, BW=124KiB/s (127kB/s)(6348KiB/51344msec) clat (usec): min=592, max=117300, avg=14453.90, stdev=15433.87 lat (usec): min=593, max=117301, avg=14456.37, stdev=15433.68 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 766], 10.00th=[ 832], 20.00th=[ 988], | 30.00th=[ 2212], 40.00th=[ 4555], 50.00th=[ 10290], 60.00th=[ 14222], | 70.00th=[ 19268], 80.00th=[ 26608], 90.00th=[ 35390], 95.00th=[ 44303], | 99.00th=[ 62653], 99.50th=[ 68682], 99.90th=[105382], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 48, max= 246, per=25.21%, avg=122.00, stdev=37.33, samples=102 iops : min= 12, max= 61, avg=30.10, stdev= 9.35, samples=102 lat (usec) : 100=0.46%, 250=3.06%, 500=0.07%, 750=1.86%, 1000=8.85% lat (msec) : 2=4.49%, 4=4.92%, 10=10.97%, 20=31.77%, 50=30.66% lat (msec) : 100=2.70%, 250=0.20% cpu : usr=0.20%, sys=9.40%, ctx=4847, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22066: Fri May 2 00:14:16 2025 read: IOPS=29, BW=117KiB/s (120kB/s)(6028KiB/51587msec) clat (usec): min=62, max=135396, avg=19411.12, stdev=13244.45 lat (usec): min=62, max=135398, avg=19414.31, stdev=13244.22 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 129], 10.00th=[ 6521], 20.00th=[ 10028], | 30.00th=[ 12256], 40.00th=[ 14484], 50.00th=[ 16909], 60.00th=[ 20055], | 70.00th=[ 23462], 80.00th=[ 27657], 90.00th=[ 34341], 95.00th=[ 43254], | 99.00th=[ 67634], 99.50th=[ 82314], 99.90th=[ 90702], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 47, max= 191, per=25.03%, avg=115.88, stdev=31.79, samples=103 iops : min= 11, max= 47, avg=28.53, stdev= 7.94, samples=103 write: IOPS=30, BW=121KiB/s (124kB/s)(6260KiB/51587msec) clat (usec): min=578, max=177255, avg=14204.86, stdev=15754.24 lat (usec): min=578, max=177258, avg=14207.74, stdev=15754.21 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 775], 10.00th=[ 832], 20.00th=[ 979], | 30.00th=[ 2147], 40.00th=[ 4686], 50.00th=[ 10028], 60.00th=[ 13829], | 70.00th=[ 18482], 80.00th=[ 25035], 90.00th=[ 34866], 95.00th=[ 44827], | 99.00th=[ 62129], 99.50th=[ 74974], 99.90th=[114820], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 55, max= 247, per=24.87%, avg=120.38, stdev=37.18, samples=103 iops : min= 13, max= 61, avg=29.65, stdev= 9.36, samples=103 lat (usec) : 100=0.88%, 250=2.70%, 500=0.10%, 750=1.50%, 1000=8.92% lat (msec) : 2=4.46%, 4=4.95%, 10=11.78%, 20=30.92%, 50=30.70% lat (msec) : 100=2.99%, 250=0.10% cpu : usr=0.12%, sys=9.35%, ctx=4996, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22067: Fri May 2 00:14:16 2025 read: IOPS=29, BW=117KiB/s (120kB/s)(6052KiB/51845msec) clat (usec): min=73, max=144728, avg=19501.79, stdev=13722.82 lat (usec): min=73, max=144729, avg=19504.66, stdev=13721.24 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 127], 10.00th=[ 6587], 20.00th=[ 10028], | 30.00th=[ 12387], 40.00th=[ 14484], 50.00th=[ 17171], 60.00th=[ 20055], | 70.00th=[ 23725], 80.00th=[ 27657], 90.00th=[ 34866], 95.00th=[ 42730], | 99.00th=[ 65799], 99.50th=[ 81265], 99.90th=[122160], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 55, max= 183, per=24.95%, avg=115.51, stdev=31.36, samples=103 iops : min= 13, max= 45, avg=28.50, stdev= 7.84, samples=103 write: IOPS=30, BW=120KiB/s (123kB/s)(6236KiB/51845msec) clat (usec): min=466, max=180596, avg=14259.04, stdev=15990.58 lat (usec): min=467, max=180599, avg=14261.19, stdev=15991.08 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 766], 10.00th=[ 824], 20.00th=[ 971], | 30.00th=[ 2278], 40.00th=[ 4686], 50.00th=[ 9110], 60.00th=[ 13829], | 70.00th=[ 17957], 80.00th=[ 25822], 90.00th=[ 35914], 95.00th=[ 44303], | 99.00th=[ 63701], 99.50th=[ 78119], 99.90th=[112722], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 15, max= 224, per=24.65%, avg=119.30, stdev=36.99, samples=103 iops : min= 3, max= 56, avg=29.45, stdev= 9.29, samples=103 lat (usec) : 100=0.91%, 250=2.70%, 500=0.07%, 750=1.95%, 1000=8.53% lat (msec) : 2=4.26%, 4=5.21%, 10=11.98%, 20=31.09%, 50=30.27% lat (msec) : 100=2.77%, 250=0.26% cpu : usr=0.14%, sys=9.53%, ctx=5074, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=463KiB/s (474kB/s), 116KiB/s-118KiB/s (118kB/s-120kB/s), io=23.4MiB (24.6MB), run=50962-51845msec WRITE: bw=485KiB/s (497kB/s), 120KiB/s-124KiB/s (123kB/s-127kB/s), io=24.6MiB (25.7MB), run=50962-51845msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22059: Fri May 2 00:14:36 2025 read: IOPS=20, BW=82.4KiB/s (84.4kB/s)(5992KiB/72715msec) clat (msec): min=3, max=339, avg=18.05, stdev=18.28 lat (msec): min=3, max=339, avg=18.05, stdev=18.28 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 26], 90.00th=[ 40], 95.00th=[ 54], | 99.00th=[ 82], 99.50th=[ 93], 99.90th=[ 125], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 15, max= 256, per=24.83%, avg=81.70, stdev=52.48, samples=145 iops : min= 3, max= 64, avg=20.03, stdev=13.18, samples=145 write: IOPS=21, BW=86.6KiB/s (88.7kB/s)(6296KiB/72715msec) clat (msec): min=3, max=152, avg=28.89, stdev=19.95 lat (msec): min=3, max=152, avg=28.90, stdev=19.95 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 27], | 70.00th=[ 34], 80.00th=[ 43], 90.00th=[ 56], 95.00th=[ 70], | 99.00th=[ 103], 99.50th=[ 112], 99.90th=[ 142], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 22, max= 288, per=25.13%, avg=86.43, stdev=52.56, samples=144 iops : min= 5, max= 72, avg=21.22, stdev=13.18, samples=144 lat (msec) : 4=0.62%, 10=22.04%, 20=35.38%, 50=32.00%, 100=9.34% lat (msec) : 250=0.59%, 500=0.03% cpu : usr=0.11%, sys=4.49%, ctx=3418, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22060: Fri May 2 00:14:36 2025 read: IOPS=20, BW=81.6KiB/s (83.6kB/s)(5940KiB/72763msec) clat (msec): min=2, max=339, avg=17.51, stdev=17.64 lat (msec): min=2, max=339, avg=17.51, stdev=17.64 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 40], 95.00th=[ 53], | 99.00th=[ 81], 99.50th=[ 91], 99.90th=[ 115], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 7, max= 288, per=24.59%, avg=80.90, stdev=53.69, samples=145 iops : min= 1, max= 72, avg=19.83, stdev=13.49, samples=145 write: IOPS=21, BW=87.2KiB/s (89.3kB/s)(6348KiB/72763msec) clat (msec): min=5, max=189, avg=29.33, stdev=20.01 lat (msec): min=5, max=189, avg=29.34, stdev=20.01 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 28], | 70.00th=[ 35], 80.00th=[ 43], 90.00th=[ 57], 95.00th=[ 68], | 99.00th=[ 95], 99.50th=[ 110], 99.90th=[ 165], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 24, max= 288, per=25.28%, avg=86.98, stdev=51.26, samples=144 iops : min= 6, max= 72, avg=21.35, stdev=12.86, samples=144 lat (msec) : 4=0.46%, 10=22.20%, 20=35.55%, 50=32.23%, 100=8.98% lat (msec) : 250=0.55%, 500=0.03% cpu : usr=0.12%, sys=4.49%, ctx=3447, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22061: Fri May 2 00:14:36 2025 read: IOPS=20, BW=82.6KiB/s (84.6kB/s)(6028KiB/72936msec) clat (msec): min=2, max=114, avg=17.50, stdev=15.91 lat (msec): min=2, max=114, avg=17.50, stdev=15.91 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 25], 90.00th=[ 39], 95.00th=[ 53], | 99.00th=[ 82], 99.50th=[ 87], 99.90th=[ 97], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 7, max= 256, per=25.04%, avg=82.37, stdev=53.31, samples=144 iops : min= 1, max= 64, avg=20.19, stdev=13.41, samples=144 write: IOPS=21, BW=85.8KiB/s (87.9kB/s)(6260KiB/72936msec) clat (msec): min=5, max=357, avg=29.62, stdev=22.60 lat (msec): min=5, max=357, avg=29.62, stdev=22.60 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 34], 80.00th=[ 43], 90.00th=[ 58], 95.00th=[ 71], | 99.00th=[ 107], 99.50th=[ 128], 99.90th=[ 188], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 7, max= 248, per=24.62%, avg=84.69, stdev=52.16, samples=145 iops : min= 1, max= 62, avg=20.77, stdev=13.13, samples=145 lat (msec) : 4=0.72%, 10=22.36%, 20=35.77%, 50=30.99%, 100=9.44% lat (msec) : 250=0.68%, 500=0.03% cpu : usr=0.11%, sys=4.44%, ctx=3426, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22062: Fri May 2 00:14:36 2025 read: IOPS=20, BW=83.6KiB/s (85.6kB/s)(6052KiB/72393msec) clat (msec): min=2, max=152, avg=17.76, stdev=16.55 lat (msec): min=2, max=152, avg=17.77, stdev=16.55 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 25], 90.00th=[ 40], 95.00th=[ 53], | 99.00th=[ 84], 99.50th=[ 99], 99.90th=[ 130], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 7, max= 296, per=25.29%, avg=83.20, stdev=52.22, samples=143 iops : min= 1, max= 74, avg=20.39, stdev=13.05, samples=143 write: IOPS=21, BW=86.1KiB/s (88.2kB/s)(6236KiB/72393msec) clat (msec): min=5, max=356, avg=29.06, stdev=20.33 lat (msec): min=5, max=356, avg=29.06, stdev=20.32 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 24], 60.00th=[ 28], | 70.00th=[ 35], 80.00th=[ 43], 90.00th=[ 56], 95.00th=[ 67], | 99.00th=[ 90], 99.50th=[ 110], 99.90th=[ 125], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8, max= 279, per=24.79%, avg=85.28, stdev=50.55, samples=144 iops : min= 2, max= 69, avg=20.92, stdev=12.63, samples=144 lat (msec) : 4=0.62%, 10=22.04%, 20=35.16%, 50=32.98%, 100=8.72% lat (msec) : 250=0.46%, 500=0.03% cpu : usr=0.11%, sys=4.49%, ctx=3433, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=329KiB/s (337kB/s), 81.6KiB/s-83.6KiB/s (83.6kB/s-85.6kB/s), io=23.4MiB (24.6MB), run=72393-72936msec WRITE: bw=345KiB/s (353kB/s), 85.8KiB/s-87.2KiB/s (87.9kB/s-89.3kB/s), io=24.6MiB (25.7MB), run=72393-72936msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22141: Fri May 2 00:14:53 2025 read: IOPS=26, BW=424KiB/s (434kB/s)(5920KiB/13953msec) clat (usec): min=68, max=143253, avg=20258.20, stdev=16758.10 lat (usec): min=68, max=143255, avg=20262.62, stdev=16759.35 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 106], 10.00th=[ 126], 20.00th=[ 8291], | 30.00th=[ 12780], 40.00th=[ 15008], 50.00th=[ 18482], 60.00th=[ 21627], | 70.00th=[ 25035], 80.00th=[ 30278], 90.00th=[ 36439], 95.00th=[ 42730], | 99.00th=[ 77071], 99.50th=[122160], 99.90th=[143655], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 191, max= 787, per=26.62%, avg=418.22, stdev=125.59, samples=27 iops : min= 11, max= 49, avg=25.52, stdev= 7.95, samples=27 write: IOPS=28, BW=456KiB/s (467kB/s)(6368KiB/13953msec) clat (usec): min=933, max=107499, avg=15916.00, stdev=18006.56 lat (usec): min=935, max=107601, avg=15921.29, stdev=18009.70 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1156], 10.00th=[ 1205], 20.00th=[ 1385], | 30.00th=[ 3032], 40.00th=[ 5211], 50.00th=[ 9765], 60.00th=[ 13960], | 70.00th=[ 19530], 80.00th=[ 28181], 90.00th=[ 41157], 95.00th=[ 55837], | 99.00th=[ 76022], 99.50th=[103285], 99.90th=[107480], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 254, max= 736, per=27.53%, avg=455.04, stdev=143.57, samples=27 iops : min= 15, max= 46, avg=27.81, stdev= 9.11, samples=27 lat (usec) : 100=1.82%, 250=6.12%, 500=0.13%, 1000=0.91% lat (msec) : 2=11.33%, 4=6.51%, 10=9.90%, 20=25.65%, 50=33.07% lat (msec) : 100=4.04%, 250=0.52% cpu : usr=0.17%, sys=9.03%, ctx=1200, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22142: Fri May 2 00:14:53 2025 read: IOPS=25, BW=408KiB/s (418kB/s)(5744KiB/14074msec) clat (usec): min=84, max=89779, avg=20041.76, stdev=15200.09 lat (usec): min=84, max=89780, avg=20048.95, stdev=15215.22 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 110], 10.00th=[ 124], 20.00th=[ 7701], | 30.00th=[12125], 40.00th=[14877], 50.00th=[18220], 60.00th=[21365], | 70.00th=[26346], 80.00th=[31851], 90.00th=[40109], 95.00th=[45351], | 99.00th=[67634], 99.50th=[88605], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 253, max= 637, per=25.65%, avg=402.89, stdev=112.30, samples=28 iops : min= 15, max= 39, avg=24.68, stdev= 7.11, samples=28 write: IOPS=29, BW=465KiB/s (476kB/s)(6544KiB/14074msec) clat (usec): min=876, max=137278, avg=16645.05, stdev=18273.31 lat (usec): min=877, max=137281, avg=16647.17, stdev=18273.32 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1139], 10.00th=[ 1221], 20.00th=[ 1467], | 30.00th=[ 3458], 40.00th=[ 5997], 50.00th=[ 10290], 60.00th=[ 15139], | 70.00th=[ 20841], 80.00th=[ 31065], 90.00th=[ 42730], 95.00th=[ 53740], | 99.00th=[ 71828], 99.50th=[ 77071], 99.90th=[137364], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 222, max= 664, per=27.82%, avg=459.89, stdev=129.38, samples=28 iops : min= 13, max= 41, avg=28.25, stdev= 8.19, samples=28 lat (usec) : 100=0.91%, 250=6.12%, 500=0.26%, 750=0.26%, 1000=0.65% lat (msec) : 2=11.85%, 4=5.34%, 10=12.63%, 20=25.00%, 50=32.29% lat (msec) : 100=4.56%, 250=0.13% cpu : usr=0.16%, sys=8.92%, ctx=1212, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22143: Fri May 2 00:14:53 2025 read: IOPS=25, BW=411KiB/s (420kB/s)(6256KiB/15237msec) clat (usec): min=85, max=147066, avg=22416.67, stdev=18535.98 lat (usec): min=85, max=147068, avg=22418.47, stdev=18536.20 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 120], 10.00th=[ 143], 20.00th=[ 9765], | 30.00th=[ 13042], 40.00th=[ 16188], 50.00th=[ 19006], 60.00th=[ 22414], | 70.00th=[ 26346], 80.00th=[ 32637], 90.00th=[ 43254], 95.00th=[ 60031], | 99.00th=[ 84411], 99.50th=[110625], 99.90th=[147850], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 159, max= 640, per=25.33%, avg=397.97, stdev=115.22, samples=30 iops : min= 9, max= 40, avg=24.37, stdev= 7.20, samples=30 write: IOPS=24, BW=396KiB/s (405kB/s)(6032KiB/15237msec) clat (usec): min=948, max=172343, avg=16804.90, stdev=20809.55 lat (usec): min=950, max=172344, avg=16806.88, stdev=20809.52 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1106], 10.00th=[ 1188], 20.00th=[ 1336], | 30.00th=[ 2442], 40.00th=[ 5014], 50.00th=[ 10552], 60.00th=[ 14746], | 70.00th=[ 20841], 80.00th=[ 28967], 90.00th=[ 41157], 95.00th=[ 55313], | 99.00th=[ 93848], 99.50th=[119014], 99.90th=[173016], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 160, max= 736, per=23.06%, avg=381.17, stdev=142.63, samples=30 iops : min= 10, max= 46, avg=23.30, stdev= 8.89, samples=30 lat (usec) : 100=0.65%, 250=5.47%, 500=0.26%, 750=0.13%, 1000=1.04% lat (msec) : 2=12.89%, 4=4.95%, 10=9.37%, 20=25.91%, 50=32.16% lat (msec) : 100=6.51%, 250=0.65% cpu : usr=0.07%, sys=8.94%, ctx=1314, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22144: Fri May 2 00:14:53 2025 read: IOPS=27, BW=435KiB/s (445kB/s)(6032KiB/13875msec) clat (usec): min=86, max=129678, avg=21525.00, stdev=18190.05 lat (usec): min=86, max=129679, avg=21526.34, stdev=18190.06 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 116], 10.00th=[ 126], 20.00th=[ 8094], | 30.00th=[ 12125], 40.00th=[ 15664], 50.00th=[ 20055], 60.00th=[ 22414], | 70.00th=[ 26346], 80.00th=[ 31589], 90.00th=[ 41157], 95.00th=[ 56361], | 99.00th=[ 95945], 99.50th=[119014], 99.90th=[129500], 99.95th=[129500], | 99.99th=[129500] bw ( KiB/s): min= 192, max= 724, per=27.07%, avg=425.26, stdev=126.48, samples=27 iops : min= 12, max= 45, avg=25.96, stdev= 7.94, samples=27 write: IOPS=28, BW=451KiB/s (462kB/s)(6256KiB/13875msec) clat (usec): min=885, max=178770, avg=14277.06, stdev=18250.47 lat (usec): min=887, max=178772, avg=14279.26, stdev=18250.58 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1106], 10.00th=[ 1188], 20.00th=[ 1352], | 30.00th=[ 2606], 40.00th=[ 5407], 50.00th=[ 8455], 60.00th=[ 12518], | 70.00th=[ 16450], 80.00th=[ 22938], 90.00th=[ 35914], 95.00th=[ 49546], | 99.00th=[ 88605], 99.50th=[102237], 99.90th=[179307], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 191, max= 704, per=27.09%, avg=447.81, stdev=144.96, samples=27 iops : min= 11, max= 44, avg=27.37, stdev= 9.20, samples=27 lat (usec) : 100=0.52%, 250=7.29%, 750=0.13%, 1000=0.39% lat (msec) : 2=13.54%, 4=5.08%, 10=12.24%, 20=24.48%, 50=30.99% lat (msec) : 100=4.69%, 250=0.65% cpu : usr=0.12%, sys=9.07%, ctx=1265, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1572KiB/s (1610kB/s), 408KiB/s-435KiB/s (418kB/s-445kB/s), io=23.4MiB (24.5MB), run=13875-15237msec WRITE: bw=1654KiB/s (1694kB/s), 396KiB/s-465KiB/s (405kB/s-476kB/s), io=24.6MiB (25.8MB), run=13875-15237msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22146: Fri May 2 00:15:00 2025 read: IOPS=17, BW=272KiB/s (279kB/s)(5920KiB/21749msec) clat (msec): min=4, max=121, avg=21.10, stdev=20.55 lat (msec): min=4, max=121, avg=21.10, stdev=20.55 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 27], 90.00th=[ 48], 95.00th=[ 69], | 99.00th=[ 107], 99.50th=[ 117], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 63, max= 672, per=24.61%, avg=268.49, stdev=151.11, samples=43 iops : min= 3, max= 42, avg=16.21, stdev= 9.52, samples=43 write: IOPS=18, BW=293KiB/s (300kB/s)(6368KiB/21749msec) clat (msec): min=8, max=127, avg=34.39, stdev=23.01 lat (msec): min=8, max=127, avg=34.40, stdev=23.01 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 31], | 70.00th=[ 38], 80.00th=[ 50], 90.00th=[ 67], 95.00th=[ 85], | 99.00th=[ 118], 99.50th=[ 125], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 31, max= 638, per=25.33%, avg=290.74, stdev=164.25, samples=43 iops : min= 1, max= 39, avg=17.60, stdev=10.31, samples=43 lat (msec) : 10=13.80%, 20=36.72%, 50=34.90%, 100=12.24%, 250=2.34% cpu : usr=0.11%, sys=3.99%, ctx=878, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22147: Fri May 2 00:15:00 2025 read: IOPS=16, BW=265KiB/s (272kB/s)(5744KiB/21656msec) clat (msec): min=4, max=144, avg=21.52, stdev=20.18 lat (msec): min=4, max=144, avg=21.52, stdev=20.18 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 29], 90.00th=[ 50], 95.00th=[ 66], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 31, max= 632, per=24.11%, avg=263.07, stdev=161.33, samples=43 iops : min= 1, max= 39, avg=15.93, stdev=10.16, samples=43 write: IOPS=18, BW=302KiB/s (309kB/s)(6544KiB/21656msec) clat (msec): min=8, max=220, avg=33.79, stdev=23.34 lat (msec): min=8, max=220, avg=33.79, stdev=23.34 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 27], 60.00th=[ 31], | 70.00th=[ 38], 80.00th=[ 48], 90.00th=[ 65], 95.00th=[ 80], | 99.00th=[ 104], 99.50th=[ 144], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 93, max= 704, per=26.09%, avg=299.51, stdev=163.26, samples=43 iops : min= 5, max= 44, avg=18.21, stdev=10.25, samples=43 lat (msec) : 10=14.06%, 20=33.98%, 50=38.15%, 100=12.63%, 250=1.17% cpu : usr=0.03%, sys=4.06%, ctx=869, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22148: Fri May 2 00:15:00 2025 read: IOPS=17, BW=285KiB/s (292kB/s)(6256KiB/21935msec) clat (msec): min=4, max=133, avg=21.24, stdev=19.56 lat (msec): min=4, max=133, avg=21.25, stdev=19.56 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 27], 90.00th=[ 49], 95.00th=[ 62], | 99.00th=[ 95], 99.50th=[ 114], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 63, max= 701, per=25.96%, avg=283.21, stdev=173.40, samples=43 iops : min= 3, max= 43, avg=17.21, stdev=10.78, samples=43 write: IOPS=17, BW=275KiB/s (282kB/s)(6032KiB/21935msec) clat (msec): min=6, max=254, avg=35.53, stdev=25.60 lat (msec): min=6, max=254, avg=35.53, stdev=25.60 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 28], 60.00th=[ 32], | 70.00th=[ 39], 80.00th=[ 50], 90.00th=[ 72], 95.00th=[ 90], | 99.00th=[ 117], 99.50th=[ 127], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 63, max= 638, per=23.45%, avg=269.19, stdev=151.21, samples=43 iops : min= 3, max= 39, avg=16.33, stdev= 9.37, samples=43 lat (msec) : 10=13.28%, 20=35.29%, 50=37.37%, 100=11.98%, 250=1.95% lat (msec) : 500=0.13% cpu : usr=0.07%, sys=3.80%, ctx=869, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22149: Fri May 2 00:15:00 2025 read: IOPS=17, BW=276KiB/s (282kB/s)(6032KiB/21894msec) clat (msec): min=4, max=134, avg=21.10, stdev=19.66 lat (msec): min=4, max=134, avg=21.11, stdev=19.66 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 31], 90.00th=[ 45], 95.00th=[ 65], | 99.00th=[ 96], 99.50th=[ 128], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 31, max= 733, per=25.13%, avg=274.12, stdev=157.84, samples=42 iops : min= 1, max= 45, avg=16.67, stdev= 9.81, samples=42 write: IOPS=17, BW=286KiB/s (293kB/s)(6256KiB/21894msec) clat (msec): min=6, max=254, avg=35.15, stdev=25.38 lat (msec): min=6, max=254, avg=35.15, stdev=25.38 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 31], | 70.00th=[ 39], 80.00th=[ 51], 90.00th=[ 70], 95.00th=[ 81], | 99.00th=[ 121], 99.50th=[ 128], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 63, max= 640, per=24.55%, avg=281.79, stdev=163.70, samples=43 iops : min= 3, max= 40, avg=17.14, stdev=10.18, samples=43 lat (msec) : 10=16.02%, 20=30.73%, 50=38.67%, 100=12.76%, 250=1.69% lat (msec) : 500=0.13% cpu : usr=0.07%, sys=3.86%, ctx=874, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1092KiB/s (1118kB/s), 265KiB/s-285KiB/s (272kB/s-292kB/s), io=23.4MiB (24.5MB), run=21656-21935msec WRITE: bw=1149KiB/s (1176kB/s), 275KiB/s-302KiB/s (282kB/s-309kB/s), io=24.6MiB (25.8MB), run=21656-21935msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22178: Fri May 2 00:15:04 2025 read: IOPS=4, BW=4187KiB/s (4287kB/s)(7168KiB/1712msec) clat (usec): min=516, max=311094, avg=126702.82, stdev=96783.84 lat (usec): min=517, max=311096, avg=126704.59, stdev=96783.76 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 519], 10.00th=[ 519], 20.00th=[ 86508], | 30.00th=[ 93848], 40.00th=[ 93848], 50.00th=[ 96994], 60.00th=[120062], | 70.00th=[120062], 80.00th=[177210], 90.00th=[312476], 95.00th=[312476], | 99.00th=[312476], 99.50th=[312476], 99.90th=[312476], 99.95th=[312476], | 99.99th=[312476] bw ( KiB/s): min= 4096, max= 6144, per=45.40%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2991KiB/s (3062kB/s)(5120KiB/1712msec) clat (msec): min=63, max=269, avg=139.96, stdev=84.70 lat (msec): min=63, max=269, avg=140.02, stdev=84.71 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 75], 40.00th=[ 75], 50.00th=[ 117], 60.00th=[ 117], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 2048, max= 5796, per=24.84%, avg=3922.00, stdev=2650.24, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 100=41.67%, 250=33.33%, 500=16.67% cpu : usr=0.06%, sys=14.96%, ctx=328, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22179: Fri May 2 00:15:04 2025 read: IOPS=1, BW=1143KiB/s (1171kB/s)(2048KiB/1791msec) clat (usec): min=1824, max=265665, avg=133745.01, stdev=186563.68 lat (usec): min=1825, max=265667, avg=133746.91, stdev=186564.51 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1827], 10.00th=[ 1827], 20.00th=[ 1827], | 30.00th=[ 1827], 40.00th=[ 1827], 50.00th=[ 1827], 60.00th=[265290], | 70.00th=[265290], 80.00th=[265290], 90.00th=[265290], 95.00th=[265290], | 99.00th=[265290], 99.50th=[265290], 99.90th=[265290], 99.95th=[265290], | 99.99th=[265290] bw ( KiB/s): min= 1961, max= 2007, per=17.59%, avg=1984.00, stdev=32.53, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5717KiB/s (5855kB/s)(10.0MiB/1791msec) clat (msec): min=50, max=265, avg=136.22, stdev=70.91 lat (msec): min=50, max=265, avg=136.30, stdev=70.92 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 66], | 30.00th=[ 69], 40.00th=[ 104], 50.00th=[ 121], 60.00th=[ 131], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 230], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 6131, max= 7846, per=44.26%, avg=6988.50, stdev=1212.69, samples=2 iops : min= 5, max= 7, avg= 6.00, stdev= 1.41, samples=2 lat (msec) : 2=8.33%, 100=25.00%, 250=50.00%, 500=16.67% cpu : usr=0.00%, sys=13.63%, ctx=111, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22180: Fri May 2 00:15:04 2025 read: IOPS=4, BW=4121KiB/s (4220kB/s)(6144KiB/1491msec) clat (usec): min=644, max=206624, avg=78419.60, stdev=69730.62 lat (usec): min=645, max=206626, avg=78421.30, stdev=69730.80 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 644], 10.00th=[ 644], 20.00th=[ 38536], | 30.00th=[ 38536], 40.00th=[ 64750], 50.00th=[ 64750], 60.00th=[ 76022], | 70.00th=[ 84411], 80.00th=[ 84411], 90.00th=[206570], 95.00th=[206570], | 99.00th=[206570], 99.50th=[206570], 99.90th=[206570], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 4096, max= 4096, per=36.32%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4121KiB/s (4220kB/s)(6144KiB/1491msec) clat (msec): min=87, max=282, avg=157.59, stdev=73.54 lat (msec): min=87, max=282, avg=157.67, stdev=73.56 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 144], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 2048, max= 4096, per=19.46%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=8.33%, 100=41.67%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=11.81%, ctx=245, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22181: Fri May 2 00:15:04 2025 read: IOPS=2, BW=2819KiB/s (2887kB/s)(5120KiB/1816msec) clat (usec): min=550, max=127840, avg=72850.83, stdev=66490.78 lat (usec): min=551, max=127842, avg=72852.55, stdev=66491.03 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 553], | 30.00th=[ 570], 40.00th=[ 570], 50.00th=[107480], 60.00th=[107480], | 70.00th=[127402], 80.00th=[127402], 90.00th=[127402], 95.00th=[127402], | 99.00th=[127402], 99.50th=[127402], 99.90th=[127402], 99.95th=[127402], | 99.99th=[127402] bw ( KiB/s): min= 4071, max= 4071, per=36.10%, avg=4071.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3947KiB/s (4042kB/s)(7168KiB/1816msec) clat (msec): min=46, max=289, avg=170.57, stdev=95.16 lat (msec): min=46, max=289, avg=170.65, stdev=95.18 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 70], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 153], 60.00th=[ 232], | 70.00th=[ 232], 80.00th=[ 268], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 2043, max= 4071, per=17.23%, avg=2720.67, stdev=1169.43, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (usec) : 750=16.67% lat (msec) : 50=8.33%, 100=8.33%, 250=50.00%, 500=16.67% cpu : usr=0.00%, sys=13.17%, ctx=212, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.0MiB/s (11.5MB/s), 1143KiB/s-4187KiB/s (1171kB/s-4287kB/s), io=20.0MiB (20.0MB), run=1491-1816msec WRITE: bw=15.4MiB/s (16.2MB/s), 2991KiB/s-5717KiB/s (3062kB/s-5855kB/s), io=28.0MiB (29.4MB), run=1491-1816msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22174: Fri May 2 00:15:05 2025 read: IOPS=4, BW=4441KiB/s (4548kB/s)(7168KiB/1614msec) clat (msec): min=34, max=138, avg=84.10, stdev=39.51 lat (msec): min=34, max=138, avg=84.10, stdev=39.51 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 55], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 78], 60.00th=[ 92], | 70.00th=[ 92], 80.00th=[ 133], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 2039, max= 6119, per=46.80%, avg=4052.67, stdev=2040.51, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 write: IOPS=3, BW=3172KiB/s (3248kB/s)(5120KiB/1614msec) clat (msec): min=87, max=335, avg=193.77, stdev=91.64 lat (msec): min=87, max=335, avg=193.86, stdev=91.64 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 4000, max= 4079, per=33.32%, avg=4039.50, stdev=55.86, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 lat (msec) : 50=8.33%, 100=41.67%, 250=41.67%, 500=8.33% cpu : usr=0.06%, sys=2.36%, ctx=27, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22175: Fri May 2 00:15:05 2025 read: IOPS=0, BW=866KiB/s (887kB/s)(2048KiB/2365msec) clat (msec): min=149, max=297, avg=223.51, stdev=105.23 lat (msec): min=149, max=297, avg=223.51, stdev=105.23 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 2023, max= 2048, per=23.51%, avg=2035.50, stdev=17.68, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4330KiB/s (4434kB/s)(10.0MiB/2365msec) clat (msec): min=23, max=353, avg=167.65, stdev=128.37 lat (msec): min=23, max=354, avg=167.73, stdev=128.38 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 32], | 30.00th=[ 43], 40.00th=[ 108], 50.00th=[ 111], 60.00th=[ 153], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 330], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 2023, max= 4096, per=28.04%, avg=3399.33, stdev=1191.97, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 lat (msec) : 50=25.00%, 250=41.67%, 500=33.33% cpu : usr=0.00%, sys=5.29%, ctx=102, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22176: Fri May 2 00:15:05 2025 read: IOPS=2, BW=2795KiB/s (2862kB/s)(6144KiB/2198msec) clat (msec): min=58, max=358, avg=198.34, stdev=108.07 lat (msec): min=58, max=358, avg=198.34, stdev=108.07 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 230], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 2031, max= 4096, per=31.47%, avg=2725.00, stdev=1187.35, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 write: IOPS=2, BW=2795KiB/s (2862kB/s)(6144KiB/2198msec) clat (msec): min=47, max=285, avg=163.44, stdev=93.32 lat (msec): min=47, max=285, avg=163.53, stdev=93.30 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 167], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 2031, max= 4096, per=21.08%, avg=2555.75, stdev=1026.86, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 50=8.33%, 100=8.33%, 250=50.00%, 500=33.33% cpu : usr=0.09%, sys=3.50%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22177: Fri May 2 00:15:05 2025 read: IOPS=2, BW=2254KiB/s (2308kB/s)(5120KiB/2272msec) clat (msec): min=43, max=222, avg=128.02, stdev=75.81 lat (msec): min=43, max=222, avg=128.02, stdev=75.81 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 4015, max= 4096, per=46.84%, avg=4055.50, stdev=57.28, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3155KiB/s (3231kB/s)(7168KiB/2272msec) clat (msec): min=61, max=283, avg=194.04, stdev=78.04 lat (msec): min=61, max=283, avg=194.13, stdev=78.05 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 132], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 232], | 70.00th=[ 232], 80.00th=[ 271], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 2007, max= 4096, per=22.38%, avg=2712.67, stdev=1198.08, samples=3 iops : min= 1, max= 4, avg= 2.00, stdev= 1.73, samples=3 lat (msec) : 50=8.33%, 100=16.67%, 250=58.33%, 500=16.67% cpu : usr=0.04%, sys=2.69%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8660KiB/s (8867kB/s), 866KiB/s-4441KiB/s (887kB/s-4548kB/s), io=20.0MiB (20.0MB), run=1614-2365msec WRITE: bw=11.8MiB/s (12.4MB/s), 2795KiB/s-4330KiB/s (2862kB/s-4434kB/s), io=28.0MiB (29.4MB), run=1614-2365msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22191: Fri May 2 00:15:08 2025 read: IOPS=4, BW=18.5MiB/s (19.4MB/s)(12.0MiB/648msec) clat (msec): min=6, max=300, avg=190.66, stdev=160.77 lat (msec): min=6, max=300, avg=190.66, stdev=160.77 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 266], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8126, max= 8126, per=65.47%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 10=33.33%, 500=66.67% cpu : usr=0.31%, sys=15.61%, ctx=177, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22194: Fri May 2 00:15:08 2025 read: IOPS=0, BW=3103KiB/s (3178kB/s)(4096KiB/1320msec) clat (nsec): min=306852k, max=306852k, avg=306851500.00, stdev= 0.00 lat (nsec): min=306854k, max=306854k, avg=306854059.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8175, max= 8175, per=65.86%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6206KiB/s (6355kB/s)(8192KiB/1320msec) clat (msec): min=318, max=629, avg=473.64, stdev=219.72 lat (msec): min=318, max=629, avg=473.97, stdev=219.66 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8192, max= 8192, per=44.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.30%, sys=18.57%, ctx=146, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22195: Fri May 2 00:15:08 2025 write: IOPS=1, BW=6872KiB/s (7037kB/s)(12.0MiB/1788msec) clat (msec): min=352, max=686, avg=535.04, stdev=169.11 lat (msec): min=353, max=687, avg=535.90, stdev=169.21 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 5868, max= 8175, per=38.31%, avg=7021.50, stdev=1631.30, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.06%, sys=16.56%, ctx=60, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22196: Fri May 2 00:15:08 2025 write: IOPS=2, BW=11.3MiB/s (11.9MB/s)(12.0MiB/1061msec) clat (msec): min=270, max=339, avg=306.17, stdev=34.67 lat (msec): min=270, max=339, avg=306.50, stdev=34.75 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8159, max= 8192, per=44.61%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=22.74%, ctx=60, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.1MiB/s (12.7MB/s), 3103KiB/s-18.5MiB/s (3178kB/s-19.4MB/s), io=16.0MiB (16.8MB), run=648-1320msec WRITE: bw=17.9MiB/s (18.8MB/s), 6206KiB/s-11.3MiB/s (6355kB/s-11.9MB/s), io=32.0MiB (33.6MB), run=1061-1788msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22189: Fri May 2 00:15:08 2025 read: IOPS=2, BW=10.8MiB/s (11.3MB/s)(12.0MiB/1111msec) clat (msec): min=94, max=658, avg=340.11, stdev=288.87 lat (msec): min=94, max=658, avg=340.11, stdev=288.87 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 94], | 30.00th=[ 94], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=5.23%, ctx=31, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22190: Fri May 2 00:15:08 2025 read: IOPS=0, BW=2160KiB/s (2212kB/s)(4096KiB/1896msec) clat (nsec): min=275808k, max=275808k, avg=275807893.00, stdev= 0.00 lat (nsec): min=275810k, max=275810k, avg=275810229.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 8175, max= 8175, per=94.61%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4321KiB/s (4424kB/s)(8192KiB/1896msec) clat (msec): min=733, max=779, avg=756.85, stdev=32.61 lat (msec): min=734, max=782, avg=758.28, stdev=34.19 clat percentiles (msec): | 1.00th=[ 735], 5.00th=[ 735], 10.00th=[ 735], 20.00th=[ 735], | 30.00th=[ 735], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 776], | 70.00th=[ 776], 80.00th=[ 776], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 5997, max= 5997, per=34.70%, avg=5997.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.26%, sys=5.33%, ctx=43, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22192: Fri May 2 00:15:08 2025 write: IOPS=2, BW=10.4MiB/s (10.9MB/s)(12.0MiB/1155msec) clat (msec): min=124, max=499, avg=337.30, stdev=192.41 lat (msec): min=124, max=499, avg=337.66, stdev=192.42 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 6122, max= 6122, per=35.42%, avg=6122.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=3.81%, ctx=28, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22193: Fri May 2 00:15:08 2025 write: IOPS=5, BW=23.7MiB/s (24.8MB/s)(12.0MiB/507msec) clat (msec): min=81, max=290, avg=161.56, stdev=112.78 lat (msec): min=81, max=290, avg=161.96, stdev=112.83 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 113], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min=16384, max=16384, per=94.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.40%, sys=4.74%, ctx=14, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8641KiB/s (8849kB/s), 2160KiB/s-10.8MiB/s (2212kB/s-11.3MB/s), io=16.0MiB (16.8MB), run=1111-1896msec WRITE: bw=16.9MiB/s (17.7MB/s), 4321KiB/s-23.7MiB/s (4424kB/s-24.8MB/s), io=32.0MiB (33.6MB), run=507-1896msec