== sanity test 398b: DIO and buffer IO race ============== 13:34:41 (1714412081) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.51323 s, 33.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24081: Mon Apr 29 13:35:08 2024 read: IOPS=61, BW=246KiB/s (252kB/s)(5992KiB/24371msec) clat (usec): min=42, max=215362, avg=7359.70, stdev=8863.91 lat (usec): min=42, max=215363, avg=7360.94, stdev=8864.00 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 78], 10.00th=[ 86], 20.00th=[ 112], | 30.00th=[ 3458], 40.00th=[ 5014], 50.00th=[ 6194], 60.00th=[ 7635], | 70.00th=[ 9503], 80.00th=[ 11863], 90.00th=[ 15795], 95.00th=[ 19006], | 99.00th=[ 28181], 99.50th=[ 32113], 99.90th=[ 85459], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 112, max= 456, per=24.94%, avg=245.37, stdev=75.25, samples=48 iops : min= 28, max= 114, avg=61.17, stdev=18.82, samples=48 write: IOPS=64, BW=258KiB/s (265kB/s)(6296KiB/24371msec) clat (usec): min=445, max=90343, avg=8406.50, stdev=8573.27 lat (usec): min=446, max=90344, avg=8407.97, stdev=8573.38 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 701], 10.00th=[ 750], 20.00th=[ 914], | 30.00th=[ 1745], 40.00th=[ 4146], 50.00th=[ 6587], 60.00th=[ 8848], | 70.00th=[10945], 80.00th=[14091], 90.00th=[19006], 95.00th=[23987], | 99.00th=[34341], 99.50th=[40633], 99.90th=[81265], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 104, max= 424, per=24.92%, avg=256.71, stdev=70.80, samples=48 iops : min= 26, max= 106, avg=64.00, stdev=17.69, samples=48 lat (usec) : 50=0.07%, 100=7.94%, 250=4.79%, 500=0.59%, 750=5.37% lat (usec) : 1000=6.12% lat (msec) : 2=5.08%, 4=6.41%, 10=32.85%, 20=24.28%, 50=6.22% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.14%, sys=19.34%, ctx=5730, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24083: Mon Apr 29 13:35:08 2024 read: IOPS=60, BW=244KiB/s (249kB/s)(5940KiB/24393msec) clat (usec): min=52, max=87560, avg=7198.00, stdev=6399.83 lat (usec): min=53, max=87562, avg=7199.23, stdev=6399.94 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 78], 10.00th=[ 87], 20.00th=[ 112], | 30.00th=[ 4080], 40.00th=[ 5407], 50.00th=[ 6587], 60.00th=[ 7767], | 70.00th=[ 9110], 80.00th=[11600], 90.00th=[15139], 95.00th=[18744], | 99.00th=[26084], 99.50th=[28181], 99.90th=[50594], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 128, max= 408, per=24.70%, avg=243.08, stdev=72.89, samples=48 iops : min= 32, max= 102, avg=60.63, stdev=18.24, samples=48 write: IOPS=65, BW=260KiB/s (266kB/s)(6348KiB/24393msec) clat (usec): min=441, max=211050, avg=8563.34, stdev=10297.43 lat (usec): min=441, max=211051, avg=8564.57, stdev=10297.45 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 701], 10.00th=[ 750], 20.00th=[ 898], | 30.00th=[ 1532], 40.00th=[ 3228], 50.00th=[ 6194], 60.00th=[ 8717], | 70.00th=[ 10945], 80.00th=[ 14353], 90.00th=[ 20055], 95.00th=[ 25297], | 99.00th=[ 38536], 99.50th=[ 46924], 99.90th=[ 92799], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 168, max= 472, per=25.01%, avg=257.58, stdev=71.96, samples=48 iops : min= 42, max= 118, avg=64.25, stdev=18.03, samples=48 lat (usec) : 100=7.85%, 250=3.81%, 500=0.39%, 750=5.08%, 1000=7.06% lat (msec) : 2=5.53%, 4=6.32%, 10=34.18%, 20=22.72%, 50=6.87% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.11%, sys=19.90%, ctx=5595, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24086: Mon Apr 29 13:35:08 2024 read: IOPS=62, BW=250KiB/s (256kB/s)(6028KiB/24156msec) clat (usec): min=45, max=209730, avg=7367.44, stdev=8423.97 lat (usec): min=45, max=209731, avg=7368.79, stdev=8423.65 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 115], | 30.00th=[ 3982], 40.00th=[ 5080], 50.00th=[ 6390], 60.00th=[ 7635], | 70.00th=[ 9372], 80.00th=[ 11731], 90.00th=[ 15795], 95.00th=[ 19006], | 99.00th=[ 24249], 99.50th=[ 27132], 99.90th=[ 81265], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 88, max= 502, per=25.21%, avg=248.02, stdev=75.69, samples=48 iops : min= 22, max= 125, avg=61.83, stdev=18.90, samples=48 write: IOPS=64, BW=259KiB/s (265kB/s)(6260KiB/24156msec) clat (usec): min=459, max=57960, avg=8261.67, stdev=8483.33 lat (usec): min=460, max=57961, avg=8264.06, stdev=8482.74 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 701], 10.00th=[ 750], 20.00th=[ 889], | 30.00th=[ 1450], 40.00th=[ 3490], 50.00th=[ 6194], 60.00th=[ 8356], | 70.00th=[10814], 80.00th=[13829], 90.00th=[19530], 95.00th=[25035], | 99.00th=[36963], 99.50th=[43254], 99.90th=[52167], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 152, max= 424, per=24.97%, avg=257.21, stdev=70.15, samples=48 iops : min= 38, max= 106, avg=64.13, stdev=17.53, samples=48 lat (usec) : 50=0.10%, 100=7.49%, 250=4.30%, 500=0.59%, 750=4.95% lat (usec) : 1000=6.84% lat (msec) : 2=5.89%, 4=5.86%, 10=34.15%, 20=23.08%, 50=6.58% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.21%, sys=19.50%, ctx=5738, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24087: Mon Apr 29 13:35:08 2024 read: IOPS=63, BW=253KiB/s (259kB/s)(6052KiB/23955msec) clat (usec): min=41, max=214849, avg=7242.65, stdev=8324.91 lat (usec): min=41, max=214849, avg=7243.60, stdev=8324.89 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 114], | 30.00th=[ 3916], 40.00th=[ 5211], 50.00th=[ 6390], 60.00th=[ 7635], | 70.00th=[ 9241], 80.00th=[ 11469], 90.00th=[ 15139], 95.00th=[ 18482], | 99.00th=[ 25035], 99.50th=[ 32375], 99.90th=[ 84411], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 144, max= 424, per=25.55%, avg=251.43, stdev=71.01, samples=47 iops : min= 36, max= 106, avg=62.66, stdev=17.75, samples=47 write: IOPS=65, BW=260KiB/s (267kB/s)(6236KiB/23955msec) clat (usec): min=426, max=90121, avg=8263.35, stdev=8490.87 lat (usec): min=426, max=90122, avg=8264.59, stdev=8491.02 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 701], 10.00th=[ 766], 20.00th=[ 947], | 30.00th=[ 1450], 40.00th=[ 3785], 50.00th=[ 6390], 60.00th=[ 8455], | 70.00th=[10683], 80.00th=[13960], 90.00th=[19268], 95.00th=[24773], | 99.00th=[38536], 99.50th=[43779], 99.90th=[52167], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 120, max= 488, per=25.14%, avg=258.91, stdev=68.95, samples=47 iops : min= 30, max= 122, avg=64.53, stdev=17.23, samples=47 lat (usec) : 50=0.10%, 100=7.75%, 250=4.00%, 500=0.55%, 750=4.46% lat (usec) : 1000=6.58% lat (msec) : 2=6.41%, 4=5.96%, 10=34.31%, 20=23.47%, 50=6.25% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.15%, sys=19.37%, ctx=5689, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=984KiB/s (1008kB/s), 244KiB/s-253KiB/s (249kB/s-259kB/s), io=23.4MiB (24.6MB), run=23955-24393msec WRITE: bw=1031KiB/s (1055kB/s), 258KiB/s-260KiB/s (265kB/s-267kB/s), io=24.6MiB (25.7MB), run=23955-24393msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24080: Mon Apr 29 13:36:28 2024 read: IOPS=14, BW=57.9KiB/s (59.3kB/s)(5992KiB/103551msec) clat (msec): min=2, max=470, avg= 8.73, stdev=14.82 lat (msec): min=2, max=470, avg= 8.73, stdev=14.82 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 16], 95.00th=[ 21], | 99.00th=[ 35], 99.50th=[ 45], 99.90th=[ 215], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 7, max= 151, per=25.18%, avg=58.17, stdev=29.72, samples=204 iops : min= 1, max= 37, avg=14.33, stdev= 7.44, samples=204 write: IOPS=15, BW=60.8KiB/s (62.3kB/s)(6296KiB/103551msec) clat (msec): min=20, max=149, avg=57.40, stdev=20.77 lat (msec): min=20, max=149, avg=57.40, stdev=20.77 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 34], 20.00th=[ 40], | 30.00th=[ 45], 40.00th=[ 50], 50.00th=[ 54], 60.00th=[ 59], | 70.00th=[ 66], 80.00th=[ 73], 90.00th=[ 87], 95.00th=[ 97], | 99.00th=[ 120], 99.50th=[ 126], 99.90th=[ 146], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 24, max= 104, per=24.99%, avg=60.48, stdev=16.75, samples=206 iops : min= 6, max= 26, avg=14.91, stdev= 4.19, samples=206 lat (msec) : 4=6.48%, 10=31.28%, 20=8.59%, 50=23.80%, 100=27.57% lat (msec) : 250=2.25%, 500=0.03% cpu : usr=0.09%, sys=3.18%, ctx=3713, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24082: Mon Apr 29 13:36:28 2024 read: IOPS=14, BW=57.2KiB/s (58.6kB/s)(5940KiB/103758msec) clat (msec): min=2, max=214, avg= 8.40, stdev= 9.03 lat (msec): min=2, max=214, avg= 8.40, stdev= 9.03 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 19], | 99.00th=[ 34], 99.50th=[ 57], 99.90th=[ 104], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 7, max= 152, per=24.91%, avg=57.53, stdev=32.21, samples=205 iops : min= 1, max= 38, avg=14.23, stdev= 8.08, samples=205 write: IOPS=15, BW=61.2KiB/s (62.6kB/s)(6348KiB/103758msec) clat (msec): min=18, max=175, avg=57.44, stdev=20.64 lat (msec): min=18, max=175, avg=57.44, stdev=20.64 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 31], 10.00th=[ 35], 20.00th=[ 40], | 30.00th=[ 45], 40.00th=[ 50], 50.00th=[ 55], 60.00th=[ 60], | 70.00th=[ 66], 80.00th=[ 72], 90.00th=[ 85], 95.00th=[ 95], | 99.00th=[ 122], 99.50th=[ 129], 99.90th=[ 169], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 24, max= 104, per=25.11%, avg=60.78, stdev=15.89, samples=207 iops : min= 6, max= 26, avg=15.04, stdev= 3.99, samples=207 lat (msec) : 4=7.45%, 10=29.69%, 20=9.24%, 50=23.14%, 100=28.26% lat (msec) : 250=2.21% cpu : usr=0.04%, sys=3.23%, ctx=3722, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24084: Mon Apr 29 13:36:28 2024 read: IOPS=14, BW=58.2KiB/s (59.6kB/s)(6028KiB/103626msec) clat (usec): min=2528, max=90535, avg=8724.42, stdev=7285.07 lat (usec): min=2529, max=90536, avg=8725.90, stdev=7284.97 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3458], 10.00th=[ 3818], 20.00th=[ 4359], | 30.00th=[ 4948], 40.00th=[ 5604], 50.00th=[ 6390], 60.00th=[ 7635], | 70.00th=[ 9110], 80.00th=[11469], 90.00th=[15533], 95.00th=[21627], | 99.00th=[36963], 99.50th=[48497], 99.90th=[86508], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 7, max= 143, per=25.07%, avg=57.92, stdev=28.75, samples=207 iops : min= 1, max= 35, avg=14.31, stdev= 7.17, samples=207 write: IOPS=15, BW=60.4KiB/s (61.9kB/s)(6260KiB/103626msec) clat (msec): min=20, max=237, avg=57.73, stdev=20.45 lat (msec): min=20, max=237, avg=57.73, stdev=20.46 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 46], 40.00th=[ 51], 50.00th=[ 55], 60.00th=[ 60], | 70.00th=[ 66], 80.00th=[ 73], 90.00th=[ 84], 95.00th=[ 95], | 99.00th=[ 121], 99.50th=[ 136], 99.90th=[ 176], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 16, max= 112, per=24.84%, avg=60.11, stdev=15.77, samples=207 iops : min= 4, max= 28, avg=14.86, stdev= 3.94, samples=207 lat (msec) : 4=6.54%, 10=30.18%, 20=9.37%, 50=23.31%, 100=29.20% lat (msec) : 250=1.40% cpu : usr=0.08%, sys=3.21%, ctx=3767, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24085: Mon Apr 29 13:36:28 2024 read: IOPS=14, BW=59.1KiB/s (60.5kB/s)(6052KiB/102443msec) clat (msec): min=2, max=141, avg= 8.37, stdev= 7.66 lat (msec): min=2, max=141, avg= 8.37, stdev= 7.66 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 15], 95.00th=[ 21], | 99.00th=[ 34], 99.50th=[ 43], 99.90th=[ 106], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 8, max= 152, per=25.51%, avg=58.93, stdev=30.56, samples=204 iops : min= 2, max= 38, avg=14.57, stdev= 7.60, samples=204 write: IOPS=15, BW=60.9KiB/s (62.3kB/s)(6236KiB/102443msec) clat (msec): min=21, max=237, avg=57.50, stdev=21.08 lat (msec): min=21, max=237, avg=57.51, stdev=21.08 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 31], 10.00th=[ 35], 20.00th=[ 40], | 30.00th=[ 45], 40.00th=[ 50], 50.00th=[ 54], 60.00th=[ 59], | 70.00th=[ 66], 80.00th=[ 73], 90.00th=[ 85], 95.00th=[ 97], | 99.00th=[ 122], 99.50th=[ 127], 99.90th=[ 165], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 24, max= 104, per=25.06%, avg=60.65, stdev=16.09, samples=204 iops : min= 6, max= 26, avg=15.00, stdev= 4.03, samples=204 lat (msec) : 4=7.19%, 10=30.31%, 20=9.28%, 50=23.60%, 100=27.31% lat (msec) : 250=2.31% cpu : usr=0.09%, sys=3.26%, ctx=3718, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=231KiB/s (237kB/s), 57.2KiB/s-59.1KiB/s (58.6kB/s-60.5kB/s), io=23.4MiB (24.6MB), run=102443-103758msec WRITE: bw=242KiB/s (248kB/s), 60.4KiB/s-61.2KiB/s (61.9kB/s-62.6kB/s), io=24.6MiB (25.7MB), run=102443-103758msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24191: Mon Apr 29 13:36:36 2024 read: IOPS=49, BW=793KiB/s (812kB/s)(5920KiB/7465msec) clat (usec): min=44, max=46503, avg=14896.27, stdev=9642.41 lat (usec): min=44, max=46504, avg=14897.25, stdev=9642.41 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 135], | 30.00th=[13173], 40.00th=[15008], 50.00th=[16188], 60.00th=[18220], | 70.00th=[20055], 80.00th=[22152], 90.00th=[25297], 95.00th=[28181], | 99.00th=[39060], 99.50th=[43779], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 608, max= 958, per=24.73%, avg=774.43, stdev=90.32, samples=14 iops : min= 38, max= 59, avg=48.21, stdev= 5.51, samples=14 write: IOPS=53, BW=853KiB/s (874kB/s)(6368KiB/7465msec) clat (usec): min=726, max=67182, avg=4885.39, stdev=6474.20 lat (usec): min=727, max=67184, avg=4887.27, stdev=6474.09 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 979], 10.00th=[ 1045], 20.00th=[ 1123], | 30.00th=[ 1205], 40.00th=[ 1385], 50.00th=[ 2409], 60.00th=[ 4293], | 70.00th=[ 5669], 80.00th=[ 7373], 90.00th=[10552], 95.00th=[13829], | 99.00th=[34341], 99.50th=[55837], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 544, max= 1152, per=25.51%, avg=840.57, stdev=219.57, samples=14 iops : min= 34, max= 72, avg=52.36, stdev=13.60, samples=14 lat (usec) : 50=0.13%, 100=5.21%, 250=5.34%, 500=0.52%, 750=0.65% lat (usec) : 1000=3.26% lat (msec) : 2=21.22%, 4=4.69%, 10=16.15%, 20=27.08%, 50=15.49% lat (msec) : 100=0.26% cpu : usr=0.20%, sys=17.75%, ctx=1966, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24193: Mon Apr 29 13:36:36 2024 read: IOPS=47, BW=767KiB/s (785kB/s)(5744KiB/7491msec) clat (usec): min=53, max=54735, avg=15079.54, stdev=10028.16 lat (usec): min=53, max=54735, avg=15080.48, stdev=10028.16 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 87], 10.00th=[ 95], 20.00th=[ 131], | 30.00th=[13566], 40.00th=[15401], 50.00th=[17171], 60.00th=[18744], | 70.00th=[20055], 80.00th=[22152], 90.00th=[25035], 95.00th=[28443], | 99.00th=[43779], 99.50th=[52167], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 448, max= 992, per=24.43%, avg=765.21, stdev=152.69, samples=14 iops : min= 28, max= 62, avg=47.64, stdev= 9.61, samples=14 write: IOPS=54, BW=874KiB/s (895kB/s)(6544KiB/7491msec) clat (usec): min=633, max=73472, avg=5054.73, stdev=8117.65 lat (usec): min=635, max=73475, avg=5056.50, stdev=8117.69 clat percentiles (usec): | 1.00th=[ 824], 5.00th=[ 947], 10.00th=[ 988], 20.00th=[ 1074], | 30.00th=[ 1139], 40.00th=[ 1303], 50.00th=[ 1729], 60.00th=[ 3818], | 70.00th=[ 5080], 80.00th=[ 7177], 90.00th=[11338], 95.00th=[16188], | 99.00th=[43254], 99.50th=[57410], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 576, max= 1341, per=26.27%, avg=865.50, stdev=224.09, samples=14 iops : min= 36, max= 83, avg=53.93, stdev=13.79, samples=14 lat (usec) : 100=5.47%, 250=5.60%, 500=0.26%, 750=0.39%, 1000=6.38% lat (msec) : 2=20.96%, 4=5.08%, 10=14.32%, 20=25.52%, 50=15.36% lat (msec) : 100=0.65% cpu : usr=0.21%, sys=17.22%, ctx=1828, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24195: Mon Apr 29 13:36:36 2024 read: IOPS=51, BW=818KiB/s (838kB/s)(6256KiB/7646msec) clat (usec): min=49, max=57805, avg=14953.74, stdev=9494.50 lat (usec): min=50, max=57805, avg=14955.16, stdev=9494.78 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 85], 10.00th=[ 92], 20.00th=[ 129], | 30.00th=[13173], 40.00th=[15270], 50.00th=[17171], 60.00th=[18744], | 70.00th=[20317], 80.00th=[22414], 90.00th=[24511], 95.00th=[27919], | 99.00th=[32113], 99.50th=[40633], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 608, max= 1085, per=25.99%, avg=813.93, stdev=129.21, samples=15 iops : min= 38, max= 67, avg=50.53, stdev= 7.79, samples=15 write: IOPS=49, BW=789KiB/s (808kB/s)(6032KiB/7646msec) clat (usec): min=626, max=38080, avg=4739.28, stdev=5563.63 lat (usec): min=626, max=38082, avg=4741.50, stdev=5564.23 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 922], 10.00th=[ 1004], 20.00th=[ 1106], | 30.00th=[ 1188], 40.00th=[ 1336], 50.00th=[ 2245], 60.00th=[ 4424], | 70.00th=[ 5538], 80.00th=[ 7504], 90.00th=[11076], 95.00th=[13960], | 99.00th=[31065], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 351, max= 1149, per=23.54%, avg=775.60, stdev=222.28, samples=15 iops : min= 21, max= 71, avg=48.13, stdev=13.95, samples=15 lat (usec) : 50=0.13%, 100=6.64%, 250=4.82%, 500=0.26%, 750=0.39% lat (usec) : 1000=4.43% lat (msec) : 2=19.14%, 4=4.43%, 10=14.84%, 20=27.60%, 50=17.19% lat (msec) : 100=0.13% cpu : usr=0.29%, sys=17.24%, ctx=1918, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24196: Mon Apr 29 13:36:36 2024 read: IOPS=49, BW=794KiB/s (813kB/s)(6032KiB/7595msec) clat (usec): min=55, max=83998, avg=15026.59, stdev=10655.90 lat (usec): min=55, max=83998, avg=15028.24, stdev=10656.25 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 80], 10.00th=[ 91], 20.00th=[ 118], | 30.00th=[12911], 40.00th=[15139], 50.00th=[16909], 60.00th=[18744], | 70.00th=[20317], 80.00th=[22152], 90.00th=[25822], 95.00th=[28181], | 99.00th=[39060], 99.50th=[65274], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 606, max= 1277, per=24.91%, avg=780.07, stdev=160.83, samples=15 iops : min= 37, max= 79, avg=48.47, stdev= 9.98, samples=15 write: IOPS=51, BW=824KiB/s (843kB/s)(6256KiB/7595msec) clat (usec): min=673, max=75685, avg=4902.79, stdev=6916.08 lat (usec): min=674, max=75687, avg=4904.67, stdev=6916.18 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 906], 10.00th=[ 971], 20.00th=[ 1106], | 30.00th=[ 1172], 40.00th=[ 1319], 50.00th=[ 1811], 60.00th=[ 3884], | 70.00th=[ 5342], 80.00th=[ 8356], 90.00th=[10945], 95.00th=[14877], | 99.00th=[32900], 99.50th=[53216], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 416, max= 1056, per=24.71%, avg=814.27, stdev=161.20, samples=15 iops : min= 26, max= 66, avg=50.60, stdev=10.04, samples=15 lat (usec) : 100=7.16%, 250=4.69%, 500=0.39%, 750=0.65%, 1000=5.34% lat (msec) : 2=20.44%, 4=4.95%, 10=13.15%, 20=26.17%, 50=16.54% lat (msec) : 100=0.52% cpu : usr=0.28%, sys=17.07%, ctx=1925, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3133KiB/s (3208kB/s), 767KiB/s-818KiB/s (785kB/s-838kB/s), io=23.4MiB (24.5MB), run=7465-7646msec WRITE: bw=3296KiB/s (3375kB/s), 789KiB/s-874KiB/s (808kB/s-895kB/s), io=24.6MiB (25.8MB), run=7465-7646msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24188: Mon Apr 29 13:36:50 2024 read: IOPS=17, BW=279KiB/s (285kB/s)(5920KiB/21252msec) clat (usec): min=2813, max=39967, avg=7467.36, stdev=6166.91 lat (usec): min=2814, max=39967, avg=7469.54, stdev=6167.08 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3425], 10.00th=[ 3654], 20.00th=[ 4113], | 30.00th=[ 4490], 40.00th=[ 4817], 50.00th=[ 5276], 60.00th=[ 5735], | 70.00th=[ 6652], 80.00th=[ 8455], 90.00th=[13829], 95.00th=[20841], | 99.00th=[35390], 99.50th=[39584], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 64, max= 672, per=24.94%, avg=276.31, stdev=129.71, samples=42 iops : min= 4, max= 42, avg=17.05, stdev= 8.10, samples=42 write: IOPS=18, BW=300KiB/s (307kB/s)(6368KiB/21252msec) clat (msec): min=18, max=108, avg=46.42, stdev=15.51 lat (msec): min=18, max=108, avg=46.42, stdev=15.51 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 24], 10.00th=[ 29], 20.00th=[ 34], | 30.00th=[ 37], 40.00th=[ 41], 50.00th=[ 44], 60.00th=[ 48], | 70.00th=[ 54], 80.00th=[ 59], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 99], 99.50th=[ 104], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 191, max= 480, per=25.59%, avg=298.40, stdev=64.23, samples=42 iops : min= 11, max= 30, avg=18.43, stdev= 4.02, samples=42 lat (msec) : 4=8.46%, 10=31.64%, 20=5.21%, 50=36.33%, 100=18.10% lat (msec) : 250=0.26% cpu : usr=0.08%, sys=3.99%, ctx=892, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24189: Mon Apr 29 13:36:50 2024 read: IOPS=16, BW=266KiB/s (272kB/s)(5744KiB/21602msec) clat (usec): min=2882, max=31498, avg=6703.08, stdev=4930.34 lat (usec): min=2883, max=31499, avg=6709.36, stdev=4932.80 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3228], 10.00th=[ 3523], 20.00th=[ 3884], | 30.00th=[ 4228], 40.00th=[ 4686], 50.00th=[ 5014], 60.00th=[ 5473], | 70.00th=[ 6128], 80.00th=[ 7308], 90.00th=[12649], 95.00th=[17957], | 99.00th=[26084], 99.50th=[31327], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 64, max= 640, per=23.99%, avg=265.84, stdev=123.69, samples=43 iops : min= 4, max= 40, avg=16.33, stdev= 7.77, samples=43 write: IOPS=18, BW=303KiB/s (310kB/s)(6544KiB/21602msec) clat (msec): min=18, max=130, avg=46.89, stdev=15.79 lat (msec): min=18, max=130, avg=46.90, stdev=15.79 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 26], 10.00th=[ 29], 20.00th=[ 33], | 30.00th=[ 37], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 54], 80.00th=[ 59], 90.00th=[ 68], 95.00th=[ 75], | 99.00th=[ 95], 99.50th=[ 105], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 128, max= 448, per=25.87%, avg=301.63, stdev=69.94, samples=43 iops : min= 8, max= 28, avg=18.56, stdev= 4.33, samples=43 lat (msec) : 4=10.81%, 10=29.04%, 20=5.21%, 50=35.81%, 100=18.62% lat (msec) : 250=0.52% cpu : usr=0.07%, sys=3.87%, ctx=881, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24192: Mon Apr 29 13:36:50 2024 read: IOPS=19, BW=304KiB/s (311kB/s)(6256KiB/20568msec) clat (usec): min=2485, max=50878, avg=7107.27, stdev=6580.76 lat (usec): min=2485, max=50879, avg=7108.71, stdev=6580.73 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3359], 10.00th=[ 3687], 20.00th=[ 3982], | 30.00th=[ 4293], 40.00th=[ 4621], 50.00th=[ 5014], 60.00th=[ 5604], | 70.00th=[ 6194], 80.00th=[ 7767], 90.00th=[11863], 95.00th=[20055], | 99.00th=[43779], 99.50th=[50070], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 96, max= 766, per=27.51%, avg=304.78, stdev=155.25, samples=41 iops : min= 6, max= 47, avg=18.76, stdev= 9.66, samples=41 write: IOPS=18, BW=293KiB/s (300kB/s)(6032KiB/20568msec) clat (msec): min=19, max=136, avg=47.14, stdev=16.05 lat (msec): min=19, max=136, avg=47.14, stdev=16.05 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 25], 10.00th=[ 30], 20.00th=[ 34], | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 50], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 67], 95.00th=[ 77], | 99.00th=[ 95], 99.50th=[ 106], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 191, max= 448, per=25.07%, avg=292.34, stdev=67.77, samples=41 iops : min= 11, max= 28, avg=17.98, stdev= 4.29, samples=41 lat (msec) : 4=10.81%, 10=33.46%, 20=4.30%, 50=31.90%, 100=19.14% lat (msec) : 250=0.39% cpu : usr=0.02%, sys=4.10%, ctx=893, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24194: Mon Apr 29 13:36:50 2024 read: IOPS=18, BW=289KiB/s (295kB/s)(6032KiB/20907msec) clat (usec): min=2586, max=44623, avg=6974.06, stdev=5688.10 lat (usec): min=2587, max=44624, avg=6975.86, stdev=5687.94 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3294], 10.00th=[ 3556], 20.00th=[ 3949], | 30.00th=[ 4293], 40.00th=[ 4686], 50.00th=[ 5080], 60.00th=[ 5604], | 70.00th=[ 6390], 80.00th=[ 7832], 90.00th=[12387], 95.00th=[21103], | 99.00th=[33817], 99.50th=[43779], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 32, max= 606, per=25.96%, avg=287.63, stdev=122.37, samples=41 iops : min= 2, max= 37, avg=17.68, stdev= 7.66, samples=41 write: IOPS=18, BW=299KiB/s (306kB/s)(6256KiB/20907msec) clat (msec): min=17, max=105, avg=46.71, stdev=14.31 lat (msec): min=17, max=105, avg=46.71, stdev=14.31 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 26], 10.00th=[ 29], 20.00th=[ 35], | 30.00th=[ 39], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 50], | 70.00th=[ 54], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 73], | 99.00th=[ 93], 99.50th=[ 96], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 191, max= 447, per=25.61%, avg=298.61, stdev=60.97, samples=41 iops : min= 11, max= 27, avg=18.37, stdev= 3.83, samples=41 lat (msec) : 4=10.68%, 10=31.51%, 20=4.69%, 50=33.59%, 100=19.40% lat (msec) : 250=0.13% cpu : usr=0.06%, sys=3.93%, ctx=892, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1109KiB/s (1135kB/s), 266KiB/s-304KiB/s (272kB/s-311kB/s), io=23.4MiB (24.5MB), run=20568-21602msec WRITE: bw=1167KiB/s (1195kB/s), 293KiB/s-303KiB/s (300kB/s-310kB/s), io=24.6MiB (25.8MB), run=20568-21602msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24221: Mon Apr 29 13:36:57 2024 read: IOPS=2, BW=2557KiB/s (2619kB/s)(7168KiB/2803msec) clat (usec): min=458, max=1025.5k, avg=358309.18, stdev=427141.09 lat (usec): min=460, max=1025.5k, avg=358310.73, stdev=427141.27 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 461], 10.00th=[ 461], | 20.00th=[ 5342], 30.00th=[ 44827], 40.00th=[ 44827], | 50.00th=[ 261096], 60.00th=[ 278922], 70.00th=[ 278922], | 80.00th=[ 893387], 90.00th=[1027605], 95.00th=[1027605], | 99.00th=[1027605], 99.50th=[1027605], 99.90th=[1027605], | 99.95th=[1027605], 99.99th=[1027605] bw ( KiB/s): min= 2048, max= 6144, per=100.00%, avg=4093.00, stdev=2048.01, samples=3 iops : min= 2, max= 6, avg= 3.67, stdev= 2.08, samples=3 write: IOPS=1, BW=1827KiB/s (1870kB/s)(5120KiB/2803msec) clat (msec): min=29, max=133, avg=58.08, stdev=42.78 lat (msec): min=29, max=133, avg=58.14, stdev=42.75 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 43], 60.00th=[ 43], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 2048, max= 6131, per=77.13%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 50=41.67%, 250=8.33%, 500=16.67%, 1000=8.33% cpu : usr=0.11%, sys=23.52%, ctx=668, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24223: Mon Apr 29 13:36:57 2024 read: IOPS=0, BW=715KiB/s (732kB/s)(2048KiB/2866msec) clat (msec): min=51, max=1894, avg=973.29, stdev=1302.96 lat (msec): min=51, max=1894, avg=973.30, stdev=1302.96 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 1888], | 70.00th=[ 1888], 80.00th=[ 1888], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 2048, max= 2048, per=54.08%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3573KiB/s (3659kB/s)(10.0MiB/2866msec) clat (msec): min=22, max=379, avg=91.74, stdev=120.19 lat (msec): min=22, max=379, avg=91.83, stdev=120.18 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 35], 60.00th=[ 39], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 241], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 2048, max= 6144, per=77.25%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=58.33%, 100=16.67%, 250=8.33%, 500=8.33% cpu : usr=0.03%, sys=13.89%, ctx=362, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24225: Mon Apr 29 13:36:57 2024 read: IOPS=1, BW=1136KiB/s (1164kB/s)(6144KiB/5407msec) clat (usec): min=552, max=1298.3k, avg=317345.93, stdev=510795.24 lat (usec): min=554, max=1298.3k, avg=317347.40, stdev=510795.27 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], | 20.00th=[ 791], 30.00th=[ 791], 40.00th=[ 1237], | 50.00th=[ 1237], 60.00th=[ 158335], 70.00th=[ 446694], | 80.00th=[ 446694], 90.00th=[1300235], 95.00th=[1300235], | 99.00th=[1300235], 99.50th=[1300235], 99.90th=[1300235], | 99.95th=[1300235], 99.99th=[1300235] bw ( KiB/s): min= 2043, max= 4096, per=90.01%, avg=3408.67, stdev=1182.71, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 write: IOPS=1, BW=1136KiB/s (1164kB/s)(6144KiB/5407msec) clat (msec): min=38, max=2438, avg=583.41, stdev=952.46 lat (msec): min=38, max=2438, avg=583.47, stdev=952.45 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 138], | 70.00th=[ 776], 80.00th=[ 776], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 2048, max= 4096, per=64.32%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 2=8.33%, 50=16.67%, 100=8.33%, 250=16.67%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.02%, sys=11.41%, ctx=436, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24226: Mon Apr 29 13:36:57 2024 read: IOPS=0, BW=1003KiB/s (1027kB/s)(5120KiB/5107msec) clat (msec): min=5, max=1221, avg=474.10, stdev=566.73 lat (msec): min=5, max=1221, avg=474.10, stdev=566.73 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 134], 60.00th=[ 134], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 2048, max= 2048, per=54.08%, avg=2048.00, stdev= 0.00, samples=3 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=3 write: IOPS=1, BW=1404KiB/s (1437kB/s)(7168KiB/5107msec) clat (msec): min=22, max=2377, avg=390.70, stdev=877.51 lat (msec): min=22, max=2377, avg=390.77, stdev=877.51 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 38], 60.00th=[ 83], | 70.00th=[ 83], 80.00th=[ 169], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 2048, max= 6144, per=77.25%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 10=8.33%, 50=33.33%, 100=16.67%, 250=16.67%, 1000=8.33% cpu : usr=0.08%, sys=23.42%, ctx=486, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3788KiB/s (3879kB/s), 715KiB/s-2557KiB/s (732kB/s-2619kB/s), io=20.0MiB (20.0MB), run=2803-5407msec WRITE: bw=5303KiB/s (5430kB/s), 1136KiB/s-3573KiB/s (1164kB/s-3659kB/s), io=28.0MiB (29.4MB), run=2803-5407msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24219: Mon Apr 29 13:36:57 2024 read: IOPS=2, BW=2451KiB/s (2509kB/s)(7168KiB/2925msec) clat (msec): min=26, max=817, avg=314.05, stdev=333.92 lat (msec): min=26, max=817, avg=314.06, stdev=333.92 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 42], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 215], 60.00th=[ 279], | 70.00th=[ 279], 80.00th=[ 751], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 1796, max= 6144, per=100.00%, avg=4012.00, stdev=2175.22, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 write: IOPS=1, BW=1750KiB/s (1792kB/s)(5120KiB/2925msec) clat (msec): min=77, max=232, avg=144.84, stdev=65.52 lat (msec): min=77, max=232, avg=144.93, stdev=65.49 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 91], 40.00th=[ 91], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 2048, max= 6144, per=82.81%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 50=16.67%, 100=25.00%, 250=33.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.07%, sys=4.14%, ctx=51, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24220: Mon Apr 29 13:36:57 2024 read: IOPS=0, BW=353KiB/s (362kB/s)(2048KiB/5796msec) clat (msec): min=68, max=980, avg=524.23, stdev=644.79 lat (msec): min=68, max=980, avg=524.23, stdev=644.79 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 978], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 1809, max= 2048, per=54.59%, avg=1928.50, stdev=169.00, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1767KiB/s (1809kB/s)(10.0MiB/5796msec) clat (msec): min=60, max=2455, avg=474.33, stdev=789.85 lat (msec): min=60, max=2455, avg=474.40, stdev=789.85 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 68], | 30.00th=[ 78], 40.00th=[ 93], 50.00th=[ 96], 60.00th=[ 133], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 1284], 95.00th=[ 2467], | 99.00th=[ 2467], 99.50th=[ 2467], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 1805, max= 6131, per=79.10%, avg=3912.50, stdev=1777.99, samples=4 iops : min= 1, max= 5, avg= 3.25, stdev= 1.71, samples=4 lat (msec) : 100=50.00%, 250=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=1.98%, ctx=35, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24222: Mon Apr 29 13:36:57 2024 read: IOPS=1, BW=1082KiB/s (1108kB/s)(6144KiB/5680msec) clat (msec): min=27, max=1216, avg=284.36, stdev=468.36 lat (msec): min=27, max=1216, avg=284.37, stdev=468.36 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 63], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 1793, max= 5389, per=84.71%, avg=2992.67, stdev=2075.29, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1082KiB/s (1108kB/s)(6144KiB/5680msec) clat (msec): min=65, max=2345, avg=661.11, stdev=937.89 lat (msec): min=65, max=2345, avg=661.20, stdev=937.88 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 131], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 2333], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 1793, max= 4096, per=57.00%, avg=2819.25, stdev=1201.04, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 lat (msec) : 50=16.67%, 100=25.00%, 250=25.00%, 500=8.33% cpu : usr=0.02%, sys=2.20%, ctx=117, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24224: Mon Apr 29 13:36:57 2024 read: IOPS=2, BW=2078KiB/s (2128kB/s)(5120KiB/2464msec) clat (msec): min=32, max=578, avg=147.04, stdev=241.38 lat (msec): min=32, max=578, avg=147.04, stdev=241.38 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 44], 60.00th=[ 44], | 70.00th=[ 47], 80.00th=[ 47], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2909KiB/s (2979kB/s)(7168KiB/2464msec) clat (msec): min=67, max=1200, avg=246.65, stdev=421.21 lat (msec): min=67, max=1200, avg=246.76, stdev=421.21 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 99], | 70.00th=[ 99], 80.00th=[ 136], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 2048, max= 8192, per=96.56%, avg=4775.67, stdev=3129.36, samples=3 iops : min= 2, max= 8, avg= 4.33, stdev= 3.21, samples=3 lat (msec) : 50=33.33%, 100=41.67%, 250=8.33%, 750=8.33% cpu : usr=0.08%, sys=4.59%, ctx=26, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3533KiB/s (3618kB/s), 353KiB/s-2451KiB/s (362kB/s-2509kB/s), io=20.0MiB (20.0MB), run=2464-5796msec WRITE: bw=4947KiB/s (5066kB/s), 1082KiB/s-2909KiB/s (1108kB/s-2979kB/s), io=28.0MiB (29.4MB), run=2464-5796msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24240: Mon Apr 29 13:37:03 2024 read: IOPS=0, BW=2962KiB/s (3033kB/s)(12.0MiB/4149msec) clat (msec): min=228, max=2188, avg=1382.35, stdev=1025.03 lat (msec): min=228, max=2188, avg=1382.35, stdev=1025.03 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 1737], 50.00th=[ 1737], 60.00th=[ 1737], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.05%, sys=45.44%, ctx=809, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24241: Mon Apr 29 13:37:03 2024 read: IOPS=0, BW=870KiB/s (891kB/s)(4096KiB/4708msec) clat (nsec): min=228873k, max=228873k, avg=228872706.00, stdev= 0.00 lat (nsec): min=228875k, max=228875k, avg=228874716.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1740KiB/s (1782kB/s)(8192KiB/4708msec) clat (msec): min=380, max=4097, avg=2238.88, stdev=2628.54 lat (msec): min=380, max=4097, avg=2239.22, stdev=2628.55 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 4111], | 70.00th=[ 4111], 80.00th=[ 4111], 90.00th=[ 4111], 95.00th=[ 4111], | 99.00th=[ 4111], 99.50th=[ 4111], 99.90th=[ 4111], 99.95th=[ 4111], | 99.99th=[ 4111] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=7.54%, ctx=150, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24242: Mon Apr 29 13:37:03 2024 write: IOPS=0, BW=2610KiB/s (2673kB/s)(12.0MiB/4708msec) clat (msec): min=113, max=4141, avg=1568.54, stdev=2234.52 lat (msec): min=113, max=4141, avg=1568.82, stdev=2234.59 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 4144], 80.00th=[ 4144], 90.00th=[ 4144], 95.00th=[ 4144], | 99.00th=[ 4144], 99.50th=[ 4144], 99.90th=[ 4144], 99.95th=[ 4144], | 99.99th=[ 4144] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.02%, sys=7.01%, ctx=25, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24243: Mon Apr 29 13:37:03 2024 write: IOPS=0, BW=3444KiB/s (3527kB/s)(12.0MiB/3568msec) clat (msec): min=235, max=3014, avg=1188.60, stdev=1581.47 lat (msec): min=235, max=3014, avg=1188.88, stdev=1581.55 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 3004], 80.00th=[ 3004], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=8.69%, ctx=51, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3480KiB/s (3564kB/s), 870KiB/s-2962KiB/s (891kB/s-3033kB/s), io=16.0MiB (16.8MB), run=4149-4708msec WRITE: bw=6960KiB/s (7127kB/s), 1740KiB/s-3444KiB/s (1782kB/s-3527kB/s), io=32.0MiB (33.6MB), run=3568-4708msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24236: Mon Apr 29 13:37:03 2024 read: IOPS=0, BW=2742KiB/s (2807kB/s)(12.0MiB/4482msec) clat (msec): min=324, max=2152, avg=1490.32, stdev=1012.72 lat (msec): min=324, max=2152, avg=1490.33, stdev=1012.72 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 1989], 50.00th=[ 1989], 60.00th=[ 1989], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7846, max= 8192, per=100.00%, avg=8019.00, stdev=244.66, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.02%, sys=2.12%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24237: Mon Apr 29 13:37:03 2024 read: IOPS=0, BW=903KiB/s (925kB/s)(4096KiB/4536msec) clat (nsec): min=93526k, max=93526k, avg=93525700.00, stdev= 0.00 lat (nsec): min=93527k, max=93527k, avg=93526507.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[93848], 5.00th=[93848], 10.00th=[93848], 20.00th=[93848], | 30.00th=[93848], 40.00th=[93848], 50.00th=[93848], 60.00th=[93848], | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1806KiB/s (1849kB/s)(8192KiB/4536msec) clat (msec): min=414, max=4024, avg=2219.38, stdev=2552.28 lat (msec): min=415, max=4024, avg=2220.09, stdev=2551.83 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 4010], | 70.00th=[ 4010], 80.00th=[ 4010], 90.00th=[ 4010], 95.00th=[ 4010], | 99.00th=[ 4010], 99.50th=[ 4010], 99.90th=[ 4010], 99.95th=[ 4010], | 99.99th=[ 4010] bw ( KiB/s): min= 7757, max= 7757, per=100.00%, avg=7757.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=2.78%, ctx=47, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24238: Mon Apr 29 13:37:03 2024 write: IOPS=0, BW=2445KiB/s (2504kB/s)(12.0MiB/5025msec) clat (msec): min=122, max=4159, avg=1668.87, stdev=2177.64 lat (msec): min=122, max=4159, avg=1669.48, stdev=2177.45 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 4144], 80.00th=[ 4144], 90.00th=[ 4144], 95.00th=[ 4144], | 99.00th=[ 4144], 99.50th=[ 4144], 99.90th=[ 4144], 99.95th=[ 4144], | 99.99th=[ 4144] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.04%, sys=3.38%, ctx=36, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24239: Mon Apr 29 13:37:03 2024 write: IOPS=3, BW=13.8MiB/s (14.5MB/s)(12.0MiB/867msec) clat (msec): min=125, max=524, avg=285.60, stdev=211.01 lat (msec): min=126, max=525, avg=286.03, stdev=210.84 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.12%, sys=11.20%, ctx=29, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3612KiB/s (3699kB/s), 903KiB/s-2742KiB/s (925kB/s-2807kB/s), io=16.0MiB (16.8MB), run=4482-4536msec WRITE: bw=6521KiB/s (6677kB/s), 1806KiB/s-13.8MiB/s (1849kB/s-14.5MB/s), io=32.0MiB (33.6MB), run=867-5025msec