== sanity test 398b: DIO and buffer IO race ============== 02:57:03 (1746082623) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.09644 s, 24.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4845: Thu May 1 02:57:45 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(5992KiB/37138msec) clat (usec): min=69, max=345519, avg=11117.87, stdev=12858.34 lat (usec): min=69, max=345521, avg=11118.99, stdev=12858.35 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 95], 10.00th=[ 105], 20.00th=[ 147], | 30.00th=[ 6063], 40.00th=[ 8094], 50.00th=[ 9765], 60.00th=[ 11338], | 70.00th=[ 13960], 80.00th=[ 17433], 90.00th=[ 23200], 95.00th=[ 28181], | 99.00th=[ 39060], 99.50th=[ 49021], 99.90th=[ 90702], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 15, max= 320, per=24.95%, avg=160.18, stdev=49.49, samples=74 iops : min= 3, max= 80, avg=39.64, stdev=12.44, samples=74 write: IOPS=42, BW=170KiB/s (174kB/s)(6296KiB/37138msec) clat (usec): min=527, max=99085, avg=12861.86, stdev=13316.80 lat (usec): min=528, max=99088, avg=12864.54, stdev=13316.54 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1188], | 30.00th=[ 2474], 40.00th=[ 5407], 50.00th=[ 9634], 60.00th=[13304], | 70.00th=[16909], 80.00th=[22414], 90.00th=[30016], 95.00th=[37487], | 99.00th=[60556], 99.50th=[68682], 99.90th=[95945], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 95, max= 344, per=25.40%, avg=170.92, stdev=46.16, samples=73 iops : min= 23, max= 86, avg=42.33, stdev=11.58, samples=73 lat (usec) : 100=3.65%, 250=6.93%, 500=0.46%, 750=3.65%, 1000=5.86% lat (msec) : 2=4.85%, 4=5.18%, 10=20.74%, 20=28.91%, 50=18.55% lat (msec) : 100=1.20%, 500=0.03% cpu : usr=0.23%, sys=11.96%, ctx=4935, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4846: Thu May 1 02:57:45 2025 read: IOPS=39, BW=159KiB/s (163kB/s)(5940KiB/37351msec) clat (usec): min=51, max=222129, avg=10550.23, stdev=10658.12 lat (usec): min=51, max=222132, avg=10554.14, stdev=10661.03 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 147], | 30.00th=[ 5932], 40.00th=[ 7570], 50.00th=[ 9110], 60.00th=[ 10945], | 70.00th=[ 13173], 80.00th=[ 16909], 90.00th=[ 22414], 95.00th=[ 26870], | 99.00th=[ 36963], 99.50th=[ 47973], 99.90th=[ 77071], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 8, max= 270, per=24.93%, avg=160.03, stdev=45.89, samples=73 iops : min= 2, max= 67, avg=39.74, stdev=11.49, samples=73 write: IOPS=42, BW=170KiB/s (174kB/s)(6348KiB/37351msec) clat (usec): min=532, max=161571, avg=13455.58, stdev=14955.77 lat (usec): min=532, max=161573, avg=13458.24, stdev=14955.25 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 1090], | 30.00th=[ 2376], 40.00th=[ 6259], 50.00th=[ 10290], 60.00th=[ 13435], | 70.00th=[ 17171], 80.00th=[ 22414], 90.00th=[ 31327], 95.00th=[ 39584], | 99.00th=[ 60556], 99.50th=[ 86508], 99.90th=[147850], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 31, max= 288, per=25.24%, avg=169.88, stdev=47.94, samples=73 iops : min= 7, max= 72, avg=42.19, stdev=12.00, samples=73 lat (usec) : 100=3.68%, 250=6.80%, 500=0.55%, 750=2.90%, 1000=7.29% lat (msec) : 2=4.88%, 4=4.33%, 10=21.35%, 20=28.71%, 50=18.20% lat (msec) : 100=1.11%, 250=0.20% cpu : usr=0.07%, sys=11.93%, ctx=4859, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4847: Thu May 1 02:57:45 2025 read: IOPS=40, BW=162KiB/s (166kB/s)(6028KiB/37167msec) clat (usec): min=50, max=346782, avg=11149.30, stdev=13329.85 lat (usec): min=50, max=346784, avg=11154.36, stdev=13329.79 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 95], 10.00th=[ 105], 20.00th=[ 137], | 30.00th=[ 5800], 40.00th=[ 8094], 50.00th=[ 9634], 60.00th=[ 11338], | 70.00th=[ 13960], 80.00th=[ 17695], 90.00th=[ 23462], 95.00th=[ 27919], | 99.00th=[ 42730], 99.50th=[ 52167], 99.90th=[129500], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 7, max= 264, per=25.20%, avg=161.81, stdev=45.33, samples=74 iops : min= 1, max= 66, avg=40.20, stdev=11.42, samples=74 write: IOPS=42, BW=168KiB/s (172kB/s)(6260KiB/37167msec) clat (usec): min=572, max=81059, avg=12857.29, stdev=12838.76 lat (usec): min=574, max=81060, avg=12859.98, stdev=12838.61 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 1188], | 30.00th=[ 2278], 40.00th=[ 5866], 50.00th=[ 9896], 60.00th=[13304], | 70.00th=[17433], 80.00th=[22152], 90.00th=[30540], 95.00th=[38011], | 99.00th=[53216], 99.50th=[63701], 99.90th=[72877], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 23, max= 280, per=24.93%, avg=167.76, stdev=41.67, samples=74 iops : min= 5, max= 70, avg=41.69, stdev=10.49, samples=74 lat (usec) : 100=3.52%, 250=7.39%, 500=0.52%, 750=3.16%, 1000=6.28% lat (msec) : 2=5.34%, 4=4.65%, 10=20.61%, 20=28.61%, 50=18.68% lat (msec) : 100=1.17%, 250=0.03%, 500=0.03% cpu : usr=0.14%, sys=12.15%, ctx=4912, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4848: Thu May 1 02:57:45 2025 read: IOPS=40, BW=162KiB/s (166kB/s)(6052KiB/37255msec) clat (usec): min=49, max=293346, avg=10926.13, stdev=12477.17 lat (usec): min=49, max=293347, avg=10927.63, stdev=12477.24 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 95], 10.00th=[ 110], 20.00th=[ 143], | 30.00th=[ 5800], 40.00th=[ 7635], 50.00th=[ 9372], 60.00th=[ 10814], | 70.00th=[ 13435], 80.00th=[ 17433], 90.00th=[ 22938], 95.00th=[ 28181], | 99.00th=[ 42730], 99.50th=[ 54789], 99.90th=[108528], 99.95th=[291505], | 99.99th=[291505] bw ( KiB/s): min= 23, max= 255, per=25.11%, avg=161.23, stdev=48.01, samples=74 iops : min= 5, max= 63, avg=40.03, stdev=11.96, samples=74 write: IOPS=41, BW=167KiB/s (171kB/s)(6236KiB/37255msec) clat (usec): min=496, max=111914, avg=13140.66, stdev=13546.92 lat (usec): min=497, max=111919, avg=13144.49, stdev=13545.97 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 1237], | 30.00th=[ 2606], 40.00th=[ 6259], 50.00th=[ 9765], 60.00th=[ 13698], | 70.00th=[ 17695], 80.00th=[ 22938], 90.00th=[ 30016], 95.00th=[ 38536], | 99.00th=[ 60031], 99.50th=[ 71828], 99.90th=[107480], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 31, max= 272, per=24.74%, avg=166.51, stdev=41.30, samples=74 iops : min= 7, max= 68, avg=41.35, stdev=10.39, samples=74 lat (usec) : 50=0.03%, 100=3.06%, 250=7.94%, 500=0.36%, 750=3.58% lat (usec) : 1000=5.63% lat (msec) : 2=5.01%, 4=5.05%, 10=22.04%, 20=27.47%, 50=18.65% lat (msec) : 100=1.01%, 250=0.13%, 500=0.03% cpu : usr=0.17%, sys=11.94%, ctx=4986, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=643KiB/s (658kB/s), 159KiB/s-162KiB/s (163kB/s-166kB/s), io=23.4MiB (24.6MB), run=37138-37351msec WRITE: bw=673KiB/s (689kB/s), 167KiB/s-170KiB/s (171kB/s-174kB/s), io=24.6MiB (25.7MB), run=37138-37351msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4841: Thu May 1 02:59:26 2025 read: IOPS=10, BW=43.4KiB/s (44.5kB/s)(5992KiB/138023msec) clat (msec): min=2, max=347, avg=11.71, stdev=13.93 lat (msec): min=2, max=347, avg=11.71, stdev=13.93 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 22], 95.00th=[ 32], | 99.00th=[ 57], 99.50th=[ 84], 99.90th=[ 138], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 7, max= 151, per=25.52%, avg=44.15, stdev=26.86, samples=269 iops : min= 1, max= 37, avg=10.74, stdev= 6.73, samples=269 write: IOPS=11, BW=45.6KiB/s (46.7kB/s)(6296KiB/138023msec) clat (msec): min=23, max=253, avg=76.38, stdev=29.22 lat (msec): min=23, max=253, avg=76.38, stdev=29.22 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 53], | 30.00th=[ 59], 40.00th=[ 65], 50.00th=[ 71], 60.00th=[ 79], | 70.00th=[ 88], 80.00th=[ 97], 90.00th=[ 113], 95.00th=[ 133], | 99.00th=[ 176], 99.50th=[ 184], 99.90th=[ 215], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 15, max= 80, per=25.06%, avg=45.36, stdev=13.61, samples=275 iops : min= 3, max= 20, avg=11.05, stdev= 3.48, samples=275 lat (msec) : 4=1.20%, 10=30.89%, 20=11.43%, 50=12.99%, 100=34.44% lat (msec) : 250=8.98%, 500=0.07% cpu : usr=0.03%, sys=2.67%, ctx=3525, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4842: Thu May 1 02:59:26 2025 read: IOPS=10, BW=42.8KiB/s (43.9kB/s)(5940KiB/138632msec) clat (msec): min=3, max=299, avg=11.83, stdev=12.65 lat (msec): min=3, max=299, avg=11.83, stdev=12.65 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 24], 95.00th=[ 32], | 99.00th=[ 54], 99.50th=[ 62], 99.90th=[ 138], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 7, max= 127, per=25.16%, avg=43.53, stdev=24.79, samples=270 iops : min= 1, max= 31, avg=10.60, stdev= 6.21, samples=270 write: IOPS=11, BW=45.8KiB/s (46.9kB/s)(6348KiB/138632msec) clat (msec): min=20, max=254, avg=76.12, stdev=28.95 lat (msec): min=20, max=254, avg=76.13, stdev=28.95 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 45], 20.00th=[ 53], | 30.00th=[ 59], 40.00th=[ 65], 50.00th=[ 71], 60.00th=[ 78], | 70.00th=[ 86], 80.00th=[ 97], 90.00th=[ 114], 95.00th=[ 130], | 99.00th=[ 169], 99.50th=[ 182], 99.90th=[ 224], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 15, max= 88, per=25.15%, avg=45.52, stdev=14.21, samples=276 iops : min= 3, max= 22, avg=11.09, stdev= 3.63, samples=276 lat (msec) : 4=0.68%, 10=30.53%, 20=11.26%, 50=13.67%, 100=34.15% lat (msec) : 250=9.64%, 500=0.07% cpu : usr=0.04%, sys=2.59%, ctx=3487, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4843: Thu May 1 02:59:26 2025 read: IOPS=10, BW=43.6KiB/s (44.6kB/s)(6028KiB/138396msec) clat (usec): min=2814, max=96759, avg=11602.23, stdev=9549.86 lat (usec): min=2815, max=96761, avg=11603.84, stdev=9549.84 clat percentiles (usec): | 1.00th=[ 3818], 5.00th=[ 4490], 10.00th=[ 5080], 20.00th=[ 5866], | 30.00th=[ 6652], 40.00th=[ 7504], 50.00th=[ 8455], 60.00th=[ 9503], | 70.00th=[10945], 80.00th=[14484], 90.00th=[22938], 95.00th=[31065], | 99.00th=[49021], 99.50th=[61080], 99.90th=[88605], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 7, max= 151, per=25.43%, avg=43.99, stdev=25.22, samples=271 iops : min= 1, max= 37, avg=10.76, stdev= 6.28, samples=271 write: IOPS=11, BW=45.2KiB/s (46.3kB/s)(6260KiB/138396msec) clat (msec): min=25, max=350, avg=77.10, stdev=29.47 lat (msec): min=25, max=350, avg=77.10, stdev=29.47 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 46], 20.00th=[ 53], | 30.00th=[ 61], 40.00th=[ 66], 50.00th=[ 72], 60.00th=[ 80], | 70.00th=[ 88], 80.00th=[ 99], 90.00th=[ 114], 95.00th=[ 129], | 99.00th=[ 171], 99.50th=[ 203], 99.90th=[ 234], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 15, max= 80, per=24.86%, avg=44.99, stdev=13.02, samples=275 iops : min= 3, max= 20, avg=11.00, stdev= 3.29, samples=275 lat (msec) : 4=0.78%, 10=30.14%, 20=11.75%, 50=14.32%, 100=33.92% lat (msec) : 250=9.05%, 500=0.03% cpu : usr=0.06%, sys=2.63%, ctx=3463, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4844: Thu May 1 02:59:26 2025 read: IOPS=10, BW=43.8KiB/s (44.8kB/s)(6052KiB/138330msec) clat (msec): min=2, max=130, avg=11.48, stdev=10.19 lat (msec): min=2, max=130, avg=11.48, stdev=10.19 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 23], 95.00th=[ 32], | 99.00th=[ 53], 99.50th=[ 69], 99.90th=[ 109], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 7, max= 120, per=25.92%, avg=44.84, stdev=24.23, samples=267 iops : min= 1, max= 30, avg=10.94, stdev= 6.10, samples=267 write: IOPS=11, BW=45.1KiB/s (46.2kB/s)(6236KiB/138330msec) clat (msec): min=27, max=403, avg=77.38, stdev=29.80 lat (msec): min=27, max=403, avg=77.38, stdev=29.80 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 53], | 30.00th=[ 60], 40.00th=[ 66], 50.00th=[ 72], 60.00th=[ 81], | 70.00th=[ 89], 80.00th=[ 99], 90.00th=[ 114], 95.00th=[ 133], | 99.00th=[ 167], 99.50th=[ 194], 99.90th=[ 215], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 15, max= 80, per=24.76%, avg=44.81, stdev=13.29, samples=275 iops : min= 3, max= 20, avg=10.93, stdev= 3.38, samples=275 lat (msec) : 4=0.91%, 10=31.64%, 20=10.58%, 50=14.03%, 100=33.46% lat (msec) : 250=9.34%, 500=0.03% cpu : usr=0.05%, sys=2.62%, ctx=3513, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=173KiB/s (177kB/s), 42.8KiB/s-43.8KiB/s (43.9kB/s-44.8kB/s), io=23.4MiB (24.6MB), run=138023-138632msec WRITE: bw=181KiB/s (186kB/s), 45.1KiB/s-45.8KiB/s (46.2kB/s-46.9kB/s), io=24.6MiB (25.7MB), run=138023-138632msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4974: Thu May 1 02:59:35 2025 read: IOPS=51, BW=830KiB/s (850kB/s)(5920KiB/7136msec) clat (usec): min=52, max=50241, avg=8361.13, stdev=8186.60 lat (usec): min=52, max=50242, avg=8363.65, stdev=8188.34 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 116], | 30.00th=[ 147], 40.00th=[ 5604], 50.00th=[ 7832], 60.00th=[ 9372], | 70.00th=[11731], 80.00th=[14484], 90.00th=[18220], 95.00th=[22676], | 99.00th=[34866], 99.50th=[42730], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 224, max= 1472, per=26.22%, avg=822.21, stdev=277.74, samples=14 iops : min= 14, max= 92, avg=51.14, stdev=17.51, samples=14 write: IOPS=55, BW=892KiB/s (914kB/s)(6368KiB/7138msec) clat (usec): min=795, max=115502, avg=9996.73, stdev=13481.28 lat (usec): min=796, max=115504, avg=10001.38, stdev=13481.52 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 996], 10.00th=[ 1090], 20.00th=[ 1303], | 30.00th=[ 1631], 40.00th=[ 2343], 50.00th=[ 3458], 60.00th=[ 8029], | 70.00th=[ 11994], 80.00th=[ 17433], 90.00th=[ 25560], 95.00th=[ 31589], | 99.00th=[ 63177], 99.50th=[ 91751], 99.90th=[115868], 99.95th=[115868], | 99.99th=[115868] bw ( KiB/s): min= 160, max= 1536, per=27.08%, avg=893.21, stdev=319.35, samples=14 iops : min= 10, max= 96, avg=55.57, stdev=20.09, samples=14 lat (usec) : 100=4.43%, 250=10.81%, 500=0.65%, 750=0.26%, 1000=2.86% lat (msec) : 2=16.15%, 4=8.85%, 10=20.31%, 20=23.44%, 50=11.20% lat (msec) : 100=0.91%, 250=0.13% cpu : usr=0.17%, sys=14.82%, ctx=1160, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4975: Thu May 1 02:59:35 2025 read: IOPS=47, BW=752KiB/s (770kB/s)(5744KiB/7637msec) clat (usec): min=67, max=64175, avg=8044.21, stdev=8744.43 lat (usec): min=68, max=64175, avg=8045.29, stdev=8744.67 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 94], 10.00th=[ 101], 20.00th=[ 116], | 30.00th=[ 141], 40.00th=[ 4686], 50.00th=[ 7046], 60.00th=[ 9110], | 70.00th=[11207], 80.00th=[14746], 90.00th=[18482], 95.00th=[22938], | 99.00th=[38536], 99.50th=[57934], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 63, max= 1568, per=23.91%, avg=749.93, stdev=419.30, samples=15 iops : min= 3, max= 98, avg=46.53, stdev=26.22, samples=15 write: IOPS=53, BW=857KiB/s (877kB/s)(6544KiB/7637msec) clat (usec): min=726, max=113181, avg=11183.24, stdev=16986.11 lat (usec): min=727, max=113184, avg=11186.97, stdev=16985.08 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1004], 10.00th=[ 1123], 20.00th=[ 1270], | 30.00th=[ 1745], 40.00th=[ 2409], 50.00th=[ 4490], 60.00th=[ 7832], | 70.00th=[ 12256], 80.00th=[ 18482], 90.00th=[ 25297], 95.00th=[ 36439], | 99.00th=[102237], 99.50th=[108528], 99.90th=[112722], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 158, max= 1440, per=25.85%, avg=852.67, stdev=420.40, samples=15 iops : min= 9, max= 90, avg=52.93, stdev=26.35, samples=15 lat (usec) : 100=4.17%, 250=11.72%, 500=0.65%, 750=0.78%, 1000=2.47% lat (msec) : 2=16.02%, 4=8.98%, 10=20.31%, 20=22.14%, 50=10.81% lat (msec) : 100=1.17%, 250=0.78% cpu : usr=0.16%, sys=14.03%, ctx=1129, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4976: Thu May 1 02:59:35 2025 read: IOPS=57, BW=918KiB/s (940kB/s)(6256KiB/6813msec) clat (usec): min=59, max=55883, avg=8193.07, stdev=9083.31 lat (usec): min=59, max=55883, avg=8200.08, stdev=9078.99 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 92], 10.00th=[ 101], 20.00th=[ 118], | 30.00th=[ 141], 40.00th=[ 1254], 50.00th=[ 6390], 60.00th=[ 8848], | 70.00th=[11600], 80.00th=[14877], 90.00th=[20317], 95.00th=[24511], | 99.00th=[39584], 99.50th=[45876], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 351, max= 1280, per=29.58%, avg=927.54, stdev=296.43, samples=13 iops : min= 21, max= 80, avg=57.54, stdev=18.68, samples=13 write: IOPS=55, BW=885KiB/s (907kB/s)(6032KiB/6813msec) clat (usec): min=726, max=97223, avg=9286.27, stdev=12576.21 lat (usec): min=726, max=97225, avg=9294.09, stdev=12572.65 clat percentiles (usec): | 1.00th=[ 824], 5.00th=[ 955], 10.00th=[ 1037], 20.00th=[ 1303], | 30.00th=[ 1713], 40.00th=[ 2638], 50.00th=[ 3687], 60.00th=[ 6915], | 70.00th=[10421], 80.00th=[16319], 90.00th=[24249], 95.00th=[30802], | 99.00th=[77071], 99.50th=[89654], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 351, max= 1216, per=26.31%, avg=868.08, stdev=229.27, samples=13 iops : min= 21, max= 76, avg=53.85, stdev=14.36, samples=13 lat (usec) : 100=4.69%, 250=13.41%, 500=1.17%, 750=0.78%, 1000=3.65% lat (msec) : 2=13.67%, 4=8.72%, 10=20.57%, 20=20.70%, 50=11.72% lat (msec) : 100=0.91% cpu : usr=0.28%, sys=15.16%, ctx=1123, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4977: Thu May 1 02:59:35 2025 read: IOPS=52, BW=838KiB/s (859kB/s)(6032KiB/7194msec) clat (usec): min=66, max=57651, avg=8327.11, stdev=8720.92 lat (usec): min=66, max=57651, avg=8328.15, stdev=8720.94 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 90], 10.00th=[ 99], 20.00th=[ 115], | 30.00th=[ 133], 40.00th=[ 5407], 50.00th=[ 7570], 60.00th=[ 9110], | 70.00th=[11469], 80.00th=[14615], 90.00th=[19792], 95.00th=[23725], | 99.00th=[38011], 99.50th=[55313], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 224, max= 1309, per=26.77%, avg=839.43, stdev=273.53, samples=14 iops : min= 14, max= 81, avg=52.00, stdev=17.04, samples=14 write: IOPS=54, BW=870KiB/s (890kB/s)(6256KiB/7194msec) clat (usec): min=669, max=127900, avg=9990.21, stdev=12980.03 lat (usec): min=670, max=127902, avg=9991.76, stdev=12980.10 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 988], 10.00th=[ 1074], 20.00th=[ 1287], | 30.00th=[ 1713], 40.00th=[ 2474], 50.00th=[ 4293], 60.00th=[ 7898], | 70.00th=[ 12649], 80.00th=[ 17171], 90.00th=[ 25560], 95.00th=[ 34341], | 99.00th=[ 57410], 99.50th=[ 81265], 99.90th=[127402], 99.95th=[127402], | 99.99th=[127402] bw ( KiB/s): min= 288, max= 1628, per=26.40%, avg=871.07, stdev=342.98, samples=14 iops : min= 18, max= 101, avg=54.00, stdev=21.31, samples=14 lat (usec) : 100=5.21%, 250=11.98%, 500=0.52%, 750=0.78%, 1000=2.86% lat (msec) : 2=14.71%, 4=7.16%, 10=21.09%, 20=23.05%, 50=11.85% lat (msec) : 100=0.65%, 250=0.13% cpu : usr=0.24%, sys=14.29%, ctx=1119, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3136KiB/s (3212kB/s), 752KiB/s-918KiB/s (770kB/s-940kB/s), io=23.4MiB (24.5MB), run=6813-7637msec WRITE: bw=3300KiB/s (3379kB/s), 857KiB/s-892KiB/s (877kB/s-914kB/s), io=24.6MiB (25.8MB), run=6813-7637msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4969: Thu May 1 03:00:01 2025 read: IOPS=11, BW=178KiB/s (183kB/s)(5920KiB/33179msec) clat (msec): min=3, max=105, avg=12.88, stdev=10.75 lat (msec): min=3, max=105, avg=12.89, stdev=10.75 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 17], 90.00th=[ 23], 95.00th=[ 33], | 99.00th=[ 63], 99.50th=[ 72], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 32, max= 574, per=24.97%, avg=178.03, stdev=94.12, samples=66 iops : min= 2, max= 35, avg=10.77, stdev= 5.82, samples=66 write: IOPS=11, BW=192KiB/s (197kB/s)(6368KiB/33179msec) clat (msec): min=33, max=201, avg=71.29, stdev=23.28 lat (msec): min=33, max=201, avg=71.30, stdev=23.28 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 72], | 70.00th=[ 79], 80.00th=[ 88], 90.00th=[ 102], 95.00th=[ 120], | 99.00th=[ 142], 99.50th=[ 161], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 63, max= 288, per=25.48%, avg=191.12, stdev=44.39, samples=66 iops : min= 3, max= 18, avg=11.59, stdev= 2.84, samples=66 lat (msec) : 4=0.26%, 10=25.00%, 20=16.02%, 50=14.84%, 100=38.02% lat (msec) : 250=5.86% cpu : usr=0.04%, sys=2.90%, ctx=937, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4970: Thu May 1 03:00:01 2025 read: IOPS=10, BW=171KiB/s (175kB/s)(5744KiB/33567msec) clat (usec): min=3137, max=89608, avg=12695.79, stdev=10114.67 lat (usec): min=3139, max=89609, avg=12698.80, stdev=10114.22 clat percentiles (usec): | 1.00th=[ 4555], 5.00th=[ 5211], 10.00th=[ 5735], 20.00th=[ 6718], | 30.00th=[ 7439], 40.00th=[ 8455], 50.00th=[ 9765], 60.00th=[10945], | 70.00th=[12649], 80.00th=[15795], 90.00th=[23200], 95.00th=[33162], | 99.00th=[58983], 99.50th=[64750], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 31, max= 480, per=23.87%, avg=170.18, stdev=93.51, samples=66 iops : min= 1, max= 30, avg=10.23, stdev= 5.92, samples=66 write: IOPS=12, BW=195KiB/s (200kB/s)(6544KiB/33567msec) clat (msec): min=31, max=218, avg=70.74, stdev=24.40 lat (msec): min=31, max=218, avg=70.74, stdev=24.40 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 57], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 80], 80.00th=[ 88], 90.00th=[ 103], 95.00th=[ 117], | 99.00th=[ 146], 99.50th=[ 153], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 64, max= 287, per=25.66%, avg=192.48, stdev=46.54, samples=66 iops : min= 4, max= 17, avg=11.62, stdev= 2.88, samples=66 lat (msec) : 4=0.13%, 10=24.22%, 20=16.67%, 50=15.23%, 100=37.11% lat (msec) : 250=6.64% cpu : usr=0.08%, sys=3.02%, ctx=948, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4971: Thu May 1 03:00:01 2025 read: IOPS=11, BW=191KiB/s (196kB/s)(6256KiB/32689msec) clat (usec): min=3864, max=79595, avg=11873.21, stdev=8800.88 lat (usec): min=3867, max=79596, avg=11874.64, stdev=8800.80 clat percentiles (usec): | 1.00th=[ 4047], 5.00th=[ 4752], 10.00th=[ 5538], 20.00th=[ 6390], | 30.00th=[ 7373], 40.00th=[ 8291], 50.00th=[ 9241], 60.00th=[10683], | 70.00th=[12649], 80.00th=[14746], 90.00th=[20055], 95.00th=[27395], | 99.00th=[47973], 99.50th=[74974], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 32, max= 447, per=27.34%, avg=194.97, stdev=102.52, samples=64 iops : min= 2, max= 27, avg=11.78, stdev= 6.36, samples=64 write: IOPS=11, BW=185KiB/s (189kB/s)(6032KiB/32689msec) clat (msec): min=33, max=221, avg=74.15, stdev=24.00 lat (msec): min=33, max=221, avg=74.15, stdev=24.00 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 55], | 30.00th=[ 61], 40.00th=[ 66], 50.00th=[ 71], 60.00th=[ 78], | 70.00th=[ 83], 80.00th=[ 91], 90.00th=[ 107], 95.00th=[ 117], | 99.00th=[ 142], 99.50th=[ 174], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 96, max= 256, per=24.55%, avg=184.12, stdev=36.53, samples=65 iops : min= 6, max= 16, avg=11.11, stdev= 2.31, samples=65 lat (msec) : 4=0.39%, 10=27.73%, 20=17.58%, 50=11.98%, 100=35.55% lat (msec) : 250=6.77% cpu : usr=0.00%, sys=3.08%, ctx=973, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4972: Thu May 1 03:00:01 2025 read: IOPS=11, BW=183KiB/s (187kB/s)(6032KiB/32992msec) clat (usec): min=3887, max=83988, avg=12702.41, stdev=10686.03 lat (usec): min=3888, max=83990, avg=12704.03, stdev=10685.99 clat percentiles (usec): | 1.00th=[ 4359], 5.00th=[ 5145], 10.00th=[ 5669], 20.00th=[ 6521], | 30.00th=[ 7439], 40.00th=[ 8094], 50.00th=[ 9241], 60.00th=[10945], | 70.00th=[13566], 80.00th=[16909], 90.00th=[21103], 95.00th=[28181], | 99.00th=[78119], 99.50th=[82314], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 31, max= 480, per=26.24%, avg=187.11, stdev=96.16, samples=63 iops : min= 1, max= 30, avg=11.44, stdev= 6.04, samples=63 write: IOPS=11, BW=190KiB/s (194kB/s)(6256KiB/32992msec) clat (msec): min=31, max=156, avg=71.98, stdev=22.67 lat (msec): min=31, max=156, avg=71.99, stdev=22.68 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 64], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 82], 80.00th=[ 89], 90.00th=[ 102], 95.00th=[ 118], | 99.00th=[ 140], 99.50th=[ 153], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 32, max= 288, per=25.23%, avg=189.25, stdev=44.68, samples=65 iops : min= 2, max= 18, avg=11.58, stdev= 2.86, samples=65 lat (msec) : 4=0.13%, 10=26.30%, 20=17.19%, 50=13.15%, 100=37.50% lat (msec) : 250=5.73% cpu : usr=0.08%, sys=3.06%, ctx=956, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=714KiB/s (731kB/s), 171KiB/s-191KiB/s (175kB/s-196kB/s), io=23.4MiB (24.5MB), run=32689-33567msec WRITE: bw=751KiB/s (769kB/s), 185KiB/s-195KiB/s (189kB/s-200kB/s), io=24.6MiB (25.8MB), run=32689-33567msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5010: Thu May 1 03:00:04 2025 read: IOPS=9, BW=9806KiB/s (10.0MB/s)(7168KiB/731msec) clat (usec): min=537, max=98315, avg=50208.14, stdev=38530.39 lat (usec): min=538, max=98317, avg=50210.15, stdev=38530.73 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[ 660], | 30.00th=[43779], 40.00th=[43779], 50.00th=[50594], 60.00th=[78119], | 70.00th=[78119], 80.00th=[80217], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 8159, max= 8159, per=94.15%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7004KiB/s (7172kB/s)(5120KiB/731msec) clat (usec): min=36194, max=69582, avg=50614.16, stdev=15034.37 lat (usec): min=36291, max=69665, avg=50717.36, stdev=15022.65 clat percentiles (usec): | 1.00th=[36439], 5.00th=[36439], 10.00th=[36439], 20.00th=[36439], | 30.00th=[36439], 40.00th=[36439], 50.00th=[49021], 60.00th=[49021], | 70.00th=[62129], 80.00th=[62129], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 2039, max= 2039, per=16.81%, avg=2039.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=33.33%, 100=50.00% cpu : usr=0.00%, sys=27.26%, ctx=200, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5011: Thu May 1 03:00:04 2025 read: IOPS=0, BW=867KiB/s (887kB/s)(2048KiB/2363msec) clat (msec): min=47, max=105, avg=76.41, stdev=41.40 lat (msec): min=47, max=105, avg=76.41, stdev=41.41 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 1868, max= 2043, per=22.57%, avg=1955.50, stdev=123.74, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4333KiB/s (4437kB/s)(10.0MiB/2363msec) clat (msec): min=47, max=1111, avg=212.88, stdev=319.49 lat (msec): min=47, max=1111, avg=212.98, stdev=319.49 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 63], | 30.00th=[ 66], 40.00th=[ 79], 50.00th=[ 115], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 188], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 3737, max= 7474, per=42.03%, avg=5099.33, stdev=2063.95, samples=3 iops : min= 3, max= 7, avg= 4.33, stdev= 2.31, samples=3 lat (msec) : 50=16.67%, 100=25.00%, 250=50.00% cpu : usr=0.00%, sys=11.85%, ctx=398, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5013: Thu May 1 03:00:04 2025 read: IOPS=7, BW=7623KiB/s (7806kB/s)(6144KiB/806msec) clat (usec): min=425, max=81739, avg=33140.29, stdev=36798.51 lat (usec): min=426, max=81741, avg=33141.87, stdev=36798.54 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 545], | 30.00th=[ 545], 40.00th=[ 8160], 50.00th=[ 8160], 60.00th=[34341], | 70.00th=[73925], 80.00th=[73925], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] write: IOPS=7, BW=7623KiB/s (7806kB/s)(6144KiB/806msec) clat (msec): min=33, max=260, avg=92.64, stdev=86.55 lat (msec): min=35, max=260, avg=93.14, stdev=86.21 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 86], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 6131, max= 6131, per=50.53%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 10=8.33%, 50=33.33%, 100=33.33%, 500=8.33% cpu : usr=0.00%, sys=18.76%, ctx=192, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5016: Thu May 1 03:00:04 2025 read: IOPS=2, BW=2343KiB/s (2399kB/s)(5120KiB/2185msec) clat (usec): min=367, max=1017.7k, avg=207322.86, stdev=453098.07 lat (usec): min=367, max=1017.7k, avg=207324.23, stdev=453098.31 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], | 20.00th=[ 367], 30.00th=[ 437], 40.00th=[ 437], | 50.00th=[ 611], 60.00th=[ 611], 70.00th=[ 17433], | 80.00th=[ 17433], 90.00th=[1019216], 95.00th=[1019216], | 99.00th=[1019216], 99.50th=[1019216], 99.90th=[1019216], | 99.95th=[1019216], 99.99th=[1019216] bw ( KiB/s): min= 8192, max= 8192, per=94.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3281KiB/s (3359kB/s)(7168KiB/2185msec) clat (msec): min=32, max=343, avg=138.03, stdev=104.04 lat (msec): min=32, max=343, avg=138.10, stdev=104.06 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 34], | 30.00th=[ 127], 40.00th=[ 127], 50.00th=[ 134], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 161], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 2043, max= 8192, per=39.37%, avg=4777.00, stdev=3130.55, samples=3 iops : min= 1, max= 8, avg= 4.33, stdev= 3.51, samples=3 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 20=8.33%, 50=16.67%, 250=33.33%, 500=8.33% cpu : usr=0.14%, sys=13.28%, ctx=344, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8667KiB/s (8875kB/s), 867KiB/s-9806KiB/s (887kB/s-10.0MB/s), io=20.0MiB (20.0MB), run=731-2363msec WRITE: bw=11.8MiB/s (12.4MB/s), 3281KiB/s-7623KiB/s (3359kB/s-7806kB/s), io=28.0MiB (29.4MB), run=731-2363msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5012: Thu May 1 03:00:04 2025 read: IOPS=2, BW=2907KiB/s (2976kB/s)(7168KiB/2466msec) clat (msec): min=20, max=375, avg=122.36, stdev=123.12 lat (msec): min=20, max=375, avg=122.36, stdev=123.12 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 35], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 102], 60.00th=[ 111], | 70.00th=[ 111], 80.00th=[ 169], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 2048, max= 6144, per=41.10%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=2, BW=2076KiB/s (2126kB/s)(5120KiB/2466msec) clat (msec): min=60, max=1173, avg=291.44, stdev=493.00 lat (msec): min=60, max=1173, avg=291.53, stdev=492.98 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 72], 60.00th=[ 72], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 2048, max= 4096, per=26.42%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.08%, sys=3.65%, ctx=44, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5014: Thu May 1 03:00:04 2025 read: IOPS=0, BW=876KiB/s (897kB/s)(2048KiB/2337msec) clat (msec): min=44, max=110, avg=77.20, stdev=46.44 lat (msec): min=44, max=110, avg=77.20, stdev=46.43 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 2048, max= 2048, per=24.66%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4382KiB/s (4487kB/s)(10.0MiB/2337msec) clat (msec): min=68, max=1109, avg=214.88, stdev=316.71 lat (msec): min=68, max=1109, avg=215.36, stdev=316.54 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 77], | 30.00th=[ 90], 40.00th=[ 101], 50.00th=[ 103], 60.00th=[ 106], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 192], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 4096, max= 6144, per=46.98%, avg=5461.33, stdev=1182.41, samples=3 iops : min= 4, max= 6, avg= 5.33, stdev= 1.15, samples=3 lat (msec) : 50=8.33%, 100=25.00%, 250=58.33% cpu : usr=0.00%, sys=6.04%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5015: Thu May 1 03:00:04 2025 read: IOPS=2, BW=2558KiB/s (2619kB/s)(6144KiB/2402msec) clat (msec): min=38, max=1188, avg=272.50, stdev=451.53 lat (msec): min=38, max=1188, avg=272.50, stdev=451.53 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 93], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8175, max= 8175, per=98.45%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2558KiB/s (2619kB/s)(6144KiB/2402msec) clat (msec): min=59, max=256, avg=124.34, stdev=71.25 lat (msec): min=59, max=256, avg=124.44, stdev=71.27 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 75], | 30.00th=[ 75], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 109], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 2043, max= 6144, per=29.35%, avg=3411.67, stdev=2366.27, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=4.33%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5017: Thu May 1 03:00:04 2025 read: IOPS=2, BW=2114KiB/s (2165kB/s)(5120KiB/2422msec) clat (msec): min=35, max=131, avg=76.94, stdev=43.47 lat (msec): min=35, max=131, avg=76.94, stdev=43.47 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 65], 60.00th=[ 65], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 8192, max= 8192, per=98.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2960KiB/s (3031kB/s)(7168KiB/2422msec) clat (msec): min=87, max=1144, avg=279.30, stdev=386.86 lat (msec): min=87, max=1144, avg=279.39, stdev=386.85 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 110], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 112], 60.00th=[ 113], | 70.00th=[ 113], 80.00th=[ 279], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2048, max= 6144, per=35.23%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=16.67%, 100=16.67%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=3.72%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8305KiB/s (8504kB/s), 876KiB/s-2907KiB/s (897kB/s-2976kB/s), io=20.0MiB (20.0MB), run=2337-2466msec WRITE: bw=11.4MiB/s (11.9MB/s), 2076KiB/s-4382KiB/s (2126kB/s-4487kB/s), io=28.0MiB (29.4MB), run=2337-2466msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5026: Thu May 1 03:00:07 2025 read: IOPS=3, BW=12.7MiB/s (13.4MB/s)(12.0MiB/942msec) clat (msec): min=237, max=385, avg=302.77, stdev=75.85 lat (msec): min=237, max=385, avg=302.77, stdev=75.85 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8175, max= 8175, per=55.79%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.64%, sys=18.81%, ctx=984, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5028: Thu May 1 03:00:07 2025 read: IOPS=0, BW=3664KiB/s (3752kB/s)(4096KiB/1118msec) clat (nsec): min=225283k, max=225283k, avg=225282555.00, stdev= 0.00 lat (nsec): min=225286k, max=225286k, avg=225285533.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 226], | 30.00th=[ 226], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 8192, max= 8192, per=55.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7327KiB/s (7503kB/s)(8192KiB/1118msec) clat (msec): min=321, max=529, avg=425.24, stdev=147.01 lat (msec): min=321, max=529, avg=425.47, stdev=146.92 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 531], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.18%, sys=21.02%, ctx=166, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5030: Thu May 1 03:00:07 2025 write: IOPS=2, BW=8976KiB/s (9191kB/s)(12.0MiB/1369msec) clat (msec): min=143, max=635, avg=445.90, stdev=265.01 lat (msec): min=143, max=635, avg=446.23, stdev=265.02 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min=11130, max=11130, per=46.50%, avg=11130.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.00%, sys=16.52%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5031: Thu May 1 03:00:07 2025 write: IOPS=3, BW=13.7MiB/s (14.4MB/s)(12.0MiB/875msec) clat (msec): min=100, max=554, avg=269.18, stdev=248.26 lat (msec): min=101, max=554, avg=269.51, stdev=248.23 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 8192, max= 8192, per=34.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=26.29%, ctx=69, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.3MiB/s (15.0MB/s), 3664KiB/s-12.7MiB/s (3752kB/s-13.4MB/s), io=16.0MiB (16.8MB), run=942-1118msec WRITE: bw=23.4MiB/s (24.5MB/s), 7327KiB/s-13.7MiB/s (7503kB/s-14.4MB/s), io=32.0MiB (33.6MB), run=875-1369msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5025: Thu May 1 03:00:07 2025 read: IOPS=2, BW=9.91MiB/s (10.4MB/s)(12.0MiB/1211msec) clat (msec): min=294, max=513, avg=395.54, stdev=110.43 lat (msec): min=294, max=513, avg=395.54, stdev=110.43 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 8175, max= 8192, per=71.13%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.08%, sys=5.29%, ctx=27, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5027: Thu May 1 03:00:07 2025 read: IOPS=0, BW=2876KiB/s (2945kB/s)(4096KiB/1424msec) clat (nsec): min=358083k, max=358083k, avg=358083093.00, stdev= 0.00 lat (nsec): min=358085k, max=358085k, avg=358084970.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8175, max= 8175, per=71.06%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5753KiB/s (5891kB/s)(8192KiB/1424msec) clat (msec): min=372, max=667, avg=519.83, stdev=208.31 lat (msec): min=372, max=667, avg=520.16, stdev=208.37 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 667], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.14%, sys=7.52%, ctx=98, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5029: Thu May 1 03:00:07 2025 write: IOPS=2, BW=10.3MiB/s (10.8MB/s)(12.0MiB/1164msec) clat (msec): min=94, max=804, avg=375.76, stdev=377.31 lat (msec): min=95, max=804, avg=376.18, stdev=377.22 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min=16384, max=16384, per=71.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.91%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5032: Thu May 1 03:00:07 2025 write: IOPS=2, BW=10.5MiB/s (10.0MB/s)(12.0MiB/1145msec) clat (msec): min=206, max=600, avg=365.33, stdev=207.84 lat (msec): min=207, max=600, avg=365.81, stdev=207.67 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 207], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 8175, max= 8192, per=35.56%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.61%, sys=9.35%, ctx=107, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.2MiB/s (11.8MB/s), 2876KiB/s-9.91MiB/s (2945kB/s-10.4MB/s), io=16.0MiB (16.8MB), run=1211-1424msec WRITE: bw=22.5MiB/s (23.6MB/s), 5753KiB/s-10.5MiB/s (5891kB/s-10.0MB/s), io=32.0MiB (33.6MB), run=1145-1424msec