== sanity test 398b: DIO and buffer IO race ============== 04:16:54 (1724314614) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.25772 s, 40.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18532: Thu Aug 22 04:17:13 2024 read: IOPS=92, BW=370KiB/s (379kB/s)(5992KiB/16179msec) clat (usec): min=49, max=251296, avg=4821.94, stdev=7550.60 lat (usec): min=49, max=251298, avg=4823.33, stdev=7550.56 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 75], 10.00th=[ 95], 20.00th=[ 190], | 30.00th=[ 2835], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4621], | 70.00th=[ 5735], 80.00th=[ 7439], 90.00th=[ 9372], 95.00th=[ 11338], | 99.00th=[ 18744], 99.50th=[ 25822], 99.90th=[ 40109], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 184, max= 552, per=24.56%, avg=364.50, stdev=80.66, samples=32 iops : min= 46, max= 138, avg=90.97, stdev=20.19, samples=32 write: IOPS=97, BW=389KiB/s (398kB/s)(6296KiB/16179msec) clat (usec): min=458, max=69165, avg=5626.17, stdev=5816.55 lat (usec): min=459, max=69166, avg=5627.71, stdev=5816.14 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 644], 10.00th=[ 725], 20.00th=[ 873], | 30.00th=[ 1221], 40.00th=[ 2638], 50.00th=[ 4228], 60.00th=[ 5669], | 70.00th=[ 7111], 80.00th=[ 8979], 90.00th=[12911], 95.00th=[15926], | 99.00th=[24511], 99.50th=[31327], 99.90th=[62653], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 224, max= 600, per=24.66%, avg=383.00, stdev=79.28, samples=32 iops : min= 56, max= 150, avg=95.59, stdev=19.75, samples=32 lat (usec) : 50=0.03%, 100=5.47%, 250=4.39%, 500=0.13%, 750=6.09% lat (usec) : 1000=7.13% lat (msec) : 2=5.53%, 4=21.03%, 10=37.34%, 20=11.26%, 50=1.50% lat (msec) : 100=0.07%, 500=0.03% cpu : usr=0.15%, sys=23.14%, ctx=6325, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18533: Thu Aug 22 04:17:13 2024 read: IOPS=92, BW=369KiB/s (377kB/s)(5940KiB/16119msec) clat (usec): min=50, max=55109, avg=4790.86, stdev=4049.93 lat (usec): min=50, max=55111, avg=4792.72, stdev=4049.85 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 84], 10.00th=[ 102], 20.00th=[ 2442], | 30.00th=[ 3032], 40.00th=[ 3490], 50.00th=[ 4015], 60.00th=[ 4621], | 70.00th=[ 5735], 80.00th=[ 7373], 90.00th=[ 9503], 95.00th=[11731], | 99.00th=[16581], 99.50th=[20055], 99.90th=[38536], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 168, max= 600, per=24.65%, avg=365.84, stdev=86.74, samples=32 iops : min= 42, max= 150, avg=91.34, stdev=21.71, samples=32 write: IOPS=98, BW=394KiB/s (403kB/s)(6348KiB/16119msec) clat (usec): min=468, max=249337, avg=5610.29, stdev=8218.24 lat (usec): min=469, max=249337, avg=5611.77, stdev=8218.10 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 652], 10.00th=[ 742], 20.00th=[ 881], | 30.00th=[ 1139], 40.00th=[ 2442], 50.00th=[ 4293], 60.00th=[ 5735], | 70.00th=[ 7177], 80.00th=[ 8848], 90.00th=[ 12256], 95.00th=[ 15533], | 99.00th=[ 22938], 99.50th=[ 28443], 99.90th=[ 65799], 99.95th=[248513], | 99.99th=[248513] bw ( KiB/s): min= 224, max= 648, per=25.02%, avg=388.59, stdev=90.66, samples=32 iops : min= 56, max= 162, avg=97.03, stdev=22.66, samples=32 lat (usec) : 100=4.46%, 250=4.00%, 500=0.26%, 750=5.27%, 1000=8.50% lat (msec) : 2=6.18%, 4=20.15%, 10=38.93%, 20=11.13%, 50=1.01% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.29%, sys=23.46%, ctx=6234, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18534: Thu Aug 22 04:17:13 2024 read: IOPS=93, BW=374KiB/s (383kB/s)(6028KiB/16104msec) clat (usec): min=50, max=65222, avg=4907.77, stdev=4390.75 lat (usec): min=50, max=65223, avg=4909.45, stdev=4390.62 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 85], 10.00th=[ 104], 20.00th=[ 2540], | 30.00th=[ 3064], 40.00th=[ 3523], 50.00th=[ 4015], 60.00th=[ 4817], | 70.00th=[ 5866], 80.00th=[ 7570], 90.00th=[ 9765], 95.00th=[11207], | 99.00th=[15795], 99.50th=[26346], 99.90th=[54789], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 168, max= 544, per=25.16%, avg=373.34, stdev=85.33, samples=32 iops : min= 42, max= 136, avg=93.22, stdev=21.28, samples=32 write: IOPS=97, BW=389KiB/s (398kB/s)(6260KiB/16104msec) clat (usec): min=511, max=255480, avg=5500.10, stdev=8122.74 lat (usec): min=511, max=255481, avg=5501.96, stdev=8122.69 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 652], 10.00th=[ 734], 20.00th=[ 881], | 30.00th=[ 1188], 40.00th=[ 2638], 50.00th=[ 4293], 60.00th=[ 5538], | 70.00th=[ 6783], 80.00th=[ 8848], 90.00th=[ 12125], 95.00th=[ 15401], | 99.00th=[ 21365], 99.50th=[ 25035], 99.90th=[ 56361], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 216, max= 560, per=24.93%, avg=387.09, stdev=81.88, samples=32 iops : min= 54, max= 140, avg=96.66, stdev=20.54, samples=32 lat (usec) : 100=4.39%, 250=3.68%, 500=0.10%, 750=5.76%, 1000=7.81% lat (msec) : 2=5.83%, 4=20.67%, 10=39.42%, 20=11.26%, 50=0.94% lat (msec) : 100=0.10%, 500=0.03% cpu : usr=0.22%, sys=23.88%, ctx=6312, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18535: Thu Aug 22 04:17:13 2024 read: IOPS=94, BW=377KiB/s (386kB/s)(6052KiB/16038msec) clat (usec): min=48, max=62694, avg=4704.62, stdev=4179.25 lat (usec): min=48, max=62695, avg=4706.87, stdev=4178.86 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 79], 10.00th=[ 101], 20.00th=[ 2376], | 30.00th=[ 2933], 40.00th=[ 3490], 50.00th=[ 4015], 60.00th=[ 4686], | 70.00th=[ 5604], 80.00th=[ 7177], 90.00th=[ 9241], 95.00th=[11076], | 99.00th=[15664], 99.50th=[19792], 99.90th=[53216], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 32, max= 592, per=25.40%, avg=376.91, stdev=107.43, samples=32 iops : min= 8, max= 148, avg=94.16, stdev=26.88, samples=32 write: IOPS=97, BW=389KiB/s (398kB/s)(6236KiB/16038msec) clat (usec): min=462, max=253817, avg=5656.94, stdev=8523.85 lat (usec): min=462, max=253818, avg=5658.20, stdev=8523.73 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 652], 10.00th=[ 742], 20.00th=[ 881], | 30.00th=[ 1237], 40.00th=[ 2671], 50.00th=[ 4359], 60.00th=[ 5473], | 70.00th=[ 6783], 80.00th=[ 8717], 90.00th=[ 12387], 95.00th=[ 15926], | 99.00th=[ 26870], 99.50th=[ 31327], 99.90th=[ 70779], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 48, max= 600, per=24.99%, avg=388.16, stdev=99.25, samples=32 iops : min= 12, max= 150, avg=96.97, stdev=24.80, samples=32 lat (usec) : 50=0.10%, 100=4.69%, 250=3.87%, 500=0.33%, 750=5.40% lat (usec) : 1000=7.75% lat (msec) : 2=5.83%, 4=20.18%, 10=40.14%, 20=10.25%, 50=1.30% lat (msec) : 100=0.13%, 500=0.03% cpu : usr=0.25%, sys=23.19%, ctx=6260, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1484KiB/s (1520kB/s), 369KiB/s-377KiB/s (377kB/s-386kB/s), io=23.4MiB (24.6MB), run=16038-16179msec WRITE: bw=1554KiB/s (1591kB/s), 389KiB/s-394KiB/s (398kB/s-403kB/s), io=24.6MiB (25.7MB), run=16038-16179msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18528: Thu Aug 22 04:17:38 2024 read: IOPS=35, BW=143KiB/s (147kB/s)(5992KiB/41771msec) clat (usec): min=1367, max=270032, avg=4757.26, stdev=7806.20 lat (usec): min=1367, max=270033, avg=4758.75, stdev=7806.22 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2442], | 30.00th=[ 2671], 40.00th=[ 2900], 50.00th=[ 3163], 60.00th=[ 3654], | 70.00th=[ 4293], 80.00th=[ 5735], 90.00th=[ 9241], 95.00th=[ 12387], | 99.00th=[ 18744], 99.50th=[ 23462], 99.90th=[ 46400], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 48, max= 272, per=24.91%, avg=142.75, stdev=58.33, samples=83 iops : min= 12, max= 68, avg=35.61, stdev=14.62, samples=83 write: IOPS=37, BW=151KiB/s (154kB/s)(6296KiB/41771msec) clat (usec): min=8832, max=68311, avg=21940.26, stdev=8350.78 lat (usec): min=8832, max=68312, avg=21941.61, stdev=8350.88 clat percentiles (usec): | 1.00th=[10814], 5.00th=[12649], 10.00th=[13435], 20.00th=[15008], | 30.00th=[16450], 40.00th=[18482], 50.00th=[20055], 60.00th=[21890], | 70.00th=[23987], 80.00th=[27395], 90.00th=[33817], 95.00th=[39060], | 99.00th=[48497], 99.50th=[53740], 99.90th=[62653], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 40, max= 240, per=24.96%, avg=149.78, stdev=46.40, samples=83 iops : min= 10, max= 60, avg=37.37, stdev=11.59, samples=83 lat (msec) : 2=1.66%, 4=30.73%, 10=12.24%, 20=29.33%, 50=25.55% lat (msec) : 100=0.46%, 500=0.03% cpu : usr=0.13%, sys=5.65%, ctx=3735, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18529: Thu Aug 22 04:17:38 2024 read: IOPS=35, BW=142KiB/s (145kB/s)(5940KiB/41862msec) clat (usec): min=1425, max=253128, avg=4810.82, stdev=7585.75 lat (usec): min=1426, max=253129, avg=4812.08, stdev=7585.77 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2442], | 30.00th=[ 2638], 40.00th=[ 2868], 50.00th=[ 3228], 60.00th=[ 3654], | 70.00th=[ 4228], 80.00th=[ 5735], 90.00th=[ 9110], 95.00th=[ 12780], | 99.00th=[ 21890], 99.50th=[ 23987], 99.90th=[ 54264], 99.95th=[252707], | 99.99th=[252707] bw ( KiB/s): min= 16, max= 296, per=24.69%, avg=141.49, stdev=60.17, samples=83 iops : min= 4, max= 74, avg=35.30, stdev=15.05, samples=83 write: IOPS=37, BW=152KiB/s (155kB/s)(6348KiB/41862msec) clat (usec): min=8136, max=80952, avg=21806.86, stdev=8680.87 lat (usec): min=8137, max=80953, avg=21808.30, stdev=8680.90 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12649], 10.00th=[13435], 20.00th=[14746], | 30.00th=[16188], 40.00th=[18220], 50.00th=[20055], 60.00th=[21627], | 70.00th=[23462], 80.00th=[27132], 90.00th=[33162], 95.00th=[38536], | 99.00th=[50594], 99.50th=[59507], 99.90th=[73925], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 56, max= 224, per=25.09%, avg=150.55, stdev=47.53, samples=83 iops : min= 14, max= 56, avg=37.57, stdev=11.87, samples=83 lat (msec) : 2=1.76%, 4=30.18%, 10=12.30%, 20=29.10%, 50=25.98% lat (msec) : 100=0.65%, 500=0.03% cpu : usr=0.17%, sys=5.60%, ctx=3770, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18530: Thu Aug 22 04:17:38 2024 read: IOPS=36, BW=145KiB/s (148kB/s)(6028KiB/41655msec) clat (usec): min=1447, max=56077, avg=4761.71, stdev=4378.37 lat (usec): min=1447, max=56078, avg=4762.79, stdev=4378.37 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 2024], 10.00th=[ 2212], 20.00th=[ 2442], | 30.00th=[ 2638], 40.00th=[ 2900], 50.00th=[ 3228], 60.00th=[ 3654], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 9372], 95.00th=[12518], | 99.00th=[20055], 99.50th=[29492], 99.90th=[46400], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 40, max= 248, per=25.21%, avg=144.48, stdev=56.85, samples=83 iops : min= 10, max= 62, avg=36.05, stdev=14.18, samples=83 write: IOPS=37, BW=150KiB/s (154kB/s)(6260KiB/41655msec) clat (msec): min=9, max=254, avg=21.96, stdev=10.26 lat (msec): min=9, max=254, avg=21.96, stdev=10.26 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 28], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 50], 99.50th=[ 54], 99.90th=[ 82], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 56, max= 232, per=24.96%, avg=149.78, stdev=48.11, samples=83 iops : min= 14, max= 58, avg=37.37, stdev=12.03, samples=83 lat (msec) : 2=2.25%, 4=29.88%, 10=12.83%, 20=29.10%, 50=25.59% lat (msec) : 100=0.33%, 500=0.03% cpu : usr=0.13%, sys=5.70%, ctx=3736, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18531: Thu Aug 22 04:17:38 2024 read: IOPS=36, BW=145KiB/s (149kB/s)(6052KiB/41627msec) clat (usec): min=1542, max=54241, avg=4690.57, stdev=3990.44 lat (usec): min=1542, max=54242, avg=4691.74, stdev=3990.81 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2442], | 30.00th=[ 2638], 40.00th=[ 2868], 50.00th=[ 3195], 60.00th=[ 3687], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 9372], 95.00th=[12649], | 99.00th=[19792], 99.50th=[21627], 99.90th=[48497], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 32, max= 280, per=25.36%, avg=145.30, stdev=58.28, samples=83 iops : min= 8, max= 70, avg=36.29, stdev=14.58, samples=83 write: IOPS=37, BW=150KiB/s (153kB/s)(6236KiB/41627msec) clat (msec): min=8, max=254, avg=22.08, stdev=10.15 lat (msec): min=8, max=254, avg=22.08, stdev=10.15 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 25], 80.00th=[ 28], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 50], 99.50th=[ 55], 99.90th=[ 63], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 48, max= 224, per=24.91%, avg=149.45, stdev=46.63, samples=83 iops : min= 12, max= 56, avg=37.33, stdev=11.68, samples=83 lat (msec) : 2=1.56%, 4=31.15%, 10=12.30%, 20=29.17%, 50=25.36% lat (msec) : 100=0.42%, 500=0.03% cpu : usr=0.13%, sys=5.69%, ctx=3745, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=574KiB/s (587kB/s), 142KiB/s-145KiB/s (145kB/s-149kB/s), io=23.4MiB (24.6MB), run=41627-41862msec WRITE: bw=601KiB/s (615kB/s), 150KiB/s-152KiB/s (153kB/s-155kB/s), io=24.6MiB (25.7MB), run=41627-41862msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18576: Thu Aug 22 04:17:44 2024 read: IOPS=74, BW=1186KiB/s (1214kB/s)(5920KiB/4993msec) clat (usec): min=45, max=35446, avg=9488.89, stdev=5859.45 lat (usec): min=45, max=35446, avg=9490.10, stdev=5859.16 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 72], 10.00th=[ 86], 20.00th=[ 125], | 30.00th=[ 9110], 40.00th=[ 9896], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12649], 80.00th=[13829], 90.00th=[15401], 95.00th=[17433], | 99.00th=[20055], 99.50th=[23987], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 896, max= 1440, per=23.99%, avg=1141.33, stdev=152.63, samples=9 iops : min= 56, max= 90, avg=71.33, stdev= 9.54, samples=9 write: IOPS=79, BW=1275KiB/s (1306kB/s)(6368KiB/4993msec) clat (usec): min=537, max=55230, avg=3647.79, stdev=4592.09 lat (usec): min=537, max=55232, avg=3649.28, stdev=4592.17 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 734], 10.00th=[ 832], 20.00th=[ 938], | 30.00th=[ 1057], 40.00th=[ 1205], 50.00th=[ 2073], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 6063], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[21890], 99.50th=[38536], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 768, max= 1632, per=24.93%, avg=1248.00, stdev=293.72, samples=9 iops : min= 48, max= 102, avg=78.00, stdev=18.36, samples=9 lat (usec) : 50=0.39%, 100=7.55%, 250=2.86%, 500=0.13%, 750=2.86% lat (usec) : 1000=10.03% lat (msec) : 2=12.76%, 4=9.11%, 10=23.83%, 20=29.56%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.22%, sys=22.50%, ctx=2229, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18577: Thu Aug 22 04:17:44 2024 read: IOPS=71, BW=1141KiB/s (1169kB/s)(5744KiB/5033msec) clat (usec): min=45, max=63330, avg=10138.71, stdev=6211.20 lat (usec): min=45, max=63331, avg=10139.83, stdev=6210.70 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 78], 10.00th=[ 89], 20.00th=[ 7308], | 30.00th=[ 9372], 40.00th=[10290], 50.00th=[11207], 60.00th=[11863], | 70.00th=[12911], 80.00th=[13698], 90.00th=[15139], 95.00th=[17433], | 99.00th=[25035], 99.50th=[26346], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 800, max= 1408, per=23.79%, avg=1132.00, stdev=213.18, samples=10 iops : min= 50, max= 88, avg=70.50, stdev=13.45, samples=10 write: IOPS=81, BW=1300KiB/s (1331kB/s)(6544KiB/5033msec) clat (usec): min=595, max=27866, avg=3373.44, stdev=3618.35 lat (usec): min=595, max=27868, avg=3374.74, stdev=3618.41 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 857], 20.00th=[ 971], | 30.00th=[ 1074], 40.00th=[ 1172], 50.00th=[ 1696], 60.00th=[ 3032], | 70.00th=[ 3884], 80.00th=[ 5276], 90.00th=[ 7635], 95.00th=[10159], | 99.00th=[16450], 99.50th=[19268], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 960, max= 1792, per=25.74%, avg=1288.70, stdev=263.05, samples=10 iops : min= 60, max= 112, avg=80.30, stdev=16.57, samples=10 lat (usec) : 50=0.26%, 100=5.73%, 250=2.73%, 500=0.13%, 750=2.86% lat (usec) : 1000=9.51% lat (msec) : 2=15.10%, 4=10.68%, 10=20.83%, 20=30.73%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=23.73%, ctx=2263, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18578: Thu Aug 22 04:17:44 2024 read: IOPS=78, BW=1263KiB/s (1293kB/s)(6256KiB/4953msec) clat (usec): min=46, max=25560, avg=9414.32, stdev=5694.14 lat (usec): min=46, max=25560, avg=9415.03, stdev=5694.16 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 68], 10.00th=[ 85], 20.00th=[ 116], | 30.00th=[ 8717], 40.00th=[ 9896], 50.00th=[10945], 60.00th=[11863], | 70.00th=[12780], 80.00th=[13566], 90.00th=[15139], 95.00th=[16909], | 99.00th=[20579], 99.50th=[22938], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 894, max= 1472, per=24.65%, avg=1172.78, stdev=194.45, samples=9 iops : min= 55, max= 92, avg=73.11, stdev=12.34, samples=9 write: IOPS=76, BW=1218KiB/s (1247kB/s)(6032KiB/4953msec) clat (usec): min=552, max=70405, avg=3258.37, stdev=4785.04 lat (usec): min=553, max=70406, avg=3259.78, stdev=4784.96 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 742], 10.00th=[ 807], 20.00th=[ 930], | 30.00th=[ 1029], 40.00th=[ 1123], 50.00th=[ 1287], 60.00th=[ 2999], | 70.00th=[ 3621], 80.00th=[ 4817], 90.00th=[ 6915], 95.00th=[ 9503], | 99.00th=[18482], 99.50th=[23725], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 704, max= 1664, per=23.29%, avg=1165.67, stdev=302.46, samples=9 iops : min= 44, max= 104, avg=72.67, stdev=18.97, samples=9 lat (usec) : 50=0.65%, 100=7.29%, 250=3.52%, 500=0.13%, 750=3.39% lat (usec) : 1000=10.68% lat (msec) : 2=12.50%, 4=10.03%, 10=20.05%, 20=30.73%, 50=0.91% lat (msec) : 100=0.13% cpu : usr=0.36%, sys=22.64%, ctx=2183, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18579: Thu Aug 22 04:17:44 2024 read: IOPS=75, BW=1202KiB/s (1231kB/s)(6032KiB/5017msec) clat (usec): min=46, max=35023, avg=9388.51, stdev=6017.10 lat (usec): min=46, max=35024, avg=9389.31, stdev=6017.14 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 72], 10.00th=[ 86], 20.00th=[ 115], | 30.00th=[ 8455], 40.00th=[ 9765], 50.00th=[10552], 60.00th=[11600], | 70.00th=[12780], 80.00th=[13960], 90.00th=[15795], 95.00th=[17695], | 99.00th=[22938], 99.50th=[25297], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 958, max= 1536, per=25.07%, avg=1192.90, stdev=227.93, samples=10 iops : min= 59, max= 96, avg=74.30, stdev=14.49, samples=10 write: IOPS=77, BW=1247KiB/s (1277kB/s)(6256KiB/5017msec) clat (usec): min=583, max=74450, avg=3678.60, stdev=5133.76 lat (usec): min=584, max=74451, avg=3680.20, stdev=5133.77 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 799], 10.00th=[ 865], 20.00th=[ 1004], | 30.00th=[ 1123], 40.00th=[ 1254], 50.00th=[ 2212], 60.00th=[ 3130], | 70.00th=[ 3982], 80.00th=[ 5669], 90.00th=[ 7373], 95.00th=[10421], | 99.00th=[20841], 99.50th=[25297], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 896, max= 1728, per=24.97%, avg=1250.20, stdev=296.07, samples=10 iops : min= 56, max= 108, avg=77.90, stdev=18.52, samples=10 lat (usec) : 50=0.26%, 100=7.03%, 250=4.17%, 500=0.13%, 750=1.43% lat (usec) : 1000=8.59% lat (msec) : 2=14.84%, 4=11.07%, 10=22.27%, 20=29.04%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.26%, sys=22.68%, ctx=2160, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4759KiB/s (4873kB/s), 1141KiB/s-1263KiB/s (1169kB/s-1293kB/s), io=23.4MiB (24.5MB), run=4953-5033msec WRITE: bw=5007KiB/s (5127kB/s), 1218KiB/s-1300KiB/s (1247kB/s-1331kB/s), io=24.6MiB (25.8MB), run=4953-5033msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18580: Thu Aug 22 04:17:49 2024 read: IOPS=39, BW=639KiB/s (655kB/s)(5920KiB/9258msec) clat (usec): min=1857, max=62832, avg=4419.36, stdev=4968.48 lat (usec): min=1857, max=62833, avg=4420.26, stdev=4968.44 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2409], 40.00th=[ 2573], 50.00th=[ 2835], 60.00th=[ 3130], | 70.00th=[ 3752], 80.00th=[ 5145], 90.00th=[ 7767], 95.00th=[11600], | 99.00th=[25560], 99.50th=[35914], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 352, max= 1120, per=24.84%, avg=623.56, stdev=200.10, samples=18 iops : min= 22, max= 70, avg=38.72, stdev=12.57, samples=18 write: IOPS=42, BW=688KiB/s (704kB/s)(6368KiB/9258msec) clat (usec): min=9819, max=40518, avg=19115.53, stdev=6016.52 lat (usec): min=9819, max=40519, avg=19117.30, stdev=6016.54 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11863], 10.00th=[12518], 20.00th=[13829], | 30.00th=[15139], 40.00th=[16712], 50.00th=[17957], 60.00th=[19792], | 70.00th=[21103], 80.00th=[23200], 90.00th=[26870], 95.00th=[31851], | 99.00th=[39060], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 384, max= 928, per=25.76%, avg=680.39, stdev=163.86, samples=18 iops : min= 24, max= 58, avg=42.28, stdev=10.30, samples=18 lat (msec) : 2=1.04%, 4=33.85%, 10=10.03%, 20=35.29%, 50=19.66% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=6.34%, ctx=878, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18581: Thu Aug 22 04:17:49 2024 read: IOPS=37, BW=602KiB/s (616kB/s)(5744KiB/9541msec) clat (usec): min=1691, max=46498, avg=4341.40, stdev=4117.10 lat (usec): min=1692, max=46499, avg=4342.29, stdev=4117.09 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2606], 50.00th=[ 2769], 60.00th=[ 2999], | 70.00th=[ 3654], 80.00th=[ 5735], 90.00th=[ 8586], 95.00th=[11863], | 99.00th=[19268], 99.50th=[26084], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 288, max= 1120, per=23.82%, avg=597.79, stdev=243.44, samples=19 iops : min= 18, max= 70, avg=37.32, stdev=15.20, samples=19 write: IOPS=42, BW=686KiB/s (702kB/s)(6544KiB/9541msec) clat (usec): min=9670, max=49786, avg=19404.80, stdev=6533.32 lat (usec): min=9672, max=49788, avg=19406.70, stdev=6533.33 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11469], 10.00th=[12125], 20.00th=[13566], | 30.00th=[15270], 40.00th=[16712], 50.00th=[18482], 60.00th=[20055], | 70.00th=[21890], 80.00th=[23987], 90.00th=[27132], 95.00th=[31327], | 99.00th=[39060], 99.50th=[45351], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 480, max= 1053, per=25.89%, avg=683.63, stdev=176.93, samples=19 iops : min= 30, max= 65, avg=42.68, stdev=10.96, samples=19 lat (msec) : 2=1.30%, 4=32.94%, 10=8.72%, 20=34.64%, 50=22.40% cpu : usr=0.09%, sys=6.28%, ctx=906, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18582: Thu Aug 22 04:17:49 2024 read: IOPS=42, BW=682KiB/s (698kB/s)(6256KiB/9179msec) clat (usec): min=1839, max=20079, avg=4042.65, stdev=3046.77 lat (usec): min=1840, max=20080, avg=4043.53, stdev=3046.75 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2057], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2802], 60.00th=[ 2999], | 70.00th=[ 3523], 80.00th=[ 4948], 90.00th=[ 7701], 95.00th=[10421], | 99.00th=[18744], 99.50th=[20055], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 352, max= 1248, per=27.26%, avg=684.17, stdev=252.85, samples=18 iops : min= 22, max= 78, avg=42.61, stdev=15.89, samples=18 write: IOPS=41, BW=657KiB/s (673kB/s)(6032KiB/9179msec) clat (usec): min=9807, max=57465, avg=20040.14, stdev=6965.44 lat (usec): min=9808, max=57467, avg=20041.86, stdev=6965.41 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11731], 10.00th=[12649], 20.00th=[14484], | 30.00th=[15795], 40.00th=[17433], 50.00th=[18482], 60.00th=[20055], | 70.00th=[21890], 80.00th=[25035], 90.00th=[28967], 95.00th=[32637], | 99.00th=[44827], 99.50th=[50070], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 480, max= 928, per=24.62%, avg=650.33, stdev=159.15, samples=18 iops : min= 30, max= 58, avg=40.50, stdev=10.00, samples=18 lat (msec) : 2=1.17%, 4=36.85%, 10=9.90%, 20=32.03%, 50=19.92% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=6.46%, ctx=900, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18583: Thu Aug 22 04:17:49 2024 read: IOPS=39, BW=640KiB/s (655kB/s)(6032KiB/9428msec) clat (usec): min=1873, max=37309, avg=4588.05, stdev=4223.36 lat (usec): min=1873, max=37310, avg=4588.97, stdev=4223.35 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2073], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2704], 50.00th=[ 2868], 60.00th=[ 3294], | 70.00th=[ 3982], 80.00th=[ 5473], 90.00th=[ 9241], 95.00th=[14353], | 99.00th=[21890], 99.50th=[25822], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 351, max= 1088, per=24.56%, avg=616.44, stdev=182.13, samples=18 iops : min= 21, max= 68, avg=38.28, stdev=11.55, samples=18 write: IOPS=41, BW=664KiB/s (679kB/s)(6256KiB/9428msec) clat (usec): min=10145, max=52572, avg=19534.28, stdev=6316.99 lat (usec): min=10147, max=52574, avg=19536.14, stdev=6317.01 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11863], 10.00th=[12518], 20.00th=[13829], | 30.00th=[15270], 40.00th=[17695], 50.00th=[19006], 60.00th=[20317], | 70.00th=[21365], 80.00th=[23200], 90.00th=[27657], 95.00th=[31065], | 99.00th=[40633], 99.50th=[43254], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 447, max= 896, per=24.62%, avg=650.22, stdev=160.59, samples=18 iops : min= 27, max= 56, avg=40.39, stdev=10.28, samples=18 lat (msec) : 2=1.04%, 4=33.33%, 10=10.55%, 20=32.68%, 50=22.27% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=6.33%, ctx=924, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2510KiB/s (2571kB/s), 602KiB/s-682KiB/s (616kB/s-698kB/s), io=23.4MiB (24.5MB), run=9179-9541msec WRITE: bw=2641KiB/s (2705kB/s), 657KiB/s-688KiB/s (673kB/s-704kB/s), io=24.6MiB (25.8MB), run=9179-9541msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18608: Thu Aug 22 04:17:52 2024 read: IOPS=3, BW=3584KiB/s (3670kB/s)(7168KiB/2000msec) clat (usec): min=530, max=559574, avg=242187.87, stdev=244129.79 lat (usec): min=531, max=559576, avg=242189.17, stdev=244129.72 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 3523], | 30.00th=[ 38536], 40.00th=[ 38536], 50.00th=[179307], 60.00th=[417334], | 70.00th=[417334], 80.00th=[497026], 90.00th=[557843], 95.00th=[557843], | 99.00th=[557843], 99.50th=[557843], 99.90th=[557843], 99.95th=[557843], | 99.99th=[557843] bw ( KiB/s): min= 4096, max= 6131, per=51.26%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2560KiB/s (2621kB/s)(5120KiB/2000msec) clat (usec): min=35259, max=83801, avg=57762.69, stdev=18770.23 lat (usec): min=35276, max=83865, avg=57803.95, stdev=18786.04 clat percentiles (usec): | 1.00th=[35390], 5.00th=[35390], 10.00th=[35390], 20.00th=[35390], | 30.00th=[45876], 40.00th=[45876], 50.00th=[56361], 60.00th=[56361], | 70.00th=[67634], 80.00th=[67634], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 2043, max= 4096, per=19.54%, avg=2729.00, stdev=1183.86, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=26.35%, ctx=949, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18610: Thu Aug 22 04:17:52 2024 read: IOPS=0, BW=998KiB/s (1022kB/s)(2048KiB/2053msec) clat (msec): min=83, max=490, avg=286.65, stdev=287.59 lat (msec): min=83, max=490, avg=286.65, stdev=287.59 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 84], 50.00th=[ 84], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 2048, max= 2048, per=20.53%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4988KiB/s (5108kB/s)(10.0MiB/2053msec) clat (msec): min=28, max=431, avg=145.57, stdev=159.42 lat (msec): min=28, max=431, avg=145.66, stdev=159.41 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 33], | 30.00th=[ 39], 40.00th=[ 46], 50.00th=[ 47], 60.00th=[ 62], | 70.00th=[ 243], 80.00th=[ 243], 90.00th=[ 414], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 2048, max= 8192, per=34.22%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (msec) : 50=41.67%, 100=16.67%, 250=16.67%, 500=25.00% cpu : usr=0.00%, sys=14.52%, ctx=378, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18612: Thu Aug 22 04:17:52 2024 read: IOPS=3, BW=3956KiB/s (4051kB/s)(6144KiB/1553msec) clat (usec): min=382, max=489448, avg=117822.50, stdev=186989.93 lat (usec): min=383, max=489449, avg=117823.75, stdev=186990.01 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 383], 10.00th=[ 383], 20.00th=[ 449], | 30.00th=[ 449], 40.00th=[ 38011], 50.00th=[ 38011], 60.00th=[ 65274], | 70.00th=[113771], 80.00th=[113771], 90.00th=[488637], 95.00th=[488637], | 99.00th=[488637], 99.50th=[488637], 99.90th=[488637], 99.95th=[488637], | 99.99th=[488637] bw ( KiB/s): min= 2048, max= 4096, per=34.19%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=3, BW=3956KiB/s (4051kB/s)(6144KiB/1553msec) clat (msec): min=20, max=431, avg=139.17, stdev=169.56 lat (msec): min=20, max=431, avg=139.24, stdev=169.53 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 58], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 2048, max= 6131, per=29.30%, avg=4091.67, stdev=2041.50, samples=3 iops : min= 2, max= 5, avg= 3.67, stdev= 1.53, samples=3 lat (usec) : 500=16.67% lat (msec) : 50=33.33%, 100=16.67%, 250=8.33%, 500=25.00% cpu : usr=0.00%, sys=12.89%, ctx=376, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18614: Thu Aug 22 04:17:52 2024 read: IOPS=3, BW=3336KiB/s (3416kB/s)(5120KiB/1535msec) clat (usec): min=358, max=437273, avg=171248.33, stdev=212327.44 lat (usec): min=359, max=437274, avg=171249.13, stdev=212327.65 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 437], 40.00th=[ 437], 50.00th=[ 53740], 60.00th=[ 53740], | 70.00th=[362808], 80.00th=[362808], 90.00th=[438305], 95.00th=[438305], | 99.00th=[438305], 99.50th=[438305], 99.90th=[438305], 99.95th=[438305], | 99.99th=[438305] bw ( KiB/s): min= 8175, max= 8175, per=81.95%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4670KiB/s (4782kB/s)(7168KiB/1535msec) clat (msec): min=29, max=278, avg=93.18, stdev=85.08 lat (msec): min=29, max=278, avg=93.25, stdev=85.09 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 50], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 71], | 70.00th=[ 71], 80.00th=[ 108], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 6144, max= 8192, per=51.33%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (usec) : 500=16.67% lat (msec) : 50=16.67%, 100=33.33%, 250=8.33%, 500=25.00% cpu : usr=0.00%, sys=17.14%, ctx=579, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9976KiB/s (10.2MB/s), 998KiB/s-3956KiB/s (1022kB/s-4051kB/s), io=20.0MiB (20.0MB), run=1535-2053msec WRITE: bw=13.6MiB/s (14.3MB/s), 2560KiB/s-4988KiB/s (2621kB/s-5108kB/s), io=28.0MiB (29.4MB), run=1535-2053msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18609: Thu Aug 22 04:17:52 2024 read: IOPS=3, BW=3444KiB/s (3527kB/s)(7168KiB/2081msec) clat (msec): min=16, max=534, avg=243.59, stdev=197.46 lat (msec): min=16, max=534, avg=243.59, stdev=197.46 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 46], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 451], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 1790, max= 6144, per=30.56%, avg=3007.50, stdev=2094.53, samples=4 iops : min= 1, max= 6, avg= 2.75, stdev= 2.22, samples=4 write: IOPS=2, BW=2460KiB/s (2519kB/s)(5120KiB/2081msec) clat (msec): min=34, max=111, avg=72.38, stdev=32.00 lat (msec): min=34, max=111, avg=72.45, stdev=31.99 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 73], 60.00th=[ 73], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 1790, max= 6144, per=24.15%, avg=3327.33, stdev=2442.71, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 250=25.00%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.05%, sys=4.18%, ctx=64, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18611: Thu Aug 22 04:17:52 2024 read: IOPS=1, BW=1297KiB/s (1328kB/s)(2048KiB/1579msec) clat (usec): min=19444, max=67842, avg=43643.20, stdev=34222.32 lat (usec): min=19445, max=67844, avg=43644.89, stdev=34223.71 clat percentiles (usec): | 1.00th=[19530], 5.00th=[19530], 10.00th=[19530], 20.00th=[19530], | 30.00th=[19530], 40.00th=[19530], 50.00th=[19530], 60.00th=[67634], | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 4096, max= 4096, per=41.62%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6485KiB/s (6641kB/s)(10.0MiB/1579msec) clat (msec): min=25, max=468, avg=148.13, stdev=179.26 lat (msec): min=25, max=468, avg=148.21, stdev=179.27 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28], | 30.00th=[ 37], 40.00th=[ 43], 50.00th=[ 49], 60.00th=[ 57], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 464], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 4096, max=10240, per=44.60%, avg=6144.00, stdev=3547.24, samples=3 iops : min= 4, max= 10, avg= 6.00, stdev= 3.46, samples=3 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=8.33%, 500=16.67% cpu : usr=0.13%, sys=6.97%, ctx=59, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18613: Thu Aug 22 04:17:52 2024 read: IOPS=3, BW=3753KiB/s (3843kB/s)(6144KiB/1637msec) clat (msec): min=30, max=120, avg=61.80, stdev=32.93 lat (msec): min=30, max=120, avg=61.80, stdev=32.93 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 65], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 2048, max= 6144, per=34.68%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=3, BW=3753KiB/s (3843kB/s)(6144KiB/1637msec) clat (msec): min=48, max=454, avg=206.01, stdev=163.62 lat (msec): min=48, max=454, avg=206.09, stdev=163.62 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 199], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 2048, max= 6144, per=24.78%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (msec) : 50=41.67%, 100=16.67%, 250=25.00%, 500=16.67% cpu : usr=0.18%, sys=5.38%, ctx=228, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18615: Thu Aug 22 04:17:52 2024 read: IOPS=3, BW=3286KiB/s (3365kB/s)(5120KiB/1558msec) clat (msec): min=17, max=437, avg=103.21, stdev=186.99 lat (msec): min=17, max=437, avg=103.21, stdev=186.99 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 18], | 70.00th=[ 27], 80.00th=[ 27], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 8143, max= 8143, per=82.75%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4601KiB/s (4711kB/s)(7168KiB/1558msec) clat (msec): min=37, max=477, avg=147.70, stdev=163.21 lat (msec): min=37, max=477, avg=147.78, stdev=163.20 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 39], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 71], 60.00th=[ 118], | 70.00th=[ 118], 80.00th=[ 247], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 2043, max= 8143, per=34.56%, avg=4760.67, stdev=3103.84, samples=3 iops : min= 1, max= 7, avg= 4.00, stdev= 3.00, samples=3 lat (msec) : 20=25.00%, 50=33.33%, 100=8.33%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=5.84%, ctx=52, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9841KiB/s (10.1MB/s), 1297KiB/s-3753KiB/s (1328kB/s-3843kB/s), io=20.0MiB (20.0MB), run=1558-2081msec WRITE: bw=13.5MiB/s (14.1MB/s), 2460KiB/s-6485KiB/s (2519kB/s-6641kB/s), io=28.0MiB (29.4MB), run=1558-2081msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18625: Thu Aug 22 04:17:55 2024 read: IOPS=1, BW=4977KiB/s (5096kB/s)(12.0MiB/2469msec) clat (msec): min=194, max=1446, avg=819.00, stdev=625.91 lat (msec): min=194, max=1446, avg=819.00, stdev=625.91 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 818], 50.00th=[ 818], 60.00th=[ 818], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 7937, max= 8192, per=100.00%, avg=8064.50, stdev=180.31, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=46.47%, ctx=1123, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18627: Thu Aug 22 04:17:55 2024 read: IOPS=0, BW=1496KiB/s (1532kB/s)(4096KiB/2738msec) clat (nsec): min=188781k, max=188781k, avg=188781016.00, stdev= 0.00 lat (nsec): min=188832k, max=188832k, avg=188832249.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2992KiB/s (3064kB/s)(8192KiB/2738msec) clat (msec): min=235, max=2299, avg=1267.09, stdev=1459.54 lat (msec): min=235, max=2299, avg=1267.37, stdev=1459.50 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 2299], | 70.00th=[ 2299], 80.00th=[ 2299], 90.00th=[ 2299], 95.00th=[ 2299], | 99.00th=[ 2299], 99.50th=[ 2299], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 8192, max= 8192, per=68.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.04%, sys=8.37%, ctx=143, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18629: Thu Aug 22 04:17:55 2024 write: IOPS=1, BW=4534KiB/s (4643kB/s)(12.0MiB/2710msec) clat (msec): min=285, max=1446, avg=900.09, stdev=583.67 lat (msec): min=285, max=1446, avg=900.42, stdev=583.56 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 969], 50.00th=[ 969], 60.00th=[ 969], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 8192, max= 8192, per=68.45%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.04%, sys=8.82%, ctx=40, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18630: Thu Aug 22 04:17:55 2024 write: IOPS=1, BW=6804KiB/s (6967kB/s)(12.0MiB/1806msec) clat (msec): min=158, max=1422, avg=596.17, stdev=716.36 lat (msec): min=159, max=1423, avg=596.49, stdev=716.32 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 1418], 80.00th=[ 1418], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 8192, max= 8192, per=68.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=14.63%, ctx=594, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5984KiB/s (6128kB/s), 1496KiB/s-4977KiB/s (1532kB/s-5096kB/s), io=16.0MiB (16.8MB), run=2469-2738msec WRITE: bw=11.7MiB/s (12.3MB/s), 2992KiB/s-6804KiB/s (3064kB/s-6967kB/s), io=32.0MiB (33.6MB), run=1806-2738msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18623: Thu Aug 22 04:17:56 2024 read: IOPS=1, BW=4371KiB/s (4476kB/s)(12.0MiB/2811msec) clat (msec): min=346, max=1502, avg=922.39, stdev=578.35 lat (msec): min=346, max=1502, avg=922.39, stdev=578.35 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 919], 50.00th=[ 919], 60.00th=[ 919], | 70.00th=[ 1502], 80.00th=[ 1502], 90.00th=[ 1502], 95.00th=[ 1502], | 99.00th=[ 1502], 99.50th=[ 1502], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7393, max= 8192, per=100.00%, avg=7792.50, stdev=564.98, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=2.49%, ctx=54, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18624: Thu Aug 22 04:17:56 2024 read: IOPS=0, BW=1373KiB/s (1406kB/s)(4096KiB/2983msec) clat (nsec): min=416743k, max=416743k, avg=416743314.00, stdev= 0.00 lat (nsec): min=416746k, max=416746k, avg=416746307.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 418], 10.00th=[ 418], 20.00th=[ 418], | 30.00th=[ 418], 40.00th=[ 418], 50.00th=[ 418], 60.00th=[ 418], | 70.00th=[ 418], 80.00th=[ 418], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 7937, max= 7937, per=100.00%, avg=7937.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2746KiB/s (2812kB/s)(8192KiB/2983msec) clat (msec): min=111, max=2442, avg=1277.30, stdev=1648.18 lat (msec): min=112, max=2443, avg=1277.60, stdev=1648.24 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 2433], | 70.00th=[ 2433], 80.00th=[ 2433], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7937, max= 7937, per=73.64%, avg=7937.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.15%, ctx=58, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18626: Thu Aug 22 04:17:56 2024 write: IOPS=8, BW=32.0MiB/s (33.6MB/s)(12.0MiB/375msec) clat (msec): min=83, max=176, avg=115.68, stdev=52.45 lat (msec): min=84, max=176, avg=115.97, stdev=52.42 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 85], 10.00th=[ 85], 20.00th=[ 85], | 30.00th=[ 85], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 87], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=16.58%, ctx=73, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18628: Thu Aug 22 04:17:56 2024 write: IOPS=0, BW=4042KiB/s (4139kB/s)(12.0MiB/3040msec) clat (msec): min=48, max=2703, avg=1004.72, stdev=1475.00 lat (msec): min=49, max=2706, avg=1005.81, stdev=1476.38 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 262], 50.00th=[ 262], 60.00th=[ 262], | 70.00th=[ 2702], 80.00th=[ 2702], 90.00th=[ 2702], 95.00th=[ 2702], | 99.00th=[ 2702], 99.50th=[ 2702], 99.90th=[ 2702], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 8192, max= 8192, per=76.01%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.07%, sys=3.19%, ctx=215, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5492KiB/s (5624kB/s), 1373KiB/s-4371KiB/s (1406kB/s-4476kB/s), io=16.0MiB (16.8MB), run=2811-2983msec WRITE: bw=10.5MiB/s (11.0MB/s), 2746KiB/s-32.0MiB/s (2812kB/s-33.6MB/s), io=32.0MiB (33.6MB), run=375-3040msec