== sanity test 398b: DIO and buffer IO race ============== 05:23:14 (1713086594) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.12846 s, 44.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10309: Sun Apr 14 05:23:30 2024 read: IOPS=114, BW=458KiB/s (469kB/s)(5992KiB/13078msec) clat (usec): min=44, max=153130, avg=4818.49, stdev=4672.75 lat (usec): min=44, max=153130, avg=4819.16, stdev=4672.76 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 105], 10.00th=[ 2180], 20.00th=[ 2704], | 30.00th=[ 3130], 40.00th=[ 3654], 50.00th=[ 4359], 60.00th=[ 5014], | 70.00th=[ 5735], 80.00th=[ 6652], 90.00th=[ 8094], 95.00th=[ 9372], | 99.00th=[ 13304], 99.50th=[ 15533], 99.90th=[ 18744], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 240, max= 640, per=24.84%, avg=455.65, stdev=80.81, samples=26 iops : min= 60, max= 160, avg=113.77, stdev=20.21, samples=26 write: IOPS=120, BW=481KiB/s (493kB/s)(6296KiB/13078msec) clat (usec): min=370, max=23188, avg=3689.46, stdev=3414.34 lat (usec): min=371, max=23188, avg=3690.49, stdev=3414.79 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 510], 10.00th=[ 553], 20.00th=[ 685], | 30.00th=[ 816], 40.00th=[ 1565], 50.00th=[ 2999], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[15270], 99.50th=[16909], 99.90th=[19530], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 296, max= 632, per=24.89%, avg=477.81, stdev=71.31, samples=26 iops : min= 74, max= 158, avg=119.31, stdev=17.85, samples=26 lat (usec) : 50=0.13%, 100=1.95%, 250=0.91%, 500=2.05%, 750=10.81% lat (usec) : 1000=5.24% lat (msec) : 2=4.04%, 4=29.85%, 10=40.27%, 20=4.69%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.11%, sys=23.13%, ctx=5668, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10310: Sun Apr 14 05:23:30 2024 read: IOPS=113, BW=454KiB/s (465kB/s)(5940KiB/13078msec) clat (usec): min=44, max=20492, avg=4731.30, stdev=2732.96 lat (usec): min=44, max=20493, avg=4731.98, stdev=2732.98 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 105], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 3097], 40.00th=[ 3621], 50.00th=[ 4293], 60.00th=[ 5014], | 70.00th=[ 5735], 80.00th=[ 6718], 90.00th=[ 7963], 95.00th=[ 9372], | 99.00th=[13960], 99.50th=[16319], 99.90th=[20055], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 344, max= 680, per=24.66%, avg=452.23, stdev=74.54, samples=26 iops : min= 86, max= 170, avg=113.00, stdev=18.66, samples=26 write: IOPS=121, BW=485KiB/s (497kB/s)(6348KiB/13078msec) clat (usec): min=390, max=151711, avg=3781.39, stdev=4984.56 lat (usec): min=391, max=151712, avg=3782.22, stdev=4984.58 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 515], 10.00th=[ 570], 20.00th=[ 701], | 30.00th=[ 881], 40.00th=[ 2024], 50.00th=[ 3064], 60.00th=[ 3884], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8291], 95.00th=[ 10290], | 99.00th=[ 14222], 99.50th=[ 15795], 99.90th=[ 30016], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 360, max= 720, per=25.06%, avg=481.15, stdev=75.11, samples=26 iops : min= 90, max= 180, avg=120.23, stdev=18.84, samples=26 lat (usec) : 50=0.20%, 100=1.99%, 250=0.75%, 500=2.05%, 750=10.06% lat (usec) : 1000=4.95% lat (msec) : 2=4.43%, 4=29.69%, 10=41.41%, 20=4.33%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.07%, ctx=5808, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10311: Sun Apr 14 05:23:30 2024 read: IOPS=115, BW=460KiB/s (471kB/s)(6028KiB/13092msec) clat (usec): min=42, max=23738, avg=4614.56, stdev=2838.06 lat (usec): min=43, max=23738, avg=4616.22, stdev=2838.49 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 99], 10.00th=[ 2180], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 4015], 60.00th=[ 4817], | 70.00th=[ 5669], 80.00th=[ 6521], 90.00th=[ 7963], 95.00th=[ 9372], | 99.00th=[14484], 99.50th=[16909], 99.90th=[22414], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 288, max= 704, per=24.99%, avg=458.35, stdev=89.09, samples=26 iops : min= 72, max= 176, avg=114.54, stdev=22.28, samples=26 write: IOPS=119, BW=478KiB/s (490kB/s)(6260KiB/13092msec) clat (usec): min=433, max=150279, avg=3885.62, stdev=4993.95 lat (usec): min=434, max=150280, avg=3886.38, stdev=4993.96 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 515], 10.00th=[ 570], 20.00th=[ 709], | 30.00th=[ 873], 40.00th=[ 2180], 50.00th=[ 3130], 60.00th=[ 3949], | 70.00th=[ 5014], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 13960], 99.50th=[ 15139], 99.90th=[ 21627], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 344, max= 728, per=24.72%, avg=474.62, stdev=89.45, samples=26 iops : min= 86, max= 182, avg=118.62, stdev=22.39, samples=26 lat (usec) : 50=0.13%, 100=2.38%, 250=0.91%, 500=1.73%, 750=10.29% lat (usec) : 1000=4.88% lat (msec) : 2=3.84%, 4=31.32%, 10=39.75%, 20=4.65%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.92%, ctx=5714, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10312: Sun Apr 14 05:23:30 2024 read: IOPS=116, BW=464KiB/s (475kB/s)(6052KiB/13040msec) clat (usec): min=44, max=147410, avg=4789.61, stdev=4699.27 lat (usec): min=45, max=147410, avg=4790.30, stdev=4699.26 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 103], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 2999], 40.00th=[ 3490], 50.00th=[ 4178], 60.00th=[ 4948], | 70.00th=[ 5735], 80.00th=[ 6652], 90.00th=[ 7767], 95.00th=[ 9372], | 99.00th=[ 16057], 99.50th=[ 19006], 99.90th=[ 26084], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 288, max= 752, per=25.26%, avg=463.23, stdev=78.54, samples=26 iops : min= 72, max= 188, avg=115.73, stdev=19.64, samples=26 write: IOPS=119, BW=478KiB/s (490kB/s)(6236KiB/13040msec) clat (usec): min=375, max=27445, avg=3684.94, stdev=3354.19 lat (usec): min=375, max=27445, avg=3686.13, stdev=3353.95 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 510], 10.00th=[ 553], 20.00th=[ 660], | 30.00th=[ 824], 40.00th=[ 1631], 50.00th=[ 3064], 60.00th=[ 3884], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[ 8356], 95.00th=[10028], | 99.00th=[13960], 99.50th=[15533], 99.90th=[23725], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 352, max= 776, per=24.82%, avg=476.46, stdev=87.47, samples=26 iops : min= 88, max= 194, avg=119.04, stdev=21.90, samples=26 lat (usec) : 50=0.07%, 100=2.21%, 250=0.94%, 500=2.18%, 750=10.48% lat (usec) : 1000=5.27% lat (msec) : 2=3.48%, 4=30.01%, 10=41.02%, 20=4.10%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.88%, ctx=5784, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1834KiB/s (1878kB/s), 454KiB/s-464KiB/s (465kB/s-475kB/s), io=23.4MiB (24.6MB), run=13040-13092msec WRITE: bw=1920KiB/s (1966kB/s), 478KiB/s-485KiB/s (490kB/s-497kB/s), io=24.6MiB (25.7MB), run=13040-13092msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10305: Sun Apr 14 05:23:33 2024 read: IOPS=90, BW=363KiB/s (371kB/s)(5992KiB/16526msec) clat (usec): min=1556, max=154966, avg=4463.21, stdev=5290.13 lat (usec): min=1557, max=154967, avg=4463.98, stdev=5290.11 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 2057], | 30.00th=[ 2245], 40.00th=[ 2573], 50.00th=[ 2966], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 6063], 90.00th=[ 9241], 95.00th=[ 12125], | 99.00th=[ 17957], 99.50th=[ 22676], 99.90th=[ 32637], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 191, max= 808, per=25.11%, avg=362.39, stdev=185.18, samples=33 iops : min= 47, max= 202, avg=90.58, stdev=46.32, samples=33 write: IOPS=95, BW=381KiB/s (390kB/s)(6296KiB/16526msec) clat (usec): min=2117, max=30027, avg=6218.19, stdev=3901.78 lat (usec): min=2118, max=30028, avg=6219.03, stdev=3901.78 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2737], 10.00th=[ 2933], 20.00th=[ 3163], | 30.00th=[ 3425], 40.00th=[ 3818], 50.00th=[ 4621], 60.00th=[ 5932], | 70.00th=[ 7439], 80.00th=[ 9110], 90.00th=[11600], 95.00th=[13960], | 99.00th=[19268], 99.50th=[22152], 99.90th=[28443], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 208, max= 840, per=25.17%, avg=380.33, stdev=190.12, samples=33 iops : min= 52, max= 210, avg=95.06, stdev=47.55, samples=33 lat (msec) : 2=8.20%, 4=46.58%, 10=32.81%, 20=11.69%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=10.84%, ctx=3536, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10306: Sun Apr 14 05:23:33 2024 read: IOPS=89, BW=358KiB/s (367kB/s)(5940KiB/16592msec) clat (usec): min=1531, max=147100, avg=4439.21, stdev=5109.04 lat (usec): min=1531, max=147101, avg=4439.93, stdev=5109.07 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1876], 20.00th=[ 2040], | 30.00th=[ 2245], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3490], | 70.00th=[ 4228], 80.00th=[ 6063], 90.00th=[ 9503], 95.00th=[ 12125], | 99.00th=[ 18220], 99.50th=[ 19792], 99.90th=[ 28181], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 119, max= 800, per=24.74%, avg=357.06, stdev=191.64, samples=33 iops : min= 29, max= 200, avg=89.24, stdev=47.94, samples=33 write: IOPS=95, BW=383KiB/s (392kB/s)(6348KiB/16592msec) clat (usec): min=2158, max=31155, avg=6265.73, stdev=3967.90 lat (usec): min=2159, max=31156, avg=6266.71, stdev=3967.85 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2769], 10.00th=[ 2933], 20.00th=[ 3228], | 30.00th=[ 3490], 40.00th=[ 3916], 50.00th=[ 4686], 60.00th=[ 5800], | 70.00th=[ 7373], 80.00th=[ 9110], 90.00th=[11863], 95.00th=[14222], | 99.00th=[19792], 99.50th=[22676], 99.90th=[30802], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 199, max= 800, per=25.17%, avg=380.33, stdev=183.71, samples=33 iops : min= 49, max= 200, avg=95.06, stdev=45.95, samples=33 lat (msec) : 2=8.33%, 4=45.38%, 10=33.59%, 20=11.98%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=10.82%, ctx=3480, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10307: Sun Apr 14 05:23:33 2024 read: IOPS=91, BW=366KiB/s (375kB/s)(6028KiB/16461msec) clat (usec): min=1530, max=32387, avg=4372.78, stdev=3504.80 lat (usec): min=1531, max=32388, avg=4373.47, stdev=3504.82 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1893], 20.00th=[ 2073], | 30.00th=[ 2278], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3490], | 70.00th=[ 4359], 80.00th=[ 5997], 90.00th=[ 9503], 95.00th=[11731], | 99.00th=[16581], 99.50th=[19792], 99.90th=[25297], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 167, max= 816, per=24.82%, avg=358.19, stdev=180.15, samples=32 iops : min= 41, max= 204, avg=89.50, stdev=45.08, samples=32 write: IOPS=95, BW=380KiB/s (389kB/s)(6260KiB/16461msec) clat (msec): min=2, max=151, avg= 6.27, stdev= 5.36 lat (msec): min=2, max=151, avg= 6.27, stdev= 5.36 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 31], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 200, max= 832, per=24.53%, avg=370.69, stdev=188.19, samples=32 iops : min= 50, max= 208, avg=92.62, stdev=47.07, samples=32 lat (msec) : 2=7.75%, 4=46.94%, 10=33.59%, 20=11.04%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=10.98%, ctx=3488, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10308: Sun Apr 14 05:23:33 2024 read: IOPS=90, BW=364KiB/s (373kB/s)(6052KiB/16634msec) clat (usec): min=1586, max=23436, avg=4432.03, stdev=3579.51 lat (usec): min=1586, max=23436, avg=4432.72, stdev=3579.51 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1795], 10.00th=[ 1893], 20.00th=[ 2040], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3425], | 70.00th=[ 4359], 80.00th=[ 6390], 90.00th=[ 9765], 95.00th=[12387], | 99.00th=[17957], 99.50th=[19792], 99.90th=[22938], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 159, max= 840, per=24.99%, avg=360.67, stdev=188.50, samples=33 iops : min= 39, max= 210, avg=90.12, stdev=47.15, samples=33 write: IOPS=93, BW=375KiB/s (384kB/s)(6236KiB/16634msec) clat (msec): min=2, max=150, avg= 6.33, stdev= 5.41 lat (msec): min=2, max=150, avg= 6.33, stdev= 5.41 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 29], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 191, max= 792, per=24.64%, avg=372.30, stdev=192.06, samples=33 iops : min= 47, max= 198, avg=93.03, stdev=48.05, samples=33 lat (msec) : 2=8.50%, 4=45.80%, 10=32.26%, 20=12.70%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.84%, ctx=3534, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1444KiB/s (1478kB/s), 358KiB/s-366KiB/s (367kB/s-375kB/s), io=23.4MiB (24.6MB), run=16461-16634msec WRITE: bw=1511KiB/s (1548kB/s), 375KiB/s-383KiB/s (384kB/s-392kB/s), io=24.6MiB (25.7MB), run=16461-16634msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10341: Sun Apr 14 05:23:41 2024 read: IOPS=57, BW=914KiB/s (936kB/s)(5920KiB/6474msec) clat (usec): min=69, max=152287, avg=11767.25, stdev=8631.20 lat (usec): min=70, max=152287, avg=11768.85, stdev=8631.04 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 7570], 10.00th=[ 8160], 20.00th=[ 9110], | 30.00th=[ 9634], 40.00th=[ 10421], 50.00th=[ 11076], 60.00th=[ 11731], | 70.00th=[ 12387], 80.00th=[ 13304], 90.00th=[ 14877], 95.00th=[ 16909], | 99.00th=[ 25822], 99.50th=[ 66847], 99.90th=[152044], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 479, max= 1152, per=25.29%, avg=898.58, stdev=193.87, samples=12 iops : min= 29, max= 72, avg=56.08, stdev=12.30, samples=12 write: IOPS=61, BW=984KiB/s (1007kB/s)(6368KiB/6474msec) clat (usec): min=652, max=40003, avg=5305.06, stdev=5492.18 lat (usec): min=653, max=40004, avg=5306.38, stdev=5492.20 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 783], 10.00th=[ 947], 20.00th=[ 2008], | 30.00th=[ 2540], 40.00th=[ 2933], 50.00th=[ 3359], 60.00th=[ 4621], | 70.00th=[ 5735], 80.00th=[ 6915], 90.00th=[12125], 95.00th=[15533], | 99.00th=[33162], 99.50th=[37487], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 479, max= 1312, per=26.11%, avg=975.92, stdev=262.10, samples=12 iops : min= 29, max= 82, avg=60.92, stdev=16.54, samples=12 lat (usec) : 100=0.52%, 250=0.78%, 750=1.43%, 1000=4.43% lat (msec) : 2=4.43%, 4=18.36%, 10=31.64%, 20=35.68%, 50=2.47% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.12%, sys=20.87%, ctx=2766, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10342: Sun Apr 14 05:23:41 2024 read: IOPS=56, BW=910KiB/s (932kB/s)(5744KiB/6314msec) clat (usec): min=55, max=31318, avg=11266.32, stdev=3399.37 lat (usec): min=56, max=31319, avg=11267.44, stdev=3399.39 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 7308], 10.00th=[ 8291], 20.00th=[ 9241], | 30.00th=[ 9896], 40.00th=[10421], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12256], 80.00th=[13173], 90.00th=[14615], 95.00th=[16057], | 99.00th=[23462], 99.50th=[27395], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 670, max= 1152, per=25.81%, avg=917.17, stdev=130.03, samples=12 iops : min= 41, max= 72, avg=57.25, stdev= 8.28, samples=12 write: IOPS=64, BW=1036KiB/s (1061kB/s)(6544KiB/6314msec) clat (usec): min=698, max=186747, avg=5525.82, stdev=10478.11 lat (usec): min=699, max=186749, avg=5528.55, stdev=10477.48 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 807], 10.00th=[ 898], 20.00th=[ 1221], | 30.00th=[ 2376], 40.00th=[ 2737], 50.00th=[ 3359], 60.00th=[ 4015], | 70.00th=[ 5211], 80.00th=[ 7111], 90.00th=[ 11731], 95.00th=[ 16188], | 99.00th=[ 29754], 99.50th=[ 39060], 99.90th=[187696], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 510, max= 1568, per=27.39%, avg=1023.83, stdev=259.97, samples=12 iops : min= 31, max= 98, avg=63.92, stdev=16.41, samples=12 lat (usec) : 100=0.78%, 250=0.13%, 750=1.82%, 1000=4.82% lat (msec) : 2=6.38%, 4=19.01%, 10=27.47%, 20=37.50%, 50=1.95% lat (msec) : 250=0.13% cpu : usr=0.10%, sys=18.77%, ctx=2758, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10343: Sun Apr 14 05:23:41 2024 read: IOPS=60, BW=964KiB/s (987kB/s)(6256KiB/6491msec) clat (usec): min=59, max=26313, avg=11313.80, stdev=3661.13 lat (usec): min=59, max=26314, avg=11314.44, stdev=3661.14 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 7373], 10.00th=[ 8160], 20.00th=[ 8979], | 30.00th=[ 9765], 40.00th=[10421], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12387], 80.00th=[13173], 90.00th=[15139], 95.00th=[17695], | 99.00th=[24511], 99.50th=[25822], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 638, max= 1152, per=26.94%, avg=957.17, stdev=156.81, samples=12 iops : min= 39, max= 72, avg=59.75, stdev= 9.96, samples=12 write: IOPS=58, BW=929KiB/s (952kB/s)(6032KiB/6491msec) clat (usec): min=693, max=179896, avg=5462.61, stdev=10181.86 lat (usec): min=694, max=179897, avg=5463.73, stdev=10181.86 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 816], 10.00th=[ 930], 20.00th=[ 1876], | 30.00th=[ 2540], 40.00th=[ 2900], 50.00th=[ 3523], 60.00th=[ 4490], | 70.00th=[ 5473], 80.00th=[ 7308], 90.00th=[ 10421], 95.00th=[ 14353], | 99.00th=[ 28181], 99.50th=[ 37487], 99.90th=[179307], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 576, max= 1184, per=24.25%, avg=906.50, stdev=194.76, samples=12 iops : min= 36, max= 74, avg=56.58, stdev=12.26, samples=12 lat (usec) : 100=0.78%, 250=0.52%, 750=0.78%, 1000=4.82% lat (msec) : 2=5.34%, 4=16.67%, 10=31.77%, 20=36.98%, 50=2.21% lat (msec) : 250=0.13% cpu : usr=0.20%, sys=18.95%, ctx=2875, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10344: Sun Apr 14 05:23:41 2024 read: IOPS=55, BW=895KiB/s (916kB/s)(6032KiB/6741msec) clat (usec): min=61, max=196646, avg=11711.43, stdev=10155.71 lat (usec): min=61, max=196647, avg=11715.21, stdev=10155.74 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 7177], 10.00th=[ 8160], 20.00th=[ 9110], | 30.00th=[ 9765], 40.00th=[ 10290], 50.00th=[ 10945], 60.00th=[ 11731], | 70.00th=[ 12518], 80.00th=[ 13304], 90.00th=[ 14746], 95.00th=[ 17171], | 99.00th=[ 22152], 99.50th=[ 27395], 99.90th=[196084], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 512, max= 1120, per=25.63%, avg=910.77, stdev=151.57, samples=13 iops : min= 32, max= 70, avg=56.92, stdev= 9.47, samples=13 write: IOPS=58, BW=928KiB/s (950kB/s)(6256KiB/6741msec) clat (usec): min=648, max=177298, avg=5924.09, stdev=10851.03 lat (usec): min=649, max=177299, avg=5927.56, stdev=10850.86 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 791], 10.00th=[ 906], 20.00th=[ 1237], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 3326], 60.00th=[ 4228], | 70.00th=[ 5473], 80.00th=[ 7767], 90.00th=[ 13698], 95.00th=[ 15795], | 99.00th=[ 39060], 99.50th=[ 63177], 99.90th=[177210], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 512, max= 1376, per=25.48%, avg=952.62, stdev=215.52, samples=13 iops : min= 32, max= 86, avg=59.54, stdev=13.47, samples=13 lat (usec) : 100=0.52%, 250=0.52%, 750=1.69%, 1000=5.21% lat (msec) : 2=6.38%, 4=16.54%, 10=29.82%, 20=36.46%, 50=2.47% lat (msec) : 100=0.13%, 250=0.26% cpu : usr=0.10%, sys=20.21%, ctx=2785, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3553KiB/s (3638kB/s), 895KiB/s-964KiB/s (916kB/s-987kB/s), io=23.4MiB (24.5MB), run=6314-6741msec WRITE: bw=3738KiB/s (3828kB/s), 928KiB/s-1036KiB/s (950kB/s-1061kB/s), io=24.6MiB (25.8MB), run=6314-6741msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10345: Sun Apr 14 05:23:41 2024 read: IOPS=57, BW=915KiB/s (937kB/s)(5920KiB/6472msec) clat (usec): min=1875, max=168785, avg=7129.87, stdev=10062.81 lat (usec): min=1876, max=168786, avg=7130.56, stdev=10062.80 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 15], 95.00th=[ 17], | 99.00th=[ 31], 99.50th=[ 40], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 480, max= 1120, per=25.21%, avg=888.00, stdev=188.64, samples=12 iops : min= 30, max= 70, avg=55.50, stdev=11.79, samples=12 write: IOPS=61, BW=984KiB/s (1008kB/s)(6368KiB/6472msec) clat (usec): min=2772, max=55102, avg=9613.33, stdev=5832.25 lat (usec): min=2773, max=55104, avg=9614.50, stdev=5832.23 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3556], 10.00th=[ 4146], 20.00th=[ 5080], | 30.00th=[ 6128], 40.00th=[ 7177], 50.00th=[ 8225], 60.00th=[ 9634], | 70.00th=[10814], 80.00th=[13173], 90.00th=[16450], 95.00th=[20317], | 99.00th=[32113], 99.50th=[42206], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 544, max= 1248, per=26.05%, avg=965.33, stdev=226.62, samples=12 iops : min= 34, max= 78, avg=60.33, stdev=14.16, samples=12 lat (msec) : 2=0.39%, 4=26.04%, 10=44.40%, 20=24.74%, 50=4.17% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.22%, sys=9.46%, ctx=1146, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10346: Sun Apr 14 05:23:41 2024 read: IOPS=56, BW=905KiB/s (927kB/s)(5744KiB/6348msec) clat (usec): min=1903, max=42920, avg=6493.86, stdev=5197.88 lat (usec): min=1904, max=42920, avg=6494.51, stdev=5197.89 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2343], 10.00th=[ 2507], 20.00th=[ 2802], | 30.00th=[ 3097], 40.00th=[ 3621], 50.00th=[ 4359], 60.00th=[ 5473], | 70.00th=[ 8225], 80.00th=[10028], 90.00th=[12518], 95.00th=[16450], | 99.00th=[26608], 99.50th=[31327], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 608, max= 1184, per=26.03%, avg=917.08, stdev=163.75, samples=12 iops : min= 38, max= 74, avg=57.25, stdev=10.19, samples=12 write: IOPS=64, BW=1031KiB/s (1056kB/s)(6544KiB/6348msec) clat (msec): min=2, max=195, avg= 9.80, stdev=11.26 lat (msec): min=2, max=195, avg= 9.80, stdev=11.26 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 17], 95.00th=[ 21], | 99.00th=[ 42], 99.50th=[ 47], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 480, max= 1440, per=27.62%, avg=1023.75, stdev=261.33, samples=12 iops : min= 30, max= 90, avg=63.92, stdev=16.32, samples=12 lat (msec) : 2=0.52%, 4=25.91%, 10=48.70%, 20=20.57%, 50=4.04% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.14%, sys=9.48%, ctx=1121, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10347: Sun Apr 14 05:23:41 2024 read: IOPS=60, BW=961KiB/s (984kB/s)(6256KiB/6508msec) clat (usec): min=1935, max=170240, avg=7033.87, stdev=10000.87 lat (usec): min=1948, max=170240, avg=7034.71, stdev=10000.88 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 41], 99.50th=[ 44], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 608, max= 1184, per=27.32%, avg=962.46, stdev=187.71, samples=13 iops : min= 38, max= 74, avg=60.15, stdev=11.73, samples=13 write: IOPS=57, BW=927KiB/s (949kB/s)(6032KiB/6508msec) clat (usec): min=2694, max=51670, avg=9948.24, stdev=6482.22 lat (usec): min=2695, max=51671, avg=9949.47, stdev=6482.20 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3523], 10.00th=[ 3982], 20.00th=[ 4948], | 30.00th=[ 6128], 40.00th=[ 7046], 50.00th=[ 8586], 60.00th=[ 9765], | 70.00th=[11469], 80.00th=[13173], 90.00th=[17433], 95.00th=[22152], | 99.00th=[34866], 99.50th=[50070], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 608, max= 1184, per=24.91%, avg=923.08, stdev=198.49, samples=13 iops : min= 38, max= 74, avg=57.69, stdev=12.41, samples=13 lat (msec) : 2=0.26%, 4=27.86%, 10=42.45%, 20=24.61%, 50=4.56% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.17%, sys=9.24%, ctx=1161, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10348: Sun Apr 14 05:23:41 2024 read: IOPS=55, BW=887KiB/s (909kB/s)(6032KiB/6798msec) clat (usec): min=1823, max=166464, avg=6848.01, stdev=9842.28 lat (usec): min=1823, max=166464, avg=6848.78, stdev=9842.24 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 17], | 99.00th=[ 27], 99.50th=[ 48], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 608, max= 1088, per=25.78%, avg=908.31, stdev=125.67, samples=13 iops : min= 38, max= 68, avg=56.77, stdev= 7.85, samples=13 write: IOPS=57, BW=920KiB/s (942kB/s)(6256KiB/6798msec) clat (msec): min=2, max=250, avg=10.76, stdev=13.79 lat (msec): min=2, max=250, avg=10.76, stdev=13.79 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 19], 95.00th=[ 22], | 99.00th=[ 37], 99.50th=[ 64], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 608, max= 1376, per=25.51%, avg=945.23, stdev=188.24, samples=13 iops : min= 38, max= 86, avg=59.08, stdev=11.76, samples=13 lat (msec) : 2=0.13%, 4=29.69%, 10=40.76%, 20=24.48%, 50=4.56% lat (msec) : 100=0.13%, 250=0.13%, 500=0.13% cpu : usr=0.09%, sys=9.21%, ctx=1240, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3523KiB/s (3608kB/s), 887KiB/s-961KiB/s (909kB/s-984kB/s), io=23.4MiB (24.5MB), run=6348-6798msec WRITE: bw=3707KiB/s (3796kB/s), 920KiB/s-1031KiB/s (942kB/s-1056kB/s), io=24.6MiB (25.8MB), run=6348-6798msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10359: Sun Apr 14 05:23:44 2024 read: IOPS=4, BW=4131KiB/s (4231kB/s)(7168KiB/1735msec) clat (msec): min=18, max=464, avg=144.00, stdev=202.60 lat (msec): min=18, max=464, avg=144.00, stdev=202.60 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 21], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 24], 60.00th=[ 44], | 70.00th=[ 44], 80.00th=[ 414], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 2048, max= 4096, per=35.30%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2951KiB/s (3022kB/s)(5120KiB/1735msec) clat (msec): min=21, max=482, avg=140.97, stdev=194.56 lat (msec): min=21, max=482, avg=141.03, stdev=194.56 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 2048, max= 4096, per=22.69%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=8.33%, 250=8.33%, 500=25.00% cpu : usr=0.00%, sys=7.15%, ctx=49, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10360: Sun Apr 14 05:23:44 2024 read: IOPS=0, BW=967KiB/s (990kB/s)(2048KiB/2118msec) clat (usec): min=18449, max=59532, avg=38990.79, stdev=29050.43 lat (usec): min=18450, max=59533, avg=38991.99, stdev=29049.83 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[18482], 40.00th=[18482], 50.00th=[18482], 60.00th=[59507], | 70.00th=[59507], 80.00th=[59507], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 2043, max= 2048, per=21.16%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4835KiB/s (4951kB/s)(10.0MiB/2118msec) clat (msec): min=21, max=533, avg=201.43, stdev=203.58 lat (msec): min=21, max=533, avg=201.50, stdev=203.58 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 27], | 30.00th=[ 30], 40.00th=[ 54], 50.00th=[ 70], 60.00th=[ 165], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 485], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 4096, max= 6144, per=40.31%, avg=5457.00, stdev=1178.68, samples=3 iops : min= 4, max= 6, avg= 5.00, stdev= 1.00, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=16.67%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=7.09%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10361: Sun Apr 14 05:23:44 2024 read: IOPS=2, BW=2950KiB/s (3020kB/s)(6144KiB/2083msec) clat (msec): min=25, max=168, avg=68.09, stdev=55.87 lat (msec): min=25, max=168, avg=68.09, stdev=55.87 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 52], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 4096, max= 6144, per=52.95%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2950KiB/s (3020kB/s)(6144KiB/2083msec) clat (msec): min=22, max=900, avg=273.52, stdev=346.95 lat (msec): min=22, max=900, avg=273.58, stdev=346.94 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 63], 50.00th=[ 63], 60.00th=[ 163], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 2048, max= 6144, per=30.26%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=41.67%, 100=25.00%, 250=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.91%, ctx=78, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10362: Sun Apr 14 05:23:44 2024 read: IOPS=2, BW=2485KiB/s (2545kB/s)(5120KiB/2060msec) clat (msec): min=17, max=447, avg=136.50, stdev=182.03 lat (msec): min=17, max=447, avg=136.50, stdev=182.03 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 35], 60.00th=[ 35], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 2048, max= 6144, per=42.36%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3480KiB/s (3563kB/s)(7168KiB/2060msec) clat (msec): min=24, max=919, avg=193.95, stdev=323.32 lat (msec): min=24, max=919, avg=194.02, stdev=323.32 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 55], | 30.00th=[ 60], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 67], 80.00th=[ 171], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 2048, max= 6144, per=35.30%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=16.67%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.05%, sys=6.90%, ctx=70, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9669KiB/s (9902kB/s), 967KiB/s-4131KiB/s (990kB/s-4231kB/s), io=20.0MiB (20.0MB), run=1735-2118msec WRITE: bw=13.2MiB/s (13.9MB/s), 2950KiB/s-4835KiB/s (3020kB/s-4951kB/s), io=28.0MiB (29.4MB), run=1735-2118msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10363: Sun Apr 14 05:23:44 2024 read: IOPS=3, BW=3879KiB/s (3972kB/s)(7168KiB/1848msec) clat (usec): min=516, max=547196, avg=213368.03, stdev=255776.61 lat (usec): min=518, max=547197, avg=213369.29, stdev=255776.47 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 519], 10.00th=[ 519], 20.00th=[ 553], | 30.00th=[ 2114], 40.00th=[ 2114], 50.00th=[ 38011], 60.00th=[438305], | 70.00th=[438305], 80.00th=[467665], 90.00th=[549454], 95.00th=[549454], | 99.00th=[549454], 99.50th=[549454], 99.90th=[549454], 99.95th=[549454], | 99.99th=[549454] bw ( KiB/s): min= 2048, max= 4096, per=31.91%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2771KiB/s (2837kB/s)(5120KiB/1848msec) clat (msec): min=28, max=165, avg=70.63, stdev=56.03 lat (msec): min=28, max=165, avg=70.67, stdev=56.02 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 62], 60.00th=[ 62], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 2048, max= 2048, per=15.19%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 4=8.33%, 50=25.00%, 100=16.67%, 250=8.33%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.05%, sys=21.93%, ctx=869, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10364: Sun Apr 14 05:23:44 2024 read: IOPS=0, BW=963KiB/s (986kB/s)(2048KiB/2127msec) clat (msec): min=54, max=805, avg=430.20, stdev=530.63 lat (msec): min=54, max=805, avg=430.20, stdev=530.63 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 810], | 70.00th=[ 810], 80.00th=[ 810], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 1678, max= 2048, per=19.35%, avg=1863.00, stdev=261.63, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4814KiB/s (4930kB/s)(10.0MiB/2127msec) clat (msec): min=20, max=525, avg=126.47, stdev=186.85 lat (msec): min=20, max=525, avg=126.54, stdev=186.85 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 26], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 38], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 426], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 4079, max= 6714, per=41.88%, avg=5645.67, stdev=1386.38, samples=3 iops : min= 3, max= 6, avg= 5.00, stdev= 1.73, samples=3 lat (msec) : 50=50.00%, 100=25.00%, 500=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=29.30%, ctx=410, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10365: Sun Apr 14 05:23:44 2024 read: IOPS=3, BW=3471KiB/s (3554kB/s)(6144KiB/1770msec) clat (usec): min=397, max=72286, avg=28759.89, stdev=28085.13 lat (usec): min=398, max=72286, avg=28760.93, stdev=28084.91 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 396], 10.00th=[ 396], 20.00th=[ 652], | 30.00th=[ 652], 40.00th=[22938], 50.00th=[22938], 60.00th=[27132], | 70.00th=[49021], 80.00th=[49021], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 4096, max= 4096, per=42.54%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3471KiB/s (3554kB/s)(6144KiB/1770msec) clat (msec): min=25, max=872, avg=265.91, stdev=342.04 lat (msec): min=25, max=872, avg=266.03, stdev=342.01 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 124], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 2048, max= 4096, per=22.79%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=11.93%, ctx=87, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10366: Sun Apr 14 05:23:44 2024 read: IOPS=2, BW=2885KiB/s (2954kB/s)(5120KiB/1775msec) clat (usec): min=416, max=461566, avg=115364.20, stdev=195811.08 lat (usec): min=417, max=461567, avg=115365.35, stdev=195810.99 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 416], | 30.00th=[ 515], 40.00th=[ 515], 50.00th=[ 44303], 60.00th=[ 44303], | 70.00th=[ 69731], 80.00th=[ 69731], 90.00th=[463471], 95.00th=[463471], | 99.00th=[463471], 99.50th=[463471], 99.90th=[463471], 99.95th=[463471], | 99.99th=[463471] bw ( KiB/s): min= 2048, max= 2048, per=21.27%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4038KiB/s (4135kB/s)(7168KiB/1775msec) clat (msec): min=18, max=886, avg=170.65, stdev=318.19 lat (msec): min=18, max=886, avg=170.70, stdev=318.19 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 56], 60.00th=[ 66], | 70.00th=[ 66], 80.00th=[ 128], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 2048, max= 6144, per=30.39%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=16.07%, ctx=416, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9629KiB/s (9860kB/s), 963KiB/s-3879KiB/s (986kB/s-3972kB/s), io=20.0MiB (20.0MB), run=1770-2127msec WRITE: bw=13.2MiB/s (13.8MB/s), 2771KiB/s-4814KiB/s (2837kB/s-4930kB/s), io=28.0MiB (29.4MB), run=1770-2127msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10374: Sun Apr 14 05:23:47 2024 read: IOPS=1, BW=5835KiB/s (5975kB/s)(12.0MiB/2106msec) clat (msec): min=3, max=1878, avg=700.57, stdev=1026.16 lat (msec): min=3, max=1878, avg=700.58, stdev=1026.16 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 1871], 80.00th=[ 1871], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 4=33.33%, 250=33.33% cpu : usr=0.00%, sys=44.18%, ctx=812, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10376: Sun Apr 14 05:23:47 2024 read: IOPS=0, BW=1851KiB/s (1895kB/s)(4096KiB/2213msec) clat (nsec): min=168507k, max=168507k, avg=168507190.00, stdev= 0.00 lat (nsec): min=168509k, max=168509k, avg=168509011.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3702KiB/s (3791kB/s)(8192KiB/2213msec) clat (msec): min=125, max=1915, avg=1020.88, stdev=1265.74 lat (msec): min=126, max=1916, avg=1021.23, stdev=1265.88 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 127], 10.00th=[ 127], 20.00th=[ 127], | 30.00th=[ 127], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 1921], | 70.00th=[ 1921], 80.00th=[ 1921], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] lat (msec) : 250=66.67% cpu : usr=0.00%, sys=12.21%, ctx=149, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10378: Sun Apr 14 05:23:47 2024 write: IOPS=1, BW=5308KiB/s (5435kB/s)(12.0MiB/2315msec) clat (msec): min=107, max=1764, avg=767.60, stdev=878.55 lat (msec): min=107, max=1764, avg=767.84, stdev=878.46 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 1770], 80.00th=[ 1770], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 8192, max= 8192, per=57.88%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=13.01%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10380: Sun Apr 14 05:23:47 2024 write: IOPS=1, BW=7074KiB/s (7244kB/s)(12.0MiB/1737msec) clat (msec): min=129, max=1449, avg=576.34, stdev=756.09 lat (msec): min=129, max=1449, avg=576.66, stdev=756.04 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=15.03%, ctx=72, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7404KiB/s (7581kB/s), 1851KiB/s-5835KiB/s (1895kB/s-5975kB/s), io=16.0MiB (16.8MB), run=2106-2213msec WRITE: bw=13.8MiB/s (14.5MB/s), 3702KiB/s-7074KiB/s (3791kB/s-7244kB/s), io=32.0MiB (33.6MB), run=1737-2315msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10375: Sun Apr 14 05:23:48 2024 read: IOPS=1, BW=5148KiB/s (5271kB/s)(12.0MiB/2387msec) clat (msec): min=291, max=1544, avg=793.52, stdev=662.91 lat (msec): min=291, max=1544, avg=793.52, stdev=662.91 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 1552], 80.00th=[ 1552], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=5.57%, ctx=83, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10377: Sun Apr 14 05:23:48 2024 read: IOPS=0, BW=1767KiB/s (1809kB/s)(4096KiB/2318msec) clat (nsec): min=240279k, max=240279k, avg=240279332.00, stdev= 0.00 lat (nsec): min=240281k, max=240281k, avg=240280543.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 241], 95.00th=[ 241], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3534KiB/s (3619kB/s)(8192KiB/2318msec) clat (msec): min=158, max=1907, avg=1033.23, stdev=1236.54 lat (msec): min=159, max=1907, avg=1033.51, stdev=1236.54 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 1905], | 70.00th=[ 1905], 80.00th=[ 1905], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] lat (msec) : 250=66.67% cpu : usr=0.09%, sys=4.66%, ctx=132, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10379: Sun Apr 14 05:23:48 2024 write: IOPS=1, BW=5515KiB/s (5648kB/s)(12.0MiB/2228msec) clat (msec): min=158, max=1842, avg=741.73, stdev=953.64 lat (msec): min=158, max=1842, avg=742.07, stdev=953.58 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 1838], 80.00th=[ 1838], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=7.59%, ctx=67, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10381: Sun Apr 14 05:23:48 2024 write: IOPS=1, BW=6056KiB/s (6202kB/s)(12.0MiB/2029msec) clat (msec): min=107, max=1588, avg=674.47, stdev=799.26 lat (msec): min=107, max=1589, avg=674.77, stdev=799.42 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 1586], 80.00th=[ 1586], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=7.40%, ctx=135, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6864KiB/s (7029kB/s), 1767KiB/s-5148KiB/s (1809kB/s-5271kB/s), io=16.0MiB (16.8MB), run=2318-2387msec WRITE: bw=13.8MiB/s (14.5MB/s), 3534KiB/s-6056KiB/s (3619kB/s-6202kB/s), io=32.0MiB (33.6MB), run=2029-2318msec