== sanity test 398b: DIO and buffer IO race ============== 20:17:26 (1718324246) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.19762 s, 42.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32552: Thu Jun 13 20:17:41 2024 read: IOPS=123, BW=493KiB/s (505kB/s)(5992KiB/12153msec) clat (usec): min=55, max=165164, avg=4667.74, stdev=4863.50 lat (usec): min=55, max=165165, avg=4668.45, stdev=4863.54 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1942], 10.00th=[ 2343], 20.00th=[ 2769], | 30.00th=[ 3130], 40.00th=[ 3589], 50.00th=[ 4080], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6128], 90.00th=[ 7242], 95.00th=[ 9110], | 99.00th=[ 14353], 99.50th=[ 17695], 99.90th=[ 20579], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 256, max= 616, per=25.63%, avg=490.13, stdev=66.43, samples=24 iops : min= 64, max= 154, avg=122.42, stdev=16.59, samples=24 write: IOPS=129, BW=518KiB/s (530kB/s)(6296KiB/12153msec) clat (usec): min=442, max=17784, avg=3233.56, stdev=2864.11 lat (usec): min=443, max=17785, avg=3234.43, stdev=2863.94 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 619], | 30.00th=[ 775], 40.00th=[ 1795], 50.00th=[ 2606], 60.00th=[ 3228], | 70.00th=[ 4047], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[ 8717], | 99.00th=[12518], 99.50th=[13829], 99.90th=[16581], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 304, max= 608, per=25.61%, avg=512.75, stdev=64.73, samples=24 iops : min= 76, max= 152, avg=128.08, stdev=16.18, samples=24 lat (usec) : 100=2.05%, 250=0.07%, 500=0.42%, 750=14.19%, 1000=2.99% lat (msec) : 2=4.10%, 4=35.29%, 10=37.86%, 20=2.96%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.67%, ctx=5825, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32553: Thu Jun 13 20:17:41 2024 read: IOPS=121, BW=487KiB/s (499kB/s)(5940KiB/12201msec) clat (usec): min=50, max=16035, avg=4550.59, stdev=2459.93 lat (usec): min=50, max=16036, avg=4551.16, stdev=2459.93 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1827], 10.00th=[ 2311], 20.00th=[ 2704], | 30.00th=[ 3097], 40.00th=[ 3490], 50.00th=[ 4146], 60.00th=[ 4752], | 70.00th=[ 5407], 80.00th=[ 6194], 90.00th=[ 7439], 95.00th=[ 8979], | 99.00th=[13566], 99.50th=[14484], 99.90th=[15401], 99.95th=[16057], | 99.99th=[16057] bw ( KiB/s): min= 272, max= 680, per=25.39%, avg=485.46, stdev=80.19, samples=24 iops : min= 68, max= 170, avg=121.29, stdev=20.08, samples=24 write: IOPS=130, BW=520KiB/s (533kB/s)(6348KiB/12201msec) clat (usec): min=466, max=159189, avg=3381.42, stdev=4830.44 lat (usec): min=467, max=159190, avg=3383.67, stdev=4831.40 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 627], | 30.00th=[ 840], 40.00th=[ 2147], 50.00th=[ 2769], 60.00th=[ 3326], | 70.00th=[ 4228], 80.00th=[ 5407], 90.00th=[ 7308], 95.00th=[ 8979], | 99.00th=[ 12518], 99.50th=[ 13829], 99.90th=[ 19530], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 344, max= 672, per=25.70%, avg=514.54, stdev=71.36, samples=24 iops : min= 86, max= 168, avg=128.54, stdev=17.91, samples=24 lat (usec) : 100=1.95%, 250=0.23%, 500=0.26%, 750=13.83%, 1000=2.93% lat (msec) : 2=3.45%, 4=35.84%, 10=38.25%, 20=3.22%, 250=0.03% cpu : usr=0.19%, sys=23.69%, ctx=5827, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32555: Thu Jun 13 20:17:41 2024 read: IOPS=121, BW=484KiB/s (496kB/s)(6028KiB/12450msec) clat (usec): min=53, max=19567, avg=4555.33, stdev=2344.20 lat (usec): min=56, max=19567, avg=4556.61, stdev=2346.52 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1942], 10.00th=[ 2343], 20.00th=[ 2737], | 30.00th=[ 3163], 40.00th=[ 3654], 50.00th=[ 4228], 60.00th=[ 4817], | 70.00th=[ 5473], 80.00th=[ 6259], 90.00th=[ 7308], 95.00th=[ 8455], | 99.00th=[12387], 99.50th=[13829], 99.90th=[16450], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 224, max= 608, per=25.06%, avg=479.21, stdev=74.42, samples=24 iops : min= 56, max= 152, avg=119.75, stdev=18.63, samples=24 write: IOPS=125, BW=503KiB/s (515kB/s)(6260KiB/12450msec) clat (usec): min=474, max=172066, avg=3520.21, stdev=5147.08 lat (usec): min=475, max=172067, avg=3521.02, stdev=5147.16 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 627], | 30.00th=[ 881], 40.00th=[ 2278], 50.00th=[ 2835], 60.00th=[ 3523], | 70.00th=[ 4359], 80.00th=[ 5669], 90.00th=[ 7504], 95.00th=[ 8979], | 99.00th=[ 12780], 99.50th=[ 13304], 99.90th=[ 21627], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 312, max= 624, per=24.68%, avg=494.17, stdev=76.57, samples=24 iops : min= 78, max= 156, avg=123.50, stdev=19.12, samples=24 lat (usec) : 100=2.02%, 250=0.16%, 500=0.20%, 750=13.61%, 1000=2.21% lat (msec) : 2=3.48%, 4=34.90%, 10=40.72%, 20=2.64%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.28%, sys=23.16%, ctx=6055, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32557: Thu Jun 13 20:17:41 2024 read: IOPS=120, BW=482KiB/s (494kB/s)(6052KiB/12555msec) clat (usec): min=49, max=21163, avg=4578.45, stdev=2447.18 lat (usec): min=49, max=21164, avg=4579.07, stdev=2447.18 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1975], 10.00th=[ 2278], 20.00th=[ 2704], | 30.00th=[ 3097], 40.00th=[ 3621], 50.00th=[ 4228], 60.00th=[ 4817], | 70.00th=[ 5407], 80.00th=[ 6194], 90.00th=[ 7570], 95.00th=[ 8848], | 99.00th=[12256], 99.50th=[15008], 99.90th=[19268], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 152, max= 632, per=25.14%, avg=480.76, stdev=91.48, samples=25 iops : min= 38, max= 158, avg=120.12, stdev=22.87, samples=25 write: IOPS=124, BW=497KiB/s (509kB/s)(6236KiB/12555msec) clat (usec): min=453, max=173264, avg=3561.88, stdev=5261.78 lat (usec): min=454, max=173266, avg=3562.74, stdev=5261.82 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 635], | 30.00th=[ 881], 40.00th=[ 2147], 50.00th=[ 2802], 60.00th=[ 3425], | 70.00th=[ 4424], 80.00th=[ 5800], 90.00th=[ 7439], 95.00th=[ 9241], | 99.00th=[ 13304], 99.50th=[ 15139], 99.90th=[ 23987], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 168, max= 720, per=24.74%, avg=495.20, stdev=104.93, samples=25 iops : min= 42, max= 180, avg=123.72, stdev=26.26, samples=25 lat (usec) : 50=0.03%, 100=1.69%, 250=0.36%, 500=0.16%, 750=13.57% lat (usec) : 1000=2.41% lat (msec) : 2=3.78%, 4=34.41%, 10=40.17%, 20=3.29%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.07%, ctx=5958, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1913KiB/s (1958kB/s), 482KiB/s-493KiB/s (494kB/s-505kB/s), io=23.4MiB (24.6MB), run=12153-12555msec WRITE: bw=2002KiB/s (2050kB/s), 497KiB/s-520KiB/s (509kB/s-533kB/s), io=24.6MiB (25.7MB), run=12153-12555msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32554: Thu Jun 13 20:17:47 2024 read: IOPS=85, BW=341KiB/s (349kB/s)(5992KiB/17561msec) clat (usec): min=1699, max=163596, avg=4781.08, stdev=4987.10 lat (usec): min=1699, max=163598, avg=4781.83, stdev=4987.11 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2180], 10.00th=[ 2343], 20.00th=[ 2638], | 30.00th=[ 2933], 40.00th=[ 3294], 50.00th=[ 3720], 60.00th=[ 4146], | 70.00th=[ 4817], 80.00th=[ 6390], 90.00th=[ 8848], 95.00th=[ 10814], | 99.00th=[ 14877], 99.50th=[ 16712], 99.90th=[ 20055], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 152, max= 552, per=25.05%, avg=340.69, stdev=80.60, samples=35 iops : min= 38, max= 138, avg=85.09, stdev=20.12, samples=35 write: IOPS=89, BW=359KiB/s (367kB/s)(6296KiB/17561msec) clat (usec): min=2219, max=24597, avg=6561.69, stdev=3169.76 lat (usec): min=2219, max=24598, avg=6562.66, stdev=3169.61 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3261], 10.00th=[ 3556], 20.00th=[ 4015], | 30.00th=[ 4555], 40.00th=[ 5145], 50.00th=[ 5800], 60.00th=[ 6521], | 70.00th=[ 7242], 80.00th=[ 8455], 90.00th=[10683], 95.00th=[12518], | 99.00th=[18744], 99.50th=[20579], 99.90th=[22938], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 168, max= 648, per=25.15%, avg=357.83, stdev=93.29, samples=35 iops : min= 42, max= 162, avg=89.37, stdev=23.38, samples=35 lat (msec) : 2=0.94%, 4=36.88%, 10=52.21%, 20=9.60%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.97%, ctx=3601, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32556: Thu Jun 13 20:17:47 2024 read: IOPS=84, BW=337KiB/s (345kB/s)(5940KiB/17630msec) clat (usec): min=1644, max=158305, avg=4647.26, stdev=4847.80 lat (usec): min=1644, max=158307, avg=4648.00, stdev=4847.78 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2540], | 30.00th=[ 2835], 40.00th=[ 3195], 50.00th=[ 3621], 60.00th=[ 4080], | 70.00th=[ 4752], 80.00th=[ 5997], 90.00th=[ 8586], 95.00th=[ 10814], | 99.00th=[ 14615], 99.50th=[ 15533], 99.90th=[ 19006], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 128, max= 584, per=24.73%, avg=336.34, stdev=90.93, samples=35 iops : min= 32, max= 146, avg=84.00, stdev=22.74, samples=35 write: IOPS=90, BW=360KiB/s (369kB/s)(6348KiB/17630msec) clat (usec): min=2345, max=30474, avg=6713.96, stdev=3301.38 lat (usec): min=2345, max=30475, avg=6714.81, stdev=3301.38 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3392], 10.00th=[ 3621], 20.00th=[ 4146], | 30.00th=[ 4621], 40.00th=[ 5276], 50.00th=[ 5932], 60.00th=[ 6587], | 70.00th=[ 7308], 80.00th=[ 8717], 90.00th=[10945], 95.00th=[12911], | 99.00th=[19792], 99.50th=[21365], 99.90th=[25035], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 200, max= 608, per=25.18%, avg=358.29, stdev=84.60, samples=35 iops : min= 50, max= 152, avg=89.49, stdev=21.19, samples=35 lat (msec) : 2=1.33%, 4=36.07%, 10=52.08%, 20=10.09%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.89%, ctx=3602, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32558: Thu Jun 13 20:17:47 2024 read: IOPS=86, BW=345KiB/s (353kB/s)(6028KiB/17469msec) clat (usec): min=1651, max=21518, avg=4660.56, stdev=2821.39 lat (usec): min=1651, max=21519, avg=4661.29, stdev=2821.36 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2089], 10.00th=[ 2278], 20.00th=[ 2573], | 30.00th=[ 2900], 40.00th=[ 3261], 50.00th=[ 3687], 60.00th=[ 4178], | 70.00th=[ 4948], 80.00th=[ 6325], 90.00th=[ 8979], 95.00th=[10683], | 99.00th=[14222], 99.50th=[15533], 99.90th=[18220], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 144, max= 632, per=25.20%, avg=342.71, stdev=90.92, samples=34 iops : min= 36, max= 158, avg=85.59, stdev=22.76, samples=34 write: IOPS=89, BW=358KiB/s (367kB/s)(6260KiB/17469msec) clat (msec): min=2, max=164, avg= 6.63, stdev= 5.13 lat (msec): min=2, max=164, avg= 6.63, stdev= 5.13 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 21], 99.90th=[ 26], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 176, max= 656, per=24.93%, avg=354.71, stdev=93.85, samples=34 iops : min= 44, max= 164, avg=88.59, stdev=23.47, samples=34 lat (msec) : 2=1.40%, 4=37.60%, 10=51.04%, 20=9.60%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.87%, ctx=3579, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32559: Thu Jun 13 20:17:47 2024 read: IOPS=85, BW=343KiB/s (351kB/s)(6052KiB/17655msec) clat (usec): min=1631, max=21533, avg=4725.51, stdev=2992.54 lat (usec): min=1632, max=21533, avg=4726.23, stdev=2992.49 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2114], 10.00th=[ 2311], 20.00th=[ 2638], | 30.00th=[ 2900], 40.00th=[ 3261], 50.00th=[ 3654], 60.00th=[ 4113], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 9110], 95.00th=[11469], | 99.00th=[15401], 99.50th=[16712], 99.90th=[20055], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 152, max= 600, per=25.08%, avg=341.14, stdev=94.82, samples=35 iops : min= 38, max= 150, avg=85.20, stdev=23.72, samples=35 write: IOPS=88, BW=353KiB/s (362kB/s)(6236KiB/17655msec) clat (msec): min=2, max=165, avg= 6.69, stdev= 5.13 lat (msec): min=2, max=165, avg= 6.69, stdev= 5.13 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 22], 99.90th=[ 31], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 168, max= 576, per=24.78%, avg=352.57, stdev=85.63, samples=35 iops : min= 42, max= 144, avg=88.06, stdev=21.41, samples=35 lat (msec) : 2=1.56%, 4=36.65%, 10=51.76%, 20=9.64%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.87%, ctx=3587, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1360KiB/s (1393kB/s), 337KiB/s-345KiB/s (345kB/s-353kB/s), io=23.4MiB (24.6MB), run=17469-17655msec WRITE: bw=1424KiB/s (1458kB/s), 353KiB/s-360KiB/s (362kB/s-369kB/s), io=24.6MiB (25.7MB), run=17469-17655msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32588: Thu Jun 13 20:17:53 2024 read: IOPS=64, BW=1026KiB/s (1050kB/s)(5920KiB/5772msec) clat (usec): min=59, max=53879, avg=11170.97, stdev=4967.99 lat (usec): min=59, max=53880, avg=11174.53, stdev=4967.94 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 76], 10.00th=[ 7635], 20.00th=[ 8979], | 30.00th=[ 9765], 40.00th=[10421], 50.00th=[11076], 60.00th=[11863], | 70.00th=[12780], 80.00th=[13698], 90.00th=[15401], 95.00th=[17171], | 99.00th=[24511], 99.50th=[35914], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 734, max= 1152, per=23.74%, avg=982.73, stdev=139.38, samples=11 iops : min= 45, max= 72, avg=61.18, stdev= 8.96, samples=11 write: IOPS=68, BW=1103KiB/s (1130kB/s)(6368KiB/5772msec) clat (usec): min=657, max=26550, avg=4093.18, stdev=3758.86 lat (usec): min=658, max=26551, avg=4094.16, stdev=3758.89 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 816], 10.00th=[ 873], 20.00th=[ 1004], | 30.00th=[ 2245], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5604], 90.00th=[ 8979], 95.00th=[13566], | 99.00th=[16450], 99.50th=[21103], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 672, max= 1312, per=24.50%, avg=1066.91, stdev=212.01, samples=11 iops : min= 42, max= 82, avg=66.45, stdev=13.18, samples=11 lat (usec) : 100=2.99%, 250=0.65%, 750=0.39%, 1000=9.64% lat (msec) : 2=4.17%, 4=20.44%, 10=25.65%, 20=34.77%, 50=1.17% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=21.43%, ctx=2684, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32589: Thu Jun 13 20:17:53 2024 read: IOPS=62, BW=1002KiB/s (1026kB/s)(5744KiB/5732msec) clat (usec): min=62, max=28598, avg=11261.21, stdev=3737.06 lat (usec): min=62, max=28599, avg=11263.60, stdev=3731.84 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 7111], 10.00th=[ 8094], 20.00th=[ 9241], | 30.00th=[ 9896], 40.00th=[10552], 50.00th=[11076], 60.00th=[11731], | 70.00th=[12649], 80.00th=[13698], 90.00th=[14877], 95.00th=[16909], | 99.00th=[23462], 99.50th=[24511], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 640, max= 1216, per=23.75%, avg=982.91, stdev=165.39, samples=11 iops : min= 40, max= 76, avg=61.27, stdev=10.47, samples=11 write: IOPS=71, BW=1142KiB/s (1169kB/s)(6544KiB/5732msec) clat (usec): min=645, max=82588, avg=4107.97, stdev=5353.61 lat (usec): min=646, max=82590, avg=4108.99, stdev=5353.59 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 799], 10.00th=[ 840], 20.00th=[ 1004], | 30.00th=[ 2114], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3228], | 70.00th=[ 4080], 80.00th=[ 5276], 90.00th=[ 9896], 95.00th=[12649], | 99.00th=[17171], 99.50th=[18482], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 896, max= 1408, per=25.50%, avg=1110.73, stdev=185.60, samples=11 iops : min= 56, max= 88, avg=69.27, stdev=11.47, samples=11 lat (usec) : 100=1.30%, 250=0.65%, 750=0.65%, 1000=9.77% lat (msec) : 2=5.21%, 4=21.48%, 10=23.44%, 20=36.59%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=20.83%, ctx=2625, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32590: Thu Jun 13 20:17:53 2024 read: IOPS=67, BW=1084KiB/s (1110kB/s)(6256KiB/5773msec) clat (usec): min=50, max=27736, avg=10881.28, stdev=4300.74 lat (usec): min=51, max=27737, avg=10881.90, stdev=4300.76 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 89], 10.00th=[ 7504], 20.00th=[ 8979], | 30.00th=[ 9765], 40.00th=[10552], 50.00th=[11207], 60.00th=[11731], | 70.00th=[12518], 80.00th=[13435], 90.00th=[14877], 95.00th=[16909], | 99.00th=[21627], 99.50th=[25297], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 638, max= 1504, per=25.28%, avg=1046.55, stdev=218.15, samples=11 iops : min= 39, max= 94, avg=65.18, stdev=13.74, samples=11 write: IOPS=65, BW=1045KiB/s (1070kB/s)(6032KiB/5773msec) clat (usec): min=650, max=88431, avg=4003.46, stdev=5606.65 lat (usec): min=651, max=88433, avg=4005.99, stdev=5606.95 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 791], 10.00th=[ 832], 20.00th=[ 963], | 30.00th=[ 1991], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3294], | 70.00th=[ 3949], 80.00th=[ 5276], 90.00th=[ 7439], 95.00th=[12387], | 99.00th=[19006], 99.50th=[23200], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 638, max= 1280, per=22.70%, avg=988.55, stdev=209.84, samples=11 iops : min= 39, max= 80, avg=61.55, stdev=13.45, samples=11 lat (usec) : 100=3.26%, 250=0.91%, 750=1.17%, 1000=9.11% lat (msec) : 2=4.56%, 4=19.53%, 10=23.83%, 20=36.20%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=20.79%, ctx=2646, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32591: Thu Jun 13 20:17:53 2024 read: IOPS=65, BW=1043KiB/s (1068kB/s)(6032KiB/5786msec) clat (usec): min=49, max=26350, avg=10977.35, stdev=4063.57 lat (usec): min=50, max=26351, avg=10978.93, stdev=4066.94 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 92], 10.00th=[ 7767], 20.00th=[ 9110], | 30.00th=[ 9765], 40.00th=[10552], 50.00th=[11207], 60.00th=[11731], | 70.00th=[12518], 80.00th=[13435], 90.00th=[14877], 95.00th=[16909], | 99.00th=[23200], 99.50th=[24511], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 606, max= 1344, per=24.02%, avg=994.27, stdev=175.31, samples=11 iops : min= 37, max= 84, avg=61.91, stdev=11.15, samples=11 write: IOPS=67, BW=1081KiB/s (1107kB/s)(6256KiB/5786msec) clat (usec): min=699, max=87514, avg=4184.22, stdev=5587.04 lat (usec): min=700, max=87515, avg=4185.12, stdev=5587.08 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 807], 10.00th=[ 840], 20.00th=[ 947], | 30.00th=[ 1516], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3425], | 70.00th=[ 4490], 80.00th=[ 5866], 90.00th=[ 8291], 95.00th=[12256], | 99.00th=[19530], 99.50th=[21103], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 766, max= 1280, per=24.03%, avg=1046.64, stdev=173.77, samples=11 iops : min= 47, max= 80, avg=65.18, stdev=10.99, samples=11 lat (usec) : 50=0.13%, 100=2.60%, 250=0.65%, 750=0.65%, 1000=11.46% lat (msec) : 2=3.52%, 4=17.84%, 10=26.43%, 20=35.55%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.02%, sys=20.95%, ctx=2646, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4140KiB/s (4239kB/s), 1002KiB/s-1084KiB/s (1026kB/s-1110kB/s), io=23.4MiB (24.5MB), run=5732-5786msec WRITE: bw=4355KiB/s (4460kB/s), 1045KiB/s-1142KiB/s (1070kB/s-1169kB/s), io=24.6MiB (25.8MB), run=5732-5786msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32583: Thu Jun 13 20:17:53 2024 read: IOPS=62, BW=1005KiB/s (1029kB/s)(5920KiB/5890msec) clat (usec): min=1880, max=28076, avg=5994.11, stdev=4363.45 lat (usec): min=1881, max=28076, avg=5994.74, stdev=4363.46 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2147], 10.00th=[ 2343], 20.00th=[ 2638], | 30.00th=[ 2933], 40.00th=[ 3490], 50.00th=[ 4080], 60.00th=[ 5342], | 70.00th=[ 6980], 80.00th=[ 9372], 90.00th=[12387], 95.00th=[14484], | 99.00th=[20579], 99.50th=[24773], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 832, max= 1184, per=24.04%, avg=977.27, stdev=123.96, samples=11 iops : min= 52, max= 74, avg=61.00, stdev= 7.63, samples=11 write: IOPS=67, BW=1081KiB/s (1107kB/s)(6368KiB/5890msec) clat (usec): min=2684, max=96228, avg=9212.19, stdev=6495.89 lat (usec): min=2685, max=96230, avg=9213.27, stdev=6495.97 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3392], 10.00th=[ 3720], 20.00th=[ 4490], | 30.00th=[ 5407], 40.00th=[ 6980], 50.00th=[ 8356], 60.00th=[ 9503], | 70.00th=[10814], 80.00th=[12125], 90.00th=[15533], 95.00th=[18220], | 99.00th=[27919], 99.50th=[31065], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 864, max= 1248, per=24.65%, avg=1054.64, stdev=159.20, samples=11 iops : min= 54, max= 78, avg=65.91, stdev= 9.94, samples=11 lat (msec) : 2=0.39%, 4=29.56%, 10=43.88%, 20=23.70%, 50=2.34% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=10.46%, ctx=1142, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32584: Thu Jun 13 20:17:53 2024 read: IOPS=66, BW=1058KiB/s (1083kB/s)(5744KiB/5429msec) clat (usec): min=1871, max=39469, avg=5735.47, stdev=4439.41 lat (usec): min=1871, max=39471, avg=5736.06, stdev=4439.44 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2835], 40.00th=[ 3130], 50.00th=[ 3982], 60.00th=[ 4948], | 70.00th=[ 6652], 80.00th=[ 8848], 90.00th=[11338], 95.00th=[14615], | 99.00th=[19268], 99.50th=[27132], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 704, max= 1312, per=26.21%, avg=1065.60, stdev=190.84, samples=10 iops : min= 44, max= 82, avg=66.60, stdev=11.93, samples=10 write: IOPS=75, BW=1205KiB/s (1234kB/s)(6544KiB/5429msec) clat (usec): min=2637, max=27616, avg=8223.03, stdev=4909.45 lat (usec): min=2638, max=27617, avg=8224.40, stdev=4909.13 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3326], 10.00th=[ 3621], 20.00th=[ 4080], | 30.00th=[ 4752], 40.00th=[ 5800], 50.00th=[ 6652], 60.00th=[ 7767], | 70.00th=[ 9241], 80.00th=[12125], 90.00th=[16057], 95.00th=[18220], | 99.00th=[23725], 99.50th=[25560], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 928, max= 1504, per=27.98%, avg=1196.80, stdev=174.75, samples=10 iops : min= 58, max= 94, avg=74.80, stdev=10.92, samples=10 lat (msec) : 2=0.65%, 4=32.03%, 10=45.44%, 20=19.92%, 50=1.95% cpu : usr=0.11%, sys=9.60%, ctx=918, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32585: Thu Jun 13 20:17:53 2024 read: IOPS=72, BW=1164KiB/s (1192kB/s)(6256KiB/5375msec) clat (usec): min=1982, max=46213, avg=5862.33, stdev=4797.78 lat (usec): min=1982, max=46214, avg=5863.09, stdev=4797.69 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2245], 10.00th=[ 2442], 20.00th=[ 2638], | 30.00th=[ 2868], 40.00th=[ 3228], 50.00th=[ 3818], 60.00th=[ 4752], | 70.00th=[ 6718], 80.00th=[ 8979], 90.00th=[11863], 95.00th=[14615], | 99.00th=[24511], 99.50th=[29230], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 704, max= 1632, per=28.65%, avg=1164.80, stdev=317.21, samples=10 iops : min= 44, max= 102, avg=72.80, stdev=19.83, samples=10 write: IOPS=70, BW=1122KiB/s (1149kB/s)(6032KiB/5375msec) clat (usec): min=2589, max=44064, avg=8160.62, stdev=5440.03 lat (usec): min=2590, max=44066, avg=8161.71, stdev=5440.38 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3359], 10.00th=[ 3556], 20.00th=[ 4080], | 30.00th=[ 4555], 40.00th=[ 5211], 50.00th=[ 6194], 60.00th=[ 7242], | 70.00th=[ 9372], 80.00th=[11994], 90.00th=[15795], 95.00th=[19530], | 99.00th=[25822], 99.50th=[31327], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 896, max= 1408, per=25.73%, avg=1100.80, stdev=179.88, samples=10 iops : min= 56, max= 88, avg=68.80, stdev=11.24, samples=10 lat (msec) : 2=0.26%, 4=35.42%, 10=42.84%, 20=18.62%, 50=2.86% cpu : usr=0.15%, sys=9.73%, ctx=937, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32586: Thu Jun 13 20:17:53 2024 read: IOPS=71, BW=1148KiB/s (1176kB/s)(6032KiB/5254msec) clat (usec): min=1880, max=34266, avg=5625.79, stdev=4053.32 lat (usec): min=1881, max=34267, avg=5626.40, stdev=4053.34 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2114], 10.00th=[ 2343], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3294], 50.00th=[ 3916], 60.00th=[ 5080], | 70.00th=[ 6587], 80.00th=[ 8717], 90.00th=[11338], 95.00th=[14222], | 99.00th=[18744], 99.50th=[21890], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 864, max= 1472, per=27.94%, avg=1136.00, stdev=180.55, samples=10 iops : min= 54, max= 92, avg=71.00, stdev=11.28, samples=10 write: IOPS=74, BW=1191KiB/s (1219kB/s)(6256KiB/5254msec) clat (usec): min=2498, max=64592, avg=7998.15, stdev=5700.44 lat (usec): min=2499, max=64594, avg=7999.13, stdev=5700.48 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3425], 10.00th=[ 3621], 20.00th=[ 3982], | 30.00th=[ 4490], 40.00th=[ 4948], 50.00th=[ 5735], 60.00th=[ 7046], | 70.00th=[ 9241], 80.00th=[11207], 90.00th=[15926], 95.00th=[19792], | 99.00th=[23200], 99.50th=[24511], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 960, max= 1440, per=27.68%, avg=1184.00, stdev=153.84, samples=10 iops : min= 60, max= 90, avg=74.00, stdev= 9.61, samples=10 lat (msec) : 2=0.91%, 4=34.90%, 10=43.49%, 20=18.23%, 50=2.34% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=9.82%, ctx=939, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4067KiB/s (4164kB/s), 1005KiB/s-1164KiB/s (1029kB/s-1192kB/s), io=23.4MiB (24.5MB), run=5254-5890msec WRITE: bw=4278KiB/s (4381kB/s), 1081KiB/s-1205KiB/s (1107kB/s-1234kB/s), io=24.6MiB (25.8MB), run=5254-5890msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32601: Thu Jun 13 20:17:56 2024 read: IOPS=22, BW=22.2MiB/s (23.2MB/s)(7168KiB/316msec) clat (usec): min=13030, max=40123, avg=26479.72, stdev=10539.17 lat (usec): min=13031, max=40124, avg=26480.72, stdev=10539.04 clat percentiles (usec): | 1.00th=[13042], 5.00th=[13042], 10.00th=[13042], 20.00th=[16909], | 30.00th=[22152], 40.00th=[22152], 50.00th=[22938], 60.00th=[31327], | 70.00th=[31327], 80.00th=[39060], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] write: IOPS=15, BW=15.8MiB/s (16.6MB/s)(5120KiB/316msec) clat (usec): min=14498, max=37804, avg=24421.07, stdev=10655.07 lat (usec): min=14536, max=37834, avg=24468.74, stdev=10652.14 clat percentiles (usec): | 1.00th=[14484], 5.00th=[14484], 10.00th=[14484], 20.00th=[14484], | 30.00th=[15533], 40.00th=[15533], 50.00th=[20579], 60.00th=[20579], | 70.00th=[33817], 80.00th=[33817], 90.00th=[38011], 95.00th=[38011], | 99.00th=[38011], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] lat (msec) : 20=33.33%, 50=66.67% cpu : usr=0.00%, sys=28.25%, ctx=57, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32602: Thu Jun 13 20:17:56 2024 read: IOPS=1, BW=1142KiB/s (1170kB/s)(2048KiB/1793msec) clat (msec): min=73, max=718, avg=396.25, stdev=456.29 lat (msec): min=73, max=718, avg=396.25, stdev=456.29 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 718], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 1715, max= 1715, per=15.99%, avg=1715.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5711KiB/s (5848kB/s)(10.0MiB/1793msec) clat (msec): min=16, max=632, avg=98.26, stdev=189.11 lat (msec): min=16, max=632, avg=98.30, stdev=189.11 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 29], 60.00th=[ 50], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 74], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 5145, max= 8576, per=45.68%, avg=6860.50, stdev=2426.08, samples=2 iops : min= 5, max= 8, avg= 6.50, stdev= 2.12, samples=2 lat (msec) : 20=33.33%, 50=16.67%, 100=33.33%, 750=16.67% cpu : usr=0.00%, sys=8.98%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32603: Thu Jun 13 20:17:56 2024 read: IOPS=3, BW=3218KiB/s (3296kB/s)(6144KiB/1909msec) clat (msec): min=18, max=493, avg=124.97, stdev=182.12 lat (msec): min=18, max=493, avg=124.97, stdev=182.12 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 66], | 70.00th=[ 87], 80.00th=[ 87], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 1709, max= 5145, per=31.94%, avg=3427.00, stdev=2429.62, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3218KiB/s (3296kB/s)(6144KiB/1909msec) clat (msec): min=19, max=718, avg=190.32, stdev=263.23 lat (msec): min=19, max=718, avg=190.37, stdev=263.24 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 129], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 3430, max= 3430, per=22.84%, avg=3430.00, stdev= 0.00, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 lat (msec) : 20=16.67%, 50=16.67%, 100=25.00%, 250=25.00%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.05%, sys=6.39%, ctx=44, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32604: Thu Jun 13 20:17:56 2024 read: IOPS=16, BW=16.0MiB/s (16.8MB/s)(5120KiB/312msec) clat (usec): min=13356, max=44856, avg=26309.68, stdev=14250.03 lat (usec): min=13357, max=44857, avg=26310.47, stdev=14250.12 clat percentiles (usec): | 1.00th=[13304], 5.00th=[13304], 10.00th=[13304], 20.00th=[13304], | 30.00th=[16450], 40.00th=[16450], 50.00th=[18482], 60.00th=[18482], | 70.00th=[38536], 80.00th=[38536], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] write: IOPS=22, BW=22.4MiB/s (23.5MB/s)(7168KiB/312msec) clat (usec): min=14368, max=44946, avg=24590.33, stdev=10390.82 lat (usec): min=14394, max=44981, avg=24638.95, stdev=10391.87 clat percentiles (usec): | 1.00th=[14353], 5.00th=[14353], 10.00th=[14353], 20.00th=[17695], | 30.00th=[18220], 40.00th=[18220], 50.00th=[20841], 60.00th=[27395], | 70.00th=[27395], 80.00th=[28705], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] lat (msec) : 20=50.00%, 50=50.00% cpu : usr=0.00%, sys=26.05%, ctx=42, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.5MiB/s (10.0MB/s), 1142KiB/s-22.2MiB/s (1170kB/s-23.2MB/s), io=20.0MiB (20.0MB), run=312-1909msec WRITE: bw=14.7MiB/s (15.4MB/s), 3218KiB/s-22.4MiB/s (3296kB/s-23.5MB/s), io=28.0MiB (29.4MB), run=312-1909msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32605: Thu Jun 13 20:17:58 2024 read: IOPS=2, BW=2215KiB/s (2268kB/s)(7168KiB/3236msec) clat (msec): min=37, max=777, avg=424.73, stdev=337.91 lat (msec): min=37, max=777, avg=424.74, stdev=337.91 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 79], | 30.00th=[ 95], 40.00th=[ 95], 50.00th=[ 575], 60.00th=[ 667], | 70.00th=[ 667], 80.00th=[ 743], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 1741, max= 3482, per=35.87%, avg=2269.80, stdev=690.08, samples=5 iops : min= 1, max= 3, avg= 1.60, stdev= 0.89, samples=5 write: IOPS=1, BW=1582KiB/s (1620kB/s)(5120KiB/3236msec) clat (usec): min=24981, max=68427, avg=52053.29, stdev=17653.69 lat (usec): min=25052, max=68497, avg=52117.46, stdev=17643.42 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[45351], 40.00th=[45351], 50.00th=[56361], 60.00th=[56361], | 70.00th=[65274], 80.00th=[65274], 90.00th=[68682], 95.00th=[68682], | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 1741, max= 5224, per=33.86%, avg=3000.00, stdev=1931.64, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (msec) : 50=25.00%, 100=41.67%, 750=25.00%, 1000=8.33% cpu : usr=0.00%, sys=24.91%, ctx=1265, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32606: Thu Jun 13 20:17:58 2024 read: IOPS=1, BW=1320KiB/s (1352kB/s)(2048KiB/1551msec) clat (msec): min=111, max=517, avg=314.32, stdev=287.24 lat (msec): min=111, max=517, avg=314.32, stdev=287.24 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 518], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 2043, max= 2048, per=32.32%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6602KiB/s (6761kB/s)(10.0MiB/1551msec) clat (msec): min=24, max=618, avg=92.05, stdev=185.17 lat (msec): min=24, max=618, avg=92.12, stdev=185.17 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 29], 60.00th=[ 30], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 51], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 6144, max=10219, per=92.34%, avg=8181.50, stdev=2881.46, samples=2 iops : min= 6, max= 9, avg= 7.50, stdev= 2.12, samples=2 lat (msec) : 50=66.67%, 100=8.33%, 250=8.33%, 750=16.67% cpu : usr=0.13%, sys=21.16%, ctx=307, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32607: Thu Jun 13 20:17:58 2024 read: IOPS=3, BW=3790KiB/s (3881kB/s)(6144KiB/1621msec) clat (usec): min=363, max=597623, avg=112523.30, stdev=238494.26 lat (usec): min=364, max=597624, avg=112524.15, stdev=238494.05 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 371], | 30.00th=[ 371], 40.00th=[ 392], 50.00th=[ 392], 60.00th=[ 26084], | 70.00th=[ 50070], 80.00th=[ 50070], 90.00th=[599786], 95.00th=[599786], | 99.00th=[599786], 99.50th=[599786], 99.90th=[599786], 99.95th=[599786], | 99.99th=[599786] bw ( KiB/s): min= 6144, max= 6144, per=97.09%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3790KiB/s (3881kB/s)(6144KiB/1621msec) clat (msec): min=18, max=771, avg=157.21, stdev=301.03 lat (msec): min=18, max=771, avg=157.26, stdev=301.04 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 47], | 70.00th=[ 52], 80.00th=[ 52], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 4087, max= 4096, per=46.18%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (usec) : 500=25.00% lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=16.05%, ctx=325, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32608: Thu Jun 13 20:17:58 2024 read: IOPS=3, BW=3198KiB/s (3275kB/s)(5120KiB/1601msec) clat (usec): min=350, max=607381, avg=126645.59, stdev=268943.10 lat (usec): min=351, max=607381, avg=126646.17, stdev=268942.99 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 429], 40.00th=[ 429], 50.00th=[ 490], 60.00th=[ 490], | 70.00th=[ 24511], 80.00th=[ 24511], 90.00th=[608175], 95.00th=[608175], | 99.00th=[608175], 99.50th=[608175], 99.90th=[608175], 99.95th=[608175], | 99.99th=[608175] bw ( KiB/s): min= 2048, max= 2048, per=32.36%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4477KiB/s (4585kB/s)(7168KiB/1601msec) clat (msec): min=18, max=796, avg=137.95, stdev=290.73 lat (msec): min=18, max=796, avg=137.99, stdev=290.74 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 38], | 70.00th=[ 38], 80.00th=[ 55], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 2048, max= 6144, per=46.23%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=25.00% lat (msec) : 20=25.00%, 50=25.00%, 100=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=14.30%, ctx=300, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6329KiB/s (6481kB/s), 1320KiB/s-3790KiB/s (1352kB/s-3881kB/s), io=20.0MiB (20.0MB), run=1551-3236msec WRITE: bw=8860KiB/s (9073kB/s), 1582KiB/s-6602KiB/s (1620kB/s-6761kB/s), io=28.0MiB (29.4MB), run=1551-3236msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32621: Thu Jun 13 20:18:02 2024 read: IOPS=1, BW=4435KiB/s (4541kB/s)(12.0MiB/2771msec) clat (msec): min=228, max=1713, avg=923.23, stdev=746.97 lat (msec): min=228, max=1713, avg=923.24, stdev=746.97 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 827], 50.00th=[ 827], 60.00th=[ 827], | 70.00th=[ 1720], 80.00th=[ 1720], 90.00th=[ 1720], 95.00th=[ 1720], | 99.00th=[ 1720], 99.50th=[ 1720], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=38.25%, ctx=878, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32622: Thu Jun 13 20:18:02 2024 read: IOPS=0, BW=1775KiB/s (1818kB/s)(4096KiB/2307msec) clat (nsec): min=183655k, max=183655k, avg=183655485.00, stdev= 0.00 lat (nsec): min=183658k, max=183658k, avg=183658269.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3551KiB/s (3636kB/s)(8192KiB/2307msec) clat (msec): min=234, max=1866, avg=1050.91, stdev=1153.91 lat (msec): min=235, max=1866, avg=1051.11, stdev=1153.81 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 1871], | 70.00th=[ 1871], 80.00th=[ 1871], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] lat (msec) : 250=66.67% cpu : usr=0.04%, sys=13.92%, ctx=258, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32623: Thu Jun 13 20:18:02 2024 write: IOPS=1, BW=4147KiB/s (4247kB/s)(12.0MiB/2963msec) clat (msec): min=298, max=1756, avg=986.14, stdev=732.35 lat (msec): min=300, max=1756, avg=986.98, stdev=731.40 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 902], 50.00th=[ 902], 60.00th=[ 902], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 8192, max= 8192, per=74.08%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=13.47%, ctx=306, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32624: Thu Jun 13 20:18:02 2024 write: IOPS=1, BW=5420KiB/s (5550kB/s)(12.0MiB/2267msec) clat (msec): min=184, max=1811, avg=754.20, stdev=916.23 lat (msec): min=184, max=1811, avg=754.41, stdev=916.27 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 1804], 80.00th=[ 1804], 90.00th=[ 1804], 95.00th=[ 1804], | 99.00th=[ 1804], 99.50th=[ 1804], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 8192, max= 8192, per=74.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=13.23%, ctx=72, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5913KiB/s (6055kB/s), 1775KiB/s-4435KiB/s (1818kB/s-4541kB/s), io=16.0MiB (16.8MB), run=2307-2771msec WRITE: bw=10.8MiB/s (11.3MB/s), 3551KiB/s-5420KiB/s (3636kB/s-5550kB/s), io=32.0MiB (33.6MB), run=2267-2963msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32617: Thu Jun 13 20:18:02 2024 read: IOPS=2, BW=10.0MiB/s (10.5MB/s)(12.0MiB/1200msec) clat (msec): min=93, max=943, avg=396.63, stdev=474.38 lat (msec): min=93, max=943, avg=396.63, stdev=474.38 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 94], | 30.00th=[ 94], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min=14867, max=14867, per=100.00%, avg=14867.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.01%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32618: Thu Jun 13 20:18:02 2024 read: IOPS=0, BW=1304KiB/s (1336kB/s)(4096KiB/3140msec) clat (nsec): min=391902k, max=391902k, avg=391901833.00, stdev= 0.00 lat (nsec): min=391903k, max=391903k, avg=391903206.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 5475, max= 5475, per=100.00%, avg=5475.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2609KiB/s (2672kB/s)(8192KiB/3140msec) clat (msec): min=314, max=2423, avg=1368.88, stdev=1490.83 lat (msec): min=315, max=2423, avg=1369.13, stdev=1490.75 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 2433], | 70.00th=[ 2433], 80.00th=[ 2433], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] lat (msec) : 500=66.67% cpu : usr=0.03%, sys=5.00%, ctx=43, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32619: Thu Jun 13 20:18:02 2024 write: IOPS=0, BW=3930KiB/s (4024kB/s)(12.0MiB/3127msec) clat (msec): min=366, max=1854, avg=1041.72, stdev=753.73 lat (msec): min=366, max=1854, avg=1041.92, stdev=753.79 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 902], 50.00th=[ 902], 60.00th=[ 902], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8143, max= 8143, per=78.04%, avg=8143.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=4.86%, ctx=58, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32620: Thu Jun 13 20:18:02 2024 write: IOPS=6, BW=24.4MiB/s (25.6MB/s)(12.0MiB/492msec) clat (msec): min=51, max=353, avg=157.77, stdev=170.06 lat (msec): min=52, max=354, avg=158.07, stdev=170.03 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 68], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.20%, sys=15.89%, ctx=153, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5218KiB/s (5343kB/s), 1304KiB/s-10.0MiB/s (1336kB/s-10.5MB/s), io=16.0MiB (16.8MB), run=1200-3140msec WRITE: bw=10.2MiB/s (10.7MB/s), 2609KiB/s-24.4MiB/s (2672kB/s-25.6MB/s), io=32.0MiB (33.6MB), run=492-3140msec