== sanity test 398b: DIO and buffer IO race ============== 02:22:47 (1721888567) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.24949 s, 40.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5643: Thu Jul 25 02:23:01 2024 read: IOPS=135, BW=541KiB/s (554kB/s)(5992KiB/11082msec) clat (usec): min=48, max=48021, avg=4104.77, stdev=2689.03 lat (usec): min=48, max=48022, avg=4105.42, stdev=2689.00 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 265], 10.00th=[ 2008], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 3130], 50.00th=[ 3621], 60.00th=[ 4178], | 70.00th=[ 4817], 80.00th=[ 5538], 90.00th=[ 6783], 95.00th=[ 8094], | 99.00th=[13042], 99.50th=[15795], 99.90th=[28705], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 184, max= 712, per=24.95%, avg=538.59, stdev=108.16, samples=22 iops : min= 46, max= 178, avg=134.55, stdev=27.08, samples=22 write: IOPS=142, BW=568KiB/s (582kB/s)(6296KiB/11082msec) clat (usec): min=406, max=135790, avg=3081.03, stdev=4301.57 lat (usec): min=407, max=135791, avg=3082.78, stdev=4301.45 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 502], 10.00th=[ 519], 20.00th=[ 562], | 30.00th=[ 709], 40.00th=[ 1876], 50.00th=[ 2442], 60.00th=[ 3032], | 70.00th=[ 3752], 80.00th=[ 4883], 90.00th=[ 6521], 95.00th=[ 8094], | 99.00th=[ 12125], 99.50th=[ 15533], 99.90th=[ 20317], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 224, max= 704, per=24.96%, avg=564.36, stdev=101.58, samples=22 iops : min= 56, max= 176, avg=141.00, stdev=25.42, samples=22 lat (usec) : 50=0.07%, 100=2.18%, 250=0.16%, 500=2.47%, 750=13.51% lat (usec) : 1000=1.73% lat (msec) : 2=5.89%, 4=38.83%, 10=32.94%, 20=2.05%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.44%, ctx=5695, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5644: Thu Jul 25 02:23:01 2024 read: IOPS=134, BW=538KiB/s (551kB/s)(5940KiB/11049msec) clat (usec): min=48, max=20009, avg=4100.79, stdev=2223.07 lat (usec): min=48, max=20009, avg=4101.34, stdev=2223.08 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 1778], 10.00th=[ 2073], 20.00th=[ 2442], | 30.00th=[ 2737], 40.00th=[ 3195], 50.00th=[ 3687], 60.00th=[ 4293], | 70.00th=[ 4883], 80.00th=[ 5669], 90.00th=[ 6783], 95.00th=[ 7832], | 99.00th=[11600], 99.50th=[13173], 99.90th=[17695], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 200, max= 656, per=24.91%, avg=537.86, stdev=101.79, samples=22 iops : min= 50, max= 164, avg=134.36, stdev=25.49, samples=22 write: IOPS=143, BW=575KiB/s (588kB/s)(6348KiB/11049msec) clat (usec): min=435, max=174886, avg=3070.83, stdev=5155.40 lat (usec): min=435, max=174887, avg=3071.59, stdev=5155.42 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 494], 10.00th=[ 515], 20.00th=[ 545], | 30.00th=[ 652], 40.00th=[ 1434], 50.00th=[ 2442], 60.00th=[ 2966], | 70.00th=[ 3752], 80.00th=[ 4948], 90.00th=[ 6718], 95.00th=[ 8029], | 99.00th=[ 11076], 99.50th=[ 13042], 99.90th=[ 44303], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 256, max= 736, per=25.36%, avg=573.45, stdev=94.82, samples=22 iops : min= 64, max= 184, avg=143.27, stdev=23.69, samples=22 lat (usec) : 50=0.07%, 100=1.73%, 250=0.10%, 500=3.65%, 750=13.64% lat (usec) : 1000=1.95% lat (msec) : 2=5.70%, 4=37.17%, 10=34.24%, 20=1.66%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.51%, ctx=5488, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5645: Thu Jul 25 02:23:01 2024 read: IOPS=135, BW=542KiB/s (555kB/s)(6028KiB/11117msec) clat (usec): min=45, max=20100, avg=4060.79, stdev=2202.18 lat (usec): min=46, max=20100, avg=4062.22, stdev=2201.66 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 1713], 10.00th=[ 2040], 20.00th=[ 2409], | 30.00th=[ 2769], 40.00th=[ 3163], 50.00th=[ 3654], 60.00th=[ 4293], | 70.00th=[ 4817], 80.00th=[ 5538], 90.00th=[ 6783], 95.00th=[ 7832], | 99.00th=[11600], 99.50th=[12256], 99.90th=[18220], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 184, max= 704, per=25.02%, avg=540.09, stdev=97.23, samples=22 iops : min= 46, max= 176, avg=134.91, stdev=24.36, samples=22 write: IOPS=140, BW=563KiB/s (577kB/s)(6260KiB/11117msec) clat (usec): min=392, max=185674, avg=3137.52, stdev=5401.01 lat (usec): min=393, max=185675, avg=3138.42, stdev=5400.92 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 498], 10.00th=[ 515], 20.00th=[ 553], | 30.00th=[ 717], 40.00th=[ 1893], 50.00th=[ 2442], 60.00th=[ 3032], | 70.00th=[ 3851], 80.00th=[ 4948], 90.00th=[ 6587], 95.00th=[ 7963], | 99.00th=[ 12780], 99.50th=[ 15139], 99.90th=[ 35390], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 240, max= 728, per=24.70%, avg=558.55, stdev=113.71, samples=22 iops : min= 60, max= 182, avg=139.55, stdev=28.43, samples=22 lat (usec) : 50=0.13%, 100=2.08%, 250=0.10%, 500=3.09%, 750=12.57% lat (usec) : 1000=1.89% lat (msec) : 2=5.86%, 4=37.83%, 10=34.31%, 20=2.05%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.38%, ctx=5736, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5647: Thu Jul 25 02:23:01 2024 read: IOPS=136, BW=545KiB/s (558kB/s)(6052KiB/11097msec) clat (usec): min=45, max=177602, avg=4223.54, stdev=4999.64 lat (usec): min=45, max=177603, avg=4224.22, stdev=4999.63 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 1713], 10.00th=[ 2040], 20.00th=[ 2409], | 30.00th=[ 2704], 40.00th=[ 3130], 50.00th=[ 3720], 60.00th=[ 4359], | 70.00th=[ 4883], 80.00th=[ 5669], 90.00th=[ 6849], 95.00th=[ 8160], | 99.00th=[ 11338], 99.50th=[ 14353], 99.90th=[ 21103], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 240, max= 728, per=25.15%, avg=543.00, stdev=90.58, samples=22 iops : min= 60, max= 182, avg=135.68, stdev=22.64, samples=22 write: IOPS=140, BW=562KiB/s (575kB/s)(6236KiB/11097msec) clat (usec): min=385, max=30109, avg=2967.76, stdev=2754.34 lat (usec): min=386, max=30110, avg=2968.43, stdev=2754.37 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 490], 10.00th=[ 510], 20.00th=[ 545], | 30.00th=[ 660], 40.00th=[ 1516], 50.00th=[ 2343], 60.00th=[ 2868], | 70.00th=[ 3687], 80.00th=[ 5145], 90.00th=[ 6915], 95.00th=[ 8455], | 99.00th=[10945], 99.50th=[12256], 99.90th=[15926], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 296, max= 832, per=24.80%, avg=560.82, stdev=105.54, samples=22 iops : min= 74, max= 208, avg=140.14, stdev=26.40, samples=22 lat (usec) : 50=0.13%, 100=1.89%, 250=0.03%, 500=3.55%, 750=13.35% lat (usec) : 1000=2.05% lat (msec) : 2=5.99%, 4=36.98%, 10=34.08%, 20=1.86%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.21%, ctx=5496, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2160KiB/s (2212kB/s), 538KiB/s-545KiB/s (551kB/s-558kB/s), io=23.4MiB (24.6MB), run=11049-11117msec WRITE: bw=2261KiB/s (2316kB/s), 562KiB/s-575KiB/s (575kB/s-588kB/s), io=24.6MiB (25.7MB), run=11049-11117msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5648: Thu Jul 25 02:23:05 2024 read: IOPS=96, BW=386KiB/s (395kB/s)(5992KiB/15520msec) clat (usec): min=1538, max=210105, avg=4160.12, stdev=6001.75 lat (usec): min=1539, max=210106, avg=4160.78, stdev=6001.75 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1893], 10.00th=[ 2008], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2638], 50.00th=[ 2933], 60.00th=[ 3326], | 70.00th=[ 3982], 80.00th=[ 5276], 90.00th=[ 8029], 95.00th=[ 10159], | 99.00th=[ 15139], 99.50th=[ 17171], 99.90th=[ 21365], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 136, max= 744, per=25.24%, avg=386.45, stdev=131.11, samples=31 iops : min= 34, max= 186, avg=96.55, stdev=32.75, samples=31 write: IOPS=101, BW=406KiB/s (415kB/s)(6296KiB/15520msec) clat (usec): min=2212, max=24094, avg=5854.95, stdev=2934.76 lat (usec): min=2213, max=24094, avg=5855.75, stdev=2934.74 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2933], 10.00th=[ 3163], 20.00th=[ 3589], | 30.00th=[ 4015], 40.00th=[ 4490], 50.00th=[ 4948], 60.00th=[ 5538], | 70.00th=[ 6521], 80.00th=[ 7832], 90.00th=[ 9765], 95.00th=[11600], | 99.00th=[15926], 99.50th=[19268], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 152, max= 686, per=25.28%, avg=405.29, stdev=124.51, samples=31 iops : min= 38, max= 171, avg=101.26, stdev=31.11, samples=31 lat (msec) : 2=4.52%, 4=44.86%, 10=43.36%, 20=7.00%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.57%, ctx=3504, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5649: Thu Jul 25 02:23:05 2024 read: IOPS=94, BW=379KiB/s (388kB/s)(5940KiB/15683msec) clat (usec): min=1479, max=162779, avg=4211.51, stdev=4982.02 lat (usec): min=1480, max=162780, avg=4212.35, stdev=4981.98 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1844], 10.00th=[ 1991], 20.00th=[ 2278], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2966], 60.00th=[ 3392], | 70.00th=[ 3982], 80.00th=[ 5538], 90.00th=[ 8455], 95.00th=[ 10552], | 99.00th=[ 13960], 99.50th=[ 15926], 99.90th=[ 20317], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 80, max= 720, per=24.80%, avg=379.74, stdev=139.17, samples=31 iops : min= 20, max= 180, avg=94.84, stdev=34.86, samples=31 write: IOPS=101, BW=405KiB/s (414kB/s)(6348KiB/15683msec) clat (usec): min=2198, max=49537, avg=5893.21, stdev=3240.16 lat (usec): min=2198, max=49537, avg=5894.08, stdev=3240.20 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2966], 10.00th=[ 3228], 20.00th=[ 3654], | 30.00th=[ 4047], 40.00th=[ 4424], 50.00th=[ 4817], 60.00th=[ 5473], | 70.00th=[ 6390], 80.00th=[ 7701], 90.00th=[ 9765], 95.00th=[11863], | 99.00th=[19268], 99.50th=[20317], 99.90th=[26608], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 152, max= 688, per=25.22%, avg=404.26, stdev=120.95, samples=31 iops : min= 38, max= 172, avg=100.97, stdev=30.28, samples=31 lat (msec) : 2=4.85%, 4=44.40%, 10=43.13%, 20=7.26%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.68%, ctx=3490, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5650: Thu Jul 25 02:23:05 2024 read: IOPS=97, BW=391KiB/s (401kB/s)(6028KiB/15408msec) clat (usec): min=1460, max=21017, avg=4023.23, stdev=2729.12 lat (usec): min=1461, max=21017, avg=4023.90, stdev=2729.08 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1909], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2474], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3359], | 70.00th=[ 3884], 80.00th=[ 5276], 90.00th=[ 7832], 95.00th=[ 9896], | 99.00th=[13829], 99.50th=[16581], 99.90th=[20841], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 112, max= 678, per=25.60%, avg=391.97, stdev=129.01, samples=30 iops : min= 28, max= 169, avg=97.80, stdev=32.35, samples=30 write: IOPS=101, BW=406KiB/s (416kB/s)(6260KiB/15408msec) clat (msec): min=2, max=184, avg= 5.93, stdev= 5.52 lat (msec): min=2, max=184, avg= 5.93, stdev= 5.52 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 20], 99.50th=[ 22], 99.90th=[ 35], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 136, max= 736, per=25.33%, avg=406.10, stdev=132.00, samples=30 iops : min= 34, max= 184, avg=101.33, stdev=33.00, samples=30 lat (msec) : 2=4.39%, 4=47.30%, 10=41.11%, 20=6.71%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.75%, ctx=3519, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5651: Thu Jul 25 02:23:05 2024 read: IOPS=96, BW=388KiB/s (397kB/s)(6052KiB/15618msec) clat (usec): min=1559, max=21371, avg=4030.88, stdev=2729.13 lat (usec): min=1560, max=21372, avg=4031.56, stdev=2729.11 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1876], 10.00th=[ 2024], 20.00th=[ 2245], | 30.00th=[ 2474], 40.00th=[ 2704], 50.00th=[ 2966], 60.00th=[ 3359], | 70.00th=[ 3982], 80.00th=[ 5276], 90.00th=[ 8029], 95.00th=[10028], | 99.00th=[13566], 99.50th=[16319], 99.90th=[20055], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 144, max= 752, per=25.30%, avg=387.39, stdev=131.68, samples=31 iops : min= 36, max= 188, avg=96.71, stdev=32.95, samples=31 write: IOPS=99, BW=399KiB/s (409kB/s)(6236KiB/15618msec) clat (msec): min=2, max=159, avg= 6.05, stdev= 5.02 lat (msec): min=2, max=159, avg= 6.05, stdev= 5.02 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 43], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 168, max= 712, per=24.89%, avg=399.00, stdev=123.77, samples=31 iops : min= 42, max= 178, avg=99.61, stdev=30.95, samples=31 lat (msec) : 2=4.33%, 4=44.30%, 10=43.82%, 20=7.16%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.58%, ctx=3502, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1531KiB/s (1568kB/s), 379KiB/s-391KiB/s (388kB/s-401kB/s), io=23.4MiB (24.6MB), run=15408-15683msec WRITE: bw=1603KiB/s (1641kB/s), 399KiB/s-406KiB/s (409kB/s-416kB/s), io=24.6MiB (25.7MB), run=15408-15683msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5676: Thu Jul 25 02:23:13 2024 read: IOPS=58, BW=935KiB/s (958kB/s)(5920KiB/6331msec) clat (usec): min=62, max=33880, avg=10978.93, stdev=4179.30 lat (usec): min=63, max=33881, avg=10979.54, stdev=4179.33 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 6915], 10.00th=[ 7570], 20.00th=[ 8717], | 30.00th=[ 9372], 40.00th=[10028], 50.00th=[10552], 60.00th=[11076], | 70.00th=[11731], 80.00th=[12780], 90.00th=[14615], 95.00th=[16909], | 99.00th=[28443], 99.50th=[32375], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 576, max= 1120, per=24.60%, avg=930.67, stdev=142.10, samples=12 iops : min= 36, max= 70, avg=58.17, stdev= 8.88, samples=12 write: IOPS=62, BW=1006KiB/s (1030kB/s)(6368KiB/6331msec) clat (usec): min=739, max=72908, avg=5585.85, stdev=6960.99 lat (usec): min=740, max=72909, avg=5589.14, stdev=6960.99 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 799], 10.00th=[ 848], 20.00th=[ 1631], | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3392], 60.00th=[ 4424], | 70.00th=[ 5604], 80.00th=[ 7701], 90.00th=[12387], 95.00th=[17171], | 99.00th=[36963], 99.50th=[47449], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 640, max= 1408, per=25.66%, avg=1021.33, stdev=208.48, samples=12 iops : min= 40, max= 88, avg=63.83, stdev=13.03, samples=12 lat (usec) : 100=0.91%, 250=0.52%, 750=0.26%, 1000=7.29% lat (msec) : 2=4.82%, 4=18.10%, 10=31.38%, 20=33.20%, 50=3.39% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=18.42%, ctx=2792, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5677: Thu Jul 25 02:23:13 2024 read: IOPS=58, BW=936KiB/s (959kB/s)(5744KiB/6136msec) clat (usec): min=59, max=91986, avg=11253.18, stdev=6155.45 lat (usec): min=59, max=91987, avg=11253.81, stdev=6155.46 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 7242], 10.00th=[ 7898], 20.00th=[ 8717], | 30.00th=[ 9241], 40.00th=[ 9765], 50.00th=[10421], 60.00th=[11076], | 70.00th=[11863], 80.00th=[12911], 90.00th=[14353], 95.00th=[17171], | 99.00th=[30278], 99.50th=[45876], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 608, max= 1152, per=24.81%, avg=938.67, stdev=151.53, samples=12 iops : min= 38, max= 72, avg=58.67, stdev= 9.47, samples=12 write: IOPS=66, BW=1066KiB/s (1092kB/s)(6544KiB/6136msec) clat (usec): min=642, max=35277, avg=5055.20, stdev=5239.23 lat (usec): min=642, max=35278, avg=5056.21, stdev=5239.24 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 791], 10.00th=[ 832], 20.00th=[ 1254], | 30.00th=[ 2147], 40.00th=[ 2573], 50.00th=[ 3032], 60.00th=[ 3818], | 70.00th=[ 5211], 80.00th=[ 7832], 90.00th=[12125], 95.00th=[15139], | 99.00th=[23987], 99.50th=[30016], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 704, max= 1600, per=26.67%, avg=1061.33, stdev=264.88, samples=12 iops : min= 44, max= 100, avg=66.33, stdev=16.55, samples=12 lat (usec) : 100=0.91%, 250=0.26%, 500=0.13%, 750=0.39%, 1000=8.07% lat (msec) : 2=5.60%, 4=18.36%, 10=31.38%, 20=32.03%, 50=2.73% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=20.67%, ctx=2786, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5678: Thu Jul 25 02:23:13 2024 read: IOPS=62, BW=995KiB/s (1019kB/s)(6256KiB/6286msec) clat (usec): min=60, max=92951, avg=11006.68, stdev=5396.61 lat (usec): min=61, max=92952, avg=11007.28, stdev=5396.64 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 7308], 10.00th=[ 7832], 20.00th=[ 8586], | 30.00th=[ 9241], 40.00th=[ 9896], 50.00th=[10290], 60.00th=[11076], | 70.00th=[11731], 80.00th=[12518], 90.00th=[14091], 95.00th=[16450], | 99.00th=[23725], 99.50th=[36963], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 608, max= 1184, per=26.08%, avg=986.67, stdev=190.95, samples=12 iops : min= 38, max= 74, avg=61.67, stdev=11.93, samples=12 write: IOPS=59, BW=960KiB/s (983kB/s)(6032KiB/6286msec) clat (usec): min=600, max=51179, avg=5214.08, stdev=6206.42 lat (usec): min=601, max=51180, avg=5215.92, stdev=6206.63 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 791], 10.00th=[ 832], 20.00th=[ 1139], | 30.00th=[ 2212], 40.00th=[ 2573], 50.00th=[ 3064], 60.00th=[ 4047], | 70.00th=[ 5342], 80.00th=[ 7242], 90.00th=[12256], 95.00th=[15926], | 99.00th=[39060], 99.50th=[41157], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 576, max= 1312, per=23.25%, avg=925.33, stdev=247.29, samples=12 iops : min= 36, max= 82, avg=57.83, stdev=15.46, samples=12 lat (usec) : 100=0.65%, 250=0.13%, 750=0.39%, 1000=8.07% lat (msec) : 2=4.95%, 4=15.62%, 10=35.29%, 20=32.81%, 50=1.82% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=18.79%, ctx=2875, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5679: Thu Jul 25 02:23:13 2024 read: IOPS=61, BW=984KiB/s (1008kB/s)(6032KiB/6128msec) clat (usec): min=50, max=33112, avg=10797.39, stdev=3995.70 lat (usec): min=50, max=33113, avg=10799.03, stdev=3997.20 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 6915], 10.00th=[ 7767], 20.00th=[ 8586], | 30.00th=[ 9241], 40.00th=[ 9765], 50.00th=[10159], 60.00th=[10814], | 70.00th=[11600], 80.00th=[12387], 90.00th=[14222], 95.00th=[16909], | 99.00th=[27395], 99.50th=[32113], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 702, max= 1277, per=25.93%, avg=980.92, stdev=179.99, samples=12 iops : min= 43, max= 79, avg=61.17, stdev=11.26, samples=12 write: IOPS=63, BW=1021KiB/s (1045kB/s)(6256KiB/6128msec) clat (usec): min=700, max=49835, avg=5162.79, stdev=6006.41 lat (usec): min=701, max=49836, avg=5163.88, stdev=6006.38 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 799], 10.00th=[ 865], 20.00th=[ 1532], | 30.00th=[ 2180], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3818], | 70.00th=[ 5014], 80.00th=[ 7242], 90.00th=[12649], 95.00th=[16909], | 99.00th=[31065], 99.50th=[43779], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 798, max= 1149, per=25.58%, avg=1018.25, stdev=120.41, samples=12 iops : min= 49, max= 71, avg=63.50, stdev= 7.60, samples=12 lat (usec) : 100=0.91%, 250=0.39%, 750=0.65%, 1000=6.90% lat (msec) : 2=4.56%, 4=18.75%, 10=34.11%, 20=30.86%, 50=2.86% cpu : usr=0.10%, sys=19.26%, ctx=2842, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3783KiB/s (3874kB/s), 935KiB/s-995KiB/s (958kB/s-1019kB/s), io=23.4MiB (24.5MB), run=6128-6331msec WRITE: bw=3980KiB/s (4076kB/s), 960KiB/s-1066KiB/s (983kB/s-1092kB/s), io=24.6MiB (25.8MB), run=6128-6331msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5681: Thu Jul 25 02:23:13 2024 read: IOPS=58, BW=937KiB/s (959kB/s)(5920KiB/6319msec) clat (usec): min=1715, max=161237, avg=6728.88, stdev=9563.96 lat (usec): min=1716, max=161238, avg=6729.48, stdev=9564.01 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2540], | 30.00th=[ 2769], 40.00th=[ 3261], 50.00th=[ 3949], 60.00th=[ 5800], | 70.00th=[ 7898], 80.00th=[ 9896], 90.00th=[ 12387], 95.00th=[ 16450], | 99.00th=[ 29230], 99.50th=[ 32637], 99.90th=[160433], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 638, max= 1056, per=24.82%, avg=935.67, stdev=120.29, samples=12 iops : min= 39, max= 66, avg=58.33, stdev= 7.73, samples=12 write: IOPS=62, BW=1008KiB/s (1032kB/s)(6368KiB/6319msec) clat (usec): min=2495, max=49768, avg=9582.27, stdev=5847.71 lat (usec): min=2497, max=49770, avg=9583.48, stdev=5847.77 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3392], 10.00th=[ 3818], 20.00th=[ 4621], | 30.00th=[ 5866], 40.00th=[ 6849], 50.00th=[ 8291], 60.00th=[ 9634], | 70.00th=[11469], 80.00th=[13566], 90.00th=[16909], 95.00th=[19792], | 99.00th=[28967], 99.50th=[39060], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 702, max= 1344, per=25.80%, avg=1023.67, stdev=220.52, samples=12 iops : min= 43, max= 84, avg=63.83, stdev=14.01, samples=12 lat (msec) : 2=0.78%, 4=30.47%, 10=39.97%, 20=24.87%, 50=3.78% lat (msec) : 250=0.13% cpu : usr=0.08%, sys=9.07%, ctx=1220, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5682: Thu Jul 25 02:23:13 2024 read: IOPS=58, BW=939KiB/s (962kB/s)(5744KiB/6117msec) clat (usec): min=1815, max=105597, avg=6683.83, stdev=7785.04 lat (usec): min=1816, max=105598, avg=6684.43, stdev=7785.08 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2573], | 30.00th=[ 2802], 40.00th=[ 3228], 50.00th=[ 4047], 60.00th=[ 5604], | 70.00th=[ 7832], 80.00th=[ 9896], 90.00th=[ 12649], 95.00th=[ 16188], | 99.00th=[ 30278], 99.50th=[ 48497], 99.90th=[105382], 99.95th=[105382], | 99.99th=[105382] bw ( KiB/s): min= 448, max= 1152, per=24.90%, avg=938.67, stdev=188.74, samples=12 iops : min= 28, max= 72, avg=58.67, stdev=11.80, samples=12 write: IOPS=66, BW=1070KiB/s (1095kB/s)(6544KiB/6117msec) clat (usec): min=2913, max=49378, avg=8934.34, stdev=6074.79 lat (usec): min=2914, max=49380, avg=8935.57, stdev=6075.07 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3458], 10.00th=[ 3752], 20.00th=[ 4293], | 30.00th=[ 5342], 40.00th=[ 6128], 50.00th=[ 7373], 60.00th=[ 8356], | 70.00th=[ 9896], 80.00th=[12518], 90.00th=[16188], 95.00th=[18744], | 99.00th=[36963], 99.50th=[40633], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 736, max= 1568, per=26.89%, avg=1066.67, stdev=265.40, samples=12 iops : min= 46, max= 98, avg=66.67, stdev=16.59, samples=12 lat (msec) : 2=1.04%, 4=29.82%, 10=44.92%, 20=20.57%, 50=3.52% lat (msec) : 250=0.13% cpu : usr=0.16%, sys=9.22%, ctx=1158, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5683: Thu Jul 25 02:23:13 2024 read: IOPS=61, BW=985KiB/s (1009kB/s)(6256KiB/6352msec) clat (usec): min=1854, max=49397, avg=6507.12, stdev=5722.01 lat (usec): min=1855, max=49397, avg=6507.76, stdev=5722.05 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2507], | 30.00th=[ 2835], 40.00th=[ 3425], 50.00th=[ 4047], 60.00th=[ 5997], | 70.00th=[ 7898], 80.00th=[ 9896], 90.00th=[12911], 95.00th=[15270], | 99.00th=[28443], 99.50th=[43779], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 640, max= 1184, per=26.24%, avg=989.17, stdev=195.25, samples=12 iops : min= 40, max= 74, avg=61.75, stdev=12.27, samples=12 write: IOPS=59, BW=950KiB/s (972kB/s)(6032KiB/6352msec) clat (msec): min=2, max=106, avg= 9.93, stdev= 8.59 lat (msec): min=2, max=106, avg= 9.93, stdev= 8.59 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 17], 95.00th=[ 22], | 99.00th=[ 48], 99.50th=[ 77], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 576, max= 1248, per=23.45%, avg=930.42, stdev=238.37, samples=12 iops : min= 36, max= 78, avg=58.08, stdev=14.87, samples=12 lat (msec) : 2=0.78%, 4=29.04%, 10=43.88%, 20=21.87%, 50=4.04% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.08%, sys=9.23%, ctx=1254, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5684: Thu Jul 25 02:23:13 2024 read: IOPS=61, BW=987KiB/s (1010kB/s)(6032KiB/6114msec) clat (usec): min=1784, max=48309, avg=6113.53, stdev=5154.84 lat (usec): min=1785, max=48309, avg=6114.12, stdev=5154.84 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 3130], 50.00th=[ 3818], 60.00th=[ 5407], | 70.00th=[ 7504], 80.00th=[ 9765], 90.00th=[12125], 95.00th=[14877], | 99.00th=[25297], 99.50th=[29492], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 736, max= 1280, per=26.10%, avg=984.00, stdev=189.62, samples=12 iops : min= 46, max= 80, avg=61.50, stdev=11.85, samples=12 write: IOPS=63, BW=1023KiB/s (1048kB/s)(6256KiB/6114msec) clat (usec): min=2587, max=91125, avg=9541.86, stdev=7332.79 lat (usec): min=2588, max=91127, avg=9543.05, stdev=7332.81 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3228], 10.00th=[ 3687], 20.00th=[ 4424], | 30.00th=[ 5342], 40.00th=[ 6849], 50.00th=[ 7963], 60.00th=[ 9241], | 70.00th=[10683], 80.00th=[13173], 90.00th=[16450], 95.00th=[20317], | 99.00th=[38011], 99.50th=[54264], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 800, max= 1248, per=25.95%, avg=1029.33, stdev=132.88, samples=12 iops : min= 50, max= 78, avg=64.33, stdev= 8.30, samples=12 lat (msec) : 2=1.43%, 4=31.12%, 10=40.10%, 20=23.70%, 50=3.39% lat (msec) : 100=0.26% cpu : usr=0.20%, sys=9.19%, ctx=1200, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3771KiB/s (3861kB/s), 937KiB/s-987KiB/s (959kB/s-1010kB/s), io=23.4MiB (24.5MB), run=6114-6352msec WRITE: bw=3967KiB/s (4062kB/s), 950KiB/s-1070KiB/s (972kB/s-1095kB/s), io=24.6MiB (25.8MB), run=6114-6352msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5694: Thu Jul 25 02:23:17 2024 read: IOPS=1, BW=2004KiB/s (2053kB/s)(7168KiB/3576msec) clat (usec): min=376, max=897652, avg=337184.74, stdev=347383.21 lat (usec): min=377, max=897653, avg=337185.70, stdev=347382.99 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 52167], | 30.00th=[ 92799], 40.00th=[ 92799], 50.00th=[254804], 60.00th=[341836], | 70.00th=[341836], 80.00th=[725615], 90.00th=[901776], 95.00th=[901776], | 99.00th=[901776], 99.50th=[901776], 99.90th=[901776], 99.95th=[901776], | 99.99th=[901776] bw ( KiB/s): min= 1695, max= 6144, per=73.16%, avg=3978.33, stdev=2226.83, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 write: IOPS=1, BW=1432KiB/s (1466kB/s)(5120KiB/3576msec) clat (msec): min=29, max=790, avg=225.51, stdev=324.14 lat (msec): min=29, max=790, avg=225.56, stdev=324.13 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 1695, max= 4096, per=32.47%, avg=2471.75, stdev=1095.55, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=16.67%, 100=25.00%, 250=8.33%, 500=16.67%, 750=8.33% lat (msec) : 1000=16.67% cpu : usr=0.03%, sys=24.20%, ctx=815, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5695: Thu Jul 25 02:23:17 2024 read: IOPS=0, BW=544KiB/s (557kB/s)(2048KiB/3766msec) clat (msec): min=79, max=716, avg=398.21, stdev=450.64 lat (msec): min=79, max=716, avg=398.21, stdev=450.64 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 80], 60.00th=[ 718], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 2043, max= 2048, per=37.61%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2719KiB/s (2784kB/s)(10.0MiB/3766msec) clat (msec): min=18, max=1039, avg=295.76, stdev=368.94 lat (msec): min=18, max=1039, avg=295.81, stdev=368.93 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 43], 40.00th=[ 65], 50.00th=[ 67], 60.00th=[ 161], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 852], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 1673, max= 6144, per=52.54%, avg=4000.00, stdev=1828.34, samples=4 iops : min= 1, max= 6, avg= 3.50, stdev= 2.08, samples=4 lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 250=8.33%, 500=16.67% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=25.71%, ctx=139, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5696: Thu Jul 25 02:23:17 2024 read: IOPS=1, BW=1690KiB/s (1730kB/s)(6144KiB/3636msec) clat (usec): min=384, max=795310, avg=178529.19, stdev=314444.81 lat (usec): min=385, max=795310, avg=178530.04, stdev=314444.69 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 383], 10.00th=[ 383], 20.00th=[ 437], | 30.00th=[ 437], 40.00th=[ 3982], 50.00th=[ 3982], 60.00th=[ 45351], | 70.00th=[225444], 80.00th=[225444], 90.00th=[792724], 95.00th=[792724], | 99.00th=[792724], 99.50th=[792724], 99.90th=[792724], 99.95th=[792724], | 99.99th=[792724] bw ( KiB/s): min= 4096, max= 5060, per=84.19%, avg=4578.00, stdev=681.65, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1690KiB/s (1730kB/s)(6144KiB/3636msec) clat (msec): min=17, max=2133, avg=417.35, stdev=841.99 lat (msec): min=17, max=2133, avg=417.38, stdev=842.00 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 86], 50.00th=[ 86], 60.00th=[ 114], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 2140], 95.00th=[ 2140], | 99.00th=[ 2140], 99.50th=[ 2140], 99.90th=[ 2140], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 3373, max= 4096, per=50.64%, avg=3855.00, stdev=417.42, samples=3 iops : min= 3, max= 4, avg= 3.67, stdev= 0.58, samples=3 lat (usec) : 500=16.67% lat (msec) : 4=8.33%, 20=8.33%, 50=16.67%, 100=8.33%, 250=25.00% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=11.53%, ctx=373, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5697: Thu Jul 25 02:23:17 2024 read: IOPS=1, BW=1957KiB/s (2004kB/s)(5120KiB/2616msec) clat (usec): min=370, max=286341, avg=112192.93, stdev=107434.10 lat (usec): min=370, max=286342, avg=112193.74, stdev=107433.94 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], 20.00th=[ 371], | 30.00th=[ 57934], 40.00th=[ 57934], 50.00th=[ 94897], 60.00th=[ 94897], | 70.00th=[121111], 80.00th=[121111], 90.00th=[287310], 95.00th=[287310], | 99.00th=[287310], 99.50th=[287310], 99.90th=[287310], 99.95th=[287310], | 99.99th=[287310] bw ( KiB/s): min= 2048, max= 6144, per=75.32%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2740KiB/s (2806kB/s)(7168KiB/2616msec) clat (msec): min=18, max=1025, avg=286.74, stdev=413.19 lat (msec): min=18, max=1025, avg=286.79, stdev=413.18 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 77], 60.00th=[ 108], | 70.00th=[ 108], 80.00th=[ 726], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 2048, max= 6144, per=47.08%, avg=3584.00, stdev=1960.81, samples=4 iops : min= 2, max= 6, avg= 3.50, stdev= 1.91, samples=4 lat (usec) : 500=8.33% lat (msec) : 20=16.67%, 50=8.33%, 100=25.00%, 250=16.67%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=18.70%, ctx=262, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5438KiB/s (5569kB/s), 544KiB/s-2004KiB/s (557kB/s-2053kB/s), io=20.0MiB (20.0MB), run=2616-3766msec WRITE: bw=7613KiB/s (7796kB/s), 1432KiB/s-2740KiB/s (1466kB/s-2806kB/s), io=28.0MiB (29.4MB), run=2616-3766msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5699: Thu Jul 25 02:23:18 2024 read: IOPS=1, BW=2009KiB/s (2057kB/s)(7168KiB/3568msec) clat (msec): min=15, max=1034, avg=414.70, stdev=383.11 lat (msec): min=15, max=1034, avg=414.70, stdev=383.11 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 226], 40.00th=[ 226], 50.00th=[ 284], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 701], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2048, max= 4096, per=56.90%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 write: IOPS=1, BW=1435KiB/s (1469kB/s)(5120KiB/3568msec) clat (msec): min=35, max=238, avg=108.48, stdev=91.62 lat (msec): min=35, max=238, avg=108.53, stdev=91.61 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 50], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 2043, max= 6144, per=45.17%, avg=3411.67, stdev=2366.27, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (msec) : 20=16.67%, 50=25.00%, 250=25.00%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=3.67%, ctx=66, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5700: Thu Jul 25 02:23:18 2024 read: IOPS=0, BW=540KiB/s (552kB/s)(2048KiB/3796msec) clat (msec): min=76, max=1151, avg=614.09, stdev=760.22 lat (msec): min=76, max=1151, avg=614.09, stdev=760.22 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 1150], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2048, max= 2048, per=37.96%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2698KiB/s (2762kB/s)(10.0MiB/3796msec) clat (msec): min=21, max=1040, avg=244.22, stdev=371.63 lat (msec): min=21, max=1040, avg=244.28, stdev=371.62 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 27], 40.00th=[ 50], 50.00th=[ 74], 60.00th=[ 106], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 835], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 4096, max= 6144, per=72.25%, avg=5457.00, stdev=1178.68, samples=3 iops : min= 4, max= 6, avg= 5.00, stdev= 1.00, samples=3 lat (msec) : 50=33.33%, 100=16.67%, 250=25.00%, 1000=8.33% cpu : usr=0.00%, sys=4.69%, ctx=114, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5701: Thu Jul 25 02:23:18 2024 read: IOPS=1, BW=1623KiB/s (1662kB/s)(6144KiB/3785msec) clat (usec): min=14127, max=71907, avg=31623.83, stdev=22169.97 lat (usec): min=14128, max=71908, avg=31624.66, stdev=22170.21 clat percentiles (usec): | 1.00th=[14091], 5.00th=[14091], 10.00th=[14091], 20.00th=[14615], | 30.00th=[14615], 40.00th=[17433], 50.00th=[17433], 60.00th=[34866], | 70.00th=[36963], 80.00th=[36963], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1623KiB/s (1662kB/s)(6144KiB/3785msec) clat (msec): min=37, max=2381, avg=577.21, stdev=933.68 lat (msec): min=37, max=2381, avg=577.27, stdev=933.66 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 114], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 4096, max= 4096, per=54.23%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 20=25.00%, 50=33.33%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=3.04%, ctx=136, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5702: Thu Jul 25 02:23:18 2024 read: IOPS=1, BW=1363KiB/s (1396kB/s)(5120KiB/3756msec) clat (msec): min=14, max=664, avg=163.46, stdev=281.62 lat (msec): min=14, max=664, avg=163.46, stdev=281.62 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 31], 60.00th=[ 31], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 6144, max= 6144, per=100.00%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1908KiB/s (1954kB/s)(7168KiB/3756msec) clat (msec): min=18, max=2449, avg=418.44, stdev=897.86 lat (msec): min=18, max=2449, avg=418.48, stdev=897.85 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 25], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 71], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 207], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 2043, max= 6144, per=54.20%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 20=25.00%, 50=25.00%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=3.81%, ctx=119, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5395KiB/s (5525kB/s), 540KiB/s-2009KiB/s (552kB/s-2057kB/s), io=20.0MiB (20.0MB), run=3568-3796msec WRITE: bw=7553KiB/s (7734kB/s), 1435KiB/s-2698KiB/s (1469kB/s-2762kB/s), io=28.0MiB (29.4MB), run=3568-3796msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5715: Thu Jul 25 02:23:21 2024 read: IOPS=1, BW=6298KiB/s (6449kB/s)(12.0MiB/1951msec) clat (usec): min=1531, max=1791.6k, avg=627928.55, stdev=1008783.38 lat (usec): min=1533, max=1791.6k, avg=627929.82, stdev=1008782.94 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1532], 10.00th=[ 1532], | 20.00th=[ 1532], 30.00th=[ 1532], 40.00th=[ 90702], | 50.00th=[ 90702], 60.00th=[ 90702], 70.00th=[1786774], | 80.00th=[1786774], 90.00th=[1786774], 95.00th=[1786774], | 99.00th=[1786774], 99.50th=[1786774], 99.90th=[1786774], | 99.95th=[1786774], 99.99th=[1786774] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 2=33.33%, 100=33.33% cpu : usr=0.00%, sys=47.18%, ctx=464, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5716: Thu Jul 25 02:23:21 2024 read: IOPS=0, BW=1487KiB/s (1522kB/s)(4096KiB/2755msec) clat (nsec): min=91948k, max=91948k, avg=91947729.00, stdev= 0.00 lat (nsec): min=91949k, max=91949k, avg=91949100.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[91751], 5.00th=[91751], 10.00th=[91751], 20.00th=[91751], | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751], | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2974KiB/s (3045kB/s)(8192KiB/2755msec) clat (msec): min=1239, max=1353, avg=1296.64, stdev=80.39 lat (msec): min=1240, max=1353, avg=1296.87, stdev=80.37 clat percentiles (msec): | 1.00th=[ 1234], 5.00th=[ 1234], 10.00th=[ 1234], 20.00th=[ 1234], | 30.00th=[ 1234], 40.00th=[ 1234], 50.00th=[ 1234], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8192, max= 8192, per=68.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.00%, sys=9.44%, ctx=75, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5717: Thu Jul 25 02:23:21 2024 write: IOPS=1, BW=4766KiB/s (4881kB/s)(12.0MiB/2578msec) clat (msec): min=182, max=2003, avg=852.27, stdev=1001.80 lat (msec): min=182, max=2004, avg=852.50, stdev=1001.85 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 2005], 80.00th=[ 2005], 90.00th=[ 2005], 95.00th=[ 2005], | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 8192, max= 8192, per=68.88%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=14.01%, ctx=290, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5718: Thu Jul 25 02:23:21 2024 write: IOPS=1, BW=4485KiB/s (4592kB/s)(12.0MiB/2740msec) clat (msec): min=95, max=1347, avg=893.03, stdev=692.70 lat (msec): min=95, max=1347, avg=893.21, stdev=692.78 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 1234], 50.00th=[ 1234], 60.00th=[ 1234], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8192, max= 8192, per=68.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.00%, sys=12.56%, ctx=27, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5947KiB/s (6090kB/s), 1487KiB/s-6298KiB/s (1522kB/s-6449kB/s), io=16.0MiB (16.8MB), run=1951-2755msec WRITE: bw=11.6MiB/s (12.2MB/s), 2974KiB/s-4766KiB/s (3045kB/s-4881kB/s), io=32.0MiB (33.6MB), run=2578-2755msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5711: Thu Jul 25 02:23:22 2024 read: IOPS=13, BW=52.2MiB/s (54.7MB/s)(12.0MiB/230msec) clat (msec): min=41, max=126, avg=74.63, stdev=45.78 lat (msec): min=41, max=126, avg=74.63, stdev=45.78 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 56], | 70.00th=[ 127], 80.00th=[ 127], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=25.76%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5712: Thu Jul 25 02:23:22 2024 read: IOPS=0, BW=1598KiB/s (1636kB/s)(4096KiB/2563msec) clat (nsec): min=71198k, max=71198k, avg=71197625.00, stdev= 0.00 lat (nsec): min=71200k, max=71200k, avg=71199663.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[70779], 5.00th=[70779], 10.00th=[70779], 20.00th=[70779], | 30.00th=[70779], 40.00th=[70779], 50.00th=[70779], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3196KiB/s (3273kB/s)(8192KiB/2563msec) clat (msec): min=1133, max=1347, avg=1240.31, stdev=151.41 lat (msec): min=1133, max=1347, avg=1240.63, stdev=151.65 clat percentiles (msec): | 1.00th=[ 1133], 5.00th=[ 1133], 10.00th=[ 1133], 20.00th=[ 1133], | 30.00th=[ 1133], 40.00th=[ 1133], 50.00th=[ 1133], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8175, max= 8175, per=75.69%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.04%, sys=6.79%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5713: Thu Jul 25 02:23:22 2024 write: IOPS=0, BW=4050KiB/s (4147kB/s)(12.0MiB/3034msec) clat (msec): min=90, max=2188, avg=1001.84, stdev=1075.92 lat (msec): min=90, max=2189, avg=1002.17, stdev=1075.96 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8192, max= 8192, per=75.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.03%, sys=6.82%, ctx=143, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5714: Thu Jul 25 02:23:22 2024 write: IOPS=1, BW=5555KiB/s (5688kB/s)(12.0MiB/2212msec) clat (msec): min=53, max=2070, avg=731.03, stdev=1160.23 lat (msec): min=53, max=2070, avg=731.27, stdev=1160.21 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 2072], 80.00th=[ 2072], 90.00th=[ 2072], 95.00th=[ 2072], | 99.00th=[ 2072], 99.50th=[ 2072], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67% cpu : usr=0.05%, sys=3.93%, ctx=53, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6393KiB/s (6546kB/s), 1598KiB/s-52.2MiB/s (1636kB/s-54.7MB/s), io=16.0MiB (16.8MB), run=230-2563msec WRITE: bw=10.5MiB/s (11.1MB/s), 3196KiB/s-5555KiB/s (3273kB/s-5688kB/s), io=32.0MiB (33.6MB), run=2212-3034msec