== sanity test 398b: DIO and buffer IO race ============== 04:23:05 (1724314985) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.36932 s, 36.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4718: Thu Aug 22 04:23:22 2024 read: IOPS=104, BW=416KiB/s (426kB/s)(5992KiB/14393msec) clat (usec): min=57, max=184926, avg=5282.07, stdev=5510.67 lat (usec): min=57, max=184928, avg=5283.50, stdev=5511.01 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 112], 10.00th=[ 2474], 20.00th=[ 2999], | 30.00th=[ 3458], 40.00th=[ 3982], 50.00th=[ 4686], 60.00th=[ 5473], | 70.00th=[ 6325], 80.00th=[ 7111], 90.00th=[ 8586], 95.00th=[ 10290], | 99.00th=[ 14484], 99.50th=[ 17695], 99.90th=[ 28181], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 232, max= 544, per=24.90%, avg=409.32, stdev=62.17, samples=28 iops : min= 58, max= 136, avg=102.25, stdev=15.53, samples=28 write: IOPS=109, BW=437KiB/s (448kB/s)(6296KiB/14393msec) clat (usec): min=433, max=20200, avg=4082.22, stdev=3590.43 lat (usec): min=434, max=20201, avg=4083.05, stdev=3590.44 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 586], 10.00th=[ 668], 20.00th=[ 840], | 30.00th=[ 1057], 40.00th=[ 2474], 50.00th=[ 3392], 60.00th=[ 4113], | 70.00th=[ 5014], 80.00th=[ 6718], 90.00th=[ 9110], 95.00th=[11469], | 99.00th=[15533], 99.50th=[17695], 99.90th=[19792], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 264, max= 616, per=24.88%, avg=428.18, stdev=68.20, samples=28 iops : min= 66, max= 154, avg=106.96, stdev=17.07, samples=28 lat (usec) : 100=2.02%, 250=0.91%, 500=0.39%, 750=7.06%, 1000=7.06% lat (msec) : 2=4.95%, 4=27.44%, 10=43.46%, 20=6.51%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.65%, ctx=6016, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4719: Thu Aug 22 04:23:22 2024 read: IOPS=102, BW=411KiB/s (421kB/s)(5940KiB/14458msec) clat (usec): min=50, max=172730, avg=5378.23, stdev=5290.62 lat (usec): min=50, max=172732, avg=5380.04, stdev=5291.10 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 126], 10.00th=[ 2507], 20.00th=[ 3064], | 30.00th=[ 3523], 40.00th=[ 4015], 50.00th=[ 4621], 60.00th=[ 5473], | 70.00th=[ 6325], 80.00th=[ 7439], 90.00th=[ 8848], 95.00th=[ 10552], | 99.00th=[ 15664], 99.50th=[ 18220], 99.90th=[ 22938], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 160, max= 504, per=24.78%, avg=407.32, stdev=67.10, samples=28 iops : min= 40, max= 126, avg=101.75, stdev=16.80, samples=28 write: IOPS=109, BW=439KiB/s (450kB/s)(6348KiB/14458msec) clat (usec): min=409, max=34573, avg=4032.74, stdev=3592.72 lat (usec): min=410, max=34574, avg=4035.66, stdev=3594.16 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 570], 10.00th=[ 635], 20.00th=[ 824], | 30.00th=[ 1004], 40.00th=[ 2442], 50.00th=[ 3294], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 9110], 95.00th=[10945], | 99.00th=[15139], 99.50th=[17171], 99.90th=[22152], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 224, max= 566, per=24.93%, avg=429.00, stdev=71.37, samples=28 iops : min= 56, max= 141, avg=107.18, stdev=17.80, samples=28 lat (usec) : 100=1.76%, 250=0.72%, 500=0.36%, 750=7.75%, 1000=7.52% lat (msec) : 2=4.26%, 4=27.47%, 10=43.29%, 20=6.64%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.29%, sys=23.27%, ctx=6071, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4720: Thu Aug 22 04:23:22 2024 read: IOPS=103, BW=413KiB/s (423kB/s)(6028KiB/14604msec) clat (usec): min=56, max=22620, avg=5263.33, stdev=2996.94 lat (usec): min=57, max=22620, avg=5264.28, stdev=2997.07 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 126], 10.00th=[ 2507], 20.00th=[ 3032], | 30.00th=[ 3556], 40.00th=[ 4080], 50.00th=[ 4752], 60.00th=[ 5538], | 70.00th=[ 6325], 80.00th=[ 7373], 90.00th=[ 8848], 95.00th=[10290], | 99.00th=[16188], 99.50th=[18482], 99.90th=[20579], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 240, max= 560, per=24.94%, avg=410.07, stdev=67.65, samples=29 iops : min= 60, max= 140, avg=102.45, stdev=16.87, samples=29 write: IOPS=107, BW=429KiB/s (439kB/s)(6260KiB/14604msec) clat (usec): min=406, max=193403, avg=4227.15, stdev=5996.43 lat (usec): min=407, max=193405, avg=4227.99, stdev=5996.46 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 578], 10.00th=[ 660], 20.00th=[ 832], | 30.00th=[ 1074], 40.00th=[ 2442], 50.00th=[ 3294], 60.00th=[ 4113], | 70.00th=[ 5211], 80.00th=[ 6915], 90.00th=[ 9372], 95.00th=[ 11338], | 99.00th=[ 14877], 99.50th=[ 17171], 99.90th=[ 25822], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 312, max= 568, per=24.64%, avg=424.14, stdev=70.55, samples=29 iops : min= 78, max= 142, avg=105.97, stdev=17.67, samples=29 lat (usec) : 100=1.89%, 250=0.78%, 500=0.20%, 750=7.62%, 1000=6.97% lat (msec) : 2=4.69%, 4=26.53%, 10=44.21%, 20=6.90%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.37%, ctx=6155, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4721: Thu Aug 22 04:23:22 2024 read: IOPS=104, BW=417KiB/s (428kB/s)(6052KiB/14496msec) clat (usec): min=49, max=26393, avg=5297.10, stdev=3072.38 lat (usec): min=49, max=26394, avg=5297.82, stdev=3072.38 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 190], 10.00th=[ 2507], 20.00th=[ 2966], | 30.00th=[ 3490], 40.00th=[ 4080], 50.00th=[ 4817], 60.00th=[ 5538], | 70.00th=[ 6456], 80.00th=[ 7308], 90.00th=[ 8848], 95.00th=[10421], | 99.00th=[16450], 99.50th=[19792], 99.90th=[24511], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 152, max= 560, per=24.82%, avg=408.04, stdev=75.63, samples=28 iops : min= 38, max= 140, avg=101.82, stdev=18.85, samples=28 write: IOPS=107, BW=430KiB/s (441kB/s)(6236KiB/14496msec) clat (usec): min=403, max=175071, avg=4110.72, stdev=5738.91 lat (usec): min=404, max=175073, avg=4111.61, stdev=5738.93 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 611], 20.00th=[ 775], | 30.00th=[ 947], 40.00th=[ 1778], 50.00th=[ 3163], 60.00th=[ 3982], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 9241], 95.00th=[ 11731], | 99.00th=[ 15664], 99.50th=[ 18744], 99.90th=[ 33162], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 176, max= 600, per=24.44%, avg=420.61, stdev=85.50, samples=28 iops : min= 44, max= 150, avg=104.96, stdev=21.45, samples=28 lat (usec) : 50=0.03%, 100=1.40%, 250=1.11%, 500=0.42%, 750=9.08% lat (usec) : 1000=6.67% lat (msec) : 2=4.75%, 4=26.30%, 10=42.94%, 20=7.00%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.18%, ctx=6120, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1644KiB/s (1684kB/s), 411KiB/s-417KiB/s (421kB/s-428kB/s), io=23.4MiB (24.6MB), run=14393-14604msec WRITE: bw=1721KiB/s (1763kB/s), 429KiB/s-439KiB/s (439kB/s-450kB/s), io=24.6MiB (25.7MB), run=14393-14604msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4713: Thu Aug 22 04:23:27 2024 read: IOPS=79, BW=317KiB/s (324kB/s)(5992KiB/18912msec) clat (usec): min=1534, max=212744, avg=5235.97, stdev=6638.90 lat (usec): min=1534, max=212745, avg=5236.77, stdev=6638.89 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2343], | 30.00th=[ 2671], 40.00th=[ 3163], 50.00th=[ 3687], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 7046], 90.00th=[ 10683], 95.00th=[ 13435], | 99.00th=[ 19530], 99.50th=[ 21627], 99.90th=[ 30802], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 152, max= 752, per=24.58%, avg=311.46, stdev=144.95, samples=37 iops : min= 38, max= 188, avg=77.78, stdev=36.26, samples=37 write: IOPS=83, BW=333KiB/s (341kB/s)(6296KiB/18912msec) clat (usec): min=2269, max=30367, avg=6993.10, stdev=4165.65 lat (usec): min=2269, max=30368, avg=6994.20, stdev=4165.53 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 3032], 10.00th=[ 3294], 20.00th=[ 3654], | 30.00th=[ 4146], 40.00th=[ 4883], 50.00th=[ 5604], 60.00th=[ 6587], | 70.00th=[ 8094], 80.00th=[ 9896], 90.00th=[12518], 95.00th=[15664], | 99.00th=[21627], 99.50th=[25035], 99.90th=[29754], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 160, max= 680, per=24.73%, avg=328.11, stdev=140.86, samples=37 iops : min= 40, max= 170, avg=81.95, stdev=35.25, samples=37 lat (msec) : 2=3.39%, 4=37.27%, 10=44.04%, 20=14.19%, 50=1.07% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=10.83%, ctx=3677, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4714: Thu Aug 22 04:23:27 2024 read: IOPS=78, BW=314KiB/s (321kB/s)(5940KiB/18941msec) clat (usec): min=1421, max=185626, avg=5318.67, stdev=6205.41 lat (usec): min=1421, max=185627, avg=5319.43, stdev=6205.44 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1942], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2671], 40.00th=[ 3130], 50.00th=[ 3720], 60.00th=[ 4490], | 70.00th=[ 5276], 80.00th=[ 7177], 90.00th=[ 11207], 95.00th=[ 14222], | 99.00th=[ 20579], 99.50th=[ 21890], 99.90th=[ 27132], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 104, max= 712, per=24.45%, avg=309.76, stdev=151.30, samples=37 iops : min= 26, max= 178, avg=77.38, stdev=37.85, samples=37 write: IOPS=83, BW=335KiB/s (343kB/s)(6348KiB/18941msec) clat (usec): min=2249, max=35696, avg=6919.69, stdev=4390.57 lat (usec): min=2249, max=35697, avg=6920.69, stdev=4390.57 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2933], 10.00th=[ 3261], 20.00th=[ 3556], | 30.00th=[ 3982], 40.00th=[ 4686], 50.00th=[ 5276], 60.00th=[ 6259], | 70.00th=[ 8029], 80.00th=[10028], 90.00th=[12911], 95.00th=[14615], | 99.00th=[23987], 99.50th=[27395], 99.90th=[34866], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 160, max= 704, per=24.60%, avg=326.41, stdev=139.63, samples=37 iops : min= 40, max= 176, avg=81.54, stdev=34.94, samples=37 lat (msec) : 2=3.35%, 4=38.61%, 10=41.96%, 20=14.62%, 50=1.43% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=10.88%, ctx=3670, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4715: Thu Aug 22 04:23:27 2024 read: IOPS=80, BW=322KiB/s (330kB/s)(6028KiB/18732msec) clat (usec): min=1550, max=24618, avg=5074.13, stdev=3684.76 lat (usec): min=1551, max=24618, avg=5074.88, stdev=3684.75 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1958], 10.00th=[ 2147], 20.00th=[ 2376], | 30.00th=[ 2769], 40.00th=[ 3228], 50.00th=[ 3752], 60.00th=[ 4424], | 70.00th=[ 5276], 80.00th=[ 6915], 90.00th=[10552], 95.00th=[13435], | 99.00th=[18482], 99.50th=[19792], 99.90th=[22152], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 136, max= 792, per=25.20%, avg=319.24, stdev=133.49, samples=37 iops : min= 34, max= 198, avg=79.73, stdev=33.41, samples=37 write: IOPS=83, BW=334KiB/s (342kB/s)(6260KiB/18732msec) clat (msec): min=2, max=191, avg= 7.04, stdev= 6.18 lat (msec): min=2, max=191, avg= 7.05, stdev= 6.18 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 36], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 168, max= 752, per=24.90%, avg=330.49, stdev=137.42, samples=37 iops : min= 42, max= 188, avg=82.54, stdev=34.40, samples=37 lat (msec) : 2=2.93%, 4=37.99%, 10=43.85%, 20=14.42%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=10.97%, ctx=3620, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4716: Thu Aug 22 04:23:27 2024 read: IOPS=80, BW=321KiB/s (329kB/s)(6052KiB/18829msec) clat (usec): min=1535, max=26380, avg=4993.07, stdev=3677.72 lat (usec): min=1536, max=26381, avg=4993.79, stdev=3677.74 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1958], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2671], 40.00th=[ 3130], 50.00th=[ 3687], 60.00th=[ 4359], | 70.00th=[ 5145], 80.00th=[ 6980], 90.00th=[10290], 95.00th=[13304], | 99.00th=[17433], 99.50th=[20317], 99.90th=[23987], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 144, max= 718, per=25.00%, avg=316.78, stdev=142.83, samples=37 iops : min= 36, max= 179, avg=79.08, stdev=35.70, samples=37 write: IOPS=82, BW=331KiB/s (339kB/s)(6236KiB/18829msec) clat (msec): min=2, max=190, avg= 7.19, stdev= 6.33 lat (msec): min=2, max=190, avg= 7.19, stdev= 6.33 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 16], | 99.00th=[ 23], 99.50th=[ 28], 99.90th=[ 32], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 168, max= 704, per=24.64%, avg=326.95, stdev=140.54, samples=37 iops : min= 42, max= 176, avg=81.62, stdev=35.15, samples=37 lat (msec) : 2=3.09%, 4=38.48%, 10=42.38%, 20=15.01%, 50=1.01% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=10.96%, ctx=3648, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1268KiB/s (1298kB/s), 314KiB/s-322KiB/s (321kB/s-330kB/s), io=23.4MiB (24.6MB), run=18732-18941msec WRITE: bw=1327KiB/s (1359kB/s), 331KiB/s-335KiB/s (339kB/s-343kB/s), io=24.6MiB (25.7MB), run=18732-18941msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4756: Thu Aug 22 04:23:34 2024 read: IOPS=57, BW=919KiB/s (941kB/s)(5920KiB/6440msec) clat (usec): min=53, max=54446, avg=12113.39, stdev=4819.13 lat (usec): min=53, max=54447, avg=12114.11, stdev=4819.15 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 102], 10.00th=[ 8029], 20.00th=[10290], | 30.00th=[11076], 40.00th=[11731], 50.00th=[12387], 60.00th=[13173], | 70.00th=[14091], 80.00th=[15008], 90.00th=[16450], 95.00th=[17695], | 99.00th=[20055], 99.50th=[22938], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 800, max= 1024, per=24.83%, avg=898.33, stdev=76.47, samples=12 iops : min= 50, max= 64, avg=56.00, stdev= 4.77, samples=12 write: IOPS=61, BW=989KiB/s (1013kB/s)(6368KiB/6440msec) clat (usec): min=764, max=22668, avg=4885.11, stdev=4261.87 lat (usec): min=765, max=22670, avg=4886.43, stdev=4261.91 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 865], 10.00th=[ 1029], 20.00th=[ 1319], | 30.00th=[ 2442], 40.00th=[ 3064], 50.00th=[ 3589], 60.00th=[ 4293], | 70.00th=[ 5276], 80.00th=[ 7046], 90.00th=[10945], 95.00th=[15664], | 99.00th=[20055], 99.50th=[22152], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 672, max= 1376, per=25.63%, avg=975.58, stdev=222.33, samples=12 iops : min= 42, max= 86, avg=60.83, stdev=13.91, samples=12 lat (usec) : 100=2.34%, 250=1.30%, 1000=4.95% lat (msec) : 2=8.20%, 4=15.89%, 10=21.87%, 20=44.40%, 50=0.91% lat (msec) : 100=0.13% cpu : usr=0.16%, sys=20.59%, ctx=2680, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4757: Thu Aug 22 04:23:34 2024 read: IOPS=55, BW=885KiB/s (907kB/s)(5744KiB/6488msec) clat (usec): min=59, max=34792, avg=12522.78, stdev=4003.55 lat (usec): min=59, max=34793, avg=12523.58, stdev=4003.59 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 7898], 10.00th=[ 9110], 20.00th=[10028], | 30.00th=[10945], 40.00th=[11731], 50.00th=[12387], 60.00th=[13173], | 70.00th=[14222], 80.00th=[15533], 90.00th=[16909], 95.00th=[18220], | 99.00th=[22152], 99.50th=[24773], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 672, max= 992, per=23.57%, avg=852.83, stdev=105.27, samples=12 iops : min= 42, max= 62, avg=53.08, stdev= 6.67, samples=12 write: IOPS=63, BW=1009KiB/s (1033kB/s)(6544KiB/6488msec) clat (usec): min=676, max=35849, avg=4735.19, stdev=4789.84 lat (usec): min=677, max=35850, avg=4736.44, stdev=4789.88 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 857], 10.00th=[ 971], 20.00th=[ 1287], | 30.00th=[ 2409], 40.00th=[ 2933], 50.00th=[ 3359], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 6521], 90.00th=[10290], 95.00th=[14353], | 99.00th=[25035], 99.50th=[28705], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 640, max= 1568, per=25.42%, avg=967.42, stdev=239.20, samples=12 iops : min= 40, max= 98, avg=60.25, stdev=15.00, samples=12 lat (usec) : 100=1.04%, 250=0.78%, 750=0.13%, 1000=5.60% lat (msec) : 2=8.72%, 4=18.88%, 10=21.74%, 20=41.41%, 50=1.69% cpu : usr=0.15%, sys=20.33%, ctx=2654, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4758: Thu Aug 22 04:23:34 2024 read: IOPS=59, BW=952KiB/s (975kB/s)(6256KiB/6568msec) clat (usec): min=59, max=29744, avg=12148.08, stdev=4471.78 lat (usec): min=60, max=29744, avg=12150.80, stdev=4477.39 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 103], 10.00th=[ 8225], 20.00th=[ 9896], | 30.00th=[10945], 40.00th=[11731], 50.00th=[12518], 60.00th=[13173], | 70.00th=[13960], 80.00th=[14877], 90.00th=[16712], 95.00th=[18482], | 99.00th=[22414], 99.50th=[24773], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 704, max= 1344, per=26.05%, avg=942.38, stdev=161.43, samples=13 iops : min= 44, max= 84, avg=58.77, stdev=10.14, samples=13 write: IOPS=57, BW=918KiB/s (940kB/s)(6032KiB/6568msec) clat (usec): min=716, max=89515, avg=4748.32, stdev=6035.60 lat (usec): min=717, max=89516, avg=4749.69, stdev=6035.69 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 832], 10.00th=[ 979], 20.00th=[ 1352], | 30.00th=[ 2376], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 4015], | 70.00th=[ 5080], 80.00th=[ 6456], 90.00th=[ 9110], 95.00th=[14353], | 99.00th=[21365], 99.50th=[26608], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 608, max= 1152, per=23.60%, avg=898.15, stdev=153.21, samples=13 iops : min= 38, max= 72, avg=56.00, stdev= 9.60, samples=13 lat (usec) : 100=2.34%, 250=0.91%, 500=0.26%, 750=0.65%, 1000=4.56% lat (msec) : 2=7.81%, 4=16.28%, 10=22.14%, 20=42.84%, 50=2.08% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=19.98%, ctx=2809, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4759: Thu Aug 22 04:23:34 2024 read: IOPS=56, BW=911KiB/s (933kB/s)(6032KiB/6620msec) clat (usec): min=58, max=28350, avg=12390.31, stdev=3636.12 lat (usec): min=59, max=28351, avg=12391.06, stdev=3636.20 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 7898], 10.00th=[ 9110], 20.00th=[10159], | 30.00th=[11076], 40.00th=[11731], 50.00th=[12387], 60.00th=[13304], | 70.00th=[13829], 80.00th=[15008], 90.00th=[16319], 95.00th=[17695], | 99.00th=[20579], 99.50th=[25297], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 704, max= 1152, per=24.54%, avg=888.00, stdev=111.89, samples=13 iops : min= 44, max= 72, avg=55.23, stdev= 7.07, samples=13 write: IOPS=59, BW=945KiB/s (968kB/s)(6256KiB/6620msec) clat (usec): min=768, max=37194, avg=4866.88, stdev=5048.65 lat (usec): min=769, max=37196, avg=4868.24, stdev=5048.58 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 840], 10.00th=[ 955], 20.00th=[ 1287], | 30.00th=[ 2212], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3851], | 70.00th=[ 5014], 80.00th=[ 6783], 90.00th=[10945], 95.00th=[16188], | 99.00th=[22676], 99.50th=[34866], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 640, max= 1344, per=24.62%, avg=937.08, stdev=205.71, samples=13 iops : min= 40, max= 84, avg=58.31, stdev=12.85, samples=13 lat (usec) : 100=1.43%, 250=0.13%, 1000=6.38% lat (msec) : 2=8.46%, 4=16.67%, 10=20.83%, 20=44.66%, 50=1.43% cpu : usr=0.17%, sys=19.96%, ctx=2668, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3618KiB/s (3705kB/s), 885KiB/s-952KiB/s (907kB/s-975kB/s), io=23.4MiB (24.5MB), run=6440-6620msec WRITE: bw=3807KiB/s (3898kB/s), 918KiB/s-1009KiB/s (940kB/s-1033kB/s), io=24.6MiB (25.8MB), run=6440-6620msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4752: Thu Aug 22 04:23:34 2024 read: IOPS=57, BW=912KiB/s (934kB/s)(5920KiB/6488msec) clat (usec): min=2004, max=48755, avg=6703.93, stdev=5151.30 lat (usec): min=2005, max=48757, avg=6704.73, stdev=5151.34 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2376], 10.00th=[ 2638], 20.00th=[ 2868], | 30.00th=[ 3359], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5932], | 70.00th=[ 8291], 80.00th=[10552], 90.00th=[13304], 95.00th=[15401], | 99.00th=[23987], 99.50th=[30278], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 704, max= 1024, per=24.78%, avg=892.42, stdev=104.01, samples=12 iops : min= 44, max= 64, avg=55.75, stdev= 6.47, samples=12 write: IOPS=61, BW=982KiB/s (1005kB/s)(6368KiB/6488msec) clat (usec): min=2728, max=38479, avg=9987.09, stdev=5489.27 lat (usec): min=2729, max=38480, avg=9988.67, stdev=5489.08 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3916], 10.00th=[ 4359], 20.00th=[ 5407], | 30.00th=[ 6652], 40.00th=[ 7701], 50.00th=[ 8848], 60.00th=[10159], | 70.00th=[11731], 80.00th=[13042], 90.00th=[16319], 95.00th=[22152], | 99.00th=[30016], 99.50th=[35914], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 608, max= 1280, per=25.66%, avg=972.42, stdev=214.87, samples=12 iops : min= 38, max= 80, avg=60.75, stdev=13.42, samples=12 lat (msec) : 4=24.09%, 10=43.75%, 20=28.26%, 50=3.91% cpu : usr=0.06%, sys=10.68%, ctx=1150, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4753: Thu Aug 22 04:23:34 2024 read: IOPS=61, BW=986KiB/s (1010kB/s)(5744KiB/5823msec) clat (usec): min=2069, max=25688, avg=5937.74, stdev=4007.35 lat (usec): min=2070, max=25689, avg=5938.50, stdev=4007.43 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2442], 10.00th=[ 2638], 20.00th=[ 2835], | 30.00th=[ 3228], 40.00th=[ 3589], 50.00th=[ 4146], 60.00th=[ 5276], | 70.00th=[ 6915], 80.00th=[ 8979], 90.00th=[11863], 95.00th=[13698], | 99.00th=[20317], 99.50th=[22938], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 672, max= 1280, per=27.77%, avg=1000.27, stdev=195.33, samples=11 iops : min= 42, max= 80, avg=62.36, stdev=12.28, samples=11 write: IOPS=70, BW=1124KiB/s (1151kB/s)(6544KiB/5823msec) clat (usec): min=2936, max=95672, avg=8996.34, stdev=6651.55 lat (usec): min=2937, max=95674, avg=8997.78, stdev=6651.88 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 3851], 10.00th=[ 4113], 20.00th=[ 4621], | 30.00th=[ 5407], 40.00th=[ 6390], 50.00th=[ 7504], 60.00th=[ 8356], | 70.00th=[ 9896], 80.00th=[11731], 90.00th=[16319], 95.00th=[21103], | 99.00th=[24773], 99.50th=[27395], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 926, max= 1312, per=29.46%, avg=1116.64, stdev=133.27, samples=11 iops : min= 57, max= 82, avg=69.64, stdev= 8.44, samples=11 lat (msec) : 4=26.82%, 10=50.13%, 20=19.27%, 50=3.65%, 100=0.13% cpu : usr=0.14%, sys=10.01%, ctx=925, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4754: Thu Aug 22 04:23:34 2024 read: IOPS=58, BW=941KiB/s (964kB/s)(6256KiB/6648msec) clat (usec): min=1972, max=22407, avg=6864.77, stdev=4523.25 lat (usec): min=1973, max=22407, avg=6865.74, stdev=4523.09 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2409], 10.00th=[ 2638], 20.00th=[ 2933], | 30.00th=[ 3326], 40.00th=[ 4113], 50.00th=[ 4948], 60.00th=[ 7046], | 70.00th=[ 8717], 80.00th=[10683], 90.00th=[13960], 95.00th=[16188], | 99.00th=[21103], 99.50th=[22152], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 640, max= 1120, per=25.60%, avg=922.08, stdev=143.09, samples=13 iops : min= 40, max= 70, avg=57.54, stdev= 8.84, samples=13 write: IOPS=56, BW=907KiB/s (929kB/s)(6032KiB/6648msec) clat (usec): min=2771, max=43048, avg=10393.85, stdev=6031.39 lat (usec): min=2772, max=43051, avg=10395.31, stdev=6031.33 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3589], 10.00th=[ 4015], 20.00th=[ 5211], | 30.00th=[ 6521], 40.00th=[ 7832], 50.00th=[ 9503], 60.00th=[10683], | 70.00th=[11994], 80.00th=[14353], 90.00th=[18482], 95.00th=[21627], | 99.00th=[30540], 99.50th=[40633], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 608, max= 1248, per=23.18%, avg=878.46, stdev=168.91, samples=13 iops : min= 38, max= 78, avg=54.85, stdev=10.58, samples=13 lat (msec) : 2=0.13%, 4=24.09%, 10=42.06%, 20=29.56%, 50=4.17% cpu : usr=0.17%, sys=10.35%, ctx=1215, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4755: Thu Aug 22 04:23:34 2024 read: IOPS=65, BW=1050KiB/s (1075kB/s)(6032KiB/5744msec) clat (usec): min=1933, max=23323, avg=6129.55, stdev=4122.90 lat (usec): min=1933, max=23324, avg=6130.31, stdev=4122.90 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2442], 10.00th=[ 2606], 20.00th=[ 3032], | 30.00th=[ 3392], 40.00th=[ 3621], 50.00th=[ 4178], 60.00th=[ 5342], | 70.00th=[ 6980], 80.00th=[ 9896], 90.00th=[12387], 95.00th=[14353], | 99.00th=[20317], 99.50th=[21365], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 768, max= 1312, per=28.90%, avg=1040.91, stdev=173.35, samples=11 iops : min= 48, max= 82, avg=64.91, stdev=10.74, samples=11 write: IOPS=68, BW=1089KiB/s (1115kB/s)(6256KiB/5744msec) clat (usec): min=3014, max=36220, avg=8647.97, stdev=5447.98 lat (usec): min=3015, max=36223, avg=8649.50, stdev=5447.88 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3752], 10.00th=[ 3982], 20.00th=[ 4490], | 30.00th=[ 5014], 40.00th=[ 5473], 50.00th=[ 6652], 60.00th=[ 7767], | 70.00th=[ 9765], 80.00th=[12256], 90.00th=[17171], 95.00th=[19268], | 99.00th=[26608], 99.50th=[34866], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 864, max= 1277, per=28.77%, avg=1090.36, stdev=134.66, samples=11 iops : min= 54, max= 79, avg=68.00, stdev= 8.23, samples=11 lat (msec) : 2=0.26%, 4=28.39%, 10=47.01%, 20=22.01%, 50=2.34% cpu : usr=0.16%, sys=10.26%, ctx=932, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3603KiB/s (3689kB/s), 912KiB/s-1050KiB/s (934kB/s-1075kB/s), io=23.4MiB (24.5MB), run=5744-6648msec WRITE: bw=3791KiB/s (3882kB/s), 907KiB/s-1124KiB/s (929kB/s-1151kB/s), io=24.6MiB (25.8MB), run=5744-6648msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4770: Thu Aug 22 04:23:38 2024 read: IOPS=2, BW=3046KiB/s (3119kB/s)(7168KiB/2353msec) clat (usec): min=426, max=523949, avg=290584.20, stdev=252065.97 lat (usec): min=428, max=523949, avg=290585.32, stdev=252065.68 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 4293], | 30.00th=[ 65799], 40.00th=[ 65799], 50.00th=[438305], 60.00th=[497026], | 70.00th=[497026], 80.00th=[505414], 90.00th=[522191], 95.00th=[522191], | 99.00th=[522191], 99.50th=[522191], 99.90th=[522191], 99.95th=[522191], | 99.99th=[522191] bw ( KiB/s): min= 1741, max= 4096, per=31.80%, avg=2767.25, stdev=1209.03, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 write: IOPS=2, BW=2176KiB/s (2228kB/s)(5120KiB/2353msec) clat (usec): min=28486, max=61339, avg=43770.79, stdev=14593.12 lat (usec): min=28519, max=61400, avg=43821.08, stdev=14593.30 clat percentiles (usec): | 1.00th=[28443], 5.00th=[28443], 10.00th=[28443], 20.00th=[28443], | 30.00th=[33817], 40.00th=[33817], 50.00th=[38011], 60.00th=[38011], | 70.00th=[57410], 80.00th=[57410], 90.00th=[61080], 95.00th=[61080], | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 1741, max= 5233, per=23.85%, avg=2906.00, stdev=2015.24, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 50=25.00%, 100=25.00%, 500=16.67%, 750=16.67% cpu : usr=0.04%, sys=25.68%, ctx=1069, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4771: Thu Aug 22 04:23:38 2024 read: IOPS=2, BW=2312KiB/s (2367kB/s)(2048KiB/886msec) clat (usec): min=42235, max=82821, avg=62528.36, stdev=28698.47 lat (usec): min=42236, max=82822, avg=62529.71, stdev=28698.45 clat percentiles (usec): | 1.00th=[42206], 5.00th=[42206], 10.00th=[42206], 20.00th=[42206], | 30.00th=[42206], 40.00th=[42206], 50.00th=[42206], 60.00th=[82314], | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 2048, max= 2048, per=23.53%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.3MiB/s (11.8MB/s)(10.0MiB/886msec) clat (msec): min=18, max=314, avg=71.66, stdev=87.15 lat (msec): min=18, max=314, avg=71.73, stdev=87.14 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 29], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 46], 60.00th=[ 55], | 70.00th=[ 65], 80.00th=[ 65], 90.00th=[ 66], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 6144, max= 6144, per=50.42%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=41.67%, 500=8.33% cpu : usr=0.00%, sys=34.01%, ctx=111, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4772: Thu Aug 22 04:23:38 2024 read: IOPS=9, BW=9225KiB/s (9447kB/s)(6144KiB/666msec) clat (usec): min=321, max=239979, avg=68509.47, stdev=93754.96 lat (usec): min=322, max=239981, avg=68510.51, stdev=93755.11 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 322], 10.00th=[ 322], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 79168], | 70.00th=[ 90702], 80.00th=[ 90702], 90.00th=[240124], 95.00th=[240124], | 99.00th=[240124], 99.50th=[240124], 99.90th=[240124], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 4096, max= 4096, per=47.06%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9225KiB/s (9447kB/s)(6144KiB/666msec) clat (usec): min=21360, max=72467, avg=38280.01, stdev=18051.76 lat (usec): min=21377, max=72543, avg=38316.34, stdev=18075.26 clat percentiles (usec): | 1.00th=[21365], 5.00th=[21365], 10.00th=[21365], 20.00th=[27395], | 30.00th=[27395], 40.00th=[32637], 50.00th=[32637], 60.00th=[34341], | 70.00th=[41681], 80.00th=[41681], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 8192, max= 8192, per=67.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 50=41.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=40.15%, ctx=176, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4773: Thu Aug 22 04:23:38 2024 read: IOPS=7, BW=7665KiB/s (7849kB/s)(5120KiB/668msec) clat (msec): min=9, max=140, avg=72.21, stdev=49.03 lat (msec): min=9, max=140, avg=72.22, stdev=49.03 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 10], 10.00th=[ 10], 20.00th=[ 10], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 80], 60.00th=[ 80], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 8192, max= 8192, per=94.13%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.5MiB/s (10.0MB/s)(7168KiB/668msec) clat (usec): min=23792, max=63087, avg=38281.64, stdev=14896.76 lat (usec): min=23849, max=63149, avg=38331.74, stdev=14900.02 clat percentiles (usec): | 1.00th=[23725], 5.00th=[23725], 10.00th=[23725], 20.00th=[25035], | 30.00th=[25560], 40.00th=[25560], 50.00th=[36963], 60.00th=[43779], | 70.00th=[43779], 80.00th=[50070], 90.00th=[63177], 95.00th=[63177], | 99.00th=[63177], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min=10240, max=10240, per=84.04%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 10=8.33%, 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=45.43%, ctx=202, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8704KiB/s (8913kB/s), 2312KiB/s-9225KiB/s (2367kB/s-9447kB/s), io=20.0MiB (20.0MB), run=666-2353msec WRITE: bw=11.9MiB/s (12.5MB/s), 2176KiB/s-11.3MiB/s (2228kB/s-11.8MB/s), io=28.0MiB (29.4MB), run=666-2353msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4774: Thu Aug 22 04:23:38 2024 read: IOPS=2, BW=3023KiB/s (3096kB/s)(7168KiB/2371msec) clat (msec): min=16, max=468, avg=210.05, stdev=210.13 lat (msec): min=16, max=468, avg=210.06, stdev=210.13 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 32], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 86], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 418], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 1741, max= 4096, per=33.76%, avg=2916.25, stdev=1357.07, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 write: IOPS=2, BW=2159KiB/s (2211kB/s)(5120KiB/2371msec) clat (msec): min=38, max=504, avg=163.52, stdev=194.02 lat (msec): min=38, max=504, avg=163.58, stdev=194.01 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 89], 60.00th=[ 89], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 2043, max= 5224, per=30.05%, avg=3633.50, stdev=2249.31, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 500=25.00% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=5.86%, ctx=76, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4775: Thu Aug 22 04:23:38 2024 read: IOPS=2, BW=2158KiB/s (2210kB/s)(2048KiB/949msec) clat (usec): min=23176, max=91143, avg=57159.89, stdev=48059.48 lat (usec): min=23177, max=91144, avg=57161.15, stdev=48059.63 clat percentiles (usec): | 1.00th=[23200], 5.00th=[23200], 10.00th=[23200], 20.00th=[23200], | 30.00th=[23200], 40.00th=[23200], 50.00th=[23200], 60.00th=[90702], | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 2048, max= 2048, per=23.71%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.5MiB/s (11.0MB/s)(10.0MiB/949msec) clat (msec): min=17, max=258, avg=62.57, stdev=72.64 lat (msec): min=17, max=258, avg=62.63, stdev=72.65 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 25], 40.00th=[ 27], 50.00th=[ 33], 60.00th=[ 39], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 94], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 6144, max= 6144, per=50.81%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 500=8.33% cpu : usr=0.11%, sys=19.41%, ctx=116, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4776: Thu Aug 22 04:23:38 2024 read: IOPS=8, BW=8853KiB/s (9065kB/s)(6144KiB/694msec) clat (msec): min=36, max=264, avg=89.87, stdev=88.12 lat (msec): min=36, max=264, avg=89.87, stdev=88.12 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 68], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8853KiB/s (9065kB/s)(6144KiB/694msec) clat (usec): min=16279, max=47612, avg=24243.66, stdev=11712.23 lat (usec): min=16353, max=47802, avg=24331.29, stdev=11760.22 clat percentiles (usec): | 1.00th=[16319], 5.00th=[16319], 10.00th=[16319], 20.00th=[18220], | 30.00th=[18220], 40.00th=[19792], 50.00th=[19792], 60.00th=[19792], | 70.00th=[23725], 80.00th=[23725], 90.00th=[47449], 95.00th=[47449], | 99.00th=[47449], 99.50th=[47449], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min=12288, max=12288, per=100.00%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=41.67%, 100=16.67%, 500=8.33% cpu : usr=0.14%, sys=13.11%, ctx=50, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4777: Thu Aug 22 04:23:38 2024 read: IOPS=5, BW=5505KiB/s (5638kB/s)(5120KiB/930msec) clat (usec): min=16729, max=51043, avg=32510.21, stdev=14718.34 lat (usec): min=16730, max=51045, avg=32511.52, stdev=14718.56 clat percentiles (usec): | 1.00th=[16712], 5.00th=[16712], 10.00th=[16712], 20.00th=[16712], | 30.00th=[19268], 40.00th=[19268], 50.00th=[33424], 60.00th=[33424], | 70.00th=[42206], 80.00th=[42206], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] write: IOPS=7, BW=7708KiB/s (7893kB/s)(7168KiB/930msec) clat (msec): min=16, max=383, avg=97.67, stdev=128.37 lat (msec): min=16, max=383, avg=97.74, stdev=128.37 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 67], 60.00th=[ 69], | 70.00th=[ 69], 80.00th=[ 77], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 4096, max= 4096, per=33.87%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=16.67%, 100=41.67%, 500=8.33% cpu : usr=0.22%, sys=11.73%, ctx=73, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8638KiB/s (8845kB/s), 2158KiB/s-8853KiB/s (2210kB/s-9065kB/s), io=20.0MiB (20.0MB), run=694-2371msec WRITE: bw=11.8MiB/s (12.4MB/s), 2159KiB/s-10.5MiB/s (2211kB/s-11.0MB/s), io=28.0MiB (29.4MB), run=694-2371msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4786: Thu Aug 22 04:23:41 2024 read: IOPS=1, BW=5385KiB/s (5514kB/s)(12.0MiB/2282msec) clat (msec): min=158, max=1262, avg=743.97, stdev=554.70 lat (msec): min=158, max=1262, avg=743.98, stdev=554.70 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=49.45%, ctx=1260, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4788: Thu Aug 22 04:23:41 2024 read: IOPS=0, BW=2453KiB/s (2512kB/s)(4096KiB/1670msec) clat (nsec): min=159721k, max=159721k, avg=159720845.00, stdev= 0.00 lat (nsec): min=159723k, max=159723k, avg=159723464.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4905KiB/s (5023kB/s)(8192KiB/1670msec) clat (msec): min=179, max=1294, avg=736.77, stdev=788.72 lat (msec): min=179, max=1294, avg=737.03, stdev=788.70 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 1301], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8192, max= 8192, per=58.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=14.38%, ctx=91, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4790: Thu Aug 22 04:23:41 2024 write: IOPS=1, BW=5297KiB/s (5424kB/s)(12.0MiB/2320msec) clat (msec): min=267, max=1325, avg=767.49, stdev=531.60 lat (msec): min=267, max=1325, avg=767.78, stdev=531.55 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 709], 50.00th=[ 709], 60.00th=[ 709], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 5512, max= 8192, per=48.51%, avg=6852.00, stdev=1895.05, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=11.90%, ctx=39, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4792: Thu Aug 22 04:23:41 2024 write: IOPS=1, BW=7548KiB/s (7729kB/s)(12.0MiB/1628msec) clat (msec): min=123, max=1302, avg=535.94, stdev=664.34 lat (msec): min=124, max=1302, avg=536.22, stdev=664.34 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8192, max= 8192, per=58.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.12%, sys=19.48%, ctx=76, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7180KiB/s (7352kB/s), 2453KiB/s-5385KiB/s (2512kB/s-5514kB/s), io=16.0MiB (16.8MB), run=1670-2282msec WRITE: bw=13.8MiB/s (14.5MB/s), 4905KiB/s-7548KiB/s (5023kB/s-7729kB/s), io=32.0MiB (33.6MB), run=1628-2320msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4785: Thu Aug 22 04:23:41 2024 read: IOPS=1, BW=5055KiB/s (5176kB/s)(12.0MiB/2431msec) clat (msec): min=400, max=1546, avg=803.10, stdev=644.68 lat (msec): min=400, max=1546, avg=803.10, stdev=644.68 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 1552], 80.00th=[ 1552], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 5512, max= 5512, per=81.79%, avg=5512.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=5.60%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4787: Thu Aug 22 04:23:41 2024 read: IOPS=4, BW=16.5MiB/s (17.3MB/s)(4096KiB/243msec) clat (nsec): min=64088k, max=64088k, avg=64087873.00, stdev= 0.00 lat (nsec): min=64091k, max=64091k, avg=64090582.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[64226], 5.00th=[64226], 10.00th=[64226], 20.00th=[64226], | 30.00th=[64226], 40.00th=[64226], 50.00th=[64226], 60.00th=[64226], | 70.00th=[64226], 80.00th=[64226], 90.00th=[64226], 95.00th=[64226], | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] write: IOPS=8, BW=32.9MiB/s (34.5MB/s)(8192KiB/243msec) clat (msec): min=49, max=122, avg=86.03, stdev=52.25 lat (msec): min=49, max=123, avg=86.32, stdev=52.23 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 124], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.83%, sys=35.12%, ctx=35, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4789: Thu Aug 22 04:23:41 2024 write: IOPS=5, BW=23.6MiB/s (24.8MB/s)(12.0MiB/508msec) clat (msec): min=82, max=236, avg=162.70, stdev=77.52 lat (msec): min=82, max=237, avg=162.93, stdev=77.48 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.39%, sys=20.32%, ctx=72, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4791: Thu Aug 22 04:23:41 2024 write: IOPS=6, BW=24.3MiB/s (25.5MB/s)(12.0MiB/494msec) clat (msec): min=47, max=387, avg=161.41, stdev=195.91 lat (msec): min=48, max=387, avg=161.69, stdev=195.79 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 49], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] lat (msec) : 50=66.67%, 500=33.33% cpu : usr=0.00%, sys=13.79%, ctx=63, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6740KiB/s (6901kB/s), 5055KiB/s-16.5MiB/s (5176kB/s-17.3MB/s), io=16.0MiB (16.8MB), run=243-2431msec WRITE: bw=62.0MiB/s (66.1MB/s), 23.6MiB/s-32.9MiB/s (24.8MB/s-34.5MB/s), io=32.0MiB (33.6MB), run=243-508msec