== sanity test 398b: DIO and buffer IO race ============== 07:38:55 (1711107535) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.57079 s, 32.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8038: Fri Mar 22 07:39:29 2024 read: IOPS=49, BW=196KiB/s (201kB/s)(5992KiB/30535msec) clat (usec): min=57, max=201233, avg=11280.35, stdev=9043.43 lat (usec): min=58, max=201235, avg=11283.58, stdev=9046.70 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 108], 10.00th=[ 3949], 20.00th=[ 5669], | 30.00th=[ 6849], 40.00th=[ 8029], 50.00th=[ 9634], 60.00th=[ 11338], | 70.00th=[ 13435], 80.00th=[ 16319], 90.00th=[ 21103], 95.00th=[ 25035], | 99.00th=[ 36439], 99.50th=[ 44303], 99.90th=[ 74974], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 103, max= 334, per=25.46%, avg=195.51, stdev=50.44, samples=61 iops : min= 25, max= 83, avg=48.54, stdev=12.64, samples=61 write: IOPS=51, BW=206KiB/s (211kB/s)(6296KiB/30535msec) clat (usec): min=490, max=83669, avg=8586.27, stdev=9601.18 lat (usec): min=491, max=83671, avg=8587.77, stdev=9601.21 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 693], 10.00th=[ 750], 20.00th=[ 873], | 30.00th=[ 1270], 40.00th=[ 2409], 50.00th=[ 5604], 60.00th=[ 8094], | 70.00th=[10814], 80.00th=[15401], 90.00th=[21627], 95.00th=[27657], | 99.00th=[40109], 99.50th=[43779], 99.90th=[70779], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 56, max= 400, per=25.52%, avg=205.21, stdev=61.65, samples=61 iops : min= 14, max= 100, avg=50.95, stdev=15.49, samples=61 lat (usec) : 100=1.99%, 250=1.33%, 500=0.16%, 750=5.27%, 1000=7.26% lat (msec) : 2=6.71%, 4=5.44%, 10=31.64%, 20=28.45%, 50=11.43% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.18%, sys=14.19%, ctx=5249, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8039: Fri Mar 22 07:39:29 2024 read: IOPS=47, BW=192KiB/s (196kB/s)(5940KiB/30971msec) clat (usec): min=58, max=57339, avg=11384.81, stdev=7462.60 lat (usec): min=58, max=57339, avg=11387.69, stdev=7462.39 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 111], 10.00th=[ 4293], 20.00th=[ 5735], | 30.00th=[ 7111], 40.00th=[ 8356], 50.00th=[ 9896], 60.00th=[11600], | 70.00th=[13698], 80.00th=[16581], 90.00th=[20841], 95.00th=[25560], | 99.00th=[35390], 99.50th=[41681], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 103, max= 332, per=24.76%, avg=190.13, stdev=48.88, samples=61 iops : min= 25, max= 83, avg=47.26, stdev=12.25, samples=61 write: IOPS=51, BW=205KiB/s (210kB/s)(6348KiB/30971msec) clat (usec): min=433, max=194837, avg=8788.91, stdev=10353.40 lat (usec): min=434, max=194838, avg=8791.31, stdev=10356.65 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 685], 10.00th=[ 734], 20.00th=[ 881], | 30.00th=[ 1450], 40.00th=[ 2999], 50.00th=[ 6259], 60.00th=[ 8455], | 70.00th=[ 11207], 80.00th=[ 15008], 90.00th=[ 21365], 95.00th=[ 27132], | 99.00th=[ 39584], 99.50th=[ 45351], 99.90th=[ 82314], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 95, max= 344, per=24.99%, avg=200.89, stdev=55.64, samples=61 iops : min= 23, max= 86, avg=49.95, stdev=13.88, samples=61 lat (usec) : 100=1.89%, 250=1.01%, 500=0.20%, 750=6.12%, 1000=5.83% lat (msec) : 2=6.38%, 4=5.21%, 10=32.00%, 20=29.72%, 50=11.36% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.15%, sys=14.25%, ctx=5223, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8041: Fri Mar 22 07:39:29 2024 read: IOPS=49, BW=197KiB/s (201kB/s)(6028KiB/30636msec) clat (usec): min=64, max=116909, avg=11635.04, stdev=8456.71 lat (usec): min=64, max=116911, avg=11636.65, stdev=8456.76 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 111], 10.00th=[ 4178], 20.00th=[ 5735], | 30.00th=[ 6980], 40.00th=[ 8225], 50.00th=[ 9896], 60.00th=[ 11731], | 70.00th=[ 13829], 80.00th=[ 16909], 90.00th=[ 21103], 95.00th=[ 25035], | 99.00th=[ 38011], 99.50th=[ 49021], 99.90th=[ 73925], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 88, max= 320, per=25.46%, avg=195.56, stdev=44.22, samples=61 iops : min= 22, max= 80, avg=48.64, stdev=11.00, samples=61 write: IOPS=51, BW=204KiB/s (209kB/s)(6260KiB/30636msec) clat (usec): min=481, max=230465, avg=8297.09, stdev=10457.43 lat (usec): min=483, max=230466, avg=8298.41, stdev=10457.49 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 693], 10.00th=[ 734], 20.00th=[ 848], | 30.00th=[ 1270], 40.00th=[ 2212], 50.00th=[ 5473], 60.00th=[ 8029], | 70.00th=[ 10552], 80.00th=[ 14484], 90.00th=[ 20579], 95.00th=[ 25035], | 99.00th=[ 38536], 99.50th=[ 45876], 99.90th=[ 58459], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 80, max= 336, per=25.19%, avg=202.49, stdev=56.87, samples=61 iops : min= 20, max= 84, avg=50.38, stdev=14.18, samples=61 lat (usec) : 100=1.76%, 250=1.11%, 500=0.10%, 750=5.83%, 1000=6.87% lat (msec) : 2=7.06%, 4=5.01%, 10=31.58%, 20=29.13%, 50=11.17% lat (msec) : 100=0.33%, 250=0.07% cpu : usr=0.21%, sys=14.21%, ctx=5178, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8043: Fri Mar 22 07:39:29 2024 read: IOPS=48, BW=194KiB/s (198kB/s)(6052KiB/31259msec) clat (usec): min=55, max=117478, avg=11149.01, stdev=7776.73 lat (usec): min=56, max=117479, avg=11152.19, stdev=7777.45 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 115], 10.00th=[ 4146], 20.00th=[ 5932], | 30.00th=[ 7111], 40.00th=[ 8291], 50.00th=[ 9765], 60.00th=[ 11338], | 70.00th=[ 13304], 80.00th=[ 16319], 90.00th=[ 19792], 95.00th=[ 23725], | 99.00th=[ 37487], 99.50th=[ 45876], 99.90th=[ 64226], 99.95th=[117965], | 99.99th=[117965] bw ( KiB/s): min= 56, max= 320, per=25.17%, avg=193.31, stdev=55.76, samples=62 iops : min= 14, max= 80, avg=48.08, stdev=13.92, samples=62 write: IOPS=49, BW=199KiB/s (204kB/s)(6236KiB/31259msec) clat (usec): min=499, max=229109, avg=9150.22, stdev=11165.52 lat (usec): min=500, max=229110, avg=9152.54, stdev=11165.58 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 701], 10.00th=[ 734], 20.00th=[ 848], | 30.00th=[ 1369], 40.00th=[ 2868], 50.00th=[ 6325], 60.00th=[ 8848], | 70.00th=[ 11731], 80.00th=[ 15533], 90.00th=[ 22676], 95.00th=[ 28443], | 99.00th=[ 41157], 99.50th=[ 49546], 99.90th=[ 71828], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 72, max= 384, per=24.78%, avg=199.21, stdev=62.97, samples=62 iops : min= 18, max= 96, avg=49.56, stdev=15.75, samples=62 lat (usec) : 100=1.82%, 250=1.37%, 500=0.20%, 750=5.99%, 1000=6.67% lat (msec) : 2=5.50%, 4=4.59%, 10=32.16%, 20=30.31%, 50=11.00% lat (msec) : 100=0.33%, 250=0.07% cpu : usr=0.19%, sys=14.15%, ctx=5173, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=768KiB/s (787kB/s), 192KiB/s-197KiB/s (196kB/s-201kB/s), io=23.4MiB (24.6MB), run=30535-31259msec WRITE: bw=804KiB/s (824kB/s), 199KiB/s-206KiB/s (204kB/s-211kB/s), io=24.6MiB (25.7MB), run=30535-31259msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8037: Fri Mar 22 07:39:44 2024 read: IOPS=32, BW=131KiB/s (134kB/s)(5992KiB/45802msec) clat (msec): min=2, max=200, avg=11.36, stdev=10.77 lat (msec): min=2, max=200, avg=11.37, stdev=10.77 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 24], 95.00th=[ 33], | 99.00th=[ 52], 99.50th=[ 58], 99.90th=[ 78], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 31, max= 303, per=24.78%, avg=129.87, stdev=64.17, samples=91 iops : min= 7, max= 75, avg=32.26, stdev=16.09, samples=91 write: IOPS=34, BW=137KiB/s (141kB/s)(6296KiB/45802msec) clat (msec): min=5, max=126, avg=18.21, stdev=11.48 lat (msec): min=5, max=126, avg=18.21, stdev=11.48 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 20], 80.00th=[ 26], 90.00th=[ 34], 95.00th=[ 41], | 99.00th=[ 58], 99.50th=[ 71], 99.90th=[ 106], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 40, max= 264, per=24.84%, avg=136.12, stdev=62.97, samples=91 iops : min= 10, max= 66, avg=33.82, stdev=15.77, samples=91 lat (msec) : 4=2.47%, 10=39.58%, 20=36.95%, 50=19.34%, 100=1.56% lat (msec) : 250=0.10% cpu : usr=0.07%, sys=6.40%, ctx=3372, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8040: Fri Mar 22 07:39:44 2024 read: IOPS=32, BW=130KiB/s (133kB/s)(5940KiB/45733msec) clat (msec): min=2, max=210, avg=11.32, stdev=11.10 lat (msec): min=2, max=210, avg=11.32, stdev=11.10 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 16], 90.00th=[ 24], 95.00th=[ 31], | 99.00th=[ 56], 99.50th=[ 64], 99.90th=[ 87], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 24, max= 304, per=24.61%, avg=128.98, stdev=65.52, samples=91 iops : min= 6, max= 76, avg=31.90, stdev=16.33, samples=91 write: IOPS=34, BW=139KiB/s (142kB/s)(6348KiB/45733msec) clat (msec): min=5, max=125, avg=18.15, stdev=11.37 lat (msec): min=5, max=125, avg=18.15, stdev=11.37 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 20], 80.00th=[ 25], 90.00th=[ 34], 95.00th=[ 41], | 99.00th=[ 60], 99.50th=[ 72], 99.90th=[ 87], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 63, max= 287, per=25.09%, avg=137.48, stdev=61.90, samples=91 iops : min= 15, max= 71, avg=34.03, stdev=15.47, samples=91 lat (msec) : 4=2.47%, 10=38.22%, 20=38.28%, 50=19.27%, 100=1.69% lat (msec) : 250=0.07% cpu : usr=0.10%, sys=6.39%, ctx=3375, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8042: Fri Mar 22 07:39:44 2024 read: IOPS=32, BW=132KiB/s (135kB/s)(6028KiB/45674msec) clat (usec): min=2844, max=71336, avg=11144.43, stdev=8960.25 lat (usec): min=2845, max=71338, avg=11146.50, stdev=8961.59 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4080], 10.00th=[ 4555], 20.00th=[ 5276], | 30.00th=[ 5997], 40.00th=[ 6980], 50.00th=[ 7963], 60.00th=[ 9110], | 70.00th=[11338], 80.00th=[14877], 90.00th=[23200], 95.00th=[30802], | 99.00th=[43254], 99.50th=[53216], 99.90th=[68682], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 48, max= 264, per=25.03%, avg=131.14, stdev=59.58, samples=91 iops : min= 12, max= 66, avg=32.53, stdev=14.94, samples=91 write: IOPS=34, BW=137KiB/s (140kB/s)(6260KiB/45674msec) clat (msec): min=5, max=225, avg=18.38, stdev=12.15 lat (msec): min=5, max=225, avg=18.38, stdev=12.15 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 26], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 55], 99.50th=[ 69], 99.90th=[ 106], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 40, max= 264, per=24.83%, avg=136.04, stdev=61.87, samples=91 iops : min= 10, max= 66, avg=33.76, stdev=15.52, samples=91 lat (msec) : 4=2.28%, 10=37.92%, 20=38.05%, 50=20.77%, 100=0.91% lat (msec) : 250=0.07% cpu : usr=0.08%, sys=6.48%, ctx=3358, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8044: Fri Mar 22 07:39:44 2024 read: IOPS=33, BW=133KiB/s (136kB/s)(6052KiB/45653msec) clat (usec): min=2661, max=87929, avg=11242.56, stdev=9868.52 lat (usec): min=2662, max=87930, avg=11245.27, stdev=9867.83 clat percentiles (usec): | 1.00th=[ 3359], 5.00th=[ 3916], 10.00th=[ 4424], 20.00th=[ 5276], | 30.00th=[ 5932], 40.00th=[ 6718], 50.00th=[ 7635], 60.00th=[ 8848], | 70.00th=[10683], 80.00th=[14353], 90.00th=[23987], 95.00th=[32900], | 99.00th=[50594], 99.50th=[59507], 99.90th=[72877], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 40, max= 312, per=25.20%, avg=132.03, stdev=65.36, samples=91 iops : min= 10, max= 78, avg=32.75, stdev=16.40, samples=91 write: IOPS=34, BW=137KiB/s (140kB/s)(6236KiB/45653msec) clat (msec): min=4, max=225, avg=18.30, stdev=12.20 lat (msec): min=4, max=225, avg=18.30, stdev=12.20 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 20], 80.00th=[ 26], 90.00th=[ 34], 95.00th=[ 41], | 99.00th=[ 61], 99.50th=[ 69], 99.90th=[ 92], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 56, max= 288, per=24.78%, avg=135.80, stdev=61.73, samples=91 iops : min= 14, max= 72, avg=33.69, stdev=15.49, samples=91 lat (msec) : 4=2.83%, 10=38.87%, 20=36.59%, 50=20.05%, 100=1.63% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=6.36%, ctx=3400, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=524KiB/s (537kB/s), 130KiB/s-133KiB/s (133kB/s-136kB/s), io=23.4MiB (24.6MB), run=45653-45802msec WRITE: bw=549KiB/s (562kB/s), 137KiB/s-139KiB/s (140kB/s-142kB/s), io=24.6MiB (25.7MB), run=45653-45802msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8091: Fri Mar 22 07:40:01 2024 read: IOPS=24, BW=388KiB/s (398kB/s)(5920KiB/15242msec) clat (usec): min=82, max=91002, avg=28987.60, stdev=12692.32 lat (usec): min=83, max=91004, avg=28988.85, stdev=12692.43 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[14222], 10.00th=[17433], 20.00th=[20841], | 30.00th=[23200], 40.00th=[25297], 50.00th=[27132], 60.00th=[29230], | 70.00th=[32637], 80.00th=[36439], 90.00th=[43779], 95.00th=[52691], | 99.00th=[69731], 99.50th=[88605], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 192, max= 606, per=24.96%, avg=385.63, stdev=91.76, samples=30 iops : min= 12, max= 37, avg=23.70, stdev= 5.89, samples=30 write: IOPS=26, BW=418KiB/s (428kB/s)(6368KiB/15242msec) clat (usec): min=765, max=84879, avg=11306.76, stdev=12048.39 lat (usec): min=766, max=84883, avg=11308.90, stdev=12048.54 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1074], 10.00th=[ 1156], 20.00th=[ 2040], | 30.00th=[ 4015], 40.00th=[ 5538], 50.00th=[ 7046], 60.00th=[ 9241], | 70.00th=[12649], 80.00th=[18220], 90.00th=[27657], 95.00th=[34341], | 99.00th=[57934], 99.50th=[70779], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 128, max= 640, per=25.69%, avg=417.53, stdev=137.92, samples=30 iops : min= 8, max= 40, avg=25.70, stdev= 8.60, samples=30 lat (usec) : 100=0.39%, 250=1.30%, 500=0.13%, 1000=0.39% lat (msec) : 2=9.51%, 4=5.47%, 10=17.06%, 20=16.80%, 50=45.44% lat (msec) : 100=3.52% cpu : usr=0.12%, sys=11.89%, ctx=2564, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8092: Fri Mar 22 07:40:01 2024 read: IOPS=23, BW=371KiB/s (379kB/s)(5744KiB/15502msec) clat (usec): min=98, max=76447, avg=28933.58, stdev=10599.88 lat (usec): min=100, max=76449, avg=28934.99, stdev=10599.95 clat percentiles (usec): | 1.00th=[ 116], 5.00th=[15926], 10.00th=[17957], 20.00th=[20841], | 30.00th=[23462], 40.00th=[25560], 50.00th=[27657], 60.00th=[30016], | 70.00th=[32637], 80.00th=[36963], 90.00th=[42730], 95.00th=[46400], | 99.00th=[63177], 99.50th=[69731], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 159, max= 574, per=23.96%, avg=370.26, stdev=96.33, samples=31 iops : min= 9, max= 35, avg=22.90, stdev= 6.13, samples=31 write: IOPS=26, BW=422KiB/s (432kB/s)(6544KiB/15502msec) clat (usec): min=885, max=131790, avg=12460.52, stdev=14468.35 lat (usec): min=886, max=131792, avg=12465.18, stdev=14466.79 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1057], 10.00th=[ 1156], 20.00th=[ 1663], | 30.00th=[ 4293], 40.00th=[ 5800], 50.00th=[ 7832], 60.00th=[ 10028], | 70.00th=[ 13698], 80.00th=[ 20055], 90.00th=[ 31065], 95.00th=[ 41157], | 99.00th=[ 73925], 99.50th=[ 79168], 99.90th=[131597], 99.95th=[131597], | 99.99th=[131597] bw ( KiB/s): min= 128, max= 768, per=25.90%, avg=420.81, stdev=166.09, samples=31 iops : min= 8, max= 48, avg=26.06, stdev=10.45, samples=31 lat (usec) : 100=0.13%, 250=0.65%, 1000=1.04% lat (msec) : 2=10.81%, 4=3.26%, 10=17.19%, 20=16.93%, 50=47.27% lat (msec) : 100=2.60%, 250=0.13% cpu : usr=0.17%, sys=11.34%, ctx=2486, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8093: Fri Mar 22 07:40:01 2024 read: IOPS=25, BW=409KiB/s (418kB/s)(6256KiB/15310msec) clat (usec): min=84, max=71850, avg=28527.00, stdev=10944.98 lat (usec): min=84, max=71852, avg=28534.52, stdev=10929.67 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[15139], 10.00th=[17171], 20.00th=[20841], | 30.00th=[22938], 40.00th=[25035], 50.00th=[27395], 60.00th=[29754], | 70.00th=[32900], 80.00th=[36439], 90.00th=[41681], 95.00th=[49021], | 99.00th=[60556], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 223, max= 640, per=26.27%, avg=405.83, stdev=104.20, samples=30 iops : min= 13, max= 40, avg=25.03, stdev= 6.68, samples=30 write: IOPS=24, BW=394KiB/s (403kB/s)(6032KiB/15310msec) clat (usec): min=946, max=123719, avg=10958.03, stdev=11387.78 lat (usec): min=948, max=123720, avg=10966.53, stdev=11389.55 clat percentiles (usec): | 1.00th=[ 1012], 5.00th=[ 1090], 10.00th=[ 1254], 20.00th=[ 2638], | 30.00th=[ 5014], 40.00th=[ 6194], 50.00th=[ 7963], 60.00th=[ 9896], | 70.00th=[ 12780], 80.00th=[ 16450], 90.00th=[ 23725], 95.00th=[ 30278], | 99.00th=[ 51643], 99.50th=[ 57410], 99.90th=[123208], 99.95th=[123208], | 99.99th=[123208] bw ( KiB/s): min= 158, max= 702, per=23.73%, avg=385.67, stdev=121.54, samples=30 iops : min= 9, max= 43, avg=23.77, stdev= 7.68, samples=30 lat (usec) : 100=0.78%, 250=0.52%, 500=0.13%, 1000=0.39% lat (msec) : 2=7.68%, 4=4.30%, 10=18.10%, 20=18.36%, 50=47.27% lat (msec) : 100=2.34%, 250=0.13% cpu : usr=0.12%, sys=11.96%, ctx=2607, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8094: Fri Mar 22 07:40:01 2024 read: IOPS=24, BW=389KiB/s (399kB/s)(6032KiB/15490msec) clat (usec): min=83, max=75336, avg=27819.58, stdev=10455.58 lat (usec): min=84, max=75338, avg=27820.89, stdev=10455.63 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[14615], 10.00th=[17957], 20.00th=[21103], | 30.00th=[23200], 40.00th=[24773], 50.00th=[26870], 60.00th=[29230], | 70.00th=[31589], 80.00th=[34866], 90.00th=[40109], 95.00th=[44303], | 99.00th=[58983], 99.50th=[63701], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 223, max= 544, per=24.55%, avg=379.37, stdev=91.24, samples=30 iops : min= 13, max= 34, avg=23.43, stdev= 5.78, samples=30 write: IOPS=25, BW=404KiB/s (414kB/s)(6256KiB/15490msec) clat (usec): min=713, max=124111, avg=12751.98, stdev=14368.29 lat (usec): min=716, max=124113, avg=12754.17, stdev=14368.32 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1090], 10.00th=[ 1172], 20.00th=[ 2442], | 30.00th=[ 4621], 40.00th=[ 6325], 50.00th=[ 8455], 60.00th=[ 11338], | 70.00th=[ 14091], 80.00th=[ 18744], 90.00th=[ 28181], 95.00th=[ 39060], | 99.00th=[ 79168], 99.50th=[ 88605], 99.90th=[124257], 99.95th=[124257], | 99.99th=[124257] bw ( KiB/s): min= 224, max= 640, per=24.26%, avg=394.27, stdev=118.22, samples=30 iops : min= 14, max= 40, avg=24.37, stdev= 7.37, samples=30 lat (usec) : 100=0.78%, 250=0.78%, 500=0.13%, 750=0.26%, 1000=0.13% lat (msec) : 2=8.98%, 4=4.43%, 10=14.58%, 20=19.14%, 50=47.92% lat (msec) : 100=2.73%, 250=0.13% cpu : usr=0.17%, sys=11.52%, ctx=2584, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1545KiB/s (1582kB/s), 371KiB/s-409KiB/s (379kB/s-418kB/s), io=23.4MiB (24.5MB), run=15242-15502msec WRITE: bw=1626KiB/s (1665kB/s), 394KiB/s-422KiB/s (403kB/s-432kB/s), io=24.6MiB (25.8MB), run=15242-15502msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8095: Fri Mar 22 07:40:01 2024 read: IOPS=24, BW=384KiB/s (394kB/s)(5920KiB/15397msec) clat (msec): min=2, max=108, avg=15.59, stdev=13.70 lat (msec): min=2, max=108, avg=15.59, stdev=13.70 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 13], | 70.00th=[ 19], 80.00th=[ 25], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 69], 99.50th=[ 81], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 223, max= 606, per=24.38%, avg=371.77, stdev=88.63, samples=30 iops : min= 13, max= 37, avg=22.87, stdev= 5.60, samples=30 write: IOPS=25, BW=414KiB/s (424kB/s)(6368KiB/15397msec) clat (usec): min=6736, max=75642, avg=24130.76, stdev=13573.48 lat (usec): min=6738, max=75644, avg=24133.93, stdev=13574.37 clat percentiles (usec): | 1.00th=[ 7504], 5.00th=[ 9372], 10.00th=[10290], 20.00th=[12649], | 30.00th=[15139], 40.00th=[17433], 50.00th=[20579], 60.00th=[24249], | 70.00th=[27919], 80.00th=[33817], 90.00th=[44303], 95.00th=[51643], | 99.00th=[67634], 99.50th=[73925], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 159, max= 608, per=25.22%, avg=404.80, stdev=116.45, samples=30 iops : min= 9, max= 38, avg=24.93, stdev= 7.35, samples=30 lat (msec) : 4=1.04%, 10=28.26%, 20=30.60%, 50=35.55%, 100=4.43% lat (msec) : 250=0.13% cpu : usr=0.16%, sys=5.77%, ctx=1022, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8096: Fri Mar 22 07:40:01 2024 read: IOPS=22, BW=366KiB/s (375kB/s)(5744KiB/15698msec) clat (usec): min=3307, max=83931, avg=15349.08, stdev=12611.52 lat (usec): min=3308, max=83932, avg=15350.37, stdev=12611.43 clat percentiles (usec): | 1.00th=[ 3654], 5.00th=[ 4555], 10.00th=[ 5014], 20.00th=[ 6325], | 30.00th=[ 7439], 40.00th=[ 8717], 50.00th=[ 9765], 60.00th=[13304], | 70.00th=[17695], 80.00th=[23725], 90.00th=[33162], 95.00th=[40633], | 99.00th=[63177], 99.50th=[78119], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 64, max= 637, per=23.65%, avg=360.65, stdev=112.56, samples=31 iops : min= 4, max= 39, avg=22.19, stdev= 7.02, samples=31 write: IOPS=26, BW=417KiB/s (427kB/s)(6544KiB/15698msec) clat (msec): min=5, max=148, avg=24.85, stdev=16.42 lat (msec): min=5, max=148, avg=24.86, stdev=16.42 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 25], | 70.00th=[ 30], 80.00th=[ 35], 90.00th=[ 45], 95.00th=[ 56], | 99.00th=[ 80], 99.50th=[ 85], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 128, max= 764, per=25.17%, avg=403.94, stdev=142.29, samples=31 iops : min= 8, max= 47, avg=24.90, stdev= 8.84, samples=31 lat (msec) : 4=0.78%, 10=28.26%, 20=30.99%, 50=35.29%, 100=4.56% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=5.85%, ctx=1042, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8097: Fri Mar 22 07:40:01 2024 read: IOPS=25, BW=406KiB/s (416kB/s)(6256KiB/15408msec) clat (msec): min=3, max=114, avg=14.53, stdev=12.12 lat (msec): min=3, max=114, avg=14.53, stdev=12.12 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 30], 95.00th=[ 39], | 99.00th=[ 62], 99.50th=[ 68], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 191, max= 768, per=26.53%, avg=404.57, stdev=139.11, samples=30 iops : min= 11, max= 48, avg=24.77, stdev= 8.76, samples=30 write: IOPS=24, BW=391KiB/s (401kB/s)(6032KiB/15408msec) clat (msec): min=5, max=125, avg=25.73, stdev=15.27 lat (msec): min=5, max=125, avg=25.74, stdev=15.27 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 16], 40.00th=[ 19], 50.00th=[ 23], 60.00th=[ 28], | 70.00th=[ 31], 80.00th=[ 37], 90.00th=[ 46], 95.00th=[ 56], | 99.00th=[ 75], 99.50th=[ 85], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 223, max= 574, per=23.81%, avg=382.20, stdev=90.19, samples=30 iops : min= 13, max= 35, avg=23.37, stdev= 5.67, samples=30 lat (msec) : 4=0.39%, 10=28.52%, 20=33.59%, 50=33.20%, 100=4.04% lat (msec) : 250=0.26% cpu : usr=0.10%, sys=6.03%, ctx=1051, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8098: Fri Mar 22 07:40:01 2024 read: IOPS=24, BW=388KiB/s (398kB/s)(6032KiB/15536msec) clat (msec): min=3, max=124, avg=14.57, stdev=13.08 lat (msec): min=3, max=124, avg=14.57, stdev=13.08 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 16], 80.00th=[ 23], 90.00th=[ 32], 95.00th=[ 40], | 99.00th=[ 57], 99.50th=[ 79], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 160, max= 637, per=25.23%, avg=384.81, stdev=131.46, samples=31 iops : min= 10, max= 39, avg=23.90, stdev= 8.27, samples=31 write: IOPS=25, BW=403KiB/s (412kB/s)(6256KiB/15536msec) clat (msec): min=6, max=132, avg=25.63, stdev=15.35 lat (msec): min=6, max=132, avg=25.63, stdev=15.35 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 31], 80.00th=[ 37], 90.00th=[ 45], 95.00th=[ 54], | 99.00th=[ 74], 99.50th=[ 108], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 192, max= 608, per=25.01%, avg=401.35, stdev=103.16, samples=31 iops : min= 12, max= 38, avg=24.94, stdev= 6.50, samples=31 lat (msec) : 4=0.91%, 10=29.17%, 20=30.34%, 50=35.29%, 100=3.91% lat (msec) : 250=0.39% cpu : usr=0.06%, sys=6.01%, ctx=1038, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1526KiB/s (1562kB/s), 366KiB/s-406KiB/s (375kB/s-416kB/s), io=23.4MiB (24.5MB), run=15397-15698msec WRITE: bw=1605KiB/s (1644kB/s), 391KiB/s-417KiB/s (401kB/s-427kB/s), io=24.6MiB (25.8MB), run=15397-15698msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8118: Fri Mar 22 07:40:08 2024 read: IOPS=1, BW=1802KiB/s (1845kB/s)(7168KiB/3978msec) clat (usec): min=532, max=1284.5k, avg=536203.70, stdev=639138.37 lat (usec): min=534, max=1284.5k, avg=536205.50, stdev=639138.40 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], | 20.00th=[ 594], 30.00th=[ 23462], 40.00th=[ 23462], | 50.00th=[ 74974], 60.00th=[1182794], 70.00th=[1182794], | 80.00th=[1182794], 90.00th=[1283458], 95.00th=[1283458], | 99.00th=[1283458], 99.50th=[1283458], 99.90th=[1283458], | 99.95th=[1283458], 99.99th=[1283458] bw ( KiB/s): min= 2043, max= 8175, per=100.00%, avg=4087.00, stdev=3540.31, samples=3 iops : min= 1, max= 7, avg= 3.00, stdev= 3.46, samples=3 write: IOPS=1, BW=1287KiB/s (1318kB/s)(5120KiB/3978msec) clat (usec): min=30442, max=58087, avg=43446.73, stdev=10751.64 lat (usec): min=30553, max=58180, avg=43509.94, stdev=10757.06 clat percentiles (usec): | 1.00th=[30540], 5.00th=[30540], 10.00th=[30540], 20.00th=[30540], | 30.00th=[35914], 40.00th=[35914], 50.00th=[45351], 60.00th=[45351], | 70.00th=[47449], 80.00th=[47449], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min=10219, max=10219, per=100.00%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=41.67%, 100=16.67% cpu : usr=0.10%, sys=16.52%, ctx=964, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8120: Fri Mar 22 07:40:08 2024 read: IOPS=0, BW=503KiB/s (515kB/s)(2048KiB/4072msec) clat (msec): min=67, max=785, avg=426.80, stdev=507.93 lat (msec): min=67, max=785, avg=426.80, stdev=507.93 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 785], | 70.00th=[ 785], 80.00th=[ 785], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 2043, max= 2043, per=54.42%, avg=2043.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2515KiB/s (2575kB/s)(10.0MiB/4072msec) clat (msec): min=43, max=1306, avg=320.99, stdev=516.46 lat (msec): min=43, max=1306, avg=321.05, stdev=516.46 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 50], 50.00th=[ 56], 60.00th=[ 64], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 1284], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 4096, max= 8175, per=100.00%, avg=6134.00, stdev=2039.50, samples=3 iops : min= 4, max= 7, avg= 5.33, stdev= 1.53, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=8.52%, ctx=255, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8123: Fri Mar 22 07:40:08 2024 read: IOPS=1, BW=1126KiB/s (1153kB/s)(6144KiB/5455msec) clat (usec): min=474, max=1143.9k, avg=402614.00, stdev=544868.65 lat (usec): min=476, max=1143.9k, avg=402615.60, stdev=544868.80 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], | 20.00th=[ 594], 30.00th=[ 594], 40.00th=[ 709], | 50.00th=[ 709], 60.00th=[ 221250], 70.00th=[1044382], | 80.00th=[1044382], 90.00th=[1149240], 95.00th=[1149240], | 99.00th=[1149240], 99.50th=[1149240], 99.90th=[1149240], | 99.95th=[1149240], 99.99th=[1149240] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1126KiB/s (1153kB/s)(6144KiB/5455msec) clat (msec): min=34, max=2614, avg=506.20, stdev=1034.23 lat (msec): min=34, max=2614, avg=506.27, stdev=1034.23 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 2601], 95.00th=[ 2601], | 99.00th=[ 2601], 99.50th=[ 2601], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 2048, max= 6144, per=77.93%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=25.00%, 250=25.00% cpu : usr=0.07%, sys=6.49%, ctx=362, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8124: Fri Mar 22 07:40:08 2024 read: IOPS=0, BW=941KiB/s (964kB/s)(5120KiB/5439msec) clat (usec): min=358, max=1193.3k, avg=460123.69, stdev=586368.74 lat (usec): min=359, max=1193.3k, avg=460125.01, stdev=586368.98 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], | 20.00th=[ 359], 30.00th=[ 742], 40.00th=[ 742], | 50.00th=[ 107480], 60.00th=[ 107480], 70.00th=[1002439], | 80.00th=[1002439], 90.00th=[1199571], 95.00th=[1199571], | 99.00th=[1199571], 99.50th=[1199571], 99.90th=[1199571], | 99.95th=[1199571], 99.99th=[1199571] bw ( KiB/s): min= 2048, max= 4096, per=72.74%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1318KiB/s (1350kB/s)(7168KiB/5439msec) clat (msec): min=41, max=2512, avg=447.50, stdev=912.92 lat (msec): min=41, max=2513, avg=447.58, stdev=912.92 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 53], | 30.00th=[ 73], 40.00th=[ 73], 50.00th=[ 82], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 218], 90.00th=[ 2500], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 2043, max= 6144, per=90.89%, avg=4777.00, stdev=2367.71, samples=3 iops : min= 1, max= 6, avg= 4.33, stdev= 2.89, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=8.33%, 100=25.00%, 250=25.00%, 1000=8.33% cpu : usr=0.02%, sys=7.25%, ctx=353, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3754KiB/s (3844kB/s), 503KiB/s-1802KiB/s (515kB/s-1845kB/s), io=20.0MiB (20.0MB), run=3978-5455msec WRITE: bw=5256KiB/s (5382kB/s), 1126KiB/s-2515KiB/s (1153kB/s-2575kB/s), io=28.0MiB (29.4MB), run=3978-5455msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8117: Fri Mar 22 07:40:08 2024 read: IOPS=1, BW=1644KiB/s (1683kB/s)(7168KiB/4361msec) clat (msec): min=38, max=1307, avg=564.05, stdev=597.10 lat (msec): min=38, max=1307, avg=564.06, stdev=597.10 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 69], | 30.00th=[ 115], 40.00th=[ 115], 50.00th=[ 131], 60.00th=[ 1116], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 2043, max= 4096, per=76.01%, avg=2727.33, stdev=1185.30, samples=3 iops : min= 1, max= 4, avg= 2.00, stdev= 1.73, samples=3 write: IOPS=1, BW=1174KiB/s (1202kB/s)(5120KiB/4361msec) clat (msec): min=50, max=185, avg=81.57, stdev=58.49 lat (msec): min=50, max=186, avg=81.70, stdev=58.47 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 2048, max= 2048, per=40.76%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=8.33%, 100=41.67%, 250=25.00% cpu : usr=0.00%, sys=3.44%, ctx=64, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8119: Fri Mar 22 07:40:08 2024 read: IOPS=0, BW=479KiB/s (490kB/s)(2048KiB/4276msec) clat (msec): min=53, max=1480, avg=766.84, stdev=1008.89 lat (msec): min=53, max=1480, avg=766.84, stdev=1008.89 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 1485], | 70.00th=[ 1485], 80.00th=[ 1485], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 2043, max= 2043, per=56.94%, avg=2043.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2395KiB/s (2452kB/s)(10.0MiB/4276msec) clat (msec): min=35, max=1292, avg=273.93, stdev=454.99 lat (msec): min=35, max=1292, avg=274.04, stdev=454.99 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 48], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 953], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 2048, max= 8175, per=94.94%, avg=4770.00, stdev=3120.08, samples=3 iops : min= 2, max= 7, avg= 4.00, stdev= 2.65, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=2.90%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8121: Fri Mar 22 07:40:08 2024 read: IOPS=1, BW=1077KiB/s (1102kB/s)(6144KiB/5707msec) clat (msec): min=40, max=1270, avg=434.49, stdev=585.34 lat (msec): min=40, max=1270, avg=434.49, stdev=585.34 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 92], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1077KiB/s (1102kB/s)(6144KiB/5707msec) clat (msec): min=36, max=2611, avg=516.32, stdev=1026.96 lat (msec): min=36, max=2611, avg=516.42, stdev=1026.96 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 140], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 2601], 95.00th=[ 2601], | 99.00th=[ 2601], 99.50th=[ 2601], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 2043, max= 4096, per=67.85%, avg=3408.67, stdev=1182.71, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 lat (msec) : 50=25.00%, 100=33.33%, 250=16.67% cpu : usr=0.02%, sys=2.68%, ctx=97, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8122: Fri Mar 22 07:40:08 2024 read: IOPS=0, BW=924KiB/s (946kB/s)(5120KiB/5543msec) clat (msec): min=34, max=1083, avg=262.42, stdev=459.26 lat (msec): min=34, max=1083, avg=262.43, stdev=459.26 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 2048, max= 6131, per=100.00%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1293KiB/s (1324kB/s)(7168KiB/5543msec) clat (msec): min=62, max=2517, avg=603.46, stdev=916.33 lat (msec): min=62, max=2517, avg=603.55, stdev=916.32 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 93], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 157], 60.00th=[ 213], | 70.00th=[ 213], 80.00th=[ 1083], 90.00th=[ 2534], 95.00th=[ 2534], | 99.00th=[ 2534], 99.50th=[ 2534], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 2048, max= 6144, per=81.47%, avg=4093.00, stdev=2048.01, samples=3 iops : min= 2, max= 6, avg= 3.67, stdev= 2.08, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=25.00% cpu : usr=0.00%, sys=3.05%, ctx=113, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3589KiB/s (3675kB/s), 479KiB/s-1644KiB/s (490kB/s-1683kB/s), io=20.0MiB (20.0MB), run=4276-5707msec WRITE: bw=5024KiB/s (5145kB/s), 1077KiB/s-2395KiB/s (1102kB/s-2452kB/s), io=28.0MiB (29.4MB), run=4276-5707msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8134: Fri Mar 22 07:40:15 2024 read: IOPS=0, BW=2792KiB/s (2859kB/s)(12.0MiB/4401msec) clat (msec): min=141, max=3834, avg=1466.21, stdev=2055.58 lat (msec): min=141, max=3834, avg=1466.21, stdev=2055.58 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 422], 50.00th=[ 422], 60.00th=[ 422], | 70.00th=[ 3842], 80.00th=[ 3842], 90.00th=[ 3842], 95.00th=[ 3842], | 99.00th=[ 3842], 99.50th=[ 3842], 99.90th=[ 3842], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 8143, max= 8192, per=100.00%, avg=8167.50, stdev=34.65, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.14%, sys=25.39%, ctx=963, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8135: Fri Mar 22 07:40:15 2024 read: IOPS=0, BW=771KiB/s (790kB/s)(4096KiB/5311msec) clat (nsec): min=196526k, max=196526k, avg=196525997.00, stdev= 0.00 lat (nsec): min=196527k, max=196527k, avg=196526994.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1542KiB/s (1579kB/s)(8192KiB/5311msec) clat (msec): min=840, max=4256, avg=2548.46, stdev=2415.15 lat (msec): min=841, max=4256, avg=2548.80, stdev=2415.05 clat percentiles (msec): | 1.00th=[ 844], 5.00th=[ 844], 10.00th=[ 844], 20.00th=[ 844], | 30.00th=[ 844], 40.00th=[ 844], 50.00th=[ 844], 60.00th=[ 4245], | 70.00th=[ 4245], 80.00th=[ 4245], 90.00th=[ 4245], 95.00th=[ 4245], | 99.00th=[ 4245], 99.50th=[ 4245], 99.90th=[ 4245], 99.95th=[ 4245], | 99.99th=[ 4245] bw ( KiB/s): min= 7198, max= 7198, per=100.00%, avg=7198.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.08%, sys=5.80%, ctx=275, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8136: Fri Mar 22 07:40:15 2024 write: IOPS=0, BW=2439KiB/s (2498kB/s)(12.0MiB/5038msec) clat (msec): min=619, max=2646, avg=1675.35, stdev=1016.08 lat (msec): min=620, max=2647, avg=1675.68, stdev=1016.21 clat percentiles (msec): | 1.00th=[ 617], 5.00th=[ 617], 10.00th=[ 617], 20.00th=[ 617], | 30.00th=[ 617], 40.00th=[ 1754], 50.00th=[ 1754], 60.00th=[ 1754], | 70.00th=[ 2635], 80.00th=[ 2635], 90.00th=[ 2635], 95.00th=[ 2635], | 99.00th=[ 2635], 99.50th=[ 2635], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 8159, max= 8192, per=100.00%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33% cpu : usr=0.02%, sys=6.65%, ctx=87, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8137: Fri Mar 22 07:40:15 2024 write: IOPS=0, BW=3505KiB/s (3589kB/s)(12.0MiB/3506msec) clat (msec): min=298, max=2902, avg=1167.74, stdev=1502.63 lat (msec): min=298, max=2903, avg=1168.13, stdev=1502.73 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 2903], 80.00th=[ 2903], 90.00th=[ 2903], 95.00th=[ 2903], | 99.00th=[ 2903], 99.50th=[ 2903], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=6.45%, ctx=83, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3085KiB/s (3159kB/s), 771KiB/s-2792KiB/s (790kB/s-2859kB/s), io=16.0MiB (16.8MB), run=4401-5311msec WRITE: bw=6170KiB/s (6318kB/s), 1542KiB/s-3505KiB/s (1579kB/s-3589kB/s), io=32.0MiB (33.6MB), run=3506-5311msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8138: Fri Mar 22 07:40:15 2024 read: IOPS=0, BW=2315KiB/s (2371kB/s)(12.0MiB/5308msec) clat (msec): min=226, max=3304, avg=1768.19, stdev=1538.80 lat (msec): min=226, max=3304, avg=1768.19, stdev=1538.80 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 1770], 50.00th=[ 1770], 60.00th=[ 1770], | 70.00th=[ 3306], 80.00th=[ 3306], 90.00th=[ 3306], 95.00th=[ 3306], | 99.00th=[ 3306], 99.50th=[ 3306], 99.90th=[ 3306], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.09%, sys=2.17%, ctx=37, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8139: Fri Mar 22 07:40:15 2024 read: IOPS=0, BW=741KiB/s (759kB/s)(4096KiB/5529msec) clat (nsec): min=906047k, max=906047k, avg=906046646.00, stdev= 0.00 lat (nsec): min=906050k, max=906050k, avg=906049837.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 911], 5.00th=[ 911], 10.00th=[ 911], 20.00th=[ 911], | 30.00th=[ 911], 40.00th=[ 911], 50.00th=[ 911], 60.00th=[ 911], | 70.00th=[ 911], 80.00th=[ 911], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1482KiB/s (1517kB/s)(8192KiB/5529msec) clat (msec): min=764, max=3848, avg=2306.86, stdev=2180.55 lat (msec): min=765, max=3849, avg=2307.20, stdev=2180.56 clat percentiles (msec): | 1.00th=[ 768], 5.00th=[ 768], 10.00th=[ 768], 20.00th=[ 768], | 30.00th=[ 768], 40.00th=[ 768], 50.00th=[ 768], 60.00th=[ 3842], | 70.00th=[ 3842], 80.00th=[ 3842], 90.00th=[ 3842], 95.00th=[ 3842], | 99.00th=[ 3842], 99.50th=[ 3842], 99.90th=[ 3842], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 7892, max= 7892, per=100.00%, avg=7892.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 1000=66.67% cpu : usr=0.11%, sys=3.44%, ctx=73, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8140: Fri Mar 22 07:40:15 2024 write: IOPS=0, BW=2373KiB/s (2430kB/s)(12.0MiB/5178msec) clat (msec): min=176, max=3233, avg=1718.65, stdev=1529.03 lat (msec): min=180, max=3235, avg=1720.47, stdev=1527.74 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 1754], 50.00th=[ 1754], 60.00th=[ 1754], | 70.00th=[ 3239], 80.00th=[ 3239], 90.00th=[ 3239], 95.00th=[ 3239], | 99.00th=[ 3239], 99.50th=[ 3239], 99.90th=[ 3239], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.04%, sys=2.70%, ctx=48, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8141: Fri Mar 22 07:40:15 2024 write: IOPS=0, BW=2493KiB/s (2553kB/s)(12.0MiB/4929msec) clat (msec): min=174, max=3215, avg=1642.23, stdev=1523.29 lat (msec): min=174, max=3215, avg=1642.53, stdev=1523.27 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 1536], 50.00th=[ 1536], 60.00th=[ 1536], | 70.00th=[ 3205], 80.00th=[ 3205], 90.00th=[ 3205], 95.00th=[ 3205], | 99.00th=[ 3205], 99.50th=[ 3205], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=4.30%, ctx=69, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2963KiB/s (3034kB/s), 741KiB/s-2315KiB/s (759kB/s-2371kB/s), io=16.0MiB (16.8MB), run=5308-5529msec WRITE: bw=5927KiB/s (6069kB/s), 1482KiB/s-2493KiB/s (1517kB/s-2553kB/s), io=32.0MiB (33.6MB), run=4929-5529msec