== sanity test 398b: DIO and buffer IO race ============== 18:29:07 (1710109747) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.975521 s, 51.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29656: Sun Mar 10 18:29:24 2024 read: IOPS=98, BW=394KiB/s (404kB/s)(5992KiB/15199msec) clat (usec): min=40, max=153115, avg=4389.39, stdev=5673.62 lat (usec): min=41, max=153117, avg=4390.47, stdev=5673.82 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 63], 10.00th=[ 75], 20.00th=[ 109], | 30.00th=[ 2343], 40.00th=[ 2835], 50.00th=[ 3359], 60.00th=[ 4178], | 70.00th=[ 5211], 80.00th=[ 6980], 90.00th=[ 9634], 95.00th=[ 11600], | 99.00th=[ 17171], 99.50th=[ 21627], 99.90th=[ 42206], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 247, max= 560, per=24.71%, avg=390.23, stdev=78.88, samples=30 iops : min= 61, max= 140, avg=97.43, stdev=19.81, samples=30 write: IOPS=103, BW=414KiB/s (424kB/s)(6296KiB/15199msec) clat (usec): min=401, max=56991, avg=5429.38, stdev=5361.43 lat (usec): min=402, max=56991, avg=5430.57, stdev=5361.41 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 519], 10.00th=[ 578], 20.00th=[ 758], | 30.00th=[ 1336], 40.00th=[ 2933], 50.00th=[ 4228], 60.00th=[ 5342], | 70.00th=[ 6980], 80.00th=[ 8848], 90.00th=[12125], 95.00th=[15533], | 99.00th=[22938], 99.50th=[26608], 99.90th=[45351], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 247, max= 528, per=24.79%, avg=409.97, stdev=62.14, samples=30 iops : min= 61, max= 132, avg=102.37, stdev=15.61, samples=30 lat (usec) : 50=0.62%, 100=7.94%, 250=3.19%, 500=1.89%, 750=8.30% lat (usec) : 1000=3.58% lat (msec) : 2=4.56%, 4=23.14%, 10=34.02%, 20=11.56%, 50=1.14% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.22%, sys=22.40%, ctx=6458, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29658: Sun Mar 10 18:29:24 2024 read: IOPS=98, BW=392KiB/s (402kB/s)(5940KiB/15136msec) clat (usec): min=40, max=25595, avg=4231.19, stdev=3809.48 lat (usec): min=40, max=25595, avg=4232.45, stdev=3809.54 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 63], 10.00th=[ 73], 20.00th=[ 108], | 30.00th=[ 2343], 40.00th=[ 2835], 50.00th=[ 3392], 60.00th=[ 4178], | 70.00th=[ 5276], 80.00th=[ 6783], 90.00th=[ 9765], 95.00th=[11731], | 99.00th=[15664], 99.50th=[19792], 99.90th=[24773], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 240, max= 544, per=24.84%, avg=392.23, stdev=83.41, samples=30 iops : min= 60, max= 136, avg=97.87, stdev=20.85, samples=30 write: IOPS=104, BW=419KiB/s (429kB/s)(6348KiB/15136msec) clat (usec): min=428, max=159239, avg=5529.50, stdev=6593.90 lat (usec): min=429, max=159240, avg=5531.11, stdev=6593.51 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 519], 10.00th=[ 594], 20.00th=[ 775], | 30.00th=[ 1172], 40.00th=[ 2900], 50.00th=[ 4228], 60.00th=[ 5473], | 70.00th=[ 6915], 80.00th=[ 9241], 90.00th=[ 12256], 95.00th=[ 15533], | 99.00th=[ 22676], 99.50th=[ 26870], 99.90th=[ 57410], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 279, max= 640, per=25.24%, avg=417.50, stdev=87.06, samples=30 iops : min= 69, max= 160, avg=104.20, stdev=21.79, samples=30 lat (usec) : 50=0.59%, 100=8.20%, 250=2.51%, 500=2.08%, 750=7.88% lat (usec) : 1000=4.72% lat (msec) : 2=4.36%, 4=22.53%, 10=33.66%, 20=12.30%, 50=1.07% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.23%, sys=22.72%, ctx=6346, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29660: Sun Mar 10 18:29:24 2024 read: IOPS=99, BW=397KiB/s (406kB/s)(6028KiB/15198msec) clat (usec): min=41, max=73333, avg=4294.07, stdev=4425.76 lat (usec): min=41, max=73334, avg=4295.12, stdev=4425.72 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 62], 10.00th=[ 69], 20.00th=[ 101], | 30.00th=[ 2278], 40.00th=[ 2802], 50.00th=[ 3392], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 7111], 90.00th=[ 9634], 95.00th=[11469], | 99.00th=[16057], 99.50th=[22676], 99.90th=[42206], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 231, max= 536, per=24.94%, avg=393.83, stdev=74.33, samples=30 iops : min= 57, max= 134, avg=98.23, stdev=18.65, samples=30 write: IOPS=102, BW=412KiB/s (422kB/s)(6260KiB/15198msec) clat (usec): min=391, max=175512, avg=5529.81, stdev=6785.63 lat (usec): min=392, max=175514, avg=5530.79, stdev=6785.66 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 502], 10.00th=[ 570], 20.00th=[ 734], | 30.00th=[ 1270], 40.00th=[ 2933], 50.00th=[ 4359], 60.00th=[ 5473], | 70.00th=[ 6783], 80.00th=[ 8848], 90.00th=[ 12387], 95.00th=[ 15926], | 99.00th=[ 22414], 99.50th=[ 26346], 99.90th=[ 38011], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 263, max= 592, per=24.70%, avg=408.50, stdev=89.80, samples=30 iops : min= 65, max= 148, avg=101.90, stdev=22.52, samples=30 lat (usec) : 50=0.91%, 100=8.79%, 250=2.77%, 500=2.70%, 750=8.07% lat (usec) : 1000=3.61% lat (msec) : 2=4.82%, 4=21.35%, 10=34.24%, 20=11.17%, 50=1.50% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=22.45%, ctx=6375, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29662: Sun Mar 10 18:29:24 2024 read: IOPS=100, BW=402KiB/s (411kB/s)(6052KiB/15066msec) clat (usec): min=40, max=78522, avg=4217.86, stdev=4439.11 lat (usec): min=40, max=78522, avg=4218.84, stdev=4439.15 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 62], 10.00th=[ 70], 20.00th=[ 99], | 30.00th=[ 2278], 40.00th=[ 2737], 50.00th=[ 3294], 60.00th=[ 3982], | 70.00th=[ 5080], 80.00th=[ 6915], 90.00th=[ 9634], 95.00th=[11731], | 99.00th=[16319], 99.50th=[21365], 99.90th=[35390], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 232, max= 574, per=25.35%, avg=400.20, stdev=83.96, samples=30 iops : min= 58, max= 143, avg=99.83, stdev=20.99, samples=30 write: IOPS=103, BW=414KiB/s (424kB/s)(6236KiB/15066msec) clat (usec): min=370, max=177230, avg=5520.77, stdev=6963.82 lat (usec): min=371, max=177231, avg=5521.82, stdev=6963.80 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 506], 10.00th=[ 562], 20.00th=[ 725], | 30.00th=[ 1029], 40.00th=[ 2638], 50.00th=[ 4113], 60.00th=[ 5211], | 70.00th=[ 6915], 80.00th=[ 9241], 90.00th=[ 12780], 95.00th=[ 15926], | 99.00th=[ 24249], 99.50th=[ 28181], 99.90th=[ 40633], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 216, max= 598, per=24.94%, avg=412.47, stdev=79.74, samples=30 iops : min= 54, max= 149, avg=102.90, stdev=19.97, samples=30 lat (usec) : 50=0.91%, 100=9.18%, 250=2.05%, 500=2.38%, 750=8.95% lat (usec) : 1000=3.84% lat (msec) : 2=4.49%, 4=22.88%, 10=32.16%, 20=11.75%, 50=1.33% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=22.58%, ctx=6281, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1580KiB/s (1618kB/s), 392KiB/s-402KiB/s (402kB/s-411kB/s), io=23.4MiB (24.6MB), run=15066-15199msec WRITE: bw=1654KiB/s (1694kB/s), 412KiB/s-419KiB/s (422kB/s-429kB/s), io=24.6MiB (25.7MB), run=15066-15199msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29655: Sun Mar 10 18:29:56 2024 read: IOPS=32, BW=128KiB/s (131kB/s)(5992KiB/46762msec) clat (usec): min=1586, max=175468, avg=4104.95, stdev=5975.47 lat (usec): min=1586, max=175469, avg=4105.98, stdev=5975.47 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2573], 60.00th=[ 2737], | 70.00th=[ 3064], 80.00th=[ 3818], 90.00th=[ 9110], 95.00th=[ 12256], | 99.00th=[ 20055], 99.50th=[ 27657], 99.90th=[ 41681], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 31, max= 256, per=25.10%, avg=127.76, stdev=55.94, samples=93 iops : min= 7, max= 64, avg=31.83, stdev=14.05, samples=93 write: IOPS=33, BW=135KiB/s (138kB/s)(6296KiB/46762msec) clat (usec): min=10654, max=91713, avg=25753.23, stdev=10097.99 lat (usec): min=10655, max=91713, avg=25754.50, stdev=10097.98 clat percentiles (usec): | 1.00th=[13566], 5.00th=[14877], 10.00th=[15795], 20.00th=[17433], | 30.00th=[19530], 40.00th=[21890], 50.00th=[23987], 60.00th=[25560], | 70.00th=[27657], 80.00th=[30540], 90.00th=[40109], 95.00th=[47449], | 99.00th=[58459], 99.50th=[64750], 99.90th=[77071], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 56, max= 192, per=25.20%, avg=134.30, stdev=40.17, samples=93 iops : min= 14, max= 48, avg=33.46, stdev=10.11, samples=93 lat (msec) : 2=4.43%, 4=35.16%, 10=5.18%, 20=19.82%, 50=33.56% lat (msec) : 100=1.82%, 250=0.03% cpu : usr=0.11%, sys=4.09%, ctx=3317, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29657: Sun Mar 10 18:29:56 2024 read: IOPS=31, BW=127KiB/s (130kB/s)(5940KiB/46857msec) clat (usec): min=1632, max=163458, avg=4053.29, stdev=5693.90 lat (usec): min=1632, max=163460, avg=4054.31, stdev=5693.91 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1893], 10.00th=[ 2008], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2737], | 70.00th=[ 3064], 80.00th=[ 3884], 90.00th=[ 8455], 95.00th=[ 12387], | 99.00th=[ 22152], 99.50th=[ 26608], 99.90th=[ 39060], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 16, max= 280, per=24.92%, avg=126.84, stdev=59.01, samples=93 iops : min= 4, max= 70, avg=31.68, stdev=14.77, samples=93 write: IOPS=33, BW=135KiB/s (139kB/s)(6348KiB/46857msec) clat (msec): min=12, max=103, avg=25.68, stdev=10.19 lat (msec): min=12, max=103, avg=25.69, stdev=10.19 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 26], | 70.00th=[ 28], 80.00th=[ 31], 90.00th=[ 42], 95.00th=[ 48], | 99.00th=[ 59], 99.50th=[ 63], 99.90th=[ 67], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 64, max= 200, per=25.35%, avg=135.10, stdev=41.51, samples=93 iops : min= 16, max= 50, avg=33.74, stdev=10.40, samples=93 lat (msec) : 2=4.85%, 4=34.15%, 10=5.50%, 20=20.54%, 50=33.04% lat (msec) : 100=1.86%, 250=0.07% cpu : usr=0.12%, sys=4.09%, ctx=3341, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29659: Sun Mar 10 18:29:56 2024 read: IOPS=31, BW=128KiB/s (131kB/s)(6028KiB/47133msec) clat (usec): min=1625, max=41401, avg=4100.70, stdev=4120.62 lat (usec): min=1626, max=41402, avg=4101.71, stdev=4120.60 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3130], 80.00th=[ 4113], 90.00th=[ 9372], 95.00th=[12387], | 99.00th=[21890], 99.50th=[25297], 99.90th=[38011], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 32, max= 248, per=25.07%, avg=127.63, stdev=54.55, samples=94 iops : min= 8, max= 62, avg=31.85, stdev=13.64, samples=94 write: IOPS=33, BW=133KiB/s (136kB/s)(6260KiB/47133msec) clat (msec): min=11, max=173, avg=26.12, stdev=10.81 lat (msec): min=11, max=173, avg=26.12, stdev=10.81 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 23], 50.00th=[ 25], 60.00th=[ 27], | 70.00th=[ 29], 80.00th=[ 32], 90.00th=[ 41], 95.00th=[ 48], | 99.00th=[ 62], 99.50th=[ 63], 99.90th=[ 86], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 56, max= 220, per=24.86%, avg=132.49, stdev=41.67, samples=94 iops : min= 14, max= 55, avg=33.07, stdev=10.43, samples=94 lat (msec) : 2=4.52%, 4=34.28%, 10=5.70%, 20=19.79%, 50=33.46% lat (msec) : 100=2.21%, 250=0.03% cpu : usr=0.14%, sys=4.02%, ctx=3286, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29661: Sun Mar 10 18:29:56 2024 read: IOPS=32, BW=129KiB/s (133kB/s)(6052KiB/46748msec) clat (usec): min=1628, max=69441, avg=3961.15, stdev=4262.07 lat (usec): min=1629, max=69442, avg=3962.20, stdev=4262.08 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1942], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2769], | 70.00th=[ 3097], 80.00th=[ 3884], 90.00th=[ 8160], 95.00th=[12125], | 99.00th=[21890], 99.50th=[27395], 99.90th=[46400], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 16, max= 272, per=25.42%, avg=129.39, stdev=54.41, samples=93 iops : min= 4, max= 68, avg=32.29, stdev=13.63, samples=93 write: IOPS=33, BW=133KiB/s (137kB/s)(6236KiB/46748msec) clat (msec): min=12, max=174, avg=26.09, stdev=10.48 lat (msec): min=12, max=174, avg=26.09, stdev=10.48 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 25], 60.00th=[ 27], | 70.00th=[ 28], 80.00th=[ 31], 90.00th=[ 41], 95.00th=[ 48], | 99.00th=[ 59], 99.50th=[ 61], 99.90th=[ 86], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 56, max= 192, per=24.99%, avg=133.17, stdev=39.38, samples=93 iops : min= 14, max= 48, avg=33.24, stdev= 9.86, samples=93 lat (msec) : 2=4.13%, 4=35.71%, 10=5.66%, 20=18.10%, 50=34.34% lat (msec) : 100=2.02%, 250=0.03% cpu : usr=0.08%, sys=4.10%, ctx=3318, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=509KiB/s (522kB/s), 127KiB/s-129KiB/s (130kB/s-133kB/s), io=23.4MiB (24.6MB), run=46748-47133msec WRITE: bw=533KiB/s (546kB/s), 133KiB/s-135KiB/s (136kB/s-139kB/s), io=24.6MiB (25.7MB), run=46748-47133msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29709: Sun Mar 10 18:30:02 2024 read: IOPS=76, BW=1217KiB/s (1246kB/s)(5920KiB/4866msec) clat (usec): min=46, max=39708, avg=9431.48, stdev=6029.30 lat (usec): min=46, max=39709, avg=9432.98, stdev=6028.50 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 68], 10.00th=[ 77], 20.00th=[ 141], | 30.00th=[ 8848], 40.00th=[ 9765], 50.00th=[10683], 60.00th=[11731], | 70.00th=[12649], 80.00th=[13698], 90.00th=[15008], 95.00th=[16909], | 99.00th=[24249], 99.50th=[35390], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 736, max= 1504, per=25.61%, avg=1166.22, stdev=203.72, samples=9 iops : min= 46, max= 94, avg=72.89, stdev=12.73, samples=9 write: IOPS=81, BW=1309KiB/s (1340kB/s)(6368KiB/4866msec) clat (usec): min=583, max=34223, avg=3432.78, stdev=4244.21 lat (usec): min=584, max=34225, avg=3434.13, stdev=4244.19 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 709], 10.00th=[ 775], 20.00th=[ 873], | 30.00th=[ 996], 40.00th=[ 1139], 50.00th=[ 1352], 60.00th=[ 2933], | 70.00th=[ 3949], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[10683], | 99.00th=[22676], 99.50th=[33162], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 800, max= 1728, per=26.80%, avg=1283.56, stdev=276.41, samples=9 iops : min= 50, max= 108, avg=80.22, stdev=17.28, samples=9 lat (usec) : 50=0.26%, 100=6.90%, 250=3.78%, 750=3.91%, 1000=12.24% lat (msec) : 2=13.02%, 4=7.94%, 10=21.87%, 20=28.65%, 50=1.43% cpu : usr=0.18%, sys=21.50%, ctx=1936, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29711: Sun Mar 10 18:30:02 2024 read: IOPS=71, BW=1147KiB/s (1174kB/s)(5744KiB/5009msec) clat (usec): min=57, max=34344, avg=10094.70, stdev=5889.28 lat (usec): min=57, max=34345, avg=10095.53, stdev=5889.28 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 71], 10.00th=[ 88], 20.00th=[ 204], | 30.00th=[ 9503], 40.00th=[10421], 50.00th=[11207], 60.00th=[12125], | 70.00th=[13042], 80.00th=[14091], 90.00th=[16057], 95.00th=[17957], | 99.00th=[22938], 99.50th=[26870], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 992, max= 1376, per=25.23%, avg=1148.80, stdev=139.44, samples=10 iops : min= 62, max= 86, avg=71.80, stdev= 8.72, samples=10 write: IOPS=81, BW=1306KiB/s (1338kB/s)(6544KiB/5009msec) clat (usec): min=559, max=42525, avg=3367.53, stdev=3812.04 lat (usec): min=559, max=42526, avg=3368.91, stdev=3811.96 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 717], 10.00th=[ 783], 20.00th=[ 898], | 30.00th=[ 1045], 40.00th=[ 1156], 50.00th=[ 1909], 60.00th=[ 3163], | 70.00th=[ 4015], 80.00th=[ 5407], 90.00th=[ 7308], 95.00th=[ 8717], | 99.00th=[15401], 99.50th=[23987], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 896, max= 1632, per=27.19%, avg=1302.40, stdev=251.09, samples=10 iops : min= 56, max= 102, avg=81.40, stdev=15.69, samples=10 lat (usec) : 100=6.38%, 250=2.99%, 750=3.52%, 1000=10.81% lat (msec) : 2=12.50%, 4=10.55%, 10=20.70%, 20=30.99%, 50=1.56% cpu : usr=0.22%, sys=21.31%, ctx=1997, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29713: Sun Mar 10 18:30:02 2024 read: IOPS=74, BW=1189KiB/s (1218kB/s)(6256KiB/5260msec) clat (usec): min=42, max=40360, avg=9935.34, stdev=6092.69 lat (usec): min=43, max=40361, avg=9937.52, stdev=6092.31 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 67], 10.00th=[ 78], 20.00th=[ 215], | 30.00th=[ 8979], 40.00th=[10028], 50.00th=[10945], 60.00th=[11863], | 70.00th=[12780], 80.00th=[13960], 90.00th=[15795], 95.00th=[17695], | 99.00th=[28181], 99.50th=[36439], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 672, max= 1536, per=25.09%, avg=1142.40, stdev=219.15, samples=10 iops : min= 42, max= 96, avg=71.40, stdev=13.70, samples=10 write: IOPS=71, BW=1147KiB/s (1174kB/s)(6032KiB/5260msec) clat (usec): min=557, max=41226, avg=3621.49, stdev=4756.19 lat (usec): min=558, max=41228, avg=3622.80, stdev=4756.20 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 668], 10.00th=[ 750], 20.00th=[ 857], | 30.00th=[ 1004], 40.00th=[ 1188], 50.00th=[ 1713], 60.00th=[ 3097], | 70.00th=[ 3884], 80.00th=[ 5407], 90.00th=[ 7963], 95.00th=[11731], | 99.00th=[24511], 99.50th=[35390], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 864, max= 1440, per=22.71%, avg=1088.00, stdev=206.83, samples=10 iops : min= 54, max= 90, avg=68.00, stdev=12.93, samples=10 lat (usec) : 50=0.65%, 100=6.51%, 250=3.12%, 500=0.13%, 750=4.95% lat (usec) : 1000=9.77% lat (msec) : 2=10.42%, 4=9.51%, 10=21.74%, 20=30.86%, 50=2.34% cpu : usr=0.19%, sys=20.56%, ctx=2031, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29715: Sun Mar 10 18:30:02 2024 read: IOPS=75, BW=1213KiB/s (1242kB/s)(6032KiB/4973msec) clat (usec): min=47, max=37539, avg=9485.45, stdev=6023.35 lat (usec): min=47, max=37540, avg=9486.23, stdev=6023.34 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 68], 10.00th=[ 85], 20.00th=[ 106], | 30.00th=[ 8848], 40.00th=[ 9765], 50.00th=[10552], 60.00th=[11469], | 70.00th=[12780], 80.00th=[14091], 90.00th=[16319], 95.00th=[17433], | 99.00th=[21365], 99.50th=[23725], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 928, max= 1472, per=26.00%, avg=1183.67, stdev=161.82, samples=9 iops : min= 58, max= 92, avg=73.89, stdev=10.18, samples=9 write: IOPS=78, BW=1258KiB/s (1288kB/s)(6256KiB/4973msec) clat (usec): min=603, max=38843, avg=3548.75, stdev=4533.47 lat (usec): min=604, max=38845, avg=3550.12, stdev=4533.45 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 717], 10.00th=[ 791], 20.00th=[ 873], | 30.00th=[ 996], 40.00th=[ 1123], 50.00th=[ 1336], 60.00th=[ 2835], | 70.00th=[ 3654], 80.00th=[ 5604], 90.00th=[ 8225], 95.00th=[12518], | 99.00th=[22676], 99.50th=[31065], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 1088, max= 1568, per=26.20%, avg=1254.78, stdev=160.99, samples=9 iops : min= 68, max= 98, avg=78.33, stdev= 9.95, samples=9 lat (usec) : 50=0.13%, 100=8.72%, 250=2.60%, 750=3.39%, 1000=12.37% lat (msec) : 2=13.15%, 4=8.46%, 10=19.40%, 20=30.08%, 50=1.69% cpu : usr=0.12%, sys=20.88%, ctx=1940, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4554KiB/s (4663kB/s), 1147KiB/s-1217KiB/s (1174kB/s-1246kB/s), io=23.4MiB (24.5MB), run=4866-5260msec WRITE: bw=4791KiB/s (4906kB/s), 1147KiB/s-1309KiB/s (1174kB/s-1340kB/s), io=24.6MiB (25.8MB), run=4866-5260msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29710: Sun Mar 10 18:30:09 2024 read: IOPS=33, BW=534KiB/s (546kB/s)(5920KiB/11096msec) clat (usec): min=1940, max=30696, avg=4445.08, stdev=3757.93 lat (usec): min=1941, max=30697, avg=4446.12, stdev=3757.90 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2474], | 30.00th=[ 2573], 40.00th=[ 2737], 50.00th=[ 2900], 60.00th=[ 3195], | 70.00th=[ 3884], 80.00th=[ 5276], 90.00th=[ 8848], 95.00th=[12256], | 99.00th=[19268], 99.50th=[26870], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 320, max= 864, per=25.20%, avg=529.27, stdev=173.10, samples=22 iops : min= 20, max= 54, avg=33.00, stdev=10.81, samples=22 write: IOPS=35, BW=574KiB/s (588kB/s)(6368KiB/11096msec) clat (usec): min=11169, max=60473, avg=23724.50, stdev=8008.01 lat (usec): min=11171, max=60475, avg=23726.37, stdev=8008.09 clat percentiles (usec): | 1.00th=[12256], 5.00th=[14091], 10.00th=[14746], 20.00th=[16319], | 30.00th=[18744], 40.00th=[21103], 50.00th=[22414], 60.00th=[23987], | 70.00th=[26084], 80.00th=[28967], 90.00th=[34866], 95.00th=[39584], | 99.00th=[46924], 99.50th=[57410], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 352, max= 796, per=25.86%, avg=571.41, stdev=153.79, samples=22 iops : min= 22, max= 49, avg=35.64, stdev= 9.60, samples=22 lat (msec) : 2=0.39%, 4=33.46%, 10=10.29%, 20=21.09%, 50=34.37% lat (msec) : 100=0.39% cpu : usr=0.15%, sys=5.28%, ctx=867, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29712: Sun Mar 10 18:30:09 2024 read: IOPS=31, BW=504KiB/s (516kB/s)(5744KiB/11402msec) clat (usec): min=1794, max=36580, avg=4445.85, stdev=4185.28 lat (usec): min=1795, max=36581, avg=4446.84, stdev=4185.25 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2442], | 30.00th=[ 2573], 40.00th=[ 2704], 50.00th=[ 2835], 60.00th=[ 3097], | 70.00th=[ 3752], 80.00th=[ 5211], 90.00th=[ 8356], 95.00th=[12518], | 99.00th=[22414], 99.50th=[32637], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 159, max= 992, per=23.89%, avg=501.64, stdev=210.22, samples=22 iops : min= 9, max= 62, avg=31.27, stdev=13.18, samples=22 write: IOPS=35, BW=574KiB/s (588kB/s)(6544KiB/11402msec) clat (usec): min=11891, max=82956, avg=23947.26, stdev=8331.39 lat (usec): min=11892, max=82958, avg=23949.21, stdev=8331.21 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13960], 10.00th=[14877], 20.00th=[16712], | 30.00th=[19006], 40.00th=[21103], 50.00th=[22414], 60.00th=[24249], | 70.00th=[26346], 80.00th=[29492], 90.00th=[35914], 95.00th=[39584], | 99.00th=[46924], 99.50th=[52167], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 352, max= 736, per=25.40%, avg=561.27, stdev=139.11, samples=22 iops : min= 22, max= 46, avg=35.00, stdev= 8.70, samples=22 lat (msec) : 2=0.39%, 4=33.59%, 10=9.37%, 20=20.96%, 50=35.29% lat (msec) : 100=0.39% cpu : usr=0.15%, sys=5.18%, ctx=872, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29714: Sun Mar 10 18:30:09 2024 read: IOPS=36, BW=577KiB/s (591kB/s)(6256KiB/10838msec) clat (usec): min=1890, max=41747, avg=4615.95, stdev=4511.52 lat (usec): min=1891, max=41748, avg=4616.88, stdev=4511.50 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2245], 10.00th=[ 2311], 20.00th=[ 2442], | 30.00th=[ 2573], 40.00th=[ 2737], 50.00th=[ 2933], 60.00th=[ 3130], | 70.00th=[ 3818], 80.00th=[ 5276], 90.00th=[ 9241], 95.00th=[13435], | 99.00th=[23200], 99.50th=[30016], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 192, max= 1152, per=27.64%, avg=580.48, stdev=247.92, samples=21 iops : min= 12, max= 72, avg=36.24, stdev=15.49, samples=21 write: IOPS=34, BW=557KiB/s (570kB/s)(6032KiB/10838msec) clat (usec): min=11727, max=57986, avg=23936.35, stdev=8231.22 lat (usec): min=11728, max=57988, avg=23938.01, stdev=8231.22 clat percentiles (usec): | 1.00th=[13173], 5.00th=[14091], 10.00th=[14877], 20.00th=[16319], | 30.00th=[18482], 40.00th=[20317], 50.00th=[22938], 60.00th=[24249], | 70.00th=[26870], 80.00th=[29492], 90.00th=[34866], 95.00th=[40633], | 99.00th=[48497], 99.50th=[57410], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 383, max= 864, per=24.89%, avg=550.05, stdev=150.02, samples=21 iops : min= 23, max= 54, avg=34.33, stdev= 9.43, samples=21 lat (msec) : 2=0.26%, 4=37.11%, 10=8.85%, 20=21.74%, 50=31.64% lat (msec) : 100=0.39% cpu : usr=0.07%, sys=5.50%, ctx=889, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29716: Sun Mar 10 18:30:09 2024 read: IOPS=33, BW=538KiB/s (551kB/s)(6032KiB/11209msec) clat (usec): min=1980, max=24125, avg=4270.27, stdev=3530.01 lat (usec): min=1981, max=24126, avg=4271.35, stdev=3529.94 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2507], | 30.00th=[ 2606], 40.00th=[ 2737], 50.00th=[ 2868], 60.00th=[ 3097], | 70.00th=[ 3589], 80.00th=[ 4621], 90.00th=[ 8029], 95.00th=[11600], | 99.00th=[21627], 99.50th=[22414], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 224, max= 1024, per=25.40%, avg=533.41, stdev=181.01, samples=22 iops : min= 14, max= 64, avg=33.14, stdev=11.29, samples=22 write: IOPS=34, BW=558KiB/s (572kB/s)(6256KiB/11209msec) clat (usec): min=11456, max=64307, avg=24528.46, stdev=7761.69 lat (usec): min=11457, max=64309, avg=24530.37, stdev=7761.75 clat percentiles (usec): | 1.00th=[12911], 5.00th=[14615], 10.00th=[15926], 20.00th=[17957], | 30.00th=[20055], 40.00th=[21890], 50.00th=[23462], 60.00th=[25297], | 70.00th=[26870], 80.00th=[29492], 90.00th=[34341], 95.00th=[38536], | 99.00th=[50070], 99.50th=[55837], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 383, max= 736, per=25.13%, avg=555.27, stdev=138.01, samples=22 iops : min= 23, max= 46, avg=34.50, stdev= 8.65, samples=22 lat (msec) : 2=0.39%, 4=36.33%, 10=8.46%, 20=18.49%, 50=35.81% lat (msec) : 100=0.52% cpu : usr=0.12%, sys=5.23%, ctx=855, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2101KiB/s (2151kB/s), 504KiB/s-577KiB/s (516kB/s-591kB/s), io=23.4MiB (24.5MB), run=10838-11402msec WRITE: bw=2210KiB/s (2263kB/s), 557KiB/s-574KiB/s (570kB/s-588kB/s), io=24.6MiB (25.8MB), run=10838-11402msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29732: Sun Mar 10 18:30:12 2024 read: IOPS=3, BW=3945KiB/s (4040kB/s)(7168KiB/1817msec) clat (usec): min=445, max=619731, avg=227530.56, stdev=270064.94 lat (usec): min=447, max=619731, avg=227531.92, stdev=270064.90 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 445], 10.00th=[ 445], 20.00th=[ 498], | 30.00th=[ 4817], 40.00th=[ 4817], 50.00th=[ 59507], 60.00th=[438305], | 70.00th=[438305], 80.00th=[471860], 90.00th=[616563], 95.00th=[616563], | 99.00th=[616563], 99.50th=[616563], 99.90th=[616563], 99.95th=[616563], | 99.99th=[616563] bw ( KiB/s): min= 2048, max= 4096, per=40.64%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2818KiB/s (2885kB/s)(5120KiB/1817msec) clat (usec): min=20388, max=70710, avg=44573.67, stdev=21888.64 lat (usec): min=20452, max=70772, avg=44624.00, stdev=21894.81 clat percentiles (usec): | 1.00th=[20317], 5.00th=[20317], 10.00th=[20317], 20.00th=[20317], | 30.00th=[23725], 40.00th=[23725], 50.00th=[49546], 60.00th=[49546], | 70.00th=[58459], 80.00th=[58459], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 2048, max= 2048, per=19.35%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 50=25.00%, 100=25.00%, 500=16.67%, 750=8.33% cpu : usr=0.06%, sys=20.87%, ctx=923, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29734: Sun Mar 10 18:30:12 2024 read: IOPS=0, BW=756KiB/s (774kB/s)(2048KiB/2709msec) clat (msec): min=62, max=1136, avg=599.54, stdev=759.03 lat (msec): min=62, max=1136, avg=599.54, stdev=759.03 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 63], 60.00th=[ 1133], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 1659, max= 2048, per=24.52%, avg=1853.50, stdev=275.06, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3780KiB/s (3871kB/s)(10.0MiB/2709msec) clat (msec): min=18, max=963, avg=150.72, stdev=291.72 lat (msec): min=18, max=963, avg=150.80, stdev=291.72 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 21], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 32], 60.00th=[ 37], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 178], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 6638, max=10240, per=79.74%, avg=8439.00, stdev=2547.00, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=8.33%, 250=16.67%, 1000=8.33% cpu : usr=0.00%, sys=31.91%, ctx=411, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29735: Sun Mar 10 18:30:12 2024 read: IOPS=3, BW=3097KiB/s (3171kB/s)(6144KiB/1984msec) clat (usec): min=371, max=109021, avg=46790.29, stdev=52172.74 lat (usec): min=372, max=109023, avg=47053.06, stdev=52486.43 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], 20.00th=[ 490], | 30.00th=[ 490], 40.00th=[ 545], 50.00th=[ 545], 60.00th=[ 71828], | 70.00th=[ 98042], 80.00th=[ 98042], 90.00th=[108528], 95.00th=[108528], | 99.00th=[108528], 99.50th=[108528], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 4096, max= 4096, per=54.19%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3097KiB/s (3171kB/s)(6144KiB/1984msec) clat (msec): min=28, max=896, avg=283.38, stdev=355.88 lat (msec): min=28, max=896, avg=283.41, stdev=355.89 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 82], 50.00th=[ 82], 60.00th=[ 95], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2048, max= 4096, per=29.03%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=8.33%, 100=41.67%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=9.18%, ctx=116, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29736: Sun Mar 10 18:30:12 2024 read: IOPS=2, BW=2606KiB/s (2668kB/s)(5120KiB/1965msec) clat (usec): min=381, max=461511, avg=104046.05, stdev=201040.76 lat (usec): min=382, max=461512, avg=104047.28, stdev=201041.04 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 383], 10.00th=[ 383], 20.00th=[ 383], | 30.00th=[ 570], 40.00th=[ 570], 50.00th=[ 5080], 60.00th=[ 5080], | 70.00th=[ 52691], 80.00th=[ 52691], 90.00th=[463471], 95.00th=[463471], | 99.00th=[463471], 99.50th=[463471], 99.90th=[463471], 99.95th=[463471], | 99.99th=[463471] bw ( KiB/s): min= 2048, max= 2048, per=27.09%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3648KiB/s (3735kB/s)(7168KiB/1965msec) clat (msec): min=17, max=924, avg=205.96, stdev=323.68 lat (msec): min=17, max=924, avg=206.20, stdev=323.70 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 42], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 74], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 222], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 2048, max= 6144, per=38.70%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 10=8.33%, 20=8.33%, 50=8.33%, 100=25.00%, 250=16.67% lat (msec) : 500=8.33%, 1000=8.33% cpu : usr=0.05%, sys=13.34%, ctx=346, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7560KiB/s (7741kB/s), 756KiB/s-3945KiB/s (774kB/s-4040kB/s), io=20.0MiB (20.0MB), run=1817-2709msec WRITE: bw=10.3MiB/s (10.8MB/s), 2818KiB/s-3780KiB/s (2885kB/s-3871kB/s), io=28.0MiB (29.4MB), run=1817-2709msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29737: Sun Mar 10 18:30:12 2024 read: IOPS=3, BW=3388KiB/s (3469kB/s)(7168KiB/2116msec) clat (msec): min=19, max=575, avg=239.75, stdev=241.22 lat (msec): min=19, max=575, avg=239.76, stdev=241.22 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 45], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 87], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 468], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 2048, max= 4096, per=45.51%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=2, BW=2420KiB/s (2478kB/s)(5120KiB/2116msec) clat (msec): min=45, max=139, avg=86.26, stdev=36.50 lat (msec): min=45, max=139, avg=86.35, stdev=36.49 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 77], 60.00th=[ 77], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 2048, max= 4096, per=29.28%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=16.67%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.05%, sys=6.62%, ctx=67, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29738: Sun Mar 10 18:30:12 2024 read: IOPS=0, BW=749KiB/s (767kB/s)(2048KiB/2733msec) clat (msec): min=109, max=626, avg=368.05, stdev=365.04 lat (msec): min=109, max=626, avg=368.05, stdev=365.04 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 1678, max= 2048, per=24.86%, avg=1863.00, stdev=261.63, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3747KiB/s (3837kB/s)(10.0MiB/2733msec) clat (msec): min=30, max=953, avg=198.73, stdev=288.42 lat (msec): min=30, max=953, avg=198.79, stdev=288.42 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 37], | 30.00th=[ 41], 40.00th=[ 58], 50.00th=[ 69], 60.00th=[ 85], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 409], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 2048, max= 8192, per=53.87%, avg=5651.33, stdev=3206.89, samples=3 iops : min= 2, max= 8, avg= 5.33, stdev= 3.06, samples=3 lat (msec) : 50=25.00%, 100=25.00%, 250=25.00%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=5.75%, ctx=155, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29739: Sun Mar 10 18:30:12 2024 read: IOPS=2, BW=2655KiB/s (2719kB/s)(6144KiB/2314msec) clat (msec): min=21, max=227, avg=89.58, stdev=73.55 lat (msec): min=21, max=227, avg=89.58, stdev=73.55 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 90], | 70.00th=[ 97], 80.00th=[ 97], 90.00th=[ 228], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 4096, max= 4096, per=54.66%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2655KiB/s (2719kB/s)(6144KiB/2314msec) clat (msec): min=46, max=957, avg=295.67, stdev=375.21 lat (msec): min=47, max=957, avg=295.73, stdev=375.22 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 124], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 2048, max= 4096, per=32.54%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 250=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.66%, ctx=113, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29740: Sun Mar 10 18:30:12 2024 read: IOPS=2, BW=2228KiB/s (2281kB/s)(5120KiB/2298msec) clat (msec): min=27, max=369, avg=147.56, stdev=147.31 lat (msec): min=27, max=369, avg=147.56, stdev=147.31 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 228], 80.00th=[ 228], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 2048, max= 6144, per=54.66%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3119KiB/s (3194kB/s)(7168KiB/2298msec) clat (msec): min=75, max=945, avg=222.04, stdev=319.65 lat (msec): min=75, max=945, avg=222.12, stdev=319.65 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 78], | 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 115], 60.00th=[ 118], | 70.00th=[ 118], 80.00th=[ 136], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 2048, max= 6144, per=39.04%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=33.33%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=6.27%, ctx=197, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7494KiB/s (7673kB/s), 749KiB/s-3388KiB/s (767kB/s-3469kB/s), io=20.0MiB (20.0MB), run=2116-2733msec WRITE: bw=10.2MiB/s (10.7MB/s), 2420KiB/s-3747KiB/s (2478kB/s-3837kB/s), io=28.0MiB (29.4MB), run=2116-2733msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29748: Sun Mar 10 18:30:15 2024 read: IOPS=2, BW=10.6MiB/s (11.2MB/s)(12.0MiB/1128msec) clat (msec): min=246, max=593, avg=375.48, stdev=189.93 lat (msec): min=246, max=593, avg=375.49, stdev=189.93 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 8192, max= 8192, per=69.06%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=68.06%, ctx=186, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29749: Sun Mar 10 18:30:15 2024 read: IOPS=0, BW=2966KiB/s (3037kB/s)(4096KiB/1381msec) clat (nsec): min=135491k, max=135491k, avg=135490942.00, stdev= 0.00 lat (nsec): min=135493k, max=135493k, avg=135492678.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 8192, max= 8192, per=69.06%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5932KiB/s (6074kB/s)(8192KiB/1381msec) clat (msec): min=281, max=951, avg=616.55, stdev=473.49 lat (msec): min=282, max=951, avg=616.82, stdev=473.48 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 953], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=11.96%, ctx=71, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29750: Sun Mar 10 18:30:15 2024 write: IOPS=2, BW=8611KiB/s (8818kB/s)(12.0MiB/1427msec) clat (msec): min=119, max=1016, avg=473.31, stdev=477.83 lat (msec): min=119, max=1017, avg=473.59, stdev=477.83 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 1020], 80.00th=[ 1020], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=16.62%, ctx=61, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29751: Sun Mar 10 18:30:15 2024 write: IOPS=1, BW=8079KiB/s (8273kB/s)(12.0MiB/1521msec) clat (msec): min=183, max=867, avg=506.37, stdev=343.66 lat (msec): min=183, max=868, avg=506.60, stdev=343.73 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 468], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 8192, max= 8192, per=38.03%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=15.72%, ctx=79, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.6MiB/s (12.1MB/s), 2966KiB/s-10.6MiB/s (3037kB/s-11.2MB/s), io=16.0MiB (16.8MB), run=1128-1381msec WRITE: bw=21.0MiB/s (22.1MB/s), 5932KiB/s-8611KiB/s (6074kB/s-8818kB/s), io=32.0MiB (33.6MB), run=1381-1521msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29752: Sun Mar 10 18:30:15 2024 read: IOPS=1, BW=7969KiB/s (8160kB/s)(12.0MiB/1542msec) clat (msec): min=110, max=1177, avg=510.31, stdev=581.26 lat (msec): min=110, max=1177, avg=510.31, stdev=581.26 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=6.16%, ctx=53, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29753: Sun Mar 10 18:30:15 2024 read: IOPS=0, BW=2407KiB/s (2464kB/s)(4096KiB/1702msec) clat (nsec): min=1229.5M, max=1229.5M, avg=1229477849.00, stdev= 0.00 lat (nsec): min=1229.5M, max=1229.5M, avg=1229480704.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1234], 5.00th=[ 1234], 10.00th=[ 1234], 20.00th=[ 1234], | 30.00th=[ 1234], 40.00th=[ 1234], 50.00th=[ 1234], 60.00th=[ 1234], | 70.00th=[ 1234], 80.00th=[ 1234], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 8175, max= 8175, per=84.93%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4813KiB/s (4929kB/s)(8192KiB/1702msec) clat (msec): min=225, max=238, avg=231.89, stdev= 9.43 lat (msec): min=225, max=238, avg=232.17, stdev= 9.42 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 226], | 30.00th=[ 226], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 239], 80.00th=[ 239], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 8175, max= 8175, per=43.29%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=7.00%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29754: Sun Mar 10 18:30:15 2024 write: IOPS=1, BW=7082KiB/s (7252kB/s)(12.0MiB/1735msec) clat (msec): min=75, max=1360, avg=577.50, stdev=686.74 lat (msec): min=75, max=1360, avg=577.80, stdev=686.76 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 75], | 30.00th=[ 75], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 1368], 80.00th=[ 1368], 90.00th=[ 1368], 95.00th=[ 1368], | 99.00th=[ 1368], 99.50th=[ 1368], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 8192, max= 8192, per=43.38%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.12%, sys=7.84%, ctx=60, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29755: Sun Mar 10 18:30:15 2024 write: IOPS=1, BW=7461KiB/s (7640kB/s)(12.0MiB/1647msec) clat (msec): min=110, max=1331, avg=539.50, stdev=686.50 lat (msec): min=110, max=1331, avg=539.77, stdev=686.50 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8192, max= 8192, per=43.38%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=8.99%, ctx=76, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9626KiB/s (9857kB/s), 2407KiB/s-7969KiB/s (2464kB/s-8160kB/s), io=16.0MiB (16.8MB), run=1542-1702msec WRITE: bw=18.4MiB/s (19.3MB/s), 4813KiB/s-7461KiB/s (4929kB/s-7640kB/s), io=32.0MiB (33.6MB), run=1647-1735msec