== sanity test 398b: DIO and buffer IO race ============== 10:55:35 (1724770535) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.10373 s, 45.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8655: Tue Aug 27 10:55:49 2024 read: IOPS=130, BW=522KiB/s (535kB/s)(5992KiB/11469msec) clat (usec): min=48, max=138445, avg=4377.93, stdev=4171.46 lat (usec): min=48, max=138446, avg=4378.53, stdev=4171.48 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1860], 10.00th=[ 2147], 20.00th=[ 2573], | 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 3884], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 7046], 95.00th=[ 8160], | 99.00th=[ 12256], 99.50th=[ 14353], 99.90th=[ 25297], 99.95th=[139461], | 99.99th=[139461] bw ( KiB/s): min= 296, max= 632, per=24.90%, avg=514.09, stdev=73.37, samples=22 iops : min= 74, max= 158, avg=128.50, stdev=18.33, samples=22 write: IOPS=137, BW=549KiB/s (562kB/s)(6296KiB/11469msec) clat (usec): min=467, max=17484, avg=3088.91, stdev=2594.94 lat (usec): min=468, max=17485, avg=3089.62, stdev=2594.93 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 783], 40.00th=[ 1975], 50.00th=[ 2606], 60.00th=[ 3130], | 70.00th=[ 3982], 80.00th=[ 5211], 90.00th=[ 6718], 95.00th=[ 8225], | 99.00th=[10814], 99.50th=[12256], 99.90th=[14222], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 360, max= 760, per=24.92%, avg=538.82, stdev=76.65, samples=22 iops : min= 90, max= 190, avg=134.68, stdev=19.17, samples=22 lat (usec) : 50=0.03%, 100=1.56%, 250=0.13%, 500=0.46%, 750=14.45% lat (usec) : 1000=2.44% lat (msec) : 2=5.05%, 4=37.37%, 10=36.43%, 20=1.95%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.58%, ctx=5865, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8656: Tue Aug 27 10:55:49 2024 read: IOPS=128, BW=514KiB/s (526kB/s)(5940KiB/11564msec) clat (usec): min=49, max=126703, avg=4289.68, stdev=3924.45 lat (usec): min=50, max=126704, avg=4290.77, stdev=3924.51 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1778], 10.00th=[ 2089], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 3195], 50.00th=[ 3818], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5735], 90.00th=[ 6915], 95.00th=[ 8225], | 99.00th=[ 12780], 99.50th=[ 14353], 99.90th=[ 24511], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 224, max= 688, per=24.78%, avg=511.78, stdev=89.77, samples=23 iops : min= 56, max= 172, avg=127.83, stdev=22.47, samples=23 write: IOPS=137, BW=549KiB/s (562kB/s)(6348KiB/11564msec) clat (usec): min=443, max=44223, avg=3233.76, stdev=2962.03 lat (usec): min=443, max=44224, avg=3234.78, stdev=2962.49 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 619], | 30.00th=[ 857], 40.00th=[ 2114], 50.00th=[ 2638], 60.00th=[ 3195], | 70.00th=[ 3949], 80.00th=[ 5276], 90.00th=[ 6915], 95.00th=[ 8586], | 99.00th=[11994], 99.50th=[14877], 99.90th=[19268], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 288, max= 760, per=25.23%, avg=545.39, stdev=98.15, samples=23 iops : min= 72, max= 190, avg=136.26, stdev=24.49, samples=23 lat (usec) : 50=0.03%, 100=1.46%, 250=0.26%, 500=0.36%, 750=14.16% lat (usec) : 1000=1.92% lat (msec) : 2=5.57%, 4=38.35%, 10=35.35%, 20=2.44%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.22%, ctx=6000, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8657: Tue Aug 27 10:55:49 2024 read: IOPS=129, BW=518KiB/s (531kB/s)(6028KiB/11627msec) clat (usec): min=48, max=22578, avg=4186.62, stdev=2332.70 lat (usec): min=49, max=22579, avg=4188.05, stdev=2333.55 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1762], 10.00th=[ 2089], 20.00th=[ 2442], | 30.00th=[ 2868], 40.00th=[ 3294], 50.00th=[ 3851], 60.00th=[ 4359], | 70.00th=[ 4948], 80.00th=[ 5669], 90.00th=[ 6718], 95.00th=[ 8225], | 99.00th=[11863], 99.50th=[15008], 99.90th=[22152], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 200, max= 672, per=24.87%, avg=513.65, stdev=92.57, samples=23 iops : min= 50, max= 168, avg=128.30, stdev=23.08, samples=23 write: IOPS=134, BW=538KiB/s (551kB/s)(6260KiB/11627msec) clat (usec): min=434, max=159439, avg=3374.83, stdev=4893.03 lat (usec): min=435, max=159440, avg=3376.57, stdev=4893.64 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 553], 20.00th=[ 635], | 30.00th=[ 799], 40.00th=[ 2057], 50.00th=[ 2671], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5407], 90.00th=[ 7373], 95.00th=[ 8717], | 99.00th=[ 12387], 99.50th=[ 13173], 99.90th=[ 26608], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 280, max= 696, per=24.65%, avg=532.87, stdev=93.88, samples=23 iops : min= 70, max= 174, avg=133.09, stdev=23.50, samples=23 lat (usec) : 50=0.03%, 100=1.66%, 250=0.29%, 500=0.39%, 750=14.10% lat (usec) : 1000=2.54% lat (msec) : 2=5.18%, 4=36.49%, 10=36.85%, 20=2.28%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.04%, ctx=6117, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8658: Tue Aug 27 10:55:49 2024 read: IOPS=130, BW=521KiB/s (534kB/s)(6052KiB/11613msec) clat (usec): min=54, max=18301, avg=4258.39, stdev=2300.55 lat (usec): min=54, max=18301, avg=4261.03, stdev=2300.91 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1827], 10.00th=[ 2147], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3294], 50.00th=[ 3818], 60.00th=[ 4490], | 70.00th=[ 5080], 80.00th=[ 5735], 90.00th=[ 6915], 95.00th=[ 8356], | 99.00th=[12649], 99.50th=[14222], 99.90th=[16319], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 248, max= 632, per=25.07%, avg=517.61, stdev=88.64, samples=23 iops : min= 62, max= 158, avg=129.30, stdev=22.17, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6236KiB/11613msec) clat (usec): min=400, max=117005, avg=3267.37, stdev=4034.33 lat (usec): min=400, max=117007, avg=3268.24, stdev=4034.26 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 553], 20.00th=[ 627], | 30.00th=[ 832], 40.00th=[ 2057], 50.00th=[ 2638], 60.00th=[ 3195], | 70.00th=[ 4113], 80.00th=[ 5342], 90.00th=[ 6980], 95.00th=[ 8225], | 99.00th=[ 11469], 99.50th=[ 13566], 99.90th=[ 39060], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 304, max= 688, per=24.75%, avg=535.00, stdev=91.00, samples=23 iops : min= 76, max= 172, avg=133.65, stdev=22.74, samples=23 lat (usec) : 100=1.60%, 250=0.20%, 500=0.49%, 750=13.70%, 1000=2.15% lat (msec) : 2=5.24%, 4=37.83%, 10=36.43%, 20=2.31%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.90%, ctx=6150, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2065KiB/s (2115kB/s), 514KiB/s-522KiB/s (526kB/s-535kB/s), io=23.4MiB (24.6MB), run=11469-11627msec WRITE: bw=2162KiB/s (2214kB/s), 537KiB/s-549KiB/s (550kB/s-562kB/s), io=24.6MiB (25.7MB), run=11469-11627msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8650: Tue Aug 27 10:55:53 2024 read: IOPS=98, BW=396KiB/s (405kB/s)(5992KiB/15147msec) clat (usec): min=1481, max=150295, avg=4105.05, stdev=4831.01 lat (usec): min=1481, max=150298, avg=4105.80, stdev=4831.05 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1811], 10.00th=[ 1926], 20.00th=[ 2089], | 30.00th=[ 2278], 40.00th=[ 2474], 50.00th=[ 2737], 60.00th=[ 3130], | 70.00th=[ 3851], 80.00th=[ 5473], 90.00th=[ 8455], 95.00th=[ 10683], | 99.00th=[ 16057], 99.50th=[ 17695], 99.90th=[ 24773], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 160, max= 800, per=25.25%, avg=394.60, stdev=159.48, samples=30 iops : min= 40, max= 200, avg=98.60, stdev=39.88, samples=30 write: IOPS=103, BW=416KiB/s (426kB/s)(6296KiB/15147msec) clat (usec): min=2267, max=23121, avg=5673.37, stdev=3259.98 lat (usec): min=2268, max=23122, avg=5674.24, stdev=3259.94 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2802], 10.00th=[ 2999], 20.00th=[ 3261], | 30.00th=[ 3523], 40.00th=[ 3884], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 6390], 80.00th=[ 7898], 90.00th=[10159], 95.00th=[12387], | 99.00th=[17957], 99.50th=[19530], 99.90th=[22414], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 184, max= 736, per=25.19%, avg=412.43, stdev=153.41, samples=30 iops : min= 46, max= 184, avg=103.07, stdev=38.33, samples=30 lat (msec) : 2=7.45%, 4=49.77%, 10=34.02%, 20=8.46%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.55%, ctx=3689, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8651: Tue Aug 27 10:55:53 2024 read: IOPS=97, BW=392KiB/s (401kB/s)(5940KiB/15154msec) clat (usec): min=1546, max=151913, avg=3963.78, stdev=4665.54 lat (usec): min=1547, max=151915, avg=3964.51, stdev=4665.57 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2704], 60.00th=[ 3064], | 70.00th=[ 3884], 80.00th=[ 5276], 90.00th=[ 8029], 95.00th=[ 9765], | 99.00th=[ 12911], 99.50th=[ 14091], 99.90th=[ 19006], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 112, max= 792, per=24.98%, avg=390.43, stdev=163.10, samples=30 iops : min= 28, max= 198, avg=97.47, stdev=40.79, samples=30 write: IOPS=104, BW=419KiB/s (429kB/s)(6348KiB/15154msec) clat (usec): min=2178, max=42863, avg=5797.18, stdev=3341.50 lat (usec): min=2178, max=42864, avg=5798.05, stdev=3341.44 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2868], 10.00th=[ 3032], 20.00th=[ 3294], | 30.00th=[ 3589], 40.00th=[ 3949], 50.00th=[ 4490], 60.00th=[ 5407], | 70.00th=[ 6718], 80.00th=[ 8094], 90.00th=[10421], 95.00th=[11863], | 99.00th=[17957], 99.50th=[19792], 99.90th=[26084], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 184, max= 784, per=25.40%, avg=415.77, stdev=152.83, samples=30 iops : min= 46, max= 196, avg=103.80, stdev=38.26, samples=30 lat (msec) : 2=6.51%, 4=49.61%, 10=36.04%, 20=7.58%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.36%, ctx=3707, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8652: Tue Aug 27 10:55:53 2024 read: IOPS=99, BW=399KiB/s (408kB/s)(6028KiB/15117msec) clat (usec): min=1579, max=18963, avg=4017.53, stdev=2876.86 lat (usec): min=1579, max=18963, avg=4018.38, stdev=2877.00 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1860], 10.00th=[ 1958], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2802], 60.00th=[ 3163], | 70.00th=[ 3884], 80.00th=[ 5407], 90.00th=[ 8225], 95.00th=[10552], | 99.00th=[14877], 99.50th=[16057], 99.90th=[17433], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 120, max= 848, per=25.41%, avg=397.17, stdev=153.64, samples=30 iops : min= 30, max= 212, avg=99.20, stdev=38.41, samples=30 write: IOPS=103, BW=414KiB/s (424kB/s)(6260KiB/15117msec) clat (msec): min=2, max=156, avg= 5.75, stdev= 4.94 lat (msec): min=2, max=156, avg= 5.75, stdev= 4.94 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 17], 99.50th=[ 19], 99.90th=[ 27], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 168, max= 800, per=25.14%, avg=411.57, stdev=154.70, samples=30 iops : min= 42, max= 200, avg=102.80, stdev=38.70, samples=30 lat (msec) : 2=6.15%, 4=50.39%, 10=35.32%, 20=8.01%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.65%, ctx=3665, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8653: Tue Aug 27 10:55:53 2024 read: IOPS=98, BW=394KiB/s (404kB/s)(6052KiB/15353msec) clat (usec): min=1580, max=22188, avg=4098.98, stdev=3028.54 lat (usec): min=1581, max=22189, avg=4099.69, stdev=3028.49 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2507], 50.00th=[ 2802], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5669], 90.00th=[ 8586], 95.00th=[11207], | 99.00th=[15533], 99.50th=[17433], 99.90th=[19006], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 176, max= 776, per=25.02%, avg=391.07, stdev=156.34, samples=30 iops : min= 44, max= 194, avg=97.70, stdev=39.09, samples=30 write: IOPS=101, BW=406KiB/s (416kB/s)(6236KiB/15353msec) clat (msec): min=2, max=156, avg= 5.83, stdev= 5.12 lat (msec): min=2, max=156, avg= 5.83, stdev= 5.12 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 22], 99.90th=[ 39], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 192, max= 752, per=24.64%, avg=403.33, stdev=158.00, samples=30 iops : min= 48, max= 188, avg=100.77, stdev=39.51, samples=30 lat (msec) : 2=6.58%, 4=49.06%, 10=35.71%, 20=8.27%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.28%, ctx=3774, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1564KiB/s (1602kB/s), 392KiB/s-399KiB/s (401kB/s-408kB/s), io=23.4MiB (24.6MB), run=15117-15353msec WRITE: bw=1637KiB/s (1677kB/s), 406KiB/s-419KiB/s (416kB/s-429kB/s), io=24.6MiB (25.7MB), run=15117-15353msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8682: Tue Aug 27 10:55:58 2024 read: IOPS=77, BW=1234KiB/s (1263kB/s)(5920KiB/4798msec) clat (usec): min=1634, max=23358, avg=4989.28, stdev=3385.76 lat (usec): min=1634, max=23359, avg=4989.89, stdev=3385.78 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2442], | 30.00th=[ 2704], 40.00th=[ 3032], 50.00th=[ 3621], 60.00th=[ 4293], | 70.00th=[ 5866], 80.00th=[ 7767], 90.00th=[ 9503], 95.00th=[11469], | 99.00th=[16319], 99.50th=[20579], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 832, max= 1472, per=24.99%, avg=1240.56, stdev=227.67, samples=9 iops : min= 52, max= 92, avg=77.44, stdev=14.19, samples=9 write: IOPS=82, BW=1327KiB/s (1359kB/s)(6368KiB/4798msec) clat (usec): min=2577, max=37569, avg=7383.15, stdev=5038.77 lat (usec): min=2578, max=37571, avg=7384.18, stdev=5038.78 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3130], 10.00th=[ 3392], 20.00th=[ 3720], | 30.00th=[ 4146], 40.00th=[ 4555], 50.00th=[ 5407], 60.00th=[ 6783], | 70.00th=[ 8586], 80.00th=[10159], 90.00th=[13960], 95.00th=[17171], | 99.00th=[25560], 99.50th=[29492], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 928, max= 1536, per=25.39%, avg=1325.89, stdev=173.92, samples=9 iops : min= 58, max= 96, avg=82.78, stdev=10.88, samples=9 lat (msec) : 2=1.95%, 4=39.19%, 10=43.62%, 20=13.28%, 50=1.95% cpu : usr=0.06%, sys=9.84%, ctx=893, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8683: Tue Aug 27 10:55:58 2024 read: IOPS=76, BW=1217KiB/s (1246kB/s)(5744KiB/4719msec) clat (usec): min=1582, max=21963, avg=4994.06, stdev=3419.69 lat (usec): min=1583, max=21964, avg=4994.81, stdev=3419.62 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2040], 10.00th=[ 2278], 20.00th=[ 2540], | 30.00th=[ 2835], 40.00th=[ 3163], 50.00th=[ 3654], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 7046], 90.00th=[10290], 95.00th=[12649], | 99.00th=[16319], 99.50th=[20841], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 864, max= 1504, per=24.71%, avg=1226.67, stdev=187.96, samples=9 iops : min= 54, max= 94, avg=76.67, stdev=11.75, samples=9 write: IOPS=86, BW=1387KiB/s (1420kB/s)(6544KiB/4719msec) clat (usec): min=2456, max=31948, avg=7073.15, stdev=4302.61 lat (usec): min=2457, max=31950, avg=7074.23, stdev=4302.64 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3163], 10.00th=[ 3425], 20.00th=[ 3720], | 30.00th=[ 4113], 40.00th=[ 4621], 50.00th=[ 5211], 60.00th=[ 6456], | 70.00th=[ 8094], 80.00th=[10421], 90.00th=[13829], 95.00th=[15926], | 99.00th=[19006], 99.50th=[19792], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 1120, max= 1632, per=26.28%, avg=1372.44, stdev=185.52, samples=9 iops : min= 70, max= 102, avg=85.78, stdev=11.60, samples=9 lat (msec) : 2=1.56%, 4=39.84%, 10=42.19%, 20=15.89%, 50=0.52% cpu : usr=0.15%, sys=10.22%, ctx=925, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8684: Tue Aug 27 10:55:58 2024 read: IOPS=81, BW=1301KiB/s (1333kB/s)(6256KiB/4807msec) clat (usec): min=1682, max=33276, avg=4925.03, stdev=3763.98 lat (usec): min=1682, max=33277, avg=4925.70, stdev=3763.97 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1991], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2638], 40.00th=[ 2868], 50.00th=[ 3261], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 7898], 90.00th=[10683], 95.00th=[12125], | 99.00th=[16909], 99.50th=[20579], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 704, max= 1824, per=26.56%, avg=1318.78, stdev=342.11, samples=9 iops : min= 44, max= 114, avg=82.33, stdev=21.42, samples=9 write: IOPS=78, BW=1255KiB/s (1285kB/s)(6032KiB/4807msec) clat (usec): min=2684, max=33079, avg=7529.50, stdev=5065.22 lat (usec): min=2685, max=33080, avg=7530.51, stdev=5065.25 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3130], 10.00th=[ 3359], 20.00th=[ 3720], | 30.00th=[ 4178], 40.00th=[ 4752], 50.00th=[ 5669], 60.00th=[ 6783], | 70.00th=[ 8356], 80.00th=[10552], 90.00th=[14746], 95.00th=[17695], | 99.00th=[25822], 99.50th=[30802], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 928, max= 1437, per=23.68%, avg=1237.00, stdev=154.64, samples=9 iops : min= 58, max= 89, avg=77.22, stdev= 9.54, samples=9 lat (msec) : 2=2.73%, 4=41.80%, 10=38.15%, 20=15.76%, 50=1.56% cpu : usr=0.06%, sys=10.01%, ctx=917, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8685: Tue Aug 27 10:55:58 2024 read: IOPS=78, BW=1250KiB/s (1280kB/s)(6032KiB/4824msec) clat (usec): min=1847, max=16145, avg=5127.39, stdev=3280.65 lat (usec): min=1848, max=16146, avg=5128.04, stdev=3280.68 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 3130], 50.00th=[ 3687], 60.00th=[ 4752], | 70.00th=[ 6128], 80.00th=[ 8094], 90.00th=[10290], 95.00th=[12256], | 99.00th=[14615], 99.50th=[15664], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 928, max= 1405, per=24.76%, avg=1229.56, stdev=176.92, samples=9 iops : min= 58, max= 87, avg=76.67, stdev=10.90, samples=9 write: IOPS=81, BW=1297KiB/s (1328kB/s)(6256KiB/4824msec) clat (usec): min=2668, max=27925, avg=7326.83, stdev=4492.64 lat (usec): min=2669, max=27926, avg=7327.82, stdev=4492.66 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3064], 10.00th=[ 3359], 20.00th=[ 3818], | 30.00th=[ 4228], 40.00th=[ 4817], 50.00th=[ 5800], 60.00th=[ 6718], | 70.00th=[ 8455], 80.00th=[10028], 90.00th=[13698], 95.00th=[17171], | 99.00th=[21627], 99.50th=[26608], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 1088, max= 1437, per=24.77%, avg=1293.56, stdev=100.97, samples=9 iops : min= 68, max= 89, avg=80.67, stdev= 6.22, samples=9 lat (msec) : 2=1.56%, 4=37.76%, 10=44.92%, 20=14.97%, 50=0.78% cpu : usr=0.12%, sys=10.01%, ctx=916, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4965KiB/s (5084kB/s), 1217KiB/s-1301KiB/s (1246kB/s-1333kB/s), io=23.4MiB (24.5MB), run=4719-4824msec WRITE: bw=5224KiB/s (5349kB/s), 1255KiB/s-1387KiB/s (1285kB/s-1420kB/s), io=24.6MiB (25.8MB), run=4719-4824msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8686: Tue Aug 27 10:55:59 2024 read: IOPS=70, BW=1127KiB/s (1154kB/s)(5920KiB/5251msec) clat (usec): min=52, max=21666, avg=10023.01, stdev=4695.07 lat (usec): min=52, max=21668, avg=10024.66, stdev=4695.14 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 65], 10.00th=[ 81], 20.00th=[ 8160], | 30.00th=[ 9110], 40.00th=[ 9896], 50.00th=[10683], 60.00th=[11469], | 70.00th=[12125], 80.00th=[13173], 90.00th=[14484], 95.00th=[16581], | 99.00th=[21365], 99.50th=[21365], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 704, max= 1469, per=22.93%, avg=1039.20, stdev=213.32, samples=10 iops : min= 44, max= 91, avg=64.70, stdev=13.17, samples=10 write: IOPS=75, BW=1213KiB/s (1242kB/s)(6368KiB/5251msec) clat (usec): min=647, max=23336, avg=3794.71, stdev=3930.24 lat (usec): min=648, max=23337, avg=3795.64, stdev=3930.25 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 971], | 30.00th=[ 1450], 40.00th=[ 2278], 50.00th=[ 2573], 60.00th=[ 3097], | 70.00th=[ 3818], 80.00th=[ 5276], 90.00th=[ 8586], 95.00th=[12911], | 99.00th=[20579], 99.50th=[23200], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 734, max= 1820, per=23.73%, avg=1131.90, stdev=322.43, samples=10 iops : min= 45, max= 113, avg=70.50, stdev=20.06, samples=10 lat (usec) : 100=5.21%, 250=1.04%, 500=0.13%, 750=1.95%, 1000=8.72% lat (msec) : 2=7.03%, 4=19.40%, 10=23.57%, 20=31.51%, 50=1.43% cpu : usr=0.13%, sys=21.14%, ctx=2659, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8687: Tue Aug 27 10:55:59 2024 read: IOPS=70, BW=1120KiB/s (1147kB/s)(5744KiB/5128msec) clat (usec): min=47, max=26224, avg=10357.17, stdev=4444.76 lat (usec): min=48, max=26224, avg=10357.75, stdev=4444.78 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 66], 10.00th=[ 6652], 20.00th=[ 8291], | 30.00th=[ 9372], 40.00th=[10159], 50.00th=[10814], 60.00th=[11207], | 70.00th=[11994], 80.00th=[12911], 90.00th=[15139], 95.00th=[16581], | 99.00th=[23462], 99.50th=[24249], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 640, max= 1440, per=24.00%, avg=1087.70, stdev=206.78, samples=10 iops : min= 40, max= 90, avg=67.90, stdev=12.91, samples=10 write: IOPS=79, BW=1276KiB/s (1307kB/s)(6544KiB/5128msec) clat (usec): min=645, max=21409, avg=3281.85, stdev=3198.07 lat (usec): min=646, max=21410, avg=3282.75, stdev=3198.09 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 947], | 30.00th=[ 1205], 40.00th=[ 2008], 50.00th=[ 2507], 60.00th=[ 2966], | 70.00th=[ 3425], 80.00th=[ 4490], 90.00th=[ 6718], 95.00th=[ 9765], | 99.00th=[15795], 99.50th=[19268], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 896, max= 1920, per=25.63%, avg=1222.10, stdev=356.22, samples=10 iops : min= 56, max= 120, avg=76.30, stdev=22.25, samples=10 lat (usec) : 50=0.13%, 100=4.04%, 250=0.39%, 750=1.56%, 1000=11.07% lat (msec) : 2=8.59%, 4=18.62%, 10=23.83%, 20=30.60%, 50=1.17% cpu : usr=0.10%, sys=21.77%, ctx=2494, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8688: Tue Aug 27 10:55:59 2024 read: IOPS=73, BW=1184KiB/s (1212kB/s)(6256KiB/5284msec) clat (usec): min=52, max=20432, avg=9865.78, stdev=4654.21 lat (usec): min=52, max=20432, avg=9867.75, stdev=4654.67 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 75], 10.00th=[ 101], 20.00th=[ 7635], | 30.00th=[ 9241], 40.00th=[ 9896], 50.00th=[10683], 60.00th=[11338], | 70.00th=[12125], 80.00th=[13173], 90.00th=[14877], 95.00th=[15926], | 99.00th=[19268], 99.50th=[20055], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 640, max= 1856, per=24.57%, avg=1113.30, stdev=322.50, samples=10 iops : min= 40, max= 116, avg=69.50, stdev=20.18, samples=10 write: IOPS=71, BW=1142KiB/s (1169kB/s)(6032KiB/5284msec) clat (usec): min=616, max=16555, avg=3585.09, stdev=3153.80 lat (usec): min=616, max=16556, avg=3586.02, stdev=3153.82 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 799], 10.00th=[ 832], 20.00th=[ 1012], | 30.00th=[ 1762], 40.00th=[ 2278], 50.00th=[ 2573], 60.00th=[ 3064], | 70.00th=[ 3785], 80.00th=[ 5276], 90.00th=[ 8029], 95.00th=[10814], | 99.00th=[15401], 99.50th=[16450], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 736, max= 1504, per=22.54%, avg=1074.90, stdev=264.69, samples=10 iops : min= 46, max= 94, avg=67.10, stdev=16.51, samples=10 lat (usec) : 100=4.95%, 250=1.95%, 750=1.04%, 1000=8.33% lat (msec) : 2=6.51%, 4=19.01%, 10=25.78%, 20=32.16%, 50=0.26% cpu : usr=0.11%, sys=21.35%, ctx=2607, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8689: Tue Aug 27 10:55:59 2024 read: IOPS=72, BW=1163KiB/s (1191kB/s)(6032KiB/5188msec) clat (usec): min=51, max=25659, avg=10041.61, stdev=4838.01 lat (usec): min=51, max=25660, avg=10042.19, stdev=4838.03 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 63], 10.00th=[ 82], 20.00th=[ 8291], | 30.00th=[ 9110], 40.00th=[10159], 50.00th=[10814], 60.00th=[11338], | 70.00th=[12125], 80.00th=[12911], 90.00th=[15008], 95.00th=[16712], | 99.00th=[21365], 99.50th=[25035], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 608, max= 1824, per=24.21%, avg=1097.30, stdev=298.35, samples=10 iops : min= 38, max= 114, avg=68.50, stdev=18.66, samples=10 write: IOPS=75, BW=1206KiB/s (1235kB/s)(6256KiB/5188msec) clat (usec): min=626, max=22453, avg=3368.45, stdev=3483.73 lat (usec): min=627, max=23018, avg=3370.79, stdev=3491.81 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 922], | 30.00th=[ 1139], 40.00th=[ 2073], 50.00th=[ 2474], 60.00th=[ 2966], | 70.00th=[ 3589], 80.00th=[ 4621], 90.00th=[ 6652], 95.00th=[ 9503], | 99.00th=[20055], 99.50th=[22414], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 768, max= 1600, per=24.28%, avg=1158.10, stdev=267.46, samples=10 iops : min= 48, max= 100, avg=72.30, stdev=16.67, samples=10 lat (usec) : 100=5.86%, 250=0.65%, 500=0.13%, 750=1.43%, 1000=10.81% lat (msec) : 2=7.68%, 4=18.49%, 10=21.87%, 20=31.38%, 50=1.69% cpu : usr=0.17%, sys=20.88%, ctx=2530, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4533KiB/s (4642kB/s), 1120KiB/s-1184KiB/s (1147kB/s-1212kB/s), io=23.4MiB (24.5MB), run=5128-5284msec WRITE: bw=4769KiB/s (4884kB/s), 1142KiB/s-1276KiB/s (1169kB/s-1307kB/s), io=24.6MiB (25.8MB), run=5128-5284msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8700: Tue Aug 27 10:56:04 2024 read: IOPS=2, BW=2097KiB/s (2147kB/s)(7168KiB/3419msec) clat (usec): min=449, max=934134, avg=279627.03, stdev=432420.19 lat (usec): min=450, max=934135, avg=279628.20, stdev=432420.55 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 603], | 30.00th=[ 3949], 40.00th=[ 3949], 50.00th=[ 63177], 60.00th=[ 67634], | 70.00th=[ 67634], 80.00th=[884999], 90.00th=[935330], 95.00th=[935330], | 99.00th=[935330], 99.50th=[935330], 99.90th=[935330], 99.95th=[935330], | 99.99th=[935330] bw ( KiB/s): min= 2043, max= 4096, per=44.42%, avg=2558.75, stdev=1024.84, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=1498KiB/s (1533kB/s)(5120KiB/3419msec) clat (msec): min=17, max=879, avg=269.81, stdev=376.44 lat (msec): min=17, max=879, avg=269.86, stdev=376.44 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 33], 60.00th=[ 33], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2048, max= 2048, per=25.39%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 4=8.33%, 20=8.33%, 50=16.67%, 100=16.67%, 500=8.33% lat (msec) : 1000=25.00% cpu : usr=0.00%, sys=14.34%, ctx=663, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8702: Tue Aug 27 10:56:04 2024 read: IOPS=0, BW=576KiB/s (590kB/s)(2048KiB/3555msec) clat (msec): min=89, max=803, avg=446.08, stdev=504.82 lat (msec): min=89, max=803, avg=446.08, stdev=504.82 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 802], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 2003, max= 2048, per=35.16%, avg=2025.50, stdev=31.82, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2880KiB/s (2950kB/s)(10.0MiB/3555msec) clat (msec): min=17, max=1084, avg=257.07, stdev=424.34 lat (msec): min=17, max=1084, avg=257.11, stdev=424.35 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 19], | 30.00th=[ 28], 40.00th=[ 33], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 1028], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 2048, max= 6011, per=50.38%, avg=4062.75, stdev=1618.34, samples=4 iops : min= 2, max= 5, avg= 3.75, stdev= 1.26, samples=4 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.03%, sys=28.19%, ctx=224, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8704: Tue Aug 27 10:56:04 2024 read: IOPS=2, BW=2285KiB/s (2340kB/s)(6144KiB/2689msec) clat (usec): min=707, max=1010.0k, avg=224820.72, stdev=388391.81 lat (usec): min=709, max=1010.0k, avg=224822.26, stdev=388391.30 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 709], 10.00th=[ 709], | 20.00th=[ 15008], 30.00th=[ 15008], 40.00th=[ 71828], | 50.00th=[ 71828], 60.00th=[ 111674], 70.00th=[ 139461], | 80.00th=[ 139461], 90.00th=[1010828], 95.00th=[1010828], | 99.00th=[1010828], 99.50th=[1010828], 99.90th=[1010828], | 99.95th=[1010828], 99.99th=[1010828] bw ( KiB/s): min= 4096, max= 4096, per=71.11%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2285KiB/s (2340kB/s)(6144KiB/2689msec) clat (msec): min=38, max=1004, avg=208.57, stdev=389.94 lat (msec): min=38, max=1004, avg=208.67, stdev=389.92 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 58], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 2048, max= 4096, per=42.32%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=16.67% cpu : usr=0.04%, sys=17.93%, ctx=481, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8706: Tue Aug 27 10:56:04 2024 read: IOPS=1, BW=1912KiB/s (1958kB/s)(5120KiB/2678msec) clat (usec): min=1634, max=1012.6k, avg=254549.95, stdev=429917.32 lat (usec): min=1635, max=1012.6k, avg=254551.05, stdev=429917.14 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1631], 10.00th=[ 1631], | 20.00th=[ 1631], 30.00th=[ 9503], 40.00th=[ 9503], | 50.00th=[ 66323], 60.00th=[ 66323], 70.00th=[ 183501], | 80.00th=[ 183501], 90.00th=[1010828], 95.00th=[1010828], | 99.00th=[1010828], 99.50th=[1010828], 99.90th=[1010828], | 99.95th=[1010828], 99.99th=[1010828] bw ( KiB/s): min= 2048, max= 6144, per=71.11%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2677KiB/s (2741kB/s)(7168KiB/2678msec) clat (msec): min=35, max=984, avg=185.30, stdev=352.87 lat (msec): min=35, max=984, avg=185.36, stdev=352.87 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 56], 60.00th=[ 62], | 70.00th=[ 62], 80.00th=[ 77], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 2048, max= 6144, per=59.25%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (msec) : 2=8.33%, 10=8.33%, 50=25.00%, 100=33.33%, 250=8.33% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=18.53%, ctx=428, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5761KiB/s (5899kB/s), 576KiB/s-2285KiB/s (590kB/s-2340kB/s), io=20.0MiB (20.0MB), run=2678-3555msec WRITE: bw=8065KiB/s (8259kB/s), 1498KiB/s-2880KiB/s (1533kB/s-2950kB/s), io=28.0MiB (29.4MB), run=2678-3555msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8699: Tue Aug 27 10:56:04 2024 read: IOPS=2, BW=3012KiB/s (3084kB/s)(7168KiB/2380msec) clat (usec): min=22952, max=74755, avg=45206.38, stdev=21669.01 lat (usec): min=22954, max=74756, avg=45207.60, stdev=21669.35 clat percentiles (usec): | 1.00th=[22938], 5.00th=[22938], 10.00th=[22938], 20.00th=[27395], | 30.00th=[27919], 40.00th=[27919], 50.00th=[43779], 60.00th=[45876], | 70.00th=[45876], 80.00th=[73925], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 2048, max=10240, per=100.00%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 write: IOPS=2, BW=2151KiB/s (2203kB/s)(5120KiB/2380msec) clat (msec): min=31, max=1031, avg=407.96, stdev=502.53 lat (msec): min=31, max=1031, avg=408.03, stdev=502.52 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 63], 60.00th=[ 63], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 2048, max= 6144, per=51.41%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 50=58.33%, 100=25.00%, 1000=8.33% cpu : usr=0.04%, sys=5.13%, ctx=77, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8701: Tue Aug 27 10:56:04 2024 read: IOPS=0, BW=835KiB/s (855kB/s)(2048KiB/2454msec) clat (usec): min=20686, max=65307, avg=42997.01, stdev=31551.29 lat (usec): min=20689, max=65307, avg=42998.80, stdev=31549.89 clat percentiles (usec): | 1.00th=[20579], 5.00th=[20579], 10.00th=[20579], 20.00th=[20579], | 30.00th=[20579], 40.00th=[20579], 50.00th=[20579], 60.00th=[65274], | 70.00th=[65274], 80.00th=[65274], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 2043, max= 2048, per=35.94%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4173KiB/s (4273kB/s)(10.0MiB/2454msec) clat (msec): min=22, max=1011, avg=227.93, stdev=399.52 lat (msec): min=22, max=1011, avg=227.98, stdev=399.51 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 34], 60.00th=[ 37], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 961], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 4096, max=10219, per=89.83%, avg=7157.50, stdev=4329.61, samples=2 iops : min= 4, max= 9, avg= 6.50, stdev= 3.54, samples=2 lat (msec) : 50=58.33%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=6.16%, ctx=92, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8703: Tue Aug 27 10:56:04 2024 read: IOPS=2, BW=2577KiB/s (2639kB/s)(6144KiB/2384msec) clat (usec): min=11986, max=69300, avg=31964.28, stdev=21354.76 lat (usec): min=11987, max=69301, avg=31965.57, stdev=21354.77 clat percentiles (usec): | 1.00th=[11994], 5.00th=[11994], 10.00th=[11994], 20.00th=[14353], | 30.00th=[14353], 40.00th=[23725], 50.00th=[23725], 60.00th=[30540], | 70.00th=[42206], 80.00th=[42206], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 2048, max= 8192, per=89.95%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=2, BW=2577KiB/s (2639kB/s)(6144KiB/2384msec) clat (msec): min=19, max=1097, avg=361.66, stdev=486.88 lat (msec): min=19, max=1097, avg=361.71, stdev=486.86 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 97], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max= 8192, per=64.26%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=3.94%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8705: Tue Aug 27 10:56:04 2024 read: IOPS=1, BW=1423KiB/s (1457kB/s)(5120KiB/3598msec) clat (msec): min=26, max=1018, avg=245.59, stdev=432.39 lat (msec): min=26, max=1018, avg=245.59, stdev=432.39 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 63], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 2048, max= 6131, per=71.85%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1992KiB/s (2040kB/s)(7168KiB/3598msec) clat (msec): min=39, max=959, avg=333.09, stdev=426.32 lat (msec): min=39, max=959, avg=333.15, stdev=426.33 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 63], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 91], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 953], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 2043, max= 6144, per=51.38%, avg=4094.33, stdev=2050.50, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 lat (msec) : 50=16.67%, 100=50.00%, 250=8.33%, 1000=16.67% cpu : usr=0.00%, sys=4.39%, ctx=77, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5692KiB/s (5829kB/s), 835KiB/s-3012KiB/s (855kB/s-3084kB/s), io=20.0MiB (20.0MB), run=2380-3598msec WRITE: bw=7969KiB/s (8160kB/s), 1992KiB/s-4173KiB/s (2040kB/s-4273kB/s), io=28.0MiB (29.4MB), run=2380-3598msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8717: Tue Aug 27 10:56:09 2024 read: IOPS=0, BW=2827KiB/s (2895kB/s)(12.0MiB/4347msec) clat (msec): min=178, max=2614, avg=1420.29, stdev=1218.43 lat (msec): min=178, max=2614, avg=1420.29, stdev=1218.43 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 1469], 50.00th=[ 1469], 60.00th=[ 1469], | 70.00th=[ 2601], 80.00th=[ 2601], 90.00th=[ 2601], 95.00th=[ 2601], | 99.00th=[ 2601], 99.50th=[ 2601], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.05%, sys=28.26%, ctx=1206, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8719: Tue Aug 27 10:56:09 2024 read: IOPS=0, BW=920KiB/s (942kB/s)(4096KiB/4454msec) clat (nsec): min=165634k, max=165634k, avg=165633984.00, stdev= 0.00 lat (nsec): min=165636k, max=165636k, avg=165636412.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1839KiB/s (1883kB/s)(8192KiB/4454msec) clat (msec): min=1013, max=3173, avg=2093.42, stdev=1527.01 lat (msec): min=1013, max=3173, avg=2093.58, stdev=1526.95 clat percentiles (msec): | 1.00th=[ 1011], 5.00th=[ 1011], 10.00th=[ 1011], 20.00th=[ 1011], | 30.00th=[ 1011], 40.00th=[ 1011], 50.00th=[ 1011], 60.00th=[ 3171], | 70.00th=[ 3171], 80.00th=[ 3171], 90.00th=[ 3171], 95.00th=[ 3171], | 99.00th=[ 3171], 99.50th=[ 3171], 99.90th=[ 3171], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=6.76%, ctx=142, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8721: Tue Aug 27 10:56:09 2024 write: IOPS=0, BW=2811KiB/s (2878kB/s)(12.0MiB/4372msec) clat (msec): min=343, max=2722, avg=1452.95, stdev=1197.48 lat (msec): min=343, max=2722, avg=1453.25, stdev=1197.45 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 1301], 50.00th=[ 1301], 60.00th=[ 1301], | 70.00th=[ 2735], 80.00th=[ 2735], 90.00th=[ 2735], 95.00th=[ 2735], | 99.00th=[ 2735], 99.50th=[ 2735], 99.90th=[ 2735], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min= 5468, max= 8192, per=92.85%, avg=6830.00, stdev=1926.16, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.02%, sys=8.21%, ctx=89, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8722: Tue Aug 27 10:56:09 2024 write: IOPS=0, BW=3781KiB/s (3872kB/s)(12.0MiB/3250msec) clat (msec): min=104, max=2675, avg=1053.62, stdev=1411.75 lat (msec): min=104, max=2676, avg=1053.90, stdev=1411.74 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 2668], 80.00th=[ 2668], 90.00th=[ 2668], 95.00th=[ 2668], | 99.00th=[ 2668], 99.50th=[ 2668], 99.90th=[ 2668], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=8.37%, ctx=22, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3678KiB/s (3767kB/s), 920KiB/s-2827KiB/s (942kB/s-2895kB/s), io=16.0MiB (16.8MB), run=4347-4454msec WRITE: bw=7357KiB/s (7534kB/s), 1839KiB/s-3781KiB/s (1883kB/s-3872kB/s), io=32.0MiB (33.6MB), run=3250-4454msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8715: Tue Aug 27 10:56:09 2024 read: IOPS=0, BW=2736KiB/s (2801kB/s)(12.0MiB/4492msec) clat (msec): min=179, max=3083, avg=1469.51, stdev=1478.85 lat (msec): min=179, max=3083, avg=1469.51, stdev=1478.85 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 1150], 50.00th=[ 1150], 60.00th=[ 1150], | 70.00th=[ 3071], 80.00th=[ 3071], 90.00th=[ 3071], 95.00th=[ 3071], | 99.00th=[ 3071], 99.50th=[ 3071], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=2.09%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8716: Tue Aug 27 10:56:09 2024 read: IOPS=0, BW=887KiB/s (908kB/s)(4096KiB/4618msec) clat (nsec): min=122763k, max=122763k, avg=122763435.00, stdev= 0.00 lat (nsec): min=122767k, max=122767k, avg=122766950.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1774KiB/s (1817kB/s)(8192KiB/4618msec) clat (msec): min=1357, max=3107, avg=2232.41, stdev=1237.94 lat (msec): min=1357, max=3108, avg=2232.76, stdev=1238.13 clat percentiles (msec): | 1.00th=[ 1351], 5.00th=[ 1351], 10.00th=[ 1351], 20.00th=[ 1351], | 30.00th=[ 1351], 40.00th=[ 1351], 50.00th=[ 1351], 60.00th=[ 3104], | 70.00th=[ 3104], 80.00th=[ 3104], 90.00th=[ 3104], 95.00th=[ 3104], | 99.00th=[ 3104], 99.50th=[ 3104], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=4.14%, ctx=197, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8718: Tue Aug 27 10:56:09 2024 write: IOPS=0, BW=2741KiB/s (2807kB/s)(12.0MiB/4483msec) clat (msec): min=266, max=2991, avg=1458.02, stdev=1394.21 lat (msec): min=266, max=2991, avg=1458.33, stdev=1394.24 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 1116], 50.00th=[ 1116], 60.00th=[ 1116], | 70.00th=[ 3004], 80.00th=[ 3004], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 5557, max= 8192, per=96.89%, avg=6874.50, stdev=1863.23, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.02%, sys=4.51%, ctx=66, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8720: Tue Aug 27 10:56:09 2024 write: IOPS=1, BW=4096KiB/s (4194kB/s)(12.0MiB/3000msec) clat (msec): min=51, max=2652, avg=994.17, stdev=1440.24 lat (msec): min=52, max=2652, avg=994.55, stdev=1440.17 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 2668], 80.00th=[ 2668], 90.00th=[ 2668], 95.00th=[ 2668], | 99.00th=[ 2668], 99.50th=[ 2668], 99.90th=[ 2668], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.03%, sys=4.87%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3548KiB/s (3633kB/s), 887KiB/s-2736KiB/s (908kB/s-2801kB/s), io=16.0MiB (16.8MB), run=4492-4618msec WRITE: bw=7096KiB/s (7266kB/s), 1774KiB/s-4096KiB/s (1817kB/s-4194kB/s), io=32.0MiB (33.6MB), run=3000-4618msec