== sanity test 398b: DIO and buffer IO race ============== 04:59:04 (1734688744) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.15527 s, 23.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27865: Fri Dec 20 04:59:50 2024 read: IOPS=36, BW=145KiB/s (148kB/s)(5992KiB/41464msec) clat (usec): min=64, max=314619, avg=15259.52, stdev=12963.09 lat (usec): min=65, max=314620, avg=15260.99, stdev=12963.34 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 125], 10.00th=[ 5342], 20.00th=[ 7242], | 30.00th=[ 8979], 40.00th=[ 10945], 50.00th=[ 12649], 60.00th=[ 14877], | 70.00th=[ 17957], 80.00th=[ 21890], 90.00th=[ 28967], 95.00th=[ 35390], | 99.00th=[ 47449], 99.50th=[ 61604], 99.90th=[ 95945], 99.95th=[316670], | 99.99th=[316670] bw ( KiB/s): min= 39, max= 304, per=24.42%, avg=141.39, stdev=50.92, samples=82 iops : min= 9, max= 76, avg=34.94, stdev=12.81, samples=82 write: IOPS=37, BW=152KiB/s (155kB/s)(6296KiB/41464msec) clat (usec): min=442, max=110156, avg=11779.11, stdev=13045.80 lat (usec): min=442, max=110158, avg=11782.19, stdev=13049.04 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 758], 10.00th=[ 824], 20.00th=[ 1020], | 30.00th=[ 1926], 40.00th=[ 4555], 50.00th=[ 8356], 60.00th=[ 11731], | 70.00th=[ 15139], 80.00th=[ 20317], 90.00th=[ 28443], 95.00th=[ 36963], | 99.00th=[ 59507], 99.50th=[ 68682], 99.90th=[102237], 99.95th=[110625], | 99.99th=[110625] bw ( KiB/s): min= 7, max= 334, per=24.50%, avg=148.48, stdev=59.19, samples=82 iops : min= 1, max= 83, avg=36.72, stdev=14.88, samples=82 lat (usec) : 100=1.27%, 250=1.63%, 500=0.16%, 750=2.38%, 1000=7.42% lat (msec) : 2=5.92%, 4=4.33%, 10=22.43%, 20=31.80%, 50=21.35% lat (msec) : 100=1.20%, 250=0.07%, 500=0.03% cpu : usr=0.14%, sys=11.67%, ctx=5352, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27866: Fri Dec 20 04:59:50 2024 read: IOPS=36, BW=145KiB/s (148kB/s)(5940KiB/40967msec) clat (usec): min=54, max=186080, avg=15028.39, stdev=11317.96 lat (usec): min=55, max=186082, avg=15029.96, stdev=11317.84 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 122], 10.00th=[ 5211], 20.00th=[ 7373], | 30.00th=[ 9110], 40.00th=[ 10683], 50.00th=[ 12780], 60.00th=[ 14877], | 70.00th=[ 17695], 80.00th=[ 21365], 90.00th=[ 27919], 95.00th=[ 34341], | 99.00th=[ 52691], 99.50th=[ 61080], 99.90th=[ 90702], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 15, max= 304, per=24.69%, avg=142.94, stdev=50.62, samples=81 iops : min= 3, max= 76, avg=35.33, stdev=12.74, samples=81 write: IOPS=38, BW=155KiB/s (159kB/s)(6348KiB/40967msec) clat (usec): min=477, max=150253, avg=11705.32, stdev=13422.39 lat (usec): min=478, max=150255, avg=11707.33, stdev=13422.41 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 1004], | 30.00th=[ 1811], 40.00th=[ 3949], 50.00th=[ 7963], 60.00th=[ 10945], | 70.00th=[ 14615], 80.00th=[ 20317], 90.00th=[ 28705], 95.00th=[ 36963], | 99.00th=[ 60556], 99.50th=[ 67634], 99.90th=[ 93848], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 39, max= 376, per=24.94%, avg=151.14, stdev=55.22, samples=81 iops : min= 9, max= 94, avg=37.38, stdev=13.86, samples=81 lat (usec) : 100=1.40%, 250=1.63%, 500=0.23%, 750=1.60%, 1000=8.72% lat (msec) : 2=5.99%, 4=4.92%, 10=22.01%, 20=32.10%, 50=19.86% lat (msec) : 100=1.50%, 250=0.07% cpu : usr=0.16%, sys=11.52%, ctx=5068, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27867: Fri Dec 20 04:59:50 2024 read: IOPS=36, BW=147KiB/s (151kB/s)(6028KiB/40945msec) clat (usec): min=67, max=80497, avg=15426.42, stdev=10536.22 lat (usec): min=68, max=80499, avg=15428.86, stdev=10535.89 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 131], 10.00th=[ 5211], 20.00th=[ 7767], | 30.00th=[ 9503], 40.00th=[11338], 50.00th=[13435], 60.00th=[15664], | 70.00th=[18482], 80.00th=[21890], 90.00th=[28705], 95.00th=[34866], | 99.00th=[53216], 99.50th=[61604], 99.90th=[79168], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 32, max= 295, per=25.05%, avg=145.04, stdev=49.24, samples=81 iops : min= 8, max= 73, avg=35.93, stdev=12.35, samples=81 write: IOPS=38, BW=153KiB/s (157kB/s)(6260KiB/40945msec) clat (usec): min=464, max=196474, avg=11257.97, stdev=13635.42 lat (usec): min=465, max=196477, avg=11265.83, stdev=13633.15 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 938], | 30.00th=[ 1532], 40.00th=[ 2999], 50.00th=[ 7504], 60.00th=[ 10421], | 70.00th=[ 14091], 80.00th=[ 19530], 90.00th=[ 28705], 95.00th=[ 35914], | 99.00th=[ 53216], 99.50th=[ 64750], 99.90th=[154141], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 32, max= 288, per=24.80%, avg=150.28, stdev=56.40, samples=81 iops : min= 8, max= 72, avg=37.23, stdev=14.11, samples=81 lat (usec) : 100=1.20%, 250=1.63%, 500=0.26%, 750=2.05%, 1000=9.28% lat (msec) : 2=6.32%, 4=4.36%, 10=20.90%, 20=32.10%, 50=20.54% lat (msec) : 100=1.30%, 250=0.07% cpu : usr=0.19%, sys=11.33%, ctx=5135, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27868: Fri Dec 20 04:59:50 2024 read: IOPS=36, BW=147KiB/s (151kB/s)(6052KiB/41143msec) clat (usec): min=55, max=106884, avg=15482.65, stdev=10724.57 lat (usec): min=56, max=106886, avg=15484.34, stdev=10724.53 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 137], 10.00th=[ 5473], 20.00th=[ 7308], | 30.00th=[ 9110], 40.00th=[ 10814], 50.00th=[ 13042], 60.00th=[ 15664], | 70.00th=[ 19006], 80.00th=[ 22938], 90.00th=[ 28967], 95.00th=[ 35390], | 99.00th=[ 51643], 99.50th=[ 57934], 99.90th=[ 78119], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 31, max= 296, per=25.20%, avg=145.90, stdev=51.48, samples=82 iops : min= 7, max= 74, avg=36.22, stdev=12.89, samples=82 write: IOPS=37, BW=152KiB/s (155kB/s)(6236KiB/41143msec) clat (usec): min=493, max=193487, avg=11316.89, stdev=13309.61 lat (usec): min=493, max=193491, avg=11319.31, stdev=13309.22 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 750], 10.00th=[ 807], 20.00th=[ 938], | 30.00th=[ 1467], 40.00th=[ 3130], 50.00th=[ 7308], 60.00th=[ 11076], | 70.00th=[ 14877], 80.00th=[ 20317], 90.00th=[ 28181], 95.00th=[ 34866], | 99.00th=[ 55313], 99.50th=[ 62129], 99.90th=[135267], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 31, max= 376, per=24.84%, avg=150.51, stdev=62.31, samples=82 iops : min= 7, max= 94, avg=37.37, stdev=15.65, samples=82 lat (usec) : 100=1.33%, 250=1.46%, 500=0.20%, 750=2.54%, 1000=8.72% lat (msec) : 2=6.15%, 4=4.23%, 10=21.65%, 20=29.95%, 50=22.43% lat (msec) : 100=1.24%, 250=0.10% cpu : usr=0.17%, sys=11.47%, ctx=5202, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=579KiB/s (593kB/s), 145KiB/s-147KiB/s (148kB/s-151kB/s), io=23.4MiB (24.6MB), run=40945-41464msec WRITE: bw=606KiB/s (621kB/s), 152KiB/s-155KiB/s (155kB/s-159kB/s), io=24.6MiB (25.7MB), run=40945-41464msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27841: Fri Dec 20 05:00:07 2024 read: IOPS=25, BW=102KiB/s (105kB/s)(5992KiB/58661msec) clat (msec): min=3, max=406, avg=14.90, stdev=16.98 lat (msec): min=3, max=406, avg=14.90, stdev=16.98 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 20], 90.00th=[ 33], 95.00th=[ 44], | 99.00th=[ 68], 99.50th=[ 88], 99.90th=[ 110], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 31, max= 247, per=24.96%, avg=102.09, stdev=57.48, samples=116 iops : min= 7, max= 61, avg=25.14, stdev=14.43, samples=116 write: IOPS=26, BW=107KiB/s (110kB/s)(6296KiB/58661msec) clat (msec): min=5, max=169, avg=22.95, stdev=16.06 lat (msec): min=5, max=169, avg=22.95, stdev=16.07 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 24], 80.00th=[ 33], 90.00th=[ 46], 95.00th=[ 57], | 99.00th=[ 80], 99.50th=[ 90], 99.90th=[ 122], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 24, max= 271, per=25.08%, avg=107.34, stdev=58.64, samples=116 iops : min= 6, max= 67, avg=26.46, stdev=14.70, samples=116 lat (msec) : 4=0.81%, 10=29.95%, 20=39.19%, 50=24.58%, 100=5.14% lat (msec) : 250=0.29%, 500=0.03% cpu : usr=0.18%, sys=5.51%, ctx=3468, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27861: Fri Dec 20 05:00:07 2024 read: IOPS=25, BW=102KiB/s (105kB/s)(5940KiB/58121msec) clat (msec): min=2, max=280, avg=14.15, stdev=14.97 lat (msec): min=2, max=280, avg=14.16, stdev=14.97 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 19], 90.00th=[ 31], 95.00th=[ 42], | 99.00th=[ 69], 99.50th=[ 90], 99.90th=[ 108], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 8, max= 304, per=24.88%, avg=101.78, stdev=61.84, samples=116 iops : min= 2, max= 76, avg=25.13, stdev=15.53, samples=116 write: IOPS=27, BW=109KiB/s (112kB/s)(6348KiB/58121msec) clat (msec): min=4, max=148, avg=23.24, stdev=15.72 lat (msec): min=4, max=148, avg=23.24, stdev=15.72 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 44], 95.00th=[ 56], | 99.00th=[ 78], 99.50th=[ 89], 99.90th=[ 128], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 31, max= 264, per=25.56%, avg=109.41, stdev=58.05, samples=115 iops : min= 7, max= 66, avg=27.03, stdev=14.57, samples=115 lat (msec) : 4=0.98%, 10=30.47%, 20=38.77%, 50=24.61%, 100=4.92% lat (msec) : 250=0.23%, 500=0.03% cpu : usr=0.13%, sys=5.55%, ctx=3430, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27862: Fri Dec 20 05:00:07 2024 read: IOPS=25, BW=103KiB/s (106kB/s)(6028KiB/58422msec) clat (usec): min=2683, max=96877, avg=14388.77, stdev=12851.77 lat (usec): min=2684, max=96879, avg=14390.51, stdev=12851.76 clat percentiles (usec): | 1.00th=[ 3785], 5.00th=[ 4621], 10.00th=[ 5211], 20.00th=[ 6325], | 30.00th=[ 7242], 40.00th=[ 8291], 50.00th=[ 9503], 60.00th=[11338], | 70.00th=[13960], 80.00th=[19792], 90.00th=[31065], 95.00th=[42206], | 99.00th=[65274], 99.50th=[76022], 99.90th=[95945], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 24, max= 296, per=25.31%, avg=103.51, stdev=58.04, samples=115 iops : min= 6, max= 74, avg=25.58, stdev=14.53, samples=115 write: IOPS=26, BW=107KiB/s (110kB/s)(6260KiB/58422msec) clat (msec): min=5, max=291, avg=23.33, stdev=18.20 lat (msec): min=5, max=291, avg=23.34, stdev=18.20 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 26], 80.00th=[ 33], 90.00th=[ 44], 95.00th=[ 58], | 99.00th=[ 91], 99.50th=[ 115], 99.90th=[ 140], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 16, max= 247, per=24.82%, avg=106.22, stdev=60.56, samples=116 iops : min= 4, max= 61, avg=26.26, stdev=15.17, samples=116 lat (msec) : 4=0.98%, 10=29.88%, 20=39.75%, 50=24.32%, 100=4.75% lat (msec) : 250=0.29%, 500=0.03% cpu : usr=0.08%, sys=5.58%, ctx=3410, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27863: Fri Dec 20 05:00:07 2024 read: IOPS=26, BW=104KiB/s (107kB/s)(6052KiB/58105msec) clat (usec): min=3295, max=96090, avg=14074.43, stdev=12553.58 lat (usec): min=3296, max=96091, avg=14077.14, stdev=12554.07 clat percentiles (usec): | 1.00th=[ 3851], 5.00th=[ 4555], 10.00th=[ 5211], 20.00th=[ 6128], | 30.00th=[ 6980], 40.00th=[ 7963], 50.00th=[ 9110], 60.00th=[10945], | 70.00th=[13698], 80.00th=[19268], 90.00th=[31589], 95.00th=[41157], | 99.00th=[61604], 99.50th=[71828], 99.90th=[92799], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 24, max= 255, per=25.58%, avg=104.61, stdev=58.95, samples=115 iops : min= 6, max= 63, avg=25.83, stdev=14.77, samples=115 write: IOPS=26, BW=107KiB/s (110kB/s)(6236KiB/58105msec) clat (msec): min=5, max=295, avg=23.47, stdev=17.96 lat (msec): min=5, max=295, avg=23.47, stdev=17.96 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 26], 80.00th=[ 33], 90.00th=[ 45], 95.00th=[ 57], | 99.00th=[ 87], 99.50th=[ 109], 99.90th=[ 169], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8, max= 256, per=24.94%, avg=106.74, stdev=57.82, samples=116 iops : min= 2, max= 64, avg=26.37, stdev=14.49, samples=116 lat (msec) : 4=0.85%, 10=30.37%, 20=38.67%, 50=25.26%, 100=4.52% lat (msec) : 250=0.29%, 500=0.03% cpu : usr=0.12%, sys=5.69%, ctx=3466, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=409KiB/s (419kB/s), 102KiB/s-104KiB/s (105kB/s-107kB/s), io=23.4MiB (24.6MB), run=58105-58661msec WRITE: bw=429KiB/s (439kB/s), 107KiB/s-109KiB/s (110kB/s-112kB/s), io=24.6MiB (25.7MB), run=58105-58661msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27931: Fri Dec 20 05:00:29 2024 read: IOPS=18, BW=300KiB/s (307kB/s)(5920KiB/19718msec) clat (msec): min=3, max=123, avg=19.40, stdev=16.19 lat (msec): min=3, max=123, avg=19.40, stdev=16.19 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 28], 90.00th=[ 41], 95.00th=[ 54], | 99.00th=[ 81], 99.50th=[ 84], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 128, max= 606, per=25.48%, avg=298.15, stdev=99.57, samples=39 iops : min= 8, max= 37, avg=18.23, stdev= 6.21, samples=39 write: IOPS=20, BW=323KiB/s (331kB/s)(6368KiB/19718msec) clat (msec): min=8, max=166, avg=31.27, stdev=20.34 lat (msec): min=8, max=166, avg=31.28, stdev=20.34 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 29], | 70.00th=[ 35], 80.00th=[ 43], 90.00th=[ 57], 95.00th=[ 67], | 99.00th=[ 122], 99.50th=[ 129], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 127, max= 544, per=26.19%, avg=322.44, stdev=92.63, samples=39 iops : min= 7, max= 34, avg=19.74, stdev= 5.80, samples=39 lat (msec) : 4=0.13%, 10=17.19%, 20=32.03%, 50=40.23%, 100=9.64% lat (msec) : 250=0.78% cpu : usr=0.13%, sys=4.62%, ctx=885, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27932: Fri Dec 20 05:00:29 2024 read: IOPS=18, BW=292KiB/s (299kB/s)(5744KiB/19666msec) clat (msec): min=3, max=105, avg=19.59, stdev=15.56 lat (msec): min=3, max=105, avg=19.59, stdev=15.56 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 18], | 70.00th=[ 23], 80.00th=[ 30], 90.00th=[ 40], 95.00th=[ 49], | 99.00th=[ 83], 99.50th=[ 93], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 127, max= 576, per=24.87%, avg=291.00, stdev=110.91, samples=39 iops : min= 7, max= 36, avg=17.82, stdev= 7.01, samples=39 write: IOPS=20, BW=333KiB/s (341kB/s)(6544KiB/19666msec) clat (msec): min=9, max=118, avg=30.69, stdev=18.85 lat (msec): min=9, max=118, avg=30.69, stdev=18.85 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 29], | 70.00th=[ 33], 80.00th=[ 42], 90.00th=[ 55], 95.00th=[ 71], | 99.00th=[ 102], 99.50th=[ 104], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 128, max= 576, per=26.99%, avg=332.21, stdev=88.52, samples=39 iops : min= 8, max= 36, avg=20.38, stdev= 5.67, samples=39 lat (msec) : 4=0.13%, 10=14.71%, 20=32.94%, 50=43.75%, 100=7.55% lat (msec) : 250=0.91% cpu : usr=0.04%, sys=4.63%, ctx=923, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27933: Fri Dec 20 05:00:29 2024 read: IOPS=19, BW=306KiB/s (313kB/s)(6256KiB/20455msec) clat (usec): min=4637, max=97728, avg=18491.94, stdev=14121.46 lat (usec): min=4638, max=97730, avg=18493.59, stdev=14121.48 clat percentiles (usec): | 1.00th=[ 5276], 5.00th=[ 5997], 10.00th=[ 6783], 20.00th=[ 8291], | 30.00th=[ 9634], 40.00th=[11076], 50.00th=[12649], 60.00th=[16057], | 70.00th=[20841], 80.00th=[27395], 90.00th=[37487], 95.00th=[47449], | 99.00th=[65799], 99.50th=[90702], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 159, max= 510, per=26.06%, avg=304.90, stdev=99.58, samples=40 iops : min= 9, max= 31, avg=18.60, stdev= 6.07, samples=40 write: IOPS=18, BW=295KiB/s (302kB/s)(6032KiB/20455msec) clat (msec): min=8, max=123, avg=34.93, stdev=20.60 lat (msec): min=8, max=123, avg=34.93, stdev=20.60 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 26], 50.00th=[ 31], 60.00th=[ 34], | 70.00th=[ 42], 80.00th=[ 49], 90.00th=[ 64], 95.00th=[ 77], | 99.00th=[ 107], 99.50th=[ 124], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 96, max= 448, per=23.48%, avg=289.02, stdev=93.60, samples=40 iops : min= 6, max= 28, avg=17.63, stdev= 5.88, samples=40 lat (msec) : 10=17.58%, 20=28.39%, 50=43.49%, 100=9.77%, 250=0.78% cpu : usr=0.01%, sys=5.12%, ctx=1082, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27934: Fri Dec 20 05:00:29 2024 read: IOPS=19, BW=319KiB/s (327kB/s)(6032KiB/18881msec) clat (usec): min=4196, max=96896, avg=19360.35, stdev=15520.39 lat (usec): min=4198, max=96898, avg=19362.27, stdev=15520.10 clat percentiles (usec): | 1.00th=[ 4752], 5.00th=[ 6390], 10.00th=[ 6915], 20.00th=[ 8029], | 30.00th=[ 9503], 40.00th=[10945], 50.00th=[13698], 60.00th=[17171], | 70.00th=[21890], 80.00th=[29754], 90.00th=[39060], 95.00th=[45876], | 99.00th=[90702], 99.50th=[92799], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 31, max= 576, per=27.24%, avg=318.68, stdev=108.73, samples=37 iops : min= 1, max= 36, avg=19.57, stdev= 6.92, samples=37 write: IOPS=20, BW=331KiB/s (339kB/s)(6256KiB/18881msec) clat (msec): min=7, max=123, avg=29.52, stdev=17.23 lat (msec): min=7, max=123, avg=29.52, stdev=17.23 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 25], 60.00th=[ 28], | 70.00th=[ 34], 80.00th=[ 42], 90.00th=[ 52], 95.00th=[ 65], | 99.00th=[ 89], 99.50th=[ 110], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 159, max= 447, per=26.80%, avg=329.89, stdev=72.56, samples=37 iops : min= 9, max= 27, avg=20.27, stdev= 4.48, samples=37 lat (msec) : 10=16.80%, 20=33.46%, 50=41.93%, 100=7.42%, 250=0.39% cpu : usr=0.19%, sys=4.64%, ctx=912, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1171KiB/s (1199kB/s), 292KiB/s-319KiB/s (299kB/s-327kB/s), io=23.4MiB (24.5MB), run=18881-20455msec WRITE: bw=1232KiB/s (1262kB/s), 295KiB/s-333KiB/s (302kB/s-341kB/s), io=24.6MiB (25.8MB), run=18881-20455msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27936: Fri Dec 20 05:00:29 2024 read: IOPS=18, BW=292KiB/s (299kB/s)(5920KiB/20255msec) clat (usec): min=83, max=96584, avg=39748.78, stdev=14953.26 lat (usec): min=84, max=96586, avg=39750.33, stdev=14953.30 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[19530], 10.00th=[25035], 20.00th=[30540], | 30.00th=[33817], 40.00th=[35914], 50.00th=[38011], 60.00th=[40633], | 70.00th=[44827], 80.00th=[50594], 90.00th=[57934], 95.00th=[66847], | 99.00th=[82314], 99.50th=[91751], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 189, max= 446, per=24.21%, avg=284.28, stdev=58.33, samples=40 iops : min= 11, max= 27, avg=17.52, stdev= 3.66, samples=40 write: IOPS=19, BW=314KiB/s (322kB/s)(6368KiB/20255msec) clat (usec): min=812, max=108222, avg=13673.73, stdev=14671.84 lat (usec): min=813, max=108225, avg=13676.50, stdev=14671.89 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1172], 10.00th=[ 1336], 20.00th=[ 2278], | 30.00th=[ 5473], 40.00th=[ 7701], 50.00th=[ 9503], 60.00th=[ 11469], | 70.00th=[ 14615], 80.00th=[ 19006], 90.00th=[ 32637], 95.00th=[ 47449], | 99.00th=[ 67634], 99.50th=[ 73925], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 159, max= 576, per=25.03%, avg=309.10, stdev=101.11, samples=40 iops : min= 9, max= 36, avg=19.07, stdev= 6.43, samples=40 lat (usec) : 100=0.39%, 250=1.56%, 1000=0.78% lat (msec) : 2=8.33%, 4=4.17%, 10=14.32%, 20=14.71%, 50=43.62% lat (msec) : 100=11.98%, 250=0.13% cpu : usr=0.15%, sys=10.50%, ctx=2566, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27937: Fri Dec 20 05:00:29 2024 read: IOPS=17, BW=284KiB/s (291kB/s)(5744KiB/20200msec) clat (usec): min=78, max=90817, avg=40027.63, stdev=15390.63 lat (usec): min=79, max=90819, avg=40029.14, stdev=15390.67 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[17957], 10.00th=[25035], 20.00th=[30540], | 30.00th=[33817], 40.00th=[36439], 50.00th=[39584], 60.00th=[42206], | 70.00th=[45351], 80.00th=[50070], 90.00th=[60031], 95.00th=[67634], | 99.00th=[79168], 99.50th=[85459], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 64, max= 480, per=23.80%, avg=279.43, stdev=78.62, samples=40 iops : min= 4, max= 30, avg=17.05, stdev= 4.90, samples=40 write: IOPS=20, BW=324KiB/s (332kB/s)(6544KiB/20200msec) clat (usec): min=880, max=117274, avg=13967.05, stdev=16126.26 lat (usec): min=883, max=117276, avg=13970.82, stdev=16126.13 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1221], 10.00th=[ 1369], 20.00th=[ 2737], | 30.00th=[ 5800], 40.00th=[ 7635], 50.00th=[ 9372], 60.00th=[ 11207], | 70.00th=[ 14222], 80.00th=[ 19792], 90.00th=[ 33162], 95.00th=[ 44827], | 99.00th=[ 85459], 99.50th=[ 93848], 99.90th=[116917], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 63, max= 605, per=25.67%, avg=317.00, stdev=133.64, samples=40 iops : min= 3, max= 37, avg=19.40, stdev= 8.43, samples=40 lat (usec) : 100=0.52%, 250=1.56%, 500=0.13%, 1000=0.13% lat (msec) : 2=8.20%, 4=4.82%, 10=15.23%, 20=14.71%, 50=43.23% lat (msec) : 100=11.20%, 250=0.26% cpu : usr=0.13%, sys=10.32%, ctx=2458, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27938: Fri Dec 20 05:00:29 2024 read: IOPS=19, BW=313KiB/s (320kB/s)(6256KiB/20004msec) clat (usec): min=83, max=101345, avg=39237.43, stdev=15741.66 lat (usec): min=84, max=101347, avg=39280.50, stdev=15773.56 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 506], 10.00th=[ 23987], 20.00th=[ 28443], | 30.00th=[ 32113], 40.00th=[ 35914], 50.00th=[ 38536], 60.00th=[ 41681], | 70.00th=[ 45351], 80.00th=[ 50594], 90.00th=[ 58983], 95.00th=[ 65274], | 99.00th=[ 77071], 99.50th=[ 93848], 99.90th=[101188], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 191, max= 448, per=26.59%, avg=312.12, stdev=62.36, samples=40 iops : min= 11, max= 28, avg=19.10, stdev= 4.06, samples=40 write: IOPS=18, BW=302KiB/s (309kB/s)(6032KiB/20004msec) clat (usec): min=945, max=135919, avg=12223.10, stdev=13863.51 lat (usec): min=947, max=135922, avg=12225.98, stdev=13863.43 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1188], 10.00th=[ 1336], 20.00th=[ 1958], | 30.00th=[ 3785], 40.00th=[ 6456], 50.00th=[ 8717], 60.00th=[ 10814], | 70.00th=[ 13829], 80.00th=[ 18744], 90.00th=[ 25560], 95.00th=[ 42206], | 99.00th=[ 63701], 99.50th=[ 65274], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 95, max= 608, per=24.24%, avg=299.35, stdev=132.12, samples=40 iops : min= 5, max= 38, avg=18.30, stdev= 8.34, samples=40 lat (usec) : 100=1.04%, 250=1.43%, 750=0.13%, 1000=0.26% lat (msec) : 2=10.03%, 4=4.82%, 10=12.89%, 20=13.28%, 50=43.10% lat (msec) : 100=12.76%, 250=0.26% cpu : usr=0.13%, sys=10.48%, ctx=2581, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27939: Fri Dec 20 05:00:29 2024 read: IOPS=18, BW=296KiB/s (303kB/s)(6032KiB/20398msec) clat (usec): min=79, max=124354, avg=40108.23, stdev=16047.47 lat (usec): min=80, max=124356, avg=40109.74, stdev=16047.50 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 18744], 10.00th=[ 23725], 20.00th=[ 30016], | 30.00th=[ 33162], 40.00th=[ 35914], 50.00th=[ 39060], 60.00th=[ 41157], | 70.00th=[ 45876], 80.00th=[ 50070], 90.00th=[ 57934], 95.00th=[ 66847], | 99.00th=[ 87557], 99.50th=[111674], 99.90th=[124257], 99.95th=[124257], | 99.99th=[124257] bw ( KiB/s): min= 128, max= 416, per=24.35%, avg=285.85, stdev=69.14, samples=40 iops : min= 8, max= 26, avg=17.47, stdev= 4.37, samples=40 write: IOPS=19, BW=307KiB/s (314kB/s)(6256KiB/20398msec) clat (usec): min=805, max=115197, avg=13189.61, stdev=15886.72 lat (usec): min=807, max=115199, avg=13192.41, stdev=15886.94 clat percentiles (usec): | 1.00th=[ 1012], 5.00th=[ 1188], 10.00th=[ 1352], 20.00th=[ 2376], | 30.00th=[ 4490], 40.00th=[ 6718], 50.00th=[ 8291], 60.00th=[ 10421], | 70.00th=[ 13698], 80.00th=[ 19268], 90.00th=[ 28967], 95.00th=[ 46924], | 99.00th=[ 85459], 99.50th=[ 99091], 99.90th=[114820], 99.95th=[114820], | 99.99th=[114820] bw ( KiB/s): min= 127, max= 608, per=24.26%, avg=299.55, stdev=111.75, samples=40 iops : min= 7, max= 38, avg=18.32, stdev= 7.04, samples=40 lat (usec) : 100=0.52%, 250=0.91%, 500=0.13%, 1000=0.26% lat (msec) : 2=9.11%, 4=4.95%, 10=15.62%, 20=12.63%, 50=43.88% lat (msec) : 100=11.59%, 250=0.39% cpu : usr=0.09%, sys=10.14%, ctx=2501, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1174KiB/s (1202kB/s), 284KiB/s-313KiB/s (291kB/s-320kB/s), io=23.4MiB (24.5MB), run=20004-20398msec WRITE: bw=1235KiB/s (1265kB/s), 302KiB/s-324KiB/s (309kB/s-332kB/s), io=24.6MiB (25.8MB), run=20004-20398msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27963: Fri Dec 20 05:00:36 2024 read: IOPS=1, BW=1452KiB/s (1487kB/s)(7168KiB/4935msec) clat (usec): min=1180, max=1618.7k, avg=632708.99, stdev=693745.56 lat (usec): min=1182, max=1618.7k, avg=632710.67, stdev=693745.17 clat percentiles (usec): | 1.00th=[ 1188], 5.00th=[ 1188], 10.00th=[ 1188], | 20.00th=[ 11731], 30.00th=[ 166724], 40.00th=[ 166724], | 50.00th=[ 304088], 60.00th=[ 817890], 70.00th=[ 817890], | 80.00th=[1501561], 90.00th=[1619002], 95.00th=[1619002], | 99.00th=[1619002], 99.50th=[1619002], 99.90th=[1619002], | 99.95th=[1619002], 99.99th=[1619002] bw ( KiB/s): min= 1563, max= 4096, per=62.39%, avg=2588.75, stdev=1247.14, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 write: IOPS=1, BW=1037KiB/s (1062kB/s)(5120KiB/4935msec) clat (msec): min=39, max=126, avg=85.57, stdev=37.36 lat (msec): min=40, max=126, avg=85.84, stdev=37.24 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 72], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 1565, max= 4690, per=44.87%, avg=2606.67, stdev=1804.22, samples=3 iops : min= 1, max= 4, avg= 2.00, stdev= 1.73, samples=3 lat (msec) : 2=8.33%, 20=8.33%, 50=8.33%, 100=16.67%, 250=25.00% lat (msec) : 500=8.33%, 1000=8.33% cpu : usr=0.06%, sys=15.28%, ctx=757, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27965: Fri Dec 20 05:00:36 2024 read: IOPS=1, BW=1447KiB/s (1482kB/s)(2048KiB/1415msec) clat (msec): min=195, max=327, avg=261.21, stdev=93.45 lat (msec): min=195, max=327, avg=261.22, stdev=93.45 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 2019, max= 2019, per=48.66%, avg=2019.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7237KiB/s (7410kB/s)(10.0MiB/1415msec) clat (msec): min=40, max=165, avg=81.58, stdev=42.33 lat (msec): min=40, max=166, avg=81.72, stdev=42.36 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 52], 40.00th=[ 56], 50.00th=[ 64], 60.00th=[ 83], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 144], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 4039, max= 6144, per=87.65%, avg=5091.50, stdev=1488.46, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=50.00%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=22.21%, ctx=121, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27967: Fri Dec 20 05:00:36 2024 read: IOPS=5, BW=5570KiB/s (5704kB/s)(6144KiB/1103msec) clat (usec): min=776, max=145205, avg=50034.04, stdev=66761.77 lat (usec): min=777, max=145207, avg=50036.10, stdev=66762.21 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 775], 10.00th=[ 775], 20.00th=[ 914], | 30.00th=[ 914], 40.00th=[ 1516], 50.00th=[ 1516], 60.00th=[ 27395], | 70.00th=[124257], 80.00th=[124257], 90.00th=[145753], 95.00th=[145753], | 99.00th=[145753], 99.50th=[145753], 99.90th=[145753], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min=10219, max=10219, per=100.00%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5570KiB/s (5704kB/s)(6144KiB/1103msec) clat (msec): min=51, max=276, avg=122.19, stdev=79.35 lat (msec): min=51, max=276, avg=122.40, stdev=79.28 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 90], 50.00th=[ 90], 60.00th=[ 110], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 3961, max= 8175, per=100.00%, avg=6068.00, stdev=2979.75, samples=2 iops : min= 3, max= 7, avg= 5.00, stdev= 2.83, samples=2 lat (usec) : 1000=16.67% lat (msec) : 2=8.33%, 50=8.33%, 100=25.00%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=17.15%, ctx=97, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27968: Fri Dec 20 05:00:36 2024 read: IOPS=4, BW=4728KiB/s (4841kB/s)(5120KiB/1083msec) clat (usec): min=478, max=240295, avg=87746.30, stdev=120358.10 lat (usec): min=479, max=240295, avg=87747.60, stdev=120357.66 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 478], 10.00th=[ 478], 20.00th=[ 478], | 30.00th=[ 594], 40.00th=[ 594], 50.00th=[ 644], 60.00th=[ 644], | 70.00th=[196084], 80.00th=[196084], 90.00th=[240124], 95.00th=[240124], | 99.00th=[240124], 99.50th=[240124], 99.90th=[240124], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 2039, max= 6131, per=98.46%, avg=4085.00, stdev=2893.48, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=6, BW=6619KiB/s (6777kB/s)(7168KiB/1083msec) clat (msec): min=33, max=119, avg=72.31, stdev=30.73 lat (msec): min=33, max=119, avg=72.43, stdev=30.72 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 39], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 70], 60.00th=[ 88], | 70.00th=[ 88], 80.00th=[ 93], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 6131, max= 8159, per=100.00%, avg=7145.00, stdev=1434.01, samples=2 iops : min= 5, max= 7, avg= 6.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=16.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=16.64%, ctx=98, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4150KiB/s (4250kB/s), 1447KiB/s-5570KiB/s (1482kB/s-5704kB/s), io=20.0MiB (20.0MB), run=1083-4935msec WRITE: bw=5810KiB/s (5949kB/s), 1037KiB/s-7237KiB/s (1062kB/s-7410kB/s), io=28.0MiB (29.4MB), run=1083-4935msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27961: Fri Dec 20 05:00:36 2024 read: IOPS=1, BW=1382KiB/s (1416kB/s)(7168KiB/5185msec) clat (msec): min=49, max=1424, avg=369.73, stdev=492.30 lat (msec): min=49, max=1424, avg=369.73, stdev=492.30 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 51], | 30.00th=[ 122], 40.00th=[ 122], 50.00th=[ 124], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 485], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 1501, max= 4096, per=59.07%, avg=2332.60, stdev=1011.58, samples=5 iops : min= 1, max= 4, avg= 1.60, stdev= 1.34, samples=5 write: IOPS=0, BW=987KiB/s (1011kB/s)(5120KiB/5185msec) clat (msec): min=58, max=1595, avg=476.39, stdev=639.84 lat (msec): min=58, max=1595, avg=476.54, stdev=639.83 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 1503, max= 3961, per=42.92%, avg=2373.00, stdev=1086.08, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 lat (msec) : 50=8.33%, 100=16.67%, 250=33.33%, 500=25.00% cpu : usr=0.00%, sys=3.22%, ctx=93, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27962: Fri Dec 20 05:00:36 2024 read: IOPS=1, BW=1235KiB/s (1265kB/s)(2048KiB/1658msec) clat (msec): min=72, max=138, avg=105.51, stdev=46.16 lat (msec): min=72, max=138, avg=105.51, stdev=46.16 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 73], | 30.00th=[ 73], 40.00th=[ 73], 50.00th=[ 73], 60.00th=[ 138], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 2039, max= 2048, per=51.75%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6176KiB/s (6324kB/s)(10.0MiB/1658msec) clat (msec): min=37, max=286, avg=124.85, stdev=67.33 lat (msec): min=37, max=286, avg=125.02, stdev=67.34 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 58], | 30.00th=[ 87], 40.00th=[ 116], 50.00th=[ 123], 60.00th=[ 125], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 148], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 2039, max= 8192, per=86.37%, avg=4775.67, stdev=3132.30, samples=3 iops : min= 1, max= 8, avg= 4.33, stdev= 3.51, samples=3 lat (msec) : 50=8.33%, 100=25.00%, 250=58.33%, 500=8.33% cpu : usr=0.06%, sys=10.92%, ctx=86, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27964: Fri Dec 20 05:00:36 2024 read: IOPS=3, BW=4003KiB/s (4099kB/s)(6144KiB/1535msec) clat (msec): min=53, max=268, avg=125.34, stdev=82.07 lat (msec): min=53, max=268, avg=125.34, stdev=82.07 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 92], 50.00th=[ 92], 60.00th=[ 117], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 4087, max= 6107, per=100.00%, avg=5097.00, stdev=1428.36, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=3, BW=4003KiB/s (4099kB/s)(6144KiB/1535msec) clat (msec): min=62, max=210, avg=120.02, stdev=68.37 lat (msec): min=64, max=210, avg=120.42, stdev=68.08 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 88], 50.00th=[ 88], 60.00th=[ 89], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 6107, max= 6144, per=100.00%, avg=6125.50, stdev=26.16, samples=2 iops : min= 5, max= 6, avg= 5.50, stdev= 0.71, samples=2 lat (msec) : 100=58.33%, 250=33.33%, 500=8.33% cpu : usr=0.07%, sys=8.60%, ctx=60, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27966: Fri Dec 20 05:00:36 2024 read: IOPS=1, BW=1431KiB/s (1466kB/s)(5120KiB/3577msec) clat (msec): min=34, max=1326, avg=311.42, stdev=568.28 lat (msec): min=34, max=1326, avg=311.42, stdev=568.28 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 56], 60.00th=[ 56], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 2048, max= 6131, per=100.00%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=1, BW=2004KiB/s (2052kB/s)(7168KiB/3577msec) clat (msec): min=51, max=804, avg=260.62, stdev=283.35 lat (msec): min=52, max=804, avg=260.74, stdev=283.35 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 167], 60.00th=[ 174], | 70.00th=[ 174], 80.00th=[ 489], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 2048, max= 6144, per=74.03%, avg=4093.00, stdev=2048.01, samples=3 iops : min= 2, max= 6, avg= 3.67, stdev= 2.08, samples=3 lat (msec) : 50=16.67%, 100=41.67%, 250=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.08%, sys=3.55%, ctx=74, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3950KiB/s (4045kB/s), 1235KiB/s-4003KiB/s (1265kB/s-4099kB/s), io=20.0MiB (20.0MB), run=1535-5185msec WRITE: bw=5530KiB/s (5663kB/s), 987KiB/s-6176KiB/s (1011kB/s-6324kB/s), io=28.0MiB (29.4MB), run=1535-5185msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27981: Fri Dec 20 05:00:46 2024 read: IOPS=0, BW=1525KiB/s (1562kB/s)(12.0MiB/8056msec) clat (msec): min=686, max=3767, avg=2663.13, stdev=1715.50 lat (msec): min=686, max=3767, avg=2663.13, stdev=1715.50 clat percentiles (msec): | 1.00th=[ 684], 5.00th=[ 684], 10.00th=[ 684], 20.00th=[ 684], | 30.00th=[ 684], 40.00th=[ 3540], 50.00th=[ 3540], 60.00th=[ 3540], | 70.00th=[ 3775], 80.00th=[ 3775], 90.00th=[ 3775], 95.00th=[ 3775], | 99.00th=[ 3775], 99.50th=[ 3775], 99.90th=[ 3775], 99.95th=[ 3775], | 99.99th=[ 3775] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33% cpu : usr=0.20%, sys=18.39%, ctx=1232, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27984: Fri Dec 20 05:00:46 2024 read: IOPS=0, BW=760KiB/s (778kB/s)(4096KiB/5391msec) clat (nsec): min=541563k, max=541563k, avg=541563390.00, stdev= 0.00 lat (nsec): min=541566k, max=541566k, avg=541566250.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1520KiB/s (1556kB/s)(8192KiB/5391msec) clat (msec): min=241, max=4425, avg=2333.61, stdev=2958.07 lat (msec): min=242, max=4425, avg=2334.08, stdev=2958.18 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 243], 10.00th=[ 243], 20.00th=[ 243], | 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 4396], | 70.00th=[ 4396], 80.00th=[ 4396], 90.00th=[ 4396], 95.00th=[ 4396], | 99.00th=[ 4396], 99.50th=[ 4396], 99.90th=[ 4396], 99.95th=[ 4396], | 99.99th=[ 4396] lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.04%, sys=6.77%, ctx=116, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27985: Fri Dec 20 05:00:46 2024 write: IOPS=0, BW=1438KiB/s (1473kB/s)(12.0MiB/8543msec) clat (msec): min=842, max=4174, avg=2782.12, stdev=1732.17 lat (msec): min=842, max=4174, avg=2782.68, stdev=1732.27 clat percentiles (msec): | 1.00th=[ 844], 5.00th=[ 844], 10.00th=[ 844], 20.00th=[ 844], | 30.00th=[ 844], 40.00th=[ 3339], 50.00th=[ 3339], 60.00th=[ 3339], | 70.00th=[ 4178], 80.00th=[ 4178], 90.00th=[ 4178], 95.00th=[ 4178], | 99.00th=[ 4178], 99.50th=[ 4178], 99.90th=[ 4178], 99.95th=[ 4178], | 99.99th=[ 4178] bw ( KiB/s): min= 6086, max= 8175, per=100.00%, avg=7130.50, stdev=1477.15, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=33.33% cpu : usr=0.00%, sys=3.77%, ctx=83, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27986: Fri Dec 20 05:00:46 2024 write: IOPS=0, BW=2317KiB/s (2372kB/s)(12.0MiB/5304msec) clat (msec): min=261, max=4606, avg=1734.75, stdev=2487.27 lat (msec): min=262, max=4606, avg=1735.17, stdev=2487.26 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 262], 10.00th=[ 262], 20.00th=[ 262], | 30.00th=[ 262], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 4597], 80.00th=[ 4597], 90.00th=[ 4597], 95.00th=[ 4597], | 99.00th=[ 4597], 99.50th=[ 4597], 99.90th=[ 4597], 99.95th=[ 4597], | 99.99th=[ 4597] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=6.32%, ctx=84, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2034KiB/s (2083kB/s), 760KiB/s-1525KiB/s (778kB/s-1562kB/s), io=16.0MiB (16.8MB), run=5391-8056msec WRITE: bw=3836KiB/s (3928kB/s), 1438KiB/s-2317KiB/s (1473kB/s-2372kB/s), io=32.0MiB (33.6MB), run=5304-8543msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27979: Fri Dec 20 05:00:47 2024 read: IOPS=3, BW=13.9MiB/s (14.6MB/s)(12.0MiB/863msec) clat (msec): min=117, max=558, avg=279.51, stdev=242.87 lat (msec): min=117, max=558, avg=279.51, stdev=242.87 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 163], 50.00th=[ 163], 60.00th=[ 163], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min=16286, max=16286, per=100.00%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=8.82%, ctx=44, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27980: Fri Dec 20 05:00:47 2024 read: IOPS=0, BW=463KiB/s (474kB/s)(4096KiB/8849msec) clat (nsec): min=763545k, max=763545k, avg=763544670.00, stdev= 0.00 lat (nsec): min=763549k, max=763549k, avg=763548970.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 768], 5.00th=[ 768], 10.00th=[ 768], 20.00th=[ 768], | 30.00th=[ 768], 40.00th=[ 768], 50.00th=[ 768], 60.00th=[ 768], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=926KiB/s (948kB/s)(8192KiB/8849msec) clat (msec): min=523, max=7485, avg=4004.42, stdev=4922.31 lat (msec): min=524, max=7485, avg=4004.80, stdev=4922.54 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[ 523], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 7483], | 70.00th=[ 7483], 80.00th=[ 7483], 90.00th=[ 7483], 95.00th=[ 7483], | 99.00th=[ 7483], 99.50th=[ 7483], 99.90th=[ 7483], 99.95th=[ 7483], | 99.99th=[ 7483] bw ( KiB/s): min= 5793, max= 5793, per=100.00%, avg=5793.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.06%, sys=2.07%, ctx=64, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27982: Fri Dec 20 05:00:47 2024 write: IOPS=0, BW=3955KiB/s (4050kB/s)(12.0MiB/3107msec) clat (msec): min=109, max=2373, avg=1011.97, stdev=1199.43 lat (msec): min=110, max=2373, avg=1012.47, stdev=1199.47 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 2366], 80.00th=[ 2366], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=2.58%, ctx=54, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27983: Fri Dec 20 05:00:47 2024 write: IOPS=0, BW=4031KiB/s (4128kB/s)(12.0MiB/3048msec) clat (msec): min=145, max=2501, avg=989.82, stdev=1312.31 lat (msec): min=146, max=2501, avg=990.42, stdev=1312.05 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 2500], 80.00th=[ 2500], 90.00th=[ 2500], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 8159, max= 8192, per=100.00%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.07%, sys=4.36%, ctx=49, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1852KiB/s (1896kB/s), 463KiB/s-13.9MiB/s (474kB/s-14.6MB/s), io=16.0MiB (16.8MB), run=863-8849msec WRITE: bw=3703KiB/s (3792kB/s), 926KiB/s-4031KiB/s (948kB/s-4128kB/s), io=32.0MiB (33.6MB), run=3048-8849msec