== sanity test 398b: DIO and buffer IO race ============== 05:54:53 (1739530493) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.59848 s, 31.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30572: Fri Feb 14 05:55:27 2025 read: IOPS=47, BW=191KiB/s (196kB/s)(5992KiB/31342msec) clat (usec): min=64, max=271192, avg=8935.06, stdev=10608.46 lat (usec): min=64, max=271196, avg=8936.33, stdev=10608.46 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 137], | 30.00th=[ 4555], 40.00th=[ 5932], 50.00th=[ 7308], 60.00th=[ 8848], | 70.00th=[ 11338], 80.00th=[ 14222], 90.00th=[ 19006], 95.00th=[ 23987], | 99.00th=[ 36963], 99.50th=[ 46400], 99.90th=[ 60556], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 40, max= 371, per=24.70%, avg=189.23, stdev=66.04, samples=62 iops : min= 10, max= 92, avg=47.13, stdev=16.49, samples=62 write: IOPS=50, BW=201KiB/s (206kB/s)(6296KiB/31342msec) clat (usec): min=459, max=183588, avg=11293.68, stdev=12960.87 lat (usec): min=459, max=183589, avg=11295.27, stdev=12960.71 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 906], | 30.00th=[ 1598], 40.00th=[ 4490], 50.00th=[ 7504], 60.00th=[ 10683], | 70.00th=[ 14484], 80.00th=[ 19530], 90.00th=[ 27395], 95.00th=[ 36439], | 99.00th=[ 57410], 99.50th=[ 60556], 99.90th=[ 82314], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 8, max= 440, per=24.68%, avg=197.95, stdev=71.57, samples=62 iops : min= 2, max= 110, avg=49.32, stdev=17.91, samples=62 lat (usec) : 100=4.20%, 250=7.16%, 500=0.39%, 750=4.46%, 1000=7.39% lat (msec) : 2=5.37%, 4=4.10%, 10=27.73%, 20=25.26%, 50=12.89% lat (msec) : 100=0.98%, 250=0.03%, 500=0.03% cpu : usr=0.19%, sys=13.67%, ctx=4814, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30573: Fri Feb 14 05:55:27 2025 read: IOPS=48, BW=195KiB/s (199kB/s)(5940KiB/30512msec) clat (usec): min=61, max=71197, avg=8732.74, stdev=8478.37 lat (usec): min=61, max=71199, avg=8734.32, stdev=8479.36 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 133], | 30.00th=[ 4293], 40.00th=[ 5735], 50.00th=[ 7177], 60.00th=[ 8979], | 70.00th=[10945], 80.00th=[13698], 90.00th=[19006], 95.00th=[25560], | 99.00th=[36439], 99.50th=[44303], 99.90th=[67634], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 40, max= 400, per=25.37%, avg=194.34, stdev=71.45, samples=61 iops : min= 10, max= 100, avg=48.30, stdev=17.85, samples=61 write: IOPS=52, BW=208KiB/s (213kB/s)(6348KiB/30512msec) clat (usec): min=489, max=262278, avg=10954.30, stdev=13610.91 lat (usec): min=490, max=262280, avg=10956.51, stdev=13611.36 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 717], 10.00th=[ 775], 20.00th=[ 930], | 30.00th=[ 1598], 40.00th=[ 3851], 50.00th=[ 7439], 60.00th=[ 10421], | 70.00th=[ 13829], 80.00th=[ 18220], 90.00th=[ 26870], 95.00th=[ 35390], | 99.00th=[ 52167], 99.50th=[ 62653], 99.90th=[121111], 99.95th=[263193], | 99.99th=[263193] bw ( KiB/s): min= 72, max= 400, per=25.88%, avg=207.57, stdev=74.46, samples=61 iops : min= 18, max= 100, avg=51.61, stdev=18.64, samples=61 lat (usec) : 100=4.17%, 250=6.87%, 500=0.39%, 750=4.07%, 1000=7.58% lat (msec) : 2=6.05%, 4=5.37%, 10=27.96%, 20=24.32%, 50=12.50% lat (msec) : 100=0.65%, 250=0.03%, 500=0.03% cpu : usr=0.16%, sys=14.01%, ctx=4743, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30576: Fri Feb 14 05:55:27 2025 read: IOPS=49, BW=197KiB/s (201kB/s)(6028KiB/30655msec) clat (usec): min=47, max=281013, avg=8562.41, stdev=10935.29 lat (usec): min=47, max=281015, avg=8563.38, stdev=10935.32 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 89], 10.00th=[ 98], 20.00th=[ 122], | 30.00th=[ 3851], 40.00th=[ 5604], 50.00th=[ 6849], 60.00th=[ 8586], | 70.00th=[ 10552], 80.00th=[ 13435], 90.00th=[ 18482], 95.00th=[ 23987], | 99.00th=[ 37487], 99.50th=[ 45351], 99.90th=[ 79168], 99.95th=[278922], | 99.99th=[278922] bw ( KiB/s): min= 72, max= 416, per=25.50%, avg=195.31, stdev=71.07, samples=61 iops : min= 18, max= 104, avg=48.59, stdev=17.73, samples=61 write: IOPS=51, BW=204KiB/s (209kB/s)(6260KiB/30655msec) clat (usec): min=428, max=120554, avg=11237.98, stdev=12486.51 lat (usec): min=429, max=120554, avg=11239.21, stdev=12486.54 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 979], | 30.00th=[ 1958], 40.00th=[ 4293], 50.00th=[ 7111], 60.00th=[ 10683], | 70.00th=[ 14615], 80.00th=[ 19006], 90.00th=[ 27132], 95.00th=[ 36963], | 99.00th=[ 53216], 99.50th=[ 61604], 99.90th=[ 79168], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 88, max= 352, per=25.22%, avg=202.28, stdev=59.47, samples=61 iops : min= 22, max= 88, avg=50.33, stdev=14.88, samples=61 lat (usec) : 50=0.03%, 100=5.37%, 250=7.03%, 500=0.65%, 750=3.61% lat (usec) : 1000=6.90% lat (msec) : 2=5.34%, 4=6.05%, 10=27.64%, 20=23.80%, 50=12.63% lat (msec) : 100=0.88%, 250=0.03%, 500=0.03% cpu : usr=0.10%, sys=13.83%, ctx=4802, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30578: Fri Feb 14 05:55:27 2025 read: IOPS=49, BW=198KiB/s (203kB/s)(6052KiB/30563msec) clat (usec): min=50, max=254461, avg=8493.06, stdev=10224.60 lat (usec): min=50, max=254463, avg=8495.15, stdev=10224.88 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 124], | 30.00th=[ 3785], 40.00th=[ 5669], 50.00th=[ 6980], 60.00th=[ 8717], | 70.00th=[ 10683], 80.00th=[ 13698], 90.00th=[ 18482], 95.00th=[ 23987], | 99.00th=[ 36963], 99.50th=[ 42206], 99.90th=[ 57934], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 56, max= 352, per=25.78%, avg=197.48, stdev=70.48, samples=61 iops : min= 14, max= 88, avg=49.08, stdev=17.60, samples=61 write: IOPS=51, BW=204KiB/s (209kB/s)(6236KiB/30563msec) clat (usec): min=460, max=100649, avg=11252.27, stdev=12547.55 lat (usec): min=461, max=100650, avg=11253.65, stdev=12547.42 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 947], | 30.00th=[ 1680], 40.00th=[ 3916], 50.00th=[ 7832], 60.00th=[ 11076], | 70.00th=[ 14222], 80.00th=[ 18744], 90.00th=[ 26870], 95.00th=[ 35914], | 99.00th=[ 56361], 99.50th=[ 63701], 99.90th=[ 83362], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 64, max= 424, per=25.36%, avg=203.36, stdev=68.69, samples=61 iops : min= 16, max= 106, avg=50.56, stdev=17.19, samples=61 lat (usec) : 100=4.88%, 250=7.78%, 500=0.46%, 750=4.43%, 1000=6.77% lat (msec) : 2=5.53%, 4=5.70%, 10=25.91%, 20=25.46%, 50=11.95% lat (msec) : 100=1.07%, 250=0.03%, 500=0.03% cpu : usr=0.25%, sys=13.89%, ctx=4825, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=766KiB/s (785kB/s), 191KiB/s-198KiB/s (196kB/s-203kB/s), io=23.4MiB (24.6MB), run=30512-31342msec WRITE: bw=802KiB/s (821kB/s), 201KiB/s-208KiB/s (206kB/s-213kB/s), io=24.6MiB (25.7MB), run=30512-31342msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30574: Fri Feb 14 05:56:42 2025 read: IOPS=14, BW=56.8KiB/s (58.1kB/s)(5992KiB/105522msec) clat (msec): min=2, max=284, avg= 9.42, stdev=10.84 lat (msec): min=2, max=284, avg= 9.42, stdev=10.84 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 19], 95.00th=[ 24], | 99.00th=[ 40], 99.50th=[ 53], 99.90th=[ 138], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 7, max= 152, per=25.44%, avg=57.24, stdev=32.70, samples=208 iops : min= 1, max= 38, avg=14.03, stdev= 8.25, samples=208 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(6296KiB/105522msec) clat (msec): min=15, max=231, avg=57.97, stdev=26.56 lat (msec): min=15, max=231, avg=57.97, stdev=26.56 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 26], 10.00th=[ 31], 20.00th=[ 37], | 30.00th=[ 43], 40.00th=[ 48], 50.00th=[ 54], 60.00th=[ 59], | 70.00th=[ 66], 80.00th=[ 74], 90.00th=[ 91], 95.00th=[ 109], | 99.00th=[ 153], 99.50th=[ 161], 99.90th=[ 209], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 8, max= 152, per=25.11%, avg=59.27, stdev=23.24, samples=211 iops : min= 2, max= 38, avg=14.54, stdev= 5.91, samples=211 lat (msec) : 4=6.02%, 10=30.76%, 20=8.69%, 50=25.29%, 100=25.72% lat (msec) : 250=3.48%, 500=0.03% cpu : usr=0.03%, sys=3.22%, ctx=3544, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30575: Fri Feb 14 05:56:42 2025 read: IOPS=13, BW=55.8KiB/s (57.1kB/s)(5940KiB/106502msec) clat (msec): min=2, max=263, avg= 9.44, stdev=10.36 lat (msec): min=2, max=263, avg= 9.44, stdev=10.36 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 19], 95.00th=[ 27], | 99.00th=[ 44], 99.50th=[ 50], 99.90th=[ 78], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 7, max= 160, per=24.87%, avg=55.97, stdev=32.59, samples=210 iops : min= 1, max= 40, avg=13.80, stdev= 8.20, samples=210 write: IOPS=14, BW=59.6KiB/s (61.0kB/s)(6348KiB/106502msec) clat (msec): min=13, max=229, avg=58.16, stdev=26.59 lat (msec): min=13, max=229, avg=58.16, stdev=26.59 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 27], 10.00th=[ 31], 20.00th=[ 37], | 30.00th=[ 43], 40.00th=[ 48], 50.00th=[ 54], 60.00th=[ 59], | 70.00th=[ 66], 80.00th=[ 77], 90.00th=[ 91], 95.00th=[ 110], | 99.00th=[ 153], 99.50th=[ 171], 99.90th=[ 211], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 8, max= 136, per=25.05%, avg=59.11, stdev=23.06, samples=212 iops : min= 2, max= 34, avg=14.58, stdev= 5.82, samples=212 lat (msec) : 4=6.18%, 10=29.72%, 20=8.89%, 50=26.99%, 100=24.51% lat (msec) : 250=3.68%, 500=0.03% cpu : usr=0.09%, sys=3.14%, ctx=3507, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30577: Fri Feb 14 05:56:42 2025 read: IOPS=14, BW=56.9KiB/s (58.2kB/s)(6028KiB/106021msec) clat (msec): min=2, max=148, avg= 9.91, stdev=10.09 lat (msec): min=2, max=148, avg= 9.92, stdev=10.09 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 20], 95.00th=[ 28], | 99.00th=[ 47], 99.50th=[ 64], 99.90th=[ 138], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 7, max= 184, per=25.50%, avg=57.38, stdev=32.57, samples=208 iops : min= 1, max= 46, avg=14.14, stdev= 8.17, samples=208 write: IOPS=14, BW=59.0KiB/s (60.5kB/s)(6260KiB/106021msec) clat (msec): min=16, max=348, avg=58.08, stdev=26.99 lat (msec): min=16, max=348, avg=58.09, stdev=26.99 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 28], 10.00th=[ 31], 20.00th=[ 37], | 30.00th=[ 43], 40.00th=[ 48], 50.00th=[ 54], 60.00th=[ 60], | 70.00th=[ 66], 80.00th=[ 74], 90.00th=[ 89], 95.00th=[ 104], | 99.00th=[ 153], 99.50th=[ 182], 99.90th=[ 228], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 8, max= 136, per=24.84%, avg=58.62, stdev=22.86, samples=211 iops : min= 2, max= 34, avg=14.45, stdev= 5.75, samples=211 lat (msec) : 4=6.18%, 10=29.56%, 20=9.02%, 50=26.07%, 100=25.94% lat (msec) : 250=3.19%, 500=0.03% cpu : usr=0.07%, sys=3.17%, ctx=3518, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30579: Fri Feb 14 05:56:42 2025 read: IOPS=14, BW=57.3KiB/s (58.7kB/s)(6052KiB/105546msec) clat (msec): min=2, max=152, avg= 9.13, stdev= 8.56 lat (msec): min=2, max=152, avg= 9.13, stdev= 8.56 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 17], 95.00th=[ 25], | 99.00th=[ 42], 99.50th=[ 57], 99.90th=[ 81], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 7, max= 216, per=25.96%, avg=58.40, stdev=33.64, samples=206 iops : min= 1, max= 54, avg=14.33, stdev= 8.47, samples=206 write: IOPS=14, BW=59.1KiB/s (60.5kB/s)(6236KiB/105546msec) clat (msec): min=14, max=317, avg=58.73, stdev=27.77 lat (msec): min=14, max=317, avg=58.74, stdev=27.77 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 26], 10.00th=[ 30], 20.00th=[ 37], | 30.00th=[ 43], 40.00th=[ 49], 50.00th=[ 55], 60.00th=[ 60], | 70.00th=[ 68], 80.00th=[ 77], 90.00th=[ 89], 95.00th=[ 110], | 99.00th=[ 155], 99.50th=[ 174], 99.90th=[ 266], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 15, max= 144, per=24.88%, avg=58.72, stdev=23.40, samples=211 iops : min= 3, max= 36, avg=14.41, stdev= 5.91, samples=211 lat (msec) : 4=6.87%, 10=29.85%, 20=9.77%, 50=23.89%, 100=25.98% lat (msec) : 250=3.58%, 500=0.07% cpu : usr=0.05%, sys=3.24%, ctx=3543, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=225KiB/s (231kB/s), 55.8KiB/s-57.3KiB/s (57.1kB/s-58.7kB/s), io=23.4MiB (24.6MB), run=105522-106502msec WRITE: bw=236KiB/s (242kB/s), 59.0KiB/s-59.7KiB/s (60.5kB/s-61.1kB/s), io=24.6MiB (25.7MB), run=105522-106502msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30678: Fri Feb 14 05:56:50 2025 read: IOPS=64, BW=1028KiB/s (1053kB/s)(5920KiB/5756msec) clat (usec): min=62, max=102722, avg=7936.05, stdev=8982.06 lat (usec): min=62, max=102724, avg=7937.01, stdev=8982.11 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 98], 10.00th=[ 105], 20.00th=[ 135], | 30.00th=[ 4686], 40.00th=[ 6063], 50.00th=[ 7111], 60.00th=[ 8455], | 70.00th=[ 9765], 80.00th=[ 11338], 90.00th=[ 14222], 95.00th=[ 19006], | 99.00th=[ 31851], 99.50th=[ 80217], 99.90th=[102237], 99.95th=[102237], | 99.99th=[102237] bw ( KiB/s): min= 734, max= 1440, per=26.04%, avg=1013.91, stdev=231.37, samples=11 iops : min= 45, max= 90, avg=63.09, stdev=14.56, samples=11 write: IOPS=69, BW=1106KiB/s (1133kB/s)(6368KiB/5756msec) clat (usec): min=684, max=79218, avg=6739.91, stdev=9096.61 lat (usec): min=685, max=79220, avg=6742.61, stdev=9096.24 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1004], 10.00th=[ 1074], 20.00th=[ 1188], | 30.00th=[ 1401], 40.00th=[ 1926], 50.00th=[ 2638], 60.00th=[ 5014], | 70.00th=[ 7635], 80.00th=[11731], 90.00th=[16450], 95.00th=[21365], | 99.00th=[56361], 99.50th=[68682], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 573, max= 1696, per=27.10%, avg=1110.09, stdev=403.97, samples=11 iops : min= 35, max= 106, avg=69.09, stdev=25.52, samples=11 lat (usec) : 100=3.39%, 250=7.03%, 500=0.65%, 750=0.13%, 1000=2.60% lat (msec) : 2=19.14%, 4=8.98%, 10=32.29%, 20=21.09%, 50=3.78% lat (msec) : 100=0.78%, 250=0.13% cpu : usr=0.19%, sys=18.04%, ctx=1185, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30679: Fri Feb 14 05:56:50 2025 read: IOPS=58, BW=934KiB/s (956kB/s)(5744KiB/6150msec) clat (usec): min=58, max=65513, avg=6798.54, stdev=7409.66 lat (usec): min=58, max=65514, avg=6799.61, stdev=7409.81 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 91], 10.00th=[ 102], 20.00th=[ 117], | 30.00th=[ 159], 40.00th=[ 4686], 50.00th=[ 5932], 60.00th=[ 7439], | 70.00th=[ 9372], 80.00th=[11469], 90.00th=[15139], 95.00th=[17957], | 99.00th=[33817], 99.50th=[53216], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 63, max= 1529, per=23.72%, avg=923.58, stdev=462.03, samples=12 iops : min= 3, max= 95, avg=57.33, stdev=29.03, samples=12 write: IOPS=66, BW=1064KiB/s (1090kB/s)(6544KiB/6150msec) clat (usec): min=753, max=95611, avg=8812.22, stdev=13915.37 lat (usec): min=754, max=95612, avg=8813.79, stdev=13915.36 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 988], 10.00th=[ 1090], 20.00th=[ 1188], | 30.00th=[ 1450], 40.00th=[ 2147], 50.00th=[ 3130], 60.00th=[ 5932], | 70.00th=[ 8979], 80.00th=[12780], 90.00th=[20317], 95.00th=[27657], | 99.00th=[81265], 99.50th=[89654], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 159, max= 1792, per=25.69%, avg=1052.67, stdev=505.65, samples=12 iops : min= 9, max= 112, avg=65.42, stdev=31.81, samples=12 lat (usec) : 100=4.30%, 250=10.68%, 500=0.52%, 750=0.26%, 1000=3.78% lat (msec) : 2=17.84%, 4=8.33%, 10=27.47%, 20=19.92%, 50=4.95% lat (msec) : 100=1.95% cpu : usr=0.34%, sys=16.23%, ctx=1119, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30680: Fri Feb 14 05:56:50 2025 read: IOPS=68, BW=1095KiB/s (1122kB/s)(6256KiB/5711msec) clat (usec): min=54, max=96415, avg=8237.71, stdev=10594.67 lat (usec): min=55, max=96416, avg=8238.64, stdev=10594.70 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 122], | 30.00th=[ 3851], 40.00th=[ 5800], 50.00th=[ 7046], 60.00th=[ 8094], | 70.00th=[ 9634], 80.00th=[11600], 90.00th=[15664], 95.00th=[21627], | 99.00th=[71828], 99.50th=[87557], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 764, max= 1756, per=28.04%, avg=1091.91, stdev=330.95, samples=11 iops : min= 47, max= 109, avg=67.82, stdev=20.74, samples=11 write: IOPS=66, BW=1056KiB/s (1082kB/s)(6032KiB/5711msec) clat (usec): min=754, max=54206, avg=6518.84, stdev=7203.31 lat (usec): min=755, max=54208, avg=6520.35, stdev=7203.33 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1029], 10.00th=[ 1090], 20.00th=[ 1221], | 30.00th=[ 1500], 40.00th=[ 1975], 50.00th=[ 2835], 60.00th=[ 5604], | 70.00th=[ 8455], 80.00th=[12387], 90.00th=[16909], 95.00th=[20841], | 99.00th=[27132], 99.50th=[50070], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 447, max= 1529, per=25.02%, avg=1025.00, stdev=308.65, samples=11 iops : min= 27, max= 95, avg=63.64, stdev=19.49, samples=11 lat (usec) : 100=4.30%, 250=8.85%, 500=0.78%, 750=0.39%, 1000=2.47% lat (msec) : 2=17.84%, 4=8.07%, 10=30.60%, 20=21.35%, 50=4.43% lat (msec) : 100=0.91% cpu : usr=0.09%, sys=17.76%, ctx=1144, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30682: Fri Feb 14 05:56:50 2025 read: IOPS=65, BW=1055KiB/s (1081kB/s)(6032KiB/5715msec) clat (usec): min=60, max=51656, avg=7424.57, stdev=6458.20 lat (usec): min=61, max=51657, avg=7425.73, stdev=6458.15 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 94], 10.00th=[ 101], 20.00th=[ 124], | 30.00th=[ 4621], 40.00th=[ 5800], 50.00th=[ 6915], 60.00th=[ 8225], | 70.00th=[ 9503], 80.00th=[11863], 90.00th=[14353], 95.00th=[19006], | 99.00th=[27395], 99.50th=[39584], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 606, max= 1434, per=26.99%, avg=1051.00, stdev=276.30, samples=11 iops : min= 37, max= 89, avg=65.18, stdev=17.40, samples=11 write: IOPS=68, BW=1095KiB/s (1121kB/s)(6256KiB/5715msec) clat (usec): min=665, max=110445, avg=7140.11, stdev=10043.99 lat (usec): min=666, max=110447, avg=7144.95, stdev=10044.52 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1029], 10.00th=[ 1106], 20.00th=[ 1205], | 30.00th=[ 1418], 40.00th=[ 2057], 50.00th=[ 3032], 60.00th=[ 5342], | 70.00th=[ 8455], 80.00th=[ 11994], 90.00th=[ 16909], 95.00th=[ 20579], | 99.00th=[ 53740], 99.50th=[ 68682], 99.90th=[110625], 99.95th=[110625], | 99.99th=[110625] bw ( KiB/s): min= 670, max= 1568, per=27.00%, avg=1106.27, stdev=275.72, samples=11 iops : min= 41, max= 98, avg=68.64, stdev=17.41, samples=11 lat (usec) : 100=4.69%, 250=7.03%, 500=0.65%, 750=0.52%, 1000=1.95% lat (msec) : 2=17.84%, 4=8.20%, 10=31.51%, 20=23.31%, 50=3.65% lat (msec) : 100=0.52%, 250=0.13% cpu : usr=0.46%, sys=17.73%, ctx=1142, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3895KiB/s (3988kB/s), 934KiB/s-1095KiB/s (956kB/s-1122kB/s), io=23.4MiB (24.5MB), run=5711-6150msec WRITE: bw=4098KiB/s (4196kB/s), 1056KiB/s-1106KiB/s (1082kB/s-1133kB/s), io=24.6MiB (25.8MB), run=5711-6150msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30681: Fri Feb 14 05:57:11 2025 read: IOPS=13, BW=217KiB/s (222kB/s)(5920KiB/27276msec) clat (usec): min=2901, max=65540, avg=9527.55, stdev=8621.03 lat (usec): min=2903, max=65541, avg=9530.38, stdev=8620.40 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 3982], 10.00th=[ 4359], 20.00th=[ 5080], | 30.00th=[ 5538], 40.00th=[ 6063], 50.00th=[ 6783], 60.00th=[ 7701], | 70.00th=[ 8848], 80.00th=[11207], 90.00th=[16450], 95.00th=[25297], | 99.00th=[55837], 99.50th=[62129], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 32, max= 448, per=24.68%, avg=214.74, stdev=98.34, samples=54 iops : min= 2, max= 28, avg=13.09, stdev= 6.16, samples=54 write: IOPS=14, BW=233KiB/s (239kB/s)(6368KiB/27276msec) clat (msec): min=25, max=179, avg=59.37, stdev=18.31 lat (msec): min=25, max=179, avg=59.37, stdev=18.31 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 49], 40.00th=[ 53], 50.00th=[ 58], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 72], 90.00th=[ 84], 95.00th=[ 93], | 99.00th=[ 126], 99.50th=[ 140], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 95, max= 320, per=25.32%, avg=231.93, stdev=51.77, samples=54 iops : min= 5, max= 20, avg=14.17, stdev= 3.34, samples=54 lat (msec) : 4=2.47%, 10=33.59%, 20=8.59%, 50=20.83%, 100=33.20% lat (msec) : 250=1.30% cpu : usr=0.05%, sys=3.41%, ctx=903, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30683: Fri Feb 14 05:57:11 2025 read: IOPS=13, BW=209KiB/s (214kB/s)(5744KiB/27501msec) clat (usec): min=3223, max=54585, avg=9128.52, stdev=6908.19 lat (usec): min=3224, max=54586, avg=9131.53, stdev=6908.09 clat percentiles (usec): | 1.00th=[ 3556], 5.00th=[ 4080], 10.00th=[ 4359], 20.00th=[ 5014], | 30.00th=[ 5538], 40.00th=[ 6128], 50.00th=[ 6783], 60.00th=[ 7701], | 70.00th=[ 9372], 80.00th=[11600], 90.00th=[15926], 95.00th=[24511], | 99.00th=[42206], 99.50th=[45351], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 63, max= 512, per=23.93%, avg=208.15, stdev=115.29, samples=54 iops : min= 3, max= 32, avg=12.61, stdev= 7.41, samples=54 write: IOPS=14, BW=238KiB/s (244kB/s)(6544KiB/27501msec) clat (msec): min=25, max=129, avg=59.14, stdev=18.78 lat (msec): min=25, max=129, avg=59.14, stdev=18.78 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 37], 20.00th=[ 43], | 30.00th=[ 48], 40.00th=[ 53], 50.00th=[ 58], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 73], 90.00th=[ 85], 95.00th=[ 96], | 99.00th=[ 116], 99.50th=[ 126], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 95, max= 352, per=25.83%, avg=236.57, stdev=53.74, samples=54 iops : min= 5, max= 22, avg=14.39, stdev= 3.47, samples=54 lat (msec) : 4=1.30%, 10=33.72%, 20=8.59%, 50=21.61%, 100=32.55% lat (msec) : 250=2.21% cpu : usr=0.02%, sys=3.43%, ctx=893, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30684: Fri Feb 14 05:57:11 2025 read: IOPS=15, BW=240KiB/s (246kB/s)(6256KiB/26049msec) clat (usec): min=2805, max=66181, avg=9219.48, stdev=7448.81 lat (usec): min=2806, max=66183, avg=9221.78, stdev=7448.84 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4015], 10.00th=[ 4424], 20.00th=[ 5145], | 30.00th=[ 5604], 40.00th=[ 6259], 50.00th=[ 7177], 60.00th=[ 8094], | 70.00th=[ 9241], 80.00th=[11338], 90.00th=[15270], 95.00th=[19268], | 99.00th=[49546], 99.50th=[65274], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 32, max= 479, per=27.61%, avg=240.17, stdev=118.86, samples=52 iops : min= 2, max= 29, avg=14.63, stdev= 7.37, samples=52 write: IOPS=14, BW=232KiB/s (237kB/s)(6032KiB/26049msec) clat (msec): min=20, max=160, avg=59.21, stdev=17.95 lat (msec): min=20, max=160, avg=59.21, stdev=17.95 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 44], | 30.00th=[ 49], 40.00th=[ 53], 50.00th=[ 57], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 72], 90.00th=[ 83], 95.00th=[ 92], | 99.00th=[ 118], 99.50th=[ 128], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 127, max= 352, per=25.21%, avg=230.96, stdev=47.41, samples=52 iops : min= 7, max= 22, avg=14.06, stdev= 3.13, samples=52 lat (msec) : 4=2.34%, 10=35.81%, 20=10.29%, 50=18.75%, 100=31.64% lat (msec) : 250=1.17% cpu : usr=0.00%, sys=3.69%, ctx=914, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30685: Fri Feb 14 05:57:11 2025 read: IOPS=14, BW=224KiB/s (230kB/s)(6032KiB/26890msec) clat (usec): min=2954, max=97570, avg=9098.02, stdev=8470.82 lat (usec): min=2956, max=97572, avg=9100.08, stdev=8470.81 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3982], 10.00th=[ 4490], 20.00th=[ 5145], | 30.00th=[ 5735], 40.00th=[ 6259], 50.00th=[ 6783], 60.00th=[ 7504], | 70.00th=[ 8717], 80.00th=[10683], 90.00th=[15664], 95.00th=[19530], | 99.00th=[49021], 99.50th=[82314], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 32, max= 448, per=25.56%, avg=222.42, stdev=97.63, samples=53 iops : min= 2, max= 28, avg=13.60, stdev= 6.17, samples=53 write: IOPS=14, BW=233KiB/s (238kB/s)(6256KiB/26890msec) clat (msec): min=29, max=138, avg=59.58, stdev=17.85 lat (msec): min=29, max=138, avg=59.58, stdev=17.85 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 44], | 30.00th=[ 49], 40.00th=[ 54], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 83], 95.00th=[ 91], | 99.00th=[ 122], 99.50th=[ 133], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 127, max= 352, per=25.27%, avg=231.45, stdev=55.75, samples=53 iops : min= 7, max= 22, avg=14.17, stdev= 3.57, samples=53 lat (msec) : 4=2.60%, 10=35.03%, 20=9.11%, 50=18.23%, 100=33.72% lat (msec) : 250=1.30% cpu : usr=0.03%, sys=3.54%, ctx=915, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=871KiB/s (892kB/s), 209KiB/s-240KiB/s (214kB/s-246kB/s), io=23.4MiB (24.5MB), run=26049-27501msec WRITE: bw=916KiB/s (938kB/s), 232KiB/s-238KiB/s (237kB/s-244kB/s), io=24.6MiB (25.8MB), run=26049-27501msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30717: Fri Feb 14 05:57:14 2025 read: IOPS=4, BW=4551KiB/s (4660kB/s)(7168KiB/1575msec) clat (msec): min=62, max=146, avg=87.54, stdev=33.93 lat (msec): min=62, max=146, avg=87.54, stdev=33.93 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 126], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 2043, max= 8143, per=42.92%, avg=5093.00, stdev=4313.35, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 write: IOPS=3, BW=3251KiB/s (3329kB/s)(5120KiB/1575msec) clat (msec): min=139, max=245, avg=183.50, stdev=43.66 lat (msec): min=139, max=245, avg=183.56, stdev=43.66 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 146], 40.00th=[ 146], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 247], 95.00th=[ 247], | 99.00th=[ 247], 99.50th=[ 247], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 2035, max= 4087, per=18.43%, avg=3061.00, stdev=1450.98, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (msec) : 100=41.67%, 250=58.33% cpu : usr=0.00%, sys=17.03%, ctx=262, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30718: Fri Feb 14 05:57:14 2025 read: IOPS=1, BW=1187KiB/s (1215kB/s)(2048KiB/1726msec) clat (msec): min=64, max=153, avg=109.13, stdev=63.17 lat (msec): min=64, max=153, avg=109.13, stdev=63.17 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 2048, max= 2048, per=17.26%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5933KiB/s (6075kB/s)(10.0MiB/1726msec) clat (msec): min=26, max=339, avg=144.70, stdev=104.67 lat (msec): min=26, max=339, avg=144.91, stdev=104.56 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 43], | 30.00th=[ 72], 40.00th=[ 86], 50.00th=[ 93], 60.00th=[ 111], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 266], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 2048, max= 8192, per=31.07%, avg=5160.67, stdev=3072.81, samples=3 iops : min= 2, max= 8, avg= 5.00, stdev= 3.00, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=33.33%, 500=16.67% cpu : usr=0.00%, sys=13.22%, ctx=200, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30719: Fri Feb 14 05:57:14 2025 read: IOPS=5, BW=5874KiB/s (6015kB/s)(6144KiB/1046msec) clat (usec): min=466, max=128478, avg=63089.66, stdev=42294.70 lat (usec): min=468, max=128480, avg=63091.21, stdev=42294.79 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 49021], | 30.00th=[ 49021], 40.00th=[ 51119], 50.00th=[ 51119], 60.00th=[ 66847], | 70.00th=[ 82314], 80.00th=[ 82314], 90.00th=[128451], 95.00th=[128451], | 99.00th=[128451], 99.50th=[128451], 99.90th=[128451], 99.95th=[128451], | 99.99th=[128451] bw ( KiB/s): min= 2043, max= 8192, per=43.13%, avg=5117.50, stdev=4348.00, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 write: IOPS=5, BW=5874KiB/s (6015kB/s)(6144KiB/1046msec) clat (msec): min=45, max=215, avg=109.74, stdev=63.80 lat (msec): min=46, max=215, avg=109.82, stdev=63.80 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 94], 50.00th=[ 94], 60.00th=[ 121], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 6131, max= 6144, per=36.95%, avg=6137.50, stdev= 9.19, samples=2 iops : min= 5, max= 6, avg= 5.50, stdev= 0.71, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=21.91%, ctx=455, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30720: Fri Feb 14 05:57:14 2025 read: IOPS=4, BW=4376KiB/s (4481kB/s)(5120KiB/1170msec) clat (usec): min=676, max=77729, avg=35150.93, stdev=35894.04 lat (usec): min=678, max=77730, avg=35152.47, stdev=35893.69 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 955], 40.00th=[ 955], 50.00th=[30278], 60.00th=[30278], | 70.00th=[66323], 80.00th=[66323], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 8175, max= 8175, per=68.90%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6126KiB/s (6274kB/s)(7168KiB/1170msec) clat (msec): min=56, max=230, avg=110.10, stdev=58.09 lat (msec): min=56, max=230, avg=110.28, stdev=58.34 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 58], | 30.00th=[ 99], 40.00th=[ 99], 50.00th=[ 102], 60.00th=[ 109], | 70.00th=[ 109], 80.00th=[ 120], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 6119, max= 6131, per=36.87%, avg=6125.00, stdev= 8.49, samples=2 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=2 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 50=8.33%, 100=41.67%, 250=33.33% cpu : usr=0.00%, sys=17.28%, ctx=88, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.6MiB/s (12.1MB/s), 1187KiB/s-5874KiB/s (1215kB/s-6015kB/s), io=20.0MiB (20.0MB), run=1046-1726msec WRITE: bw=16.2MiB/s (17.0MB/s), 3251KiB/s-6126KiB/s (3329kB/s-6274kB/s), io=28.0MiB (29.4MB), run=1046-1726msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30713: Fri Feb 14 05:57:14 2025 read: IOPS=4, BW=4652KiB/s (4763kB/s)(7168KiB/1541msec) clat (msec): min=24, max=138, avg=65.06, stdev=41.97 lat (msec): min=24, max=138, avg=65.06, stdev=41.97 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 32], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 47], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 2048, max=10219, per=61.61%, avg=6133.50, stdev=5777.77, samples=2 iops : min= 2, max= 9, avg= 5.50, stdev= 4.95, samples=2 write: IOPS=3, BW=3323KiB/s (3402kB/s)(5120KiB/1541msec) clat (msec): min=145, max=291, avg=212.57, stdev=59.90 lat (msec): min=145, max=291, avg=212.66, stdev=59.90 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 2043, max= 4096, per=24.48%, avg=3411.67, stdev=1185.30, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 50=33.33%, 100=16.67%, 250=33.33%, 500=16.67% cpu : usr=0.06%, sys=6.62%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30714: Fri Feb 14 05:57:14 2025 read: IOPS=0, BW=996KiB/s (1020kB/s)(2048KiB/2057msec) clat (usec): min=41881, max=42171, avg=42026.73, stdev=204.87 lat (usec): min=41884, max=42173, avg=42029.21, stdev=204.05 clat percentiles (usec): | 1.00th=[41681], 5.00th=[41681], 10.00th=[41681], 20.00th=[41681], | 30.00th=[41681], 40.00th=[41681], 50.00th=[41681], 60.00th=[42206], | 70.00th=[42206], 80.00th=[42206], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 2048, max= 2048, per=20.57%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4978KiB/s (5098kB/s)(10.0MiB/2057msec) clat (msec): min=98, max=365, avg=169.94, stdev=96.89 lat (msec): min=98, max=365, avg=170.28, stdev=97.17 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 101], | 30.00th=[ 103], 40.00th=[ 106], 50.00th=[ 116], 60.00th=[ 130], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 275], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 4047, max= 6144, per=36.56%, avg=5095.50, stdev=1482.80, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=16.67%, 250=41.67%, 500=25.00% cpu : usr=0.00%, sys=7.30%, ctx=83, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30715: Fri Feb 14 05:57:14 2025 read: IOPS=2, BW=3069KiB/s (3143kB/s)(6144KiB/2002msec) clat (msec): min=30, max=296, avg=108.31, stdev=98.80 lat (msec): min=30, max=296, avg=108.31, stdev=98.80 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 113], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 2043, max= 6144, per=41.12%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=2, BW=3069KiB/s (3143kB/s)(6144KiB/2002msec) clat (msec): min=118, max=292, avg=208.22, stdev=75.58 lat (msec): min=119, max=292, avg=208.31, stdev=75.57 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 163], 50.00th=[ 163], 60.00th=[ 241], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 2048, max= 4087, per=19.57%, avg=2727.67, stdev=1177.22, samples=3 iops : min= 2, max= 3, avg= 2.33, stdev= 0.58, samples=3 lat (msec) : 50=16.67%, 100=8.33%, 250=50.00%, 500=25.00% cpu : usr=0.10%, sys=5.00%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30716: Fri Feb 14 05:57:14 2025 read: IOPS=2, BW=2725KiB/s (2790kB/s)(5120KiB/1879msec) clat (usec): min=31875, max=85095, avg=58645.00, stdev=24249.73 lat (usec): min=31877, max=85097, avg=58646.77, stdev=24249.88 clat percentiles (usec): | 1.00th=[31851], 5.00th=[31851], 10.00th=[31851], 20.00th=[31851], | 30.00th=[37487], 40.00th=[37487], 50.00th=[57934], 60.00th=[57934], | 70.00th=[80217], 80.00th=[80217], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 8192, max= 8192, per=82.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3815KiB/s (3906kB/s)(7168KiB/1879msec) clat (msec): min=112, max=329, avg=187.95, stdev=74.22 lat (msec): min=112, max=329, avg=188.04, stdev=74.22 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 121], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 174], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 218], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 4096, max= 6083, per=36.52%, avg=5089.50, stdev=1405.02, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (msec) : 50=16.67%, 100=25.00%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=7.45%, ctx=59, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9956KiB/s (10.2MB/s), 996KiB/s-4652KiB/s (1020kB/s-4763kB/s), io=20.0MiB (20.0MB), run=1541-2057msec WRITE: bw=13.6MiB/s (14.3MB/s), 3069KiB/s-4978KiB/s (3143kB/s-5098kB/s), io=28.0MiB (29.4MB), run=1541-2057msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30730: Fri Feb 14 05:57:17 2025 read: IOPS=3, BW=15.4MiB/s (16.2MB/s)(12.0MiB/779msec) clat (msec): min=33, max=367, avg=216.80, stdev=169.40 lat (msec): min=33, max=367, avg=216.81, stdev=169.40 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 8175, max= 8175, per=70.66%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.13%, sys=13.37%, ctx=537, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30731: Fri Feb 14 05:57:17 2025 read: IOPS=0, BW=2893KiB/s (2962kB/s)(4096KiB/1416msec) clat (nsec): min=381957k, max=381957k, avg=381956903.00, stdev= 0.00 lat (nsec): min=381959k, max=381959k, avg=381959234.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 5970, max= 5970, per=51.60%, avg=5970.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5785KiB/s (5924kB/s)(8192KiB/1416msec) clat (msec): min=362, max=543, avg=453.43, stdev=127.93 lat (msec): min=363, max=544, avg=453.85, stdev=127.92 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 5970, max= 5970, per=27.04%, avg=5970.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.14%, sys=15.76%, ctx=462, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30733: Fri Feb 14 05:57:17 2025 write: IOPS=2, BW=8280KiB/s (8479kB/s)(12.0MiB/1484msec) clat (msec): min=191, max=796, avg=479.27, stdev=303.70 lat (msec): min=192, max=797, avg=479.61, stdev=303.68 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 8175, max= 8175, per=37.02%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.07%, sys=20.70%, ctx=68, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30735: Fri Feb 14 05:57:17 2025 write: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1023msec) clat (msec): min=221, max=428, avg=328.84, stdev=103.52 lat (msec): min=222, max=428, avg=329.23, stdev=103.37 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 8126, max= 8192, per=36.95%, avg=8159.00, stdev=46.67, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.10%, sys=25.54%, ctx=72, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.3MiB/s (11.8MB/s), 2893KiB/s-15.4MiB/s (2962kB/s-16.2MB/s), io=16.0MiB (16.8MB), run=779-1416msec WRITE: bw=21.6MiB/s (22.6MB/s), 5785KiB/s-11.7MiB/s (5924kB/s-12.3MB/s), io=32.0MiB (33.6MB), run=1023-1484msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30728: Fri Feb 14 05:57:17 2025 read: IOPS=5, BW=23.1MiB/s (24.2MB/s)(12.0MiB/519msec) clat (msec): min=112, max=242, avg=162.91, stdev=69.81 lat (msec): min=112, max=242, avg=162.91, stdev=69.81 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 134], | 70.00th=[ 243], 80.00th=[ 243], 90.00th=[ 243], 95.00th=[ 243], | 99.00th=[ 243], 99.50th=[ 243], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min=16253, max=16253, per=100.00%, avg=16253.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=13.90%, ctx=165, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30729: Fri Feb 14 05:57:17 2025 read: IOPS=0, BW=2499KiB/s (2559kB/s)(4096KiB/1639msec) clat (nsec): min=514437k, max=514437k, avg=514437183.00, stdev= 0.00 lat (nsec): min=514440k, max=514440k, avg=514439519.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 8192, max= 8192, per=81.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4998KiB/s (5118kB/s)(8192KiB/1639msec) clat (msec): min=464, max=575, avg=520.39, stdev=78.61 lat (msec): min=465, max=576, avg=520.72, stdev=78.45 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.31%, sys=8.36%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30732: Fri Feb 14 05:57:17 2025 write: IOPS=2, BW=9323KiB/s (9547kB/s)(12.0MiB/1318msec) clat (msec): min=189, max=818, avg=424.07, stdev=343.39 lat (msec): min=189, max=818, avg=424.55, stdev=343.62 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 266], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8192, max= 8192, per=40.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.38%, sys=7.14%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30734: Fri Feb 14 05:57:17 2025 write: IOPS=2, BW=9295KiB/s (9518kB/s)(12.0MiB/1322msec) clat (msec): min=324, max=512, avg=423.32, stdev=94.19 lat (msec): min=324, max=512, avg=423.86, stdev=94.26 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min=16384, max=16384, per=81.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.30%, sys=8.18%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9996KiB/s (10.2MB/s), 2499KiB/s-23.1MiB/s (2559kB/s-24.2MB/s), io=16.0MiB (16.8MB), run=519-1639msec WRITE: bw=19.5MiB/s (20.5MB/s), 4998KiB/s-9323KiB/s (5118kB/s-9547kB/s), io=32.0MiB (33.6MB), run=1318-1639msec