== sanity test 398b: DIO and buffer IO race ============== 10:55:50 (1743519350) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 6.47257 s, 7.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=530460: Tue Apr 1 10:58:32 2025 read: IOPS=11, BW=44.7KiB/s (45.7kB/s)(5992KiB/134157msec) clat (usec): min=612, max=3035.2k, avg=51021.85, stdev=79870.32 lat (usec): min=612, max=3035.2k, avg=51022.26, stdev=79870.32 clat percentiles (usec): | 1.00th=[ 1352], 5.00th=[ 21627], 10.00th=[ 26346], | 20.00th=[ 32637], 30.00th=[ 36963], 40.00th=[ 42206], | 50.00th=[ 46400], 60.00th=[ 51119], 70.00th=[ 57410], | 80.00th=[ 63177], 90.00th=[ 74974], 95.00th=[ 86508], | 99.00th=[ 113771], 99.50th=[ 127402], 99.90th=[ 173016], | 99.95th=[3036677], 99.99th=[3036677] bw ( KiB/s): min= 7, max= 95, per=25.33%, avg=45.09, stdev=14.41, samples=256 iops : min= 1, max= 23, avg=10.60, stdev= 3.63, samples=256 write: IOPS=11, BW=46.9KiB/s (48.1kB/s)(6296KiB/134157msec); 0 zone resets clat (msec): min=3, max=140, avg=36.24, stdev=24.97 lat (msec): min=3, max=140, avg=36.25, stdev=24.97 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 20], 40.00th=[ 25], 50.00th=[ 30], 60.00th=[ 36], | 70.00th=[ 44], 80.00th=[ 57], 90.00th=[ 74], 95.00th=[ 87], | 99.00th=[ 106], 99.50th=[ 117], 99.90th=[ 132], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 7, max= 134, per=25.29%, avg=47.30, stdev=18.64, samples=256 iops : min= 1, max= 33, avg=11.16, stdev= 4.72, samples=256 lat (usec) : 750=0.03%, 1000=0.07% lat (msec) : 2=0.59%, 4=0.29%, 10=3.91%, 20=12.63%, 50=48.93% lat (msec) : 100=31.35%, 250=2.18%, >=2000=0.03% cpu : usr=0.03%, sys=28.74%, ctx=5519, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530461: Tue Apr 1 10:58:32 2025 read: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5940KiB/133472msec) clat (usec): min=566, max=156437, avg=49544.40, stdev=21395.48 lat (usec): min=567, max=156437, avg=49544.84, stdev=21395.52 clat percentiles (usec): | 1.00th=[ 1582], 5.00th=[ 23200], 10.00th=[ 27132], 20.00th=[ 32637], | 30.00th=[ 37487], 40.00th=[ 41681], 50.00th=[ 46400], 60.00th=[ 51643], | 70.00th=[ 57410], 80.00th=[ 62653], 90.00th=[ 74974], 95.00th=[ 88605], | 99.00th=[122160], 99.50th=[133694], 99.90th=[154141], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 7, max= 94, per=25.25%, avg=44.95, stdev=14.94, samples=256 iops : min= 1, max= 23, avg=10.57, stdev= 3.77, samples=256 write: IOPS=11, BW=47.6KiB/s (48.7kB/s)(6348KiB/133472msec); 0 zone resets clat (usec): min=1155, max=2948.0k, avg=37320.85, stdev=77104.19 lat (usec): min=1155, max=2948.0k, avg=37321.49, stdev=77104.23 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 19], 40.00th=[ 25], 50.00th=[ 29], 60.00th=[ 35], | 70.00th=[ 44], 80.00th=[ 56], 90.00th=[ 72], 95.00th=[ 87], | 99.00th=[ 108], 99.50th=[ 117], 99.90th=[ 153], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 103, per=25.56%, avg=47.79, stdev=18.51, samples=257 iops : min= 1, max= 25, avg=11.28, stdev= 4.69, samples=257 lat (usec) : 750=0.03% lat (msec) : 2=0.59%, 4=0.10%, 10=4.72%, 20=12.37%, 50=48.14% lat (msec) : 100=31.58%, 250=2.44%, >=2000=0.03% cpu : usr=0.02%, sys=28.68%, ctx=5443, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530462: Tue Apr 1 10:58:32 2025 read: IOPS=11, BW=45.2KiB/s (46.3kB/s)(6028KiB/133425msec) clat (usec): min=673, max=183703, avg=48203.30, stdev=20205.13 lat (usec): min=674, max=183703, avg=48203.96, stdev=20205.72 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 22676], 10.00th=[ 26870], 20.00th=[ 32900], | 30.00th=[ 36963], 40.00th=[ 41157], 50.00th=[ 45351], 60.00th=[ 50070], | 70.00th=[ 54789], 80.00th=[ 61604], 90.00th=[ 72877], 95.00th=[ 84411], | 99.00th=[112722], 99.50th=[125305], 99.90th=[152044], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 15, max= 94, per=25.83%, avg=45.98, stdev=13.97, samples=255 iops : min= 3, max= 23, avg=10.81, stdev= 3.52, samples=255 write: IOPS=11, BW=46.9KiB/s (48.0kB/s)(6260KiB/133425msec); 0 zone resets clat (usec): min=1072, max=2994.6k, avg=38401.57, stdev=78921.47 lat (usec): min=1073, max=2994.6k, avg=38402.10, stdev=78921.49 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 20], 40.00th=[ 25], 50.00th=[ 30], 60.00th=[ 37], | 70.00th=[ 46], 80.00th=[ 58], 90.00th=[ 74], 95.00th=[ 88], | 99.00th=[ 109], 99.50th=[ 126], 99.90th=[ 157], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 125, per=25.48%, avg=47.64, stdev=17.66, samples=255 iops : min= 1, max= 31, avg=11.25, stdev= 4.49, samples=255 lat (usec) : 750=0.03%, 1000=0.03% lat (msec) : 2=0.72%, 4=0.29%, 10=3.55%, 20=12.63%, 50=50.03% lat (msec) : 100=30.27%, 250=2.41%, >=2000=0.03% cpu : usr=0.02%, sys=28.63%, ctx=5455, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530463: Tue Apr 1 10:58:32 2025 read: IOPS=11, BW=45.4KiB/s (46.5kB/s)(6052KiB/133264msec) clat (usec): min=673, max=144244, avg=48315.82, stdev=19416.69 lat (usec): min=673, max=144245, avg=48316.25, stdev=19416.70 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 23462], 10.00th=[ 27395], 20.00th=[ 32900], | 30.00th=[ 37487], 40.00th=[ 41157], 50.00th=[ 45876], 60.00th=[ 51119], | 70.00th=[ 55313], 80.00th=[ 62129], 90.00th=[ 71828], 95.00th=[ 82314], | 99.00th=[113771], 99.50th=[124257], 99.90th=[137364], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 15, max= 94, per=26.04%, avg=46.35, stdev=14.87, samples=254 iops : min= 3, max= 23, avg=10.92, stdev= 3.76, samples=254 write: IOPS=11, BW=46.8KiB/s (47.9kB/s)(6236KiB/133264msec); 0 zone resets clat (usec): min=1527, max=2959.1k, avg=38158.41, stdev=78051.55 lat (usec): min=1528, max=2959.1k, avg=38158.93, stdev=78051.55 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 20], 40.00th=[ 25], 50.00th=[ 30], 60.00th=[ 36], | 70.00th=[ 45], 80.00th=[ 58], 90.00th=[ 73], 95.00th=[ 87], | 99.00th=[ 109], 99.50th=[ 117], 99.90th=[ 148], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 118, per=25.24%, avg=47.19, stdev=17.51, samples=257 iops : min= 1, max= 29, avg=11.14, stdev= 4.43, samples=257 lat (usec) : 750=0.07%, 1000=0.03% lat (msec) : 2=0.59%, 4=0.16%, 10=4.33%, 20=11.98%, 50=48.86% lat (msec) : 100=32.03%, 250=1.92%, >=2000=0.03% cpu : usr=0.02%, sys=28.74%, ctx=5442, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=179KiB/s (183kB/s), 44.5KiB/s-45.4KiB/s (45.6kB/s-46.5kB/s), io=23.4MiB (24.6MB), run=133264-134157msec WRITE: bw=187KiB/s (192kB/s), 46.8KiB/s-47.6KiB/s (47.9kB/s-48.7kB/s), io=24.6MiB (25.7MB), run=133264-134157msec rand-rw: (groupid=0, jobs=1): err= 0: pid=530464: Tue Apr 1 10:58:44 2025 read: IOPS=10, BW=41.2KiB/s (42.1kB/s)(5992KiB/145575msec) clat (msec): min=4, max=3062, avg=44.48, stdev=84.30 lat (msec): min=4, max=3062, avg=44.48, stdev=84.30 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 19], 40.00th=[ 29], 50.00th=[ 35], 60.00th=[ 43], | 70.00th=[ 53], 80.00th=[ 70], 90.00th=[ 91], 95.00th=[ 106], | 99.00th=[ 140], 99.50th=[ 153], 99.90th=[ 169], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 224, per=25.55%, avg=41.91, stdev=38.56, samples=276 iops : min= 1, max= 56, avg= 9.78, stdev= 9.71, samples=276 write: IOPS=10, BW=43.2KiB/s (44.3kB/s)(6296KiB/145575msec); 0 zone resets clat (msec): min=5, max=202, avg=49.78, stdev=36.47 lat (msec): min=5, max=202, avg=49.78, stdev=36.47 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 19], 40.00th=[ 34], 50.00th=[ 42], 60.00th=[ 53], | 70.00th=[ 65], 80.00th=[ 81], 90.00th=[ 104], 95.00th=[ 121], | 99.00th=[ 153], 99.50th=[ 161], 99.90th=[ 186], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 7, max= 240, per=25.39%, avg=43.68, stdev=39.20, samples=278 iops : min= 1, max= 60, avg=10.24, stdev= 9.88, samples=278 lat (msec) : 10=7.65%, 20=23.05%, 50=31.61%, 100=28.84%, 250=8.82% lat (msec) : >=2000=0.03% cpu : usr=0.02%, sys=15.78%, ctx=3775, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530465: Tue Apr 1 10:58:44 2025 read: IOPS=10, BW=40.9KiB/s (41.9kB/s)(5940KiB/145300msec) clat (msec): min=4, max=2966, avg=43.88, stdev=82.35 lat (msec): min=4, max=2966, avg=43.88, stdev=82.35 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 17], 40.00th=[ 28], 50.00th=[ 34], 60.00th=[ 41], | 70.00th=[ 53], 80.00th=[ 68], 90.00th=[ 92], 95.00th=[ 107], | 99.00th=[ 140], 99.50th=[ 153], 99.90th=[ 167], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 224, per=25.46%, avg=41.76, stdev=38.60, samples=277 iops : min= 1, max= 56, avg= 9.75, stdev= 9.72, samples=277 write: IOPS=10, BW=43.7KiB/s (44.7kB/s)(6348KiB/145300msec); 0 zone resets clat (msec): min=6, max=215, avg=50.14, stdev=35.71 lat (msec): min=6, max=215, avg=50.14, stdev=35.71 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 23], 40.00th=[ 34], 50.00th=[ 43], 60.00th=[ 56], | 70.00th=[ 67], 80.00th=[ 80], 90.00th=[ 101], 95.00th=[ 117], | 99.00th=[ 155], 99.50th=[ 169], 99.90th=[ 197], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 7, max= 246, per=25.78%, avg=44.34, stdev=38.72, samples=279 iops : min= 1, max= 61, avg=10.40, stdev= 9.76, samples=279 lat (msec) : 10=6.87%, 20=22.92%, 50=32.36%, 100=29.30%, 250=8.53% lat (msec) : >=2000=0.03% cpu : usr=0.02%, sys=15.84%, ctx=3758, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530466: Tue Apr 1 10:58:44 2025 read: IOPS=10, BW=41.5KiB/s (42.5kB/s)(6028KiB/145365msec) clat (msec): min=5, max=166, avg=42.58, stdev=32.74 lat (msec): min=5, max=166, avg=42.58, stdev=32.74 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 19], 40.00th=[ 28], 50.00th=[ 35], 60.00th=[ 42], | 70.00th=[ 52], 80.00th=[ 70], 90.00th=[ 94], 95.00th=[ 109], | 99.00th=[ 134], 99.50th=[ 146], 99.90th=[ 161], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 7, max= 232, per=25.85%, avg=42.39, stdev=39.32, samples=276 iops : min= 1, max= 58, avg= 9.91, stdev= 9.91, samples=276 write: IOPS=10, BW=43.1KiB/s (44.1kB/s)(6260KiB/145365msec); 0 zone resets clat (msec): min=6, max=2970, avg=51.53, stdev=82.22 lat (msec): min=6, max=2970, avg=51.53, stdev=82.22 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 22], 40.00th=[ 33], 50.00th=[ 41], 60.00th=[ 53], | 70.00th=[ 65], 80.00th=[ 82], 90.00th=[ 102], 95.00th=[ 117], | 99.00th=[ 153], 99.50th=[ 180], 99.90th=[ 203], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 224, per=25.24%, avg=43.41, stdev=37.75, samples=280 iops : min= 1, max= 56, avg=10.17, stdev= 9.51, samples=280 lat (msec) : 10=6.97%, 20=23.40%, 50=33.11%, 100=27.18%, 250=9.31% lat (msec) : >=2000=0.03% cpu : usr=0.02%, sys=15.77%, ctx=3713, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530467: Tue Apr 1 10:58:44 2025 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(6052KiB/145552msec) clat (msec): min=4, max=165, avg=42.81, stdev=32.05 lat (msec): min=4, max=165, avg=42.81, stdev=32.05 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 13], | 30.00th=[ 19], 40.00th=[ 29], 50.00th=[ 36], 60.00th=[ 43], | 70.00th=[ 54], 80.00th=[ 69], 90.00th=[ 93], 95.00th=[ 109], | 99.00th=[ 134], 99.50th=[ 142], 99.90th=[ 163], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 7, max= 224, per=25.88%, avg=42.44, stdev=37.64, samples=275 iops : min= 1, max= 56, avg= 9.92, stdev= 9.48, samples=275 write: IOPS=10, BW=42.8KiB/s (43.9kB/s)(6236KiB/145552msec); 0 zone resets clat (msec): min=5, max=2991, avg=51.43, stdev=82.54 lat (msec): min=5, max=2991, avg=51.44, stdev=82.54 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 22], 40.00th=[ 34], 50.00th=[ 43], 60.00th=[ 54], | 70.00th=[ 66], 80.00th=[ 81], 90.00th=[ 101], 95.00th=[ 116], | 99.00th=[ 157], 99.50th=[ 176], 99.90th=[ 194], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 224, per=25.19%, avg=43.33, stdev=38.65, samples=278 iops : min= 1, max= 56, avg=10.15, stdev= 9.73, samples=278 lat (msec) : 10=7.13%, 20=22.40%, 50=32.52%, 100=29.39%, 250=8.53% lat (msec) : >=2000=0.03% cpu : usr=0.02%, sys=15.66%, ctx=3742, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=165KiB/s (169kB/s), 40.9KiB/s-41.6KiB/s (41.9kB/s-42.6kB/s), io=23.4MiB (24.6MB), run=145300-145575msec WRITE: bw=173KiB/s (177kB/s), 42.8KiB/s-43.7KiB/s (43.9kB/s-44.7kB/s), io=24.6MiB (25.7MB), run=145300-145575msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=530506: Tue Apr 1 10:59:23 2025 read: IOPS=16, BW=258KiB/s (264kB/s)(5920KiB/22964msec) clat (usec): min=210, max=134792, avg=32627.35, stdev=16444.27 lat (usec): min=210, max=134793, avg=32627.88, stdev=16444.24 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 13698], 10.00th=[ 17695], 20.00th=[ 20841], | 30.00th=[ 24249], 40.00th=[ 27657], 50.00th=[ 30802], 60.00th=[ 33817], | 70.00th=[ 37487], 80.00th=[ 41681], 90.00th=[ 50070], 95.00th=[ 63177], | 99.00th=[ 89654], 99.50th=[103285], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 125, max= 410, per=24.51%, avg=255.60, stdev=63.87, samples=45 iops : min= 7, max= 25, avg=15.18, stdev= 4.04, samples=45 write: IOPS=17, BW=277KiB/s (284kB/s)(6368KiB/22964msec); 0 zone resets clat (msec): min=3, max=100, avg=27.33, stdev=18.89 lat (msec): min=3, max=100, avg=27.33, stdev=18.89 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 16], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 27], | 70.00th=[ 34], 80.00th=[ 45], 90.00th=[ 56], 95.00th=[ 65], | 99.00th=[ 85], 99.50th=[ 92], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 126, max= 446, per=25.24%, avg=276.87, stdev=74.96, samples=45 iops : min= 7, max= 27, avg=16.53, stdev= 4.74, samples=45 lat (usec) : 250=0.13%, 500=0.13%, 750=1.30%, 1000=0.52% lat (msec) : 4=0.13%, 10=11.98%, 20=17.19%, 50=55.86%, 100=12.37% lat (msec) : 250=0.39% cpu : usr=0.02%, sys=31.02%, ctx=1315, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530507: Tue Apr 1 10:59:23 2025 read: IOPS=15, BW=254KiB/s (260kB/s)(5744KiB/22604msec) clat (usec): min=519, max=106160, avg=33490.15, stdev=15286.77 lat (usec): min=519, max=106161, avg=33490.62, stdev=15286.77 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 16188], 10.00th=[ 17957], 20.00th=[ 21627], | 30.00th=[ 25035], 40.00th=[ 28181], 50.00th=[ 30802], 60.00th=[ 34866], | 70.00th=[ 38536], 80.00th=[ 43254], 90.00th=[ 52167], 95.00th=[ 61080], | 99.00th=[ 87557], 99.50th=[ 99091], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 126, max= 407, per=24.20%, avg=252.36, stdev=63.66, samples=44 iops : min= 7, max= 25, avg=14.98, stdev= 4.03, samples=44 write: IOPS=18, BW=290KiB/s (296kB/s)(6544KiB/22604msec); 0 zone resets clat (usec): min=4473, max=97068, avg=25840.93, stdev=18506.39 lat (usec): min=4474, max=97069, avg=25841.65, stdev=18506.39 clat percentiles (usec): | 1.00th=[ 7177], 5.00th=[ 7963], 10.00th=[ 8291], 20.00th=[ 8848], | 30.00th=[13042], 40.00th=[17433], 50.00th=[20317], 60.00th=[23462], | 70.00th=[29754], 80.00th=[41157], 90.00th=[55313], 95.00th=[65274], | 99.00th=[85459], 99.50th=[86508], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 124, max= 574, per=26.26%, avg=288.11, stdev=101.69, samples=44 iops : min= 7, max= 35, avg=17.23, stdev= 6.43, samples=44 lat (usec) : 750=0.65%, 1000=0.52% lat (msec) : 10=12.63%, 20=19.01%, 50=54.82%, 100=12.24%, 250=0.13% cpu : usr=0.02%, sys=31.90%, ctx=1289, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530508: Tue Apr 1 10:59:23 2025 read: IOPS=17, BW=273KiB/s (280kB/s)(6256KiB/22920msec) clat (usec): min=536, max=87940, avg=32289.66, stdev=14278.79 lat (usec): min=537, max=87941, avg=32290.12, stdev=14278.79 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[10683], 10.00th=[17695], 20.00th=[21890], | 30.00th=[24511], 40.00th=[27919], 50.00th=[31327], 60.00th=[34866], | 70.00th=[38011], 80.00th=[42206], 90.00th=[48497], 95.00th=[56886], | 99.00th=[80217], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 93, max= 443, per=26.21%, avg=273.38, stdev=73.25, samples=45 iops : min= 5, max= 27, avg=16.29, stdev= 4.63, samples=45 write: IOPS=16, BW=263KiB/s (269kB/s)(6032KiB/22920msec); 0 zone resets clat (msec): min=5, max=105, avg=27.28, stdev=19.71 lat (msec): min=5, max=105, avg=27.28, stdev=19.71 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 13], 40.00th=[ 18], 50.00th=[ 22], 60.00th=[ 27], | 70.00th=[ 34], 80.00th=[ 45], 90.00th=[ 54], 95.00th=[ 69], | 99.00th=[ 83], 99.50th=[ 105], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 156, max= 412, per=23.83%, avg=261.42, stdev=70.02, samples=45 iops : min= 9, max= 25, avg=15.53, stdev= 4.40, samples=45 lat (usec) : 750=2.08%, 1000=0.26% lat (msec) : 10=12.11%, 20=15.49%, 50=58.98%, 100=10.68%, 250=0.39% cpu : usr=0.04%, sys=30.82%, ctx=1303, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530509: Tue Apr 1 10:59:23 2025 read: IOPS=16, BW=269KiB/s (275kB/s)(6032KiB/22425msec) clat (usec): min=385, max=109126, avg=31728.37, stdev=16153.37 lat (usec): min=385, max=109127, avg=31728.83, stdev=16153.38 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 742], 10.00th=[ 16057], 20.00th=[ 21365], | 30.00th=[ 25560], 40.00th=[ 27395], 50.00th=[ 29492], 60.00th=[ 33424], | 70.00th=[ 36439], 80.00th=[ 41157], 90.00th=[ 47449], 95.00th=[ 60556], | 99.00th=[ 89654], 99.50th=[100140], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min= 155, max= 383, per=25.64%, avg=267.41, stdev=65.84, samples=44 iops : min= 9, max= 23, avg=15.93, stdev= 4.19, samples=44 write: IOPS=17, BW=279KiB/s (286kB/s)(6256KiB/22425msec); 0 zone resets clat (msec): min=6, max=123, avg=26.74, stdev=18.74 lat (msec): min=6, max=123, avg=26.74, stdev=18.74 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 15], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 25], | 70.00th=[ 32], 80.00th=[ 44], 90.00th=[ 56], 95.00th=[ 64], | 99.00th=[ 80], 99.50th=[ 83], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 93, max= 412, per=25.47%, avg=279.43, stdev=77.47, samples=44 iops : min= 5, max= 25, avg=16.68, stdev= 4.92, samples=44 lat (usec) : 500=0.26%, 750=2.21%, 1000=0.65% lat (msec) : 10=12.24%, 20=16.28%, 50=56.38%, 100=11.59%, 250=0.39% cpu : usr=0.02%, sys=31.48%, ctx=1295, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1043KiB/s (1068kB/s), 254KiB/s-273KiB/s (260kB/s-280kB/s), io=23.4MiB (24.5MB), run=22425-22964msec WRITE: bw=1097KiB/s (1124kB/s), 263KiB/s-290KiB/s (269kB/s-296kB/s), io=24.6MiB (25.8MB), run=22425-22964msec rand-rw: (groupid=0, jobs=1): err= 0: pid=530501: Tue Apr 1 10:59:28 2025 read: IOPS=13, BW=208KiB/s (213kB/s)(5920KiB/28403msec) clat (msec): min=9, max=108, avg=34.22, stdev=21.41 lat (msec): min=9, max=108, avg=34.22, stdev=21.41 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 17], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 28], 60.00th=[ 32], | 70.00th=[ 40], 80.00th=[ 51], 90.00th=[ 69], 95.00th=[ 82], | 99.00th=[ 95], 99.50th=[ 108], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 62, max= 544, per=24.67%, avg=206.23, stdev=105.33, samples=56 iops : min= 3, max= 34, avg=12.12, stdev= 6.72, samples=56 write: IOPS=14, BW=224KiB/s (230kB/s)(6368KiB/28403msec); 0 zone resets clat (msec): min=10, max=132, avg=39.49, stdev=23.85 lat (msec): min=10, max=132, avg=39.49, stdev=23.85 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 23], 40.00th=[ 26], 50.00th=[ 32], 60.00th=[ 41], | 70.00th=[ 50], 80.00th=[ 58], 90.00th=[ 72], 95.00th=[ 93], | 99.00th=[ 112], 99.50th=[ 113], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 31, max= 480, per=25.33%, avg=222.68, stdev=94.46, samples=56 iops : min= 1, max= 30, avg=13.16, stdev= 5.98, samples=56 lat (msec) : 10=0.52%, 20=24.74%, 50=49.35%, 100=23.57%, 250=1.82% cpu : usr=0.02%, sys=14.45%, ctx=924, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530502: Tue Apr 1 10:59:28 2025 read: IOPS=12, BW=201KiB/s (205kB/s)(5744KiB/28638msec) clat (msec): min=6, max=127, avg=34.44, stdev=22.80 lat (msec): min=6, max=127, avg=34.44, stdev=22.80 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 23], 50.00th=[ 27], 60.00th=[ 32], | 70.00th=[ 38], 80.00th=[ 53], 90.00th=[ 69], 95.00th=[ 83], | 99.00th=[ 99], 99.50th=[ 113], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 63, max= 480, per=23.67%, avg=197.91, stdev=99.37, samples=56 iops : min= 3, max= 30, avg=11.61, stdev= 6.31, samples=56 write: IOPS=14, BW=229KiB/s (234kB/s)(6544KiB/28638msec); 0 zone resets clat (msec): min=7, max=156, avg=39.62, stdev=26.19 lat (msec): min=7, max=156, avg=39.62, stdev=26.19 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 25], 50.00th=[ 30], 60.00th=[ 37], | 70.00th=[ 48], 80.00th=[ 62], 90.00th=[ 79], 95.00th=[ 91], | 99.00th=[ 127], 99.50th=[ 130], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 94, max= 510, per=25.13%, avg=220.91, stdev=107.22, samples=56 iops : min= 5, max= 31, avg=13.05, stdev= 6.80, samples=56 lat (msec) : 10=1.17%, 20=26.82%, 50=47.40%, 100=22.66%, 250=1.95% cpu : usr=0.00%, sys=14.20%, ctx=932, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530503: Tue Apr 1 10:59:28 2025 read: IOPS=13, BW=219KiB/s (224kB/s)(6256KiB/28611msec) clat (msec): min=6, max=111, avg=34.80, stdev=22.70 lat (msec): min=6, max=111, avg=34.80, stdev=22.70 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 24], 50.00th=[ 28], 60.00th=[ 32], | 70.00th=[ 41], 80.00th=[ 51], 90.00th=[ 71], 95.00th=[ 83], | 99.00th=[ 107], 99.50th=[ 108], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 63, max= 603, per=25.76%, avg=215.34, stdev=111.25, samples=56 iops : min= 3, max= 37, avg=12.70, stdev= 7.06, samples=56 write: IOPS=13, BW=211KiB/s (216kB/s)(6032KiB/28611msec); 0 zone resets clat (msec): min=7, max=133, avg=39.66, stdev=25.97 lat (msec): min=7, max=133, avg=39.66, stdev=25.97 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 22], 40.00th=[ 25], 50.00th=[ 31], 60.00th=[ 39], | 70.00th=[ 51], 80.00th=[ 62], 90.00th=[ 78], 95.00th=[ 94], | 99.00th=[ 122], 99.50th=[ 130], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 63, max= 480, per=23.20%, avg=203.96, stdev=106.75, samples=56 iops : min= 3, max= 30, avg=11.98, stdev= 6.77, samples=56 lat (msec) : 10=1.17%, 20=28.78%, 50=44.79%, 100=23.05%, 250=2.21% cpu : usr=0.02%, sys=14.42%, ctx=950, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530505: Tue Apr 1 10:59:28 2025 read: IOPS=13, BW=213KiB/s (218kB/s)(6032KiB/28294msec) clat (msec): min=6, max=112, avg=34.46, stdev=21.93 lat (msec): min=6, max=112, avg=34.46, stdev=21.93 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 27], 60.00th=[ 31], | 70.00th=[ 39], 80.00th=[ 53], 90.00th=[ 71], 95.00th=[ 83], | 99.00th=[ 99], 99.50th=[ 107], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 94, max= 480, per=25.45%, avg=212.73, stdev=95.62, samples=56 iops : min= 5, max= 30, avg=12.54, stdev= 6.09, samples=56 write: IOPS=13, BW=221KiB/s (226kB/s)(6256KiB/28294msec); 0 zone resets clat (msec): min=10, max=126, avg=38.93, stdev=23.51 lat (msec): min=10, max=126, avg=38.93, stdev=23.51 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 21], | 30.00th=[ 23], 40.00th=[ 27], 50.00th=[ 31], 60.00th=[ 38], | 70.00th=[ 50], 80.00th=[ 58], 90.00th=[ 72], 95.00th=[ 86], | 99.00th=[ 123], 99.50th=[ 125], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 94, max= 480, per=25.16%, avg=221.14, stdev=92.40, samples=56 iops : min= 5, max= 30, avg=13.07, stdev= 5.86, samples=56 lat (msec) : 10=0.52%, 20=24.35%, 50=50.39%, 100=23.31%, 250=1.43% cpu : usr=0.00%, sys=14.68%, ctx=918, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=836KiB/s (856kB/s), 201KiB/s-219KiB/s (205kB/s-224kB/s), io=23.4MiB (24.5MB), run=28294-28638msec WRITE: bw=880KiB/s (901kB/s), 211KiB/s-229KiB/s (216kB/s-234kB/s), io=24.6MiB (25.8MB), run=28294-28638msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=530525: Tue Apr 1 10:59:51 2025 read: IOPS=2, BW=2066KiB/s (2116kB/s)(7168KiB/3469msec) clat (msec): min=226, max=341, avg=280.90, stdev=42.42 lat (msec): min=226, max=341, avg=280.90, stdev=42.42 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 236], | 30.00th=[ 262], 40.00th=[ 262], 50.00th=[ 288], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 321], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 1805, max= 3696, per=77.06%, avg=2670.25, stdev=908.29, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 1.15, samples=4 write: IOPS=1, BW=1476KiB/s (1511kB/s)(5120KiB/3469msec); 0 zone resets clat (msec): min=285, max=305, avg=294.80, stdev= 8.41 lat (msec): min=285, max=305, avg=294.83, stdev= 8.42 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 1805, max= 3984, per=50.60%, avg=2454.75, stdev=1024.30, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 lat (msec) : 250=16.67%, 500=83.33% cpu : usr=0.00%, sys=64.79%, ctx=499, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530526: Tue Apr 1 10:59:51 2025 read: IOPS=0, BW=347KiB/s (355kB/s)(2048KiB/5910msec) clat (msec): min=314, max=449, avg=382.08, stdev=94.98 lat (msec): min=314, max=449, avg=382.08, stdev=94.98 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 1622, max= 2019, per=52.54%, avg=1820.50, stdev=280.72, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1733KiB/s (1774kB/s)(10.0MiB/5910msec); 0 zone resets clat (msec): min=230, max=858, avg=504.11, stdev=215.71 lat (msec): min=230, max=858, avg=504.16, stdev=215.70 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 422], | 70.00th=[ 550], 80.00th=[ 760], 90.00th=[ 760], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 1992, max= 3696, per=45.89%, avg=2226.00, stdev=594.28, samples=8 iops : min= 1, max= 3, avg= 1.38, stdev= 0.74, samples=8 lat (msec) : 250=8.33%, 500=58.33%, 750=8.33%, 1000=25.00% cpu : usr=0.00%, sys=42.60%, ctx=115, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530527: Tue Apr 1 10:59:51 2025 read: IOPS=1, BW=1187KiB/s (1216kB/s)(6144KiB/5176msec) clat (msec): min=194, max=382, avg=275.72, stdev=65.05 lat (msec): min=194, max=382, avg=275.72, stdev=65.05 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 284], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 1996, max= 2019, per=58.02%, avg=2010.40, stdev= 9.15, samples=5 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=5 write: IOPS=1, BW=1187KiB/s (1216kB/s)(6144KiB/5176msec); 0 zone resets clat (msec): min=197, max=1592, avg=583.12, stdev=518.14 lat (msec): min=197, max=1593, avg=583.15, stdev=518.16 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 493], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 1992, max= 3330, per=48.12%, avg=2334.25, stdev=663.94, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 lat (msec) : 250=25.00%, 500=58.33%, 750=8.33%, 2000=8.33% cpu : usr=0.00%, sys=43.42%, ctx=762, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530528: Tue Apr 1 10:59:51 2025 read: IOPS=1, BW=1067KiB/s (1093kB/s)(5120KiB/4797msec) clat (usec): min=1202, max=381864, avg=198290.07, stdev=187005.51 lat (usec): min=1203, max=381864, avg=198290.44, stdev=187005.52 clat percentiles (usec): | 1.00th=[ 1205], 5.00th=[ 1205], 10.00th=[ 1205], 20.00th=[ 1205], | 30.00th=[ 1319], 40.00th=[ 1319], 50.00th=[246416], 60.00th=[246416], | 70.00th=[358613], 80.00th=[358613], 90.00th=[383779], 95.00th=[383779], | 99.00th=[383779], 99.50th=[383779], 99.90th=[383779], 99.95th=[383779], | 99.99th=[383779] bw ( KiB/s): min= 2007, max= 6059, per=100.00%, avg=4033.00, stdev=2865.20, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1494KiB/s (1530kB/s)(7168KiB/4797msec); 0 zone resets clat (msec): min=223, max=996, avg=535.00, stdev=311.08 lat (msec): min=223, max=996, avg=535.06, stdev=311.08 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 241], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 502], 60.00th=[ 667], | 70.00th=[ 667], 80.00th=[ 844], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 1654, max= 2015, per=39.33%, avg=1907.80, stdev=158.76, samples=5 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=5 lat (msec) : 2=16.67%, 250=25.00%, 500=25.00%, 750=16.67%, 1000=16.67% cpu : usr=0.00%, sys=38.05%, ctx=236, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3465KiB/s (3548kB/s), 347KiB/s-2066KiB/s (355kB/s-2116kB/s), io=20.0MiB (20.0MB), run=3469-5910msec WRITE: bw=4851KiB/s (4968kB/s), 1187KiB/s-1733KiB/s (1216kB/s-1774kB/s), io=28.0MiB (29.4MB), run=3469-5910msec rand-rw: (groupid=0, jobs=1): err= 0: pid=530521: Tue Apr 1 10:59:51 2025 read: IOPS=1, BW=1167KiB/s (1195kB/s)(7168KiB/6141msec) clat (msec): min=69, max=905, avg=505.46, stdev=339.48 lat (msec): min=69, max=905, avg=505.46, stdev=339.48 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 116], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 634], 60.00th=[ 751], | 70.00th=[ 751], 80.00th=[ 768], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 1537, max= 3849, per=67.77%, avg=2259.40, stdev=909.56, samples=5 iops : min= 1, max= 3, avg= 1.40, stdev= 0.89, samples=5 write: IOPS=0, BW=834KiB/s (854kB/s)(5120KiB/6141msec); 0 zone resets clat (msec): min=143, max=1277, avg=484.88, stdev=451.40 lat (msec): min=143, max=1277, avg=484.93, stdev=451.40 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 2043, max= 4087, per=58.36%, avg=2724.33, stdev=1180.10, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (msec) : 100=8.33%, 250=16.67%, 500=33.33%, 750=16.67%, 1000=16.67% lat (msec) : 2000=8.33% cpu : usr=0.00%, sys=17.21%, ctx=81, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530522: Tue Apr 1 10:59:51 2025 read: IOPS=0, BW=334KiB/s (342kB/s)(2048KiB/6124msec) clat (msec): min=171, max=758, avg=464.97, stdev=414.58 lat (msec): min=171, max=758, avg=464.98, stdev=414.58 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 760], | 70.00th=[ 760], 80.00th=[ 760], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 2043, max= 2048, per=61.35%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1672KiB/s (1712kB/s)(10.0MiB/6124msec); 0 zone resets clat (msec): min=118, max=1804, avg=505.26, stdev=502.62 lat (msec): min=118, max=1804, avg=505.31, stdev=502.62 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 148], | 30.00th=[ 218], 40.00th=[ 257], 50.00th=[ 296], 60.00th=[ 338], | 70.00th=[ 447], 80.00th=[ 709], 90.00th=[ 718], 95.00th=[ 1804], | 99.00th=[ 1804], 99.50th=[ 1804], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 1924, max= 4087, per=69.53%, avg=3245.60, stdev=1152.90, samples=5 iops : min= 1, max= 3, avg= 2.20, stdev= 1.10, samples=5 lat (msec) : 250=33.33%, 500=33.33%, 750=16.67%, 1000=8.33%, 2000=8.33% cpu : usr=0.02%, sys=18.77%, ctx=177, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530523: Tue Apr 1 10:59:51 2025 read: IOPS=0, BW=1012KiB/s (1036kB/s)(6144KiB/6073msec) clat (msec): min=109, max=582, avg=294.31, stdev=200.47 lat (msec): min=109, max=582, avg=294.31, stdev=200.47 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 284], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 1924, max= 4087, per=75.71%, avg=2524.25, stdev=1043.34, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 write: IOPS=0, BW=1012KiB/s (1036kB/s)(6144KiB/6073msec); 0 zone resets clat (msec): min=471, max=1128, avg=700.41, stdev=253.66 lat (msec): min=471, max=1128, avg=700.47, stdev=253.66 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 472], 10.00th=[ 472], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 634], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 1903, max= 2048, per=42.68%, avg=1992.20, stdev=72.25, samples=5 iops : min= 1, max= 2, avg= 1.20, stdev= 0.45, samples=5 lat (msec) : 250=25.00%, 500=25.00%, 750=33.33%, 1000=8.33%, 2000=8.33% cpu : usr=0.00%, sys=16.09%, ctx=99, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530524: Tue Apr 1 10:59:51 2025 read: IOPS=0, BW=860KiB/s (881kB/s)(5120KiB/5953msec) clat (msec): min=76, max=977, avg=450.53, stdev=373.65 lat (msec): min=76, max=977, avg=450.53, stdev=373.65 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 1924, max= 4087, per=80.52%, avg=2684.67, stdev=1215.91, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1204KiB/s (1233kB/s)(7168KiB/5953msec); 0 zone resets clat (msec): min=81, max=968, avg=524.28, stdev=349.79 lat (msec): min=81, max=968, avg=524.34, stdev=349.78 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 140], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 447], 60.00th=[ 676], | 70.00th=[ 676], 80.00th=[ 919], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 1517, max= 2048, per=40.93%, avg=1910.80, stdev=228.54, samples=5 iops : min= 1, max= 2, avg= 1.20, stdev= 0.45, samples=5 lat (msec) : 100=16.67%, 250=16.67%, 500=16.67%, 750=25.00%, 1000=25.00% cpu : usr=0.02%, sys=16.52%, ctx=97, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3335KiB/s (3415kB/s), 334KiB/s-1167KiB/s (342kB/s-1195kB/s), io=20.0MiB (20.0MB), run=5953-6141msec WRITE: bw=4669KiB/s (4781kB/s), 834KiB/s-1672KiB/s (854kB/s-1712kB/s), io=28.0MiB (29.4MB), run=5953-6141msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=530536: Tue Apr 1 11:00:13 2025 read: IOPS=1, BW=4163KiB/s (4263kB/s)(12.0MiB/2952msec) clat (msec): min=637, max=1443, avg=979.65, stdev=416.33 lat (msec): min=637, max=1443, avg=979.65, stdev=416.33 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 1000=33.33%, 2000=33.33% cpu : usr=0.00%, sys=32.94%, ctx=1119, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530537: Tue Apr 1 11:00:13 2025 read: IOPS=0, BW=723KiB/s (741kB/s)(4096KiB/5663msec) clat (nsec): min=849798k, max=849798k, avg=849798227.00, stdev= 0.00 lat (nsec): min=849799k, max=849799k, avg=849798942.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 852], 5.00th=[ 852], 10.00th=[ 852], 20.00th=[ 852], | 30.00th=[ 852], 40.00th=[ 852], 50.00th=[ 852], 60.00th=[ 852], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1447KiB/s (1481kB/s)(8192KiB/5663msec); 0 zone resets clat (msec): min=2298, max=2483, avg=2390.57, stdev=130.77 lat (msec): min=2298, max=2483, avg=2390.73, stdev=130.71 clat percentiles (msec): | 1.00th=[ 2299], 5.00th=[ 2299], 10.00th=[ 2299], 20.00th=[ 2299], | 30.00th=[ 2299], 40.00th=[ 2299], 50.00th=[ 2299], 60.00th=[ 2500], | 70.00th=[ 2500], 80.00th=[ 2500], 90.00th=[ 2500], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 1000=33.33%, >=2000=66.67% cpu : usr=0.00%, sys=31.30%, ctx=133, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530538: Tue Apr 1 11:00:13 2025 write: IOPS=0, BW=2783KiB/s (2849kB/s)(12.0MiB/4416msec); 0 zone resets clat (msec): min=1106, max=1754, avg=1468.24, stdev=330.36 lat (msec): min=1107, max=1755, avg=1468.46, stdev=330.29 clat percentiles (msec): | 1.00th=[ 1099], 5.00th=[ 1099], 10.00th=[ 1099], 20.00th=[ 1099], | 30.00th=[ 1099], 40.00th=[ 1536], 50.00th=[ 1536], 60.00th=[ 1536], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7816, max= 8192, per=100.00%, avg=8004.00, stdev=265.87, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 2000=100.00% cpu : usr=0.00%, sys=50.94%, ctx=67, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530540: Tue Apr 1 11:00:13 2025 write: IOPS=0, BW=2718KiB/s (2783kB/s)(12.0MiB/4521msec); 0 zone resets clat (msec): min=971, max=2387, avg=1498.21, stdev=774.88 lat (msec): min=971, max=2388, avg=1498.43, stdev=774.87 clat percentiles (msec): | 1.00th=[ 969], 5.00th=[ 969], 10.00th=[ 969], 20.00th=[ 969], | 30.00th=[ 969], 40.00th=[ 1133], 50.00th=[ 1133], 60.00th=[ 1133], | 70.00th=[ 2400], 80.00th=[ 2400], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7816, max= 8192, per=100.00%, avg=8004.00, stdev=265.87, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 1000=33.33%, 2000=33.33%, >=2000=33.33% cpu : usr=0.04%, sys=52.90%, ctx=67, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2893KiB/s (2963kB/s), 723KiB/s-4163KiB/s (741kB/s-4263kB/s), io=16.0MiB (16.8MB), run=2952-5663msec WRITE: bw=5786KiB/s (5925kB/s), 1447KiB/s-2783KiB/s (1481kB/s-2849kB/s), io=32.0MiB (33.6MB), run=4416-5663msec rand-rw: (groupid=0, jobs=1): err= 0: pid=530541: Tue Apr 1 11:00:13 2025 read: IOPS=0, BW=2258KiB/s (2312kB/s)(12.0MiB/5443msec) clat (msec): min=1360, max=2398, avg=1762.77, stdev=557.10 lat (msec): min=1360, max=2398, avg=1762.77, stdev=557.10 clat percentiles (msec): | 1.00th=[ 1368], 5.00th=[ 1368], 10.00th=[ 1368], 20.00th=[ 1368], | 30.00th=[ 1368], 40.00th=[ 1536], 50.00th=[ 1536], 60.00th=[ 1536], | 70.00th=[ 2400], 80.00th=[ 2400], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7801, max= 8192, per=100.00%, avg=7996.50, stdev=276.48, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 2000=66.67%, >=2000=33.33% cpu : usr=0.00%, sys=14.59%, ctx=69, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530542: Tue Apr 1 11:00:13 2025 read: IOPS=0, BW=669KiB/s (685kB/s)(4096KiB/6119msec) clat (nsec): min=1799.2M, max=1799.2M, avg=1799155254.00, stdev= 0.00 lat (nsec): min=1799.2M, max=1799.2M, avg=1799156149.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1804], 5.00th=[ 1804], 10.00th=[ 1804], 20.00th=[ 1804], | 30.00th=[ 1804], 40.00th=[ 1804], 50.00th=[ 1804], 60.00th=[ 1804], | 70.00th=[ 1804], 80.00th=[ 1804], 90.00th=[ 1804], 95.00th=[ 1804], | 99.00th=[ 1804], 99.50th=[ 1804], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1339KiB/s (1371kB/s)(8192KiB/6119msec); 0 zone resets clat (msec): min=1112, max=3113, avg=2112.72, stdev=1415.19 lat (msec): min=1112, max=3113, avg=2112.95, stdev=1415.18 clat percentiles (msec): | 1.00th=[ 1116], 5.00th=[ 1116], 10.00th=[ 1116], 20.00th=[ 1116], | 30.00th=[ 1116], 40.00th=[ 1116], 50.00th=[ 1116], 60.00th=[ 3104], | 70.00th=[ 3104], 80.00th=[ 3104], 90.00th=[ 3104], 95.00th=[ 3104], | 99.00th=[ 3104], 99.50th=[ 3104], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 2000=66.67%, >=2000=33.33% cpu : usr=0.00%, sys=20.48%, ctx=174, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530543: Tue Apr 1 11:00:13 2025 write: IOPS=0, BW=2336KiB/s (2392kB/s)(12.0MiB/5260msec); 0 zone resets clat (msec): min=1051, max=2321, avg=1735.82, stdev=640.58 lat (msec): min=1051, max=2321, avg=1736.06, stdev=640.56 clat percentiles (msec): | 1.00th=[ 1053], 5.00th=[ 1053], 10.00th=[ 1053], 20.00th=[ 1053], | 30.00th=[ 1053], 40.00th=[ 1838], 50.00th=[ 1838], 60.00th=[ 1838], | 70.00th=[ 2333], 80.00th=[ 2333], 90.00th=[ 2333], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 2000=66.67%, >=2000=33.33% cpu : usr=0.02%, sys=9.81%, ctx=26, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=530544: Tue Apr 1 11:00:13 2025 write: IOPS=0, BW=2309KiB/s (2365kB/s)(12.0MiB/5321msec); 0 zone resets clat (msec): min=1119, max=2331, avg=1690.05, stdev=609.06 lat (msec): min=1119, max=2331, avg=1690.35, stdev=609.16 clat percentiles (msec): | 1.00th=[ 1116], 5.00th=[ 1116], 10.00th=[ 1116], 20.00th=[ 1116], | 30.00th=[ 1116], 40.00th=[ 1620], 50.00th=[ 1620], 60.00th=[ 1620], | 70.00th=[ 2333], 80.00th=[ 2333], 90.00th=[ 2333], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 2000=66.67%, >=2000=33.33% cpu : usr=0.00%, sys=19.59%, ctx=83, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2678KiB/s (2742kB/s), 669KiB/s-2258KiB/s (685kB/s-2312kB/s), io=16.0MiB (16.8MB), run=5443-6119msec WRITE: bw=5355KiB/s (5484kB/s), 1339KiB/s-2336KiB/s (1371kB/s-2392kB/s), io=32.0MiB (33.6MB), run=5260-6119msec