== sanity test 398b: DIO and buffer IO race ============== 02:54:06 (1747810446) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18479 s, 42.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13923: Wed May 21 02:54:22 2025 read: IOPS=106, BW=424KiB/s (435kB/s)(5992KiB/14118msec) clat (usec): min=50, max=176845, avg=4221.05, stdev=6134.08 lat (usec): min=51, max=176845, avg=4221.73, stdev=6134.08 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 87], 20.00th=[ 1729], | 30.00th=[ 2474], 40.00th=[ 2868], 50.00th=[ 3294], 60.00th=[ 3818], | 70.00th=[ 4752], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[ 10290], | 99.00th=[ 14615], 99.50th=[ 18744], 99.90th=[ 83362], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 280, max= 600, per=25.41%, avg=424.07, stdev=74.47, samples=28 iops : min= 70, max= 150, avg=105.93, stdev=18.58, samples=28 write: IOPS=111, BW=446KiB/s (457kB/s)(6296KiB/14118msec) clat (usec): min=474, max=57302, avg=4888.48, stdev=4896.80 lat (usec): min=475, max=57303, avg=4889.63, stdev=4896.79 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 603], 20.00th=[ 775], | 30.00th=[ 1020], 40.00th=[ 2278], 50.00th=[ 3752], 60.00th=[ 4883], | 70.00th=[ 6325], 80.00th=[ 8225], 90.00th=[11207], 95.00th=[14353], | 99.00th=[21103], 99.50th=[23725], 99.90th=[38011], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 328, max= 605, per=25.45%, avg=444.89, stdev=72.64, samples=28 iops : min= 82, max= 151, avg=111.14, stdev=18.16, samples=28 lat (usec) : 100=6.02%, 250=3.48%, 500=0.29%, 750=9.54%, 1000=5.50% lat (msec) : 2=5.05%, 4=27.41%, 10=33.33%, 20=8.53%, 50=0.72% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.21%, sys=21.69%, ctx=5879, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13925: Wed May 21 02:54:22 2025 read: IOPS=103, BW=416KiB/s (426kB/s)(5940KiB/14284msec) clat (usec): min=54, max=55401, avg=4084.99, stdev=3861.41 lat (usec): min=54, max=55402, avg=4085.84, stdev=3861.41 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 75], 10.00th=[ 92], 20.00th=[ 1876], | 30.00th=[ 2606], 40.00th=[ 2933], 50.00th=[ 3359], 60.00th=[ 3982], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[14353], 99.50th=[19792], 99.90th=[50594], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 248, max= 664, per=24.79%, avg=413.82, stdev=76.04, samples=28 iops : min= 62, max= 166, avg=103.32, stdev=19.06, samples=28 write: IOPS=111, BW=444KiB/s (455kB/s)(6348KiB/14284msec) clat (usec): min=438, max=196306, avg=5118.45, stdev=6839.34 lat (usec): min=439, max=196307, avg=5119.97, stdev=6838.95 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 619], 20.00th=[ 791], | 30.00th=[ 1074], 40.00th=[ 2507], 50.00th=[ 3916], 60.00th=[ 5014], | 70.00th=[ 6521], 80.00th=[ 8291], 90.00th=[ 11731], 95.00th=[ 14222], | 99.00th=[ 21627], 99.50th=[ 26346], 99.90th=[ 47449], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 288, max= 584, per=24.95%, avg=436.07, stdev=80.06, samples=28 iops : min= 72, max= 146, avg=108.89, stdev=20.02, samples=28 lat (usec) : 100=5.83%, 250=3.58%, 500=0.26%, 750=9.24%, 1000=5.53% lat (msec) : 2=4.88%, 4=25.85%, 10=35.55%, 20=8.46%, 50=0.72% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.18%, sys=21.78%, ctx=5844, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13927: Wed May 21 02:54:22 2025 read: IOPS=104, BW=419KiB/s (429kB/s)(6028KiB/14380msec) clat (usec): min=54, max=78216, avg=4144.95, stdev=3982.59 lat (usec): min=54, max=78217, avg=4145.86, stdev=3982.51 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 86], 20.00th=[ 1942], | 30.00th=[ 2540], 40.00th=[ 2900], 50.00th=[ 3326], 60.00th=[ 3916], | 70.00th=[ 5014], 80.00th=[ 6587], 90.00th=[ 8717], 95.00th=[ 9896], | 99.00th=[14353], 99.50th=[17433], 99.90th=[49546], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 184, max= 640, per=24.89%, avg=415.46, stdev=102.04, samples=28 iops : min= 46, max= 160, avg=103.71, stdev=25.51, samples=28 write: IOPS=108, BW=435KiB/s (446kB/s)(6260KiB/14380msec) clat (usec): min=471, max=217226, avg=5132.71, stdev=7308.56 lat (usec): min=472, max=217228, avg=5133.73, stdev=7308.66 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 627], 20.00th=[ 775], | 30.00th=[ 1045], 40.00th=[ 2638], 50.00th=[ 3949], 60.00th=[ 5145], | 70.00th=[ 6390], 80.00th=[ 8094], 90.00th=[ 11469], 95.00th=[ 13829], | 99.00th=[ 20055], 99.50th=[ 25822], 99.90th=[ 64226], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 240, max= 648, per=24.55%, avg=429.14, stdev=89.93, samples=28 iops : min= 60, max= 162, avg=107.14, stdev=22.52, samples=28 lat (usec) : 100=5.99%, 250=3.35%, 500=0.42%, 750=9.18%, 1000=5.37% lat (msec) : 2=4.52%, 4=26.76%, 10=34.90%, 20=8.82%, 50=0.59% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.23%, sys=22.23%, ctx=6014, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13929: Wed May 21 02:54:22 2025 read: IOPS=106, BW=426KiB/s (436kB/s)(6052KiB/14212msec) clat (usec): min=54, max=52414, avg=4140.37, stdev=3817.34 lat (usec): min=54, max=52415, avg=4141.04, stdev=3817.35 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 74], 10.00th=[ 88], 20.00th=[ 709], | 30.00th=[ 2474], 40.00th=[ 2868], 50.00th=[ 3359], 60.00th=[ 4015], | 70.00th=[ 5080], 80.00th=[ 6521], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[16057], 99.50th=[19792], 99.90th=[49021], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 151, max= 560, per=25.32%, avg=422.64, stdev=82.47, samples=28 iops : min= 37, max= 140, avg=105.50, stdev=20.71, samples=28 write: IOPS=109, BW=439KiB/s (449kB/s)(6236KiB/14212msec) clat (usec): min=481, max=217767, avg=5034.56, stdev=7283.98 lat (usec): min=481, max=217768, avg=5035.50, stdev=7283.92 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 562], 10.00th=[ 611], 20.00th=[ 775], | 30.00th=[ 1090], 40.00th=[ 2409], 50.00th=[ 3884], 60.00th=[ 5014], | 70.00th=[ 6390], 80.00th=[ 8094], 90.00th=[ 10945], 95.00th=[ 13698], | 99.00th=[ 21365], 99.50th=[ 23462], 99.90th=[ 62129], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 183, max= 648, per=24.93%, avg=435.75, stdev=93.02, samples=28 iops : min= 45, max= 162, avg=108.79, stdev=23.37, samples=28 lat (usec) : 100=5.99%, 250=3.61%, 500=0.42%, 750=9.37%, 1000=5.11% lat (msec) : 2=5.27%, 4=25.65%, 10=35.74%, 20=8.01%, 50=0.72% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.19%, sys=21.95%, ctx=5772, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1670KiB/s (1710kB/s), 416KiB/s-426KiB/s (426kB/s-436kB/s), io=23.4MiB (24.6MB), run=14118-14380msec WRITE: bw=1748KiB/s (1790kB/s), 435KiB/s-446KiB/s (446kB/s-457kB/s), io=24.6MiB (25.7MB), run=14118-14380msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13922: Wed May 21 02:54:46 2025 read: IOPS=40, BW=160KiB/s (164kB/s)(5992KiB/37405msec) clat (usec): min=1622, max=180755, avg=4326.53, stdev=5615.16 lat (usec): min=1623, max=180756, avg=4327.32, stdev=5615.17 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2638], 50.00th=[ 2933], 60.00th=[ 3523], | 70.00th=[ 4228], 80.00th=[ 5276], 90.00th=[ 8160], 95.00th=[ 11338], | 99.00th=[ 16319], 99.50th=[ 19792], 99.90th=[ 41681], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 56, max= 280, per=25.00%, avg=160.03, stdev=63.92, samples=74 iops : min= 14, max= 70, avg=39.95, stdev=15.97, samples=74 write: IOPS=42, BW=168KiB/s (172kB/s)(6296KiB/37405msec) clat (usec): min=9151, max=97159, avg=19584.15, stdev=8208.60 lat (usec): min=9152, max=97160, avg=19585.23, stdev=8208.51 clat percentiles (usec): | 1.00th=[10028], 5.00th=[11207], 10.00th=[12256], 20.00th=[13435], | 30.00th=[14746], 40.00th=[16057], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20841], 80.00th=[23462], 90.00th=[30016], 95.00th=[35390], | 99.00th=[49021], 99.50th=[54264], 99.90th=[88605], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 48, max= 240, per=24.98%, avg=167.59, stdev=51.81, samples=74 iops : min= 12, max= 60, avg=41.84, stdev=12.97, samples=74 lat (msec) : 2=3.22%, 4=29.04%, 10=13.70%, 20=35.94%, 50=17.61% lat (msec) : 100=0.46%, 250=0.03% cpu : usr=0.09%, sys=5.33%, ctx=3742, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13924: Wed May 21 02:54:46 2025 read: IOPS=39, BW=159KiB/s (163kB/s)(5940KiB/37304msec) clat (usec): min=1566, max=200358, avg=4454.38, stdev=6150.12 lat (usec): min=1566, max=200359, avg=4455.18, stdev=6150.17 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3621], | 70.00th=[ 4293], 80.00th=[ 5276], 90.00th=[ 8717], 95.00th=[ 11731], | 99.00th=[ 17433], 99.50th=[ 21103], 99.90th=[ 42206], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 16, max= 296, per=24.91%, avg=159.42, stdev=70.03, samples=74 iops : min= 4, max= 74, avg=39.82, stdev=17.53, samples=74 write: IOPS=42, BW=170KiB/s (174kB/s)(6348KiB/37304msec) clat (usec): min=8166, max=89358, avg=19275.73, stdev=7895.61 lat (usec): min=8167, max=89359, avg=19276.73, stdev=7895.62 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11207], 10.00th=[11863], 20.00th=[13304], | 30.00th=[14484], 40.00th=[15926], 50.00th=[17433], 60.00th=[19006], | 70.00th=[20579], 80.00th=[23462], 90.00th=[29754], 95.00th=[35390], | 99.00th=[46400], 99.50th=[50594], 99.90th=[68682], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 64, max= 240, per=25.31%, avg=169.80, stdev=52.70, samples=74 iops : min= 16, max= 60, avg=42.42, stdev=13.20, samples=74 lat (msec) : 2=3.26%, 4=28.39%, 10=13.54%, 20=37.17%, 50=17.35% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.12%, sys=5.35%, ctx=3749, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13926: Wed May 21 02:54:46 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(6028KiB/37464msec) clat (usec): min=1600, max=86477, avg=4561.91, stdev=4002.58 lat (usec): min=1600, max=86478, avg=4562.74, stdev=4002.61 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1991], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2507], 40.00th=[ 2737], 50.00th=[ 3261], 60.00th=[ 3916], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 9110], 95.00th=[11863], | 99.00th=[16909], 99.50th=[20579], 99.90th=[38011], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 48, max= 312, per=25.15%, avg=160.93, stdev=58.11, samples=74 iops : min= 12, max= 78, avg=40.20, stdev=14.55, samples=74 write: IOPS=41, BW=167KiB/s (171kB/s)(6260KiB/37464msec) clat (msec): min=8, max=214, avg=19.48, stdev= 9.57 lat (msec): min=8, max=214, avg=19.48, stdev= 9.57 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 24], 90.00th=[ 31], 95.00th=[ 36], | 99.00th=[ 49], 99.50th=[ 54], 99.90th=[ 107], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 48, max= 240, per=24.81%, avg=166.45, stdev=54.69, samples=74 iops : min= 12, max= 60, avg=41.58, stdev=13.69, samples=74 lat (msec) : 2=2.70%, 4=27.41%, 10=15.23%, 20=37.83%, 50=16.44% lat (msec) : 100=0.33%, 250=0.07% cpu : usr=0.09%, sys=5.41%, ctx=3796, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13928: Wed May 21 02:54:46 2025 read: IOPS=40, BW=162KiB/s (166kB/s)(6052KiB/37295msec) clat (usec): min=1564, max=53308, avg=4360.97, stdev=3732.63 lat (usec): min=1565, max=53309, avg=4361.73, stdev=3732.63 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2474], 40.00th=[ 2671], 50.00th=[ 2999], 60.00th=[ 3654], | 70.00th=[ 4293], 80.00th=[ 5211], 90.00th=[ 8717], 95.00th=[11469], | 99.00th=[18482], 99.50th=[23725], 99.90th=[34866], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 15, max= 320, per=25.28%, avg=161.80, stdev=65.61, samples=74 iops : min= 3, max= 80, avg=40.42, stdev=16.43, samples=74 write: IOPS=41, BW=167KiB/s (171kB/s)(6236KiB/37295msec) clat (msec): min=9, max=215, avg=19.63, stdev= 9.07 lat (msec): min=9, max=215, avg=19.63, stdev= 9.07 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 31], 95.00th=[ 35], | 99.00th=[ 45], 99.50th=[ 48], 99.90th=[ 73], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 56, max= 240, per=24.90%, avg=167.09, stdev=52.72, samples=74 iops : min= 14, max= 60, avg=41.74, stdev=13.20, samples=74 lat (msec) : 2=2.93%, 4=29.75%, 10=13.28%, 20=35.64%, 50=18.16% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.12%, sys=5.39%, ctx=3706, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=641KiB/s (656kB/s), 159KiB/s-162KiB/s (163kB/s-166kB/s), io=23.4MiB (24.6MB), run=37295-37464msec WRITE: bw=671KiB/s (687kB/s), 167KiB/s-170KiB/s (171kB/s-174kB/s), io=24.6MiB (25.7MB), run=37295-37464msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13968: Wed May 21 02:54:50 2025 read: IOPS=125, BW=2005KiB/s (2054kB/s)(5920KiB/2952msec) clat (usec): min=56, max=47878, avg=3784.41, stdev=3888.05 lat (usec): min=56, max=47879, avg=3785.10, stdev=3888.01 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 72], 10.00th=[ 83], 20.00th=[ 123], | 30.00th=[ 2671], 40.00th=[ 3163], 50.00th=[ 3589], 60.00th=[ 4146], | 70.00th=[ 4817], 80.00th=[ 5669], 90.00th=[ 7046], 95.00th=[ 8356], | 99.00th=[20317], 99.50th=[23987], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 1472, max= 2554, per=25.14%, avg=1950.80, stdev=396.11, samples=5 iops : min= 92, max= 159, avg=121.80, stdev=24.52, samples=5 write: IOPS=134, BW=2157KiB/s (2209kB/s)(6368KiB/2952msec) clat (usec): min=663, max=26996, avg=3769.97, stdev=4168.89 lat (usec): min=663, max=26997, avg=3771.26, stdev=4168.98 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 791], 10.00th=[ 881], 20.00th=[ 1012], | 30.00th=[ 1172], 40.00th=[ 1385], 50.00th=[ 1991], 60.00th=[ 3130], | 70.00th=[ 4047], 80.00th=[ 5866], 90.00th=[ 8848], 95.00th=[12256], | 99.00th=[25560], 99.50th=[26084], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 1760, max= 2496, per=26.41%, avg=2156.00, stdev=306.44, samples=5 iops : min= 110, max= 156, avg=134.60, stdev=19.28, samples=5 lat (usec) : 100=6.77%, 250=5.73%, 500=0.52%, 750=1.56%, 1000=8.59% lat (msec) : 2=16.15%, 4=24.48%, 10=31.77%, 20=3.26%, 50=1.17% cpu : usr=0.07%, sys=27.21%, ctx=1580, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13969: Wed May 21 02:54:50 2025 read: IOPS=123, BW=1977KiB/s (2025kB/s)(5744KiB/2905msec) clat (usec): min=60, max=56133, avg=4023.25, stdev=4704.36 lat (usec): min=60, max=56134, avg=4023.90, stdev=4704.36 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 73], 10.00th=[ 83], 20.00th=[ 120], | 30.00th=[ 2737], 40.00th=[ 3130], 50.00th=[ 3556], 60.00th=[ 3982], | 70.00th=[ 5014], 80.00th=[ 5997], 90.00th=[ 7373], 95.00th=[ 9110], | 99.00th=[13960], 99.50th=[49546], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 1792, max= 2240, per=25.80%, avg=2002.20, stdev=209.36, samples=5 iops : min= 112, max= 140, avg=125.00, stdev=12.92, samples=5 write: IOPS=140, BW=2253KiB/s (2307kB/s)(6544KiB/2905msec) clat (usec): min=649, max=20693, avg=3535.31, stdev=3373.59 lat (usec): min=650, max=20694, avg=3537.63, stdev=3373.15 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 807], 10.00th=[ 889], 20.00th=[ 996], | 30.00th=[ 1188], 40.00th=[ 1467], 50.00th=[ 2024], 60.00th=[ 3163], | 70.00th=[ 4178], 80.00th=[ 5735], 90.00th=[ 8160], 95.00th=[10159], | 99.00th=[16188], 99.50th=[17695], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 1984, max= 2746, per=27.73%, avg=2264.40, stdev=292.03, samples=5 iops : min= 124, max= 171, avg=141.40, stdev=17.99, samples=5 lat (usec) : 100=6.77%, 250=4.43%, 500=0.13%, 750=1.04%, 1000=9.90% lat (msec) : 2=16.02%, 4=26.56%, 10=29.95%, 20=4.69%, 50=0.39% lat (msec) : 100=0.13% cpu : usr=0.28%, sys=28.17%, ctx=1643, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13970: Wed May 21 02:54:50 2025 read: IOPS=126, BW=2027KiB/s (2076kB/s)(6256KiB/3086msec) clat (usec): min=55, max=26470, avg=3838.53, stdev=3066.25 lat (usec): min=56, max=26471, avg=3839.24, stdev=3066.35 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 74], 10.00th=[ 89], 20.00th=[ 127], | 30.00th=[ 2638], 40.00th=[ 3130], 50.00th=[ 3589], 60.00th=[ 4293], | 70.00th=[ 5014], 80.00th=[ 6194], 90.00th=[ 7504], 95.00th=[ 8979], | 99.00th=[12387], 99.50th=[13698], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 1472, max= 2560, per=25.97%, avg=2015.17, stdev=365.07, samples=6 iops : min= 92, max= 160, avg=125.83, stdev=22.77, samples=6 write: IOPS=122, BW=1955KiB/s (2002kB/s)(6032KiB/3086msec) clat (usec): min=646, max=38397, avg=3987.42, stdev=4478.43 lat (usec): min=647, max=38398, avg=3988.43, stdev=4478.45 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 791], 10.00th=[ 840], 20.00th=[ 996], | 30.00th=[ 1237], 40.00th=[ 1483], 50.00th=[ 2311], 60.00th=[ 3458], | 70.00th=[ 4686], 80.00th=[ 6456], 90.00th=[ 9372], 95.00th=[11469], | 99.00th=[23725], 99.50th=[32637], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 1664, max= 2112, per=23.05%, avg=1881.83, stdev=160.35, samples=6 iops : min= 104, max= 132, avg=117.50, stdev= 9.91, samples=6 lat (usec) : 100=6.77%, 250=5.60%, 500=0.26%, 750=1.82%, 1000=8.20% lat (msec) : 2=13.80%, 4=24.87%, 10=33.59%, 20=4.43%, 50=0.65% cpu : usr=0.10%, sys=26.90%, ctx=1621, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13971: Wed May 21 02:54:50 2025 read: IOPS=129, BW=2069KiB/s (2118kB/s)(6032KiB/2916msec) clat (usec): min=59, max=36731, avg=4023.42, stdev=4197.74 lat (usec): min=60, max=36732, avg=4024.12, stdev=4197.72 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 84], 20.00th=[ 117], | 30.00th=[ 2507], 40.00th=[ 3064], 50.00th=[ 3556], 60.00th=[ 4293], | 70.00th=[ 5145], 80.00th=[ 6194], 90.00th=[ 7570], 95.00th=[ 9110], | 99.00th=[20841], 99.50th=[35390], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 1728, max= 2522, per=25.96%, avg=2014.80, stdev=308.61, samples=5 iops : min= 108, max= 157, avg=125.80, stdev=19.03, samples=5 write: IOPS=134, BW=2145KiB/s (2197kB/s)(6256KiB/2916msec) clat (usec): min=660, max=28862, avg=3383.30, stdev=3286.66 lat (usec): min=661, max=28909, avg=3384.42, stdev=3287.52 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 807], 10.00th=[ 873], 20.00th=[ 1004], | 30.00th=[ 1156], 40.00th=[ 1352], 50.00th=[ 1745], 60.00th=[ 3163], | 70.00th=[ 4359], 80.00th=[ 5800], 90.00th=[ 7898], 95.00th=[ 8979], | 99.00th=[14615], 99.50th=[15795], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 1824, max= 2363, per=25.93%, avg=2117.40, stdev=257.19, samples=5 iops : min= 114, max= 147, avg=132.20, stdev=15.91, samples=5 lat (usec) : 100=7.29%, 250=5.21%, 500=0.65%, 750=1.17%, 1000=8.98% lat (msec) : 2=17.19%, 4=21.87%, 10=34.11%, 20=2.73%, 50=0.78% cpu : usr=0.07%, sys=27.31%, ctx=1615, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7762KiB/s (7948kB/s), 1977KiB/s-2069KiB/s (2025kB/s-2118kB/s), io=23.4MiB (24.5MB), run=2905-3086msec WRITE: bw=8166KiB/s (8362kB/s), 1955KiB/s-2253KiB/s (2002kB/s-2307kB/s), io=24.6MiB (25.8MB), run=2905-3086msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13972: Wed May 21 02:54:55 2025 read: IOPS=42, BW=675KiB/s (692kB/s)(5920KiB/8766msec) clat (usec): min=1838, max=23978, avg=4133.84, stdev=3564.12 lat (usec): min=1838, max=23979, avg=4134.59, stdev=3564.12 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2409], | 30.00th=[ 2540], 40.00th=[ 2671], 50.00th=[ 2868], 60.00th=[ 3032], | 70.00th=[ 3326], 80.00th=[ 4047], 90.00th=[ 8029], 95.00th=[12125], | 99.00th=[19792], 99.50th=[20841], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 288, max= 1184, per=24.87%, avg=660.71, stdev=244.74, samples=17 iops : min= 18, max= 74, avg=41.29, stdev=15.30, samples=17 write: IOPS=45, BW=726KiB/s (744kB/s)(6368KiB/8766msec) clat (usec): min=8676, max=52692, avg=18141.92, stdev=6905.56 lat (usec): min=8678, max=52693, avg=18143.34, stdev=6905.59 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[11076], 10.00th=[11731], 20.00th=[12649], | 30.00th=[13829], 40.00th=[15139], 50.00th=[16581], 60.00th=[18220], | 70.00th=[19530], 80.00th=[21890], 90.00th=[26870], 95.00th=[32637], | 99.00th=[45351], 99.50th=[48497], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 384, max= 960, per=25.85%, avg=722.82, stdev=219.40, samples=17 iops : min= 24, max= 60, avg=45.18, stdev=13.71, samples=17 lat (msec) : 2=0.52%, 4=37.63%, 10=7.68%, 20=39.19%, 50=14.84% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=6.46%, ctx=908, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13973: Wed May 21 02:54:55 2025 read: IOPS=39, BW=637KiB/s (653kB/s)(5744KiB/9012msec) clat (usec): min=1949, max=20949, avg=3933.44, stdev=3111.76 lat (usec): min=1950, max=20949, avg=3934.25, stdev=3111.73 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2737], 60.00th=[ 2933], | 70.00th=[ 3294], 80.00th=[ 4490], 90.00th=[ 7177], 95.00th=[11076], | 99.00th=[18482], 99.50th=[19792], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 288, max= 1024, per=24.01%, avg=637.94, stdev=229.05, samples=18 iops : min= 18, max= 64, avg=39.72, stdev=14.35, samples=18 write: IOPS=45, BW=726KiB/s (744kB/s)(6544KiB/9012msec) clat (usec): min=8277, max=65284, avg=18514.90, stdev=6961.55 lat (usec): min=8278, max=65286, avg=18516.38, stdev=6961.57 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10945], 10.00th=[11600], 20.00th=[12780], | 30.00th=[14222], 40.00th=[15926], 50.00th=[17433], 60.00th=[18482], | 70.00th=[20055], 80.00th=[22676], 90.00th=[27132], 95.00th=[32637], | 99.00th=[41157], 99.50th=[44303], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 384, max= 1024, per=25.93%, avg=725.00, stdev=206.38, samples=18 iops : min= 24, max= 64, avg=45.17, stdev=12.85, samples=18 lat (msec) : 2=0.39%, 4=36.07%, 10=8.46%, 20=38.41%, 50=16.41% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=6.41%, ctx=892, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13974: Wed May 21 02:54:55 2025 read: IOPS=45, BW=721KiB/s (738kB/s)(6256KiB/8675msec) clat (usec): min=1985, max=23325, avg=3886.17, stdev=3404.51 lat (usec): min=1986, max=23326, avg=3887.16, stdev=3404.41 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2606], 50.00th=[ 2737], 60.00th=[ 2868], | 70.00th=[ 3163], 80.00th=[ 3654], 90.00th=[ 6456], 95.00th=[11469], | 99.00th=[20317], 99.50th=[22152], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 224, max= 1376, per=27.34%, avg=726.29, stdev=324.77, samples=17 iops : min= 14, max= 86, avg=45.29, stdev=20.31, samples=17 write: IOPS=43, BW=695KiB/s (712kB/s)(6032KiB/8675msec) clat (usec): min=9078, max=70340, avg=18737.66, stdev=7154.54 lat (usec): min=9080, max=70342, avg=18739.16, stdev=7154.55 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11469], 10.00th=[12125], 20.00th=[13173], | 30.00th=[14353], 40.00th=[15926], 50.00th=[17433], 60.00th=[18744], | 70.00th=[19792], 80.00th=[22414], 90.00th=[27395], 95.00th=[32900], | 99.00th=[43254], 99.50th=[52691], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 320, max= 928, per=24.70%, avg=690.59, stdev=198.94, samples=17 iops : min= 20, max= 58, avg=43.06, stdev=12.46, samples=17 lat (msec) : 2=0.39%, 4=41.28%, 10=6.12%, 20=37.37%, 50=14.58% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=6.64%, ctx=903, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13975: Wed May 21 02:54:55 2025 read: IOPS=42, BW=677KiB/s (694kB/s)(6032KiB/8904msec) clat (usec): min=1964, max=46374, avg=4134.25, stdev=4185.86 lat (usec): min=1965, max=46375, avg=4135.02, stdev=4185.85 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2474], 40.00th=[ 2573], 50.00th=[ 2704], 60.00th=[ 2900], | 70.00th=[ 3195], 80.00th=[ 4047], 90.00th=[ 7898], 95.00th=[11731], | 99.00th=[23462], 99.50th=[24249], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 192, max= 1120, per=25.06%, avg=665.94, stdev=243.01, samples=17 iops : min= 12, max= 70, avg=41.47, stdev=15.19, samples=17 write: IOPS=43, BW=703KiB/s (719kB/s)(6256KiB/8904msec) clat (usec): min=10103, max=70405, avg=18553.67, stdev=7146.56 lat (usec): min=10105, max=70405, avg=18555.37, stdev=7146.64 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11076], 10.00th=[11863], 20.00th=[13173], | 30.00th=[14353], 40.00th=[15533], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19792], 80.00th=[22152], 90.00th=[27132], 95.00th=[32375], | 99.00th=[46400], 99.50th=[54264], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 352, max= 960, per=24.83%, avg=694.24, stdev=211.69, samples=17 iops : min= 22, max= 60, avg=43.24, stdev=13.28, samples=17 lat (msec) : 2=0.26%, 4=38.54%, 10=7.42%, 20=38.41%, 50=14.97% lat (msec) : 100=0.39% cpu : usr=0.10%, sys=6.56%, ctx=926, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2658KiB/s (2722kB/s), 637KiB/s-721KiB/s (653kB/s-738kB/s), io=23.4MiB (24.5MB), run=8675-9012msec WRITE: bw=2796KiB/s (2863kB/s), 695KiB/s-726KiB/s (712kB/s-744kB/s), io=24.6MiB (25.8MB), run=8675-9012msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14012: Wed May 21 02:54:58 2025 read: IOPS=4, BW=4262KiB/s (4364kB/s)(7168KiB/1682msec) clat (usec): min=508, max=635071, avg=186191.29, stdev=274024.20 lat (usec): min=509, max=635072, avg=186192.32, stdev=274024.21 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 29492], | 30.00th=[ 31065], 40.00th=[ 31065], 50.00th=[ 33817], 60.00th=[ 39584], | 70.00th=[ 39584], 80.00th=[534774], 90.00th=[633340], 95.00th=[633340], | 99.00th=[633340], 99.50th=[633340], 99.90th=[633340], 99.95th=[633340], | 99.99th=[633340] bw ( KiB/s): min=10240, max=10240, per=93.20%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=2, BW=3044KiB/s (3117kB/s)(5120KiB/1682msec) clat (msec): min=29, max=118, avg=71.52, stdev=34.60 lat (msec): min=29, max=118, avg=71.58, stdev=34.58 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 79], 60.00th=[ 79], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 8192, max= 8192, per=53.26%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 750=16.67% cpu : usr=0.00%, sys=15.23%, ctx=671, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14014: Wed May 21 02:54:58 2025 read: IOPS=1, BW=1226KiB/s (1256kB/s)(2048KiB/1670msec) clat (usec): min=838, max=489973, avg=245405.82, stdev=345870.80 lat (usec): min=838, max=489974, avg=245406.82, stdev=345871.30 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 840], 10.00th=[ 840], 20.00th=[ 840], | 30.00th=[ 840], 40.00th=[ 840], 50.00th=[ 840], 60.00th=[488637], | 70.00th=[488637], 80.00th=[488637], 90.00th=[488637], 95.00th=[488637], | 99.00th=[488637], 99.50th=[488637], 99.90th=[488637], 99.95th=[488637], | 99.99th=[488637] bw ( KiB/s): min= 2048, max= 2048, per=18.64%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=6132KiB/s (6279kB/s)(10.0MiB/1670msec) clat (msec): min=28, max=636, avg=116.75, stdev=188.14 lat (msec): min=28, max=636, avg=116.81, stdev=188.14 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 32], | 30.00th=[ 33], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 39], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 144], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min=14336, max=14336, per=93.21%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (usec) : 1000=8.33% lat (msec) : 50=58.33%, 250=16.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=14.02%, ctx=352, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14016: Wed May 21 02:54:58 2025 read: IOPS=3, BW=3751KiB/s (3841kB/s)(6144KiB/1638msec) clat (usec): min=470, max=625060, avg=116563.19, stdev=249720.72 lat (usec): min=471, max=625062, avg=116564.50, stdev=249720.71 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 660], | 30.00th=[ 660], 40.00th=[ 840], 50.00th=[ 840], 60.00th=[ 34866], | 70.00th=[ 37487], 80.00th=[ 37487], 90.00th=[624952], 95.00th=[624952], | 99.00th=[624952], 99.50th=[624952], 99.90th=[624952], 99.95th=[624952], | 99.99th=[624952] bw ( KiB/s): min= 4096, max= 4096, per=37.28%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3751KiB/s (3841kB/s)(6144KiB/1638msec) clat (msec): min=22, max=547, avg=146.96, stdev=199.61 lat (msec): min=22, max=547, avg=147.01, stdev=199.63 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 102], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 8192, max= 8192, per=53.26%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33%, 1000=8.33% lat (msec) : 50=33.33%, 100=8.33%, 250=16.67%, 750=16.67% cpu : usr=0.00%, sys=13.93%, ctx=645, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14018: Wed May 21 02:54:58 2025 read: IOPS=2, BW=2747KiB/s (2813kB/s)(5120KiB/1864msec) clat (usec): min=460, max=601973, avg=147395.09, stdev=255489.35 lat (usec): min=461, max=601974, avg=147396.42, stdev=255489.36 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 461], 10.00th=[ 461], 20.00th=[ 461], | 30.00th=[ 29754], 40.00th=[ 29754], 50.00th=[ 30278], 60.00th=[ 30278], | 70.00th=[ 74974], 80.00th=[ 74974], 90.00th=[599786], 95.00th=[599786], | 99.00th=[599786], 99.50th=[599786], 99.90th=[599786], 99.95th=[599786], | 99.99th=[599786] write: IOPS=3, BW=3845KiB/s (3938kB/s)(7168KiB/1864msec) clat (msec): min=23, max=541, avg=155.59, stdev=175.32 lat (msec): min=23, max=541, avg=155.69, stdev=175.37 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 94], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 157], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 8192, max= 8192, per=53.26%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=33.33%, 250=16.67%, 750=16.67% cpu : usr=0.11%, sys=12.12%, ctx=598, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.7MiB/s (11.2MB/s), 1226KiB/s-4262KiB/s (1256kB/s-4364kB/s), io=20.0MiB (20.0MB), run=1638-1864msec WRITE: bw=15.0MiB/s (15.8MB/s), 3044KiB/s-6132KiB/s (3117kB/s-6279kB/s), io=28.0MiB (29.4MB), run=1638-1864msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14011: Wed May 21 02:54:58 2025 read: IOPS=4, BW=4155KiB/s (4255kB/s)(7168KiB/1725msec) clat (msec): min=14, max=651, avg=191.34, stdev=281.79 lat (msec): min=14, max=651, avg=191.34, stdev=281.79 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 27], 60.00th=[ 65], | 70.00th=[ 65], 80.00th=[ 550], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 4096, max= 6144, per=48.58%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2968KiB/s (3039kB/s)(5120KiB/1725msec) clat (msec): min=35, max=106, avg=75.34, stdev=25.54 lat (msec): min=35, max=106, avg=75.42, stdev=25.54 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 79], 60.00th=[ 79], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8192, max= 8192, per=55.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=16.67%, 100=33.33%, 250=8.33%, 750=16.67% cpu : usr=0.06%, sys=4.76%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14013: Wed May 21 02:54:58 2025 read: IOPS=1, BW=1054KiB/s (1079kB/s)(2048KiB/1943msec) clat (msec): min=40, max=473, avg=256.59, stdev=306.27 lat (msec): min=40, max=473, avg=256.59, stdev=306.27 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 2048, max= 2048, per=19.43%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5270KiB/s (5397kB/s)(10.0MiB/1943msec) clat (msec): min=29, max=736, avg=137.91, stdev=215.07 lat (msec): min=29, max=736, avg=137.97, stdev=215.07 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 34], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 192], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 6144, max= 6144, per=41.64%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=50.00%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=6.75%, ctx=181, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14015: Wed May 21 02:54:58 2025 read: IOPS=3, BW=3513KiB/s (3597kB/s)(6144KiB/1749msec) clat (msec): min=17, max=588, avg=144.98, stdev=224.13 lat (msec): min=17, max=588, avg=144.98, stdev=224.13 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 36], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 4096, max= 4096, per=38.86%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3513KiB/s (3597kB/s)(6144KiB/1749msec) clat (msec): min=39, max=526, avg=140.34, stdev=190.59 lat (msec): min=39, max=526, avg=140.40, stdev=190.60 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 80], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 8192, max= 8192, per=55.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=16.67%, 750=16.67% cpu : usr=0.00%, sys=5.26%, ctx=53, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14017: Wed May 21 02:54:58 2025 read: IOPS=2, BW=2671KiB/s (2735kB/s)(5120KiB/1917msec) clat (msec): min=22, max=620, avg=167.74, stdev=256.67 lat (msec): min=22, max=620, avg=167.74, stdev=256.67 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 37], 60.00th=[ 37], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] write: IOPS=3, BW=3739KiB/s (3829kB/s)(7168KiB/1917msec) clat (msec): min=23, max=522, avg=150.75, stdev=169.16 lat (msec): min=23, max=523, avg=150.86, stdev=169.30 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 61], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 96], 60.00th=[ 124], | 70.00th=[ 124], 80.00th=[ 153], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 8192, max= 8192, per=55.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=25.00%, 250=25.00%, 750=16.67% cpu : usr=0.10%, sys=5.53%, ctx=56, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.3MiB/s (10.8MB/s), 1054KiB/s-4155KiB/s (1079kB/s-4255kB/s), io=20.0MiB (20.0MB), run=1725-1943msec WRITE: bw=14.4MiB/s (15.1MB/s), 2968KiB/s-5270KiB/s (3039kB/s-5397kB/s), io=28.0MiB (29.4MB), run=1725-1943msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14025: Wed May 21 02:55:00 2025 read: IOPS=7, BW=28.7MiB/s (30.1MB/s)(12.0MiB/418msec) clat (msec): min=99, max=172, avg=131.77, stdev=36.91 lat (msec): min=99, max=172, avg=131.77, stdev=36.91 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=35.01%, ctx=1348, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14026: Wed May 21 02:55:00 2025 read: IOPS=1, BW=4785KiB/s (4900kB/s)(4096KiB/856msec) clat (nsec): min=137047k, max=137047k, avg=137046933.00, stdev= 0.00 lat (nsec): min=137049k, max=137049k, avg=137049358.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8159, max= 8159, per=42.63%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9570KiB/s (9800kB/s)(8192KiB/856msec) clat (msec): min=340, max=343, avg=342.36, stdev= 2.25 lat (msec): min=341, max=344, avg=342.63, stdev= 2.25 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=25.85%, ctx=628, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14027: Wed May 21 02:55:00 2025 write: IOPS=4, BW=18.6MiB/s (19.5MB/s)(12.0MiB/644msec) clat (msec): min=179, max=248, avg=206.82, stdev=36.80 lat (msec): min=180, max=249, avg=207.12, stdev=36.78 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 249], 80.00th=[ 249], 90.00th=[ 249], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min=16384, max=16384, per=42.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=32.97%, ctx=127, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14028: Wed May 21 02:55:00 2025 write: IOPS=5, BW=22.1MiB/s (23.2MB/s)(12.0MiB/543msec) clat (msec): min=110, max=219, avg=174.89, stdev=57.12 lat (msec): min=110, max=219, avg=175.22, stdev=57.09 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min=16384, max=16384, per=42.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=42.44%, ctx=84, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.7MiB/s (19.6MB/s), 4785KiB/s-28.7MiB/s (4900kB/s-30.1MB/s), io=16.0MiB (16.8MB), run=418-856msec WRITE: bw=37.4MiB/s (39.2MB/s), 9570KiB/s-22.1MiB/s (9800kB/s-23.2MB/s), io=32.0MiB (33.6MB), run=543-856msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14029: Wed May 21 02:55:00 2025 read: IOPS=4, BW=16.0MiB/s (17.8MB/s)(12.0MiB/707msec) clat (msec): min=115, max=307, avg=230.91, stdev=101.53 lat (msec): min=115, max=307, avg=230.91, stdev=101.53 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=42.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=9.35%, ctx=39, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14030: Wed May 21 02:55:00 2025 read: IOPS=1, BW=4859KiB/s (4975kB/s)(4096KiB/843msec) clat (nsec): min=358976k, max=358976k, avg=358976491.00, stdev= 0.00 lat (nsec): min=358978k, max=358978k, avg=358977811.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8175, max= 8175, per=42.06%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9718KiB/s (9951kB/s)(8192KiB/843msec) clat (msec): min=196, max=252, avg=224.31, stdev=39.93 lat (msec): min=196, max=252, avg=224.62, stdev=39.82 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 253], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.12%, sys=12.47%, ctx=114, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14031: Wed May 21 02:55:00 2025 write: IOPS=3, BW=15.0MiB/s (15.7MB/s)(12.0MiB/800msec) clat (msec): min=97, max=331, avg=249.93, stdev=131.80 lat (msec): min=98, max=331, avg=250.21, stdev=131.77 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8192, max= 8192, per=21.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=10.01%, ctx=237, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14032: Wed May 21 02:55:00 2025 write: IOPS=5, BW=20.5MiB/s (21.5MB/s)(12.0MiB/585msec) clat (msec): min=91, max=244, avg=188.39, stdev=84.47 lat (msec): min=91, max=245, avg=188.72, stdev=84.48 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 92], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min=16351, max=16351, per=42.07%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=19.83%, ctx=73, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.0MiB/s (19.9MB/s), 4859KiB/s-16.0MiB/s (4975kB/s-17.8MB/s), io=16.0MiB (16.8MB), run=707-843msec WRITE: bw=37.0MiB/s (39.8MB/s), 9718KiB/s-20.5MiB/s (9951kB/s-21.5MB/s), io=32.0MiB (33.6MB), run=585-843msec