== sanity test 398b: DIO and buffer IO race ============== 04:04:34 (1749542674) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18746 s, 42.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20787: Tue Jun 10 04:04:49 2025 read: IOPS=116, BW=466KiB/s (477kB/s)(5992KiB/12859msec) clat (usec): min=59, max=31616, avg=4690.53, stdev=2604.29 lat (usec): min=59, max=31617, avg=4691.24, stdev=2604.34 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 1893], 10.00th=[ 2278], 20.00th=[ 2769], | 30.00th=[ 3163], 40.00th=[ 3720], 50.00th=[ 4228], 60.00th=[ 4883], | 70.00th=[ 5538], 80.00th=[ 6390], 90.00th=[ 7767], 95.00th=[ 9110], | 99.00th=[12911], 99.50th=[15270], 99.90th=[20317], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 320, max= 552, per=24.82%, avg=461.36, stdev=57.63, samples=25 iops : min= 80, max= 138, avg=115.32, stdev=14.38, samples=25 write: IOPS=122, BW=490KiB/s (501kB/s)(6296KiB/12859msec) clat (usec): min=433, max=171780, avg=3666.23, stdev=5229.56 lat (usec): min=434, max=171781, avg=3667.04, stdev=5229.55 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 562], 10.00th=[ 603], 20.00th=[ 742], | 30.00th=[ 996], 40.00th=[ 2180], 50.00th=[ 2966], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5866], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[ 13566], 99.50th=[ 16188], 99.90th=[ 23462], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 384, max= 632, per=24.96%, avg=485.72, stdev=57.43, samples=25 iops : min= 96, max= 158, avg=121.40, stdev=14.35, samples=25 lat (usec) : 100=1.37%, 250=0.65%, 500=0.26%, 750=10.22%, 1000=4.95% lat (msec) : 2=5.14%, 4=32.49%, 10=41.05%, 20=3.74%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.39%, ctx=5753, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20788: Tue Jun 10 04:04:49 2025 read: IOPS=115, BW=460KiB/s (471kB/s)(5940KiB/12913msec) clat (usec): min=59, max=165942, avg=4702.71, stdev=4967.70 lat (usec): min=59, max=165943, avg=4704.07, stdev=4967.73 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 131], 10.00th=[ 2212], 20.00th=[ 2671], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 4146], 60.00th=[ 4686], | 70.00th=[ 5473], 80.00th=[ 6390], 90.00th=[ 7767], 95.00th=[ 9372], | 99.00th=[ 13698], 99.50th=[ 15795], 99.90th=[ 25297], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 256, max= 592, per=24.32%, avg=452.12, stdev=73.31, samples=25 iops : min= 64, max= 148, avg=113.00, stdev=18.34, samples=25 write: IOPS=122, BW=492KiB/s (503kB/s)(6348KiB/12913msec) clat (usec): min=433, max=25287, avg=3685.30, stdev=3251.65 lat (usec): min=434, max=25287, avg=3686.22, stdev=3251.62 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 734], | 30.00th=[ 1012], 40.00th=[ 2212], 50.00th=[ 2966], 60.00th=[ 3752], | 70.00th=[ 4686], 80.00th=[ 5997], 90.00th=[ 8029], 95.00th=[ 9765], | 99.00th=[14484], 99.50th=[15664], 99.90th=[21103], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 312, max= 592, per=24.48%, avg=476.44, stdev=68.04, samples=25 iops : min= 78, max= 148, avg=119.08, stdev=17.03, samples=25 lat (usec) : 100=1.86%, 250=0.91%, 500=0.20%, 750=10.77%, 1000=4.52% lat (msec) : 2=5.21%, 4=31.61%, 10=40.62%, 20=4.10%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.75%, ctx=5846, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20789: Tue Jun 10 04:04:49 2025 read: IOPS=116, BW=468KiB/s (479kB/s)(6028KiB/12890msec) clat (usec): min=56, max=172639, avg=4842.93, stdev=5133.94 lat (usec): min=56, max=172639, avg=4844.89, stdev=5134.90 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 208], 10.00th=[ 2245], 20.00th=[ 2704], | 30.00th=[ 3130], 40.00th=[ 3654], 50.00th=[ 4228], 60.00th=[ 4883], | 70.00th=[ 5538], 80.00th=[ 6390], 90.00th=[ 7832], 95.00th=[ 9896], | 99.00th=[ 14746], 99.50th=[ 16909], 99.90th=[ 22938], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 304, max= 568, per=24.95%, avg=463.80, stdev=59.96, samples=25 iops : min= 76, max= 142, avg=115.88, stdev=14.93, samples=25 write: IOPS=121, BW=486KiB/s (497kB/s)(6260KiB/12890msec) clat (usec): min=466, max=20963, avg=3534.54, stdev=3142.62 lat (usec): min=467, max=20964, avg=3535.72, stdev=3142.64 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 693], | 30.00th=[ 906], 40.00th=[ 1909], 50.00th=[ 2835], 60.00th=[ 3589], | 70.00th=[ 4490], 80.00th=[ 5932], 90.00th=[ 8160], 95.00th=[ 9896], | 99.00th=[13173], 99.50th=[14484], 99.90th=[17433], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 352, max= 688, per=24.62%, avg=479.20, stdev=77.90, samples=25 iops : min= 88, max= 172, avg=119.72, stdev=19.47, samples=25 lat (usec) : 100=1.63%, 250=0.88%, 500=0.36%, 750=11.39%, 1000=5.11% lat (msec) : 2=4.85%, 4=31.35%, 10=39.71%, 20=4.56%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.93%, ctx=5861, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20790: Tue Jun 10 04:04:49 2025 read: IOPS=117, BW=470KiB/s (481kB/s)(6052KiB/12881msec) clat (usec): min=58, max=175258, avg=4855.95, stdev=5093.56 lat (usec): min=58, max=175258, avg=4857.16, stdev=5093.81 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 1647], 10.00th=[ 2212], 20.00th=[ 2737], | 30.00th=[ 3195], 40.00th=[ 3687], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 5735], 80.00th=[ 6521], 90.00th=[ 7898], 95.00th=[ 9634], | 99.00th=[ 13435], 99.50th=[ 15533], 99.90th=[ 21103], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 336, max= 558, per=24.83%, avg=461.52, stdev=53.72, samples=25 iops : min= 84, max= 139, avg=115.28, stdev=13.37, samples=25 write: IOPS=121, BW=484KiB/s (496kB/s)(6236KiB/12881msec) clat (usec): min=440, max=25181, avg=3512.30, stdev=3137.21 lat (usec): min=441, max=25182, avg=3513.08, stdev=3137.30 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 701], | 30.00th=[ 922], 40.00th=[ 1844], 50.00th=[ 2802], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 5932], 90.00th=[ 7898], 95.00th=[ 9634], | 99.00th=[13173], 99.50th=[14484], 99.90th=[23200], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 368, max= 606, per=24.52%, avg=477.24, stdev=66.24, samples=25 iops : min= 92, max= 151, avg=119.20, stdev=16.58, samples=25 lat (usec) : 100=1.69%, 250=0.59%, 500=0.29%, 750=11.43%, 1000=4.79% lat (msec) : 2=5.44%, 4=31.09%, 10=40.56%, 20=4.00%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=23.32%, ctx=5748, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1860KiB/s (1904kB/s), 460KiB/s-470KiB/s (471kB/s-481kB/s), io=23.4MiB (24.6MB), run=12859-12913msec WRITE: bw=1947KiB/s (1994kB/s), 484KiB/s-492KiB/s (496kB/s-503kB/s), io=24.6MiB (25.7MB), run=12859-12913msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20791: Tue Jun 10 04:04:54 2025 read: IOPS=88, BW=354KiB/s (362kB/s)(5992KiB/16930msec) clat (usec): min=1684, max=162953, avg=4584.73, stdev=5241.32 lat (usec): min=1685, max=162954, avg=4585.38, stdev=5241.32 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 6390], 90.00th=[ 9110], 95.00th=[ 11600], | 99.00th=[ 16450], 99.50th=[ 18220], 99.90th=[ 28443], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 184, max= 726, per=24.84%, avg=349.27, stdev=152.64, samples=33 iops : min= 46, max= 181, avg=87.30, stdev=38.12, samples=33 write: IOPS=92, BW=372KiB/s (381kB/s)(6296KiB/16930msec) clat (usec): min=2380, max=32477, avg=6350.73, stdev=3782.05 lat (usec): min=2380, max=32478, avg=6351.62, stdev=3781.99 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 2999], 10.00th=[ 3228], 20.00th=[ 3523], | 30.00th=[ 3818], 40.00th=[ 4228], 50.00th=[ 4817], 60.00th=[ 5800], | 70.00th=[ 7177], 80.00th=[ 8979], 90.00th=[11469], 95.00th=[13829], | 99.00th=[19530], 99.50th=[22676], 99.90th=[30278], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 216, max= 760, per=24.98%, avg=367.70, stdev=154.00, samples=33 iops : min= 54, max= 190, avg=91.91, stdev=38.47, samples=33 lat (msec) : 2=2.93%, 4=45.80%, 10=39.32%, 20=11.30%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.08%, ctx=3708, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20792: Tue Jun 10 04:04:54 2025 read: IOPS=87, BW=350KiB/s (358kB/s)(5940KiB/16972msec) clat (usec): min=1514, max=179629, avg=4545.45, stdev=5548.51 lat (usec): min=1515, max=179630, avg=4546.39, stdev=5548.45 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1975], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3752], | 70.00th=[ 4555], 80.00th=[ 6063], 90.00th=[ 9110], 95.00th=[ 11469], | 99.00th=[ 16319], 99.50th=[ 17695], 99.90th=[ 26608], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 160, max= 728, per=24.69%, avg=347.15, stdev=157.09, samples=33 iops : min= 40, max= 182, avg=86.79, stdev=39.27, samples=33 write: IOPS=93, BW=374KiB/s (383kB/s)(6348KiB/16972msec) clat (usec): min=2268, max=30645, avg=6410.65, stdev=3788.96 lat (usec): min=2269, max=30646, avg=6411.52, stdev=3788.97 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3064], 10.00th=[ 3261], 20.00th=[ 3556], | 30.00th=[ 3818], 40.00th=[ 4178], 50.00th=[ 4817], 60.00th=[ 5866], | 70.00th=[ 7373], 80.00th=[ 9110], 90.00th=[11863], 95.00th=[14091], | 99.00th=[19268], 99.50th=[21103], 99.90th=[29754], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 216, max= 704, per=24.85%, avg=365.82, stdev=146.30, samples=33 iops : min= 54, max= 176, avg=91.45, stdev=36.58, samples=33 lat (msec) : 2=3.19%, 4=45.70%, 10=39.13%, 20=11.33%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.17%, ctx=3704, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20793: Tue Jun 10 04:04:54 2025 read: IOPS=89, BW=357KiB/s (366kB/s)(6028KiB/16877msec) clat (usec): min=1508, max=27678, avg=4477.39, stdev=3331.31 lat (usec): min=1508, max=27679, avg=4478.10, stdev=3331.25 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1942], 10.00th=[ 2057], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3720], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 9241], 95.00th=[11863], | 99.00th=[16712], 99.50th=[18482], 99.90th=[23200], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 176, max= 752, per=25.12%, avg=353.18, stdev=152.34, samples=33 iops : min= 44, max= 188, avg=88.27, stdev=38.10, samples=33 write: IOPS=92, BW=371KiB/s (380kB/s)(6260KiB/16877msec) clat (msec): min=2, max=182, avg= 6.44, stdev= 5.84 lat (msec): min=2, max=182, avg= 6.44, stdev= 5.84 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 37], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 216, max= 744, per=24.85%, avg=365.79, stdev=155.10, samples=33 iops : min= 54, max= 186, avg=91.42, stdev=38.80, samples=33 lat (msec) : 2=3.68%, 4=45.74%, 10=38.44%, 20=11.56%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.19%, ctx=3703, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20794: Tue Jun 10 04:04:54 2025 read: IOPS=88, BW=354KiB/s (363kB/s)(6052KiB/17075msec) clat (usec): min=1483, max=28295, avg=4593.15, stdev=3537.83 lat (usec): min=1483, max=28295, avg=4593.78, stdev=3537.82 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3785], | 70.00th=[ 4621], 80.00th=[ 6456], 90.00th=[ 9765], 95.00th=[12256], | 99.00th=[16712], 99.50th=[19268], 99.90th=[27657], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 168, max= 768, per=25.08%, avg=352.62, stdev=156.32, samples=34 iops : min= 42, max= 192, avg=88.12, stdev=39.08, samples=34 write: IOPS=91, BW=365KiB/s (374kB/s)(6236KiB/17075msec) clat (msec): min=2, max=160, avg= 6.45, stdev= 5.48 lat (msec): min=2, max=160, avg= 6.45, stdev= 5.48 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 26], 99.90th=[ 37], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 192, max= 696, per=24.71%, avg=363.68, stdev=156.60, samples=34 iops : min= 48, max= 174, avg=90.88, stdev=39.15, samples=34 lat (msec) : 2=3.84%, 4=45.21%, 10=38.22%, 20=12.01%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=10.99%, ctx=3646, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1406KiB/s (1440kB/s), 350KiB/s-357KiB/s (358kB/s-366kB/s), io=23.4MiB (24.6MB), run=16877-17075msec WRITE: bw=1472KiB/s (1508kB/s), 365KiB/s-374KiB/s (374kB/s-383kB/s), io=24.6MiB (25.7MB), run=16877-17075msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20823: Tue Jun 10 04:04:58 2025 read: IOPS=108, BW=1738KiB/s (1780kB/s)(5920KiB/3406msec) clat (usec): min=69, max=14779, avg=4532.58, stdev=2632.73 lat (usec): min=70, max=14779, avg=4533.37, stdev=2632.76 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 85], 10.00th=[ 130], 20.00th=[ 2704], | 30.00th=[ 3261], 40.00th=[ 3720], 50.00th=[ 4146], 60.00th=[ 4752], | 70.00th=[ 5538], 80.00th=[ 6783], 90.00th=[ 7898], 95.00th=[ 8979], | 99.00th=[12125], 99.50th=[12911], 99.90th=[14746], 99.95th=[14746], | 99.99th=[14746] bw ( KiB/s): min= 1152, max= 1952, per=24.63%, avg=1653.33, stdev=297.90, samples=6 iops : min= 72, max= 122, avg=103.33, stdev=18.62, samples=6 write: IOPS=116, BW=1870KiB/s (1915kB/s)(6368KiB/3406msec) clat (usec): min=640, max=17318, avg=4158.44, stdev=3458.37 lat (usec): min=641, max=17319, avg=4162.94, stdev=3462.14 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 799], 10.00th=[ 848], 20.00th=[ 1004], | 30.00th=[ 1369], 40.00th=[ 2573], 50.00th=[ 3425], 60.00th=[ 4293], | 70.00th=[ 5211], 80.00th=[ 6390], 90.00th=[ 9241], 95.00th=[11207], | 99.00th=[15533], 99.50th=[17171], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1440, max= 2336, per=25.75%, avg=1818.67, stdev=347.27, samples=6 iops : min= 90, max= 146, avg=113.67, stdev=21.70, samples=6 lat (usec) : 100=3.65%, 250=1.56%, 750=1.04%, 1000=9.24% lat (msec) : 2=8.59%, 4=28.39%, 10=41.80%, 20=5.73% cpu : usr=0.18%, sys=24.35%, ctx=1638, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20824: Tue Jun 10 04:04:58 2025 read: IOPS=104, BW=1671KiB/s (1711kB/s)(5744KiB/3438msec) clat (usec): min=70, max=22061, avg=4918.36, stdev=2938.75 lat (usec): min=70, max=22062, avg=4922.02, stdev=2940.21 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 82], 10.00th=[ 420], 20.00th=[ 2900], | 30.00th=[ 3490], 40.00th=[ 4113], 50.00th=[ 4621], 60.00th=[ 5211], | 70.00th=[ 5932], 80.00th=[ 7046], 90.00th=[ 8455], 95.00th=[ 9896], | 99.00th=[13698], 99.50th=[16057], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 1376, max= 1756, per=23.98%, avg=1610.00, stdev=156.88, samples=6 iops : min= 86, max= 109, avg=100.50, stdev= 9.67, samples=6 write: IOPS=118, BW=1903KiB/s (1949kB/s)(6544KiB/3438msec) clat (usec): min=704, max=17589, avg=3902.47, stdev=3164.49 lat (usec): min=704, max=17590, avg=3903.47, stdev=3164.49 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 824], 10.00th=[ 848], 20.00th=[ 996], | 30.00th=[ 1319], 40.00th=[ 2409], 50.00th=[ 3097], 60.00th=[ 3982], | 70.00th=[ 5080], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[10028], | 99.00th=[14222], 99.50th=[15270], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1472, max= 2272, per=25.82%, avg=1823.33, stdev=293.90, samples=6 iops : min= 92, max= 142, avg=113.83, stdev=18.49, samples=6 lat (usec) : 100=3.91%, 250=0.65%, 500=0.13%, 750=0.65%, 1000=10.03% lat (msec) : 2=9.37%, 4=24.61%, 10=45.57%, 20=4.95%, 50=0.13% cpu : usr=0.09%, sys=24.32%, ctx=1680, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20825: Tue Jun 10 04:04:58 2025 read: IOPS=109, BW=1753KiB/s (1795kB/s)(6256KiB/3568msec) clat (usec): min=59, max=25809, avg=4994.74, stdev=2993.02 lat (usec): min=60, max=25809, avg=4997.17, stdev=2992.21 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 94], 10.00th=[ 2343], 20.00th=[ 2868], | 30.00th=[ 3326], 40.00th=[ 3982], 50.00th=[ 4621], 60.00th=[ 5276], | 70.00th=[ 5932], 80.00th=[ 7046], 90.00th=[ 8356], 95.00th=[ 9765], | 99.00th=[15664], 99.50th=[16712], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1152, max= 2048, per=24.55%, avg=1648.00, stdev=309.43, samples=6 iops : min= 72, max= 128, avg=103.00, stdev=19.34, samples=6 write: IOPS=105, BW=1691KiB/s (1731kB/s)(6032KiB/3568msec) clat (usec): min=698, max=17811, avg=4110.14, stdev=3255.85 lat (usec): min=699, max=17812, avg=4111.06, stdev=3255.98 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 824], 10.00th=[ 889], 20.00th=[ 1045], | 30.00th=[ 1598], 40.00th=[ 2671], 50.00th=[ 3261], 60.00th=[ 3949], | 70.00th=[ 5407], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[10290], | 99.00th=[14091], 99.50th=[17695], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1248, max= 2080, per=23.26%, avg=1642.67, stdev=316.57, samples=6 iops : min= 78, max= 130, avg=102.67, stdev=19.79, samples=6 lat (usec) : 100=2.73%, 250=0.91%, 500=0.13%, 750=0.26%, 1000=9.37% lat (msec) : 2=6.77%, 4=30.73%, 10=43.88%, 20=5.08%, 50=0.13% cpu : usr=0.08%, sys=24.05%, ctx=1671, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20826: Tue Jun 10 04:04:58 2025 read: IOPS=114, BW=1826KiB/s (1870kB/s)(6032KiB/3303msec) clat (usec): min=57, max=15978, avg=4700.40, stdev=2815.58 lat (usec): min=57, max=15978, avg=4701.49, stdev=2815.47 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 88], 10.00th=[ 210], 20.00th=[ 2802], | 30.00th=[ 3359], 40.00th=[ 3687], 50.00th=[ 4228], 60.00th=[ 4817], | 70.00th=[ 5669], 80.00th=[ 6718], 90.00th=[ 8225], 95.00th=[10159], | 99.00th=[13829], 99.50th=[14615], 99.90th=[15926], 99.95th=[15926], | 99.99th=[15926] bw ( KiB/s): min= 1440, max= 2043, per=26.44%, avg=1775.17, stdev=220.25, samples=6 iops : min= 90, max= 127, avg=110.83, stdev=13.60, samples=6 write: IOPS=118, BW=1894KiB/s (1939kB/s)(6256KiB/3303msec) clat (usec): min=640, max=24217, avg=3860.75, stdev=3322.89 lat (usec): min=641, max=24218, avg=3865.38, stdev=3320.69 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 807], 10.00th=[ 832], 20.00th=[ 889], | 30.00th=[ 1139], 40.00th=[ 2057], 50.00th=[ 3032], 60.00th=[ 3752], | 70.00th=[ 4817], 80.00th=[ 6652], 90.00th=[ 8455], 95.00th=[ 9765], | 99.00th=[13173], 99.50th=[15533], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1664, max= 2368, per=26.65%, avg=1882.00, stdev=262.73, samples=6 iops : min= 104, max= 148, avg=117.50, stdev=16.44, samples=6 lat (usec) : 100=3.39%, 250=1.56%, 750=1.17%, 1000=11.72% lat (msec) : 2=7.16%, 4=29.56%, 10=40.36%, 20=4.95%, 50=0.13% cpu : usr=0.09%, sys=24.74%, ctx=1571, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6713KiB/s (6874kB/s), 1671KiB/s-1826KiB/s (1711kB/s-1870kB/s), io=23.4MiB (24.5MB), run=3303-3568msec WRITE: bw=7063KiB/s (7232kB/s), 1691KiB/s-1903KiB/s (1731kB/s-1949kB/s), io=24.6MiB (25.8MB), run=3303-3568msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20819: Tue Jun 10 04:04:59 2025 read: IOPS=83, BW=1332KiB/s (1364kB/s)(5920KiB/4445msec) clat (usec): min=1864, max=19984, avg=4608.80, stdev=3389.85 lat (usec): min=1864, max=19985, avg=4609.40, stdev=3389.86 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3195], 60.00th=[ 3720], | 70.00th=[ 4555], 80.00th=[ 6063], 90.00th=[ 9896], 95.00th=[12518], | 99.00th=[16581], 99.50th=[18220], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 864, max= 2272, per=22.52%, avg=1199.62, stdev=464.82, samples=8 iops : min= 54, max= 142, avg=74.88, stdev=29.01, samples=8 write: IOPS=89, BW=1433KiB/s (1467kB/s)(6368KiB/4445msec) clat (usec): min=2599, max=30596, avg=6733.88, stdev=4091.45 lat (usec): min=2600, max=30597, avg=6735.05, stdev=4091.50 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3097], 10.00th=[ 3294], 20.00th=[ 3621], | 30.00th=[ 3916], 40.00th=[ 4293], 50.00th=[ 4948], 60.00th=[ 6128], | 70.00th=[ 7635], 80.00th=[10159], 90.00th=[12780], 95.00th=[14877], | 99.00th=[20055], 99.50th=[22676], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 926, max= 2368, per=23.27%, avg=1303.75, stdev=486.05, samples=8 iops : min= 57, max= 148, avg=81.37, stdev=30.48, samples=8 lat (msec) : 2=0.65%, 4=46.61%, 10=37.24%, 20=15.10%, 50=0.39% cpu : usr=0.20%, sys=11.36%, ctx=961, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20820: Tue Jun 10 04:04:59 2025 read: IOPS=79, BW=1277KiB/s (1308kB/s)(5744KiB/4497msec) clat (usec): min=1877, max=19882, avg=4797.72, stdev=3650.27 lat (usec): min=1877, max=19883, avg=4798.33, stdev=3650.27 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3130], 60.00th=[ 3589], | 70.00th=[ 4621], 80.00th=[ 7046], 90.00th=[11076], 95.00th=[13173], | 99.00th=[16319], 99.50th=[19268], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 704, max= 2432, per=21.93%, avg=1168.00, stdev=547.08, samples=8 iops : min= 44, max= 152, avg=73.00, stdev=34.19, samples=8 write: IOPS=90, BW=1455KiB/s (1490kB/s)(6544KiB/4497msec) clat (usec): min=2645, max=23988, avg=6623.23, stdev=3895.21 lat (usec): min=2646, max=23989, avg=6624.27, stdev=3895.24 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3064], 10.00th=[ 3228], 20.00th=[ 3589], | 30.00th=[ 3785], 40.00th=[ 4228], 50.00th=[ 4883], 60.00th=[ 6194], | 70.00th=[ 7898], 80.00th=[10159], 90.00th=[12649], 95.00th=[14353], | 99.00th=[17433], 99.50th=[19006], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 928, max= 2272, per=22.56%, avg=1264.00, stdev=418.98, samples=8 iops : min= 58, max= 142, avg=79.00, stdev=26.19, samples=8 lat (msec) : 2=1.17%, 4=48.57%, 10=33.72%, 20=16.41%, 50=0.13% cpu : usr=0.09%, sys=11.17%, ctx=915, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20821: Tue Jun 10 04:04:59 2025 read: IOPS=89, BW=1432KiB/s (1466kB/s)(6256KiB/4369msec) clat (usec): min=1817, max=19425, avg=4651.16, stdev=3541.86 lat (usec): min=1817, max=19425, avg=4651.77, stdev=3541.85 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3556], | 70.00th=[ 4293], 80.00th=[ 6456], 90.00th=[10552], 95.00th=[13042], | 99.00th=[17433], 99.50th=[17695], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 800, max= 2272, per=24.33%, avg=1296.00, stdev=487.71, samples=8 iops : min= 50, max= 142, avg=81.00, stdev=30.48, samples=8 write: IOPS=86, BW=1381KiB/s (1414kB/s)(6032KiB/4369msec) clat (usec): min=2759, max=31281, avg=6725.69, stdev=3993.31 lat (usec): min=2760, max=31282, avg=6726.71, stdev=3993.32 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3064], 10.00th=[ 3294], 20.00th=[ 3589], | 30.00th=[ 3916], 40.00th=[ 4359], 50.00th=[ 5080], 60.00th=[ 6259], | 70.00th=[ 7635], 80.00th=[10159], 90.00th=[12911], 95.00th=[14615], | 99.00th=[17171], 99.50th=[23987], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 928, max= 2496, per=22.99%, avg=1288.00, stdev=499.20, samples=8 iops : min= 58, max= 156, avg=80.50, stdev=31.20, samples=8 lat (msec) : 2=0.91%, 4=49.61%, 10=33.46%, 20=15.76%, 50=0.26% cpu : usr=0.09%, sys=11.56%, ctx=963, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20822: Tue Jun 10 04:04:59 2025 read: IOPS=85, BW=1370KiB/s (1403kB/s)(6032KiB/4404msec) clat (usec): min=1956, max=22134, avg=4883.88, stdev=3812.01 lat (usec): min=1956, max=22135, avg=4884.49, stdev=3812.03 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2507], 40.00th=[ 2802], 50.00th=[ 3228], 60.00th=[ 3916], | 70.00th=[ 4686], 80.00th=[ 6718], 90.00th=[10683], 95.00th=[14353], | 99.00th=[17433], 99.50th=[17433], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 768, max= 2272, per=23.21%, avg=1236.00, stdev=449.93, samples=8 iops : min= 48, max= 142, avg=77.25, stdev=28.12, samples=8 write: IOPS=88, BW=1421KiB/s (1455kB/s)(6256KiB/4404msec) clat (usec): min=2530, max=22088, avg=6407.33, stdev=3639.27 lat (usec): min=2531, max=22089, avg=6408.46, stdev=3639.24 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3097], 10.00th=[ 3326], 20.00th=[ 3556], | 30.00th=[ 3916], 40.00th=[ 4293], 50.00th=[ 4817], 60.00th=[ 5800], | 70.00th=[ 7111], 80.00th=[ 9503], 90.00th=[11994], 95.00th=[13829], | 99.00th=[18482], 99.50th=[19268], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 800, max= 2432, per=23.42%, avg=1312.00, stdev=527.48, samples=8 iops : min= 50, max= 152, avg=82.00, stdev=32.97, samples=8 lat (msec) : 2=0.52%, 4=46.09%, 10=39.06%, 20=14.06%, 50=0.26% cpu : usr=0.11%, sys=11.51%, ctx=910, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5326KiB/s (5454kB/s), 1277KiB/s-1432KiB/s (1308kB/s-1466kB/s), io=23.4MiB (24.5MB), run=4369-4497msec WRITE: bw=5604KiB/s (5738kB/s), 1381KiB/s-1455KiB/s (1414kB/s-1490kB/s), io=24.6MiB (25.8MB), run=4369-4497msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20835: Tue Jun 10 04:05:01 2025 read: IOPS=6, BW=6282KiB/s (6433kB/s)(7168KiB/1141msec) clat (usec): min=413, max=597511, avg=110831.91, stdev=216972.00 lat (usec): min=415, max=597512, avg=110832.93, stdev=216971.94 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 498], | 30.00th=[ 22938], 40.00th=[ 22938], 50.00th=[ 28443], 60.00th=[ 30278], | 70.00th=[ 30278], 80.00th=[ 95945], 90.00th=[599786], 95.00th=[599786], | 99.00th=[599786], 99.50th=[599786], 99.90th=[599786], 99.95th=[599786], | 99.99th=[599786] bw ( KiB/s): min=10240, max=10240, per=65.20%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4487KiB/s (4595kB/s)(5120KiB/1141msec) clat (msec): min=33, max=123, avg=61.18, stdev=36.78 lat (msec): min=33, max=123, avg=61.22, stdev=36.77 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 50], 60.00th=[ 50], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 4096, max= 4096, per=18.63%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.09%, sys=20.44%, ctx=655, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20836: Tue Jun 10 04:05:01 2025 read: IOPS=1, BW=1571KiB/s (1608kB/s)(2048KiB/1304msec) clat (usec): min=30786, max=31466, avg=31126.10, stdev=480.96 lat (usec): min=30786, max=31467, avg=31126.85, stdev=481.21 clat percentiles (usec): | 1.00th=[30802], 5.00th=[30802], 10.00th=[30802], 20.00th=[30802], | 30.00th=[30802], 40.00th=[30802], 50.00th=[30802], 60.00th=[31589], | 70.00th=[31589], 80.00th=[31589], 90.00th=[31589], 95.00th=[31589], | 99.00th=[31589], 99.50th=[31589], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 1738, max= 2048, per=12.05%, avg=1893.00, stdev=219.20, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=7, BW=7853KiB/s (8041kB/s)(10.0MiB/1304msec) clat (msec): min=23, max=632, avg=118.39, stdev=183.68 lat (msec): min=23, max=632, avg=118.46, stdev=183.67 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 32], 40.00th=[ 52], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 127], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min=10431, max=10431, per=47.44%, avg=10431.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=16.67%, 750=8.33% cpu : usr=0.15%, sys=21.80%, ctx=373, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20837: Tue Jun 10 04:05:01 2025 read: IOPS=6, BW=6447KiB/s (6602kB/s)(6144KiB/953msec) clat (usec): min=297, max=834, avg=488.20, stdev=192.37 lat (usec): min=298, max=835, avg=489.00, stdev=192.46 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 297], 10.00th=[ 297], 20.00th=[ 367], | 30.00th=[ 367], 40.00th=[ 392], 50.00th=[ 392], 60.00th=[ 482], | 70.00th=[ 553], 80.00th=[ 553], 90.00th=[ 832], 95.00th=[ 832], | 99.00th=[ 832], 99.50th=[ 832], 99.90th=[ 832], 99.95th=[ 832], | 99.99th=[ 832] write: IOPS=6, BW=6447KiB/s (6602kB/s)(6144KiB/953msec) clat (msec): min=20, max=709, avg=148.79, stdev=275.16 lat (msec): min=20, max=709, avg=148.83, stdev=275.17 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 48], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 4096, max= 4096, per=18.63%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=33.33%, 750=8.33%, 1000=8.33% lat (msec) : 50=33.33%, 100=8.33%, 750=8.33% cpu : usr=0.00%, sys=14.60%, ctx=61, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20838: Tue Jun 10 04:05:01 2025 read: IOPS=5, BW=5412KiB/s (5542kB/s)(5120KiB/946msec) clat (usec): min=288, max=34952, avg=18798.08, stdev=17066.00 lat (usec): min=289, max=34953, avg=18798.75, stdev=17065.97 clat percentiles (usec): | 1.00th=[ 289], 5.00th=[ 289], 10.00th=[ 289], 20.00th=[ 289], | 30.00th=[ 445], 40.00th=[ 445], 50.00th=[26870], 60.00th=[26870], | 70.00th=[31327], 80.00th=[31327], 90.00th=[34866], 95.00th=[34866], | 99.00th=[34866], 99.50th=[34866], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] write: IOPS=7, BW=7577KiB/s (7759kB/s)(7168KiB/946msec) clat (msec): min=22, max=636, avg=118.21, stdev=228.81 lat (msec): min=22, max=637, avg=118.27, stdev=228.81 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 28], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 34], 80.00th=[ 42], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 2048, max= 2048, per=9.31%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=75.00%, 750=8.33% cpu : usr=0.00%, sys=19.89%, ctx=620, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.3MiB/s (16.1MB/s), 1571KiB/s-6447KiB/s (1608kB/s-6602kB/s), io=20.0MiB (20.0MB), run=946-1304msec WRITE: bw=21.5MiB/s (22.5MB/s), 4487KiB/s-7853KiB/s (4595kB/s-8041kB/s), io=28.0MiB (29.4MB), run=946-1304msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20839: Tue Jun 10 04:05:01 2025 read: IOPS=5, BW=5406KiB/s (5535kB/s)(7168KiB/1326msec) clat (msec): min=12, max=573, avg=137.67, stdev=205.40 lat (msec): min=12, max=573, avg=137.67, stdev=205.40 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 58], 60.00th=[ 64], | 70.00th=[ 64], 80.00th=[ 222], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 6131, max= 6131, per=39.70%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3861KiB/s (3954kB/s)(5120KiB/1326msec) clat (msec): min=20, max=161, avg=56.00, stdev=59.88 lat (msec): min=20, max=161, avg=56.04, stdev=59.87 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 43], 80.00th=[ 43], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] lat (msec) : 20=16.67%, 50=41.67%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=7.62%, ctx=44, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20840: Tue Jun 10 04:05:01 2025 read: IOPS=1, BW=1666KiB/s (1706kB/s)(2048KiB/1229msec) clat (usec): min=16218, max=51339, avg=33779.32, stdev=24834.30 lat (usec): min=16220, max=51340, avg=33780.64, stdev=24833.20 clat percentiles (usec): | 1.00th=[16188], 5.00th=[16188], 10.00th=[16188], 20.00th=[16188], | 30.00th=[16188], 40.00th=[16188], 50.00th=[16188], 60.00th=[51119], | 70.00th=[51119], 80.00th=[51119], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 2048, max= 2048, per=13.26%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8332KiB/s (8532kB/s)(10.0MiB/1229msec) clat (msec): min=20, max=625, avg=113.90, stdev=183.76 lat (msec): min=20, max=625, avg=113.96, stdev=183.76 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 24], | 30.00th=[ 26], 40.00th=[ 36], 50.00th=[ 40], 60.00th=[ 52], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 131], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min=10240, max=10240, per=47.36%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=11.16%, ctx=76, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20841: Tue Jun 10 04:05:01 2025 read: IOPS=4, BW=4947KiB/s (5066kB/s)(6144KiB/1242msec) clat (usec): min=33450, max=56874, avg=42092.41, stdev=9746.52 lat (usec): min=33451, max=56875, avg=42093.57, stdev=9746.67 clat percentiles (usec): | 1.00th=[33424], 5.00th=[33424], 10.00th=[33424], 20.00th=[33817], | 30.00th=[33817], 40.00th=[37487], 50.00th=[37487], 60.00th=[39060], | 70.00th=[51643], 80.00th=[51643], 90.00th=[56886], 95.00th=[56886], | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 4096, max= 4096, per=26.52%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4947KiB/s (5066kB/s)(6144KiB/1242msec) clat (msec): min=19, max=692, avg=163.84, stdev=264.34 lat (msec): min=19, max=692, avg=163.95, stdev=264.31 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 68], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 2048, max= 4096, per=14.21%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.16%, sys=8.70%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20842: Tue Jun 10 04:05:01 2025 read: IOPS=4, BW=4183KiB/s (4283kB/s)(5120KiB/1224msec) clat (usec): min=15088, max=51819, avg=29961.76, stdev=13899.11 lat (usec): min=15089, max=51820, avg=29962.88, stdev=13899.29 clat percentiles (usec): | 1.00th=[15139], 5.00th=[15139], 10.00th=[15139], 20.00th=[15139], | 30.00th=[21365], 40.00th=[21365], 50.00th=[30278], 60.00th=[30278], | 70.00th=[31327], 80.00th=[31327], 90.00th=[51643], 95.00th=[51643], | 99.00th=[51643], 99.50th=[51643], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] write: IOPS=5, BW=5856KiB/s (5997kB/s)(7168KiB/1224msec) clat (msec): min=16, max=701, avg=141.49, stdev=254.38 lat (msec): min=17, max=701, avg=141.55, stdev=254.36 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 24], | 70.00th=[ 24], 80.00th=[ 184], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 2048, max= 4096, per=14.21%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 20=25.00%, 50=50.00%, 100=8.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=6.87%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.1MiB/s (15.8MB/s), 1666KiB/s-5406KiB/s (1706kB/s-5535kB/s), io=20.0MiB (20.0MB), run=1224-1326msec WRITE: bw=21.1MiB/s (22.1MB/s), 3861KiB/s-8332KiB/s (3954kB/s-8532kB/s), io=28.0MiB (29.4MB), run=1224-1326msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20849: Tue Jun 10 04:05:03 2025 read: IOPS=5, BW=20.0MiB/s (20.0MB/s)(12.0MiB/600msec) clat (msec): min=52, max=374, avg=188.06, stdev=167.06 lat (msec): min=52, max=374, avg=188.06, stdev=167.06 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min=14707, max=14707, per=62.75%, avg=14707.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.68%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20850: Tue Jun 10 04:05:03 2025 read: IOPS=1, BW=5860KiB/s (6000kB/s)(4096KiB/699msec) clat (nsec): min=133027k, max=133027k, avg=133027337.00, stdev= 0.00 lat (nsec): min=133029k, max=133029k, avg=133028636.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 7571, max= 7571, per=32.30%, avg=7571.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.4MiB/s (12.0MB/s)(8192KiB/699msec) clat (msec): min=161, max=383, avg=272.76, stdev=157.08 lat (msec): min=161, max=384, avg=272.99, stdev=157.04 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 163], 50.00th=[ 163], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 7571, max= 7571, per=16.15%, avg=7571.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=10.59%, ctx=280, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20851: Tue Jun 10 04:05:03 2025 write: IOPS=6, BW=27.9MiB/s (29.3MB/s)(12.0MiB/430msec) clat (msec): min=59, max=246, avg=138.05, stdev=97.34 lat (msec): min=59, max=247, avg=138.36, stdev=97.31 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 108], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 247], 95.00th=[ 247], | 99.00th=[ 247], 99.50th=[ 247], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.47%, sys=13.75%, ctx=43, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20852: Tue Jun 10 04:05:03 2025 write: IOPS=4, BW=18.4MiB/s (19.3MB/s)(12.0MiB/651msec) clat (msec): min=58, max=316, avg=206.68, stdev=133.21 lat (msec): min=58, max=317, avg=207.02, stdev=133.28 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 8192, max= 8192, per=17.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=24.92%, ctx=66, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.9MiB/s (24.0MB/s), 5860KiB/s-20.0MiB/s (6000kB/s-20.0MB/s), io=16.0MiB (16.8MB), run=600-699msec WRITE: bw=45.8MiB/s (48.0MB/s), 11.4MiB/s-27.9MiB/s (12.0MB/s-29.3MB/s), io=32.0MiB (33.6MB), run=430-699msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20853: Tue Jun 10 04:05:03 2025 read: IOPS=6, BW=26.7MiB/s (28.0MB/s)(12.0MiB/449msec) clat (msec): min=123, max=150, avg=140.81, stdev=15.07 lat (msec): min=123, max=150, avg=140.81, stdev=15.07 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] lat (msec) : 250=100.00% cpu : usr=0.22%, sys=25.45%, ctx=1487, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20854: Tue Jun 10 04:05:03 2025 read: IOPS=1, BW=5397KiB/s (5526kB/s)(4096KiB/759msec) clat (nsec): min=148779k, max=148779k, avg=148778520.00, stdev= 0.00 lat (nsec): min=148780k, max=148780k, avg=148779534.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8175, max= 8175, per=37.87%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.5MiB/s (11.1MB/s)(8192KiB/759msec) clat (msec): min=96, max=483, avg=290.03, stdev=274.04 lat (msec): min=96, max=484, avg=290.28, stdev=274.01 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 96], 50.00th=[ 96], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.26%, sys=33.51%, ctx=488, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20855: Tue Jun 10 04:05:03 2025 write: IOPS=3, BW=15.3MiB/s (16.0MB/s)(12.0MiB/786msec) clat (msec): min=229, max=279, avg=254.22, stdev=24.92 lat (msec): min=229, max=279, avg=254.52, stdev=24.94 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 8192, max= 8192, per=19.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.25%, sys=33.50%, ctx=50, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20856: Tue Jun 10 04:05:03 2025 write: IOPS=6, BW=26.5MiB/s (27.8MB/s)(12.0MiB/453msec) clat (msec): min=91, max=188, avg=148.41, stdev=50.88 lat (msec): min=91, max=188, avg=148.69, stdev=50.89 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 92], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=51.77%, ctx=134, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.1MiB/s (22.1MB/s), 5397KiB/s-26.7MiB/s (5526kB/s-28.0MB/s), io=16.0MiB (16.8MB), run=449-759msec WRITE: bw=40.7MiB/s (42.7MB/s), 10.5MiB/s-26.5MiB/s (11.1MB/s-27.8MB/s), io=32.0MiB (33.6MB), run=453-786msec