== sanity test 398b: DIO and buffer IO race ============== 14:42:11 (1744137731) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.07278 s, 46.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1870: Tue Apr 8 14:42:25 2025 read: IOPS=134, BW=536KiB/s (549kB/s)(5992KiB/11175msec) clat (usec): min=54, max=135130, avg=4159.26, stdev=4153.97 lat (usec): min=54, max=135131, avg=4159.87, stdev=4154.06 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 260], 10.00th=[ 2008], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 3130], 50.00th=[ 3556], 60.00th=[ 4178], | 70.00th=[ 4817], 80.00th=[ 5604], 90.00th=[ 6718], 95.00th=[ 7832], | 99.00th=[ 12649], 99.50th=[ 16188], 99.90th=[ 26084], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 328, max= 688, per=24.55%, avg=527.41, stdev=69.58, samples=22 iops : min= 82, max= 172, avg=131.77, stdev=17.40, samples=22 write: IOPS=140, BW=563KiB/s (577kB/s)(6296KiB/11175msec) clat (usec): min=397, max=20186, avg=3101.16, stdev=2771.31 lat (usec): min=398, max=20186, avg=3101.86, stdev=2771.29 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 515], 10.00th=[ 529], 20.00th=[ 562], | 30.00th=[ 676], 40.00th=[ 1663], 50.00th=[ 2540], 60.00th=[ 3163], | 70.00th=[ 3916], 80.00th=[ 5145], 90.00th=[ 7111], 95.00th=[ 8586], | 99.00th=[11469], 99.50th=[12518], 99.90th=[14484], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 400, max= 736, per=24.67%, avg=554.73, stdev=86.69, samples=22 iops : min= 100, max= 184, avg=138.59, stdev=21.74, samples=22 lat (usec) : 100=2.25%, 250=0.16%, 500=1.30%, 750=15.53%, 1000=1.79% lat (msec) : 2=5.21%, 4=37.76%, 10=33.46%, 20=2.41%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=21.89%, ctx=5528, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1871: Tue Apr 8 14:42:25 2025 read: IOPS=133, BW=535KiB/s (548kB/s)(5940KiB/11104msec) clat (usec): min=53, max=17098, avg=4102.61, stdev=2395.91 lat (usec): min=53, max=17098, avg=4103.42, stdev=2395.83 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1598], 10.00th=[ 1975], 20.00th=[ 2343], | 30.00th=[ 2671], 40.00th=[ 3032], 50.00th=[ 3621], 60.00th=[ 4228], | 70.00th=[ 4817], 80.00th=[ 5669], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[13173], 99.50th=[15401], 99.90th=[16909], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 384, max= 696, per=24.79%, avg=532.50, stdev=70.68, samples=22 iops : min= 96, max= 174, avg=133.05, stdev=17.67, samples=22 write: IOPS=142, BW=572KiB/s (585kB/s)(6348KiB/11104msec) clat (usec): min=418, max=151421, avg=3130.27, stdev=4646.54 lat (usec): min=419, max=151421, avg=3131.52, stdev=4646.60 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 693], 40.00th=[ 1467], 50.00th=[ 2442], 60.00th=[ 3032], | 70.00th=[ 3818], 80.00th=[ 5211], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[ 11600], 99.50th=[ 13042], 99.90th=[ 22938], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 400, max= 760, per=25.21%, avg=567.05, stdev=86.27, samples=22 iops : min= 100, max= 190, avg=141.68, stdev=21.63, samples=22 lat (usec) : 100=1.92%, 250=0.26%, 500=1.20%, 750=15.40%, 1000=2.44% lat (msec) : 2=6.32%, 4=36.39%, 10=33.92%, 20=2.08%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.05%, ctx=5620, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1872: Tue Apr 8 14:42:25 2025 read: IOPS=135, BW=541KiB/s (554kB/s)(6028KiB/11148msec) clat (usec): min=51, max=22706, avg=4030.59, stdev=2288.46 lat (usec): min=51, max=22706, avg=4031.56, stdev=2288.58 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 135], 10.00th=[ 1991], 20.00th=[ 2409], | 30.00th=[ 2704], 40.00th=[ 3064], 50.00th=[ 3621], 60.00th=[ 4228], | 70.00th=[ 4817], 80.00th=[ 5538], 90.00th=[ 6652], 95.00th=[ 7832], | 99.00th=[11863], 99.50th=[12911], 99.90th=[19530], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 320, max= 656, per=24.96%, avg=536.09, stdev=79.96, samples=22 iops : min= 80, max= 164, avg=133.95, stdev=20.00, samples=22 write: IOPS=140, BW=562KiB/s (575kB/s)(6260KiB/11148msec) clat (usec): min=409, max=141391, avg=3199.29, stdev=4492.62 lat (usec): min=410, max=141393, avg=3199.97, stdev=4492.64 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 510], 10.00th=[ 537], 20.00th=[ 578], | 30.00th=[ 717], 40.00th=[ 1696], 50.00th=[ 2540], 60.00th=[ 3064], | 70.00th=[ 3916], 80.00th=[ 5211], 90.00th=[ 7111], 95.00th=[ 8717], | 99.00th=[ 12649], 99.50th=[ 14222], 99.90th=[ 18744], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 400, max= 728, per=24.69%, avg=555.36, stdev=75.76, samples=22 iops : min= 100, max= 182, avg=138.77, stdev=18.95, samples=22 lat (usec) : 100=2.34%, 250=0.20%, 500=1.56%, 750=14.19%, 1000=2.28% lat (msec) : 2=5.73%, 4=37.21%, 10=34.18%, 20=2.25%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=21.97%, ctx=5389, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1873: Tue Apr 8 14:42:25 2025 read: IOPS=136, BW=547KiB/s (560kB/s)(6052KiB/11071msec) clat (usec): min=51, max=26098, avg=4045.35, stdev=2310.87 lat (usec): min=52, max=26099, avg=4046.02, stdev=2310.80 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1729], 10.00th=[ 1975], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 3130], 50.00th=[ 3556], 60.00th=[ 4178], | 70.00th=[ 4883], 80.00th=[ 5538], 90.00th=[ 6652], 95.00th=[ 7963], | 99.00th=[12649], 99.50th=[15008], 99.90th=[16712], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 304, max= 696, per=25.33%, avg=544.09, stdev=93.18, samples=22 iops : min= 76, max= 174, avg=135.95, stdev=23.29, samples=22 write: IOPS=140, BW=563KiB/s (577kB/s)(6236KiB/11071msec) clat (usec): min=426, max=138110, avg=3132.80, stdev=4406.41 lat (usec): min=426, max=138112, avg=3133.99, stdev=4406.18 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 586], | 30.00th=[ 693], 40.00th=[ 1434], 50.00th=[ 2474], 60.00th=[ 3064], | 70.00th=[ 3884], 80.00th=[ 5080], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[ 12125], 99.50th=[ 13829], 99.90th=[ 22938], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 408, max= 808, per=24.99%, avg=561.95, stdev=87.46, samples=22 iops : min= 102, max= 202, avg=140.41, stdev=21.90, samples=22 lat (usec) : 100=1.89%, 250=0.20%, 500=1.33%, 750=15.17%, 1000=1.95% lat (msec) : 2=6.38%, 4=37.40%, 10=33.30%, 20=2.28%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.12%, ctx=5508, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2149KiB/s (2200kB/s), 535KiB/s-547KiB/s (548kB/s-560kB/s), io=23.4MiB (24.6MB), run=11071-11175msec WRITE: bw=2250KiB/s (2304kB/s), 562KiB/s-572KiB/s (575kB/s-585kB/s), io=24.6MiB (25.7MB), run=11071-11175msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1866: Tue Apr 8 14:42:28 2025 read: IOPS=102, BW=411KiB/s (420kB/s)(5992KiB/14595msec) clat (usec): min=1569, max=154511, avg=3941.33, stdev=4884.51 lat (usec): min=1570, max=154513, avg=3941.91, stdev=4884.54 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1762], 10.00th=[ 1844], 20.00th=[ 1975], | 30.00th=[ 2114], 40.00th=[ 2278], 50.00th=[ 2573], 60.00th=[ 2900], | 70.00th=[ 3523], 80.00th=[ 5145], 90.00th=[ 8717], 95.00th=[ 10683], | 99.00th=[ 14484], 99.50th=[ 15664], 99.90th=[ 18482], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 192, max= 904, per=25.06%, avg=408.72, stdev=189.34, samples=29 iops : min= 48, max= 226, avg=102.10, stdev=47.39, samples=29 write: IOPS=107, BW=431KiB/s (442kB/s)(6296KiB/14595msec) clat (usec): min=2199, max=27766, avg=5486.18, stdev=3403.20 lat (usec): min=2199, max=27767, avg=5486.86, stdev=3403.19 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2638], 10.00th=[ 2835], 20.00th=[ 3064], | 30.00th=[ 3261], 40.00th=[ 3523], 50.00th=[ 3949], 60.00th=[ 4817], | 70.00th=[ 6128], 80.00th=[ 7832], 90.00th=[10159], 95.00th=[12125], | 99.00th=[17695], 99.50th=[21627], 99.90th=[26084], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 224, max= 896, per=25.09%, avg=428.59, stdev=190.77, samples=29 iops : min= 56, max= 224, avg=107.07, stdev=47.74, samples=29 lat (msec) : 2=11.20%, 4=50.91%, 10=29.36%, 20=8.17%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.46%, ctx=3545, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1867: Tue Apr 8 14:42:28 2025 read: IOPS=100, BW=404KiB/s (413kB/s)(5940KiB/14716msec) clat (usec): min=1587, max=159592, avg=3951.76, stdev=4950.33 lat (usec): min=1588, max=159593, avg=3952.34, stdev=4950.33 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1975], | 30.00th=[ 2114], 40.00th=[ 2278], 50.00th=[ 2606], 60.00th=[ 3064], | 70.00th=[ 3785], 80.00th=[ 5342], 90.00th=[ 8160], 95.00th=[ 10290], | 99.00th=[ 14353], 99.50th=[ 16057], 99.90th=[ 21627], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 168, max= 824, per=24.62%, avg=401.55, stdev=196.86, samples=29 iops : min= 42, max= 206, avg=100.31, stdev=49.25, samples=29 write: IOPS=107, BW=431KiB/s (442kB/s)(6348KiB/14716msec) clat (usec): min=2091, max=29241, avg=5540.67, stdev=3451.80 lat (usec): min=2092, max=29242, avg=5541.39, stdev=3451.82 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3064], | 30.00th=[ 3261], 40.00th=[ 3490], 50.00th=[ 3982], 60.00th=[ 4883], | 70.00th=[ 6390], 80.00th=[ 7963], 90.00th=[10159], 95.00th=[11731], | 99.00th=[18744], 99.50th=[22152], 99.90th=[26870], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 224, max= 800, per=24.90%, avg=425.28, stdev=188.79, samples=29 iops : min= 56, max= 200, avg=106.24, stdev=47.24, samples=29 lat (msec) : 2=10.32%, 4=50.46%, 10=31.25%, 20=7.49%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=11.42%, ctx=3548, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1868: Tue Apr 8 14:42:28 2025 read: IOPS=103, BW=413KiB/s (423kB/s)(6028KiB/14591msec) clat (usec): min=1513, max=24441, avg=3864.72, stdev=2850.20 lat (usec): min=1514, max=24442, avg=3865.42, stdev=2850.28 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1844], 20.00th=[ 1958], | 30.00th=[ 2114], 40.00th=[ 2311], 50.00th=[ 2638], 60.00th=[ 3097], | 70.00th=[ 3818], 80.00th=[ 5276], 90.00th=[ 8356], 95.00th=[10290], | 99.00th=[13304], 99.50th=[14353], 99.90th=[20841], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 168, max= 808, per=25.25%, avg=411.76, stdev=184.08, samples=29 iops : min= 42, max= 202, avg=102.86, stdev=46.05, samples=29 write: IOPS=107, BW=429KiB/s (439kB/s)(6260KiB/14591msec) clat (msec): min=2, max=163, avg= 5.56, stdev= 5.20 lat (msec): min=2, max=163, avg= 5.57, stdev= 5.20 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 26], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 216, max= 800, per=24.95%, avg=426.10, stdev=193.46, samples=29 iops : min= 54, max= 200, avg=106.45, stdev=48.41, samples=29 lat (msec) : 2=11.10%, 4=49.64%, 10=31.25%, 20=7.68%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.42%, ctx=3550, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1869: Tue Apr 8 14:42:28 2025 read: IOPS=103, BW=412KiB/s (422kB/s)(6052KiB/14684msec) clat (usec): min=1433, max=24419, avg=3806.31, stdev=2908.61 lat (usec): min=1434, max=24419, avg=3806.92, stdev=2908.58 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1975], | 30.00th=[ 2114], 40.00th=[ 2311], 50.00th=[ 2573], 60.00th=[ 2900], | 70.00th=[ 3621], 80.00th=[ 5145], 90.00th=[ 8291], 95.00th=[10552], | 99.00th=[13960], 99.50th=[15270], 99.90th=[21365], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 192, max= 864, per=25.03%, avg=408.17, stdev=188.56, samples=29 iops : min= 48, max= 216, avg=101.97, stdev=47.18, samples=29 write: IOPS=106, BW=425KiB/s (435kB/s)(6236KiB/14684msec) clat (msec): min=2, max=164, avg= 5.69, stdev= 5.35 lat (msec): min=2, max=164, avg= 5.69, stdev= 5.35 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 28], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 208, max= 840, per=24.64%, avg=420.86, stdev=192.66, samples=29 iops : min= 52, max= 210, avg=105.14, stdev=48.22, samples=29 lat (msec) : 2=11.07%, 4=50.55%, 10=29.59%, 20=8.24%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.30%, ctx=3542, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1632KiB/s (1671kB/s), 404KiB/s-413KiB/s (413kB/s-423kB/s), io=23.4MiB (24.6MB), run=14591-14716msec WRITE: bw=1708KiB/s (1749kB/s), 425KiB/s-431KiB/s (435kB/s-442kB/s), io=24.6MiB (25.7MB), run=14591-14716msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1902: Tue Apr 8 14:42:32 2025 read: IOPS=117, BW=1887KiB/s (1932kB/s)(5920KiB/3138msec) clat (usec): min=70, max=18163, avg=4471.82, stdev=2881.78 lat (usec): min=70, max=18163, avg=4472.44, stdev=2881.80 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 84], 10.00th=[ 135], 20.00th=[ 2540], | 30.00th=[ 2999], 40.00th=[ 3556], 50.00th=[ 4113], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 6390], 90.00th=[ 7963], 95.00th=[ 9765], | 99.00th=[14746], 99.50th=[15139], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1408, max= 2144, per=24.49%, avg=1850.00, stdev=266.89, samples=6 iops : min= 88, max= 134, avg=115.50, stdev=16.61, samples=6 write: IOPS=126, BW=2029KiB/s (2078kB/s)(6368KiB/3138msec) clat (usec): min=650, max=18973, avg=3665.70, stdev=3130.48 lat (usec): min=651, max=18974, avg=3667.60, stdev=3130.87 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 914], | 30.00th=[ 1106], 40.00th=[ 1631], 50.00th=[ 2737], 60.00th=[ 3654], | 70.00th=[ 4817], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[ 9896], | 99.00th=[13829], 99.50th=[17695], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1664, max= 2240, per=24.96%, avg=1983.33, stdev=261.74, samples=6 iops : min= 104, max= 140, avg=123.83, stdev=16.40, samples=6 lat (usec) : 100=4.30%, 250=0.78%, 500=0.26%, 750=1.17%, 1000=12.11% lat (msec) : 2=8.33%, 4=28.39%, 10=40.36%, 20=4.30% cpu : usr=0.13%, sys=24.61%, ctx=1553, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1903: Tue Apr 8 14:42:32 2025 read: IOPS=113, BW=1823KiB/s (1867kB/s)(5744KiB/3150msec) clat (usec): min=68, max=16674, avg=4487.11, stdev=2523.20 lat (usec): min=69, max=16675, avg=4487.67, stdev=2523.21 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 83], 10.00th=[ 145], 20.00th=[ 2606], | 30.00th=[ 3097], 40.00th=[ 3752], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5604], 80.00th=[ 6456], 90.00th=[ 7308], 95.00th=[ 8455], | 99.00th=[12649], 99.50th=[12780], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw ( KiB/s): min= 1472, max= 2080, per=23.66%, avg=1786.67, stdev=211.70, samples=6 iops : min= 92, max= 130, avg=111.67, stdev=13.23, samples=6 write: IOPS=129, BW=2077KiB/s (2127kB/s)(6544KiB/3150msec) clat (usec): min=617, max=17509, avg=3670.16, stdev=2955.97 lat (usec): min=618, max=17510, avg=3671.34, stdev=2955.89 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 914], | 30.00th=[ 1254], 40.00th=[ 2409], 50.00th=[ 2900], 60.00th=[ 3523], | 70.00th=[ 4621], 80.00th=[ 5866], 90.00th=[ 8160], 95.00th=[10290], | 99.00th=[11863], 99.50th=[12387], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1568, max= 2496, per=25.44%, avg=2021.33, stdev=375.05, samples=6 iops : min= 98, max= 156, avg=126.33, stdev=23.44, samples=6 lat (usec) : 100=3.78%, 250=0.91%, 750=1.69%, 1000=10.55% lat (msec) : 2=7.03%, 4=31.77%, 10=40.10%, 20=4.17% cpu : usr=0.16%, sys=25.34%, ctx=1622, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1904: Tue Apr 8 14:42:32 2025 read: IOPS=126, BW=2021KiB/s (2070kB/s)(6256KiB/3095msec) clat (usec): min=59, max=15352, avg=4372.25, stdev=2552.60 lat (usec): min=59, max=15353, avg=4372.84, stdev=2552.60 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 83], 10.00th=[ 2040], 20.00th=[ 2638], | 30.00th=[ 2999], 40.00th=[ 3523], 50.00th=[ 3982], 60.00th=[ 4555], | 70.00th=[ 5342], 80.00th=[ 6259], 90.00th=[ 7373], 95.00th=[ 8586], | 99.00th=[13698], 99.50th=[15139], 99.90th=[15401], 99.95th=[15401], | 99.99th=[15401] bw ( KiB/s): min= 1472, max= 2272, per=26.83%, avg=2026.67, stdev=284.54, samples=6 iops : min= 92, max= 142, avg=126.67, stdev=17.78, samples=6 write: IOPS=121, BW=1949KiB/s (1996kB/s)(6032KiB/3095msec) clat (usec): min=656, max=15224, avg=3648.74, stdev=3030.40 lat (usec): min=657, max=15225, avg=3649.56, stdev=3030.40 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 930], | 30.00th=[ 1188], 40.00th=[ 2245], 50.00th=[ 2802], 60.00th=[ 3523], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 7898], 95.00th=[10028], | 99.00th=[13566], 99.50th=[14484], 99.90th=[15270], 99.95th=[15270], | 99.99th=[15270] bw ( KiB/s): min= 1504, max= 2304, per=23.96%, avg=1904.00, stdev=256.60, samples=6 iops : min= 94, max= 144, avg=119.00, stdev=16.04, samples=6 lat (usec) : 100=3.91%, 250=0.65%, 750=1.17%, 1000=10.81% lat (msec) : 2=7.29%, 4=33.72%, 10=38.28%, 20=4.17% cpu : usr=0.23%, sys=24.69%, ctx=1565, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1905: Tue Apr 8 14:42:32 2025 read: IOPS=118, BW=1902KiB/s (1948kB/s)(6032KiB/3171msec) clat (usec): min=58, max=16965, avg=4378.17, stdev=2848.51 lat (usec): min=58, max=16966, avg=4378.73, stdev=2848.52 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 79], 10.00th=[ 87], 20.00th=[ 2507], | 30.00th=[ 3064], 40.00th=[ 3556], 50.00th=[ 4146], 60.00th=[ 4686], | 70.00th=[ 5473], 80.00th=[ 6325], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[14091], 99.50th=[15795], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1341, max= 2304, per=24.43%, avg=1844.83, stdev=328.92, samples=6 iops : min= 83, max= 144, avg=115.17, stdev=20.81, samples=6 write: IOPS=123, BW=1973KiB/s (2020kB/s)(6256KiB/3171msec) clat (usec): min=651, max=17622, avg=3773.29, stdev=3182.73 lat (usec): min=652, max=17622, avg=3774.13, stdev=3182.74 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 775], 10.00th=[ 816], 20.00th=[ 889], | 30.00th=[ 1139], 40.00th=[ 2147], 50.00th=[ 2933], 60.00th=[ 3720], | 70.00th=[ 4752], 80.00th=[ 6390], 90.00th=[ 8586], 95.00th=[10290], | 99.00th=[13829], 99.50th=[14091], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1628, max= 2368, per=24.42%, avg=1940.67, stdev=282.53, samples=6 iops : min= 101, max= 148, avg=121.17, stdev=17.83, samples=6 lat (usec) : 100=5.73%, 250=1.17%, 500=0.13%, 750=1.69%, 1000=10.94% lat (msec) : 2=7.68%, 4=28.39%, 10=39.71%, 20=4.56% cpu : usr=0.16%, sys=24.42%, ctx=1495, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7553KiB/s (7735kB/s), 1823KiB/s-2021KiB/s (1867kB/s-2070kB/s), io=23.4MiB (24.5MB), run=3095-3171msec WRITE: bw=7947KiB/s (8138kB/s), 1949KiB/s-2077KiB/s (1996kB/s-2127kB/s), io=24.6MiB (25.8MB), run=3095-3171msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1897: Tue Apr 8 14:42:33 2025 read: IOPS=88, BW=1414KiB/s (1448kB/s)(5920KiB/4186msec) clat (usec): min=1681, max=18436, avg=4411.36, stdev=3188.54 lat (usec): min=1682, max=18437, avg=4411.94, stdev=3188.54 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3097], 60.00th=[ 3687], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 9765], 95.00th=[12125], | 99.00th=[15401], 99.50th=[17171], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 960, max= 2304, per=24.26%, avg=1388.00, stdev=476.46, samples=8 iops : min= 60, max= 144, avg=86.75, stdev=29.78, samples=8 write: IOPS=95, BW=1521KiB/s (1558kB/s)(6368KiB/4186msec) clat (usec): min=2591, max=30101, avg=6388.36, stdev=3822.75 lat (usec): min=2592, max=30102, avg=6389.33, stdev=3822.76 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3097], 10.00th=[ 3261], 20.00th=[ 3556], | 30.00th=[ 3949], 40.00th=[ 4359], 50.00th=[ 4883], 60.00th=[ 5735], | 70.00th=[ 6849], 80.00th=[ 8979], 90.00th=[12125], 95.00th=[13698], | 99.00th=[20579], 99.50th=[23200], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 1056, max= 2560, per=24.78%, avg=1492.00, stdev=485.29, samples=8 iops : min= 66, max= 160, avg=93.25, stdev=30.33, samples=8 lat (msec) : 2=1.82%, 4=45.96%, 10=39.58%, 20=12.11%, 50=0.52% cpu : usr=0.10%, sys=11.95%, ctx=943, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1898: Tue Apr 8 14:42:33 2025 read: IOPS=85, BW=1373KiB/s (1406kB/s)(5744KiB/4183msec) clat (usec): min=1816, max=21067, avg=4586.28, stdev=3312.45 lat (usec): min=1816, max=21068, avg=4586.87, stdev=3312.45 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 3294], 60.00th=[ 3752], | 70.00th=[ 4555], 80.00th=[ 6390], 90.00th=[ 9765], 95.00th=[11863], | 99.00th=[16057], 99.50th=[17695], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 928, max= 2240, per=23.70%, avg=1356.00, stdev=480.13, samples=8 iops : min= 58, max= 140, avg=84.75, stdev=30.01, samples=8 write: IOPS=97, BW=1564KiB/s (1602kB/s)(6544KiB/4183msec) clat (usec): min=2387, max=23341, avg=6122.11, stdev=3303.61 lat (usec): min=2388, max=23342, avg=6123.11, stdev=3303.58 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3556], | 30.00th=[ 3916], 40.00th=[ 4424], 50.00th=[ 4948], 60.00th=[ 5866], | 70.00th=[ 6915], 80.00th=[ 8160], 90.00th=[10814], 95.00th=[13304], | 99.00th=[16909], 99.50th=[17433], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 1088, max= 2688, per=25.38%, avg=1528.00, stdev=525.74, samples=8 iops : min= 68, max= 168, avg=95.50, stdev=32.86, samples=8 lat (msec) : 2=1.56%, 4=44.92%, 10=42.32%, 20=10.94%, 50=0.26% cpu : usr=0.10%, sys=12.00%, ctx=916, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1899: Tue Apr 8 14:42:33 2025 read: IOPS=94, BW=1516KiB/s (1553kB/s)(6256KiB/4126msec) clat (usec): min=1734, max=22536, avg=4705.49, stdev=3549.43 lat (usec): min=1735, max=22537, avg=4706.14, stdev=3549.45 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2008], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2507], 40.00th=[ 2769], 50.00th=[ 3261], 60.00th=[ 3851], | 70.00th=[ 4686], 80.00th=[ 6587], 90.00th=[10159], 95.00th=[12518], | 99.00th=[17171], 99.50th=[21365], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1056, max= 2880, per=26.50%, avg=1516.00, stdev=603.52, samples=8 iops : min= 66, max= 180, avg=94.75, stdev=37.72, samples=8 write: IOPS=91, BW=1462KiB/s (1497kB/s)(6032KiB/4126msec) clat (usec): min=2472, max=20271, avg=6010.46, stdev=3274.88 lat (usec): min=2473, max=20272, avg=6011.49, stdev=3274.85 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2868], 10.00th=[ 3032], 20.00th=[ 3359], | 30.00th=[ 3720], 40.00th=[ 4146], 50.00th=[ 4883], 60.00th=[ 5800], | 70.00th=[ 6980], 80.00th=[ 8356], 90.00th=[10552], 95.00th=[12780], | 99.00th=[16319], 99.50th=[19268], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 992, max= 2240, per=23.65%, avg=1424.00, stdev=439.10, samples=8 iops : min= 62, max= 140, avg=89.00, stdev=27.44, samples=8 lat (msec) : 2=2.60%, 4=47.14%, 10=38.93%, 20=10.94%, 50=0.39% cpu : usr=0.12%, sys=12.27%, ctx=941, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1900: Tue Apr 8 14:42:33 2025 read: IOPS=90, BW=1452KiB/s (1487kB/s)(6032KiB/4154msec) clat (usec): min=1776, max=18177, avg=4672.29, stdev=3447.35 lat (usec): min=1777, max=18178, avg=4673.07, stdev=3447.20 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2606], 40.00th=[ 2835], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 4621], 80.00th=[ 6325], 90.00th=[10552], 95.00th=[12649], | 99.00th=[17171], 99.50th=[17695], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1056, max= 2592, per=24.89%, avg=1424.00, stdev=513.43, samples=8 iops : min= 66, max= 162, avg=89.00, stdev=32.09, samples=8 write: IOPS=94, BW=1506KiB/s (1542kB/s)(6256KiB/4154msec) clat (usec): min=2354, max=33190, avg=6055.80, stdev=3444.13 lat (usec): min=2354, max=33191, avg=6056.78, stdev=3444.10 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2999], 10.00th=[ 3261], 20.00th=[ 3490], | 30.00th=[ 3785], 40.00th=[ 4228], 50.00th=[ 4883], 60.00th=[ 5735], | 70.00th=[ 6718], 80.00th=[ 8225], 90.00th=[10552], 95.00th=[12649], | 99.00th=[17695], 99.50th=[21103], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 1056, max= 2304, per=24.58%, avg=1480.00, stdev=500.37, samples=8 iops : min= 66, max= 144, avg=92.50, stdev=31.27, samples=8 lat (msec) : 2=3.26%, 4=44.66%, 10=41.02%, 20=10.68%, 50=0.39% cpu : usr=0.10%, sys=11.97%, ctx=971, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5722KiB/s (5859kB/s), 1373KiB/s-1516KiB/s (1406kB/s-1553kB/s), io=23.4MiB (24.5MB), run=4126-4186msec WRITE: bw=6020KiB/s (6165kB/s), 1462KiB/s-1564KiB/s (1497kB/s-1602kB/s), io=24.6MiB (25.8MB), run=4126-4186msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1914: Tue Apr 8 14:42:35 2025 read: IOPS=12, BW=12.7MiB/s (13.3MB/s)(7168KiB/550msec) clat (usec): min=361, max=83635, avg=32782.85, stdev=25556.38 lat (usec): min=362, max=83636, avg=32783.56, stdev=25556.43 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[23462], | 30.00th=[24511], 40.00th=[24511], 50.00th=[26084], 60.00th=[29754], | 70.00th=[29754], 80.00th=[41681], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min=12263, max=12263, per=35.57%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9309KiB/s (9533kB/s)(5120KiB/550msec) clat (usec): min=27017, max=61584, avg=49898.23, stdev=13949.17 lat (usec): min=28292, max=61597, avg=50174.91, stdev=13435.85 clat percentiles (usec): | 1.00th=[27132], 5.00th=[27132], 10.00th=[27132], 20.00th=[27132], | 30.00th=[49021], 40.00th=[49021], 50.00th=[51643], 60.00th=[51643], | 70.00th=[60556], 80.00th=[60556], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 8175, max= 8175, per=16.94%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 100=33.33% cpu : usr=0.00%, sys=30.05%, ctx=485, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1915: Tue Apr 8 14:42:35 2025 read: IOPS=3, BW=3448KiB/s (3531kB/s)(2048KiB/594msec) clat (usec): min=29797, max=32039, avg=30918.59, stdev=1584.79 lat (usec): min=29799, max=32039, avg=30919.47, stdev=1584.56 clat percentiles (usec): | 1.00th=[29754], 5.00th=[29754], 10.00th=[29754], 20.00th=[29754], | 30.00th=[29754], 40.00th=[29754], 50.00th=[29754], 60.00th=[32113], | 70.00th=[32113], 80.00th=[32113], 90.00th=[32113], 95.00th=[32113], | 99.00th=[32113], 99.50th=[32113], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 3580, max= 3580, per=10.38%, avg=3580.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.8MiB/s (17.7MB/s)(10.0MiB/594msec) clat (usec): min=22554, max=79227, avg=47893.95, stdev=19158.72 lat (usec): min=22610, max=79287, avg=47940.38, stdev=19163.84 clat percentiles (usec): | 1.00th=[22676], 5.00th=[22676], 10.00th=[22676], 20.00th=[25297], | 30.00th=[25560], 40.00th=[38536], 50.00th=[50070], 60.00th=[55313], | 70.00th=[62129], 80.00th=[62129], 90.00th=[62653], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min=16111, max=16111, per=33.38%, avg=16111.00, stdev= 0.00, samples=1 iops : min= 15, max= 15, avg=15.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=50.00% cpu : usr=0.00%, sys=33.05%, ctx=479, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1916: Tue Apr 8 14:42:35 2025 read: IOPS=13, BW=13.1MiB/s (13.8MB/s)(6144KiB/457msec) clat (usec): min=394, max=50321, avg=28416.01, stdev=17684.31 lat (usec): min=395, max=50322, avg=28416.67, stdev=17684.29 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 396], 10.00th=[ 396], 20.00th=[22152], | 30.00th=[22152], 40.00th=[23200], 50.00th=[23200], 60.00th=[31327], | 70.00th=[43254], 80.00th=[43254], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] write: IOPS=13, BW=13.1MiB/s (13.8MB/s)(6144KiB/457msec) clat (usec): min=25049, max=97032, avg=46821.55, stdev=29021.96 lat (usec): min=25068, max=97050, avg=46861.35, stdev=29008.66 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[27395], | 30.00th=[27395], 40.00th=[28705], 50.00th=[28705], 60.00th=[35914], | 70.00th=[66847], 80.00th=[66847], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] lat (usec) : 500=8.33% lat (msec) : 50=66.67%, 100=25.00% cpu : usr=0.00%, sys=37.06%, ctx=393, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1917: Tue Apr 8 14:42:35 2025 read: IOPS=9, BW=9.98MiB/s (10.5MB/s)(5120KiB/501msec) clat (usec): min=377, max=31800, avg=11185.15, stdev=15063.13 lat (usec): min=378, max=31801, avg=11185.81, stdev=15063.14 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 379], 10.00th=[ 379], 20.00th=[ 379], | 30.00th=[ 392], 40.00th=[ 392], 50.00th=[ 529], 60.00th=[ 529], | 70.00th=[22938], 80.00th=[22938], 90.00th=[31851], 95.00th=[31851], | 99.00th=[31851], 99.50th=[31851], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 8192, max= 8192, per=23.76%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.0MiB/s (14.7MB/s)(7168KiB/501msec) clat (msec): min=32, max=112, avg=61.75, stdev=28.25 lat (msec): min=32, max=112, avg=61.82, stdev=28.23 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 36], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 71], | 70.00th=[ 71], 80.00th=[ 80], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min=14336, max=14336, per=29.70%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=41.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=30.80%, ctx=445, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=33.7MiB/s (35.3MB/s), 3448KiB/s-13.1MiB/s (3531kB/s-13.8MB/s), io=20.0MiB (20.0MB), run=457-594msec WRITE: bw=47.1MiB/s (49.4MB/s), 9309KiB/s-16.8MiB/s (9533kB/s-17.7MB/s), io=28.0MiB (29.4MB), run=457-594msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1918: Tue Apr 8 14:42:35 2025 read: IOPS=10, BW=10.6MiB/s (11.2MB/s)(7168KiB/658msec) clat (msec): min=13, max=114, avg=59.37, stdev=41.17 lat (msec): min=13, max=114, avg=59.37, stdev=41.17 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 52], 60.00th=[ 79], | 70.00th=[ 79], 80.00th=[ 106], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min=10240, max=10240, per=33.65%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7781KiB/s (7968kB/s)(5120KiB/658msec) clat (usec): min=18767, max=57944, avg=32294.19, stdev=16024.88 lat (usec): min=18813, max=57999, avg=32329.98, stdev=16030.23 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[18744], | 30.00th=[19006], 40.00th=[19006], 50.00th=[30540], 60.00th=[30540], | 70.00th=[34866], 80.00th=[34866], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 2048, max= 2048, per=4.81%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=25.00%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=13.09%, ctx=64, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1919: Tue Apr 8 14:42:35 2025 read: IOPS=2, BW=3043KiB/s (3116kB/s)(2048KiB/673msec) clat (usec): min=15358, max=49028, avg=32193.17, stdev=23808.05 lat (usec): min=15358, max=49029, avg=32194.08, stdev=23808.47 clat percentiles (usec): | 1.00th=[15401], 5.00th=[15401], 10.00th=[15401], 20.00th=[15401], | 30.00th=[15401], 40.00th=[15401], 50.00th=[15401], 60.00th=[49021], | 70.00th=[49021], 80.00th=[49021], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 4096, max= 4096, per=13.46%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.9MiB/s (15.6MB/s)(10.0MiB/673msec) clat (usec): min=25485, max=85174, avg=54434.04, stdev=19000.37 lat (usec): min=25527, max=85201, avg=54486.44, stdev=19007.21 clat percentiles (usec): | 1.00th=[25560], 5.00th=[25560], 10.00th=[25560], 20.00th=[38536], | 30.00th=[39584], 40.00th=[40633], 50.00th=[45876], 60.00th=[59507], | 70.00th=[69731], 80.00th=[69731], 90.00th=[70779], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min=10240, max=10240, per=24.04%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=41.67% cpu : usr=0.45%, sys=15.03%, ctx=74, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1920: Tue Apr 8 14:42:35 2025 read: IOPS=9, BW=9.88MiB/s (10.4MB/s)(6144KiB/607msec) clat (usec): min=18352, max=76371, avg=40137.21, stdev=21130.59 lat (usec): min=18352, max=76372, avg=40138.07, stdev=21130.44 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[22676], | 30.00th=[22676], 40.00th=[34866], 50.00th=[34866], 60.00th=[38011], | 70.00th=[50594], 80.00th=[50594], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 8175, max= 8175, per=26.86%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.88MiB/s (10.4MB/s)(6144KiB/607msec) clat (msec): min=18, max=108, avg=58.70, stdev=35.08 lat (msec): min=18, max=108, avg=58.75, stdev=35.09 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 60], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 109], 95.00th=[ 109], | 99.00th=[ 109], 99.50th=[ 109], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 8175, max= 8175, per=19.19%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=14.36%, ctx=175, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1921: Tue Apr 8 14:42:35 2025 read: IOPS=8, BW=8393KiB/s (8595kB/s)(5120KiB/610msec) clat (usec): min=12432, max=85317, avg=32747.06, stdev=29968.23 lat (usec): min=12433, max=85318, avg=32747.68, stdev=29968.22 clat percentiles (usec): | 1.00th=[12387], 5.00th=[12387], 10.00th=[12387], 20.00th=[12387], | 30.00th=[15664], 40.00th=[15664], 50.00th=[22938], 60.00th=[22938], | 70.00th=[27395], 80.00th=[27395], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 4096, max= 4096, per=13.46%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.5MiB/s (12.0MB/s)(7168KiB/610msec) clat (msec): min=16, max=115, avg=46.80, stdev=37.11 lat (msec): min=16, max=115, avg=46.85, stdev=37.12 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 27], 60.00th=[ 41], | 70.00th=[ 41], 80.00th=[ 81], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 8192, max= 8192, per=19.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=50.00%, 100=16.67%, 250=8.33% cpu : usr=0.16%, sys=13.46%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.7MiB/s (31.2MB/s), 3043KiB/s-10.6MiB/s (3116kB/s-11.2MB/s), io=20.0MiB (20.0MB), run=607-673msec WRITE: bw=41.6MiB/s (43.6MB/s), 7781KiB/s-14.9MiB/s (7968kB/s-15.6MB/s), io=28.0MiB (29.4MB), run=607-673msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1928: Tue Apr 8 14:42:37 2025 read: IOPS=7, BW=29.3MiB/s (30.8MB/s)(12.0MiB/409msec) clat (msec): min=113, max=160, avg=133.91, stdev=23.67 lat (msec): min=113, max=160, avg=133.91, stdev=23.67 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 128], 50.00th=[ 128], 60.00th=[ 128], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=34.07%, ctx=1742, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1929: Tue Apr 8 14:42:37 2025 read: IOPS=1, BW=5107KiB/s (5230kB/s)(4096KiB/802msec) clat (nsec): min=154294k, max=154294k, avg=154293835.00, stdev= 0.00 lat (nsec): min=154296k, max=154296k, avg=154296355.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8192, max= 8192, per=40.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.97MiB/s (10.5MB/s)(8192KiB/802msec) clat (msec): min=313, max=321, avg=317.58, stdev= 5.38 lat (msec): min=313, max=321, avg=317.75, stdev= 5.48 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 8192, max= 8192, per=20.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.12%, sys=26.72%, ctx=970, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1930: Tue Apr 8 14:42:37 2025 write: IOPS=4, BW=18.2MiB/s (19.1MB/s)(12.0MiB/658msec) clat (msec): min=117, max=308, avg=216.59, stdev=95.99 lat (msec): min=117, max=309, avg=216.85, stdev=96.05 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=20.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.30%, sys=40.49%, ctx=49, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1931: Tue Apr 8 14:42:37 2025 write: IOPS=5, BW=22.7MiB/s (23.8MB/s)(12.0MiB/529msec) clat (msec): min=132, max=234, avg=168.80, stdev=56.61 lat (msec): min=132, max=234, avg=169.06, stdev=56.61 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min=16384, max=16384, per=40.10%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=40.91%, ctx=65, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.9MiB/s (20.9MB/s), 5107KiB/s-29.3MiB/s (5230kB/s-30.8MB/s), io=16.0MiB (16.8MB), run=409-802msec WRITE: bw=39.9MiB/s (41.8MB/s), 9.97MiB/s-22.7MiB/s (10.5MB/s-23.8MB/s), io=32.0MiB (33.6MB), run=529-802msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1932: Tue Apr 8 14:42:37 2025 read: IOPS=4, BW=16.7MiB/s (17.5MB/s)(12.0MiB/719msec) clat (msec): min=81, max=378, avg=235.27, stdev=148.92 lat (msec): min=81, max=378, avg=235.27, stdev=148.92 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 8192, max= 8192, per=42.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.98%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1933: Tue Apr 8 14:42:37 2025 read: IOPS=1, BW=4842KiB/s (4958kB/s)(4096KiB/846msec) clat (nsec): min=364806k, max=364806k, avg=364806300.00, stdev= 0.00 lat (nsec): min=364807k, max=364807k, avg=364807470.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8110, max= 8110, per=41.88%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9683KiB/s (9916kB/s)(8192KiB/846msec) clat (msec): min=117, max=323, avg=220.59, stdev=145.74 lat (msec): min=117, max=323, avg=220.82, stdev=145.75 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 118], 50.00th=[ 118], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.24%, sys=14.08%, ctx=98, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1934: Tue Apr 8 14:42:37 2025 write: IOPS=3, BW=15.6MiB/s (16.4MB/s)(12.0MiB/769msec) clat (msec): min=82, max=449, avg=253.95, stdev=184.90 lat (msec): min=82, max=449, avg=254.18, stdev=184.96 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 8192, max= 8192, per=21.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.13%, sys=10.03%, ctx=28, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1935: Tue Apr 8 14:42:37 2025 write: IOPS=4, BW=19.0MiB/s (19.9MB/s)(12.0MiB/631msec) clat (msec): min=137, max=261, avg=207.26, stdev=63.59 lat (msec): min=137, max=262, avg=207.50, stdev=63.56 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min=16384, max=16384, per=42.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.16%, sys=15.71%, ctx=74, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.9MiB/s (19.8MB/s), 4842KiB/s-16.7MiB/s (4958kB/s-17.5MB/s), io=16.0MiB (16.8MB), run=719-846msec WRITE: bw=37.8MiB/s (39.7MB/s), 9683KiB/s-19.0MiB/s (9916kB/s-19.9MB/s), io=32.0MiB (33.6MB), run=631-846msec