== sanity test 398b: DIO and buffer IO race ============== 15:22:44 (1741634564) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.19575 s, 42.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25073: Mon Mar 10 15:23:00 2025 read: IOPS=120, BW=480KiB/s (492kB/s)(5992KiB/12478msec) clat (usec): min=57, max=157469, avg=4555.30, stdev=4675.76 lat (usec): min=57, max=157470, avg=4556.38, stdev=4675.75 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 151], 10.00th=[ 2147], 20.00th=[ 2573], | 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 4015], 60.00th=[ 4686], | 70.00th=[ 5407], 80.00th=[ 6259], 90.00th=[ 7504], 95.00th=[ 8717], | 99.00th=[ 13173], 99.50th=[ 14615], 99.90th=[ 19268], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 328, max= 584, per=24.81%, avg=474.62, stdev=57.83, samples=24 iops : min= 82, max= 146, avg=118.62, stdev=14.47, samples=24 write: IOPS=126, BW=505KiB/s (517kB/s)(6296KiB/12478msec) clat (usec): min=416, max=23572, avg=3548.72, stdev=3280.74 lat (usec): min=416, max=23573, avg=3549.99, stdev=3280.40 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 627], | 30.00th=[ 824], 40.00th=[ 1975], 50.00th=[ 2868], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 6128], 90.00th=[ 7832], 95.00th=[ 9503], | 99.00th=[14353], 99.50th=[17433], 99.90th=[23200], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 376, max= 632, per=24.78%, avg=496.29, stdev=58.03, samples=24 iops : min= 94, max= 158, avg=124.04, stdev=14.56, samples=24 lat (usec) : 100=1.99%, 250=0.62%, 500=0.36%, 750=13.54%, 1000=3.52% lat (msec) : 2=4.49%, 4=32.68%, 10=39.39%, 20=3.22%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.31%, sys=22.25%, ctx=5684, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25075: Mon Mar 10 15:23:00 2025 read: IOPS=118, BW=474KiB/s (486kB/s)(5940KiB/12526msec) clat (usec): min=50, max=161708, avg=4677.35, stdev=4943.81 lat (usec): min=51, max=161709, avg=4678.11, stdev=4943.85 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 86], 10.00th=[ 2024], 20.00th=[ 2540], | 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 4015], 60.00th=[ 4883], | 70.00th=[ 5538], 80.00th=[ 6456], 90.00th=[ 7767], 95.00th=[ 9110], | 99.00th=[ 14746], 99.50th=[ 16712], 99.90th=[ 20055], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 240, max= 600, per=24.75%, avg=473.56, stdev=74.85, samples=25 iops : min= 60, max= 150, avg=118.36, stdev=18.73, samples=25 write: IOPS=126, BW=507KiB/s (519kB/s)(6348KiB/12526msec) clat (usec): min=425, max=19808, avg=3468.51, stdev=3108.43 lat (usec): min=425, max=19808, avg=3469.39, stdev=3108.37 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 635], | 30.00th=[ 848], 40.00th=[ 1762], 50.00th=[ 2802], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 6063], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[12518], 99.50th=[15533], 99.90th=[19530], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 288, max= 704, per=25.18%, avg=504.28, stdev=92.55, samples=25 iops : min= 72, max= 176, avg=126.04, stdev=23.17, samples=25 lat (usec) : 100=2.54%, 250=0.75%, 500=0.42%, 750=13.51%, 1000=4.07% lat (msec) : 2=4.43%, 4=32.58%, 10=37.53%, 20=4.13%, 250=0.03% cpu : usr=0.24%, sys=22.35%, ctx=5557, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25076: Mon Mar 10 15:23:00 2025 read: IOPS=120, BW=482KiB/s (493kB/s)(6028KiB/12513msec) clat (usec): min=53, max=156398, avg=4581.03, stdev=4709.08 lat (usec): min=53, max=156399, avg=4581.83, stdev=4709.21 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 106], 10.00th=[ 2114], 20.00th=[ 2606], | 30.00th=[ 2966], 40.00th=[ 3359], 50.00th=[ 3982], 60.00th=[ 4752], | 70.00th=[ 5473], 80.00th=[ 6325], 90.00th=[ 7635], 95.00th=[ 8979], | 99.00th=[ 13173], 99.50th=[ 16057], 99.90th=[ 19268], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 320, max= 600, per=25.17%, avg=481.52, stdev=69.95, samples=25 iops : min= 80, max= 150, avg=120.36, stdev=17.48, samples=25 write: IOPS=125, BW=500KiB/s (512kB/s)(6260KiB/12513msec) clat (usec): min=418, max=22484, avg=3544.02, stdev=3201.28 lat (usec): min=419, max=22485, avg=3544.90, stdev=3201.48 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 652], | 30.00th=[ 857], 40.00th=[ 1958], 50.00th=[ 2802], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 5997], 90.00th=[ 8160], 95.00th=[ 9896], | 99.00th=[12911], 99.50th=[14222], 99.90th=[21890], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 400, max= 656, per=24.92%, avg=499.16, stdev=70.30, samples=25 iops : min= 100, max= 164, avg=124.76, stdev=17.60, samples=25 lat (usec) : 100=2.34%, 250=0.55%, 500=0.42%, 750=12.83%, 1000=4.23% lat (msec) : 2=4.30%, 4=33.14%, 10=38.28%, 20=3.78%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.71%, ctx=5433, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25077: Mon Mar 10 15:23:00 2025 read: IOPS=120, BW=482KiB/s (494kB/s)(6052KiB/12551msec) clat (usec): min=50, max=168106, avg=4645.65, stdev=4868.95 lat (usec): min=51, max=168107, avg=4646.30, stdev=4868.96 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 217], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3523], 50.00th=[ 4113], 60.00th=[ 4752], | 70.00th=[ 5473], 80.00th=[ 6390], 90.00th=[ 7635], 95.00th=[ 8848], | 99.00th=[ 11994], 99.50th=[ 15008], 99.90th=[ 16712], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 296, max= 632, per=25.11%, avg=480.40, stdev=76.89, samples=25 iops : min= 74, max= 158, avg=119.96, stdev=19.25, samples=25 write: IOPS=124, BW=497KiB/s (509kB/s)(6236KiB/12551msec) clat (usec): min=452, max=24645, avg=3501.86, stdev=3259.23 lat (usec): min=453, max=24646, avg=3503.30, stdev=3259.04 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 627], | 30.00th=[ 799], 40.00th=[ 1565], 50.00th=[ 2835], 60.00th=[ 3490], | 70.00th=[ 4359], 80.00th=[ 5866], 90.00th=[ 8225], 95.00th=[10028], | 99.00th=[13173], 99.50th=[15270], 99.90th=[20841], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 376, max= 696, per=24.72%, avg=495.12, stdev=68.18, samples=25 iops : min= 94, max= 174, avg=123.64, stdev=17.08, samples=25 lat (usec) : 100=1.92%, 250=0.55%, 500=0.46%, 750=13.67%, 1000=4.13% lat (msec) : 2=4.30%, 4=32.42%, 10=38.70%, 20=3.74%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.47%, ctx=5582, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1913KiB/s (1959kB/s), 474KiB/s-482KiB/s (486kB/s-494kB/s), io=23.4MiB (24.6MB), run=12478-12551msec WRITE: bw=2003KiB/s (2051kB/s), 497KiB/s-507KiB/s (509kB/s-519kB/s), io=24.6MiB (25.7MB), run=12478-12551msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25070: Mon Mar 10 15:23:03 2025 read: IOPS=93, BW=373KiB/s (382kB/s)(5992KiB/16070msec) clat (usec): min=1602, max=160736, avg=4379.40, stdev=5223.11 lat (usec): min=1603, max=160738, avg=4380.08, stdev=5223.15 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1958], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3359], | 70.00th=[ 4113], 80.00th=[ 5932], 90.00th=[ 9372], 95.00th=[ 11863], | 99.00th=[ 16188], 99.50th=[ 18220], 99.90th=[ 22414], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 184, max= 760, per=24.92%, avg=371.34, stdev=177.80, samples=32 iops : min= 46, max= 190, avg=92.75, stdev=44.44, samples=32 write: IOPS=97, BW=392KiB/s (401kB/s)(6296KiB/16070msec) clat (usec): min=2214, max=30678, avg=6002.24, stdev=3814.16 lat (usec): min=2214, max=30679, avg=6003.03, stdev=3814.12 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2769], 10.00th=[ 2966], 20.00th=[ 3228], | 30.00th=[ 3458], 40.00th=[ 3818], 50.00th=[ 4293], 60.00th=[ 5342], | 70.00th=[ 6783], 80.00th=[ 8848], 90.00th=[11207], 95.00th=[13566], | 99.00th=[19530], 99.50th=[23987], 99.90th=[27132], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 200, max= 832, per=25.01%, avg=390.09, stdev=184.77, samples=32 iops : min= 50, max= 208, avg=97.44, stdev=46.20, samples=32 lat (msec) : 2=6.15%, 4=49.97%, 10=32.29%, 20=10.94%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.49%, ctx=3617, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25071: Mon Mar 10 15:23:03 2025 read: IOPS=92, BW=369KiB/s (378kB/s)(5940KiB/16110msec) clat (usec): min=1572, max=171153, avg=4369.16, stdev=5469.06 lat (usec): min=1572, max=171154, avg=4369.86, stdev=5469.08 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1926], 20.00th=[ 2057], | 30.00th=[ 2212], 40.00th=[ 2474], 50.00th=[ 2835], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 5932], 90.00th=[ 8848], 95.00th=[ 11731], | 99.00th=[ 17433], 99.50th=[ 19268], 99.90th=[ 26608], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 112, max= 752, per=24.66%, avg=367.41, stdev=186.77, samples=32 iops : min= 28, max= 188, avg=91.78, stdev=46.72, samples=32 write: IOPS=98, BW=394KiB/s (403kB/s)(6348KiB/16110msec) clat (usec): min=2353, max=26893, avg=6021.95, stdev=3731.51 lat (usec): min=2354, max=26894, avg=6022.68, stdev=3731.51 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2835], 10.00th=[ 3032], 20.00th=[ 3228], | 30.00th=[ 3458], 40.00th=[ 3818], 50.00th=[ 4359], 60.00th=[ 5473], | 70.00th=[ 7111], 80.00th=[ 8586], 90.00th=[10945], 95.00th=[13173], | 99.00th=[19268], 99.50th=[22938], 99.90th=[26346], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 200, max= 784, per=25.06%, avg=390.91, stdev=178.67, samples=32 iops : min= 50, max= 196, avg=97.66, stdev=44.70, samples=32 lat (msec) : 2=7.75%, 4=47.66%, 10=33.69%, 20=10.22%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=11.24%, ctx=3555, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25072: Mon Mar 10 15:23:03 2025 read: IOPS=94, BW=377KiB/s (386kB/s)(6028KiB/15999msec) clat (usec): min=1609, max=26271, avg=4301.97, stdev=3324.71 lat (usec): min=1610, max=26272, avg=4302.67, stdev=3324.92 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3359], | 70.00th=[ 4359], 80.00th=[ 6194], 90.00th=[ 9110], 95.00th=[11600], | 99.00th=[16057], 99.50th=[18482], 99.90th=[22152], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 168, max= 816, per=24.66%, avg=367.39, stdev=173.70, samples=31 iops : min= 42, max= 204, avg=91.77, stdev=43.46, samples=31 write: IOPS=97, BW=391KiB/s (401kB/s)(6260KiB/15999msec) clat (msec): min=2, max=167, avg= 6.03, stdev= 5.54 lat (msec): min=2, max=167, avg= 6.04, stdev= 5.54 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 25], 99.90th=[ 27], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 200, max= 800, per=24.36%, avg=380.03, stdev=177.14, samples=31 iops : min= 50, max= 200, avg=94.94, stdev=44.31, samples=31 lat (msec) : 2=6.90%, 4=49.28%, 10=32.81%, 20=10.32%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.33%, ctx=3550, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25074: Mon Mar 10 15:23:03 2025 read: IOPS=94, BW=377KiB/s (386kB/s)(6052KiB/16071msec) clat (usec): min=1683, max=24802, avg=4350.44, stdev=3491.37 lat (usec): min=1683, max=24802, avg=4351.07, stdev=3491.41 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3392], | 70.00th=[ 4178], 80.00th=[ 6128], 90.00th=[ 9634], 95.00th=[12125], | 99.00th=[17171], 99.50th=[19006], 99.90th=[22938], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 152, max= 768, per=25.14%, avg=374.66, stdev=183.70, samples=32 iops : min= 38, max= 192, avg=93.59, stdev=45.95, samples=32 write: IOPS=97, BW=388KiB/s (397kB/s)(6236KiB/16071msec) clat (msec): min=2, max=175, avg= 6.05, stdev= 5.68 lat (msec): min=2, max=175, avg= 6.05, stdev= 5.68 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 31], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 184, max= 776, per=24.79%, avg=386.66, stdev=187.89, samples=32 iops : min= 46, max= 194, avg=96.59, stdev=47.00, samples=32 lat (msec) : 2=6.87%, 4=50.23%, 10=31.32%, 20=10.94%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.35%, ctx=3589, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1491KiB/s (1526kB/s), 369KiB/s-377KiB/s (378kB/s-386kB/s), io=23.4MiB (24.6MB), run=15999-16110msec WRITE: bw=1561KiB/s (1598kB/s), 388KiB/s-394KiB/s (397kB/s-403kB/s), io=24.6MiB (25.7MB), run=15999-16110msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25128: Mon Mar 10 15:23:07 2025 read: IOPS=113, BW=1818KiB/s (1862kB/s)(5920KiB/3256msec) clat (usec): min=55, max=17811, avg=4628.64, stdev=2833.32 lat (usec): min=55, max=17811, avg=4633.17, stdev=2833.76 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 83], 10.00th=[ 979], 20.00th=[ 2704], | 30.00th=[ 3163], 40.00th=[ 3752], 50.00th=[ 4146], 60.00th=[ 4883], | 70.00th=[ 5538], 80.00th=[ 6521], 90.00th=[ 8029], 95.00th=[ 9896], | 99.00th=[14484], 99.50th=[16319], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1216, max= 1952, per=24.02%, avg=1744.00, stdev=273.60, samples=6 iops : min= 76, max= 122, avg=109.00, stdev=17.10, samples=6 write: IOPS=122, BW=1956KiB/s (2003kB/s)(6368KiB/3256msec) clat (usec): min=607, max=17767, avg=3688.12, stdev=3163.33 lat (usec): min=608, max=17767, avg=3688.97, stdev=3163.35 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 873], | 30.00th=[ 1188], 40.00th=[ 2147], 50.00th=[ 2999], 60.00th=[ 3720], | 70.00th=[ 4490], 80.00th=[ 5800], 90.00th=[ 8094], 95.00th=[10028], | 99.00th=[15926], 99.50th=[16712], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1440, max= 2240, per=24.93%, avg=1904.00, stdev=303.41, samples=6 iops : min= 90, max= 140, avg=119.00, stdev=18.96, samples=6 lat (usec) : 100=3.91%, 250=0.78%, 750=1.17%, 1000=12.11% lat (msec) : 2=6.90%, 4=30.86%, 10=39.19%, 20=5.08% cpu : usr=0.12%, sys=24.55%, ctx=1565, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25129: Mon Mar 10 15:23:07 2025 read: IOPS=108, BW=1742KiB/s (1783kB/s)(5744KiB/3298msec) clat (usec): min=67, max=18075, avg=4579.03, stdev=2727.30 lat (usec): min=68, max=18075, avg=4579.97, stdev=2727.49 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 80], 10.00th=[ 208], 20.00th=[ 2671], | 30.00th=[ 3163], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5014], | 70.00th=[ 5604], 80.00th=[ 6390], 90.00th=[ 7570], 95.00th=[ 9110], | 99.00th=[15926], 99.50th=[17957], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1376, max= 2176, per=23.58%, avg=1712.00, stdev=295.20, samples=6 iops : min= 86, max= 136, avg=107.00, stdev=18.45, samples=6 write: IOPS=124, BW=1984KiB/s (2032kB/s)(6544KiB/3298msec) clat (usec): min=634, max=20066, avg=3879.23, stdev=3441.87 lat (usec): min=635, max=20067, avg=3880.11, stdev=3441.90 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 881], | 30.00th=[ 1123], 40.00th=[ 2311], 50.00th=[ 2933], 60.00th=[ 3785], | 70.00th=[ 5080], 80.00th=[ 6259], 90.00th=[ 8225], 95.00th=[ 9896], | 99.00th=[15795], 99.50th=[17171], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1472, max= 2272, per=25.35%, avg=1936.00, stdev=297.27, samples=6 iops : min= 92, max= 142, avg=121.00, stdev=18.58, samples=6 lat (usec) : 100=4.04%, 250=0.65%, 750=1.56%, 1000=12.50% lat (msec) : 2=6.25%, 4=27.99%, 10=42.84%, 20=3.91%, 50=0.26% cpu : usr=0.12%, sys=24.41%, ctx=1543, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25130: Mon Mar 10 15:23:07 2025 read: IOPS=118, BW=1896KiB/s (1942kB/s)(6256KiB/3299msec) clat (usec): min=63, max=18613, avg=4693.43, stdev=2970.83 lat (usec): min=64, max=18614, avg=4694.02, stdev=2970.86 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 84], 10.00th=[ 2024], 20.00th=[ 2704], | 30.00th=[ 3097], 40.00th=[ 3654], 50.00th=[ 4293], 60.00th=[ 4948], | 70.00th=[ 5735], 80.00th=[ 6456], 90.00th=[ 7963], 95.00th=[10159], | 99.00th=[14484], 99.50th=[16909], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1216, max= 2208, per=24.90%, avg=1808.00, stdev=339.11, samples=6 iops : min= 76, max= 138, avg=113.00, stdev=21.19, samples=6 write: IOPS=114, BW=1828KiB/s (1872kB/s)(6032KiB/3299msec) clat (usec): min=643, max=15185, avg=3660.05, stdev=2989.04 lat (usec): min=644, max=15187, avg=3660.92, stdev=2989.07 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 824], 20.00th=[ 914], | 30.00th=[ 1221], 40.00th=[ 2057], 50.00th=[ 2835], 60.00th=[ 3621], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[12911], 99.50th=[15139], 99.90th=[15139], 99.95th=[15139], | 99.99th=[15139] bw ( KiB/s): min= 1312, max= 2272, per=23.18%, avg=1770.67, stdev=377.91, samples=6 iops : min= 82, max= 142, avg=110.67, stdev=23.62, samples=6 lat (usec) : 100=3.39%, 250=1.17%, 500=0.13%, 750=1.82%, 1000=10.94% lat (msec) : 2=7.16%, 4=30.21%, 10=40.76%, 20=4.43% cpu : usr=0.15%, sys=24.20%, ctx=1623, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25131: Mon Mar 10 15:23:07 2025 read: IOPS=116, BW=1863KiB/s (1908kB/s)(6032KiB/3238msec) clat (usec): min=57, max=28051, avg=4799.79, stdev=3241.31 lat (usec): min=57, max=28052, avg=4800.44, stdev=3241.29 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 73], 10.00th=[ 449], 20.00th=[ 2606], | 30.00th=[ 3195], 40.00th=[ 3687], 50.00th=[ 4359], 60.00th=[ 4948], | 70.00th=[ 5800], 80.00th=[ 6783], 90.00th=[ 8291], 95.00th=[10421], | 99.00th=[16188], 99.50th=[17171], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 1344, max= 2368, per=25.20%, avg=1829.33, stdev=355.43, samples=6 iops : min= 84, max= 148, avg=114.33, stdev=22.21, samples=6 write: IOPS=120, BW=1932KiB/s (1978kB/s)(6256KiB/3238msec) clat (usec): min=618, max=17189, avg=3612.11, stdev=2957.63 lat (usec): min=618, max=17191, avg=3614.85, stdev=2956.06 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 914], | 30.00th=[ 1139], 40.00th=[ 1991], 50.00th=[ 2835], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 7832], 95.00th=[ 9634], | 99.00th=[13435], 99.50th=[14353], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1568, max= 2272, per=25.28%, avg=1930.67, stdev=297.22, samples=6 iops : min= 98, max= 142, avg=120.67, stdev=18.58, samples=6 lat (usec) : 100=3.78%, 250=1.04%, 500=0.13%, 750=0.78%, 1000=11.98% lat (msec) : 2=7.68%, 4=28.91%, 10=40.89%, 20=4.69%, 50=0.13% cpu : usr=0.22%, sys=25.05%, ctx=1566, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7260KiB/s (7435kB/s), 1742KiB/s-1896KiB/s (1783kB/s-1942kB/s), io=23.4MiB (24.5MB), run=3238-3299msec WRITE: bw=7639KiB/s (7822kB/s), 1828KiB/s-1984KiB/s (1872kB/s-2032kB/s), io=24.6MiB (25.8MB), run=3238-3299msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25132: Mon Mar 10 15:23:08 2025 read: IOPS=86, BW=1383KiB/s (1416kB/s)(5920KiB/4281msec) clat (usec): min=1780, max=22910, avg=4496.11, stdev=3464.84 lat (usec): min=1780, max=22911, avg=4496.75, stdev=3464.83 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2089], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3621], | 70.00th=[ 4621], 80.00th=[ 6194], 90.00th=[ 9503], 95.00th=[11469], | 99.00th=[17957], 99.50th=[22414], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 832, max= 2688, per=22.95%, avg=1283.62, stdev=592.39, samples=8 iops : min= 52, max= 168, avg=80.12, stdev=37.03, samples=8 write: IOPS=92, BW=1488KiB/s (1523kB/s)(6368KiB/4281msec) clat (usec): min=2370, max=27998, avg=6364.91, stdev=4071.22 lat (usec): min=2371, max=27999, avg=6366.10, stdev=4071.14 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 2966], 10.00th=[ 3163], 20.00th=[ 3458], | 30.00th=[ 3818], 40.00th=[ 4228], 50.00th=[ 4621], 60.00th=[ 5473], | 70.00th=[ 6783], 80.00th=[ 8848], 90.00th=[12387], 95.00th=[15008], | 99.00th=[20317], 99.50th=[26608], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 830, max= 2624, per=23.71%, avg=1395.75, stdev=591.55, samples=8 iops : min= 51, max= 164, avg=87.13, stdev=37.09, samples=8 lat (msec) : 2=2.60%, 4=46.74%, 10=38.54%, 20=11.20%, 50=0.91% cpu : usr=0.14%, sys=11.82%, ctx=909, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25133: Mon Mar 10 15:23:08 2025 read: IOPS=84, BW=1345KiB/s (1377kB/s)(5744KiB/4270msec) clat (usec): min=1760, max=19617, avg=4646.33, stdev=3441.92 lat (usec): min=1760, max=19618, avg=4646.92, stdev=3441.93 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2008], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2769], 50.00th=[ 3032], 60.00th=[ 3392], | 70.00th=[ 4817], 80.00th=[ 6849], 90.00th=[10028], 95.00th=[12518], | 99.00th=[15926], 99.50th=[17695], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 864, max= 2400, per=22.80%, avg=1275.62, stdev=513.91, samples=8 iops : min= 54, max= 150, avg=79.62, stdev=32.14, samples=8 write: IOPS=95, BW=1533KiB/s (1569kB/s)(6544KiB/4270msec) clat (usec): min=2549, max=24239, avg=6313.47, stdev=3841.10 lat (usec): min=2550, max=24241, avg=6314.84, stdev=3841.28 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3130], 20.00th=[ 3458], | 30.00th=[ 3752], 40.00th=[ 4228], 50.00th=[ 4621], 60.00th=[ 5735], | 70.00th=[ 7308], 80.00th=[ 8848], 90.00th=[11600], 95.00th=[14484], | 99.00th=[19530], 99.50th=[20841], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1056, max= 2816, per=24.39%, avg=1435.62, stdev=573.64, samples=8 iops : min= 66, max= 176, avg=89.62, stdev=35.87, samples=8 lat (msec) : 2=1.95%, 4=47.92%, 10=37.63%, 20=11.98%, 50=0.52% cpu : usr=0.14%, sys=11.74%, ctx=885, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25134: Mon Mar 10 15:23:08 2025 read: IOPS=92, BW=1475KiB/s (1511kB/s)(6256KiB/4241msec) clat (usec): min=1815, max=22038, avg=4669.21, stdev=3635.48 lat (usec): min=1816, max=22039, avg=4669.84, stdev=3635.54 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1991], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2442], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3654], | 70.00th=[ 4621], 80.00th=[ 6390], 90.00th=[10290], 95.00th=[12387], | 99.00th=[19792], 99.50th=[20579], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 768, max= 2656, per=24.81%, avg=1388.00, stdev=611.70, samples=8 iops : min= 48, max= 166, avg=86.75, stdev=38.23, samples=8 write: IOPS=88, BW=1422KiB/s (1456kB/s)(6032KiB/4241msec) clat (usec): min=2392, max=27880, avg=6128.27, stdev=3550.29 lat (usec): min=2393, max=27881, avg=6129.62, stdev=3550.82 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2966], 10.00th=[ 3163], 20.00th=[ 3392], | 30.00th=[ 3687], 40.00th=[ 4047], 50.00th=[ 4686], 60.00th=[ 5538], | 70.00th=[ 6980], 80.00th=[ 8848], 90.00th=[11207], 95.00th=[13304], | 99.00th=[17171], 99.50th=[19006], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 896, max= 2592, per=22.77%, avg=1340.00, stdev=529.82, samples=8 iops : min= 56, max= 162, avg=83.75, stdev=33.11, samples=8 lat (msec) : 2=2.60%, 4=48.96%, 10=34.90%, 20=13.15%, 50=0.39% cpu : usr=0.07%, sys=11.75%, ctx=912, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25135: Mon Mar 10 15:23:08 2025 read: IOPS=88, BW=1410KiB/s (1444kB/s)(6032KiB/4279msec) clat (usec): min=1788, max=22850, avg=4649.43, stdev=3836.79 lat (usec): min=1788, max=22851, avg=4650.10, stdev=3836.77 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2245], | 30.00th=[ 2376], 40.00th=[ 2606], 50.00th=[ 2933], 60.00th=[ 3392], | 70.00th=[ 4359], 80.00th=[ 6783], 90.00th=[10290], 95.00th=[12911], | 99.00th=[19792], 99.50th=[21627], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 832, max= 2432, per=23.53%, avg=1316.00, stdev=504.93, samples=8 iops : min= 52, max= 152, avg=82.25, stdev=31.56, samples=8 write: IOPS=91, BW=1462KiB/s (1497kB/s)(6256KiB/4279msec) clat (usec): min=2430, max=19388, avg=6180.69, stdev=3622.83 lat (usec): min=2431, max=19389, avg=6181.79, stdev=3622.74 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2933], 10.00th=[ 3163], 20.00th=[ 3458], | 30.00th=[ 3752], 40.00th=[ 4178], 50.00th=[ 4686], 60.00th=[ 5538], | 70.00th=[ 6915], 80.00th=[ 8717], 90.00th=[11863], 95.00th=[13960], | 99.00th=[18482], 99.50th=[19006], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 864, max= 2656, per=23.31%, avg=1372.00, stdev=584.43, samples=8 iops : min= 54, max= 166, avg=85.75, stdev=36.53, samples=8 lat (msec) : 2=2.21%, 4=48.70%, 10=35.94%, 20=12.76%, 50=0.39% cpu : usr=0.21%, sys=11.57%, ctx=924, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5595KiB/s (5729kB/s), 1345KiB/s-1475KiB/s (1377kB/s-1511kB/s), io=23.4MiB (24.5MB), run=4241-4281msec WRITE: bw=5886KiB/s (6028kB/s), 1422KiB/s-1533KiB/s (1456kB/s-1569kB/s), io=24.6MiB (25.8MB), run=4241-4281msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25144: Mon Mar 10 15:23:10 2025 read: IOPS=6, BW=6906KiB/s (7071kB/s)(7168KiB/1038msec) clat (usec): min=356, max=361119, avg=98574.04, stdev=125703.55 lat (usec): min=357, max=361120, avg=98575.26, stdev=125703.75 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 26084], | 30.00th=[ 27132], 40.00th=[ 27132], 50.00th=[ 31851], 60.00th=[108528], | 70.00th=[108528], 80.00th=[135267], 90.00th=[362808], 95.00th=[362808], | 99.00th=[362808], 99.50th=[362808], 99.90th=[362808], 99.95th=[362808], | 99.99th=[362808] bw ( KiB/s): min= 2048, max= 2048, per=10.38%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4933KiB/s (5051kB/s)(5120KiB/1038msec) clat (msec): min=37, max=135, avg=60.80, stdev=42.22 lat (msec): min=37, max=136, avg=60.85, stdev=42.23 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 43], 60.00th=[ 43], | 70.00th=[ 48], 80.00th=[ 48], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=25.55%, ctx=1096, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25145: Mon Mar 10 15:23:10 2025 read: IOPS=2, BW=2083KiB/s (2133kB/s)(2048KiB/983msec) clat (usec): min=29934, max=33659, avg=31796.74, stdev=2633.74 lat (usec): min=29935, max=33661, avg=31798.10, stdev=2634.66 clat percentiles (usec): | 1.00th=[30016], 5.00th=[30016], 10.00th=[30016], 20.00th=[30016], | 30.00th=[30016], 40.00th=[30016], 50.00th=[30016], 60.00th=[33817], | 70.00th=[33817], 80.00th=[33817], 90.00th=[33817], 95.00th=[33817], | 99.00th=[33817], 99.50th=[33817], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 2043, max= 2043, per=10.35%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.2MiB/s (10.7MB/s)(10.0MiB/983msec) clat (msec): min=25, max=410, avg=91.15, stdev=115.21 lat (msec): min=25, max=410, avg=91.22, stdev=115.19 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28], | 30.00th=[ 32], 40.00th=[ 39], 50.00th=[ 53], 60.00th=[ 64], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 93], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 4087, max= 4087, per=14.80%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=41.67%, 500=8.33% cpu : usr=0.00%, sys=26.48%, ctx=343, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25146: Mon Mar 10 15:23:10 2025 read: IOPS=6, BW=6314KiB/s (6466kB/s)(6144KiB/973msec) clat (usec): min=494, max=47739, avg=34969.85, stdev=17375.42 lat (usec): min=495, max=47741, avg=34971.11, stdev=17375.47 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 494], 10.00th=[ 494], 20.00th=[35914], | 30.00th=[35914], 40.00th=[39584], 50.00th=[39584], 60.00th=[40633], | 70.00th=[44827], 80.00th=[44827], 90.00th=[47973], 95.00th=[47973], | 99.00th=[47973], 99.50th=[47973], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] write: IOPS=6, BW=6314KiB/s (6466kB/s)(6144KiB/973msec) clat (msec): min=22, max=463, avg=118.63, stdev=170.66 lat (msec): min=22, max=463, avg=118.69, stdev=170.66 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 75], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 4096, max= 4096, per=14.83%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=66.67%, 100=16.67%, 500=8.33% cpu : usr=0.00%, sys=20.68%, ctx=439, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25147: Mon Mar 10 15:23:10 2025 read: IOPS=5, BW=6117KiB/s (6264kB/s)(5120KiB/837msec) clat (usec): min=14618, max=53505, avg=29078.36, stdev=14923.38 lat (usec): min=14619, max=53507, avg=29079.59, stdev=14923.52 clat percentiles (usec): | 1.00th=[14615], 5.00th=[14615], 10.00th=[14615], 20.00th=[14615], | 30.00th=[20055], 40.00th=[20055], 50.00th=[27919], 60.00th=[27919], | 70.00th=[29230], 80.00th=[29230], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] write: IOPS=8, BW=8564KiB/s (8769kB/s)(7168KiB/837msec) clat (msec): min=28, max=386, avg=95.83, stdev=128.67 lat (msec): min=28, max=386, avg=95.89, stdev=128.66 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 36], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 55], 80.00th=[ 64], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 6131, max= 6131, per=22.20%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=41.67%, 500=8.33% cpu : usr=0.00%, sys=25.84%, ctx=306, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.3MiB/s (20.2MB/s), 2083KiB/s-6906KiB/s (2133kB/s-7071kB/s), io=20.0MiB (20.0MB), run=837-1038msec WRITE: bw=26.0MiB/s (28.3MB/s), 4933KiB/s-10.2MiB/s (5051kB/s-10.7MB/s), io=28.0MiB (29.4MB), run=837-1038msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25148: Mon Mar 10 15:23:10 2025 read: IOPS=7, BW=7851KiB/s (8039kB/s)(7168KiB/913msec) clat (msec): min=18, max=377, avg=93.83, stdev=127.44 lat (msec): min=18, max=377, avg=93.83, stdev=127.44 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 65], 60.00th=[ 70], | 70.00th=[ 70], 80.00th=[ 78], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 6144, max= 6144, per=32.52%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5608KiB/s (5742kB/s)(5120KiB/913msec) clat (msec): min=16, max=127, avg=49.41, stdev=44.90 lat (msec): min=16, max=127, avg=49.47, stdev=44.90 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 48], 80.00th=[ 48], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 2048, max= 2048, per=7.74%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=10.09%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25149: Mon Mar 10 15:23:10 2025 read: IOPS=1, BW=1889KiB/s (1935kB/s)(2048KiB/1084msec) clat (msec): min=32, max=115, avg=74.07, stdev=58.36 lat (msec): min=32, max=115, avg=74.07, stdev=58.36 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 2048, max= 2048, per=10.84%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=9, BW=9446KiB/s (9673kB/s)(10.0MiB/1084msec) clat (msec): min=25, max=412, avg=85.98, stdev=116.53 lat (msec): min=25, max=412, avg=86.07, stdev=116.53 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 36], 50.00th=[ 41], 60.00th=[ 62], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 80], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min=14336, max=14336, per=54.20%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.18%, sys=13.57%, ctx=83, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25150: Mon Mar 10 15:23:10 2025 read: IOPS=5, BW=5919KiB/s (6061kB/s)(6144KiB/1038msec) clat (usec): min=13507, max=46360, avg=27178.89, stdev=12043.00 lat (usec): min=13508, max=46362, avg=27180.32, stdev=12043.23 clat percentiles (usec): | 1.00th=[13566], 5.00th=[13566], 10.00th=[13566], 20.00th=[17695], | 30.00th=[17695], 40.00th=[22152], 50.00th=[22152], 60.00th=[28443], | 70.00th=[34866], 80.00th=[34866], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min=10240, max=10240, per=54.20%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5919KiB/s (6061kB/s)(6144KiB/1038msec) clat (msec): min=17, max=469, avg=133.53, stdev=169.73 lat (msec): min=17, max=469, avg=133.60, stdev=169.73 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 88], | 70.00th=[ 134], 80.00th=[ 134], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 4096, max= 6144, per=19.36%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=25.00%, 50=50.00%, 100=8.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=13.02%, ctx=113, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25151: Mon Mar 10 15:23:10 2025 read: IOPS=4, BW=5039KiB/s (5160kB/s)(5120KiB/1016msec) clat (usec): min=15712, max=51309, avg=35977.42, stdev=16390.57 lat (usec): min=15712, max=51311, avg=35978.58, stdev=16390.81 clat percentiles (usec): | 1.00th=[15664], 5.00th=[15664], 10.00th=[15664], 20.00th=[15664], | 30.00th=[22152], 40.00th=[22152], 50.00th=[40109], 60.00th=[40109], | 70.00th=[50594], 80.00th=[50594], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 8192, max= 8192, per=43.36%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7055KiB/s (7224kB/s)(7168KiB/1016msec) clat (msec): min=23, max=461, avg=110.19, stdev=155.97 lat (msec): min=23, max=461, avg=110.28, stdev=156.06 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 56], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 86], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 2048, max=12288, per=27.10%, avg=7168.00, stdev=7240.77, samples=2 iops : min= 2, max= 12, avg= 7.00, stdev= 7.07, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=41.67%, 500=8.33% cpu : usr=0.00%, sys=10.94%, ctx=121, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.4MiB/s (19.3MB/s), 1889KiB/s-7851KiB/s (1935kB/s-8039kB/s), io=20.0MiB (20.0MB), run=913-1084msec WRITE: bw=25.8MiB/s (27.1MB/s), 5608KiB/s-9446KiB/s (5742kB/s-9673kB/s), io=28.0MiB (29.4MB), run=913-1084msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25158: Mon Mar 10 15:23:13 2025 read: IOPS=4, BW=17.8MiB/s (18.6MB/s)(12.0MiB/676msec) clat (msec): min=42, max=476, avg=222.14, stdev=226.17 lat (msec): min=42, max=476, avg=222.15, stdev=226.17 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 8192, max= 8192, per=67.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=22.52%, ctx=1212, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25159: Mon Mar 10 15:23:13 2025 read: IOPS=0, BW=3043KiB/s (3116kB/s)(4096KiB/1346msec) clat (nsec): min=159269k, max=159269k, avg=159268754.00, stdev= 0.00 lat (nsec): min=159271k, max=159271k, avg=159271249.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 8192, max= 8192, per=67.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6086KiB/s (6232kB/s)(8192KiB/1346msec) clat (msec): min=425, max=751, avg=588.70, stdev=230.73 lat (msec): min=425, max=752, avg=589.03, stdev=230.64 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 751], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 8192, max= 8192, per=33.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.07%, sys=17.99%, ctx=907, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25160: Mon Mar 10 15:23:13 2025 write: IOPS=2, BW=10.3MiB/s (10.8MB/s)(12.0MiB/1164msec) clat (msec): min=137, max=621, avg=379.93, stdev=241.73 lat (msec): min=138, max=621, avg=380.22, stdev=241.76 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min=15427, max=15427, per=63.37%, avg=15427.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=25.02%, ctx=88, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25161: Mon Mar 10 15:23:13 2025 write: IOPS=3, BW=12.1MiB/s (12.7MB/s)(12.0MiB/991msec) clat (msec): min=101, max=662, avg=327.51, stdev=295.62 lat (msec): min=102, max=662, avg=327.72, stdev=295.69 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=30.40%, ctx=26, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.9MiB/s (12.5MB/s), 3043KiB/s-17.8MiB/s (3116kB/s-18.6MB/s), io=16.0MiB (16.8MB), run=676-1346msec WRITE: bw=23.8MiB/s (24.9MB/s), 6086KiB/s-12.1MiB/s (6232kB/s-12.7MB/s), io=32.0MiB (33.6MB), run=991-1346msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25162: Mon Mar 10 15:23:13 2025 read: IOPS=2, BW=10.2MiB/s (10.7MB/s)(12.0MiB/1171msec) clat (msec): min=60, max=745, avg=386.34, stdev=343.90 lat (msec): min=60, max=745, avg=386.34, stdev=343.90 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 8192, max= 8192, per=67.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.17%, sys=5.81%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25163: Mon Mar 10 15:23:13 2025 read: IOPS=0, BW=3018KiB/s (3091kB/s)(4096KiB/1357msec) clat (nsec): min=724172k, max=724172k, avg=724171753.00, stdev= 0.00 lat (nsec): min=724173k, max=724173k, avg=724173315.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 726], 5.00th=[ 726], 10.00th=[ 726], 20.00th=[ 726], | 30.00th=[ 726], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 8192, max= 8192, per=67.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6037KiB/s (6182kB/s)(8192KiB/1357msec) clat (msec): min=273, max=346, avg=309.98, stdev=51.62 lat (msec): min=273, max=346, avg=310.25, stdev=51.65 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.07%, sys=8.41%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25164: Mon Mar 10 15:23:13 2025 write: IOPS=2, BW=9438KiB/s (9664kB/s)(12.0MiB/1302msec) clat (msec): min=90, max=852, avg=429.91, stdev=387.55 lat (msec): min=90, max=852, avg=430.20, stdev=387.56 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 8192, max= 8192, per=33.93%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.15%, sys=5.76%, ctx=43, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25165: Mon Mar 10 15:23:13 2025 write: IOPS=2, BW=10.4MiB/s (10.9MB/s)(12.0MiB/1153msec) clat (msec): min=144, max=743, avg=380.69, stdev=318.75 lat (msec): min=144, max=743, avg=380.97, stdev=318.73 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 8192, max= 8192, per=33.93%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.17%, sys=13.45%, ctx=72, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.8MiB/s (12.4MB/s), 3018KiB/s-10.2MiB/s (3091kB/s-10.7MB/s), io=16.0MiB (16.8MB), run=1171-1357msec WRITE: bw=23.6MiB/s (24.7MB/s), 6037KiB/s-10.4MiB/s (6182kB/s-10.9MB/s), io=32.0MiB (33.6MB), run=1153-1357msec