== sanity test 398b: DIO and buffer IO race ============== 02:27:39 (1744093659) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.27266 s, 39.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30547: Tue Apr 8 02:27:54 2025 read: IOPS=116, BW=466KiB/s (477kB/s)(5992KiB/12851msec) clat (usec): min=59, max=143325, avg=4721.42, stdev=4453.54 lat (usec): min=59, max=143326, avg=4722.19, stdev=4453.54 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 130], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 3064], 40.00th=[ 3556], 50.00th=[ 4178], 60.00th=[ 4883], | 70.00th=[ 5604], 80.00th=[ 6521], 90.00th=[ 7832], 95.00th=[ 9241], | 99.00th=[ 13829], 99.50th=[ 15139], 99.90th=[ 23725], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 264, max= 574, per=24.71%, avg=461.28, stdev=61.09, samples=25 iops : min= 66, max= 143, avg=115.24, stdev=15.25, samples=25 write: IOPS=122, BW=490KiB/s (502kB/s)(6296KiB/12851msec) clat (usec): min=429, max=25812, avg=3612.14, stdev=3235.47 lat (usec): min=430, max=25812, avg=3613.00, stdev=3235.41 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 709], | 30.00th=[ 898], 40.00th=[ 1860], 50.00th=[ 2933], 60.00th=[ 3654], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 8225], 95.00th=[10159], | 99.00th=[13173], 99.50th=[14746], 99.90th=[20841], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 320, max= 616, per=24.85%, avg=485.52, stdev=80.66, samples=25 iops : min= 80, max= 154, avg=121.32, stdev=20.12, samples=25 lat (usec) : 100=1.95%, 250=0.94%, 500=0.23%, 750=11.26%, 1000=4.95% lat (msec) : 2=4.85%, 4=31.48%, 10=39.81%, 20=4.26%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.32%, ctx=5898, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30548: Tue Apr 8 02:27:54 2025 read: IOPS=115, BW=463KiB/s (474kB/s)(5940KiB/12829msec) clat (usec): min=53, max=21440, avg=4536.91, stdev=2566.83 lat (usec): min=54, max=21440, avg=4537.81, stdev=2566.77 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 182], 10.00th=[ 2212], 20.00th=[ 2606], | 30.00th=[ 3032], 40.00th=[ 3490], 50.00th=[ 4080], 60.00th=[ 4752], | 70.00th=[ 5473], 80.00th=[ 6194], 90.00th=[ 7504], 95.00th=[ 9110], | 99.00th=[13435], 99.50th=[15795], 99.90th=[19006], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 327, max= 734, per=24.61%, avg=459.56, stdev=71.45, samples=25 iops : min= 81, max= 183, avg=114.76, stdev=17.84, samples=25 write: IOPS=123, BW=495KiB/s (507kB/s)(6348KiB/12829msec) clat (usec): min=442, max=170137, avg=3799.20, stdev=5289.25 lat (usec): min=443, max=170138, avg=3800.26, stdev=5289.22 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 545], 10.00th=[ 594], 20.00th=[ 734], | 30.00th=[ 938], 40.00th=[ 2212], 50.00th=[ 2999], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 8291], 95.00th=[ 10028], | 99.00th=[ 13960], 99.50th=[ 15795], 99.90th=[ 23725], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 359, max= 638, per=24.85%, avg=485.48, stdev=73.44, samples=25 iops : min= 89, max= 159, avg=121.24, stdev=18.38, samples=25 lat (usec) : 100=1.82%, 250=0.65%, 500=0.16%, 750=10.61%, 1000=5.50% lat (msec) : 2=4.82%, 4=32.16%, 10=40.07%, 20=4.07%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.60%, ctx=5894, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30549: Tue Apr 8 02:27:54 2025 read: IOPS=117, BW=469KiB/s (480kB/s)(6028KiB/12861msec) clat (usec): min=57, max=27374, avg=4638.84, stdev=2858.40 lat (usec): min=57, max=27375, avg=4639.59, stdev=2858.44 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 125], 10.00th=[ 2114], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 4146], 60.00th=[ 4752], | 70.00th=[ 5604], 80.00th=[ 6456], 90.00th=[ 7898], 95.00th=[ 9503], | 99.00th=[14615], 99.50th=[17171], 99.90th=[23725], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 288, max= 592, per=24.94%, avg=465.56, stdev=73.30, samples=25 iops : min= 72, max= 148, avg=116.36, stdev=18.34, samples=25 write: IOPS=121, BW=487KiB/s (498kB/s)(6260KiB/12861msec) clat (usec): min=440, max=148102, avg=3686.45, stdev=4860.26 lat (usec): min=441, max=148104, avg=3688.73, stdev=4859.74 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 725], | 30.00th=[ 922], 40.00th=[ 1827], 50.00th=[ 2835], 60.00th=[ 3556], | 70.00th=[ 4686], 80.00th=[ 6063], 90.00th=[ 8356], 95.00th=[ 10028], | 99.00th=[ 13435], 99.50th=[ 15270], 99.90th=[ 19530], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 344, max= 664, per=24.59%, avg=480.56, stdev=78.98, samples=25 iops : min= 86, max= 166, avg=120.12, stdev=19.74, samples=25 lat (usec) : 100=1.69%, 250=1.07%, 500=0.13%, 750=11.00%, 1000=5.57% lat (msec) : 2=5.18%, 4=31.97%, 10=38.87%, 20=4.36%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.39%, ctx=5705, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30550: Tue Apr 8 02:27:54 2025 read: IOPS=118, BW=472KiB/s (483kB/s)(6052KiB/12821msec) clat (usec): min=53, max=28130, avg=4602.10, stdev=2690.77 lat (usec): min=53, max=28130, avg=4602.94, stdev=2691.39 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 157], 10.00th=[ 2212], 20.00th=[ 2638], | 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 4113], 60.00th=[ 4817], | 70.00th=[ 5473], 80.00th=[ 6325], 90.00th=[ 7832], 95.00th=[ 9110], | 99.00th=[13173], 99.50th=[16057], 99.90th=[23725], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 304, max= 576, per=25.02%, avg=467.20, stdev=51.90, samples=25 iops : min= 76, max= 144, avg=116.80, stdev=12.97, samples=25 write: IOPS=121, BW=486KiB/s (498kB/s)(6236KiB/12821msec) clat (usec): min=447, max=168179, avg=3713.86, stdev=5303.74 lat (usec): min=448, max=168181, avg=3714.91, stdev=5303.78 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 693], | 30.00th=[ 906], 40.00th=[ 1926], 50.00th=[ 2966], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 6063], 90.00th=[ 8356], 95.00th=[ 9634], | 99.00th=[ 13960], 99.50th=[ 15270], 99.90th=[ 28967], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 368, max= 648, per=24.63%, avg=481.28, stdev=68.84, samples=25 iops : min= 92, max= 162, avg=120.32, stdev=17.21, samples=25 lat (usec) : 100=1.60%, 250=0.91%, 500=0.29%, 750=11.23%, 1000=5.40% lat (msec) : 2=4.36%, 4=33.07%, 10=39.19%, 20=3.65%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=22.29%, ctx=5804, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1867KiB/s (1912kB/s), 463KiB/s-472KiB/s (474kB/s-483kB/s), io=23.4MiB (24.6MB), run=12821-12861msec WRITE: bw=1955KiB/s (2002kB/s), 486KiB/s-495KiB/s (498kB/s-507kB/s), io=24.6MiB (25.7MB), run=12821-12861msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30543: Tue Apr 8 02:27:58 2025 read: IOPS=90, BW=361KiB/s (369kB/s)(5992KiB/16619msec) clat (usec): min=1569, max=172985, avg=4587.82, stdev=5630.21 lat (usec): min=1570, max=172987, avg=4588.59, stdev=5630.18 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1942], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3490], | 70.00th=[ 4555], 80.00th=[ 6194], 90.00th=[ 9634], 95.00th=[ 12256], | 99.00th=[ 17695], 99.50th=[ 20579], 99.90th=[ 30016], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 183, max= 736, per=24.96%, avg=359.64, stdev=176.96, samples=33 iops : min= 45, max= 184, avg=89.82, stdev=44.31, samples=33 write: IOPS=94, BW=379KiB/s (388kB/s)(6296KiB/16619msec) clat (usec): min=2346, max=28090, avg=6150.72, stdev=3821.95 lat (usec): min=2347, max=28090, avg=6151.65, stdev=3821.92 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2933], 10.00th=[ 3097], 20.00th=[ 3359], | 30.00th=[ 3589], 40.00th=[ 3916], 50.00th=[ 4490], 60.00th=[ 5473], | 70.00th=[ 7111], 80.00th=[ 8717], 90.00th=[11469], 95.00th=[13698], | 99.00th=[19792], 99.50th=[22676], 99.90th=[27919], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 184, max= 784, per=25.01%, avg=377.33, stdev=179.44, samples=33 iops : min= 46, max= 196, avg=94.24, stdev=44.93, samples=33 lat (msec) : 2=4.10%, 4=49.38%, 10=34.51%, 20=11.26%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.62%, ctx=3703, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30544: Tue Apr 8 02:27:58 2025 read: IOPS=89, BW=360KiB/s (369kB/s)(5940KiB/16501msec) clat (usec): min=1541, max=161691, avg=4430.68, stdev=5261.37 lat (usec): min=1542, max=161692, avg=4431.37, stdev=5261.37 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1909], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2573], 50.00th=[ 2933], 60.00th=[ 3425], | 70.00th=[ 4228], 80.00th=[ 5997], 90.00th=[ 9241], 95.00th=[ 11731], | 99.00th=[ 17171], 99.50th=[ 19268], 99.90th=[ 22152], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 136, max= 774, per=24.52%, avg=353.37, stdev=176.54, samples=32 iops : min= 34, max= 193, avg=88.28, stdev=44.13, samples=32 write: IOPS=96, BW=385KiB/s (394kB/s)(6348KiB/16501msec) clat (usec): min=2423, max=34426, avg=6200.83, stdev=3816.97 lat (usec): min=2424, max=34427, avg=6201.65, stdev=3816.94 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3392], | 30.00th=[ 3621], 40.00th=[ 3949], 50.00th=[ 4555], 60.00th=[ 5538], | 70.00th=[ 7242], 80.00th=[ 8979], 90.00th=[11469], 95.00th=[13566], | 99.00th=[19268], 99.50th=[22414], 99.90th=[28967], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 208, max= 736, per=24.71%, avg=372.88, stdev=162.11, samples=32 iops : min= 52, max= 184, avg=93.16, stdev=40.51, samples=32 lat (msec) : 2=4.33%, 4=49.77%, 10=33.98%, 20=11.26%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.43%, ctx=3739, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30545: Tue Apr 8 02:27:58 2025 read: IOPS=91, BW=367KiB/s (376kB/s)(6028KiB/16414msec) clat (usec): min=1624, max=29114, avg=4283.21, stdev=3292.63 lat (usec): min=1625, max=29115, avg=4283.83, stdev=3292.62 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2024], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2606], 50.00th=[ 2933], 60.00th=[ 3359], | 70.00th=[ 4228], 80.00th=[ 5735], 90.00th=[ 9241], 95.00th=[11338], | 99.00th=[16450], 99.50th=[20317], 99.90th=[23200], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 160, max= 760, per=25.02%, avg=360.50, stdev=167.23, samples=32 iops : min= 40, max= 190, avg=90.12, stdev=41.81, samples=32 write: IOPS=95, BW=381KiB/s (391kB/s)(6260KiB/16414msec) clat (msec): min=2, max=182, avg= 6.32, stdev= 5.82 lat (msec): min=2, max=182, avg= 6.32, stdev= 5.82 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 29], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 208, max= 784, per=24.69%, avg=372.50, stdev=170.92, samples=32 iops : min= 52, max= 196, avg=93.12, stdev=42.73, samples=32 lat (msec) : 2=4.39%, 4=49.71%, 10=33.85%, 20=11.26%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.70%, ctx=3688, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30546: Tue Apr 8 02:27:58 2025 read: IOPS=90, BW=363KiB/s (372kB/s)(6052KiB/16652msec) clat (usec): min=1666, max=29584, avg=4381.77, stdev=3369.24 lat (usec): min=1667, max=29584, avg=4382.53, stdev=3369.15 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2606], 50.00th=[ 2966], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 6325], 90.00th=[ 9372], 95.00th=[11600], | 99.00th=[16450], 99.50th=[20055], 99.90th=[26346], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 144, max= 720, per=25.10%, avg=361.64, stdev=175.26, samples=33 iops : min= 36, max= 180, avg=90.36, stdev=43.83, samples=33 write: IOPS=93, BW=374KiB/s (383kB/s)(6236KiB/16652msec) clat (msec): min=2, max=162, avg= 6.37, stdev= 5.56 lat (msec): min=2, max=162, avg= 6.37, stdev= 5.56 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 35], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 168, max= 744, per=24.72%, avg=373.03, stdev=180.74, samples=33 iops : min= 42, max= 186, avg=93.21, stdev=45.21, samples=33 lat (msec) : 2=4.13%, 4=50.52%, 10=33.01%, 20=11.56%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.43%, ctx=3691, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1442KiB/s (1477kB/s), 360KiB/s-367KiB/s (369kB/s-376kB/s), io=23.4MiB (24.6MB), run=16414-16652msec WRITE: bw=1510KiB/s (1546kB/s), 374KiB/s-385KiB/s (383kB/s-394kB/s), io=24.6MiB (25.7MB), run=16414-16652msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30579: Tue Apr 8 02:28:03 2025 read: IOPS=106, BW=1709KiB/s (1750kB/s)(5920KiB/3465msec) clat (usec): min=58, max=18412, avg=4859.28, stdev=2961.55 lat (usec): min=59, max=18413, avg=4860.15, stdev=2961.57 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 85], 10.00th=[ 133], 20.00th=[ 2900], | 30.00th=[ 3392], 40.00th=[ 3916], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 6063], 80.00th=[ 6980], 90.00th=[ 8717], 95.00th=[10159], | 99.00th=[14615], 99.50th=[15795], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1312, max= 1920, per=24.25%, avg=1674.00, stdev=211.87, samples=6 iops : min= 82, max= 120, avg=104.50, stdev=13.23, samples=6 write: IOPS=114, BW=1838KiB/s (1882kB/s)(6368KiB/3465msec) clat (usec): min=706, max=14830, avg=4054.29, stdev=3067.29 lat (usec): min=706, max=14831, avg=4055.21, stdev=3067.33 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 832], 10.00th=[ 906], 20.00th=[ 1156], | 30.00th=[ 1500], 40.00th=[ 2638], 50.00th=[ 3359], 60.00th=[ 4080], | 70.00th=[ 5145], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[10028], | 99.00th=[13698], 99.50th=[14615], 99.90th=[14877], 99.95th=[14877], | 99.99th=[14877] bw ( KiB/s): min= 1408, max= 2432, per=25.03%, avg=1818.00, stdev=369.06, samples=6 iops : min= 88, max= 152, avg=113.50, stdev=23.08, samples=6 lat (usec) : 100=3.26%, 250=2.34%, 750=0.52%, 1000=7.29% lat (msec) : 2=10.42%, 4=27.21%, 10=43.88%, 20=5.08% cpu : usr=0.35%, sys=24.28%, ctx=1765, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30580: Tue Apr 8 02:28:03 2025 read: IOPS=103, BW=1656KiB/s (1696kB/s)(5744KiB/3469msec) clat (usec): min=62, max=16733, avg=4807.20, stdev=2893.84 lat (usec): min=62, max=16733, avg=4808.70, stdev=2893.61 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 87], 10.00th=[ 133], 20.00th=[ 2802], | 30.00th=[ 3392], 40.00th=[ 3884], 50.00th=[ 4686], 60.00th=[ 5211], | 70.00th=[ 5932], 80.00th=[ 6849], 90.00th=[ 8455], 95.00th=[ 9896], | 99.00th=[14353], 99.50th=[15139], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw ( KiB/s): min= 1152, max= 1952, per=23.01%, avg=1588.67, stdev=261.74, samples=6 iops : min= 72, max= 122, avg=99.17, stdev=16.30, samples=6 write: IOPS=117, BW=1886KiB/s (1932kB/s)(6544KiB/3469msec) clat (usec): min=645, max=22059, avg=4213.65, stdev=3175.94 lat (usec): min=645, max=22060, avg=4214.61, stdev=3176.04 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 816], 10.00th=[ 865], 20.00th=[ 1057], | 30.00th=[ 1860], 40.00th=[ 2999], 50.00th=[ 3720], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 6587], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[13566], 99.50th=[15664], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 1408, max= 2240, per=24.44%, avg=1775.33, stdev=305.06, samples=6 iops : min= 88, max= 140, avg=110.83, stdev=19.00, samples=6 lat (usec) : 100=3.26%, 250=2.21%, 750=0.39%, 1000=9.37% lat (msec) : 2=6.90%, 4=26.82%, 10=45.83%, 20=5.08%, 50=0.13% cpu : usr=0.06%, sys=24.80%, ctx=1722, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30581: Tue Apr 8 02:28:03 2025 read: IOPS=116, BW=1862KiB/s (1907kB/s)(6256KiB/3359msec) clat (usec): min=60, max=21612, avg=4536.14, stdev=2715.75 lat (usec): min=61, max=21613, avg=4536.71, stdev=2715.77 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 80], 10.00th=[ 135], 20.00th=[ 2704], | 30.00th=[ 3163], 40.00th=[ 3687], 50.00th=[ 4293], 60.00th=[ 5014], | 70.00th=[ 5669], 80.00th=[ 6587], 90.00th=[ 7570], 95.00th=[ 8586], | 99.00th=[11338], 99.50th=[16909], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 1280, max= 2016, per=25.25%, avg=1743.33, stdev=279.11, samples=6 iops : min= 80, max= 126, avg=108.83, stdev=17.37, samples=6 write: IOPS=112, BW=1796KiB/s (1839kB/s)(6032KiB/3359msec) clat (usec): min=632, max=21640, avg=4126.51, stdev=3486.40 lat (usec): min=633, max=21641, avg=4127.52, stdev=3486.56 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 799], 10.00th=[ 840], 20.00th=[ 1012], | 30.00th=[ 1385], 40.00th=[ 2409], 50.00th=[ 3163], 60.00th=[ 4015], | 70.00th=[ 5211], 80.00th=[ 7046], 90.00th=[ 8717], 95.00th=[10683], | 99.00th=[15664], 99.50th=[17171], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 1376, max= 2208, per=23.85%, avg=1732.67, stdev=293.77, samples=6 iops : min= 86, max= 138, avg=108.17, stdev=18.27, samples=6 lat (usec) : 100=4.43%, 250=0.91%, 500=0.13%, 750=1.04%, 1000=8.85% lat (msec) : 2=8.46%, 4=28.26%, 10=43.49%, 20=4.17%, 50=0.26% cpu : usr=0.00%, sys=25.01%, ctx=1659, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30582: Tue Apr 8 02:28:03 2025 read: IOPS=114, BW=1835KiB/s (1879kB/s)(6032KiB/3287msec) clat (usec): min=61, max=17127, avg=4523.48, stdev=2632.88 lat (usec): min=61, max=17127, avg=4524.10, stdev=2632.90 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 84], 10.00th=[ 127], 20.00th=[ 2868], | 30.00th=[ 3326], 40.00th=[ 3785], 50.00th=[ 4228], 60.00th=[ 4817], | 70.00th=[ 5473], 80.00th=[ 6390], 90.00th=[ 7570], 95.00th=[ 8586], | 99.00th=[13829], 99.50th=[14484], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1472, max= 2176, per=25.94%, avg=1791.17, stdev=265.24, samples=6 iops : min= 92, max= 136, avg=111.83, stdev=16.45, samples=6 write: IOPS=118, BW=1903KiB/s (1949kB/s)(6256KiB/3287msec) clat (usec): min=694, max=19410, avg=3918.05, stdev=3330.97 lat (usec): min=695, max=19410, avg=3920.30, stdev=3332.19 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 799], 10.00th=[ 832], 20.00th=[ 955], | 30.00th=[ 1254], 40.00th=[ 2057], 50.00th=[ 3097], 60.00th=[ 3851], | 70.00th=[ 5211], 80.00th=[ 6456], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[16909], 99.50th=[18482], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1664, max= 2240, per=26.06%, avg=1892.67, stdev=229.59, samples=6 iops : min= 104, max= 140, avg=118.17, stdev=14.40, samples=6 lat (usec) : 100=3.65%, 250=1.56%, 500=0.26%, 750=1.17%, 1000=11.07% lat (msec) : 2=8.20%, 4=27.34%, 10=42.84%, 20=3.91% cpu : usr=0.24%, sys=25.11%, ctx=1602, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6905KiB/s (7070kB/s), 1656KiB/s-1862KiB/s (1696kB/s-1907kB/s), io=23.4MiB (24.5MB), run=3287-3469msec WRITE: bw=7264KiB/s (7439kB/s), 1796KiB/s-1903KiB/s (1839kB/s-1949kB/s), io=24.6MiB (25.8MB), run=3287-3469msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30575: Tue Apr 8 02:28:04 2025 read: IOPS=84, BW=1357KiB/s (1389kB/s)(5920KiB/4363msec) clat (usec): min=1815, max=21456, avg=4639.88, stdev=3628.11 lat (usec): min=1816, max=21456, avg=4640.48, stdev=3628.10 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2900], 50.00th=[ 3130], 60.00th=[ 3556], | 70.00th=[ 4293], 80.00th=[ 6063], 90.00th=[ 9765], 95.00th=[13042], | 99.00th=[18744], 99.50th=[21103], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 768, max= 2368, per=22.29%, avg=1223.25, stdev=488.58, samples=8 iops : min= 48, max= 148, avg=76.25, stdev=30.55, samples=8 write: IOPS=91, BW=1460KiB/s (1495kB/s)(6368KiB/4363msec) clat (usec): min=2513, max=31385, avg=6604.97, stdev=3910.48 lat (usec): min=2514, max=31385, avg=6606.14, stdev=3910.57 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3130], 20.00th=[ 3425], | 30.00th=[ 3752], 40.00th=[ 4228], 50.00th=[ 4883], 60.00th=[ 6587], | 70.00th=[ 8094], 80.00th=[ 9634], 90.00th=[11731], 95.00th=[14222], | 99.00th=[19792], 99.50th=[20841], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 960, max= 2624, per=23.47%, avg=1355.25, stdev=537.16, samples=8 iops : min= 60, max= 164, avg=84.50, stdev=33.67, samples=8 lat (msec) : 2=1.43%, 4=48.57%, 10=35.94%, 20=13.28%, 50=0.78% cpu : usr=0.09%, sys=12.01%, ctx=963, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30576: Tue Apr 8 02:28:04 2025 read: IOPS=82, BW=1323KiB/s (1355kB/s)(5744KiB/4342msec) clat (usec): min=1839, max=21825, avg=4523.75, stdev=3424.08 lat (usec): min=1839, max=21825, avg=4524.35, stdev=3424.09 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2057], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 3097], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5604], 90.00th=[ 9765], 95.00th=[12518], | 99.00th=[16909], 99.50th=[18220], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 862, max= 2491, per=22.50%, avg=1235.12, stdev=538.06, samples=8 iops : min= 53, max= 155, avg=77.00, stdev=33.49, samples=8 write: IOPS=94, BW=1507KiB/s (1543kB/s)(6544KiB/4342msec) clat (usec): min=2448, max=27511, avg=6577.71, stdev=4131.92 lat (usec): min=2449, max=27511, avg=6578.79, stdev=4131.87 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2999], 10.00th=[ 3163], 20.00th=[ 3490], | 30.00th=[ 3752], 40.00th=[ 4146], 50.00th=[ 4752], 60.00th=[ 5997], | 70.00th=[ 7570], 80.00th=[ 9503], 90.00th=[12518], 95.00th=[14353], | 99.00th=[19530], 99.50th=[22938], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 1056, max= 2682, per=24.08%, avg=1390.88, stdev=532.68, samples=8 iops : min= 66, max= 167, avg=86.75, stdev=33.11, samples=8 lat (msec) : 2=1.56%, 4=48.05%, 10=36.59%, 20=13.15%, 50=0.65% cpu : usr=0.12%, sys=12.51%, ctx=946, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30577: Tue Apr 8 02:28:04 2025 read: IOPS=90, BW=1456KiB/s (1491kB/s)(6256KiB/4297msec) clat (usec): min=1792, max=25625, avg=4701.80, stdev=3535.91 lat (usec): min=1793, max=25625, avg=4702.46, stdev=3535.91 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2008], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2474], 40.00th=[ 2900], 50.00th=[ 3359], 60.00th=[ 4015], | 70.00th=[ 4817], 80.00th=[ 6456], 90.00th=[ 9634], 95.00th=[13304], | 99.00th=[16909], 99.50th=[21365], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 928, max= 2560, per=24.33%, avg=1335.62, stdev=518.10, samples=8 iops : min= 58, max= 160, avg=83.38, stdev=32.44, samples=8 write: IOPS=87, BW=1404KiB/s (1437kB/s)(6032KiB/4297msec) clat (usec): min=2418, max=26189, avg=6489.13, stdev=3787.68 lat (usec): min=2419, max=26190, avg=6490.44, stdev=3787.54 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3163], 20.00th=[ 3458], | 30.00th=[ 3785], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 6325], | 70.00th=[ 7832], 80.00th=[ 9110], 90.00th=[11863], 95.00th=[14222], | 99.00th=[18744], 99.50th=[21103], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 896, max= 2688, per=22.85%, avg=1319.62, stdev=574.97, samples=8 iops : min= 56, max= 168, avg=82.38, stdev=35.95, samples=8 lat (msec) : 2=2.60%, 4=44.92%, 10=40.49%, 20=11.46%, 50=0.52% cpu : usr=0.14%, sys=12.13%, ctx=950, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30578: Tue Apr 8 02:28:04 2025 read: IOPS=87, BW=1407KiB/s (1441kB/s)(6032KiB/4286msec) clat (usec): min=1853, max=29056, avg=4762.50, stdev=3660.20 lat (usec): min=1853, max=29057, avg=4763.11, stdev=3660.20 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4555], 80.00th=[ 6718], 90.00th=[10421], 95.00th=[12649], | 99.00th=[17695], 99.50th=[18744], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 992, max= 2464, per=24.04%, avg=1319.62, stdev=470.41, samples=8 iops : min= 62, max= 154, avg=82.38, stdev=29.45, samples=8 write: IOPS=91, BW=1460KiB/s (1495kB/s)(6256KiB/4286msec) clat (usec): min=2574, max=27501, avg=6308.39, stdev=3593.97 lat (usec): min=2575, max=27502, avg=6309.48, stdev=3593.95 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3032], 10.00th=[ 3228], 20.00th=[ 3490], | 30.00th=[ 3818], 40.00th=[ 4178], 50.00th=[ 4883], 60.00th=[ 5997], | 70.00th=[ 7373], 80.00th=[ 8979], 90.00th=[11469], 95.00th=[12911], | 99.00th=[18482], 99.50th=[22676], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 1021, max= 2656, per=23.82%, avg=1375.62, stdev=528.87, samples=8 iops : min= 63, max= 166, avg=85.88, stdev=33.13, samples=8 lat (msec) : 2=1.30%, 4=47.40%, 10=38.02%, 20=12.89%, 50=0.39% cpu : usr=0.07%, sys=12.16%, ctx=941, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5490KiB/s (5622kB/s), 1323KiB/s-1456KiB/s (1355kB/s-1491kB/s), io=23.4MiB (24.5MB), run=4286-4363msec WRITE: bw=5776KiB/s (5914kB/s), 1404KiB/s-1507KiB/s (1437kB/s-1543kB/s), io=24.6MiB (25.8MB), run=4286-4363msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30591: Tue Apr 8 02:28:06 2025 read: IOPS=3, BW=3809KiB/s (3900kB/s)(7168KiB/1882msec) clat (msec): min=22, max=623, avg=234.22, stdev=257.07 lat (msec): min=22, max=623, avg=234.22, stdev=257.07 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 28], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 54], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 481], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 1747, max= 3488, per=22.31%, avg=2427.67, stdev=930.53, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 write: IOPS=2, BW=2721KiB/s (2786kB/s)(5120KiB/1882msec) clat (usec): min=29866, max=66833, avg=47594.09, stdev=17684.96 lat (usec): min=29900, max=66880, avg=47646.93, stdev=17696.08 clat percentiles (usec): | 1.00th=[29754], 5.00th=[29754], 10.00th=[29754], 20.00th=[29754], | 30.00th=[32113], 40.00th=[32113], 50.00th=[43779], 60.00th=[43779], | 70.00th=[65274], 80.00th=[65274], 90.00th=[66847], 95.00th=[66847], | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 1744, max= 1744, per=11.45%, avg=1744.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=25.00%, 500=16.67%, 750=8.33% cpu : usr=0.00%, sys=22.70%, ctx=1778, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30592: Tue Apr 8 02:28:06 2025 read: IOPS=1, BW=2016KiB/s (2064kB/s)(2048KiB/1016msec) clat (usec): min=30364, max=41091, avg=35728.11, stdev=7585.22 lat (usec): min=30366, max=41092, avg=35729.68, stdev=7584.38 clat percentiles (usec): | 1.00th=[30278], 5.00th=[30278], 10.00th=[30278], 20.00th=[30278], | 30.00th=[30278], 40.00th=[30278], 50.00th=[30278], 60.00th=[41157], | 70.00th=[41157], 80.00th=[41157], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 2048, max= 2048, per=18.82%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=9, BW=9.84MiB/s (10.3MB/s)(10.0MiB/1016msec) clat (msec): min=20, max=521, avg=92.76, stdev=152.02 lat (msec): min=20, max=521, avg=92.82, stdev=152.02 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 31], 40.00th=[ 36], 50.00th=[ 44], 60.00th=[ 46], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 85], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min=18432, max=18432, per=100.00%, avg=18432.00, stdev= 0.00, samples=1 iops : min= 18, max= 18, avg=18.00, stdev= 0.00, samples=1 lat (msec) : 50=75.00%, 100=16.67%, 750=8.33% cpu : usr=0.10%, sys=25.62%, ctx=305, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30593: Tue Apr 8 02:28:06 2025 read: IOPS=6, BW=6812KiB/s (6975kB/s)(6144KiB/902msec) clat (usec): min=346, max=99314, avg=29906.12, stdev=36377.03 lat (usec): min=347, max=99315, avg=29907.13, stdev=36376.85 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 586], | 30.00th=[ 586], 40.00th=[22152], 50.00th=[22152], 60.00th=[28181], | 70.00th=[28967], 80.00th=[28967], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] write: IOPS=6, BW=6812KiB/s (6975kB/s)(6144KiB/902msec) clat (msec): min=23, max=469, avg=105.54, stdev=178.64 lat (msec): min=23, max=469, avg=105.58, stdev=178.65 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 36], | 70.00th=[ 49], 80.00th=[ 49], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 4096, max= 4096, per=26.89%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=66.67%, 100=8.33%, 500=8.33% cpu : usr=0.00%, sys=20.53%, ctx=549, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30595: Tue Apr 8 02:28:06 2025 read: IOPS=5, BW=5670KiB/s (5806kB/s)(5120KiB/903msec) clat (usec): min=402, max=40237, avg=27075.63, stdev=15537.23 lat (usec): min=402, max=40238, avg=27076.42, stdev=15537.26 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 404], 10.00th=[ 404], 20.00th=[ 404], | 30.00th=[27919], 40.00th=[27919], 50.00th=[32900], 60.00th=[32900], | 70.00th=[33817], 80.00th=[33817], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] write: IOPS=7, BW=7938KiB/s (8128kB/s)(7168KiB/903msec) clat (msec): min=29, max=477, avg=105.47, stdev=164.51 lat (msec): min=29, max=477, avg=105.53, stdev=164.52 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 33], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 37], 60.00th=[ 62], | 70.00th=[ 62], 80.00th=[ 67], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 2048, max= 2048, per=13.44%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=66.67%, 100=16.67%, 500=8.33% cpu : usr=0.00%, sys=23.28%, ctx=515, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.6MiB/s (11.1MB/s), 2016KiB/s-6812KiB/s (2064kB/s-6975kB/s), io=20.0MiB (20.0MB), run=902-1882msec WRITE: bw=14.9MiB/s (15.6MB/s), 2721KiB/s-9.84MiB/s (2786kB/s-10.3MB/s), io=28.0MiB (29.4MB), run=902-1882msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30594: Tue Apr 8 02:28:06 2025 read: IOPS=3, BW=3729KiB/s (3819kB/s)(7168KiB/1922msec) clat (msec): min=11, max=761, avg=240.02, stdev=280.18 lat (msec): min=11, max=761, avg=240.02, stdev=280.17 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 194], 60.00th=[ 230], | 70.00th=[ 230], 80.00th=[ 451], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 2048, max= 4096, per=28.83%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2664KiB/s (2728kB/s)(5120KiB/1922msec) clat (usec): min=15015, max=64962, avg=41346.83, stdev=24405.46 lat (usec): min=15079, max=65027, avg=41403.02, stdev=24401.63 clat percentiles (usec): | 1.00th=[15008], 5.00th=[15008], 10.00th=[15008], 20.00th=[15008], | 30.00th=[16319], 40.00th=[16319], 50.00th=[47449], 60.00th=[47449], | 70.00th=[62653], 80.00th=[62653], 90.00th=[64750], 95.00th=[64750], | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 2048, max= 2048, per=13.73%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=16.67%, 100=16.67%, 250=16.67%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.10%, sys=5.26%, ctx=61, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30596: Tue Apr 8 02:28:06 2025 read: IOPS=1, BW=1112KiB/s (1139kB/s)(2048KiB/1842msec) clat (usec): min=15908, max=52871, avg=34390.08, stdev=26136.56 lat (usec): min=15909, max=52872, avg=34391.37, stdev=26136.66 clat percentiles (usec): | 1.00th=[15926], 5.00th=[15926], 10.00th=[15926], 20.00th=[15926], | 30.00th=[15926], 40.00th=[15926], 50.00th=[15926], 60.00th=[52691], | 70.00th=[52691], 80.00th=[52691], 90.00th=[52691], 95.00th=[52691], | 99.00th=[52691], 99.50th=[52691], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 2048, max= 2048, per=19.22%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5559KiB/s (5693kB/s)(10.0MiB/1842msec) clat (msec): min=21, max=804, avg=173.66, stdev=275.49 lat (msec): min=21, max=804, avg=173.72, stdev=275.48 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 31], 40.00th=[ 41], 50.00th=[ 47], 60.00th=[ 51], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 567], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min=10240, max=10240, per=68.65%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.05%, sys=7.82%, ctx=82, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30597: Tue Apr 8 02:28:06 2025 read: IOPS=3, BW=3623KiB/s (3710kB/s)(6144KiB/1696msec) clat (msec): min=13, max=720, avg=151.58, stdev=279.40 lat (msec): min=13, max=720, avg=151.58, stdev=279.40 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 59], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 8192, max= 8192, per=76.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3623KiB/s (3710kB/s)(6144KiB/1696msec) clat (msec): min=17, max=477, avg=119.88, stdev=176.87 lat (msec): min=17, max=477, avg=119.95, stdev=176.86 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 4096, max= 6144, per=34.32%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=7.85%, ctx=99, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30598: Tue Apr 8 02:28:06 2025 read: IOPS=2, BW=2965KiB/s (3036kB/s)(5120KiB/1727msec) clat (usec): min=13596, max=60252, avg=24806.87, stdev=20075.20 lat (usec): min=13597, max=60253, avg=24807.56, stdev=20075.20 clat percentiles (usec): | 1.00th=[13566], 5.00th=[13566], 10.00th=[13566], 20.00th=[13566], | 30.00th=[13698], 40.00th=[13698], 50.00th=[15008], 60.00th=[15008], | 70.00th=[21365], 80.00th=[21365], 90.00th=[60031], 95.00th=[60031], | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 8192, max= 8192, per=76.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4151KiB/s (4250kB/s)(7168KiB/1727msec) clat (msec): min=27, max=710, avg=219.14, stdev=265.40 lat (msec): min=27, max=710, avg=219.21, stdev=265.41 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 89], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 477], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 2048, max= 8192, per=34.32%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 20=25.00%, 50=16.67%, 100=33.33%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=6.66%, ctx=81, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.4MiB/s (10.9MB/s), 1112KiB/s-3729KiB/s (1139kB/s-3819kB/s), io=20.0MiB (20.0MB), run=1696-1922msec WRITE: bw=14.6MiB/s (15.3MB/s), 2664KiB/s-5559KiB/s (2728kB/s-5693kB/s), io=28.0MiB (29.4MB), run=1696-1922msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30608: Tue Apr 8 02:28:08 2025 read: IOPS=8, BW=32.7MiB/s (34.3MB/s)(12.0MiB/367msec) clat (msec): min=103, max=115, avg=108.53, stdev= 6.24 lat (msec): min=103, max=115, avg=108.53, stdev= 6.24 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 108], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=36.61%, ctx=2124, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30610: Tue Apr 8 02:28:08 2025 read: IOPS=1, BW=5588KiB/s (5722kB/s)(4096KiB/733msec) clat (nsec): min=98273k, max=98273k, avg=98272633.00, stdev= 0.00 lat (nsec): min=98274k, max=98274k, avg=98274288.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[98042], 5.00th=[98042], 10.00th=[98042], 20.00th=[98042], | 30.00th=[98042], 40.00th=[98042], 50.00th=[98042], 60.00th=[98042], | 70.00th=[98042], 80.00th=[98042], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 8192, max= 8192, per=36.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.9MiB/s (11.4MB/s)(8192KiB/733msec) clat (msec): min=246, max=342, avg=294.85, stdev=68.04 lat (msec): min=247, max=343, avg=295.23, stdev=67.93 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 342], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8192, max= 8192, per=18.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.27%, sys=29.10%, ctx=827, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30611: Tue Apr 8 02:28:08 2025 write: IOPS=4, BW=17.8MiB/s (18.6MB/s)(12.0MiB/675msec) clat (msec): min=141, max=310, avg=210.97, stdev=88.47 lat (msec): min=142, max=310, avg=211.26, stdev=88.46 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min=16384, max=16384, per=36.65%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=39.17%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30612: Tue Apr 8 02:28:08 2025 write: IOPS=5, BW=21.4MiB/s (22.5MB/s)(12.0MiB/560msec) clat (msec): min=101, max=249, avg=178.14, stdev=73.98 lat (msec): min=102, max=249, avg=178.42, stdev=73.99 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 251], 80.00th=[ 251], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min=16286, max=16286, per=36.43%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=46.69%, ctx=20, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.8MiB/s (22.9MB/s), 5588KiB/s-32.7MiB/s (5722kB/s-34.3MB/s), io=16.0MiB (16.8MB), run=367-733msec WRITE: bw=43.7MiB/s (45.8MB/s), 10.9MiB/s-21.4MiB/s (11.4MB/s-22.5MB/s), io=32.0MiB (33.6MB), run=560-733msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30605: Tue Apr 8 02:28:08 2025 read: IOPS=5, BW=20.4MiB/s (21.4MB/s)(12.0MiB/588msec) clat (msec): min=62, max=366, avg=191.28, stdev=156.99 lat (msec): min=62, max=366, avg=191.28, stdev=156.99 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min=15369, max=15369, per=64.35%, avg=15369.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.17%, sys=10.05%, ctx=36, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30606: Tue Apr 8 02:28:08 2025 read: IOPS=1, BW=5971KiB/s (6114kB/s)(4096KiB/686msec) clat (nsec): min=34147k, max=34147k, avg=34146930.00, stdev= 0.00 lat (nsec): min=34150k, max=34150k, avg=34149663.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[34341], 5.00th=[34341], 10.00th=[34341], 20.00th=[34341], | 30.00th=[34341], 40.00th=[34341], 50.00th=[34341], 60.00th=[34341], | 70.00th=[34341], 80.00th=[34341], 90.00th=[34341], 95.00th=[34341], | 99.00th=[34341], 99.50th=[34341], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 8192, max= 8192, per=34.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.7MiB/s (12.2MB/s)(8192KiB/686msec) clat (msec): min=247, max=362, avg=305.00, stdev=81.85 lat (msec): min=247, max=363, avg=305.28, stdev=81.83 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8192, max= 8192, per=22.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.15%, sys=13.43%, ctx=62, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30607: Tue Apr 8 02:28:08 2025 write: IOPS=3, BW=13.6MiB/s (14.2MB/s)(12.0MiB/884msec) clat (msec): min=96, max=442, avg=290.60, stdev=177.19 lat (msec): min=96, max=443, avg=290.81, stdev=177.26 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 8031, max= 8031, per=21.67%, avg=8031.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.11%, sys=21.97%, ctx=96, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30609: Tue Apr 8 02:28:08 2025 write: IOPS=7, BW=29.7MiB/s (31.1MB/s)(12.0MiB/404msec) clat (msec): min=54, max=259, avg=128.23, stdev=113.88 lat (msec): min=54, max=259, avg=128.53, stdev=113.84 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 71], 50.00th=[ 71], 60.00th=[ 71], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.00%, sys=15.38%, ctx=42, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.3MiB/s (24.5MB/s), 5971KiB/s-20.4MiB/s (6114kB/s-21.4MB/s), io=16.0MiB (16.8MB), run=588-686msec WRITE: bw=36.2MiB/s (37.0MB/s), 11.7MiB/s-29.7MiB/s (12.2MB/s-31.1MB/s), io=32.0MiB (33.6MB), run=404-884msec