== sanity test 398b: DIO and buffer IO race ============== 15:46:21 (1744055181) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.1553 s, 43.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28443: Mon Apr 7 15:46:35 2025 read: IOPS=131, BW=528KiB/s (540kB/s)(5992KiB/11357msec) clat (usec): min=54, max=164033, avg=4261.25, stdev=4689.56 lat (usec): min=55, max=164033, avg=4262.28, stdev=4689.65 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 1762], 10.00th=[ 2089], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 3195], 50.00th=[ 3785], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 5735], 90.00th=[ 6783], 95.00th=[ 8029], | 99.00th=[ 11207], 99.50th=[ 12780], 99.90th=[ 20841], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 272, max= 696, per=24.82%, avg=519.27, stdev=80.60, samples=22 iops : min= 68, max= 174, avg=129.82, stdev=20.15, samples=22 write: IOPS=138, BW=554KiB/s (568kB/s)(6296KiB/11357msec) clat (usec): min=402, max=17533, avg=3107.33, stdev=2680.69 lat (usec): min=403, max=17534, avg=3108.16, stdev=2680.56 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 750], 40.00th=[ 1860], 50.00th=[ 2573], 60.00th=[ 3261], | 70.00th=[ 4047], 80.00th=[ 5211], 90.00th=[ 6849], 95.00th=[ 8455], | 99.00th=[11207], 99.50th=[11994], 99.90th=[16909], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 344, max= 728, per=24.81%, avg=543.27, stdev=82.14, samples=22 iops : min= 86, max= 182, avg=135.82, stdev=20.53, samples=22 lat (usec) : 100=1.73%, 250=0.26%, 500=0.55%, 750=14.81%, 1000=2.90% lat (msec) : 2=4.88%, 4=36.33%, 10=36.39%, 20=2.08%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=22.69%, ctx=5479, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28445: Mon Apr 7 15:46:35 2025 read: IOPS=130, BW=523KiB/s (536kB/s)(5940KiB/11348msec) clat (usec): min=51, max=15736, avg=4087.00, stdev=2339.06 lat (usec): min=51, max=15737, avg=4089.50, stdev=2341.26 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1647], 10.00th=[ 1975], 20.00th=[ 2343], | 30.00th=[ 2704], 40.00th=[ 3163], 50.00th=[ 3654], 60.00th=[ 4228], | 70.00th=[ 4817], 80.00th=[ 5538], 90.00th=[ 6915], 95.00th=[ 8455], | 99.00th=[12780], 99.50th=[14091], 99.90th=[15401], 99.95th=[15795], | 99.99th=[15795] bw ( KiB/s): min= 280, max= 672, per=24.63%, avg=515.18, stdev=81.17, samples=22 iops : min= 70, max= 168, avg=128.77, stdev=20.30, samples=22 write: IOPS=139, BW=559KiB/s (573kB/s)(6348KiB/11348msec) clat (usec): min=431, max=163101, avg=3266.68, stdev=4949.62 lat (usec): min=432, max=163102, avg=3268.00, stdev=4949.67 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 586], | 30.00th=[ 734], 40.00th=[ 1827], 50.00th=[ 2507], 60.00th=[ 3064], | 70.00th=[ 3949], 80.00th=[ 5276], 90.00th=[ 7308], 95.00th=[ 8717], | 99.00th=[ 13435], 99.50th=[ 14746], 99.90th=[ 20579], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 352, max= 672, per=24.82%, avg=543.55, stdev=75.78, samples=22 iops : min= 88, max= 168, avg=135.86, stdev=18.95, samples=22 lat (usec) : 100=2.15%, 250=0.20%, 500=0.78%, 750=15.07%, 1000=1.92% lat (msec) : 2=6.02%, 4=37.63%, 10=33.63%, 20=2.54%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.43%, ctx=5718, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28447: Mon Apr 7 15:46:35 2025 read: IOPS=132, BW=528KiB/s (541kB/s)(6028KiB/11410msec) clat (usec): min=57, max=163143, avg=4158.39, stdev=4672.23 lat (usec): min=57, max=163144, avg=4159.35, stdev=4672.21 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 135], 10.00th=[ 1991], 20.00th=[ 2376], | 30.00th=[ 2737], 40.00th=[ 3163], 50.00th=[ 3654], 60.00th=[ 4228], | 70.00th=[ 4883], 80.00th=[ 5604], 90.00th=[ 6718], 95.00th=[ 8094], | 99.00th=[ 11469], 99.50th=[ 13698], 99.90th=[ 17433], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 264, max= 648, per=24.86%, avg=520.05, stdev=73.54, samples=22 iops : min= 66, max= 162, avg=129.91, stdev=18.38, samples=22 write: IOPS=137, BW=549KiB/s (562kB/s)(6260KiB/11410msec) clat (usec): min=437, max=18859, avg=3231.90, stdev=2906.80 lat (usec): min=437, max=18859, avg=3232.76, stdev=2906.89 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 594], | 30.00th=[ 742], 40.00th=[ 1795], 50.00th=[ 2540], 60.00th=[ 3195], | 70.00th=[ 4080], 80.00th=[ 5538], 90.00th=[ 7439], 95.00th=[ 8979], | 99.00th=[12387], 99.50th=[13304], 99.90th=[16319], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 328, max= 720, per=24.56%, avg=537.86, stdev=85.97, samples=22 iops : min= 82, max= 180, avg=134.36, stdev=21.54, samples=22 lat (usec) : 100=2.15%, 250=0.46%, 500=0.81%, 750=14.71%, 1000=2.57% lat (msec) : 2=5.57%, 4=36.56%, 10=34.83%, 20=2.31%, 250=0.03% cpu : usr=0.18%, sys=22.32%, ctx=5478, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28449: Mon Apr 7 15:46:35 2025 read: IOPS=131, BW=527KiB/s (540kB/s)(6052KiB/11477msec) clat (usec): min=50, max=17980, avg=4166.78, stdev=2293.86 lat (usec): min=50, max=17981, avg=4168.75, stdev=2294.67 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1762], 10.00th=[ 2057], 20.00th=[ 2409], | 30.00th=[ 2769], 40.00th=[ 3228], 50.00th=[ 3785], 60.00th=[ 4424], | 70.00th=[ 4948], 80.00th=[ 5669], 90.00th=[ 6849], 95.00th=[ 8160], | 99.00th=[12649], 99.50th=[14091], 99.90th=[17695], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 280, max= 696, per=24.73%, avg=517.36, stdev=89.54, samples=22 iops : min= 70, max= 174, avg=129.32, stdev=22.38, samples=22 write: IOPS=135, BW=543KiB/s (556kB/s)(6236KiB/11477msec) clat (usec): min=431, max=165150, avg=3261.38, stdev=4958.29 lat (usec): min=432, max=165150, avg=3262.04, stdev=4958.29 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 742], 40.00th=[ 1844], 50.00th=[ 2573], 60.00th=[ 3130], | 70.00th=[ 4146], 80.00th=[ 5276], 90.00th=[ 7046], 95.00th=[ 8717], | 99.00th=[ 11863], 99.50th=[ 13960], 99.90th=[ 17433], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 344, max= 752, per=24.27%, avg=531.59, stdev=87.73, samples=22 iops : min= 86, max= 188, avg=132.86, stdev=21.96, samples=22 lat (usec) : 100=1.79%, 250=0.23%, 500=0.55%, 750=14.78%, 1000=2.38% lat (msec) : 2=5.34%, 4=36.17%, 10=36.33%, 20=2.41%, 250=0.03% cpu : usr=0.20%, sys=22.33%, ctx=5733, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2092KiB/s (2142kB/s), 523KiB/s-528KiB/s (536kB/s-541kB/s), io=23.4MiB (24.6MB), run=11348-11477msec WRITE: bw=2190KiB/s (2243kB/s), 543KiB/s-559KiB/s (556kB/s-573kB/s), io=24.6MiB (25.7MB), run=11348-11477msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28444: Mon Apr 7 15:46:39 2025 read: IOPS=100, BW=403KiB/s (412kB/s)(5992KiB/14881msec) clat (usec): min=1535, max=166514, avg=4055.97, stdev=5099.20 lat (usec): min=1535, max=166516, avg=4056.66, stdev=5099.21 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2180], 40.00th=[ 2376], 50.00th=[ 2638], 60.00th=[ 3097], | 70.00th=[ 3949], 80.00th=[ 5538], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 14091], 99.50th=[ 15008], 99.90th=[ 19268], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 168, max= 784, per=24.80%, avg=396.52, stdev=176.80, samples=29 iops : min= 42, max= 196, avg=99.03, stdev=44.21, samples=29 write: IOPS=105, BW=423KiB/s (433kB/s)(6296KiB/14881msec) clat (usec): min=2160, max=24429, avg=5541.95, stdev=3340.33 lat (usec): min=2199, max=24430, avg=5542.86, stdev=3340.14 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2704], 10.00th=[ 2900], 20.00th=[ 3130], | 30.00th=[ 3359], 40.00th=[ 3621], 50.00th=[ 4113], 60.00th=[ 4948], | 70.00th=[ 6325], 80.00th=[ 7767], 90.00th=[10028], 95.00th=[12125], | 99.00th=[18220], 99.50th=[20579], 99.90th=[23462], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 184, max= 840, per=24.81%, avg=415.28, stdev=178.66, samples=29 iops : min= 46, max= 210, avg=103.72, stdev=44.69, samples=29 lat (msec) : 2=9.37%, 4=49.48%, 10=32.75%, 20=8.07%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.36%, ctx=3528, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28446: Mon Apr 7 15:46:39 2025 read: IOPS=99, BW=397KiB/s (407kB/s)(5940KiB/14961msec) clat (usec): min=1565, max=169178, avg=4073.92, stdev=5223.26 lat (usec): min=1566, max=169179, avg=4074.55, stdev=5223.26 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2311], 50.00th=[ 2638], 60.00th=[ 3064], | 70.00th=[ 3916], 80.00th=[ 5407], 90.00th=[ 8979], 95.00th=[ 10552], | 99.00th=[ 14615], 99.50th=[ 15533], 99.90th=[ 21103], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 120, max= 840, per=24.61%, avg=393.45, stdev=183.39, samples=29 iops : min= 30, max= 210, avg=98.28, stdev=45.80, samples=29 write: IOPS=106, BW=424KiB/s (434kB/s)(6348KiB/14961msec) clat (usec): min=2225, max=23213, avg=5562.98, stdev=3195.55 lat (usec): min=2226, max=23213, avg=5563.78, stdev=3195.53 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2769], 10.00th=[ 2966], 20.00th=[ 3195], | 30.00th=[ 3425], 40.00th=[ 3720], 50.00th=[ 4228], 60.00th=[ 5080], | 70.00th=[ 6390], 80.00th=[ 7767], 90.00th=[10159], 95.00th=[11731], | 99.00th=[16909], 99.50th=[20055], 99.90th=[22152], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 200, max= 760, per=24.77%, avg=414.69, stdev=168.86, samples=29 iops : min= 50, max= 190, avg=103.59, stdev=42.17, samples=29 lat (msec) : 2=9.57%, 4=48.47%, 10=33.27%, 20=8.33%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.50%, ctx=3567, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28448: Mon Apr 7 15:46:39 2025 read: IOPS=102, BW=411KiB/s (421kB/s)(6028KiB/14672msec) clat (usec): min=1530, max=19030, avg=3924.64, stdev=2980.09 lat (usec): min=1531, max=19030, avg=3925.26, stdev=2980.06 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2638], 60.00th=[ 2999], | 70.00th=[ 3818], 80.00th=[ 5342], 90.00th=[ 8455], 95.00th=[10683], | 99.00th=[15401], 99.50th=[16909], 99.90th=[18744], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 160, max= 776, per=25.47%, avg=407.28, stdev=169.40, samples=29 iops : min= 40, max= 194, avg=101.72, stdev=42.36, samples=29 write: IOPS=106, BW=427KiB/s (437kB/s)(6260KiB/14672msec) clat (msec): min=2, max=167, avg= 5.54, stdev= 5.18 lat (msec): min=2, max=167, avg= 5.54, stdev= 5.18 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 24], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 200, max= 784, per=25.22%, avg=422.17, stdev=177.85, samples=29 iops : min= 50, max= 196, avg=105.45, stdev=44.46, samples=29 lat (msec) : 2=9.51%, 4=49.80%, 10=32.85%, 20=7.45%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.38%, ctx=3551, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28450: Mon Apr 7 15:46:39 2025 read: IOPS=100, BW=403KiB/s (413kB/s)(6052KiB/15009msec) clat (usec): min=1537, max=18906, avg=3951.21, stdev=2889.03 lat (usec): min=1538, max=18906, avg=3951.84, stdev=2889.02 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1991], | 30.00th=[ 2147], 40.00th=[ 2409], 50.00th=[ 2704], 60.00th=[ 3130], | 70.00th=[ 4015], 80.00th=[ 5538], 90.00th=[ 8356], 95.00th=[10159], | 99.00th=[14615], 99.50th=[16450], 99.90th=[18482], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 167, max= 768, per=25.21%, avg=403.13, stdev=175.67, samples=30 iops : min= 41, max= 192, avg=100.73, stdev=43.97, samples=30 write: IOPS=103, BW=415KiB/s (425kB/s)(6236KiB/15009msec) clat (msec): min=2, max=159, avg= 5.73, stdev= 5.19 lat (msec): min=2, max=159, avg= 5.73, stdev= 5.19 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 22], 99.90th=[ 25], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 191, max= 832, per=24.81%, avg=415.40, stdev=182.74, samples=30 iops : min= 47, max= 208, avg=103.80, stdev=45.72, samples=30 lat (msec) : 2=10.19%, 4=48.47%, 10=33.01%, 20=7.94%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=11.27%, ctx=3509, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1600KiB/s (1638kB/s), 397KiB/s-411KiB/s (407kB/s-421kB/s), io=23.4MiB (24.6MB), run=14672-15009msec WRITE: bw=1675KiB/s (1715kB/s), 415KiB/s-427KiB/s (425kB/s-437kB/s), io=24.6MiB (25.7MB), run=14672-15009msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28477: Mon Apr 7 15:46:43 2025 read: IOPS=118, BW=1903KiB/s (1949kB/s)(5920KiB/3111msec) clat (usec): min=66, max=17732, avg=4528.51, stdev=2852.07 lat (usec): min=67, max=17733, avg=4529.18, stdev=2852.04 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 79], 10.00th=[ 97], 20.00th=[ 2638], | 30.00th=[ 3097], 40.00th=[ 3490], 50.00th=[ 4146], 60.00th=[ 4883], | 70.00th=[ 5473], 80.00th=[ 6390], 90.00th=[ 8094], 95.00th=[ 9241], | 99.00th=[14222], 99.50th=[14746], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1472, max= 2080, per=24.67%, avg=1860.67, stdev=213.41, samples=6 iops : min= 92, max= 130, avg=116.17, stdev=13.30, samples=6 write: IOPS=127, BW=2047KiB/s (2096kB/s)(6368KiB/3111msec) clat (usec): min=642, max=21696, avg=3508.28, stdev=3165.23 lat (usec): min=643, max=21698, avg=3509.11, stdev=3165.33 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 783], 20.00th=[ 824], | 30.00th=[ 930], 40.00th=[ 1811], 50.00th=[ 2606], 60.00th=[ 3359], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[13173], 99.50th=[21103], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 1664, max= 2496, per=25.27%, avg=2004.67, stdev=305.85, samples=6 iops : min= 104, max= 156, avg=125.17, stdev=19.21, samples=6 lat (usec) : 100=5.08%, 250=0.39%, 750=2.34%, 1000=14.06% lat (msec) : 2=5.21%, 4=29.95%, 10=38.67%, 20=4.04%, 50=0.26% cpu : usr=0.13%, sys=23.89%, ctx=1439, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28478: Mon Apr 7 15:46:43 2025 read: IOPS=113, BW=1809KiB/s (1852kB/s)(5744KiB/3176msec) clat (usec): min=56, max=15239, avg=4467.68, stdev=2419.23 lat (usec): min=56, max=15240, avg=4469.02, stdev=2419.19 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 74], 10.00th=[ 2147], 20.00th=[ 2802], | 30.00th=[ 3392], 40.00th=[ 3818], 50.00th=[ 4228], 60.00th=[ 4883], | 70.00th=[ 5342], 80.00th=[ 6063], 90.00th=[ 7177], 95.00th=[ 8848], | 99.00th=[12911], 99.50th=[13829], 99.90th=[15270], 99.95th=[15270], | 99.99th=[15270] bw ( KiB/s): min= 1469, max= 2048, per=23.69%, avg=1786.17, stdev=230.17, samples=6 iops : min= 91, max= 128, avg=111.50, stdev=14.61, samples=6 write: IOPS=128, BW=2060KiB/s (2110kB/s)(6544KiB/3176msec) clat (usec): min=646, max=14769, avg=3728.55, stdev=3080.56 lat (usec): min=647, max=14769, avg=3731.24, stdev=3080.39 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 1172], 40.00th=[ 2278], 50.00th=[ 2900], 60.00th=[ 3752], | 70.00th=[ 4883], 80.00th=[ 6194], 90.00th=[ 8225], 95.00th=[10159], | 99.00th=[13042], 99.50th=[13829], 99.90th=[14746], 99.95th=[14746], | 99.99th=[14746] bw ( KiB/s): min= 1664, max= 2464, per=25.40%, avg=2015.17, stdev=290.75, samples=6 iops : min= 104, max= 154, avg=125.83, stdev=18.12, samples=6 lat (usec) : 100=3.78%, 250=0.39%, 750=1.82%, 1000=12.63% lat (msec) : 2=5.34%, 4=30.21%, 10=41.80%, 20=4.04% cpu : usr=0.13%, sys=24.00%, ctx=1496, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28480: Mon Apr 7 15:46:43 2025 read: IOPS=124, BW=1991KiB/s (2039kB/s)(6256KiB/3142msec) clat (usec): min=57, max=17391, avg=4379.35, stdev=2697.32 lat (usec): min=58, max=17392, avg=4379.95, stdev=2697.32 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 75], 10.00th=[ 2024], 20.00th=[ 2573], | 30.00th=[ 2999], 40.00th=[ 3425], 50.00th=[ 4113], 60.00th=[ 4686], | 70.00th=[ 5211], 80.00th=[ 6063], 90.00th=[ 7373], 95.00th=[ 9110], | 99.00th=[14222], 99.50th=[15139], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1408, max= 2496, per=26.52%, avg=2000.00, stdev=362.46, samples=6 iops : min= 88, max= 156, avg=125.00, stdev=22.65, samples=6 write: IOPS=119, BW=1920KiB/s (1966kB/s)(6032KiB/3142msec) clat (usec): min=628, max=14170, avg=3739.58, stdev=3017.18 lat (usec): min=629, max=14170, avg=3743.46, stdev=3017.78 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 848], | 30.00th=[ 1303], 40.00th=[ 2245], 50.00th=[ 2999], 60.00th=[ 3752], | 70.00th=[ 4883], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[10159], | 99.00th=[12518], 99.50th=[13698], 99.90th=[14222], 99.95th=[14222], | 99.99th=[14222] bw ( KiB/s): min= 1760, max= 2080, per=23.73%, avg=1882.67, stdev=154.69, samples=6 iops : min= 110, max= 130, avg=117.67, stdev= 9.67, samples=6 lat (usec) : 100=4.43%, 250=0.52%, 750=2.34%, 1000=10.55% lat (msec) : 2=5.08%, 4=33.20%, 10=39.32%, 20=4.56% cpu : usr=0.25%, sys=23.72%, ctx=1457, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28482: Mon Apr 7 15:46:43 2025 read: IOPS=120, BW=1926KiB/s (1972kB/s)(6032KiB/3132msec) clat (usec): min=58, max=16030, avg=4420.74, stdev=2710.28 lat (usec): min=58, max=16031, avg=4421.30, stdev=2710.29 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 78], 10.00th=[ 101], 20.00th=[ 2507], | 30.00th=[ 3064], 40.00th=[ 3589], 50.00th=[ 4178], 60.00th=[ 4621], | 70.00th=[ 5473], 80.00th=[ 6652], 90.00th=[ 7701], 95.00th=[ 9110], | 99.00th=[12256], 99.50th=[15533], 99.90th=[16057], 99.95th=[16057], | 99.99th=[16057] bw ( KiB/s): min= 1504, max= 2112, per=25.10%, avg=1892.50, stdev=236.33, samples=6 iops : min= 94, max= 132, avg=118.17, stdev=14.65, samples=6 write: IOPS=124, BW=1997KiB/s (2045kB/s)(6256KiB/3132msec) clat (usec): min=664, max=14433, avg=3605.88, stdev=3214.75 lat (usec): min=665, max=14433, avg=3608.17, stdev=3218.38 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 824], | 30.00th=[ 898], 40.00th=[ 1287], 50.00th=[ 2606], 60.00th=[ 3392], | 70.00th=[ 5080], 80.00th=[ 6456], 90.00th=[ 8160], 95.00th=[10683], | 99.00th=[13042], 99.50th=[14353], 99.90th=[14484], 99.95th=[14484], | 99.99th=[14484] bw ( KiB/s): min= 1756, max= 2304, per=24.86%, avg=1972.67, stdev=224.92, samples=6 iops : min= 109, max= 144, avg=123.17, stdev=14.20, samples=6 lat (usec) : 100=4.82%, 250=0.65%, 750=1.82%, 1000=15.62% lat (msec) : 2=6.38%, 4=27.08%, 10=39.45%, 20=4.17% cpu : usr=0.03%, sys=23.79%, ctx=1434, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7542KiB/s (7723kB/s), 1809KiB/s-1991KiB/s (1852kB/s-2039kB/s), io=23.4MiB (24.5MB), run=3111-3176msec WRITE: bw=7935KiB/s (8125kB/s), 1920KiB/s-2060KiB/s (1966kB/s-2110kB/s), io=24.6MiB (25.8MB), run=3111-3176msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28479: Mon Apr 7 15:46:44 2025 read: IOPS=81, BW=1311KiB/s (1343kB/s)(5920KiB/4515msec) clat (usec): min=1957, max=25473, avg=4831.97, stdev=3250.14 lat (usec): min=1958, max=25474, avg=4832.69, stdev=3250.10 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2311], 10.00th=[ 2540], 20.00th=[ 2868], | 30.00th=[ 3195], 40.00th=[ 3359], 50.00th=[ 3621], 60.00th=[ 3884], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 9503], 95.00th=[11731], | 99.00th=[17957], 99.50th=[21890], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 862, max= 1920, per=25.19%, avg=1307.89, stdev=363.05, samples=9 iops : min= 53, max= 120, avg=81.56, stdev=22.88, samples=9 write: IOPS=88, BW=1410KiB/s (1444kB/s)(6368KiB/4515msec) clat (usec): min=2778, max=26039, avg=6792.34, stdev=3445.69 lat (usec): min=2780, max=26039, avg=6793.72, stdev=3445.57 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3621], 10.00th=[ 3851], 20.00th=[ 4359], | 30.00th=[ 4752], 40.00th=[ 5080], 50.00th=[ 5473], 60.00th=[ 6063], | 70.00th=[ 7308], 80.00th=[ 9110], 90.00th=[11731], 95.00th=[13829], | 99.00th=[17957], 99.50th=[25035], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 958, max= 1920, per=25.76%, avg=1407.44, stdev=374.39, samples=9 iops : min= 59, max= 120, avg=87.78, stdev=23.59, samples=9 lat (msec) : 2=0.39%, 4=35.94%, 10=51.30%, 20=11.72%, 50=0.65% cpu : usr=0.11%, sys=12.65%, ctx=906, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28481: Mon Apr 7 15:46:44 2025 read: IOPS=77, BW=1245KiB/s (1275kB/s)(5744KiB/4612msec) clat (usec): min=1847, max=18872, avg=4838.38, stdev=2932.76 lat (usec): min=1848, max=18872, avg=4839.11, stdev=2932.70 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2376], 10.00th=[ 2671], 20.00th=[ 2999], | 30.00th=[ 3228], 40.00th=[ 3523], 50.00th=[ 3687], 60.00th=[ 4015], | 70.00th=[ 4752], 80.00th=[ 6259], 90.00th=[ 9503], 95.00th=[11600], | 99.00th=[15795], 99.50th=[17957], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 864, max= 1856, per=24.03%, avg=1248.00, stdev=376.94, samples=9 iops : min= 54, max= 116, avg=78.00, stdev=23.56, samples=9 write: IOPS=88, BW=1419KiB/s (1453kB/s)(6544KiB/4612msec) clat (usec): min=2642, max=25968, avg=6848.82, stdev=3358.08 lat (usec): min=2643, max=25969, avg=6850.36, stdev=3357.98 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3523], 10.00th=[ 3818], 20.00th=[ 4359], | 30.00th=[ 4817], 40.00th=[ 5211], 50.00th=[ 5604], 60.00th=[ 6194], | 70.00th=[ 7373], 80.00th=[ 9372], 90.00th=[11863], 95.00th=[13304], | 99.00th=[17171], 99.50th=[20579], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1088, max= 2016, per=25.83%, avg=1411.56, stdev=366.06, samples=9 iops : min= 68, max= 126, avg=88.22, stdev=22.88, samples=9 lat (msec) : 2=0.26%, 4=34.64%, 10=51.82%, 20=12.89%, 50=0.39% cpu : usr=0.09%, sys=12.71%, ctx=906, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28483: Mon Apr 7 15:46:44 2025 read: IOPS=86, BW=1386KiB/s (1419kB/s)(6256KiB/4514msec) clat (usec): min=2022, max=17591, avg=4733.95, stdev=2876.59 lat (usec): min=2023, max=17591, avg=4734.67, stdev=2876.53 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2376], 10.00th=[ 2540], 20.00th=[ 2802], | 30.00th=[ 3130], 40.00th=[ 3326], 50.00th=[ 3556], 60.00th=[ 3884], | 70.00th=[ 4490], 80.00th=[ 6325], 90.00th=[ 8979], 95.00th=[11207], | 99.00th=[15270], 99.50th=[15795], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 864, max= 2048, per=26.77%, avg=1390.22, stdev=432.63, samples=9 iops : min= 54, max= 128, avg=86.89, stdev=27.04, samples=9 write: IOPS=83, BW=1336KiB/s (1368kB/s)(6032KiB/4514msec) clat (usec): min=3065, max=25699, avg=6898.12, stdev=3335.94 lat (usec): min=3066, max=25700, avg=6899.63, stdev=3335.84 clat percentiles (usec): | 1.00th=[ 3261], 5.00th=[ 3589], 10.00th=[ 3982], 20.00th=[ 4555], | 30.00th=[ 4883], 40.00th=[ 5342], 50.00th=[ 5735], 60.00th=[ 6194], | 70.00th=[ 7635], 80.00th=[ 8979], 90.00th=[11731], 95.00th=[13304], | 99.00th=[19792], 99.50th=[21103], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 896, max= 1824, per=24.34%, avg=1329.78, stdev=289.82, samples=9 iops : min= 56, max= 114, avg=83.11, stdev=18.11, samples=9 lat (msec) : 4=36.46%, 10=51.69%, 20=11.46%, 50=0.39% cpu : usr=0.11%, sys=12.94%, ctx=911, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28484: Mon Apr 7 15:46:44 2025 read: IOPS=83, BW=1343KiB/s (1376kB/s)(6032KiB/4490msec) clat (usec): min=1900, max=21956, avg=4931.97, stdev=3264.55 lat (usec): min=1901, max=21957, avg=4932.80, stdev=3264.42 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2311], 10.00th=[ 2573], 20.00th=[ 2802], | 30.00th=[ 3195], 40.00th=[ 3425], 50.00th=[ 3621], 60.00th=[ 3884], | 70.00th=[ 4555], 80.00th=[ 6652], 90.00th=[10028], 95.00th=[12518], | 99.00th=[17695], 99.50th=[19792], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 894, max= 1856, per=24.64%, avg=1279.38, stdev=315.12, samples=8 iops : min= 55, max= 116, avg=79.75, stdev=19.93, samples=8 write: IOPS=87, BW=1393KiB/s (1427kB/s)(6256KiB/4490msec) clat (usec): min=2726, max=24913, avg=6475.83, stdev=3255.19 lat (usec): min=2727, max=24916, avg=6477.41, stdev=3255.23 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3425], 10.00th=[ 3621], 20.00th=[ 4047], | 30.00th=[ 4621], 40.00th=[ 4948], 50.00th=[ 5342], 60.00th=[ 5866], | 70.00th=[ 6783], 80.00th=[ 8455], 90.00th=[11207], 95.00th=[13304], | 99.00th=[18482], 99.50th=[19792], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 1053, max= 1952, per=24.44%, avg=1335.25, stdev=322.91, samples=8 iops : min= 65, max= 122, avg=83.25, stdev=20.39, samples=8 lat (msec) : 2=0.39%, 4=39.97%, 10=47.66%, 20=11.72%, 50=0.26% cpu : usr=0.09%, sys=12.90%, ctx=926, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5193KiB/s (5318kB/s), 1245KiB/s-1386KiB/s (1275kB/s-1419kB/s), io=23.4MiB (24.5MB), run=4490-4612msec WRITE: bw=5464KiB/s (5595kB/s), 1336KiB/s-1419KiB/s (1368kB/s-1453kB/s), io=24.6MiB (25.8MB), run=4490-4612msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28497: Mon Apr 7 15:46:46 2025 read: IOPS=16, BW=16.9MiB/s (17.7MB/s)(7168KiB/414msec) clat (usec): min=14486, max=80553, avg=29921.07, stdev=23603.01 lat (usec): min=14486, max=80554, avg=29921.88, stdev=23603.05 clat percentiles (usec): | 1.00th=[14484], 5.00th=[14484], 10.00th=[14484], 20.00th=[15533], | 30.00th=[17171], 40.00th=[17171], 50.00th=[20055], 60.00th=[25035], | 70.00th=[25035], 80.00th=[36439], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] write: IOPS=12, BW=12.1MiB/s (12.7MB/s)(5120KiB/414msec) clat (msec): min=18, max=106, avg=37.12, stdev=39.02 lat (msec): min=18, max=106, avg=37.14, stdev=39.02 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 20], | 70.00th=[ 23], 80.00th=[ 23], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] lat (msec) : 20=50.00%, 50=33.33%, 100=8.33%, 250=8.33% cpu : usr=0.24%, sys=19.13%, ctx=105, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28498: Mon Apr 7 15:46:46 2025 read: IOPS=2, BW=2656KiB/s (2720kB/s)(2048KiB/771msec) clat (usec): min=29061, max=58290, avg=43675.79, stdev=20667.98 lat (usec): min=29062, max=58291, avg=43676.63, stdev=20668.17 clat percentiles (usec): | 1.00th=[28967], 5.00th=[28967], 10.00th=[28967], 20.00th=[28967], | 30.00th=[28967], 40.00th=[28967], 50.00th=[28967], 60.00th=[58459], | 70.00th=[58459], 80.00th=[58459], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 4096, max= 4096, per=17.68%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.0MiB/s (13.6MB/s)(10.0MiB/771msec) clat (msec): min=19, max=147, avg=61.97, stdev=37.25 lat (msec): min=19, max=147, avg=62.01, stdev=37.25 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 21], | 30.00th=[ 44], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 61], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 87], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min=10240, max=10240, per=31.57%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.13%, sys=12.47%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28499: Mon Apr 7 15:46:46 2025 read: IOPS=8, BW=8904KiB/s (9118kB/s)(6144KiB/690msec) clat (msec): min=13, max=116, avg=43.99, stdev=42.56 lat (msec): min=13, max=116, avg=43.99, stdev=42.56 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 16], 60.00th=[ 29], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 4096, max= 4096, per=17.68%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8904KiB/s (9118kB/s)(6144KiB/690msec) clat (msec): min=19, max=162, avg=69.76, stdev=52.83 lat (msec): min=19, max=163, avg=69.81, stdev=52.84 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 61], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 6144, max= 6144, per=18.94%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=25.00%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=14.37%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28500: Mon Apr 7 15:46:46 2025 read: IOPS=5, BW=5792KiB/s (5931kB/s)(5120KiB/884msec) clat (msec): min=16, max=109, avg=46.70, stdev=42.34 lat (msec): min=16, max=109, avg=46.70, stdev=42.34 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 20], 60.00th=[ 20], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] write: IOPS=7, BW=8109KiB/s (8303kB/s)(7168KiB/884msec) clat (msec): min=16, max=229, avg=88.04, stdev=75.82 lat (msec): min=16, max=229, avg=88.08, stdev=75.82 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 33], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 62], 60.00th=[ 86], | 70.00th=[ 86], 80.00th=[ 148], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 8192, max= 8192, per=25.26%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=16.67%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=10.53%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.6MiB/s (23.7MB/s), 2656KiB/s-16.9MiB/s (2720kB/s-17.7MB/s), io=20.0MiB (20.0MB), run=414-884msec WRITE: bw=31.7MiB/s (33.2MB/s), 8109KiB/s-12.0MiB/s (8303kB/s-13.6MB/s), io=28.0MiB (29.4MB), run=414-884msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28493: Mon Apr 7 15:46:46 2025 read: IOPS=7, BW=7553KiB/s (7734kB/s)(7168KiB/949msec) clat (msec): min=24, max=447, avg=97.40, stdev=155.39 lat (msec): min=24, max=447, avg=97.40, stdev=155.39 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 25], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 60], | 70.00th=[ 60], 80.00th=[ 66], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 4096, max= 4096, per=18.98%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5395KiB/s (5525kB/s)(5120KiB/949msec) clat (usec): min=26764, max=42892, avg=34249.53, stdev=5785.39 lat (usec): min=26812, max=42912, avg=34274.50, stdev=5776.51 clat percentiles (usec): | 1.00th=[26870], 5.00th=[26870], 10.00th=[26870], 20.00th=[26870], | 30.00th=[32637], 40.00th=[32637], 50.00th=[34341], 60.00th=[34341], | 70.00th=[34866], 80.00th=[34866], 90.00th=[42730], 95.00th=[42730], | 99.00th=[42730], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] lat (msec) : 50=75.00%, 100=16.67%, 500=8.33% cpu : usr=0.00%, sys=28.69%, ctx=1470, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28494: Mon Apr 7 15:46:46 2025 read: IOPS=2, BW=2938KiB/s (3009kB/s)(2048KiB/697msec) clat (usec): min=34003, max=87043, avg=60523.32, stdev=37504.71 lat (usec): min=34005, max=87043, avg=60524.54, stdev=37503.93 clat percentiles (usec): | 1.00th=[33817], 5.00th=[33817], 10.00th=[33817], 20.00th=[33817], | 30.00th=[33817], 40.00th=[33817], 50.00th=[33817], 60.00th=[87557], | 70.00th=[87557], 80.00th=[87557], 90.00th=[87557], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 4096, max= 4096, per=18.98%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.3MiB/s (15.0MB/s)(10.0MiB/697msec) clat (msec): min=25, max=125, avg=57.11, stdev=32.20 lat (msec): min=25, max=125, avg=57.14, stdev=32.20 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 32], 40.00th=[ 36], 50.00th=[ 39], 60.00th=[ 61], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 81], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min=14336, max=14336, per=47.45%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=36.49%, ctx=372, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28495: Mon Apr 7 15:46:46 2025 read: IOPS=9, BW=9974KiB/s (10.2MB/s)(6144KiB/616msec) clat (usec): min=284, max=137929, avg=33849.86, stdev=53266.83 lat (usec): min=285, max=137930, avg=33850.58, stdev=53266.80 clat percentiles (usec): | 1.00th=[ 285], 5.00th=[ 285], 10.00th=[ 285], 20.00th=[ 293], | 30.00th=[ 293], 40.00th=[ 922], 50.00th=[ 922], 60.00th=[ 29754], | 70.00th=[ 33817], 80.00th=[ 33817], 90.00th=[137364], 95.00th=[137364], | 99.00th=[137364], 99.50th=[137364], 99.90th=[137364], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min=10240, max=10240, per=47.45%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9974KiB/s (10.2MB/s)(6144KiB/616msec) clat (msec): min=22, max=160, avg=62.51, stdev=50.78 lat (msec): min=22, max=160, avg=62.55, stdev=50.78 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 56], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min=12288, max=12288, per=40.67%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 1000=8.33% lat (msec) : 50=41.67%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=27.97%, ctx=725, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28496: Mon Apr 7 15:46:46 2025 read: IOPS=8, BW=8352KiB/s (8553kB/s)(5120KiB/613msec) clat (usec): min=338, max=61262, avg=24523.22, stdev=25473.22 lat (usec): min=338, max=61262, avg=24523.87, stdev=25473.34 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[28181], 60.00th=[28181], | 70.00th=[32375], 80.00th=[32375], 90.00th=[61080], 95.00th=[61080], | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 8192, max= 8192, per=37.96%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.4MiB/s (11.0MB/s)(7168KiB/613msec) clat (msec): min=22, max=154, avg=64.93, stdev=46.22 lat (msec): min=22, max=154, avg=64.98, stdev=46.21 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 35], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 52], 80.00th=[ 101], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min=10240, max=10240, per=33.89%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=27.29%, ctx=288, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.1MiB/s (22.1MB/s), 2938KiB/s-9974KiB/s (3009kB/s-10.2MB/s), io=20.0MiB (20.0MB), run=613-949msec WRITE: bw=29.5MiB/s (30.9MB/s), 5395KiB/s-14.3MiB/s (5525kB/s-15.0MB/s), io=28.0MiB (29.4MB), run=613-949msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28508: Mon Apr 7 15:46:48 2025 read: IOPS=3, BW=13.8MiB/s (14.5MB/s)(12.0MiB/867msec) clat (msec): min=125, max=565, avg=280.79, stdev=247.30 lat (msec): min=125, max=565, avg=280.80, stdev=247.30 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.23%, sys=24.83%, ctx=2014, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28510: Mon Apr 7 15:46:48 2025 read: IOPS=0, BW=4076KiB/s (4173kB/s)(4096KiB/1005msec) clat (nsec): min=149941k, max=149941k, avg=149941471.00, stdev= 0.00 lat (nsec): min=149942k, max=149942k, avg=149942388.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 8175, max= 8175, per=50.15%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=8151KiB/s (8347kB/s)(8192KiB/1005msec) clat (msec): min=102, max=737, avg=419.70, stdev=448.96 lat (msec): min=102, max=737, avg=419.87, stdev=448.88 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 735], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=23.71%, ctx=903, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28512: Mon Apr 7 15:46:48 2025 write: IOPS=2, BW=9.85MiB/s (10.3MB/s)(12.0MiB/1218msec) clat (msec): min=217, max=710, avg=398.78, stdev=271.39 lat (msec): min=217, max=711, avg=399.03, stdev=271.38 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7922, max= 8192, per=29.95%, avg=8057.00, stdev=190.92, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=22.02%, ctx=59, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28514: Mon Apr 7 15:46:48 2025 write: IOPS=5, BW=21.0MiB/s (22.0MB/s)(12.0MiB/571msec) clat (msec): min=163, max=199, avg=185.68, stdev=19.16 lat (msec): min=164, max=199, avg=185.86, stdev=19.09 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min=16384, max=16384, per=60.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=42.46%, ctx=81, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.9MiB/s (16.7MB/s), 4076KiB/s-13.8MiB/s (4173kB/s-14.5MB/s), io=16.0MiB (16.8MB), run=867-1005msec WRITE: bw=26.3MiB/s (27.5MB/s), 8151KiB/s-21.0MiB/s (8347kB/s-22.0MB/s), io=32.0MiB (33.6MB), run=571-1218msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28507: Mon Apr 7 15:46:48 2025 read: IOPS=7, BW=28.0MiB/s (30.4MB/s)(12.0MiB/414msec) clat (msec): min=78, max=195, avg=129.01, stdev=60.46 lat (msec): min=78, max=195, avg=129.01, stdev=60.46 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 113], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=17.92%, ctx=57, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28509: Mon Apr 7 15:46:48 2025 read: IOPS=0, BW=3468KiB/s (3551kB/s)(4096KiB/1181msec) clat (nsec): min=356605k, max=356605k, avg=356604567.00, stdev= 0.00 lat (nsec): min=356607k, max=356607k, avg=356606600.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8192, max= 8192, per=59.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6936KiB/s (7103kB/s)(8192KiB/1181msec) clat (msec): min=104, max=694, avg=399.34, stdev=417.56 lat (msec): min=104, max=694, avg=399.51, stdev=417.61 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 693], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.17%, sys=10.93%, ctx=64, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28511: Mon Apr 7 15:46:48 2025 write: IOPS=2, BW=9253KiB/s (9475kB/s)(12.0MiB/1328msec) clat (msec): min=151, max=701, avg=438.98, stdev=276.15 lat (msec): min=151, max=702, avg=439.22, stdev=276.17 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 8000, max= 8192, per=32.81%, avg=8096.00, stdev=135.76, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.15%, sys=10.25%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28513: Mon Apr 7 15:46:48 2025 write: IOPS=5, BW=22.3MiB/s (23.4MB/s)(12.0MiB/538msec) clat (msec): min=85, max=214, avg=169.80, stdev=73.36 lat (msec): min=85, max=214, avg=170.09, stdev=73.29 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min=16384, max=16384, per=66.40%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=11.55%, ctx=53, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.5MiB/s (14.2MB/s), 3468KiB/s-28.0MiB/s (3551kB/s-30.4MB/s), io=16.0MiB (16.8MB), run=414-1181msec WRITE: bw=24.1MiB/s (25.3MB/s), 6936KiB/s-22.3MiB/s (7103kB/s-23.4MB/s), io=32.0MiB (33.6MB), run=538-1328msec