== sanity test 398c: run fio to test AIO ================= 18:28:28 (1775168908) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.180691 s, 232 MB/s osc.lustre-OST0000-osc-ffff8b9682e75000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=152644: Thu Apr 2 18:29:27 2026 write: IOPS=47, BW=189KiB/s (193kB/s)(10.0MiB/54309msec); 0 zone resets slat (usec): min=37, max=3634, avg=251.66, stdev=238.26 clat (msec): min=66, max=680, avg=338.86, stdev=79.52 lat (msec): min=67, max=680, avg=339.11, stdev=79.52 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 226], 10.00th=[ 247], 20.00th=[ 275], | 30.00th=[ 296], 40.00th=[ 317], 50.00th=[ 338], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 401], 90.00th=[ 439], 95.00th=[ 472], | 99.00th=[ 542], 99.50th=[ 575], 99.90th=[ 642], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 95, max= 370, per=24.90%, avg=187.46, stdev=45.08, samples=108 iops : min= 23, max= 92, avg=46.68, stdev=11.32, samples=108 lat (msec) : 100=0.31%, 250=11.02%, 500=85.90%, 750=2.77% cpu : usr=0.10%, sys=0.85%, ctx=3080, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152645: Thu Apr 2 18:29:27 2026 write: IOPS=47, BW=189KiB/s (194kB/s)(10.0MiB/54162msec); 0 zone resets slat (usec): min=37, max=4509, avg=253.03, stdev=234.44 clat (msec): min=57, max=792, avg=338.05, stdev=82.80 lat (msec): min=57, max=793, avg=338.31, stdev=82.81 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 222], 10.00th=[ 243], 20.00th=[ 275], | 30.00th=[ 296], 40.00th=[ 317], 50.00th=[ 338], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 401], 90.00th=[ 439], 95.00th=[ 472], | 99.00th=[ 550], 99.50th=[ 609], 99.90th=[ 735], 99.95th=[ 776], | 99.99th=[ 793] bw ( KiB/s): min= 103, max= 415, per=25.09%, avg=188.96, stdev=43.34, samples=107 iops : min= 25, max= 103, avg=47.05, stdev=10.86, samples=107 lat (msec) : 100=0.62%, 250=11.33%, 500=85.43%, 750=2.54%, 1000=0.08% cpu : usr=0.09%, sys=0.88%, ctx=3013, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152646: Thu Apr 2 18:29:27 2026 write: IOPS=47, BW=189KiB/s (193kB/s)(10.0MiB/54310msec); 0 zone resets slat (usec): min=36, max=3416, avg=249.17, stdev=220.78 clat (msec): min=57, max=681, avg=338.87, stdev=79.54 lat (msec): min=57, max=682, avg=339.12, stdev=79.55 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 222], 10.00th=[ 245], 20.00th=[ 279], | 30.00th=[ 296], 40.00th=[ 313], 50.00th=[ 334], 60.00th=[ 359], | 70.00th=[ 380], 80.00th=[ 401], 90.00th=[ 439], 95.00th=[ 472], | 99.00th=[ 531], 99.50th=[ 584], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 63, max= 363, per=24.90%, avg=187.46, stdev=46.43, samples=108 iops : min= 15, max= 90, avg=46.68, stdev=11.62, samples=108 lat (msec) : 100=0.62%, 250=10.55%, 500=86.25%, 750=2.58% cpu : usr=0.09%, sys=0.84%, ctx=2980, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152647: Thu Apr 2 18:29:27 2026 write: IOPS=47, BW=188KiB/s (193kB/s)(10.0MiB/54336msec); 0 zone resets slat (usec): min=37, max=3998, avg=256.59, stdev=236.30 clat (msec): min=40, max=664, avg=339.00, stdev=79.50 lat (msec): min=40, max=664, avg=339.26, stdev=79.50 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 222], 10.00th=[ 247], 20.00th=[ 275], | 30.00th=[ 296], 40.00th=[ 317], 50.00th=[ 342], 60.00th=[ 359], | 70.00th=[ 380], 80.00th=[ 401], 90.00th=[ 435], 95.00th=[ 472], | 99.00th=[ 535], 99.50th=[ 575], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 104, max= 363, per=24.90%, avg=187.46, stdev=46.06, samples=108 iops : min= 26, max= 90, avg=46.68, stdev=11.56, samples=108 lat (msec) : 50=0.08%, 100=0.20%, 250=10.82%, 500=85.94%, 750=2.97% cpu : usr=0.09%, sys=0.85%, ctx=3067, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=754KiB/s (772kB/s), 188KiB/s-189KiB/s (193kB/s-194kB/s), io=40.0MiB (41.9MB), run=54162-54336msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=152664: Thu Apr 2 18:30:10 2026 read: IOPS=31, BW=128KiB/s (131kB/s)(5048KiB/39465msec) slat (usec): min=1706, max=13894, avg=4692.55, stdev=1740.75 clat (msec): min=22, max=413, avg=157.92, stdev=34.38 lat (msec): min=25, max=419, avg=162.61, stdev=34.82 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 111], 10.00th=[ 120], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 184], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 249], 99.50th=[ 257], 99.90th=[ 355], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 63, max= 232, per=25.26%, avg=128.08, stdev=37.45, samples=78 iops : min= 15, max= 58, avg=31.81, stdev= 9.38, samples=78 write: IOPS=32, BW=132KiB/s (135kB/s)(5192KiB/39465msec); 0 zone resets slat (usec): min=56, max=3408, avg=264.93, stdev=247.73 clat (msec): min=136, max=560, avg=326.93, stdev=80.64 lat (msec): min=136, max=560, avg=327.20, stdev=80.66 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 199], 10.00th=[ 220], 20.00th=[ 249], | 30.00th=[ 275], 40.00th=[ 300], 50.00th=[ 330], 60.00th=[ 355], | 70.00th=[ 380], 80.00th=[ 405], 90.00th=[ 435], 95.00th=[ 456], | 99.00th=[ 489], 99.50th=[ 502], 99.90th=[ 542], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 72, max= 208, per=24.83%, avg=131.09, stdev=31.37, samples=78 iops : min= 18, max= 52, avg=32.56, stdev= 7.89, samples=78 lat (msec) : 50=0.20%, 100=0.82%, 250=58.28%, 500=40.43%, 750=0.27% cpu : usr=0.14%, sys=1.54%, ctx=5157, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152665: Thu Apr 2 18:30:10 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(4976KiB/39500msec) slat (usec): min=1715, max=14089, avg=4772.77, stdev=1703.33 clat (msec): min=5, max=385, avg=157.47, stdev=34.65 lat (msec): min=8, max=388, avg=162.25, stdev=34.97 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 110], 10.00th=[ 118], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 174], 80.00th=[ 186], 90.00th=[ 201], 95.00th=[ 213], | 99.00th=[ 249], 99.50th=[ 262], 99.90th=[ 271], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 48, max= 216, per=24.82%, avg=125.82, stdev=35.03, samples=78 iops : min= 12, max= 54, avg=31.24, stdev= 8.75, samples=78 write: IOPS=33, BW=133KiB/s (136kB/s)(5264KiB/39500msec); 0 zone resets slat (usec): min=59, max=4884, avg=274.20, stdev=287.72 clat (msec): min=125, max=572, avg=325.71, stdev=82.86 lat (msec): min=125, max=573, avg=325.98, stdev=82.87 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 192], 10.00th=[ 218], 20.00th=[ 247], | 30.00th=[ 275], 40.00th=[ 300], 50.00th=[ 330], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 401], 90.00th=[ 430], 95.00th=[ 456], | 99.00th=[ 506], 99.50th=[ 527], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 71, max= 216, per=25.16%, avg=132.86, stdev=32.40, samples=78 iops : min= 17, max= 54, avg=33.01, stdev= 8.12, samples=78 lat (msec) : 10=0.04%, 20=0.04%, 50=0.23%, 100=0.90%, 250=57.97% lat (msec) : 500=40.08%, 750=0.74% cpu : usr=0.19%, sys=1.51%, ctx=5173, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152666: Thu Apr 2 18:30:10 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(4984KiB/39536msec) slat (usec): min=1653, max=13599, avg=4798.22, stdev=1739.14 clat (msec): min=23, max=262, avg=157.55, stdev=33.57 lat (msec): min=26, max=272, avg=162.35, stdev=33.88 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 110], 10.00th=[ 117], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 186], 90.00th=[ 201], 95.00th=[ 218], | 99.00th=[ 247], 99.50th=[ 255], 99.90th=[ 262], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 55, max= 232, per=24.87%, avg=126.10, stdev=38.14, samples=78 iops : min= 13, max= 58, avg=31.32, stdev= 9.55, samples=78 write: IOPS=33, BW=133KiB/s (136kB/s)(5256KiB/39536msec); 0 zone resets slat (usec): min=57, max=4052, avg=270.81, stdev=273.80 clat (msec): min=115, max=547, avg=326.29, stdev=81.54 lat (msec): min=115, max=547, avg=326.56, stdev=81.54 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 192], 10.00th=[ 218], 20.00th=[ 251], | 30.00th=[ 275], 40.00th=[ 309], 50.00th=[ 330], 60.00th=[ 355], | 70.00th=[ 380], 80.00th=[ 401], 90.00th=[ 430], 95.00th=[ 456], | 99.00th=[ 498], 99.50th=[ 523], 99.90th=[ 542], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 56, max= 216, per=24.99%, avg=131.95, stdev=33.11, samples=78 iops : min= 14, max= 54, avg=32.78, stdev= 8.30, samples=78 lat (msec) : 50=0.20%, 100=1.13%, 250=57.23%, 500=40.94%, 750=0.51% cpu : usr=0.21%, sys=1.45%, ctx=5164, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152667: Thu Apr 2 18:30:10 2026 read: IOPS=32, BW=129KiB/s (132kB/s)(5060KiB/39269msec) slat (usec): min=96, max=14079, avg=4735.46, stdev=1797.59 clat (msec): min=4, max=397, avg=156.25, stdev=35.68 lat (msec): min=9, max=403, avg=160.99, stdev=36.09 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 109], 10.00th=[ 118], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 184], 90.00th=[ 201], 95.00th=[ 215], | 99.00th=[ 249], 99.50th=[ 255], 99.90th=[ 326], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 47, max= 240, per=25.36%, avg=128.56, stdev=36.78, samples=78 iops : min= 11, max= 60, avg=31.94, stdev= 9.22, samples=78 write: IOPS=32, BW=132KiB/s (135kB/s)(5180KiB/39269msec); 0 zone resets slat (usec): min=48, max=5020, avg=268.41, stdev=311.97 clat (msec): min=49, max=614, avg=327.24, stdev=86.48 lat (msec): min=49, max=615, avg=327.51, stdev=86.51 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 197], 10.00th=[ 218], 20.00th=[ 247], | 30.00th=[ 275], 40.00th=[ 309], 50.00th=[ 330], 60.00th=[ 351], | 70.00th=[ 376], 80.00th=[ 405], 90.00th=[ 439], 95.00th=[ 464], | 99.00th=[ 531], 99.50th=[ 550], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 64, max= 248, per=24.79%, avg=130.90, stdev=32.27, samples=78 iops : min= 16, max= 62, avg=32.51, stdev= 8.09, samples=78 lat (msec) : 10=0.04%, 20=0.12%, 50=0.35%, 100=1.09%, 250=58.24% lat (msec) : 500=39.22%, 750=0.94% cpu : usr=0.16%, sys=1.51%, ctx=5076, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=508KiB/s (520kB/s), 126KiB/s-129KiB/s (129kB/s-132kB/s), io=19.6MiB (20.5MB), run=39269-39536msec WRITE: bw=528KiB/s (541kB/s), 132KiB/s-133KiB/s (135kB/s-136kB/s), io=20.4MiB (21.4MB), run=39269-39536msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=152678: Thu Apr 2 18:30:15 2026 read: IOPS=5, BW=205MiB/s (215MB/s)(40.0MiB/195msec) slat (nsec): min=16351k, max=16351k, avg=16350937.00, stdev= 0.00 clat (nsec): min=167972k, max=167972k, avg=167971879.00, stdev= 0.00 lat (nsec): min=184327k, max=184327k, avg=184326729.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.70%, ctx=41, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=40.0MiB (41.9MB), run=195-195msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout