== sanity test 398c: run fio to test AIO ================= 06:11:50 (1769080310) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.423155 s, 99.1 MB/s osc.lustre-OST0000-osc-ffff97f178fa9000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147140: Thu Jan 22 06:13:32 2026 write: IOPS=27, BW=108KiB/s (111kB/s)(10.0MiB/94551msec); 0 zone resets slat (usec): min=22, max=10862, avg=190.86, stdev=518.79 clat (msec): min=168, max=1065, avg=590.28, stdev=138.39 lat (msec): min=168, max=1065, avg=590.47, stdev=138.42 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 359], 10.00th=[ 426], 20.00th=[ 485], | 30.00th=[ 518], 40.00th=[ 558], 50.00th=[ 592], 60.00th=[ 617], | 70.00th=[ 659], 80.00th=[ 709], 90.00th=[ 776], 95.00th=[ 827], | 99.00th=[ 894], 99.50th=[ 919], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 24, max= 256, per=24.87%, avg=107.44, stdev=32.32, samples=188 iops : min= 6, max= 64, avg=26.45, stdev= 8.11, samples=188 lat (msec) : 250=1.91%, 500=22.97%, 750=62.62%, 1000=12.34%, 2000=0.16% cpu : usr=0.05%, sys=0.38%, ctx=2527, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147141: Thu Jan 22 06:13:32 2026 write: IOPS=27, BW=108KiB/s (111kB/s)(10.0MiB/94600msec); 0 zone resets slat (usec): min=22, max=9743, avg=203.56, stdev=568.74 clat (msec): min=133, max=1114, avg=590.52, stdev=136.38 lat (msec): min=133, max=1114, avg=590.72, stdev=136.40 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 368], 10.00th=[ 430], 20.00th=[ 485], | 30.00th=[ 518], 40.00th=[ 558], 50.00th=[ 592], 60.00th=[ 617], | 70.00th=[ 659], 80.00th=[ 701], 90.00th=[ 776], 95.00th=[ 818], | 99.00th=[ 911], 99.50th=[ 953], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1116] bw ( KiB/s): min= 31, max= 216, per=24.82%, avg=107.23, stdev=31.25, samples=188 iops : min= 7, max= 54, avg=26.40, stdev= 7.84, samples=188 lat (msec) : 250=1.52%, 500=23.09%, 750=62.93%, 1000=12.23%, 2000=0.23% cpu : usr=0.05%, sys=0.38%, ctx=2567, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147142: Thu Jan 22 06:13:32 2026 write: IOPS=27, BW=108KiB/s (111kB/s)(10.0MiB/94550msec); 0 zone resets slat (usec): min=21, max=10863, avg=197.29, stdev=553.21 clat (msec): min=95, max=1068, avg=590.40, stdev=140.72 lat (msec): min=95, max=1068, avg=590.60, stdev=140.72 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 363], 10.00th=[ 430], 20.00th=[ 485], | 30.00th=[ 523], 40.00th=[ 558], 50.00th=[ 592], 60.00th=[ 625], | 70.00th=[ 659], 80.00th=[ 701], 90.00th=[ 776], 95.00th=[ 818], | 99.00th=[ 902], 99.50th=[ 978], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1070] bw ( KiB/s): min= 8, max= 322, per=24.98%, avg=107.91, stdev=33.94, samples=187 iops : min= 2, max= 80, avg=26.57, stdev= 8.46, samples=187 lat (msec) : 100=0.62%, 250=1.37%, 500=22.23%, 750=63.36%, 1000=12.15% lat (msec) : 2000=0.27% cpu : usr=0.07%, sys=0.37%, ctx=2447, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147143: Thu Jan 22 06:13:32 2026 write: IOPS=27, BW=108KiB/s (111kB/s)(10.0MiB/94552msec); 0 zone resets slat (usec): min=23, max=8610, avg=193.62, stdev=522.00 clat (msec): min=54, max=1040, avg=590.20, stdev=136.57 lat (msec): min=55, max=1040, avg=590.39, stdev=136.57 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 376], 10.00th=[ 439], 20.00th=[ 485], | 30.00th=[ 518], 40.00th=[ 558], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 659], 80.00th=[ 701], 90.00th=[ 768], 95.00th=[ 818], | 99.00th=[ 902], 99.50th=[ 961], 99.90th=[ 1028], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 15, max= 235, per=24.86%, avg=107.41, stdev=30.06, samples=188 iops : min= 3, max= 58, avg=26.44, stdev= 7.51, samples=188 lat (msec) : 100=0.16%, 250=1.21%, 500=23.20%, 750=63.55%, 1000=11.72% lat (msec) : 2000=0.16% cpu : usr=0.07%, sys=0.37%, ctx=2721, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=433KiB/s (443kB/s), 108KiB/s-108KiB/s (111kB/s-111kB/s), io=40.0MiB (41.9MB), run=94550-94600msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147168: Thu Jan 22 06:14:34 2026 read: IOPS=22, BW=88.2KiB/s (90.4kB/s)(5048KiB/57201msec) slat (usec): min=23, max=27578, avg=7742.87, stdev=3916.68 clat (msec): min=8, max=456, avg=228.32, stdev=58.69 lat (msec): min=14, max=462, avg=236.06, stdev=59.58 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 140], 10.00th=[ 161], 20.00th=[ 182], | 30.00th=[ 199], 40.00th=[ 213], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 255], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 372], 99.50th=[ 376], 99.90th=[ 414], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 23, max= 191, per=25.13%, avg=87.69, stdev=34.55, samples=113 iops : min= 5, max= 47, avg=21.56, stdev= 8.68, samples=113 write: IOPS=22, BW=90.8KiB/s (92.9kB/s)(5192KiB/57201msec); 0 zone resets slat (usec): min=24, max=5740, avg=205.20, stdev=343.51 clat (msec): min=43, max=850, avg=474.98, stdev=122.48 lat (msec): min=43, max=850, avg=475.18, stdev=122.48 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 264], 10.00th=[ 317], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 510], | 70.00th=[ 542], 80.00th=[ 575], 90.00th=[ 625], 95.00th=[ 676], | 99.00th=[ 776], 99.50th=[ 785], 99.90th=[ 835], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 23, max= 160, per=24.79%, avg=90.22, stdev=25.52, samples=113 iops : min= 5, max= 40, avg=22.19, stdev= 6.42, samples=113 lat (msec) : 10=0.04%, 20=0.08%, 50=0.31%, 100=0.70%, 250=33.71% lat (msec) : 500=43.32%, 750=21.09%, 1000=0.74% cpu : usr=0.16%, sys=1.06%, ctx=5098, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147169: Thu Jan 22 06:14:34 2026 read: IOPS=21, BW=86.0KiB/s (89.1kB/s)(4976KiB/57200msec) slat (msec): min=2, max=225, avg= 7.78, stdev= 7.20 clat (msec): min=29, max=455, avg=227.49, stdev=58.17 lat (msec): min=37, max=464, avg=235.27, stdev=58.57 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 146], 10.00th=[ 161], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 253], 80.00th=[ 275], 90.00th=[ 300], 95.00th=[ 330], | 99.00th=[ 393], 99.50th=[ 405], 99.90th=[ 447], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 31, max= 168, per=24.80%, avg=86.55, stdev=28.85, samples=113 iops : min= 7, max= 42, avg=21.27, stdev= 7.29, samples=113 write: IOPS=23, BW=92.0KiB/s (94.2kB/s)(5264KiB/57200msec); 0 zone resets slat (usec): min=23, max=13947, avg=235.99, stdev=602.17 clat (msec): min=73, max=855, avg=472.19, stdev=124.30 lat (msec): min=73, max=855, avg=472.43, stdev=124.31 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 264], 10.00th=[ 313], 20.00th=[ 368], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 510], | 70.00th=[ 542], 80.00th=[ 575], 90.00th=[ 625], 95.00th=[ 659], | 99.00th=[ 793], 99.50th=[ 827], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 24, max= 176, per=25.21%, avg=91.77, stdev=26.68, samples=113 iops : min= 6, max= 44, avg=22.58, stdev= 6.71, samples=113 lat (msec) : 50=0.20%, 100=0.51%, 250=34.34%, 500=42.85%, 750=21.33% lat (msec) : 1000=0.78% cpu : usr=0.14%, sys=1.08%, ctx=4914, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147170: Thu Jan 22 06:14:34 2026 read: IOPS=21, BW=86.9KiB/s (88.0kB/s)(4984KiB/57366msec) slat (usec): min=1866, max=28514, avg=7754.20, stdev=3735.17 clat (msec): min=9, max=411, avg=227.03, stdev=56.37 lat (msec): min=17, max=416, avg=234.79, stdev=57.00 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 144], 10.00th=[ 161], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 241], | 70.00th=[ 255], 80.00th=[ 275], 90.00th=[ 300], 95.00th=[ 326], | 99.00th=[ 363], 99.50th=[ 372], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 24, max= 184, per=24.77%, avg=86.46, stdev=33.89, samples=114 iops : min= 6, max= 46, avg=21.25, stdev= 8.51, samples=114 write: IOPS=22, BW=91.6KiB/s (93.8kB/s)(5256KiB/57366msec); 0 zone resets slat (usec): min=25, max=9424, avg=207.47, stdev=461.64 clat (msec): min=109, max=868, avg=474.94, stdev=125.51 lat (msec): min=109, max=868, avg=475.15, stdev=125.53 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 262], 10.00th=[ 313], 20.00th=[ 368], | 30.00th=[ 409], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 514], | 70.00th=[ 542], 80.00th=[ 575], 90.00th=[ 634], 95.00th=[ 676], | 99.00th=[ 768], 99.50th=[ 818], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 32, max= 176, per=24.89%, avg=90.59, stdev=26.03, samples=114 iops : min= 8, max= 44, avg=22.28, stdev= 6.52, samples=114 lat (msec) : 10=0.04%, 20=0.08%, 50=0.23%, 100=0.20%, 250=33.83% lat (msec) : 500=43.67%, 750=21.17%, 1000=0.78% cpu : usr=0.16%, sys=1.04%, ctx=4586, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147171: Thu Jan 22 06:14:34 2026 read: IOPS=22, BW=88.4KiB/s (90.5kB/s)(5060KiB/57236msec) slat (usec): min=1754, max=31743, avg=7754.04, stdev=3733.88 clat (msec): min=8, max=478, avg=230.59, stdev=57.89 lat (msec): min=13, max=485, avg=238.36, stdev=58.56 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 146], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 199], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 241], | 70.00th=[ 255], 80.00th=[ 279], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 384], 99.50th=[ 393], 99.90th=[ 472], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 31, max= 215, per=25.28%, avg=88.22, stdev=31.31, samples=113 iops : min= 7, max= 53, avg=21.69, stdev= 7.84, samples=113 write: IOPS=22, BW=90.5KiB/s (92.7kB/s)(5180KiB/57236msec); 0 zone resets slat (usec): min=26, max=9499, avg=211.06, stdev=447.83 clat (msec): min=163, max=843, avg=473.20, stdev=121.56 lat (msec): min=163, max=844, avg=473.41, stdev=121.57 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 271], 10.00th=[ 317], 20.00th=[ 368], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 542], 80.00th=[ 575], 90.00th=[ 625], 95.00th=[ 676], | 99.00th=[ 793], 99.50th=[ 818], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 32, max= 176, per=24.69%, avg=89.86, stdev=25.26, samples=113 iops : min= 8, max= 44, avg=22.10, stdev= 6.34, samples=113 lat (msec) : 10=0.04%, 20=0.08%, 50=0.20%, 100=0.08%, 250=34.22% lat (msec) : 500=44.96%, 750=19.53%, 1000=0.90% cpu : usr=0.14%, sys=0.99%, ctx=5277, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=350KiB/s (358kB/s), 86.9KiB/s-88.4KiB/s (88.0kB/s-90.5kB/s), io=19.6MiB (20.5MB), run=57200-57366msec WRITE: bw=364KiB/s (373kB/s), 90.5KiB/s-92.0KiB/s (92.7kB/s-94.2kB/s), io=20.4MiB (21.4MB), run=57200-57366msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147184: Thu Jan 22 06:14:40 2026 read: IOPS=3, BW=129MiB/s (136MB/s)(40.0MiB/309msec) slat (nsec): min=21219k, max=21219k, avg=21219201.00, stdev= 0.00 clat (nsec): min=272626k, max=272626k, avg=272626213.00, stdev= 0.00 lat (nsec): min=293850k, max=293850k, avg=293849722.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.57%, ctx=15, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=129MiB/s (136MB/s), 129MiB/s-129MiB/s (136MB/s-136MB/s), io=40.0MiB (41.9MB), run=309-309msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout