== sanity test 398c: run fio to test AIO ================= 05:36:03 (1789292163) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.513346 s, 81.7 MB/s osc.lustre-OST0000-osc-ffffa058901da000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146319: Sun Sep 13 05:38:37 2026 write: IOPS=17, BW=70.6KiB/s (72.3kB/s)(10.0MiB/144978msec); 0 zone resets slat (usec): min=21, max=14234, avg=165.87, stdev=652.71 clat (msec): min=305, max=1790, avg=905.53, stdev=249.47 lat (msec): min=305, max=1790, avg=905.70, stdev=249.50 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 835], 50.00th=[ 894], 60.00th=[ 953], | 70.00th=[ 1020], 80.00th=[ 1083], 90.00th=[ 1217], 95.00th=[ 1334], | 99.00th=[ 1653], 99.50th=[ 1720], 99.90th=[ 1754], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 256, per=25.26%, avg=71.23, stdev=30.35, samples=283 iops : min= 1, max= 64, avg=17.35, stdev= 7.63, samples=283 lat (msec) : 500=4.34%, 750=23.32%, 1000=39.41%, 2000=32.93% cpu : usr=0.04%, sys=0.21%, ctx=1590, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146320: Sun Sep 13 05:38:37 2026 write: IOPS=17, BW=70.7KiB/s (72.4kB/s)(10.0MiB/144879msec); 0 zone resets slat (usec): min=22, max=20360, avg=150.88, stdev=605.61 clat (msec): min=201, max=1771, avg=904.18, stdev=251.90 lat (msec): min=201, max=1771, avg=904.34, stdev=251.90 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 514], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 894], 60.00th=[ 953], | 70.00th=[ 1020], 80.00th=[ 1083], 90.00th=[ 1217], 95.00th=[ 1334], | 99.00th=[ 1636], 99.50th=[ 1703], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 256, per=24.82%, avg=69.98, stdev=33.01, samples=288 iops : min= 1, max= 64, avg=17.04, stdev= 8.28, samples=288 lat (msec) : 250=0.04%, 500=4.80%, 750=21.21%, 1000=41.48%, 2000=32.46% cpu : usr=0.04%, sys=0.20%, ctx=1550, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146321: Sun Sep 13 05:38:37 2026 write: IOPS=17, BW=70.7KiB/s (72.4kB/s)(10.0MiB/144776msec); 0 zone resets slat (usec): min=21, max=14291, avg=173.28, stdev=621.58 clat (msec): min=257, max=1768, avg=903.69, stdev=247.40 lat (msec): min=257, max=1768, avg=903.86, stdev=247.38 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 485], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 844], 50.00th=[ 902], 60.00th=[ 944], | 70.00th=[ 1003], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1334], | 99.00th=[ 1603], 99.50th=[ 1670], 99.90th=[ 1737], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 232, per=24.99%, avg=70.48, stdev=33.06, samples=286 iops : min= 1, max= 58, avg=17.16, stdev= 8.30, samples=286 lat (msec) : 500=5.16%, 750=19.77%, 1000=44.92%, 2000=30.16% cpu : usr=0.04%, sys=0.23%, ctx=1826, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146322: Sun Sep 13 05:38:37 2026 write: IOPS=17, BW=70.7KiB/s (72.4kB/s)(10.0MiB/144881msec); 0 zone resets slat (usec): min=21, max=20884, avg=173.85, stdev=761.77 clat (msec): min=149, max=1768, avg=904.93, stdev=253.02 lat (msec): min=149, max=1768, avg=905.11, stdev=253.04 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 518], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 768], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 953], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1636], 99.50th=[ 1720], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 192, per=25.15%, avg=70.93, stdev=32.10, samples=284 iops : min= 1, max= 48, avg=17.29, stdev= 8.06, samples=284 lat (msec) : 250=0.31%, 500=4.45%, 750=22.19%, 1000=41.84%, 2000=31.21% cpu : usr=0.04%, sys=0.22%, ctx=1614, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=283KiB/s (289kB/s), 70.6KiB/s-70.7KiB/s (72.3kB/s-72.4kB/s), io=40.0MiB (41.9MB), run=144776-144978msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146352: Sun Sep 13 05:40:07 2026 read: IOPS=15, BW=61.0KiB/s (63.5kB/s)(5048KiB/81423msec) slat (usec): min=2300, max=51706, avg=11254.85, stdev=6035.66 clat (msec): min=26, max=741, avg=317.09, stdev=81.18 lat (msec): min=31, max=750, avg=328.35, stdev=82.29 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 199], 10.00th=[ 224], 20.00th=[ 253], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 330], | 70.00th=[ 351], 80.00th=[ 380], 90.00th=[ 422], 95.00th=[ 451], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 651], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 8, max= 151, per=25.09%, avg=61.73, stdev=26.44, samples=161 iops : min= 2, max= 37, avg=15.00, stdev= 6.64, samples=161 write: IOPS=15, BW=63.8KiB/s (65.3kB/s)(5192KiB/81423msec); 0 zone resets slat (usec): min=26, max=6954, avg=163.33, stdev=348.64 clat (msec): min=218, max=1260, avg=682.64, stdev=182.17 lat (msec): min=218, max=1261, avg=682.80, stdev=182.18 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 393], 10.00th=[ 443], 20.00th=[ 523], | 30.00th=[ 567], 40.00th=[ 625], 50.00th=[ 693], 60.00th=[ 743], | 70.00th=[ 793], 80.00th=[ 835], 90.00th=[ 919], 95.00th=[ 986], | 99.00th=[ 1083], 99.50th=[ 1116], 99.90th=[ 1217], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 7, max= 135, per=24.64%, avg=63.08, stdev=22.68, samples=162 iops : min= 1, max= 33, avg=15.34, stdev= 5.67, samples=162 lat (msec) : 50=0.08%, 100=0.08%, 250=9.53%, 500=46.84%, 750=24.18% lat (msec) : 1000=17.23%, 2000=2.07% cpu : usr=0.09%, sys=0.66%, ctx=4263, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146353: Sun Sep 13 05:40:07 2026 read: IOPS=15, BW=61.2KiB/s (62.7kB/s)(4976KiB/81274msec) slat (usec): min=60, max=42257, avg=11148.80, stdev=6060.49 clat (msec): min=6, max=879, avg=319.75, stdev=92.80 lat (msec): min=15, max=886, avg=330.91, stdev=93.78 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 188], 10.00th=[ 215], 20.00th=[ 249], | 30.00th=[ 275], 40.00th=[ 292], 50.00th=[ 313], 60.00th=[ 334], | 70.00th=[ 359], 80.00th=[ 384], 90.00th=[ 430], 95.00th=[ 472], | 99.00th=[ 592], 99.50th=[ 625], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 176, per=24.88%, avg=61.21, stdev=26.21, samples=160 iops : min= 1, max= 44, avg=14.87, stdev= 6.58, samples=160 write: IOPS=16, BW=64.8KiB/s (66.3kB/s)(5264KiB/81274msec); 0 zone resets slat (usec): min=30, max=6575, avg=177.35, stdev=390.32 clat (msec): min=71, max=1313, avg=674.48, stdev=186.04 lat (msec): min=71, max=1314, avg=674.66, stdev=186.04 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 376], 10.00th=[ 439], 20.00th=[ 514], | 30.00th=[ 567], 40.00th=[ 625], 50.00th=[ 676], 60.00th=[ 735], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 902], 95.00th=[ 986], | 99.00th=[ 1083], 99.50th=[ 1183], 99.90th=[ 1267], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 23, max= 142, per=25.12%, avg=64.32, stdev=22.14, samples=161 iops : min= 5, max= 35, avg=15.65, stdev= 5.55, samples=161 lat (msec) : 10=0.04%, 20=0.04%, 50=0.23%, 100=0.16%, 250=9.92% lat (msec) : 500=45.78%, 750=24.88%, 1000=16.84%, 2000=2.11% cpu : usr=0.13%, sys=0.73%, ctx=4373, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146354: Sun Sep 13 05:40:07 2026 read: IOPS=15, BW=61.2KiB/s (62.7kB/s)(4984KiB/81426msec) slat (usec): min=2355, max=58811, avg=11255.18, stdev=6298.20 clat (msec): min=18, max=693, avg=317.02, stdev=82.58 lat (msec): min=26, max=697, avg=328.28, stdev=83.56 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 194], 10.00th=[ 215], 20.00th=[ 251], | 30.00th=[ 275], 40.00th=[ 296], 50.00th=[ 313], 60.00th=[ 334], | 70.00th=[ 355], 80.00th=[ 380], 90.00th=[ 426], 95.00th=[ 451], | 99.00th=[ 535], 99.50th=[ 592], 99.90th=[ 617], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 7, max= 143, per=24.89%, avg=61.22, stdev=26.28, samples=161 iops : min= 1, max= 35, avg=14.87, stdev= 6.59, samples=161 write: IOPS=16, BW=64.5KiB/s (66.1kB/s)(5256KiB/81426msec); 0 zone resets slat (usec): min=33, max=19401, avg=193.09, stdev=649.33 clat (msec): min=194, max=1295, avg=678.75, stdev=181.32 lat (msec): min=194, max=1295, avg=678.95, stdev=181.32 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 388], 10.00th=[ 447], 20.00th=[ 518], | 30.00th=[ 567], 40.00th=[ 634], 50.00th=[ 684], 60.00th=[ 735], | 70.00th=[ 785], 80.00th=[ 835], 90.00th=[ 902], 95.00th=[ 969], | 99.00th=[ 1070], 99.50th=[ 1183], 99.90th=[ 1284], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 15, max= 128, per=24.82%, avg=63.54, stdev=21.49, samples=162 iops : min= 3, max= 32, avg=15.46, stdev= 5.39, samples=162 lat (msec) : 20=0.04%, 50=0.12%, 100=0.27%, 250=9.45%, 500=46.17% lat (msec) : 750=24.61%, 1000=17.66%, 2000=1.68% cpu : usr=0.12%, sys=0.67%, ctx=4556, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146355: Sun Sep 13 05:40:07 2026 read: IOPS=15, BW=62.3KiB/s (63.8kB/s)(5060KiB/81203msec) slat (usec): min=2241, max=52198, avg=11407.06, stdev=6175.36 clat (msec): min=50, max=935, avg=318.42, stdev=88.93 lat (msec): min=57, max=945, avg=329.83, stdev=89.88 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 197], 10.00th=[ 220], 20.00th=[ 249], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 330], | 70.00th=[ 351], 80.00th=[ 380], 90.00th=[ 422], 95.00th=[ 464], | 99.00th=[ 592], 99.50th=[ 676], 99.90th=[ 919], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 8, max= 128, per=25.46%, avg=62.62, stdev=26.87, samples=159 iops : min= 2, max= 32, avg=15.23, stdev= 6.74, samples=159 write: IOPS=15, BW=63.8KiB/s (65.3kB/s)(5180KiB/81203msec); 0 zone resets slat (usec): min=24, max=11335, avg=201.93, stdev=592.09 clat (msec): min=172, max=1298, avg=679.11, stdev=186.89 lat (msec): min=172, max=1299, avg=679.31, stdev=186.91 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 376], 10.00th=[ 443], 20.00th=[ 510], | 30.00th=[ 567], 40.00th=[ 625], 50.00th=[ 684], 60.00th=[ 735], | 70.00th=[ 785], 80.00th=[ 844], 90.00th=[ 919], 95.00th=[ 986], | 99.00th=[ 1099], 99.50th=[ 1167], 99.90th=[ 1250], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 23, max= 151, per=24.75%, avg=63.35, stdev=22.42, samples=161 iops : min= 5, max= 37, avg=15.41, stdev= 5.61, samples=161 lat (msec) : 100=0.16%, 250=10.00%, 500=47.38%, 750=24.10%, 1000=16.25% lat (msec) : 2000=2.11% cpu : usr=0.13%, sys=0.74%, ctx=4698, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=246KiB/s (252kB/s), 61.2KiB/s-62.3KiB/s (62.7kB/s-63.8kB/s), io=19.6MiB (20.5MB), run=81203-81426msec WRITE: bw=257KiB/s (263kB/s), 63.8KiB/s-64.8KiB/s (65.3kB/s-66.3kB/s), io=20.4MiB (21.4MB), run=81203-81426msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146371: Sun Sep 13 05:40:15 2026 read: IOPS=2, BW=91.7MiB/s (96.2MB/s)(40.0MiB/436msec) slat (nsec): min=46181k, max=46181k, avg=46181479.00, stdev= 0.00 clat (nsec): min=373022k, max=373022k, avg=373021893.00, stdev= 0.00 lat (nsec): min=419224k, max=419224k, avg=419223707.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.76%, ctx=37, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=91.7MiB/s (96.2MB/s), 91.7MiB/s-91.7MiB/s (96.2MB/s-96.2MB/s), io=40.0MiB (41.9MB), run=436-436msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout