== sanity test 398c: run fio to test AIO ================= 22:53:50 (1784343230) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.154477 s, 272 MB/s osc.lustre-OST0000-osc-ffff8dce58926800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=155669: Fri Jul 17 22:54:21 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27718msec); 0 zone resets slat (usec): min=41, max=1935, avg=210.70, stdev=95.20 clat (msec): min=43, max=286, avg=172.78, stdev=34.42 lat (msec): min=43, max=286, avg=172.99, stdev=34.42 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 120], 10.00th=[ 136], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 199], 90.00th=[ 211], 95.00th=[ 224], | 99.00th=[ 264], 99.50th=[ 268], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 256, max= 643, per=24.87%, avg=367.27, stdev=55.73, samples=55 iops : min= 64, max= 160, avg=91.78, stdev=13.87, samples=55 lat (msec) : 50=0.78%, 100=2.58%, 250=95.04%, 500=1.60% cpu : usr=0.14%, sys=1.63%, ctx=2924, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155670: Fri Jul 17 22:54:21 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27728msec); 0 zone resets slat (usec): min=44, max=1972, avg=206.37, stdev=97.63 clat (msec): min=43, max=295, avg=172.86, stdev=33.96 lat (msec): min=43, max=295, avg=173.07, stdev=33.97 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 122], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 159], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 199], 90.00th=[ 211], 95.00th=[ 222], | 99.00th=[ 264], 99.50th=[ 275], 99.90th=[ 288], 99.95th=[ 292], | 99.99th=[ 296] bw ( KiB/s): min= 232, max= 643, per=24.87%, avg=367.27, stdev=60.16, samples=55 iops : min= 58, max= 160, avg=91.78, stdev=14.99, samples=55 lat (msec) : 50=0.78%, 100=2.58%, 250=95.27%, 500=1.37% cpu : usr=0.15%, sys=1.59%, ctx=2914, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155671: Fri Jul 17 22:54:21 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27728msec); 0 zone resets slat (usec): min=37, max=1293, avg=208.60, stdev=85.83 clat (msec): min=43, max=295, avg=172.88, stdev=34.13 lat (msec): min=43, max=295, avg=173.09, stdev=34.14 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 120], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 190], 80.00th=[ 199], 90.00th=[ 211], 95.00th=[ 224], | 99.00th=[ 262], 99.50th=[ 266], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 296] bw ( KiB/s): min= 248, max= 643, per=24.87%, avg=367.27, stdev=61.02, samples=55 iops : min= 62, max= 160, avg=91.78, stdev=15.21, samples=55 lat (msec) : 50=0.16%, 100=3.16%, 250=95.31%, 500=1.37% cpu : usr=0.17%, sys=1.61%, ctx=2823, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155672: Fri Jul 17 22:54:21 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27690msec); 0 zone resets slat (usec): min=45, max=1267, avg=211.05, stdev=94.09 clat (msec): min=31, max=290, avg=172.75, stdev=34.72 lat (msec): min=32, max=290, avg=172.96, stdev=34.72 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 122], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 211], 95.00th=[ 226], | 99.00th=[ 259], 99.50th=[ 271], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 292] bw ( KiB/s): min= 240, max= 681, per=24.85%, avg=367.09, stdev=60.77, samples=55 iops : min= 60, max= 170, avg=91.75, stdev=15.19, samples=55 lat (msec) : 50=0.55%, 100=3.05%, 250=95.00%, 500=1.41% cpu : usr=0.20%, sys=1.56%, ctx=2936, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1477KiB/s (1513kB/s), 369KiB/s-370KiB/s (378kB/s-379kB/s), io=40.0MiB (41.9MB), run=27690-27728msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=155686: Fri Jul 17 22:54:43 2026 read: IOPS=67, BW=268KiB/s (275kB/s)(5048KiB/18811msec) slat (usec): min=961, max=5177, avg=2339.61, stdev=768.74 clat (msec): min=3, max=156, avg=73.92, stdev=15.18 lat (msec): min=5, max=158, avg=76.26, stdev=15.26 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 53], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 86], 90.00th=[ 93], 95.00th=[ 97], | 99.00th=[ 112], 99.50th=[ 123], 99.90th=[ 142], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 144, max= 399, per=25.17%, avg=268.11, stdev=54.84, samples=37 iops : min= 36, max= 99, avg=67.00, stdev=13.67, samples=37 write: IOPS=69, BW=276KiB/s (283kB/s)(5192KiB/18811msec); 0 zone resets slat (usec): min=54, max=613, avg=176.05, stdev=83.55 clat (msec): min=58, max=261, avg=156.94, stdev=36.02 lat (msec): min=58, max=261, avg=157.11, stdev=36.02 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 96], 10.00th=[ 108], 20.00th=[ 125], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 241], 99.50th=[ 245], 99.90th=[ 253], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 208, max= 328, per=24.89%, avg=275.76, stdev=31.38, samples=37 iops : min= 52, max= 82, avg=68.92, stdev= 7.86, samples=37 lat (msec) : 4=0.08%, 10=0.12%, 20=0.23%, 50=0.94%, 100=49.61% lat (msec) : 250=48.87%, 500=0.16% cpu : usr=0.22%, sys=2.28%, ctx=4890, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155687: Fri Jul 17 22:54:43 2026 read: IOPS=66, BW=264KiB/s (270kB/s)(4976KiB/18841msec) slat (usec): min=944, max=8276, avg=2364.00, stdev=779.24 clat (msec): min=7, max=123, avg=73.94, stdev=14.06 lat (msec): min=9, max=126, avg=76.31, stdev=14.14 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 86], 90.00th=[ 91], 95.00th=[ 96], | 99.00th=[ 110], 99.50th=[ 113], 99.90th=[ 120], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 144, max= 383, per=24.61%, avg=262.05, stdev=56.39, samples=37 iops : min= 36, max= 95, avg=65.49, stdev=14.05, samples=37 write: IOPS=69, BW=279KiB/s (286kB/s)(5264KiB/18841msec); 0 zone resets slat (usec): min=48, max=604, avg=178.60, stdev=82.34 clat (msec): min=26, max=258, avg=156.21, stdev=37.16 lat (msec): min=26, max=258, avg=156.39, stdev=37.17 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 94], 10.00th=[ 107], 20.00th=[ 125], | 30.00th=[ 136], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 218], | 99.00th=[ 234], 99.50th=[ 241], 99.90th=[ 257], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 168, max= 400, per=25.17%, avg=278.92, stdev=40.14, samples=37 iops : min= 42, max= 100, avg=69.70, stdev=10.05, samples=37 lat (msec) : 10=0.04%, 20=0.27%, 50=1.13%, 100=48.91%, 250=49.53% lat (msec) : 500=0.12% cpu : usr=0.23%, sys=2.28%, ctx=4877, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155688: Fri Jul 17 22:54:43 2026 read: IOPS=66, BW=265KiB/s (271kB/s)(4984KiB/18840msec) slat (usec): min=945, max=5995, avg=2364.75, stdev=759.33 clat (msec): min=3, max=119, avg=74.45, stdev=14.37 lat (msec): min=6, max=122, avg=76.81, stdev=14.47 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 93], 95.00th=[ 97], | 99.00th=[ 108], 99.50th=[ 114], 99.90th=[ 117], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 152, max= 376, per=24.79%, avg=264.00, stdev=51.70, samples=37 iops : min= 38, max= 94, avg=65.97, stdev=12.90, samples=37 write: IOPS=69, BW=279KiB/s (286kB/s)(5256KiB/18840msec); 0 zone resets slat (usec): min=45, max=705, avg=178.17, stdev=80.12 clat (msec): min=60, max=257, avg=155.91, stdev=37.36 lat (msec): min=60, max=257, avg=156.09, stdev=37.36 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 95], 10.00th=[ 104], 20.00th=[ 122], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 205], 95.00th=[ 218], | 99.00th=[ 236], 99.50th=[ 247], 99.90th=[ 251], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 200, max= 376, per=24.96%, avg=276.54, stdev=37.71, samples=37 iops : min= 50, max= 94, avg=69.11, stdev= 9.42, samples=37 lat (msec) : 4=0.04%, 10=0.08%, 20=0.27%, 50=0.86%, 100=49.77% lat (msec) : 250=48.87%, 500=0.12% cpu : usr=0.19%, sys=2.34%, ctx=4817, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155689: Fri Jul 17 22:54:43 2026 read: IOPS=67, BW=270KiB/s (276kB/s)(5060KiB/18743msec) slat (usec): min=62, max=5389, avg=2313.36, stdev=740.10 clat (msec): min=4, max=203, avg=74.48, stdev=15.93 lat (msec): min=6, max=205, avg=76.80, stdev=15.99 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 53], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 86], 90.00th=[ 92], 95.00th=[ 97], | 99.00th=[ 109], 99.50th=[ 117], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 192, max= 384, per=25.35%, avg=270.03, stdev=49.22, samples=37 iops : min= 48, max= 96, avg=67.49, stdev=12.28, samples=37 write: IOPS=69, BW=276KiB/s (283kB/s)(5180KiB/18743msec); 0 zone resets slat (usec): min=55, max=705, avg=178.31, stdev=81.52 clat (msec): min=24, max=263, avg=156.16, stdev=37.76 lat (msec): min=24, max=263, avg=156.35, stdev=37.76 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 123], | 30.00th=[ 136], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 218], | 99.00th=[ 243], 99.50th=[ 249], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 200, max= 360, per=24.93%, avg=276.22, stdev=37.84, samples=37 iops : min= 50, max= 90, avg=69.03, stdev= 9.44, samples=37 lat (msec) : 10=0.20%, 20=0.31%, 50=1.25%, 100=49.77%, 250=48.28% lat (msec) : 500=0.20% cpu : usr=0.19%, sys=2.36%, ctx=4932, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1065KiB/s (1091kB/s), 264KiB/s-270KiB/s (270kB/s-276kB/s), io=19.6MiB (20.5MB), run=18743-18841msec WRITE: bw=1109KiB/s (1135kB/s), 276KiB/s-279KiB/s (283kB/s-286kB/s), io=20.4MiB (21.4MB), run=18743-18841msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=155695: Fri Jul 17 22:54:47 2026 read: IOPS=8, BW=342MiB/s (358MB/s)(40.0MiB/117msec) slat (nsec): min=11671k, max=11671k, avg=11671304.00, stdev= 0.00 clat (nsec): min=99459k, max=99459k, avg=99459478.00, stdev= 0.00 lat (nsec): min=111134k, max=111134k, avg=111134493.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[99091], 5.00th=[99091], 10.00th=[99091], 20.00th=[99091], | 30.00th=[99091], 40.00th=[99091], 50.00th=[99091], 60.00th=[99091], | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=8.62%, ctx=40, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=342MiB/s (358MB/s), 342MiB/s-342MiB/s (358MB/s-358MB/s), io=40.0MiB (41.9MB), run=117-117msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout