== sanity test 398c: run fio to test AIO ================= 14:11:40 (1772305900) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.148608 s, 282 MB/s osc.lustre-OST0000-osc-ffff95c1982b1000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=122544: Sat Feb 28 14:12:12 2026 write: IOPS=90, BW=362KiB/s (371kB/s)(10.0MiB/28293msec); 0 zone resets slat (usec): min=28, max=490, avg=113.91, stdev=55.85 clat (msec): min=65, max=344, avg=176.59, stdev=37.12 lat (msec): min=65, max=344, avg=176.70, stdev=37.12 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 110], 10.00th=[ 131], 20.00th=[ 150], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 234], | 99.00th=[ 264], 99.50th=[ 275], 99.90th=[ 313], 99.95th=[ 334], | 99.99th=[ 347] bw ( KiB/s): min= 272, max= 635, per=24.93%, avg=360.77, stdev=61.52, samples=56 iops : min= 68, max= 158, avg=90.18, stdev=15.32, samples=56 lat (msec) : 100=4.06%, 250=93.91%, 500=2.03% cpu : usr=0.16%, sys=0.97%, ctx=1853, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122545: Sat Feb 28 14:12:12 2026 write: IOPS=90, BW=362KiB/s (371kB/s)(10.0MiB/28293msec); 0 zone resets slat (usec): min=28, max=517, avg=110.36, stdev=52.82 clat (msec): min=69, max=359, avg=176.60, stdev=37.11 lat (msec): min=70, max=359, avg=176.71, stdev=37.11 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 110], 10.00th=[ 131], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 207], 90.00th=[ 222], 95.00th=[ 234], | 99.00th=[ 264], 99.50th=[ 279], 99.90th=[ 305], 99.95th=[ 347], | 99.99th=[ 359] bw ( KiB/s): min= 272, max= 629, per=24.93%, avg=360.80, stdev=62.79, samples=56 iops : min= 68, max= 157, avg=90.20, stdev=15.68, samples=56 lat (msec) : 100=4.10%, 250=93.63%, 500=2.27% cpu : usr=0.13%, sys=0.98%, ctx=1771, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122546: Sat Feb 28 14:12:12 2026 write: IOPS=90, BW=363KiB/s (372kB/s)(10.0MiB/28175msec); 0 zone resets slat (usec): min=26, max=470, avg=111.66, stdev=54.49 clat (msec): min=14, max=360, avg=175.88, stdev=39.16 lat (msec): min=14, max=360, avg=176.00, stdev=39.16 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 109], 10.00th=[ 131], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 222], 95.00th=[ 236], | 99.00th=[ 268], 99.50th=[ 292], 99.90th=[ 313], 99.95th=[ 330], | 99.99th=[ 359] bw ( KiB/s): min= 224, max= 731, per=24.90%, avg=360.34, stdev=73.40, samples=56 iops : min= 56, max= 182, avg=90.07, stdev=18.28, samples=56 lat (msec) : 20=0.08%, 50=0.31%, 100=4.06%, 250=93.32%, 500=2.23% cpu : usr=0.17%, sys=0.96%, ctx=1780, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122547: Sat Feb 28 14:12:12 2026 write: IOPS=90, BW=362KiB/s (371kB/s)(10.0MiB/28293msec); 0 zone resets slat (usec): min=26, max=614, avg=111.24, stdev=54.67 clat (msec): min=66, max=311, avg=176.60, stdev=36.94 lat (msec): min=66, max=311, avg=176.72, stdev=36.94 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 110], 10.00th=[ 131], 20.00th=[ 150], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 264], 99.50th=[ 271], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 313] bw ( KiB/s): min= 256, max= 622, per=24.93%, avg=360.68, stdev=62.10, samples=56 iops : min= 64, max= 155, avg=90.16, stdev=15.49, samples=56 lat (msec) : 100=3.91%, 250=93.75%, 500=2.34% cpu : usr=0.14%, sys=0.96%, ctx=1828, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1448KiB/s (1482kB/s), 362KiB/s-363KiB/s (371kB/s-372kB/s), io=40.0MiB (41.9MB), run=28175-28293msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=122560: Sat Feb 28 14:12:31 2026 read: IOPS=77, BW=312KiB/s (319kB/s)(5048KiB/16199msec) slat (usec): min=732, max=4158, avg=1280.31, stdev=340.52 clat (usec): min=1554, max=136860, avg=67249.33, stdev=12878.15 lat (msec): min=2, max=137, avg=68.53, stdev=12.90 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 69], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 83], 95.00th=[ 88], | 99.00th=[ 104], 99.50th=[ 110], 99.90th=[ 125], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 216, max= 408, per=25.14%, avg=311.25, stdev=50.69, samples=32 iops : min= 54, max= 102, avg=77.81, stdev=12.67, samples=32 write: IOPS=80, BW=321KiB/s (328kB/s)(5192KiB/16199msec); 0 zone resets slat (usec): min=27, max=1276, avg=93.86, stdev=57.51 clat (msec): min=48, max=230, avg=132.67, stdev=29.90 lat (msec): min=48, max=230, avg=132.77, stdev=29.90 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 86], 10.00th=[ 95], 20.00th=[ 107], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 174], 95.00th=[ 186], | 99.00th=[ 207], 99.50th=[ 209], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 248, max= 416, per=24.85%, avg=320.38, stdev=39.93, samples=32 iops : min= 62, max= 104, avg=80.09, stdev= 9.98, samples=32 lat (msec) : 2=0.04%, 4=0.04%, 20=0.20%, 50=2.70%, 100=52.77% lat (msec) : 250=44.26% cpu : usr=0.27%, sys=2.00%, ctx=2564, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122561: Sat Feb 28 14:12:31 2026 read: IOPS=77, BW=309KiB/s (316kB/s)(4976KiB/16122msec) slat (usec): min=47, max=4635, avg=1285.46, stdev=362.55 clat (usec): min=1500, max=138371, avg=66392.41, stdev=13318.69 lat (msec): min=2, max=139, avg=67.68, stdev=13.35 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 82], 95.00th=[ 89], | 99.00th=[ 104], 99.50th=[ 108], 99.90th=[ 120], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 224, max= 432, per=25.03%, avg=309.84, stdev=51.78, samples=31 iops : min= 56, max= 108, avg=77.45, stdev=12.94, samples=31 write: IOPS=81, BW=327KiB/s (334kB/s)(5264KiB/16122msec); 0 zone resets slat (usec): min=29, max=456, avg=96.12, stdev=50.95 clat (msec): min=14, max=217, avg=131.81, stdev=30.99 lat (msec): min=14, max=217, avg=131.91, stdev=30.99 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 85], 10.00th=[ 95], 20.00th=[ 107], | 30.00th=[ 115], 40.00th=[ 123], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 171], 95.00th=[ 184], | 99.00th=[ 205], 99.50th=[ 211], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 272, max= 384, per=25.33%, avg=326.45, stdev=34.22, samples=31 iops : min= 68, max= 96, avg=81.61, stdev= 8.55, samples=31 lat (msec) : 2=0.12%, 20=0.47%, 50=3.20%, 100=51.17%, 250=45.04% cpu : usr=0.21%, sys=2.06%, ctx=2559, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122562: Sat Feb 28 14:12:31 2026 read: IOPS=76, BW=308KiB/s (315kB/s)(4984KiB/16207msec) slat (usec): min=744, max=3627, avg=1300.11, stdev=331.42 clat (usec): min=1837, max=120989, avg=67184.20, stdev=12686.67 lat (msec): min=3, max=122, avg=68.49, stdev=12.73 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 78], 90.00th=[ 83], 95.00th=[ 88], | 99.00th=[ 108], 99.50th=[ 115], 99.90th=[ 121], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 168, max= 448, per=24.84%, avg=307.50, stdev=62.23, samples=32 iops : min= 42, max= 112, avg=76.87, stdev=15.56, samples=32 write: IOPS=81, BW=324KiB/s (332kB/s)(5256KiB/16207msec); 0 zone resets slat (usec): min=32, max=475, avg=95.79, stdev=48.68 clat (msec): min=20, max=230, avg=132.07, stdev=31.19 lat (msec): min=20, max=230, avg=132.17, stdev=31.19 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 83], 10.00th=[ 93], 20.00th=[ 108], | 30.00th=[ 117], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 171], 95.00th=[ 184], | 99.00th=[ 205], 99.50th=[ 211], 99.90th=[ 224], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 232, max= 408, per=24.97%, avg=321.91, stdev=43.54, samples=32 iops : min= 58, max= 102, avg=80.47, stdev=10.88, samples=32 lat (msec) : 2=0.04%, 10=0.08%, 20=0.04%, 50=2.62%, 100=52.70% lat (msec) : 250=44.53% cpu : usr=0.25%, sys=2.05%, ctx=2626, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122563: Sat Feb 28 14:12:31 2026 read: IOPS=78, BW=313KiB/s (320kB/s)(5060KiB/16176msec) slat (usec): min=765, max=4345, avg=1286.84, stdev=340.65 clat (msec): min=10, max=123, avg=67.19, stdev=12.78 lat (msec): min=11, max=124, avg=68.48, stdev=12.79 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 49], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 83], 95.00th=[ 89], | 99.00th=[ 101], 99.50th=[ 104], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 176, max= 456, per=25.28%, avg=312.97, stdev=57.79, samples=32 iops : min= 44, max= 114, avg=78.22, stdev=14.45, samples=32 write: IOPS=80, BW=320KiB/s (328kB/s)(5180KiB/16176msec); 0 zone resets slat (usec): min=24, max=1327, avg=101.57, stdev=65.48 clat (msec): min=33, max=229, avg=132.61, stdev=29.96 lat (msec): min=33, max=229, avg=132.71, stdev=29.96 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 85], 10.00th=[ 95], 20.00th=[ 108], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 174], 95.00th=[ 186], | 99.00th=[ 201], 99.50th=[ 205], 99.90th=[ 218], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 256, max= 384, per=24.71%, avg=318.47, stdev=28.24, samples=32 iops : min= 64, max= 96, avg=79.59, stdev= 7.05, samples=32 lat (msec) : 20=0.27%, 50=3.09%, 100=52.73%, 250=43.91% cpu : usr=0.22%, sys=2.08%, ctx=2643, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1238KiB/s (1268kB/s), 308KiB/s-313KiB/s (315kB/s-320kB/s), io=19.6MiB (20.5MB), run=16122-16207msec WRITE: bw=1289KiB/s (1320kB/s), 320KiB/s-327KiB/s (328kB/s-334kB/s), io=20.4MiB (21.4MB), run=16122-16207msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=122570: Sat Feb 28 14:12:34 2026 read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec) slat (nsec): min=10330k, max=10330k, avg=10329701.00, stdev= 0.00 clat (nsec): min=75861k, max=75861k, avg=75861033.00, stdev= 0.00 lat (nsec): min=86225k, max=86225k, avg=86225366.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[76022], 5.00th=[76022], 10.00th=[76022], 20.00th=[76022], | 30.00th=[76022], 40.00th=[76022], 50.00th=[76022], 60.00th=[76022], | 70.00th=[76022], 80.00th=[76022], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=10.11%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck