== sanity test 398c: run fio to test AIO ================= 22:01:42 (1767841302) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.118927 s, 353 MB/s osc.lustre-OST0000-osc-ffff8cc7c6abb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=143887: Wed Jan 7 22:02:10 2026 write: IOPS=104, BW=417KiB/s (427kB/s)(10.0MiB/24566msec); 0 zone resets slat (usec): min=20, max=910, avg=109.24, stdev=56.36 clat (msec): min=11, max=266, avg=153.33, stdev=26.67 lat (msec): min=11, max=266, avg=153.44, stdev=26.67 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 132], | 30.00th=[ 140], 40.00th=[ 146], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 174], 90.00th=[ 188], 95.00th=[ 201], | 99.00th=[ 222], 99.50th=[ 230], 99.90th=[ 243], 99.95th=[ 249], | 99.99th=[ 266] bw ( KiB/s): min= 312, max= 523, per=24.94%, avg=415.56, stdev=50.42, samples=48 iops : min= 78, max= 130, avg=103.87, stdev=12.57, samples=48 lat (msec) : 20=0.04%, 100=1.45%, 250=98.48%, 500=0.04% cpu : usr=0.19%, sys=1.08%, ctx=1898, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=143888: Wed Jan 7 22:02:10 2026 write: IOPS=104, BW=419KiB/s (429kB/s)(10.0MiB/24457msec); 0 zone resets slat (usec): min=21, max=714, avg=107.05, stdev=54.81 clat (msec): min=35, max=253, avg=152.67, stdev=27.80 lat (msec): min=35, max=253, avg=152.78, stdev=27.80 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 109], 10.00th=[ 120], 20.00th=[ 131], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 199], | 99.00th=[ 224], 99.50th=[ 230], 99.90th=[ 247], 99.95th=[ 251], | 99.99th=[ 253] bw ( KiB/s): min= 328, max= 592, per=25.11%, avg=418.33, stdev=56.74, samples=48 iops : min= 82, max= 148, avg=104.58, stdev=14.18, samples=48 lat (msec) : 50=0.70%, 100=1.84%, 250=97.38%, 500=0.08% cpu : usr=0.15%, sys=1.09%, ctx=1889, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=143889: Wed Jan 7 22:02:10 2026 write: IOPS=104, BW=417KiB/s (427kB/s)(10.0MiB/24574msec); 0 zone resets slat (usec): min=21, max=2079, avg=108.59, stdev=66.13 clat (msec): min=59, max=239, avg=153.38, stdev=26.71 lat (msec): min=59, max=239, avg=153.48, stdev=26.71 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 112], 10.00th=[ 120], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 146], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 176], 90.00th=[ 188], 95.00th=[ 201], | 99.00th=[ 222], 99.50th=[ 228], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 312, max= 516, per=24.90%, avg=414.92, stdev=49.51, samples=48 iops : min= 78, max= 129, avg=103.73, stdev=12.38, samples=48 lat (msec) : 100=1.09%, 250=98.91% cpu : usr=0.20%, sys=1.08%, ctx=1888, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=143890: Wed Jan 7 22:02:10 2026 write: IOPS=104, BW=417KiB/s (427kB/s)(10.0MiB/24574msec); 0 zone resets slat (usec): min=26, max=584, avg=109.88, stdev=53.75 clat (msec): min=56, max=253, avg=153.38, stdev=26.93 lat (msec): min=56, max=253, avg=153.49, stdev=26.93 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 112], 10.00th=[ 120], 20.00th=[ 131], | 30.00th=[ 140], 40.00th=[ 146], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 174], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 228], 99.50th=[ 230], 99.90th=[ 243], 99.95th=[ 247], | 99.99th=[ 253] bw ( KiB/s): min= 328, max= 520, per=24.96%, avg=415.92, stdev=49.83, samples=48 iops : min= 82, max= 130, avg=103.98, stdev=12.46, samples=48 lat (msec) : 100=1.52%, 250=98.44%, 500=0.04% cpu : usr=0.17%, sys=1.09%, ctx=1923, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1667KiB/s (1707kB/s), 417KiB/s-419KiB/s (427kB/s-429kB/s), io=40.0MiB (41.9MB), run=24457-24574msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=143903: Wed Jan 7 22:02:27 2026 read: IOPS=89, BW=360KiB/s (368kB/s)(5048KiB/14035msec) slat (usec): min=643, max=5145, avg=1130.99, stdev=361.42 clat (usec): min=1634, max=171555, avg=57949.37, stdev=11884.83 lat (msec): min=2, max=173, avg=59.08, stdev=11.90 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 66], 90.00th=[ 71], 95.00th=[ 74], | 99.00th=[ 85], 99.50th=[ 90], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 200, max= 472, per=25.26%, avg=360.19, stdev=61.26, samples=27 iops : min= 50, max= 118, avg=90.04, stdev=15.31, samples=27 write: IOPS=92, BW=370KiB/s (379kB/s)(5192KiB/14035msec); 0 zone resets slat (usec): min=23, max=626, avg=88.94, stdev=51.62 clat (msec): min=45, max=235, avg=115.17, stdev=25.72 lat (msec): min=45, max=235, avg=115.26, stdev=25.72 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 74], 10.00th=[ 82], 20.00th=[ 92], | 30.00th=[ 103], 40.00th=[ 110], 50.00th=[ 116], 60.00th=[ 123], | 70.00th=[ 129], 80.00th=[ 136], 90.00th=[ 146], 95.00th=[ 157], | 99.00th=[ 182], 99.50th=[ 194], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 288, max= 456, per=24.98%, avg=371.00, stdev=37.42, samples=27 iops : min= 72, max= 114, avg=92.74, stdev= 9.35, samples=27 lat (msec) : 2=0.08%, 10=0.08%, 20=0.08%, 50=10.47%, 100=52.46% lat (msec) : 250=36.84% cpu : usr=0.29%, sys=2.14%, ctx=2619, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143904: Wed Jan 7 22:02:27 2026 read: IOPS=88, BW=354KiB/s (362kB/s)(4976KiB/14064msec) slat (usec): min=667, max=4250, avg=1143.75, stdev=339.33 clat (usec): min=8608, max=90107, avg=58211.29, stdev=10050.40 lat (usec): min=10581, max=91161, avg=59356.52, stdev=10052.45 clat percentiles (usec): | 1.00th=[33817], 5.00th=[42730], 10.00th=[46924], 20.00th=[50070], | 30.00th=[53740], 40.00th=[56361], 50.00th=[58459], 60.00th=[60556], | 70.00th=[63177], 80.00th=[65799], 90.00th=[70779], 95.00th=[73925], | 99.00th=[82314], 99.50th=[87557], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 240, max= 472, per=24.67%, avg=351.78, stdev=65.35, samples=27 iops : min= 60, max= 118, avg=87.93, stdev=16.33, samples=27 write: IOPS=93, BW=374KiB/s (383kB/s)(5264KiB/14064msec); 0 zone resets slat (usec): min=21, max=1067, avg=90.33, stdev=55.13 clat (msec): min=50, max=205, avg=114.60, stdev=24.88 lat (msec): min=50, max=205, avg=114.69, stdev=24.88 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 73], 10.00th=[ 83], 20.00th=[ 93], | 30.00th=[ 102], 40.00th=[ 108], 50.00th=[ 115], 60.00th=[ 123], | 70.00th=[ 130], 80.00th=[ 136], 90.00th=[ 146], 95.00th=[ 153], | 99.00th=[ 167], 99.50th=[ 180], 99.90th=[ 203], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 312, max= 432, per=25.19%, avg=374.07, stdev=35.42, samples=27 iops : min= 78, max= 108, avg=93.52, stdev= 8.86, samples=27 lat (msec) : 10=0.04%, 20=0.20%, 50=9.57%, 100=53.71%, 250=36.48% cpu : usr=0.31%, sys=2.09%, ctx=2647, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143905: Wed Jan 7 22:02:27 2026 read: IOPS=88, BW=355KiB/s (363kB/s)(4984KiB/14056msec) slat (usec): min=42, max=4341, avg=1135.38, stdev=354.40 clat (usec): min=1879, max=120620, avg=57697.17, stdev=10830.40 lat (usec): min=1953, max=121667, avg=58833.80, stdev=10880.00 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 61], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 70], 95.00th=[ 74], | 99.00th=[ 85], 99.50th=[ 88], 99.90th=[ 100], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 248, max= 480, per=25.01%, avg=356.70, stdev=62.23, samples=27 iops : min= 62, max= 120, avg=89.15, stdev=15.54, samples=27 write: IOPS=93, BW=374KiB/s (383kB/s)(5256KiB/14056msec); 0 zone resets slat (usec): min=27, max=449, avg=86.00, stdev=47.49 clat (msec): min=15, max=205, avg=115.14, stdev=25.27 lat (msec): min=15, max=205, avg=115.23, stdev=25.28 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 75], 10.00th=[ 84], 20.00th=[ 94], | 30.00th=[ 103], 40.00th=[ 110], 50.00th=[ 117], 60.00th=[ 123], | 70.00th=[ 129], 80.00th=[ 138], 90.00th=[ 148], 95.00th=[ 155], | 99.00th=[ 167], 99.50th=[ 178], 99.90th=[ 205], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 288, max= 469, per=24.95%, avg=370.56, stdev=43.27, samples=27 iops : min= 72, max= 117, avg=92.63, stdev=10.80, samples=27 lat (msec) : 2=0.04%, 4=0.08%, 10=0.31%, 20=0.12%, 50=10.16% lat (msec) : 100=52.19%, 250=37.11% cpu : usr=0.26%, sys=2.13%, ctx=2619, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143906: Wed Jan 7 22:02:27 2026 read: IOPS=89, BW=360KiB/s (369kB/s)(5060KiB/14056msec) slat (usec): min=641, max=4768, avg=1136.18, stdev=379.92 clat (usec): min=1756, max=99135, avg=58526.61, stdev=10087.58 lat (msec): min=2, max=100, avg=59.66, stdev=10.11 clat percentiles (usec): | 1.00th=[36963], 5.00th=[43254], 10.00th=[46924], 20.00th=[50070], | 30.00th=[53740], 40.00th=[55837], 50.00th=[58459], 60.00th=[61080], | 70.00th=[63701], 80.00th=[66847], 90.00th=[70779], 95.00th=[73925], | 99.00th=[83362], 99.50th=[86508], 99.90th=[98042], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 216, max= 488, per=25.31%, avg=360.93, stdev=62.36, samples=27 iops : min= 54, max= 122, avg=90.22, stdev=15.58, samples=27 write: IOPS=92, BW=369KiB/s (377kB/s)(5180KiB/14056msec); 0 zone resets slat (usec): min=25, max=563, avg=87.24, stdev=50.42 clat (msec): min=25, max=203, avg=115.05, stdev=25.08 lat (msec): min=25, max=203, avg=115.14, stdev=25.08 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 75], 10.00th=[ 84], 20.00th=[ 93], | 30.00th=[ 102], 40.00th=[ 110], 50.00th=[ 116], 60.00th=[ 122], | 70.00th=[ 130], 80.00th=[ 138], 90.00th=[ 148], 95.00th=[ 155], | 99.00th=[ 174], 99.50th=[ 184], 99.90th=[ 194], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 296, max= 464, per=24.78%, avg=368.04, stdev=42.24, samples=27 iops : min= 74, max= 116, avg=92.00, stdev=10.55, samples=27 lat (msec) : 2=0.04%, 4=0.04%, 10=0.08%, 20=0.08%, 50=9.57% lat (msec) : 100=54.10%, 250=36.09% cpu : usr=0.28%, sys=2.11%, ctx=2638, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1427KiB/s (1461kB/s), 354KiB/s-360KiB/s (362kB/s-369kB/s), io=19.6MiB (20.5MB), run=14035-14064msec WRITE: bw=1485KiB/s (1521kB/s), 369KiB/s-374KiB/s (377kB/s-383kB/s), io=20.4MiB (21.4MB), run=14035-14064msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=143911: Wed Jan 7 22:02:30 2026 read: IOPS=11, BW=460MiB/s (482MB/s)(40.0MiB/87msec) slat (nsec): min=10076k, max=10076k, avg=10075691.00, stdev= 0.00 clat (nsec): min=73380k, max=73380k, avg=73379568.00, stdev= 0.00 lat (nsec): min=83459k, max=83459k, avg=83458787.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[72877], 5.00th=[72877], 10.00th=[72877], 20.00th=[72877], | 30.00th=[72877], 40.00th=[72877], 50.00th=[72877], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=11.63%, ctx=3, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=460MiB/s (482MB/s), 460MiB/s-460MiB/s (482MB/s-482MB/s), io=40.0MiB (41.9MB), run=87-87msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck