== sanity test 398c: run fio to test AIO ================= 02:48:56 (1769759336) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.242991 s, 173 MB/s osc.lustre-OST0000-osc-ffff919e19763800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183644: Fri Jan 30 02:49:15 2026 write: IOPS=182, BW=732KiB/s (749kB/s)(10.0MiB/13994msec); 0 zone resets slat (usec): min=23, max=4675, avg=131.10, stdev=294.77 clat (msec): min=19, max=159, avg=86.86, stdev=20.22 lat (msec): min=19, max=159, avg=86.99, stdev=20.22 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 55], 10.00th=[ 61], 20.00th=[ 69], | 30.00th=[ 75], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 93], | 70.00th=[ 99], 80.00th=[ 105], 90.00th=[ 112], 95.00th=[ 121], | 99.00th=[ 134], 99.50th=[ 138], 99.90th=[ 140], 99.95th=[ 142], | 99.99th=[ 161] bw ( KiB/s): min= 574, max= 1040, per=25.00%, avg=731.52, stdev=120.90, samples=27 iops : min= 143, max= 260, avg=182.74, stdev=30.27, samples=27 lat (msec) : 20=0.04%, 50=3.28%, 100=70.43%, 250=26.25% cpu : usr=0.31%, sys=1.51%, ctx=2967, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183645: Fri Jan 30 02:49:15 2026 write: IOPS=183, BW=735KiB/s (753kB/s)(10.0MiB/13927msec); 0 zone resets slat (usec): min=21, max=7749, avg=149.47, stdev=346.93 clat (msec): min=11, max=187, avg=86.76, stdev=20.90 lat (msec): min=11, max=187, avg=86.91, stdev=20.88 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 53], 10.00th=[ 60], 20.00th=[ 69], | 30.00th=[ 75], 40.00th=[ 82], 50.00th=[ 87], 60.00th=[ 93], | 70.00th=[ 99], 80.00th=[ 104], 90.00th=[ 113], 95.00th=[ 120], | 99.00th=[ 136], 99.50th=[ 138], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 188] bw ( KiB/s): min= 564, max= 1102, per=25.15%, avg=735.89, stdev=139.85, samples=27 iops : min= 141, max= 275, avg=183.81, stdev=34.96, samples=27 lat (msec) : 20=0.08%, 50=3.28%, 100=70.04%, 250=26.60% cpu : usr=0.25%, sys=1.65%, ctx=3158, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183646: Fri Jan 30 02:49:15 2026 write: IOPS=182, BW=732KiB/s (749kB/s)(10.0MiB/13994msec); 0 zone resets slat (usec): min=21, max=13299, avg=114.59, stdev=360.09 clat (msec): min=8, max=148, avg=86.82, stdev=20.00 lat (msec): min=8, max=148, avg=86.93, stdev=20.01 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 54], 10.00th=[ 62], 20.00th=[ 69], | 30.00th=[ 77], 40.00th=[ 82], 50.00th=[ 88], 60.00th=[ 94], | 70.00th=[ 100], 80.00th=[ 105], 90.00th=[ 111], 95.00th=[ 118], | 99.00th=[ 132], 99.50th=[ 136], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 150] bw ( KiB/s): min= 570, max= 1024, per=24.88%, avg=727.93, stdev=118.45, samples=27 iops : min= 142, max= 256, avg=181.81, stdev=29.64, samples=27 lat (msec) : 10=0.04%, 20=0.27%, 50=2.34%, 100=70.31%, 250=27.03% cpu : usr=0.19%, sys=1.26%, ctx=2110, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183647: Fri Jan 30 02:49:15 2026 write: IOPS=183, BW=732KiB/s (750kB/s)(10.0MiB/13984msec); 0 zone resets slat (usec): min=22, max=10465, avg=148.01, stdev=412.64 clat (msec): min=15, max=150, avg=86.76, stdev=20.31 lat (msec): min=15, max=150, avg=86.91, stdev=20.32 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 54], 10.00th=[ 61], 20.00th=[ 68], | 30.00th=[ 75], 40.00th=[ 82], 50.00th=[ 89], 60.00th=[ 93], | 70.00th=[ 97], 80.00th=[ 104], 90.00th=[ 113], 95.00th=[ 121], | 99.00th=[ 133], 99.50th=[ 142], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 588, max= 1024, per=25.05%, avg=732.89, stdev=117.99, samples=27 iops : min= 147, max= 256, avg=183.07, stdev=29.53, samples=27 lat (msec) : 20=0.04%, 50=3.01%, 100=72.89%, 250=24.06% cpu : usr=0.33%, sys=1.54%, ctx=3089, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2927KiB/s (2997kB/s), 732KiB/s-735KiB/s (749kB/s-753kB/s), io=40.0MiB (41.9MB), run=13927-13994msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183659: Fri Jan 30 02:49:33 2026 read: IOPS=87, BW=350KiB/s (358kB/s)(5048KiB/14434msec) slat (usec): min=36, max=26229, avg=6188.32, stdev=3171.41 clat (msec): min=11, max=158, avg=85.01, stdev=17.59 lat (msec): min=17, max=165, avg=91.20, stdev=18.19 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 88], | 70.00th=[ 93], 80.00th=[ 99], 90.00th=[ 109], 95.00th=[ 115], | 99.00th=[ 134], 99.50th=[ 138], 99.90th=[ 153], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 255, max= 447, per=25.23%, avg=347.89, stdev=53.46, samples=28 iops : min= 63, max= 111, avg=86.57, stdev=13.38, samples=28 write: IOPS=89, BW=360KiB/s (368kB/s)(5192KiB/14434msec); 0 zone resets slat (usec): min=27, max=4442, avg=143.77, stdev=301.67 clat (msec): min=44, max=153, avg=88.37, stdev=17.90 lat (msec): min=47, max=153, avg=88.51, stdev=17.90 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 97], 80.00th=[ 105], 90.00th=[ 113], 95.00th=[ 121], | 99.00th=[ 134], 99.50th=[ 144], 99.90th=[ 153], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 295, max= 447, per=24.95%, avg=358.32, stdev=43.64, samples=28 iops : min= 73, max= 111, avg=89.21, stdev=10.90, samples=28 lat (msec) : 20=0.04%, 50=0.35%, 100=77.85%, 250=21.76% cpu : usr=0.30%, sys=2.49%, ctx=4708, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183660: Fri Jan 30 02:49:33 2026 read: IOPS=85, BW=342KiB/s (350kB/s)(4976KiB/14546msec) slat (usec): min=34, max=118477, avg=6377.41, stdev=5917.92 clat (msec): min=4, max=462, avg=86.25, stdev=34.78 lat (msec): min=5, max=464, avg=92.63, stdev=36.10 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 57], 10.00th=[ 63], 20.00th=[ 70], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 88], | 70.00th=[ 93], 80.00th=[ 101], 90.00th=[ 110], 95.00th=[ 117], | 99.00th=[ 150], 99.50th=[ 397], 99.90th=[ 460], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 224, max= 447, per=24.23%, avg=334.18, stdev=54.75, samples=28 iops : min= 56, max= 111, avg=83.18, stdev=13.64, samples=28 write: IOPS=90, BW=362KiB/s (371kB/s)(5264KiB/14546msec); 0 zone resets slat (usec): min=22, max=5361, avg=139.75, stdev=280.43 clat (msec): min=11, max=430, avg=88.17, stdev=28.29 lat (msec): min=12, max=430, avg=88.31, stdev=28.30 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 59], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 90], | 70.00th=[ 95], 80.00th=[ 104], 90.00th=[ 113], 95.00th=[ 121], | 99.00th=[ 142], 99.50th=[ 159], 99.90th=[ 422], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 232, max= 504, per=24.76%, avg=355.61, stdev=65.61, samples=28 iops : min= 58, max= 126, avg=88.50, stdev=16.39, samples=28 lat (msec) : 10=0.08%, 20=0.55%, 50=2.07%, 100=75.23%, 250=21.48% lat (msec) : 500=0.59% cpu : usr=0.37%, sys=2.39%, ctx=4251, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183661: Fri Jan 30 02:49:33 2026 read: IOPS=86, BW=348KiB/s (356kB/s)(4984KiB/14327msec) slat (usec): min=36, max=24071, avg=6094.03, stdev=3055.87 clat (msec): min=15, max=172, avg=85.26, stdev=19.02 lat (msec): min=20, max=179, avg=91.35, stdev=19.57 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 89], | 70.00th=[ 94], 80.00th=[ 102], 90.00th=[ 110], 95.00th=[ 118], | 99.00th=[ 138], 99.50th=[ 144], 99.90th=[ 171], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 256, max= 471, per=25.23%, avg=347.96, stdev=54.14, samples=28 iops : min= 64, max= 117, avg=86.61, stdev=13.48, samples=28 write: IOPS=91, BW=367KiB/s (376kB/s)(5256KiB/14327msec); 0 zone resets slat (usec): min=28, max=6953, avg=142.77, stdev=322.54 clat (msec): min=26, max=161, avg=87.45, stdev=17.77 lat (msec): min=26, max=161, avg=87.59, stdev=17.77 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 79], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 90], | 70.00th=[ 95], 80.00th=[ 102], 90.00th=[ 111], 95.00th=[ 121], | 99.00th=[ 134], 99.50th=[ 144], 99.90th=[ 159], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 277, max= 488, per=25.44%, avg=365.36, stdev=56.42, samples=28 iops : min= 69, max= 122, avg=90.96, stdev=14.19, samples=28 lat (msec) : 20=0.04%, 50=1.33%, 100=77.03%, 250=21.60% cpu : usr=0.28%, sys=2.65%, ctx=4762, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183662: Fri Jan 30 02:49:33 2026 read: IOPS=87, BW=351KiB/s (360kB/s)(5060KiB/14408msec) slat (usec): min=53, max=23409, avg=6256.58, stdev=2986.96 clat (msec): min=7, max=169, avg=85.18, stdev=18.66 lat (msec): min=17, max=178, avg=91.44, stdev=19.10 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 89], | 70.00th=[ 94], 80.00th=[ 102], 90.00th=[ 111], 95.00th=[ 116], | 99.00th=[ 134], 99.50th=[ 144], 99.90th=[ 159], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 256, max= 447, per=25.40%, avg=350.32, stdev=53.04, samples=28 iops : min= 64, max= 111, avg=87.21, stdev=13.22, samples=28 write: IOPS=89, BW=360KiB/s (368kB/s)(5180KiB/14408msec); 0 zone resets slat (usec): min=24, max=4097, avg=138.61, stdev=231.97 clat (msec): min=17, max=163, avg=87.81, stdev=18.30 lat (msec): min=17, max=163, avg=87.95, stdev=18.30 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 97], 80.00th=[ 104], 90.00th=[ 112], 95.00th=[ 121], | 99.00th=[ 140], 99.50th=[ 142], 99.90th=[ 157], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 263, max= 518, per=24.86%, avg=356.93, stdev=53.40, samples=28 iops : min= 65, max= 129, avg=88.86, stdev=13.39, samples=28 lat (msec) : 10=0.08%, 20=0.04%, 50=1.09%, 100=75.90%, 250=22.89% cpu : usr=0.33%, sys=2.51%, ctx=4844, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1380KiB/s (1413kB/s), 342KiB/s-351KiB/s (350kB/s-360kB/s), io=19.6MiB (20.5MB), run=14327-14546msec WRITE: bw=1436KiB/s (1471kB/s), 360KiB/s-367KiB/s (368kB/s-376kB/s), io=20.4MiB (21.4MB), run=14327-14546msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183667: Fri Jan 30 02:49:38 2026 read: IOPS=3, BW=138MiB/s (145MB/s)(40.0MiB/289msec) slat (nsec): min=15916k, max=15916k, avg=15916158.00, stdev= 0.00 clat (nsec): min=255481k, max=255481k, avg=255480772.00, stdev= 0.00 lat (nsec): min=271401k, max=271401k, avg=271401158.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.17%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=40.0MiB (41.9MB), run=289-289msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout