== sanity test 398c: run fio to test AIO ================= 22:45:56 (1767325556) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.159893 s, 262 MB/s osc.lustre-OST0000-osc-ffff8b7760217800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=186919: Thu Jan 1 22:46:09 2026 write: IOPS=297, BW=1192KiB/s (1220kB/s)(10.0MiB/8594msec); 0 zone resets slat (usec): min=30, max=4265, avg=110.38, stdev=161.40 clat (msec): min=9, max=136, avg=53.53, stdev=17.56 lat (msec): min=9, max=136, avg=53.64, stdev=17.56 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 30], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 52], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 67], 90.00th=[ 78], 95.00th=[ 86], | 99.00th=[ 104], 99.50th=[ 117], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 888, max= 1974, per=25.24%, avg=1192.88, stdev=259.12, samples=16 iops : min= 222, max= 493, avg=298.06, stdev=64.63, samples=16 lat (msec) : 10=0.04%, 20=0.55%, 50=46.72%, 100=51.17%, 250=1.52% cpu : usr=0.35%, sys=2.37%, ctx=2755, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186920: Thu Jan 1 22:46:09 2026 write: IOPS=295, BW=1182KiB/s (1210kB/s)(10.0MiB/8664msec); 0 zone resets slat (usec): min=21, max=3642, avg=115.32, stdev=179.99 clat (msec): min=20, max=127, avg=53.70, stdev=17.03 lat (msec): min=20, max=127, avg=53.81, stdev=17.04 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 44], 40.00th=[ 48], 50.00th=[ 53], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 67], 90.00th=[ 77], 95.00th=[ 84], | 99.00th=[ 104], 99.50th=[ 113], 99.90th=[ 123], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 896, max= 1858, per=24.77%, avg=1170.71, stdev=232.92, samples=17 iops : min= 224, max= 464, avg=292.65, stdev=58.14, samples=17 lat (msec) : 50=46.37%, 100=52.46%, 250=1.17% cpu : usr=0.47%, sys=2.27%, ctx=2692, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186921: Thu Jan 1 22:46:09 2026 write: IOPS=295, BW=1183KiB/s (1211kB/s)(10.0MiB/8656msec); 0 zone resets slat (usec): min=22, max=4723, avg=116.66, stdev=166.47 clat (msec): min=3, max=125, avg=53.65, stdev=17.31 lat (msec): min=3, max=125, avg=53.77, stdev=17.32 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 31], 10.00th=[ 33], 20.00th=[ 40], | 30.00th=[ 44], 40.00th=[ 48], 50.00th=[ 52], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 67], 90.00th=[ 77], 95.00th=[ 86], | 99.00th=[ 110], 99.50th=[ 112], 99.90th=[ 123], 99.95th=[ 125], | 99.99th=[ 127] bw ( KiB/s): min= 896, max= 1880, per=24.77%, avg=1170.65, stdev=238.85, samples=17 iops : min= 224, max= 470, avg=292.65, stdev=59.73, samples=17 lat (msec) : 4=0.04%, 10=0.04%, 20=0.16%, 50=46.13%, 100=51.88% lat (msec) : 250=1.76% cpu : usr=0.45%, sys=2.77%, ctx=2651, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186922: Thu Jan 1 22:46:09 2026 write: IOPS=296, BW=1187KiB/s (1215kB/s)(10.0MiB/8628msec); 0 zone resets slat (usec): min=24, max=26765, avg=123.81, stdev=540.91 clat (msec): min=8, max=127, avg=53.52, stdev=17.50 lat (msec): min=8, max=127, avg=53.64, stdev=17.48 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 30], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 52], 60.00th=[ 56], | 70.00th=[ 61], 80.00th=[ 68], 90.00th=[ 78], 95.00th=[ 85], | 99.00th=[ 104], 99.50th=[ 120], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 128] bw ( KiB/s): min= 864, max= 1883, per=25.18%, avg=1190.19, stdev=242.72, samples=16 iops : min= 216, max= 470, avg=297.50, stdev=60.54, samples=16 lat (msec) : 10=0.12%, 20=0.51%, 50=46.25%, 100=51.68%, 250=1.45% cpu : usr=0.50%, sys=2.69%, ctx=2760, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4728KiB/s (4841kB/s), 1182KiB/s-1192KiB/s (1210kB/s-1220kB/s), io=40.0MiB (41.9MB), run=8594-8664msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=186933: Thu Jan 1 22:46:23 2026 read: IOPS=119, BW=478KiB/s (490kB/s)(5048KiB/10553msec) slat (usec): min=1224, max=21651, avg=4598.44, stdev=2433.87 clat (msec): min=26, max=127, avg=62.22, stdev=18.07 lat (msec): min=29, max=134, avg=66.82, stdev=19.06 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 46], | 30.00th=[ 50], 40.00th=[ 55], 50.00th=[ 61], 60.00th=[ 66], | 70.00th=[ 71], 80.00th=[ 78], 90.00th=[ 87], 95.00th=[ 95], | 99.00th=[ 111], 99.50th=[ 118], 99.90th=[ 125], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 220, max= 688, per=25.12%, avg=477.50, stdev=128.73, samples=20 iops : min= 55, max= 172, avg=119.10, stdev=32.31, samples=20 write: IOPS=122, BW=492KiB/s (504kB/s)(5192KiB/10553msec); 0 zone resets slat (usec): min=23, max=4139, avg=118.30, stdev=187.18 clat (msec): min=13, max=119, avg=63.81, stdev=17.08 lat (msec): min=13, max=119, avg=63.93, stdev=17.09 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 49], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 67], | 70.00th=[ 72], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 94], | 99.00th=[ 109], 99.50th=[ 112], 99.90th=[ 117], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 285, max= 760, per=24.87%, avg=492.20, stdev=121.17, samples=20 iops : min= 71, max= 190, avg=122.80, stdev=30.41, samples=20 lat (msec) : 20=0.04%, 50=27.54%, 100=69.65%, 250=2.77% cpu : usr=0.45%, sys=3.07%, ctx=4565, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186934: Thu Jan 1 22:46:23 2026 read: IOPS=117, BW=472KiB/s (483kB/s)(4976KiB/10552msec) slat (usec): min=1166, max=22525, avg=4614.48, stdev=2479.95 clat (msec): min=18, max=124, avg=62.46, stdev=18.08 lat (msec): min=22, max=131, avg=67.08, stdev=19.01 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 47], | 30.00th=[ 51], 40.00th=[ 55], 50.00th=[ 59], 60.00th=[ 66], | 70.00th=[ 72], 80.00th=[ 78], 90.00th=[ 88], 95.00th=[ 95], | 99.00th=[ 111], 99.50th=[ 116], 99.90th=[ 122], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 252, max= 680, per=24.69%, avg=469.45, stdev=129.09, samples=20 iops : min= 63, max= 170, avg=117.10, stdev=32.41, samples=20 write: IOPS=124, BW=499KiB/s (511kB/s)(5264KiB/10552msec); 0 zone resets slat (usec): min=26, max=3957, avg=116.47, stdev=166.05 clat (msec): min=27, max=116, avg=63.80, stdev=17.44 lat (msec): min=27, max=116, avg=63.92, stdev=17.45 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 67], | 70.00th=[ 73], 80.00th=[ 80], 90.00th=[ 90], 95.00th=[ 95], | 99.00th=[ 108], 99.50th=[ 111], 99.90th=[ 116], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 291, max= 768, per=25.23%, avg=499.35, stdev=117.87, samples=20 iops : min= 72, max= 192, avg=124.55, stdev=29.63, samples=20 lat (msec) : 20=0.04%, 50=27.62%, 100=69.14%, 250=3.20% cpu : usr=0.47%, sys=3.25%, ctx=4591, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186935: Thu Jan 1 22:46:23 2026 read: IOPS=118, BW=475KiB/s (487kB/s)(4984KiB/10484msec) slat (usec): min=40, max=19619, avg=4605.29, stdev=2409.48 clat (msec): min=7, max=128, avg=62.62, stdev=18.54 lat (msec): min=7, max=132, avg=67.23, stdev=19.61 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 51], 40.00th=[ 55], 50.00th=[ 61], 60.00th=[ 66], | 70.00th=[ 72], 80.00th=[ 80], 90.00th=[ 89], 95.00th=[ 95], | 99.00th=[ 111], 99.50th=[ 115], 99.90th=[ 124], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 298, max= 648, per=25.18%, avg=478.60, stdev=105.92, samples=20 iops : min= 74, max= 162, avg=119.35, stdev=26.72, samples=20 write: IOPS=125, BW=501KiB/s (513kB/s)(5256KiB/10484msec); 0 zone resets slat (usec): min=34, max=6007, avg=131.94, stdev=265.07 clat (msec): min=10, max=120, avg=63.59, stdev=17.74 lat (msec): min=10, max=120, avg=63.72, stdev=17.72 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 49], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 61], 60.00th=[ 67], | 70.00th=[ 72], 80.00th=[ 79], 90.00th=[ 89], 95.00th=[ 99], | 99.00th=[ 108], 99.50th=[ 113], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 304, max= 752, per=25.26%, avg=499.80, stdev=135.25, samples=20 iops : min= 76, max= 188, avg=124.70, stdev=33.91, samples=20 lat (msec) : 10=0.08%, 20=0.16%, 50=27.34%, 100=68.87%, 250=3.55% cpu : usr=0.47%, sys=3.26%, ctx=4499, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186936: Thu Jan 1 22:46:23 2026 read: IOPS=120, BW=480KiB/s (492kB/s)(5060KiB/10541msec) slat (usec): min=1166, max=20100, avg=4592.87, stdev=2333.73 clat (msec): min=17, max=119, avg=62.31, stdev=17.70 lat (msec): min=25, max=129, avg=66.91, stdev=18.68 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 46], | 30.00th=[ 50], 40.00th=[ 54], 50.00th=[ 59], 60.00th=[ 66], | 70.00th=[ 72], 80.00th=[ 80], 90.00th=[ 88], 95.00th=[ 95], | 99.00th=[ 106], 99.50th=[ 108], 99.90th=[ 114], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 247, max= 688, per=25.21%, avg=479.20, stdev=120.72, samples=20 iops : min= 61, max= 172, avg=119.50, stdev=30.37, samples=20 write: IOPS=122, BW=491KiB/s (503kB/s)(5180KiB/10541msec); 0 zone resets slat (usec): min=31, max=6315, avg=134.46, stdev=273.43 clat (msec): min=31, max=122, avg=63.94, stdev=17.25 lat (msec): min=31, max=122, avg=64.07, stdev=17.27 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 49], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 67], | 70.00th=[ 73], 80.00th=[ 80], 90.00th=[ 89], 95.00th=[ 96], | 99.00th=[ 108], 99.50th=[ 109], 99.90th=[ 114], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 317, max= 712, per=24.85%, avg=491.75, stdev=119.41, samples=20 iops : min= 79, max= 178, avg=122.65, stdev=30.05, samples=20 lat (msec) : 20=0.04%, 50=27.62%, 100=69.57%, 250=2.77% cpu : usr=0.48%, sys=3.20%, ctx=4722, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1902KiB/s (1947kB/s), 472KiB/s-480KiB/s (483kB/s-492kB/s), io=19.6MiB (20.5MB), run=10484-10553msec WRITE: bw=1980KiB/s (2027kB/s), 491KiB/s-501KiB/s (503kB/s-513kB/s), io=20.4MiB (21.4MB), run=10484-10553msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=186941: Thu Jan 1 22:46:27 2026 read: IOPS=5, BW=233MiB/s (244MB/s)(40.0MiB/172msec) slat (nsec): min=13188k, max=13188k, avg=13188265.00, stdev= 0.00 clat (nsec): min=151023k, max=151023k, avg=151022895.00, stdev= 0.00 lat (nsec): min=164216k, max=164216k, avg=164215523.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.43%, ctx=13, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=233MiB/s (244MB/s), 233MiB/s-233MiB/s (244MB/s-244MB/s), io=40.0MiB (41.9MB), run=172-172msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout