== sanity test 398c: run fio to test AIO ================= 10:48:09 (1788878889) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.142431 s, 294 MB/s osc.lustre-OST0000-osc-ffff9c7e48edd000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180142: Tue Sep 8 10:48:18 2026 write: IOPS=481, BW=1926KiB/s (1972kB/s)(10.0MiB/5317msec); 0 zone resets slat (usec): min=20, max=27108, avg=109.57, stdev=539.49 clat (usec): min=6253, max=77283, avg=32980.41, stdev=5720.55 lat (usec): min=6500, max=77332, avg=33090.91, stdev=5712.87 clat percentiles (usec): | 1.00th=[20055], 5.00th=[26084], 10.00th=[27919], 20.00th=[29230], | 30.00th=[30278], 40.00th=[31327], 50.00th=[31851], 60.00th=[33162], | 70.00th=[34341], 80.00th=[35914], 90.00th=[40633], 95.00th=[42730], | 99.00th=[50594], 99.50th=[54264], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 1776, max= 2080, per=25.23%, avg=1940.10, stdev=116.26, samples=10 iops : min= 444, max= 520, avg=485.00, stdev=29.03, samples=10 lat (msec) : 10=0.08%, 20=0.66%, 50=97.89%, 100=1.37% cpu : usr=0.45%, sys=4.33%, ctx=2591, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180143: Tue Sep 8 10:48:18 2026 write: IOPS=480, BW=1922KiB/s (1968kB/s)(10.0MiB/5327msec); 0 zone resets slat (usec): min=21, max=1122, avg=94.37, stdev=69.63 clat (usec): min=16959, max=58031, avg=32869.52, stdev=5410.80 lat (usec): min=17034, max=58133, avg=32965.04, stdev=5417.85 clat percentiles (usec): | 1.00th=[20055], 5.00th=[25297], 10.00th=[27919], 20.00th=[29492], | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32900], | 70.00th=[34341], 80.00th=[35914], 90.00th=[40109], 95.00th=[42730], | 99.00th=[49546], 99.50th=[53740], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 1760, max= 2088, per=25.13%, avg=1932.10, stdev=108.28, samples=10 iops : min= 440, max= 522, avg=483.00, stdev=27.05, samples=10 lat (msec) : 20=1.09%, 50=98.12%, 100=0.78% cpu : usr=0.43%, sys=3.66%, ctx=2684, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180144: Tue Sep 8 10:48:18 2026 write: IOPS=482, BW=1932KiB/s (1978kB/s)(10.0MiB/5301msec); 0 zone resets slat (usec): min=20, max=990, avg=94.67, stdev=64.74 clat (usec): min=13624, max=97801, avg=32980.38, stdev=6183.28 lat (usec): min=13756, max=97883, avg=33076.23, stdev=6187.63 clat percentiles (usec): | 1.00th=[17433], 5.00th=[25297], 10.00th=[27919], 20.00th=[29492], | 30.00th=[30278], 40.00th=[31327], 50.00th=[31851], 60.00th=[33162], | 70.00th=[34341], 80.00th=[35914], 90.00th=[40633], 95.00th=[42730], | 99.00th=[50594], 99.50th=[62129], 99.90th=[94897], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 1744, max= 2150, per=25.31%, avg=1946.20, stdev=131.43, samples=10 iops : min= 436, max= 537, avg=486.50, stdev=32.77, samples=10 lat (msec) : 20=1.80%, 50=96.95%, 100=1.25% cpu : usr=0.60%, sys=3.53%, ctx=2556, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180145: Tue Sep 8 10:48:18 2026 write: IOPS=480, BW=1923KiB/s (1969kB/s)(10.0MiB/5326msec); 0 zone resets slat (usec): min=26, max=3605, avg=99.93, stdev=98.36 clat (usec): min=3660, max=66109, avg=32885.43, stdev=5692.39 lat (usec): min=3720, max=66161, avg=32986.57, stdev=5702.07 clat percentiles (usec): | 1.00th=[18744], 5.00th=[25822], 10.00th=[27657], 20.00th=[29230], | 30.00th=[30540], 40.00th=[31065], 50.00th=[31851], 60.00th=[32900], | 70.00th=[34341], 80.00th=[36439], 90.00th=[40633], 95.00th=[43254], | 99.00th=[50070], 99.50th=[54264], 99.90th=[58983], 99.95th=[61604], | 99.99th=[66323] bw ( KiB/s): min= 1792, max= 2080, per=25.16%, avg=1934.40, stdev=106.25, samples=10 iops : min= 448, max= 520, avg=483.60, stdev=26.56, samples=10 lat (msec) : 4=0.04%, 10=0.12%, 20=1.41%, 50=97.38%, 100=1.05% cpu : usr=0.49%, sys=4.30%, ctx=2715, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=7689KiB/s (7874kB/s), 1922KiB/s-1932KiB/s (1968kB/s-1978kB/s), io=40.0MiB (41.9MB), run=5301-5327msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180156: Tue Sep 8 10:48:28 2026 read: IOPS=161, BW=644KiB/s (660kB/s)(5048KiB/7833msec) slat (usec): min=1034, max=11791, avg=3136.36, stdev=1389.29 clat (msec): min=9, max=104, avg=46.10, stdev=11.04 lat (msec): min=13, max=109, avg=49.24, stdev=11.53 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 37], | 30.00th=[ 40], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 47], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 62], 95.00th=[ 65], | 99.00th=[ 78], 99.50th=[ 91], 99.90th=[ 104], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 400, max= 830, per=25.13%, avg=643.60, stdev=117.35, samples=15 iops : min= 100, max= 207, avg=160.80, stdev=29.33, samples=15 write: IOPS=165, BW=663KiB/s (679kB/s)(5192KiB/7833msec); 0 zone resets slat (usec): min=27, max=2614, avg=104.53, stdev=111.13 clat (msec): min=17, max=106, avg=47.87, stdev=11.80 lat (msec): min=18, max=106, avg=47.97, stdev=11.81 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], 20.00th=[ 38], | 30.00th=[ 41], 40.00th=[ 44], 50.00th=[ 47], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 64], 95.00th=[ 69], | 99.00th=[ 85], 99.50th=[ 99], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 512, max= 800, per=24.74%, avg=659.80, stdev=91.43, samples=15 iops : min= 128, max= 200, avg=164.87, stdev=22.89, samples=15 lat (msec) : 10=0.04%, 20=0.04%, 50=66.29%, 100=33.48%, 250=0.16% cpu : usr=0.41%, sys=3.61%, ctx=4750, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180157: Tue Sep 8 10:48:28 2026 read: IOPS=158, BW=635KiB/s (651kB/s)(4976KiB/7833msec) slat (usec): min=996, max=10387, avg=3133.61, stdev=1378.43 clat (msec): min=23, max=114, avg=46.54, stdev=11.55 lat (msec): min=24, max=122, avg=49.67, stdev=12.12 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 37], | 30.00th=[ 40], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 48], | 70.00th=[ 52], 80.00th=[ 55], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 85], 99.50th=[ 93], 99.90th=[ 110], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 472, max= 734, per=24.58%, avg=629.60, stdev=96.46, samples=15 iops : min= 118, max= 183, avg=157.27, stdev=24.16, samples=15 write: IOPS=168, BW=672KiB/s (688kB/s)(5264KiB/7833msec); 0 zone resets slat (usec): min=30, max=3898, avg=106.12, stdev=185.00 clat (msec): min=8, max=112, avg=47.70, stdev=11.82 lat (msec): min=8, max=112, avg=47.81, stdev=11.84 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 36], 20.00th=[ 38], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 50], | 70.00th=[ 52], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 87], 99.50th=[ 97], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 464, max= 856, per=25.16%, avg=671.07, stdev=111.72, samples=15 iops : min= 116, max= 214, avg=167.67, stdev=27.97, samples=15 lat (msec) : 10=0.04%, 20=0.04%, 50=65.35%, 100=34.26%, 250=0.31% cpu : usr=0.43%, sys=3.59%, ctx=4782, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180158: Tue Sep 8 10:48:28 2026 read: IOPS=159, BW=636KiB/s (652kB/s)(4984KiB/7833msec) slat (usec): min=1130, max=11024, avg=3173.62, stdev=1406.93 clat (msec): min=22, max=117, avg=46.30, stdev=11.34 lat (msec): min=24, max=127, avg=49.48, stdev=11.90 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 33], 10.00th=[ 35], 20.00th=[ 38], | 30.00th=[ 40], 40.00th=[ 42], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 61], 95.00th=[ 66], | 99.00th=[ 82], 99.50th=[ 95], 99.90th=[ 115], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 432, max= 808, per=24.88%, avg=637.07, stdev=103.87, samples=15 iops : min= 108, max= 202, avg=159.13, stdev=26.04, samples=15 write: IOPS=167, BW=671KiB/s (687kB/s)(5256KiB/7833msec); 0 zone resets slat (usec): min=29, max=2370, avg=105.66, stdev=118.20 clat (msec): min=14, max=103, avg=47.58, stdev=11.70 lat (msec): min=14, max=103, avg=47.69, stdev=11.70 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 38], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 85], 99.50th=[ 97], 99.90th=[ 103], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 479, max= 808, per=24.93%, avg=664.87, stdev=114.90, samples=15 iops : min= 119, max= 202, avg=166.07, stdev=28.85, samples=15 lat (msec) : 20=0.08%, 50=66.72%, 100=32.93%, 250=0.27% cpu : usr=0.36%, sys=3.60%, ctx=4790, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180159: Tue Sep 8 10:48:28 2026 read: IOPS=162, BW=650KiB/s (665kB/s)(5060KiB/7790msec) slat (usec): min=41, max=11163, avg=3136.34, stdev=1435.58 clat (msec): min=4, max=114, avg=46.50, stdev=11.92 lat (msec): min=6, max=122, avg=49.64, stdev=12.47 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 38], | 30.00th=[ 40], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 48], | 70.00th=[ 52], 80.00th=[ 55], 90.00th=[ 62], 95.00th=[ 67], | 99.00th=[ 89], 99.50th=[ 101], 99.90th=[ 111], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 464, max= 792, per=25.35%, avg=649.13, stdev=98.90, samples=15 iops : min= 116, max= 198, avg=162.13, stdev=24.84, samples=15 write: IOPS=166, BW=665KiB/s (681kB/s)(5180KiB/7790msec); 0 zone resets slat (usec): min=34, max=3656, avg=110.14, stdev=161.15 clat (msec): min=7, max=110, avg=47.47, stdev=11.26 lat (msec): min=8, max=111, avg=47.59, stdev=11.26 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 41], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 81], 99.50th=[ 93], 99.90th=[ 103], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 512, max= 832, per=24.84%, avg=662.47, stdev=95.94, samples=15 iops : min= 128, max= 208, avg=165.53, stdev=23.96, samples=15 lat (msec) : 10=0.20%, 20=0.12%, 50=65.12%, 100=34.18%, 250=0.39% cpu : usr=0.42%, sys=3.63%, ctx=4834, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2562KiB/s (2623kB/s), 635KiB/s-650KiB/s (651kB/s-665kB/s), io=19.6MiB (20.5MB), run=7790-7833msec WRITE: bw=2667KiB/s (2731kB/s), 663KiB/s-672KiB/s (679kB/s-688kB/s), io=20.4MiB (21.4MB), run=7790-7833msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180163: Tue Sep 8 10:48:32 2026 read: IOPS=8, BW=320MiB/s (336MB/s)(40.0MiB/125msec) slat (nsec): min=11327k, max=11327k, avg=11327374.00, stdev= 0.00 clat (nsec): min=107983k, max=107983k, avg=107983027.00, stdev= 0.00 lat (nsec): min=119314k, max=119314k, avg=119314422.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.06%, ctx=10, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=320MiB/s (336MB/s), 320MiB/s-320MiB/s (336MB/s-336MB/s), io=40.0MiB (41.9MB), run=125-125msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout