== sanity test 398c: run fio to test AIO ================= 10:19:05 (1783952345) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.301727 s, 139 MB/s osc.lustre-OST0000-osc-ffff89eed39a7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=151229: Mon Jul 13 10:19:39 2026 write: IOPS=103, BW=416KiB/s (426kB/s)(10.0MiB/24623msec); 0 zone resets slat (usec): min=37, max=14031, avg=345.05, stdev=881.39 clat (msec): min=21, max=260, avg=153.15, stdev=29.54 lat (msec): min=21, max=260, avg=153.49, stdev=29.51 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 109], 10.00th=[ 117], 20.00th=[ 130], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 192], 95.00th=[ 205], | 99.00th=[ 236], 99.50th=[ 239], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 270, max= 552, per=25.00%, avg=415.71, stdev=65.19, samples=48 iops : min= 67, max= 138, avg=103.60, stdev=16.37, samples=48 lat (msec) : 50=0.12%, 100=2.58%, 250=97.15%, 500=0.16% cpu : usr=0.25%, sys=1.60%, ctx=2667, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151230: Mon Jul 13 10:19:39 2026 write: IOPS=104, BW=416KiB/s (426kB/s)(10.0MiB/24611msec); 0 zone resets slat (usec): min=43, max=19313, avg=375.17, stdev=1055.19 clat (msec): min=37, max=290, avg=152.87, stdev=28.88 lat (msec): min=37, max=290, avg=153.25, stdev=28.86 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 107], 10.00th=[ 116], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 176], 90.00th=[ 190], 95.00th=[ 205], | 99.00th=[ 226], 99.50th=[ 234], 99.90th=[ 249], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 310, max= 512, per=25.03%, avg=416.19, stdev=60.78, samples=48 iops : min= 77, max= 128, avg=103.73, stdev=15.24, samples=48 lat (msec) : 50=0.08%, 100=2.30%, 250=97.54%, 500=0.08% cpu : usr=0.24%, sys=1.66%, ctx=2414, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151231: Mon Jul 13 10:19:39 2026 write: IOPS=104, BW=418KiB/s (428kB/s)(10.0MiB/24489msec); 0 zone resets slat (usec): min=38, max=23457, avg=378.96, stdev=1226.05 clat (msec): min=24, max=310, avg=152.27, stdev=31.03 lat (msec): min=24, max=310, avg=152.66, stdev=31.00 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 108], 10.00th=[ 116], 20.00th=[ 131], | 30.00th=[ 140], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 190], 95.00th=[ 205], | 99.00th=[ 236], 99.50th=[ 245], 99.90th=[ 292], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 254, max= 517, per=25.15%, avg=418.19, stdev=71.81, samples=48 iops : min= 63, max= 129, avg=104.23, stdev=17.99, samples=48 lat (msec) : 50=0.43%, 100=2.50%, 250=96.80%, 500=0.27% cpu : usr=0.21%, sys=1.78%, ctx=2342, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151232: Mon Jul 13 10:19:39 2026 write: IOPS=104, BW=418KiB/s (428kB/s)(10.0MiB/24495msec); 0 zone resets slat (usec): min=36, max=18802, avg=364.34, stdev=932.94 clat (msec): min=21, max=321, avg=152.57, stdev=30.59 lat (msec): min=22, max=321, avg=152.93, stdev=30.60 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 111], 10.00th=[ 116], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 174], 90.00th=[ 190], 95.00th=[ 207], | 99.00th=[ 232], 99.50th=[ 236], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 296, max= 518, per=25.19%, avg=418.87, stdev=62.18, samples=48 iops : min= 74, max= 129, avg=104.40, stdev=15.55, samples=48 lat (msec) : 50=0.74%, 100=1.37%, 250=97.66%, 500=0.23% cpu : usr=0.26%, sys=1.58%, ctx=2571, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1663KiB/s (1703kB/s), 416KiB/s-418KiB/s (426kB/s-428kB/s), io=40.0MiB (41.9MB), run=24489-24623msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151248: Mon Jul 13 10:20:18 2026 read: IOPS=41, BW=165KiB/s (169kB/s)(5048KiB/30624msec) slat (usec): min=2661, max=53569, avg=13714.69, stdev=7270.94 clat (msec): min=57, max=328, avg=179.57, stdev=41.63 lat (msec): min=63, max=355, avg=193.29, stdev=42.99 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 115], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 44, max= 268, per=25.03%, avg=163.92, stdev=38.37, samples=60 iops : min= 11, max= 67, avg=40.47, stdev= 9.63, samples=60 write: IOPS=42, BW=170KiB/s (174kB/s)(5192KiB/30624msec); 0 zone resets slat (usec): min=55, max=16850, avg=322.60, stdev=956.64 clat (msec): min=64, max=352, avg=185.36, stdev=38.45 lat (msec): min=64, max=353, avg=185.68, stdev=38.44 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 126], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 234], 95.00th=[ 249], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 342], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 44, max= 248, per=24.82%, avg=169.30, stdev=37.95, samples=60 iops : min= 11, max= 62, avg=41.82, stdev= 9.47, samples=60 lat (msec) : 100=0.66%, 250=94.49%, 500=4.84% cpu : usr=0.28%, sys=2.38%, ctx=4637, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151249: Mon Jul 13 10:20:18 2026 read: IOPS=40, BW=163KiB/s (167kB/s)(4976KiB/30512msec) slat (usec): min=2239, max=52584, avg=13668.17, stdev=7011.16 clat (msec): min=89, max=425, avg=179.22, stdev=41.89 lat (msec): min=97, max=438, avg=192.90, stdev=43.02 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 393], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 37, max= 263, per=24.84%, avg=162.72, stdev=40.61, samples=60 iops : min= 9, max= 65, avg=40.15, stdev=10.16, samples=60 write: IOPS=43, BW=173KiB/s (177kB/s)(5264KiB/30512msec); 0 zone resets slat (usec): min=49, max=14035, avg=376.15, stdev=1002.46 clat (msec): min=26, max=425, avg=185.07, stdev=39.66 lat (msec): min=26, max=425, avg=185.45, stdev=39.69 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 384], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 94, max= 284, per=25.34%, avg=172.80, stdev=38.28, samples=60 iops : min= 23, max= 71, avg=42.68, stdev= 9.65, samples=60 lat (msec) : 50=0.04%, 100=0.51%, 250=94.49%, 500=4.96% cpu : usr=0.31%, sys=2.25%, ctx=4630, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151250: Mon Jul 13 10:20:18 2026 read: IOPS=40, BW=163KiB/s (167kB/s)(4984KiB/30587msec) slat (usec): min=2731, max=56569, avg=13871.17, stdev=6991.25 clat (msec): min=33, max=318, avg=178.40, stdev=40.12 lat (msec): min=51, max=338, avg=192.28, stdev=41.51 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 118], 10.00th=[ 127], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 245], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 317], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 37, max= 263, per=24.76%, avg=162.17, stdev=41.79, samples=60 iops : min= 9, max= 65, avg=40.02, stdev=10.50, samples=60 write: IOPS=42, BW=172KiB/s (176kB/s)(5256KiB/30587msec); 0 zone resets slat (usec): min=42, max=13167, avg=329.08, stdev=875.63 clat (msec): min=64, max=345, avg=186.08, stdev=39.51 lat (msec): min=64, max=345, avg=186.41, stdev=39.53 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 126], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 59, max= 260, per=25.02%, avg=170.63, stdev=41.59, samples=60 iops : min= 14, max= 65, avg=42.15, stdev=10.41, samples=60 lat (msec) : 50=0.04%, 100=0.74%, 250=94.34%, 500=4.88% cpu : usr=0.27%, sys=2.22%, ctx=4334, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151251: Mon Jul 13 10:20:18 2026 read: IOPS=41, BW=166KiB/s (170kB/s)(5060KiB/30447msec) slat (usec): min=60, max=56430, avg=13684.75, stdev=7309.47 clat (msec): min=46, max=337, avg=179.23, stdev=41.22 lat (msec): min=49, max=349, avg=192.93, stdev=42.12 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 251], | 99.00th=[ 288], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 94, max= 287, per=25.36%, avg=166.12, stdev=39.22, samples=60 iops : min= 23, max= 71, avg=41.00, stdev= 9.80, samples=60 write: IOPS=42, BW=170KiB/s (174kB/s)(5180KiB/30447msec); 0 zone resets slat (usec): min=38, max=17864, avg=357.73, stdev=911.93 clat (msec): min=85, max=383, avg=186.57, stdev=39.45 lat (msec): min=91, max=383, avg=186.93, stdev=39.44 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 305], 99.50th=[ 330], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 111, max= 236, per=24.85%, avg=169.50, stdev=28.66, samples=60 iops : min= 27, max= 59, avg=41.87, stdev= 7.25, samples=60 lat (msec) : 50=0.04%, 100=0.62%, 250=93.95%, 500=5.39% cpu : usr=0.20%, sys=2.25%, ctx=4495, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=655KiB/s (671kB/s), 163KiB/s-166KiB/s (167kB/s-170kB/s), io=19.6MiB (20.5MB), run=30447-30624msec WRITE: bw=682KiB/s (699kB/s), 170KiB/s-173KiB/s (174kB/s-177kB/s), io=20.4MiB (21.4MB), run=30447-30624msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=151259: Mon Jul 13 10:20:28 2026 read: IOPS=1, BW=62.1MiB/s (65.1MB/s)(40.0MiB/644msec) slat (nsec): min=65690k, max=65690k, avg=65689992.00, stdev= 0.00 clat (nsec): min=507610k, max=507610k, avg=507610219.00, stdev= 0.00 lat (nsec): min=573341k, max=573341k, avg=573340650.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.18%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.1MiB/s (65.1MB/s), 62.1MiB/s-62.1MiB/s (65.1MB/s-65.1MB/s), io=40.0MiB (41.9MB), run=644-644msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout