== sanity test 398c: run fio to test AIO ================= 15:30:59 (1788291059) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.470542 s, 89.1 MB/s osc.lustre-OST0000-osc-ffff93705143c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179655: Tue Sep 1 15:31:35 2026 write: IOPS=91, BW=366KiB/s (375kB/s)(10.0MiB/27971msec); 0 zone resets slat (usec): min=25, max=10698, avg=164.97, stdev=593.14 clat (msec): min=21, max=437, avg=174.34, stdev=49.22 lat (msec): min=21, max=438, avg=174.51, stdev=49.21 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 103], 10.00th=[ 116], 20.00th=[ 134], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 259], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 231, max= 629, per=25.04%, avg=366.33, stdev=81.75, samples=55 iops : min= 57, max= 157, avg=91.24, stdev=20.44, samples=55 lat (msec) : 50=0.08%, 100=3.91%, 250=89.30%, 500=6.72% cpu : usr=0.15%, sys=0.77%, ctx=2287, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179656: Tue Sep 1 15:31:35 2026 write: IOPS=91, BW=366KiB/s (375kB/s)(10.0MiB/27989msec); 0 zone resets slat (usec): min=28, max=19587, avg=162.60, stdev=756.71 clat (msec): min=50, max=353, avg=174.03, stdev=47.10 lat (msec): min=50, max=353, avg=174.19, stdev=47.10 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 104], 10.00th=[ 116], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 300], 99.50th=[ 317], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 183, max= 629, per=24.98%, avg=365.44, stdev=83.96, samples=55 iops : min= 45, max= 157, avg=91.02, stdev=21.01, samples=55 lat (msec) : 100=3.98%, 250=88.36%, 500=7.66% cpu : usr=0.20%, sys=0.71%, ctx=2077, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179657: Tue Sep 1 15:31:35 2026 write: IOPS=92, BW=368KiB/s (377kB/s)(10.0MiB/27818msec); 0 zone resets slat (usec): min=27, max=24056, avg=160.72, stdev=720.44 clat (msec): min=16, max=486, avg=173.37, stdev=48.75 lat (msec): min=16, max=486, avg=173.53, stdev=48.77 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 103], 10.00th=[ 114], 20.00th=[ 133], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 489] bw ( KiB/s): min= 255, max= 614, per=25.14%, avg=367.76, stdev=86.11, samples=55 iops : min= 63, max= 153, avg=91.60, stdev=21.52, samples=55 lat (msec) : 20=0.12%, 50=0.23%, 100=4.38%, 250=89.57%, 500=5.70% cpu : usr=0.17%, sys=0.78%, ctx=2063, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179658: Tue Sep 1 15:31:35 2026 write: IOPS=91, BW=366KiB/s (375kB/s)(10.0MiB/27976msec); 0 zone resets slat (usec): min=28, max=20086, avg=187.23, stdev=822.01 clat (msec): min=15, max=329, avg=174.02, stdev=46.41 lat (msec): min=15, max=329, avg=174.21, stdev=46.41 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 106], 10.00th=[ 121], 20.00th=[ 134], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 236], 95.00th=[ 262], | 99.00th=[ 309], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 215, max= 614, per=24.95%, avg=365.02, stdev=79.58, samples=55 iops : min= 53, max= 153, avg=90.91, stdev=19.89, samples=55 lat (msec) : 20=0.04%, 50=0.08%, 100=3.59%, 250=89.49%, 500=6.80% cpu : usr=0.15%, sys=0.83%, ctx=2134, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1463KiB/s (1499kB/s), 366KiB/s-368KiB/s (375kB/s-377kB/s), io=40.0MiB (41.9MB), run=27818-27989msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179673: Tue Sep 1 15:32:22 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(5048KiB/36545msec) slat (usec): min=2640, max=82895, avg=16062.05, stdev=9955.79 clat (msec): min=77, max=479, avg=214.94, stdev=55.71 lat (msec): min=92, max=496, avg=231.00, stdev=57.99 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 257], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 384], 99.50th=[ 401], 99.90th=[ 472], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 63, max= 247, per=24.97%, avg=137.08, stdev=37.81, samples=72 iops : min= 15, max= 61, avg=33.81, stdev= 9.45, samples=72 write: IOPS=35, BW=142KiB/s (145kB/s)(5192KiB/36545msec); 0 zone resets slat (usec): min=31, max=17483, avg=218.99, stdev=782.11 clat (msec): min=94, max=434, avg=223.43, stdev=55.28 lat (msec): min=95, max=434, avg=223.66, stdev=55.30 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 146], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 247], 80.00th=[ 266], 90.00th=[ 296], 95.00th=[ 326], | 99.00th=[ 384], 99.50th=[ 405], 99.90th=[ 426], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 58, max= 238, per=24.75%, avg=141.32, stdev=38.37, samples=72 iops : min= 14, max= 59, avg=34.88, stdev= 9.61, samples=72 lat (msec) : 100=0.12%, 250=74.53%, 500=25.35% cpu : usr=0.22%, sys=1.35%, ctx=3926, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179674: Tue Sep 1 15:32:22 2026 read: IOPS=34, BW=136KiB/s (139kB/s)(4976KiB/36541msec) slat (msec): min=2, max=256, avg=16.40, stdev=12.29 clat (msec): min=37, max=496, avg=213.59, stdev=56.75 lat (msec): min=43, max=518, avg=230.00, stdev=59.66 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 129], 10.00th=[ 146], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 380], 99.50th=[ 418], 99.90th=[ 451], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 62, max= 215, per=24.63%, avg=135.19, stdev=36.87, samples=72 iops : min= 15, max= 53, avg=33.35, stdev= 9.25, samples=72 write: IOPS=36, BW=144KiB/s (148kB/s)(5264KiB/36541msec); 0 zone resets slat (usec): min=31, max=16709, avg=195.05, stdev=744.89 clat (msec): min=101, max=482, avg=225.06, stdev=55.38 lat (msec): min=101, max=482, avg=225.25, stdev=55.35 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 249], 80.00th=[ 266], 90.00th=[ 300], 95.00th=[ 326], | 99.00th=[ 372], 99.50th=[ 451], 99.90th=[ 477], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 79, max= 215, per=25.12%, avg=143.44, stdev=33.20, samples=72 iops : min= 19, max= 53, avg=35.43, stdev= 8.33, samples=72 lat (msec) : 50=0.04%, 100=0.08%, 250=73.63%, 500=26.25% cpu : usr=0.22%, sys=1.41%, ctx=3802, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179675: Tue Sep 1 15:32:22 2026 read: IOPS=34, BW=137KiB/s (141kB/s)(4984KiB/36275msec) slat (usec): min=36, max=81928, avg=15469.94, stdev=9869.13 clat (msec): min=77, max=434, avg=215.31, stdev=57.05 lat (msec): min=77, max=458, avg=230.79, stdev=59.63 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 131], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 363], 99.50th=[ 388], 99.90th=[ 430], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 63, max= 245, per=25.02%, avg=137.37, stdev=35.52, samples=71 iops : min= 15, max= 61, avg=33.90, stdev= 8.90, samples=71 write: IOPS=36, BW=145KiB/s (148kB/s)(5256KiB/36275msec); 0 zone resets slat (usec): min=31, max=12314, avg=165.82, stdev=548.22 clat (msec): min=44, max=432, avg=222.32, stdev=56.40 lat (msec): min=45, max=432, avg=222.49, stdev=56.37 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 176], | 30.00th=[ 190], 40.00th=[ 203], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 247], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 326], | 99.00th=[ 372], 99.50th=[ 380], 99.90th=[ 414], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 56, max= 239, per=25.25%, avg=144.18, stdev=38.55, samples=71 iops : min= 14, max= 59, avg=35.59, stdev= 9.65, samples=71 lat (msec) : 50=0.08%, 100=0.90%, 250=72.85%, 500=26.17% cpu : usr=0.20%, sys=1.29%, ctx=3874, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179676: Tue Sep 1 15:32:22 2026 read: IOPS=34, BW=139KiB/s (142kB/s)(5060KiB/36502msec) slat (usec): min=2403, max=81577, avg=15947.33, stdev=9826.85 clat (msec): min=52, max=396, avg=214.61, stdev=54.24 lat (msec): min=62, max=440, avg=230.56, stdev=56.68 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 368], 99.50th=[ 376], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 23, max= 229, per=25.06%, avg=137.58, stdev=35.94, samples=72 iops : min= 5, max= 57, avg=33.96, stdev= 9.02, samples=72 write: IOPS=35, BW=142KiB/s (145kB/s)(5180KiB/36502msec); 0 zone resets slat (usec): min=35, max=45228, avg=233.50, stdev=1491.29 clat (msec): min=100, max=404, avg=223.13, stdev=53.18 lat (msec): min=100, max=404, avg=223.37, stdev=53.22 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 247], 80.00th=[ 266], 90.00th=[ 296], 95.00th=[ 321], | 99.00th=[ 363], 99.50th=[ 380], 99.90th=[ 401], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 71, max= 247, per=24.67%, avg=140.85, stdev=35.46, samples=72 iops : min= 17, max= 61, avg=34.76, stdev= 8.86, samples=72 lat (msec) : 100=0.16%, 250=72.73%, 500=27.11% cpu : usr=0.19%, sys=1.29%, ctx=4065, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=549KiB/s (562kB/s), 136KiB/s-139KiB/s (139kB/s-142kB/s), io=19.6MiB (20.5MB), run=36275-36545msec WRITE: bw=572KiB/s (585kB/s), 142KiB/s-145KiB/s (145kB/s-148kB/s), io=20.4MiB (21.4MB), run=36275-36545msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179685: Tue Sep 1 15:32:33 2026 read: IOPS=1, BW=70.3MiB/s (73.7MB/s)(40.0MiB/569msec) slat (nsec): min=59487k, max=59487k, avg=59487144.00, stdev= 0.00 clat (nsec): min=478826k, max=478826k, avg=478826493.00, stdev= 0.00 lat (nsec): min=538317k, max=538317k, avg=538317462.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 481], 10.00th=[ 481], 20.00th=[ 481], | 30.00th=[ 481], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] lat (msec) : 500=100.00% cpu : usr=0.18%, sys=2.29%, ctx=8, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.3MiB/s (73.7MB/s), 70.3MiB/s-70.3MiB/s (73.7MB/s-73.7MB/s), io=40.0MiB (41.9MB), run=569-569msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout