== sanity test 398c: run fio to test AIO ================= 00:16:39 (1787372199) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.54483 s, 77.0 MB/s osc.lustre-OST0000-osc-ffff94aa60620000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179744: Sat Aug 22 00:17:19 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27663msec); 0 zone resets slat (usec): min=30, max=21930, avg=225.04, stdev=798.23 clat (msec): min=15, max=337, avg=172.45, stdev=36.95 lat (msec): min=15, max=337, avg=172.67, stdev=36.96 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 220], 95.00th=[ 232], | 99.00th=[ 266], 99.50th=[ 271], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 249, max= 534, per=25.14%, avg=370.52, stdev=62.82, samples=54 iops : min= 62, max= 133, avg=92.28, stdev=15.69, samples=54 lat (msec) : 20=0.04%, 50=0.31%, 100=1.41%, 250=96.21%, 500=2.03% cpu : usr=0.20%, sys=1.00%, ctx=2570, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179745: Sat Aug 22 00:17:19 2026 write: IOPS=92, BW=369KiB/s (377kB/s)(10.0MiB/27788msec); 0 zone resets slat (usec): min=27, max=10798, avg=217.64, stdev=642.24 clat (msec): min=17, max=331, avg=172.73, stdev=35.97 lat (msec): min=18, max=331, avg=172.95, stdev=35.97 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 114], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 249, max= 582, per=24.96%, avg=367.89, stdev=63.14, samples=55 iops : min= 62, max= 145, avg=91.60, stdev=15.80, samples=55 lat (msec) : 20=0.04%, 50=0.04%, 100=1.68%, 250=95.55%, 500=2.70% cpu : usr=0.15%, sys=0.98%, ctx=3022, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179746: Sat Aug 22 00:17:19 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27655msec); 0 zone resets slat (usec): min=21, max=8788, avg=179.09, stdev=545.58 clat (msec): min=36, max=277, avg=171.73, stdev=35.75 lat (msec): min=36, max=277, avg=171.91, stdev=35.74 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 255], 99.50th=[ 271], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 230, max= 534, per=24.85%, avg=366.35, stdev=65.83, samples=54 iops : min= 57, max= 133, avg=91.22, stdev=16.47, samples=54 lat (msec) : 50=0.51%, 100=1.21%, 250=96.45%, 500=1.84% cpu : usr=0.17%, sys=0.79%, ctx=1700, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179747: Sat Aug 22 00:17:19 2026 write: IOPS=92, BW=369KiB/s (377kB/s)(10.0MiB/27785msec); 0 zone resets slat (usec): min=29, max=25384, avg=234.52, stdev=870.03 clat (msec): min=57, max=281, avg=172.57, stdev=34.59 lat (msec): min=57, max=281, avg=172.81, stdev=34.57 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 117], 10.00th=[ 127], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 215], 95.00th=[ 228], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 229, max= 518, per=24.92%, avg=367.31, stdev=63.23, samples=55 iops : min= 57, max= 129, avg=91.47, stdev=15.79, samples=55 lat (msec) : 100=1.09%, 250=97.34%, 500=1.56% cpu : usr=0.19%, sys=1.00%, ctx=2426, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1474KiB/s (1509kB/s), 369KiB/s-370KiB/s (377kB/s-379kB/s), io=40.0MiB (41.9MB), run=27655-27788msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179762: Sat Aug 22 00:18:03 2026 read: IOPS=35, BW=143KiB/s (147kB/s)(5048KiB/35219msec) slat (usec): min=3563, max=57047, avg=15549.39, stdev=8443.75 clat (msec): min=16, max=469, avg=207.03, stdev=50.78 lat (msec): min=32, max=496, avg=222.58, stdev=52.10 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 127], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 439], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 47, max= 223, per=25.24%, avg=143.09, stdev=34.08, samples=69 iops : min= 11, max= 55, avg=35.29, stdev= 8.54, samples=69 write: IOPS=36, BW=147KiB/s (151kB/s)(5192KiB/35219msec); 0 zone resets slat (usec): min=30, max=23198, avg=285.96, stdev=1352.38 clat (msec): min=107, max=405, avg=214.63, stdev=49.07 lat (msec): min=108, max=406, avg=214.92, stdev=49.06 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 388], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 87, max= 247, per=25.00%, avg=147.51, stdev=32.75, samples=69 iops : min= 21, max= 61, avg=36.38, stdev= 8.20, samples=69 lat (msec) : 20=0.04%, 50=0.04%, 100=0.39%, 250=78.48%, 500=21.05% cpu : usr=0.20%, sys=1.60%, ctx=4451, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179763: Sat Aug 22 00:18:03 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(4976KiB/35365msec) slat (usec): min=116, max=241142, avg=15945.62, stdev=12003.37 clat (msec): min=30, max=731, avg=209.16, stdev=65.15 lat (msec): min=30, max=787, avg=225.11, stdev=67.63 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 123], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 359], 99.50th=[ 634], 99.90th=[ 693], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 63, max= 207, per=24.65%, avg=139.77, stdev=33.71, samples=70 iops : min= 15, max= 51, avg=34.46, stdev= 8.42, samples=70 write: IOPS=37, BW=149KiB/s (152kB/s)(5264KiB/35365msec); 0 zone resets slat (usec): min=29, max=23788, avg=297.07, stdev=1236.53 clat (msec): min=30, max=708, avg=214.65, stdev=57.44 lat (msec): min=30, max=710, avg=214.95, stdev=57.47 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 132], 10.00th=[ 153], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 363], 99.50th=[ 397], 99.90th=[ 651], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 39, max= 270, per=25.02%, avg=147.64, stdev=40.47, samples=70 iops : min= 9, max= 67, avg=36.43, stdev=10.20, samples=70 lat (msec) : 50=0.31%, 100=1.48%, 250=76.88%, 500=20.78%, 750=0.55% cpu : usr=0.22%, sys=1.57%, ctx=4462, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179764: Sat Aug 22 00:18:03 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(4984KiB/34956msec) slat (usec): min=47, max=60588, avg=15175.17, stdev=8551.40 clat (msec): min=25, max=438, avg=204.92, stdev=52.55 lat (msec): min=26, max=466, avg=220.10, stdev=54.27 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 418], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 71, max= 262, per=25.12%, avg=142.41, stdev=36.25, samples=69 iops : min= 17, max= 65, avg=35.12, stdev= 9.15, samples=69 write: IOPS=37, BW=150KiB/s (154kB/s)(5256KiB/34956msec); 0 zone resets slat (usec): min=34, max=34327, avg=273.78, stdev=1420.06 clat (msec): min=60, max=434, avg=216.05, stdev=52.08 lat (msec): min=61, max=434, avg=216.32, stdev=52.14 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 368], 99.50th=[ 388], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 86, max= 224, per=25.31%, avg=149.35, stdev=34.15, samples=69 iops : min= 21, max= 56, avg=36.84, stdev= 8.52, samples=69 lat (msec) : 50=0.31%, 100=0.98%, 250=77.97%, 500=20.74% cpu : usr=0.22%, sys=1.55%, ctx=4078, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179765: Sat Aug 22 00:18:03 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(5060KiB/35149msec) slat (usec): min=2552, max=84000, avg=15081.99, stdev=8785.63 clat (msec): min=89, max=429, avg=206.43, stdev=48.55 lat (msec): min=98, max=444, avg=221.52, stdev=49.90 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 330], 99.50th=[ 338], 99.90th=[ 355], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 85, max= 232, per=25.40%, avg=144.03, stdev=33.97, samples=69 iops : min= 21, max= 58, avg=35.52, stdev= 8.53, samples=69 write: IOPS=36, BW=147KiB/s (151kB/s)(5180KiB/35149msec); 0 zone resets slat (usec): min=30, max=16680, avg=249.22, stdev=948.22 clat (msec): min=52, max=455, avg=214.70, stdev=49.22 lat (msec): min=52, max=455, avg=214.95, stdev=49.19 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 447], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 79, max= 238, per=24.89%, avg=146.86, stdev=33.49, samples=69 iops : min= 19, max= 59, avg=36.22, stdev= 8.36, samples=69 lat (msec) : 100=0.27%, 250=79.30%, 500=20.43% cpu : usr=0.21%, sys=1.70%, ctx=4126, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=567KiB/s (581kB/s), 141KiB/s-144KiB/s (144kB/s-147kB/s), io=19.6MiB (20.5MB), run=34956-35365msec WRITE: bw=591KiB/s (605kB/s), 147KiB/s-150KiB/s (151kB/s-154kB/s), io=20.4MiB (21.4MB), run=34956-35365msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179774: Sat Aug 22 00:18:14 2026 read: IOPS=2, BW=83.2MiB/s (87.2MB/s)(40.0MiB/481msec) slat (nsec): min=44776k, max=44776k, avg=44776071.00, stdev= 0.00 clat (nsec): min=413912k, max=413912k, avg=413911729.00, stdev= 0.00 lat (nsec): min=458692k, max=458692k, avg=458691950.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.50%, ctx=16, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.2MiB/s (87.2MB/s), 83.2MiB/s-83.2MiB/s (87.2MB/s-87.2MB/s), io=40.0MiB (41.9MB), run=481-481msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout