== sanity test 398c: run fio to test AIO ================= 19:27:25 (1781047645) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.132179 s, 317 MB/s osc.lustre-OST0000-osc-ffff8d7b20a28000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=122109: Tue Jun 9 19:27:55 2026 write: IOPS=94, BW=379KiB/s (388kB/s)(10.0MiB/27000msec); 0 zone resets slat (usec): min=27, max=591, avg=112.48, stdev=53.97 clat (msec): min=48, max=316, avg=168.50, stdev=36.31 lat (msec): min=48, max=316, avg=168.61, stdev=36.32 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 110], 10.00th=[ 124], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 230], | 99.00th=[ 259], 99.50th=[ 275], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 317] bw ( KiB/s): min= 280, max= 620, per=24.94%, avg=378.34, stdev=64.18, samples=53 iops : min= 70, max= 155, avg=94.58, stdev=16.04, samples=53 lat (msec) : 50=0.62%, 100=2.50%, 250=95.35%, 500=1.52% cpu : usr=0.20%, sys=1.02%, ctx=1949, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122110: Tue Jun 9 19:27:55 2026 write: IOPS=94, BW=379KiB/s (388kB/s)(10.0MiB/26991msec); 0 zone resets slat (usec): min=26, max=468, avg=114.90, stdev=56.16 clat (msec): min=48, max=307, avg=168.45, stdev=36.38 lat (msec): min=48, max=307, avg=168.56, stdev=36.39 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 110], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 230], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 288], 99.95th=[ 300], | 99.99th=[ 309] bw ( KiB/s): min= 280, max= 620, per=24.96%, avg=378.64, stdev=63.14, samples=53 iops : min= 70, max= 155, avg=94.66, stdev=15.79, samples=53 lat (msec) : 50=0.62%, 100=2.34%, 250=95.59%, 500=1.45% cpu : usr=0.16%, sys=1.04%, ctx=2101, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122111: Tue Jun 9 19:27:55 2026 write: IOPS=94, BW=380KiB/s (389kB/s)(10.0MiB/26954msec); 0 zone resets slat (usec): min=25, max=778, avg=114.08, stdev=55.78 clat (msec): min=48, max=302, avg=168.22, stdev=36.55 lat (msec): min=48, max=302, avg=168.34, stdev=36.56 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 110], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 228], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 292], 99.95th=[ 300], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 620, per=24.99%, avg=379.09, stdev=67.49, samples=53 iops : min= 64, max= 155, avg=94.77, stdev=16.87, samples=53 lat (msec) : 50=0.62%, 100=2.54%, 250=95.51%, 500=1.33% cpu : usr=0.14%, sys=1.05%, ctx=1990, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122112: Tue Jun 9 19:27:55 2026 write: IOPS=94, BW=379KiB/s (388kB/s)(10.0MiB/26992msec); 0 zone resets slat (usec): min=22, max=735, avg=111.62, stdev=55.64 clat (msec): min=19, max=286, avg=168.49, stdev=36.87 lat (msec): min=19, max=286, avg=168.61, stdev=36.87 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 110], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 255], 99.50th=[ 275], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 256, max= 626, per=24.93%, avg=378.15, stdev=68.04, samples=53 iops : min= 64, max= 156, avg=94.53, stdev=16.98, samples=53 lat (msec) : 20=0.20%, 50=0.27%, 100=2.70%, 250=95.55%, 500=1.29% cpu : usr=0.22%, sys=0.97%, ctx=1938, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1517KiB/s (1553kB/s), 379KiB/s-380KiB/s (388kB/s-389kB/s), io=40.0MiB (41.9MB), run=26954-27000msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=122125: Tue Jun 9 19:28:14 2026 read: IOPS=80, BW=320KiB/s (328kB/s)(5048KiB/15753msec) slat (usec): min=670, max=5205, avg=1255.90, stdev=444.09 clat (msec): min=8, max=129, avg=65.66, stdev=12.30 lat (msec): min=10, max=131, avg=66.92, stdev=12.34 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 86], | 99.00th=[ 101], 99.50th=[ 104], 99.90th=[ 118], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 216, max= 456, per=25.17%, avg=320.39, stdev=64.08, samples=31 iops : min= 54, max= 114, avg=80.06, stdev=16.06, samples=31 write: IOPS=82, BW=330KiB/s (337kB/s)(5192KiB/15753msec); 0 zone resets slat (usec): min=30, max=613, avg=97.09, stdev=57.53 clat (msec): min=40, max=232, avg=128.81, stdev=32.34 lat (msec): min=40, max=232, avg=128.91, stdev=32.34 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 79], 10.00th=[ 88], 20.00th=[ 102], | 30.00th=[ 112], 40.00th=[ 121], 50.00th=[ 128], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 186], | 99.00th=[ 218], 99.50th=[ 226], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 240, max= 424, per=24.79%, avg=328.42, stdev=48.17, samples=31 iops : min= 60, max= 106, avg=82.06, stdev=12.03, samples=31 lat (msec) : 10=0.04%, 20=0.08%, 50=4.65%, 100=53.83%, 250=41.41% cpu : usr=0.36%, sys=1.91%, ctx=3252, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122126: Tue Jun 9 19:28:14 2026 read: IOPS=78, BW=316KiB/s (323kB/s)(4976KiB/15764msec) slat (usec): min=641, max=5849, avg=1232.80, stdev=411.17 clat (usec): min=1015, max=105597, avg=64536.05, stdev=12707.49 lat (msec): min=2, max=107, avg=65.77, stdev=12.74 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 85], | 99.00th=[ 93], 99.50th=[ 95], 99.90th=[ 103], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 192, max= 432, per=24.72%, avg=314.74, stdev=71.88, samples=31 iops : min= 48, max= 108, avg=78.65, stdev=17.98, samples=31 write: IOPS=83, BW=334KiB/s (342kB/s)(5264KiB/15764msec); 0 zone resets slat (usec): min=27, max=1976, avg=97.10, stdev=75.02 clat (msec): min=33, max=232, avg=129.24, stdev=32.03 lat (msec): min=34, max=232, avg=129.33, stdev=32.04 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 80], 10.00th=[ 88], 20.00th=[ 103], | 30.00th=[ 113], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 186], | 99.00th=[ 213], 99.50th=[ 222], 99.90th=[ 230], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 224, max= 408, per=25.14%, avg=333.06, stdev=43.75, samples=31 iops : min= 56, max= 102, avg=83.23, stdev=10.93, samples=31 lat (msec) : 2=0.04%, 10=0.23%, 20=0.35%, 50=4.73%, 100=52.54% lat (msec) : 250=42.11% cpu : usr=0.23%, sys=2.07%, ctx=3101, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122127: Tue Jun 9 19:28:14 2026 read: IOPS=79, BW=317KiB/s (324kB/s)(4984KiB/15728msec) slat (usec): min=28, max=5117, avg=1230.36, stdev=423.78 clat (usec): min=1260, max=199624, avg=65104.55, stdev=13511.64 lat (usec): min=1296, max=200844, avg=66336.31, stdev=13573.40 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 93], 99.50th=[ 95], 99.90th=[ 199], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 168, max= 440, per=24.98%, avg=318.00, stdev=68.60, samples=31 iops : min= 42, max= 110, avg=79.45, stdev=17.15, samples=31 write: IOPS=83, BW=334KiB/s (342kB/s)(5256KiB/15728msec); 0 zone resets slat (usec): min=29, max=435, avg=94.72, stdev=53.17 clat (msec): min=13, max=257, avg=128.35, stdev=34.04 lat (msec): min=13, max=257, avg=128.44, stdev=34.04 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 78], 10.00th=[ 87], 20.00th=[ 100], | 30.00th=[ 111], 40.00th=[ 121], 50.00th=[ 128], 60.00th=[ 136], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 188], | 99.00th=[ 220], 99.50th=[ 228], 99.90th=[ 255], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 224, max= 416, per=25.10%, avg=332.52, stdev=45.52, samples=31 iops : min= 56, max= 104, avg=83.10, stdev=11.37, samples=31 lat (msec) : 2=0.12%, 4=0.08%, 10=0.04%, 20=0.23%, 50=4.80% lat (msec) : 100=53.67%, 250=40.98%, 500=0.08% cpu : usr=0.25%, sys=2.00%, ctx=3262, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122128: Tue Jun 9 19:28:14 2026 read: IOPS=80, BW=321KiB/s (329kB/s)(5060KiB/15740msec) slat (usec): min=621, max=5759, avg=1238.64, stdev=434.47 clat (msec): min=8, max=186, avg=65.25, stdev=13.12 lat (msec): min=10, max=187, avg=66.49, stdev=13.16 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 80], 95.00th=[ 86], | 99.00th=[ 94], 99.50th=[ 102], 99.90th=[ 186], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 200, max= 448, per=25.35%, avg=322.65, stdev=58.25, samples=31 iops : min= 50, max= 112, avg=80.61, stdev=14.58, samples=31 write: IOPS=82, BW=329KiB/s (337kB/s)(5180KiB/15740msec); 0 zone resets slat (usec): min=20, max=2657, avg=99.65, stdev=92.83 clat (msec): min=40, max=268, avg=129.19, stdev=33.99 lat (msec): min=40, max=268, avg=129.29, stdev=33.99 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 79], 10.00th=[ 89], 20.00th=[ 100], | 30.00th=[ 111], 40.00th=[ 121], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 222], 99.50th=[ 257], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 224, max= 416, per=24.77%, avg=328.16, stdev=44.17, samples=31 iops : min= 56, max= 104, avg=82.00, stdev=11.02, samples=31 lat (msec) : 10=0.04%, 20=0.23%, 50=4.84%, 100=54.53%, 250=40.08% lat (msec) : 500=0.27% cpu : usr=0.27%, sys=2.01%, ctx=3387, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1273KiB/s (1304kB/s), 316KiB/s-321KiB/s (323kB/s-329kB/s), io=19.6MiB (20.5MB), run=15728-15764msec WRITE: bw=1325KiB/s (1357kB/s), 329KiB/s-334KiB/s (337kB/s-342kB/s), io=20.4MiB (21.4MB), run=15728-15764msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=122134: Tue Jun 9 19:28:17 2026 read: IOPS=12, BW=482MiB/s (505MB/s)(40.0MiB/83msec) slat (nsec): min=13426k, max=13426k, avg=13426147.00, stdev= 0.00 clat (nsec): min=65776k, max=65776k, avg=65775622.00, stdev= 0.00 lat (nsec): min=79206k, max=79206k, avg=79205550.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[65799], 5.00th=[65799], 10.00th=[65799], 20.00th=[65799], | 30.00th=[65799], 40.00th=[65799], 50.00th=[65799], 60.00th=[65799], | 70.00th=[65799], 80.00th=[65799], 90.00th=[65799], 95.00th=[65799], | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=15.85%, ctx=2, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=482MiB/s (505MB/s), 482MiB/s-482MiB/s (505MB/s-505MB/s), io=40.0MiB (41.9MB), run=83-83msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck