== sanity test 398c: run fio to test AIO ================= 21:50:25 (1763347825) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.301395 s, 139 MB/s osc.lustre-OST0000-osc-ffff8f68c2bd8000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182163: Sun Nov 16 21:50:55 2025 write: IOPS=112, BW=448KiB/s (459kB/s)(10.0MiB/22841msec); 0 zone resets slat (usec): min=24, max=10466, avg=151.14, stdev=425.44 clat (msec): min=31, max=231, avg=141.99, stdev=29.21 lat (msec): min=31, max=231, avg=142.14, stdev=29.21 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 95], 10.00th=[ 104], 20.00th=[ 115], | 30.00th=[ 127], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 165], 90.00th=[ 182], 95.00th=[ 194], | 99.00th=[ 222], 99.50th=[ 222], 99.90th=[ 230], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 254, max= 637, per=24.84%, avg=445.38, stdev=80.37, samples=45 iops : min= 63, max= 159, avg=110.98, stdev=20.10, samples=45 lat (msec) : 50=0.04%, 100=6.95%, 250=93.01% cpu : usr=0.19%, sys=1.04%, ctx=1815, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182164: Sun Nov 16 21:50:55 2025 write: IOPS=112, BW=450KiB/s (461kB/s)(10.0MiB/22744msec); 0 zone resets slat (usec): min=22, max=20215, avg=187.53, stdev=588.41 clat (msec): min=39, max=235, avg=141.71, stdev=30.28 lat (msec): min=39, max=235, avg=141.90, stdev=30.29 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 95], 10.00th=[ 103], 20.00th=[ 115], | 30.00th=[ 127], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 157], 80.00th=[ 165], 90.00th=[ 180], 95.00th=[ 192], | 99.00th=[ 218], 99.50th=[ 230], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 236] bw ( KiB/s): min= 262, max= 636, per=24.97%, avg=447.80, stdev=76.00, samples=45 iops : min= 65, max= 159, avg=111.60, stdev=19.01, samples=45 lat (msec) : 50=0.66%, 100=7.89%, 250=91.45% cpu : usr=0.22%, sys=1.33%, ctx=2587, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182165: Sun Nov 16 21:50:55 2025 write: IOPS=112, BW=449KiB/s (460kB/s)(10.0MiB/22805msec); 0 zone resets slat (usec): min=26, max=10939, avg=162.74, stdev=469.05 clat (msec): min=29, max=226, avg=141.86, stdev=29.76 lat (msec): min=30, max=226, avg=142.03, stdev=29.76 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 97], 10.00th=[ 103], 20.00th=[ 115], | 30.00th=[ 127], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 178], 95.00th=[ 190], | 99.00th=[ 220], 99.50th=[ 224], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 228] bw ( KiB/s): min= 255, max= 630, per=24.92%, avg=446.84, stdev=83.18, samples=45 iops : min= 63, max= 157, avg=111.36, stdev=20.80, samples=45 lat (msec) : 50=0.66%, 100=6.76%, 250=92.58% cpu : usr=0.19%, sys=1.07%, ctx=2312, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182166: Sun Nov 16 21:50:55 2025 write: IOPS=112, BW=448KiB/s (459kB/s)(10.0MiB/22839msec); 0 zone resets slat (usec): min=26, max=22307, avg=188.88, stdev=604.52 clat (msec): min=31, max=235, avg=142.04, stdev=29.77 lat (msec): min=31, max=235, avg=142.23, stdev=29.77 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 95], 10.00th=[ 104], 20.00th=[ 115], | 30.00th=[ 127], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 180], 95.00th=[ 197], | 99.00th=[ 222], 99.50th=[ 226], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 270, max= 606, per=24.82%, avg=445.04, stdev=75.26, samples=45 iops : min= 67, max= 151, avg=110.91, stdev=18.79, samples=45 lat (msec) : 50=0.12%, 100=7.23%, 250=92.66% cpu : usr=0.26%, sys=1.23%, ctx=2928, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1793KiB/s (1836kB/s), 448KiB/s-450KiB/s (459kB/s-461kB/s), io=40.0MiB (41.9MB), run=22744-22841msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182179: Sun Nov 16 21:51:24 2025 read: IOPS=51, BW=207KiB/s (212kB/s)(5048KiB/24388msec) slat (usec): min=2583, max=59427, avg=10286.29, stdev=4468.08 clat (msec): min=76, max=278, avg=144.13, stdev=29.53 lat (msec): min=83, max=293, avg=154.43, stdev=30.72 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 103], 10.00th=[ 110], 20.00th=[ 120], | 30.00th=[ 128], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 239], 99.50th=[ 255], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 134, max= 294, per=25.08%, avg=206.19, stdev=38.13, samples=48 iops : min= 33, max= 73, avg=51.06, stdev= 9.57, samples=48 write: IOPS=53, BW=213KiB/s (218kB/s)(5192KiB/24388msec); 0 zone resets slat (usec): min=32, max=2920, avg=160.98, stdev=258.29 clat (msec): min=86, max=273, avg=148.54, stdev=28.57 lat (msec): min=86, max=273, avg=148.70, stdev=28.59 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 109], 10.00th=[ 116], 20.00th=[ 126], | 30.00th=[ 133], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 169], 90.00th=[ 186], 95.00th=[ 203], | 99.00th=[ 243], 99.50th=[ 249], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 112, max= 295, per=24.80%, avg=212.29, stdev=41.56, samples=48 iops : min= 28, max= 73, avg=52.58, stdev=10.32, samples=48 lat (msec) : 100=2.50%, 250=96.91%, 500=0.59% cpu : usr=0.28%, sys=1.94%, ctx=4245, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182180: Sun Nov 16 21:51:24 2025 read: IOPS=50, BW=204KiB/s (209kB/s)(4976KiB/24403msec) slat (usec): min=2273, max=58635, avg=10441.84, stdev=4623.47 clat (msec): min=74, max=276, avg=145.00, stdev=30.01 lat (msec): min=80, max=287, avg=155.44, stdev=31.23 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 103], 10.00th=[ 111], 20.00th=[ 122], | 30.00th=[ 129], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 182], 95.00th=[ 199], | 99.00th=[ 241], 99.50th=[ 259], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 113, max= 327, per=24.66%, avg=202.73, stdev=40.38, samples=48 iops : min= 28, max= 81, avg=50.19, stdev=10.08, samples=48 write: IOPS=53, BW=216KiB/s (221kB/s)(5264KiB/24403msec); 0 zone resets slat (usec): min=33, max=4225, avg=176.87, stdev=331.13 clat (msec): min=32, max=267, avg=148.01, stdev=28.07 lat (msec): min=32, max=269, avg=148.19, stdev=28.08 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 109], 10.00th=[ 116], 20.00th=[ 127], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 169], 90.00th=[ 182], 95.00th=[ 199], | 99.00th=[ 234], 99.50th=[ 255], 99.90th=[ 262], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 104, max= 303, per=25.19%, avg=215.67, stdev=43.41, samples=48 iops : min= 26, max= 75, avg=53.42, stdev=10.81, samples=48 lat (msec) : 50=0.04%, 100=3.16%, 250=96.09%, 500=0.70% cpu : usr=0.31%, sys=1.95%, ctx=4831, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182181: Sun Nov 16 21:51:24 2025 read: IOPS=51, BW=205KiB/s (210kB/s)(4984KiB/24345msec) slat (usec): min=3010, max=55585, avg=10396.65, stdev=4730.29 clat (msec): min=25, max=272, avg=144.10, stdev=30.15 lat (msec): min=40, max=294, avg=154.50, stdev=31.09 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 103], 10.00th=[ 110], 20.00th=[ 120], | 30.00th=[ 128], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 165], 90.00th=[ 182], 95.00th=[ 199], | 99.00th=[ 241], 99.50th=[ 259], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 129, max= 278, per=24.90%, avg=204.69, stdev=32.43, samples=48 iops : min= 32, max= 69, avg=50.69, stdev= 8.11, samples=48 write: IOPS=53, BW=216KiB/s (221kB/s)(5256KiB/24345msec); 0 zone resets slat (usec): min=31, max=4547, avg=170.14, stdev=313.48 clat (msec): min=84, max=276, avg=148.26, stdev=29.15 lat (msec): min=84, max=276, avg=148.43, stdev=29.17 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 107], 10.00th=[ 116], 20.00th=[ 125], | 30.00th=[ 133], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 239], 99.50th=[ 264], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 80, max= 335, per=25.13%, avg=215.08, stdev=46.01, samples=48 iops : min= 20, max= 83, avg=53.29, stdev=11.46, samples=48 lat (msec) : 50=0.04%, 100=3.01%, 250=96.21%, 500=0.74% cpu : usr=0.27%, sys=1.94%, ctx=4629, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182182: Sun Nov 16 21:51:24 2025 read: IOPS=51, BW=207KiB/s (212kB/s)(5060KiB/24397msec) slat (usec): min=38, max=122484, avg=10333.70, stdev=5472.29 clat (msec): min=22, max=270, avg=144.08, stdev=31.61 lat (msec): min=23, max=294, avg=154.42, stdev=32.62 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 101], 10.00th=[ 108], 20.00th=[ 118], | 30.00th=[ 127], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 184], 95.00th=[ 201], | 99.00th=[ 241], 99.50th=[ 255], 99.90th=[ 268], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 96, max= 278, per=25.15%, avg=206.71, stdev=40.28, samples=48 iops : min= 24, max= 69, avg=51.21, stdev=10.03, samples=48 write: IOPS=53, BW=212KiB/s (217kB/s)(5180KiB/24397msec); 0 zone resets slat (usec): min=32, max=11182, avg=205.79, stdev=619.96 clat (msec): min=46, max=263, avg=149.98, stdev=28.44 lat (msec): min=46, max=263, avg=150.18, stdev=28.47 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 110], 10.00th=[ 120], 20.00th=[ 127], | 30.00th=[ 134], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 155], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 234], 99.50th=[ 243], 99.90th=[ 253], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 143, max= 281, per=24.70%, avg=211.46, stdev=37.97, samples=48 iops : min= 35, max= 70, avg=52.37, stdev= 9.56, samples=48 lat (msec) : 50=0.16%, 100=3.36%, 250=95.98%, 500=0.51% cpu : usr=0.19%, sys=2.07%, ctx=4873, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=822KiB/s (842kB/s), 204KiB/s-207KiB/s (209kB/s-212kB/s), io=19.6MiB (20.5MB), run=24345-24403msec WRITE: bw=856KiB/s (877kB/s), 212KiB/s-216KiB/s (217kB/s-221kB/s), io=20.4MiB (21.4MB), run=24345-24403msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182189: Sun Nov 16 21:51:32 2025 read: IOPS=2, BW=114MiB/s (120MB/s)(40.0MiB/350msec) slat (nsec): min=21216k, max=21216k, avg=21215562.00, stdev= 0.00 clat (nsec): min=316573k, max=316573k, avg=316572929.00, stdev= 0.00 lat (nsec): min=337792k, max=337792k, avg=337792169.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.43%, ctx=3, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=114MiB/s (120MB/s), 114MiB/s-114MiB/s (120MB/s-120MB/s), io=40.0MiB (41.9MB), run=350-350msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout