== sanity test 398c: run fio to test AIO ================= 13:19:36 (1788628776) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.509572 s, 82.3 MB/s osc.lustre-OST0000-osc-ffff8c8417294800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179928: Sat Sep 5 13:20:16 2026 write: IOPS=89, BW=357KiB/s (365kB/s)(10.0MiB/28694msec); 0 zone resets slat (usec): min=20, max=21780, avg=197.77, stdev=889.01 clat (msec): min=13, max=351, avg=178.14, stdev=49.34 lat (msec): min=13, max=351, avg=178.34, stdev=49.36 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 120], 10.00th=[ 126], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 199], 80.00th=[ 218], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 220, max= 512, per=24.99%, avg=356.59, stdev=76.61, samples=56 iops : min= 55, max= 128, avg=88.75, stdev=19.23, samples=56 lat (msec) : 20=0.12%, 50=0.08%, 100=0.74%, 250=89.18%, 500=9.88% cpu : usr=0.17%, sys=0.95%, ctx=1655, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179929: Sat Sep 5 13:20:16 2026 write: IOPS=89, BW=357KiB/s (365kB/s)(10.0MiB/28695msec); 0 zone resets slat (usec): min=21, max=18228, avg=184.31, stdev=798.70 clat (msec): min=25, max=351, avg=178.62, stdev=49.41 lat (msec): min=25, max=351, avg=178.81, stdev=49.40 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 220], 90.00th=[ 251], 95.00th=[ 275], | 99.00th=[ 313], 99.50th=[ 317], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 351] bw ( KiB/s): min= 173, max= 510, per=24.97%, avg=356.34, stdev=81.30, samples=56 iops : min= 43, max= 127, avg=88.68, stdev=20.39, samples=56 lat (msec) : 50=0.04%, 100=1.99%, 250=87.93%, 500=10.04% cpu : usr=0.11%, sys=0.76%, ctx=1690, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179930: Sat Sep 5 13:20:16 2026 write: IOPS=89, BW=358KiB/s (366kB/s)(10.0MiB/28642msec); 0 zone resets slat (usec): min=21, max=13933, avg=183.48, stdev=689.34 clat (msec): min=11, max=354, avg=178.59, stdev=50.11 lat (msec): min=12, max=354, avg=178.77, stdev=50.08 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 115], 10.00th=[ 125], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 220], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 334], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 165, max= 515, per=25.00%, avg=356.70, stdev=81.10, samples=56 iops : min= 41, max= 128, avg=88.75, stdev=20.32, samples=56 lat (msec) : 20=0.04%, 50=0.16%, 100=1.84%, 250=86.72%, 500=11.25% cpu : usr=0.17%, sys=0.87%, ctx=1820, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179931: Sat Sep 5 13:20:16 2026 write: IOPS=89, BW=357KiB/s (365kB/s)(10.0MiB/28692msec); 0 zone resets slat (usec): min=21, max=17064, avg=210.22, stdev=795.48 clat (msec): min=34, max=350, avg=178.13, stdev=50.58 lat (msec): min=34, max=350, avg=178.34, stdev=50.58 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 116], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 215], 90.00th=[ 253], 95.00th=[ 279], | 99.00th=[ 334], 99.50th=[ 338], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 351] bw ( KiB/s): min= 198, max= 510, per=24.91%, avg=355.45, stdev=75.81, samples=56 iops : min= 49, max= 127, avg=88.45, stdev=18.99, samples=56 lat (msec) : 50=0.23%, 100=1.21%, 250=87.97%, 500=10.59% cpu : usr=0.22%, sys=0.98%, ctx=2659, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1427KiB/s (1462kB/s), 357KiB/s-358KiB/s (365kB/s-366kB/s), io=40.0MiB (41.9MB), run=28642-28695msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179946: Sat Sep 5 13:21:00 2026 read: IOPS=37, BW=151KiB/s (154kB/s)(5048KiB/33486msec) slat (usec): min=2875, max=59755, avg=14376.97, stdev=8111.81 clat (msec): min=36, max=361, avg=196.45, stdev=46.99 lat (msec): min=59, max=395, avg=210.83, stdev=49.24 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 126], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 46, max= 270, per=25.15%, avg=150.12, stdev=37.99, samples=66 iops : min= 11, max= 67, avg=37.06, stdev= 9.55, samples=66 write: IOPS=38, BW=155KiB/s (159kB/s)(5192KiB/33486msec); 0 zone resets slat (usec): min=32, max=13957, avg=207.53, stdev=731.41 clat (msec): min=82, max=341, avg=204.49, stdev=44.28 lat (msec): min=83, max=341, avg=204.70, stdev=44.27 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 169], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 264], 95.00th=[ 288], | 99.00th=[ 321], 99.50th=[ 330], 99.90th=[ 334], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 38, max= 257, per=24.94%, avg=154.86, stdev=39.28, samples=66 iops : min= 9, max= 64, avg=38.26, stdev= 9.83, samples=66 lat (msec) : 50=0.04%, 100=0.43%, 250=85.00%, 500=14.53% cpu : usr=0.25%, sys=1.51%, ctx=3881, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179947: Sat Sep 5 13:21:00 2026 read: IOPS=37, BW=148KiB/s (152kB/s)(4976KiB/33614msec) slat (msec): min=2, max=220, avg=14.77, stdev=10.04 clat (msec): min=24, max=390, avg=198.64, stdev=50.34 lat (msec): min=31, max=409, avg=213.42, stdev=52.23 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 128], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 351], 99.90th=[ 368], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 62, max= 218, per=24.54%, avg=146.48, stdev=34.77, samples=66 iops : min= 15, max= 54, avg=36.15, stdev= 8.72, samples=66 write: IOPS=39, BW=157KiB/s (160kB/s)(5264KiB/33614msec); 0 zone resets slat (usec): min=25, max=16806, avg=215.97, stdev=856.18 clat (msec): min=24, max=402, avg=204.64, stdev=47.57 lat (msec): min=24, max=402, avg=204.86, stdev=47.59 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 134], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 264], 95.00th=[ 288], | 99.00th=[ 351], 99.50th=[ 376], 99.90th=[ 393], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 72, max= 254, per=24.99%, avg=155.21, stdev=38.07, samples=66 iops : min= 18, max= 63, avg=38.33, stdev= 9.56, samples=66 lat (msec) : 50=0.20%, 100=0.70%, 250=84.69%, 500=14.41% cpu : usr=0.31%, sys=1.62%, ctx=4432, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179948: Sat Sep 5 13:21:00 2026 read: IOPS=37, BW=149KiB/s (152kB/s)(4984KiB/33544msec) slat (usec): min=2868, max=68334, avg=14299.34, stdev=7889.38 clat (msec): min=82, max=362, avg=198.26, stdev=46.05 lat (msec): min=93, max=383, avg=212.57, stdev=48.30 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 321], 99.50th=[ 338], 99.90th=[ 351], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 54, max= 247, per=24.79%, avg=148.00, stdev=38.62, samples=66 iops : min= 13, max= 61, avg=36.52, stdev= 9.68, samples=66 write: IOPS=39, BW=157KiB/s (160kB/s)(5256KiB/33544msec); 0 zone resets slat (usec): min=29, max=28998, avg=244.26, stdev=1195.58 clat (msec): min=49, max=366, avg=203.53, stdev=44.37 lat (msec): min=49, max=366, avg=203.78, stdev=44.36 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 264], 95.00th=[ 288], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 351], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 62, max= 272, per=25.03%, avg=155.45, stdev=39.29, samples=66 iops : min= 15, max= 68, avg=38.39, stdev= 9.85, samples=66 lat (msec) : 50=0.04%, 100=0.35%, 250=85.94%, 500=13.67% cpu : usr=0.25%, sys=1.61%, ctx=4378, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179949: Sat Sep 5 13:21:00 2026 read: IOPS=37, BW=152KiB/s (155kB/s)(5060KiB/33369msec) slat (usec): min=37, max=54137, avg=14424.14, stdev=8059.61 clat (msec): min=21, max=374, avg=196.98, stdev=46.47 lat (msec): min=21, max=383, avg=211.41, stdev=48.56 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 128], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 234], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 317], 99.50th=[ 338], 99.90th=[ 363], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 71, max= 240, per=25.33%, avg=151.23, stdev=34.55, samples=66 iops : min= 17, max= 60, avg=37.35, stdev= 8.69, samples=66 write: IOPS=38, BW=155KiB/s (159kB/s)(5180KiB/33369msec); 0 zone resets slat (usec): min=25, max=9267, avg=186.05, stdev=558.20 clat (msec): min=65, max=379, avg=205.07, stdev=44.57 lat (msec): min=65, max=379, avg=205.27, stdev=44.57 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 334], 99.50th=[ 368], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 71, max= 248, per=24.95%, avg=154.92, stdev=39.20, samples=66 iops : min= 17, max= 62, avg=38.27, stdev= 9.81, samples=66 lat (msec) : 50=0.08%, 100=0.82%, 250=85.74%, 500=13.36% cpu : usr=0.24%, sys=1.69%, ctx=4152, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=597KiB/s (611kB/s), 148KiB/s-152KiB/s (152kB/s-155kB/s), io=19.6MiB (20.5MB), run=33369-33614msec WRITE: bw=622KiB/s (636kB/s), 155KiB/s-157KiB/s (159kB/s-160kB/s), io=20.4MiB (21.4MB), run=33369-33614msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179960: Sat Sep 5 13:21:11 2026 read: IOPS=1, BW=67.0MiB/s (70.3MB/s)(40.0MiB/597msec) slat (nsec): min=40442k, max=40442k, avg=40441505.00, stdev= 0.00 clat (nsec): min=512742k, max=512742k, avg=512742221.00, stdev= 0.00 lat (nsec): min=553188k, max=553188k, avg=553188381.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.18%, ctx=5, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.0MiB/s (70.3MB/s), 67.0MiB/s-67.0MiB/s (70.3MB/s-70.3MB/s), io=40.0MiB (41.9MB), run=597-597msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout