== sanity test 398c: run fio to test AIO ================= 20:40:25 (1777423225) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.23118 s, 181 MB/s osc.lustre-OST0000-osc-ffff8f1538eaf000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154386: Tue Apr 28 20:41:41 2026 write: IOPS=37, BW=149KiB/s (153kB/s)(10.0MiB/68552msec); 0 zone resets slat (usec): min=39, max=10801, avg=285.80, stdev=544.50 clat (msec): min=66, max=1663, avg=427.90, stdev=196.47 lat (msec): min=72, max=1664, avg=428.19, stdev=196.48 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 232], 10.00th=[ 259], 20.00th=[ 305], | 30.00th=[ 338], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 447], 80.00th=[ 493], 90.00th=[ 584], 95.00th=[ 735], | 99.00th=[ 1301], 99.50th=[ 1418], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 296, per=25.00%, avg=149.02, stdev=55.73, samples=136 iops : min= 1, max= 74, avg=36.97, stdev=13.99, samples=136 lat (msec) : 100=0.08%, 250=8.28%, 500=73.44%, 750=13.40%, 1000=1.72% lat (msec) : 2000=3.09% cpu : usr=0.09%, sys=0.69%, ctx=3036, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154387: Tue Apr 28 20:41:41 2026 write: IOPS=37, BW=149KiB/s (153kB/s)(10.0MiB/68634msec); 0 zone resets slat (usec): min=40, max=15859, avg=279.31, stdev=553.79 clat (msec): min=135, max=1553, avg=428.29, stdev=193.88 lat (msec): min=136, max=1556, avg=428.57, stdev=193.92 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 236], 10.00th=[ 266], 20.00th=[ 300], | 30.00th=[ 334], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 493], 90.00th=[ 584], 95.00th=[ 760], | 99.00th=[ 1318], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1502], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 304, per=25.21%, avg=150.27, stdev=56.37, samples=135 iops : min= 1, max= 76, avg=37.29, stdev=14.17, samples=135 lat (msec) : 250=7.23%, 500=73.79%, 750=13.91%, 1000=1.64%, 2000=3.44% cpu : usr=0.08%, sys=0.69%, ctx=2930, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154388: Tue Apr 28 20:41:41 2026 write: IOPS=37, BW=149KiB/s (153kB/s)(10.0MiB/68547msec); 0 zone resets slat (usec): min=39, max=6081, avg=269.12, stdev=359.39 clat (msec): min=177, max=1637, avg=427.28, stdev=190.87 lat (msec): min=177, max=1637, avg=427.55, stdev=190.86 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 232], 10.00th=[ 262], 20.00th=[ 305], | 30.00th=[ 334], 40.00th=[ 368], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 451], 80.00th=[ 498], 90.00th=[ 575], 95.00th=[ 743], | 99.00th=[ 1284], 99.50th=[ 1368], 99.90th=[ 1485], 99.95th=[ 1502], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 296, per=25.03%, avg=149.15, stdev=56.48, samples=136 iops : min= 1, max= 74, avg=37.01, stdev=14.19, samples=136 lat (msec) : 250=7.85%, 500=72.77%, 750=14.38%, 1000=1.84%, 2000=3.16% cpu : usr=0.09%, sys=0.69%, ctx=2873, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154389: Tue Apr 28 20:41:41 2026 write: IOPS=37, BW=149KiB/s (153kB/s)(10.0MiB/68633msec); 0 zone resets slat (usec): min=41, max=4692, avg=272.02, stdev=350.31 clat (msec): min=131, max=1581, avg=427.80, stdev=195.54 lat (msec): min=131, max=1581, avg=428.07, stdev=195.55 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 234], 10.00th=[ 257], 20.00th=[ 305], | 30.00th=[ 338], 40.00th=[ 368], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 493], 90.00th=[ 584], 95.00th=[ 726], | 99.00th=[ 1301], 99.50th=[ 1452], 99.90th=[ 1536], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 288, per=25.00%, avg=148.98, stdev=58.89, samples=136 iops : min= 1, max= 72, avg=36.96, stdev=14.79, samples=136 lat (msec) : 250=8.44%, 500=72.58%, 750=14.10%, 1000=1.17%, 2000=3.71% cpu : usr=0.09%, sys=0.70%, ctx=2975, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=597KiB/s (611kB/s), 149KiB/s-149KiB/s (153kB/s-153kB/s), io=40.0MiB (41.9MB), run=68547-68634msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154408: Tue Apr 28 20:42:16 2026 read: IOPS=40, BW=162KiB/s (165kB/s)(5048KiB/31249msec) slat (usec): min=1294, max=16518, avg=4281.67, stdev=2281.01 clat (msec): min=3, max=293, avg=123.81, stdev=41.62 lat (msec): min=10, max=298, avg=128.09, stdev=42.56 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 71], 10.00th=[ 79], 20.00th=[ 88], | 30.00th=[ 96], 40.00th=[ 107], 50.00th=[ 118], 60.00th=[ 130], | 70.00th=[ 144], 80.00th=[ 159], 90.00th=[ 182], 95.00th=[ 201], | 99.00th=[ 234], 99.50th=[ 249], 99.90th=[ 268], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 48, max= 304, per=25.06%, avg=160.90, stdev=60.99, samples=62 iops : min= 12, max= 76, avg=39.97, stdev=15.29, samples=62 write: IOPS=41, BW=166KiB/s (170kB/s)(5192KiB/31249msec); 0 zone resets slat (usec): min=60, max=3866, avg=250.59, stdev=245.54 clat (msec): min=99, max=539, avg=259.29, stdev=87.19 lat (msec): min=100, max=539, avg=259.54, stdev=87.21 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 132], 10.00th=[ 153], 20.00th=[ 180], | 30.00th=[ 201], 40.00th=[ 222], 50.00th=[ 253], 60.00th=[ 279], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 380], 95.00th=[ 414], | 99.00th=[ 477], 99.50th=[ 506], 99.90th=[ 535], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 79, max= 288, per=24.81%, avg=165.73, stdev=47.95, samples=62 iops : min= 19, max= 72, avg=41.18, stdev=12.09, samples=62 lat (msec) : 4=0.04%, 10=0.04%, 20=0.31%, 50=0.20%, 100=16.29% lat (msec) : 250=57.27%, 500=25.51%, 750=0.35% cpu : usr=0.19%, sys=1.92%, ctx=4971, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154409: Tue Apr 28 20:42:16 2026 read: IOPS=39, BW=159KiB/s (163kB/s)(4976KiB/31249msec) slat (usec): min=1194, max=19078, avg=4172.10, stdev=2230.67 clat (msec): min=5, max=294, avg=123.47, stdev=39.86 lat (msec): min=10, max=299, avg=127.65, stdev=40.78 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 73], 10.00th=[ 80], 20.00th=[ 89], | 30.00th=[ 97], 40.00th=[ 108], 50.00th=[ 117], 60.00th=[ 129], | 70.00th=[ 142], 80.00th=[ 157], 90.00th=[ 178], 95.00th=[ 194], | 99.00th=[ 230], 99.50th=[ 257], 99.90th=[ 275], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 55, max= 352, per=24.72%, avg=158.71, stdev=59.55, samples=62 iops : min= 13, max= 88, avg=39.42, stdev=14.95, samples=62 write: IOPS=42, BW=168KiB/s (172kB/s)(5264KiB/31249msec); 0 zone resets slat (usec): min=55, max=5517, avg=247.62, stdev=254.19 clat (msec): min=82, max=534, avg=258.33, stdev=87.43 lat (msec): min=82, max=534, avg=258.57, stdev=87.45 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 125], 10.00th=[ 155], 20.00th=[ 180], | 30.00th=[ 201], 40.00th=[ 222], 50.00th=[ 249], 60.00th=[ 279], | 70.00th=[ 305], 80.00th=[ 338], 90.00th=[ 384], 95.00th=[ 409], | 99.00th=[ 472], 99.50th=[ 493], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 71, max= 272, per=25.11%, avg=167.74, stdev=49.75, samples=62 iops : min= 17, max= 68, avg=41.68, stdev=12.55, samples=62 lat (msec) : 10=0.12%, 20=0.08%, 50=0.16%, 100=15.82%, 250=58.20% lat (msec) : 500=25.39%, 750=0.23% cpu : usr=0.21%, sys=1.83%, ctx=4851, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154410: Tue Apr 28 20:42:16 2026 read: IOPS=39, BW=160KiB/s (163kB/s)(4984KiB/31236msec) slat (usec): min=38, max=17231, avg=4195.62, stdev=2220.84 clat (msec): min=13, max=303, avg=124.66, stdev=43.00 lat (msec): min=16, max=307, avg=128.86, stdev=43.99 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 70], 10.00th=[ 79], 20.00th=[ 89], | 30.00th=[ 97], 40.00th=[ 109], 50.00th=[ 118], 60.00th=[ 130], | 70.00th=[ 146], 80.00th=[ 161], 90.00th=[ 184], 95.00th=[ 205], | 99.00th=[ 236], 99.50th=[ 262], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 56, max= 312, per=24.79%, avg=159.18, stdev=55.91, samples=62 iops : min= 14, max= 78, avg=39.53, stdev=14.06, samples=62 write: IOPS=42, BW=168KiB/s (172kB/s)(5256KiB/31236msec); 0 zone resets slat (usec): min=41, max=3442, avg=244.28, stdev=221.06 clat (msec): min=61, max=538, avg=257.60, stdev=88.27 lat (msec): min=62, max=539, avg=257.85, stdev=88.30 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 129], 10.00th=[ 150], 20.00th=[ 180], | 30.00th=[ 201], 40.00th=[ 220], 50.00th=[ 245], 60.00th=[ 275], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 384], 95.00th=[ 409], | 99.00th=[ 477], 99.50th=[ 498], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 87, max= 320, per=24.99%, avg=166.95, stdev=54.49, samples=62 iops : min= 21, max= 80, avg=41.48, stdev=13.71, samples=62 lat (msec) : 20=0.31%, 50=0.47%, 100=15.62%, 250=58.44%, 500=24.96% lat (msec) : 750=0.20% cpu : usr=0.21%, sys=1.86%, ctx=4774, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154411: Tue Apr 28 20:42:16 2026 read: IOPS=40, BW=162KiB/s (166kB/s)(5060KiB/31249msec) slat (usec): min=1129, max=19128, avg=4275.63, stdev=2190.27 clat (msec): min=12, max=303, avg=125.35, stdev=41.08 lat (msec): min=14, max=307, avg=129.63, stdev=42.04 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 73], 10.00th=[ 80], 20.00th=[ 89], | 30.00th=[ 100], 40.00th=[ 109], 50.00th=[ 121], 60.00th=[ 130], | 70.00th=[ 144], 80.00th=[ 157], 90.00th=[ 180], 95.00th=[ 203], | 99.00th=[ 251], 99.50th=[ 262], 99.90th=[ 296], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 71, max= 320, per=25.19%, avg=161.69, stdev=56.06, samples=62 iops : min= 17, max= 80, avg=40.16, stdev=14.12, samples=62 write: IOPS=41, BW=166KiB/s (170kB/s)(5180KiB/31249msec); 0 zone resets slat (usec): min=56, max=5186, avg=242.39, stdev=258.51 clat (msec): min=72, max=559, avg=258.58, stdev=89.44 lat (msec): min=73, max=559, avg=258.82, stdev=89.45 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 130], 10.00th=[ 153], 20.00th=[ 180], | 30.00th=[ 199], 40.00th=[ 220], 50.00th=[ 245], 60.00th=[ 275], | 70.00th=[ 305], 80.00th=[ 342], 90.00th=[ 384], 95.00th=[ 414], | 99.00th=[ 485], 99.50th=[ 493], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 80, max= 288, per=24.65%, avg=164.69, stdev=52.95, samples=62 iops : min= 20, max= 72, avg=40.92, stdev=13.31, samples=62 lat (msec) : 20=0.12%, 50=0.20%, 100=15.70%, 250=59.57%, 500=24.22% lat (msec) : 750=0.20% cpu : usr=0.20%, sys=1.83%, ctx=4908, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=642KiB/s (658kB/s), 159KiB/s-162KiB/s (163kB/s-166kB/s), io=19.6MiB (20.5MB), run=31236-31249msec WRITE: bw=669KiB/s (685kB/s), 166KiB/s-168KiB/s (170kB/s-172kB/s), io=20.4MiB (21.4MB), run=31236-31249msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154419: Tue Apr 28 20:42:20 2026 read: IOPS=7, BW=299MiB/s (313MB/s)(40.0MiB/134msec) slat (nsec): min=10172k, max=10172k, avg=10172133.00, stdev= 0.00 clat (nsec): min=118697k, max=118697k, avg=118696541.00, stdev= 0.00 lat (nsec): min=128873k, max=128873k, avg=128872736.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.02%, ctx=41, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=299MiB/s (313MB/s), 299MiB/s-299MiB/s (313MB/s-313MB/s), io=40.0MiB (41.9MB), run=134-134msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout