== sanity test 398c: run fio to test AIO ================= 22:09:59 (1763348999) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.516118 s, 81.3 MB/s osc.lustre-OST0000-osc-ffff8ba309cab800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183468: Sun Nov 16 22:10:34 2025 write: IOPS=101, BW=407KiB/s (417kB/s)(10.0MiB/25160msec); 0 zone resets slat (usec): min=28, max=8202, avg=150.13, stdev=412.89 clat (msec): min=25, max=341, avg=156.95, stdev=38.55 lat (msec): min=26, max=341, avg=157.10, stdev=38.54 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 106], 10.00th=[ 113], 20.00th=[ 128], | 30.00th=[ 140], 40.00th=[ 146], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 201], 95.00th=[ 226], | 99.00th=[ 279], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 291, max= 595, per=25.15%, avg=408.47, stdev=74.37, samples=49 iops : min= 72, max= 148, avg=101.71, stdev=18.62, samples=49 lat (msec) : 50=0.31%, 100=2.03%, 250=95.31%, 500=2.34% cpu : usr=0.15%, sys=1.03%, ctx=2528, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183469: Sun Nov 16 22:10:34 2025 write: IOPS=101, BW=407KiB/s (416kB/s)(10.0MiB/25190msec); 0 zone resets slat (usec): min=27, max=5768, avg=150.72, stdev=353.56 clat (msec): min=19, max=334, avg=156.79, stdev=36.50 lat (msec): min=20, max=334, avg=156.95, stdev=36.50 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 130], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 182], 90.00th=[ 203], 95.00th=[ 226], | 99.00th=[ 266], 99.50th=[ 279], 99.90th=[ 305], 99.95th=[ 317], | 99.99th=[ 334] bw ( KiB/s): min= 277, max= 552, per=25.05%, avg=406.84, stdev=64.63, samples=49 iops : min= 69, max= 138, avg=101.33, stdev=16.10, samples=49 lat (msec) : 20=0.04%, 50=0.27%, 100=2.58%, 250=94.88%, 500=2.23% cpu : usr=0.16%, sys=1.07%, ctx=2788, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183470: Sun Nov 16 22:10:34 2025 write: IOPS=101, BW=407KiB/s (416kB/s)(10.0MiB/25188msec); 0 zone resets slat (usec): min=28, max=9023, avg=158.14, stdev=461.60 clat (msec): min=26, max=305, avg=156.68, stdev=35.16 lat (msec): min=26, max=306, avg=156.84, stdev=35.16 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 109], 10.00th=[ 118], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 203], 95.00th=[ 215], | 99.00th=[ 271], 99.50th=[ 296], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 252, max= 518, per=25.08%, avg=407.27, stdev=70.83, samples=49 iops : min= 63, max= 129, avg=101.45, stdev=17.63, samples=49 lat (msec) : 50=0.23%, 100=1.48%, 250=95.90%, 500=2.38% cpu : usr=0.17%, sys=1.04%, ctx=2366, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183471: Sun Nov 16 22:10:34 2025 write: IOPS=101, BW=406KiB/s (416kB/s)(10.0MiB/25221msec); 0 zone resets slat (usec): min=27, max=13672, avg=160.30, stdev=555.68 clat (msec): min=28, max=304, avg=156.86, stdev=35.86 lat (msec): min=29, max=306, avg=157.02, stdev=35.88 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 109], 10.00th=[ 118], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 199], 95.00th=[ 228], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 518, per=24.99%, avg=405.80, stdev=71.92, samples=49 iops : min= 64, max= 129, avg=101.08, stdev=17.91, samples=49 lat (msec) : 50=0.04%, 100=1.68%, 250=95.47%, 500=2.81% cpu : usr=0.16%, sys=1.05%, ctx=2840, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1624KiB/s (1663kB/s), 406KiB/s-407KiB/s (416kB/s-417kB/s), io=40.0MiB (41.9MB), run=25160-25221msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183485: Sun Nov 16 22:11:10 2025 read: IOPS=46, BW=185KiB/s (189kB/s)(5048KiB/27343msec) slat (usec): min=36, max=83053, avg=11373.22, stdev=5640.44 clat (msec): min=30, max=356, avg=160.77, stdev=44.11 lat (msec): min=30, max=374, avg=172.15, stdev=46.10 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 101], 10.00th=[ 111], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 167], | 70.00th=[ 180], 80.00th=[ 194], 90.00th=[ 224], 95.00th=[ 247], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 300], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 100, max= 294, per=25.14%, avg=183.78, stdev=43.48, samples=54 iops : min= 25, max= 73, avg=45.50, stdev=10.92, samples=54 write: IOPS=47, BW=190KiB/s (194kB/s)(5192KiB/27343msec); 0 zone resets slat (usec): min=35, max=7257, avg=194.27, stdev=469.32 clat (msec): min=62, max=341, avg=168.23, stdev=41.90 lat (msec): min=62, max=341, avg=168.43, stdev=41.95 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 109], 10.00th=[ 120], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 251], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 330], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 102, max= 286, per=24.89%, avg=189.41, stdev=40.47, samples=54 iops : min= 25, max= 71, avg=46.91, stdev=10.11, samples=54 lat (msec) : 50=0.16%, 100=3.55%, 250=91.91%, 500=4.38% cpu : usr=0.22%, sys=1.81%, ctx=4575, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183486: Sun Nov 16 22:11:10 2025 read: IOPS=45, BW=182KiB/s (187kB/s)(4976KiB/27307msec) slat (usec): min=28, max=77541, avg=11252.17, stdev=5935.98 clat (msec): min=55, max=325, avg=161.28, stdev=41.70 lat (msec): min=64, max=335, avg=172.53, stdev=43.86 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 106], 10.00th=[ 114], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 157], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 192], 90.00th=[ 222], 95.00th=[ 243], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 300], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 87, max= 287, per=24.73%, avg=180.74, stdev=41.59, samples=54 iops : min= 21, max= 71, avg=44.70, stdev=10.49, samples=54 write: IOPS=48, BW=193KiB/s (197kB/s)(5264KiB/27307msec); 0 zone resets slat (usec): min=32, max=12828, avg=211.52, stdev=643.01 clat (msec): min=68, max=313, avg=168.32, stdev=43.33 lat (msec): min=68, max=313, avg=168.53, stdev=43.38 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 106], 10.00th=[ 116], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 228], 95.00th=[ 253], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 309], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 95, max= 334, per=25.26%, avg=192.26, stdev=49.80, samples=54 iops : min= 23, max= 83, avg=47.61, stdev=12.46, samples=54 lat (msec) : 100=2.85%, 250=92.58%, 500=4.57% cpu : usr=0.24%, sys=1.74%, ctx=4662, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183487: Sun Nov 16 22:11:10 2025 read: IOPS=45, BW=182KiB/s (186kB/s)(4984KiB/27374msec) slat (usec): min=39, max=59991, avg=11492.21, stdev=5320.41 clat (msec): min=38, max=318, avg=161.34, stdev=43.98 lat (msec): min=47, max=330, avg=172.83, stdev=45.81 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 104], 10.00th=[ 112], 20.00th=[ 123], | 30.00th=[ 133], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 167], | 70.00th=[ 180], 80.00th=[ 199], 90.00th=[ 226], 95.00th=[ 245], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 103, max= 304, per=24.88%, avg=181.87, stdev=47.04, samples=54 iops : min= 25, max= 76, avg=45.00, stdev=11.82, samples=54 write: IOPS=48, BW=192KiB/s (197kB/s)(5256KiB/27374msec); 0 zone resets slat (usec): min=29, max=27539, avg=214.26, stdev=967.68 clat (msec): min=50, max=302, avg=167.90, stdev=40.35 lat (msec): min=50, max=302, avg=168.12, stdev=40.42 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 108], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 292], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 111, max= 264, per=25.15%, avg=191.37, stdev=40.40, samples=54 iops : min= 27, max= 66, avg=47.37, stdev=10.12, samples=54 lat (msec) : 50=0.04%, 100=3.32%, 250=92.30%, 500=4.34% cpu : usr=0.28%, sys=1.73%, ctx=4285, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183488: Sun Nov 16 22:11:10 2025 read: IOPS=46, BW=184KiB/s (189kB/s)(5060KiB/27430msec) slat (usec): min=2716, max=41246, avg=11521.05, stdev=5431.06 clat (msec): min=7, max=308, avg=161.14, stdev=44.60 lat (msec): min=14, max=322, avg=172.67, stdev=46.60 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 101], 10.00th=[ 112], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 178], 80.00th=[ 194], 90.00th=[ 226], 95.00th=[ 249], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 79, max= 280, per=25.07%, avg=183.24, stdev=46.40, samples=54 iops : min= 19, max= 70, avg=45.35, stdev=11.64, samples=54 write: IOPS=47, BW=189KiB/s (193kB/s)(5180KiB/27430msec); 0 zone resets slat (usec): min=31, max=6159, avg=187.73, stdev=423.03 clat (msec): min=30, max=331, avg=168.22, stdev=44.03 lat (msec): min=30, max=332, avg=168.41, stdev=44.05 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 109], 10.00th=[ 120], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 257], | 99.00th=[ 305], 99.50th=[ 309], 99.90th=[ 330], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 100, max= 272, per=24.60%, avg=187.22, stdev=41.48, samples=54 iops : min= 25, max= 68, avg=46.37, stdev=10.38, samples=54 lat (msec) : 10=0.04%, 20=0.04%, 50=0.35%, 100=3.63%, 250=90.74% lat (msec) : 500=5.20% cpu : usr=0.28%, sys=1.82%, ctx=4548, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=732KiB/s (749kB/s), 182KiB/s-185KiB/s (186kB/s-189kB/s), io=19.6MiB (20.5MB), run=27307-27430msec WRITE: bw=762KiB/s (780kB/s), 189KiB/s-193KiB/s (193kB/s-197kB/s), io=20.4MiB (21.4MB), run=27307-27430msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183496: Sun Nov 16 22:11:19 2025 read: IOPS=2, BW=88.7MiB/s (93.0MB/s)(40.0MiB/451msec) slat (nsec): min=29708k, max=29708k, avg=29707817.00, stdev= 0.00 clat (nsec): min=398282k, max=398282k, avg=398282113.00, stdev= 0.00 lat (nsec): min=427995k, max=427995k, avg=427995054.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.11%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=88.7MiB/s (93.0MB/s), 88.7MiB/s-88.7MiB/s (93.0MB/s-93.0MB/s), io=40.0MiB (41.9MB), run=451-451msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout