== sanity test 398c: run fio to test AIO ================= 14:46:13 (1788288373) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.528314 s, 79.4 MB/s osc.lustre-OST0000-osc-ffffa0a5528a0800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179292: Tue Sep 1 14:46:50 2026 write: IOPS=95, BW=383KiB/s (392kB/s)(10.0MiB/26740msec); 0 zone resets slat (usec): min=22, max=11725, avg=174.77, stdev=512.30 clat (msec): min=12, max=427, avg=166.18, stdev=49.10 lat (msec): min=12, max=427, avg=166.35, stdev=49.10 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 103], 10.00th=[ 114], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 201], 90.00th=[ 220], 95.00th=[ 249], | 99.00th=[ 342], 99.50th=[ 372], 99.90th=[ 426], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 167, max= 582, per=24.98%, avg=381.45, stdev=84.31, samples=53 iops : min= 41, max= 145, avg=95.04, stdev=21.15, samples=53 lat (msec) : 20=0.12%, 50=0.51%, 100=3.48%, 250=91.09%, 500=4.80% cpu : usr=0.15%, sys=1.07%, ctx=2559, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179293: Tue Sep 1 14:46:50 2026 write: IOPS=96, BW=386KiB/s (395kB/s)(10.0MiB/26517msec); 0 zone resets slat (usec): min=21, max=11667, avg=107.67, stdev=449.47 clat (msec): min=61, max=404, avg=165.38, stdev=48.33 lat (msec): min=62, max=404, avg=165.49, stdev=48.32 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 100], 10.00th=[ 113], 20.00th=[ 127], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 155], 60.00th=[ 169], | 70.00th=[ 186], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 249], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 405] bw ( KiB/s): min= 245, max= 614, per=25.08%, avg=382.92, stdev=103.42, samples=52 iops : min= 61, max= 153, avg=95.44, stdev=25.95, samples=52 lat (msec) : 100=5.51%, 250=90.00%, 500=4.49% cpu : usr=0.11%, sys=0.64%, ctx=543, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179294: Tue Sep 1 14:46:50 2026 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26788msec); 0 zone resets slat (usec): min=22, max=22567, avg=164.14, stdev=790.62 clat (msec): min=18, max=393, avg=166.20, stdev=48.70 lat (msec): min=18, max=393, avg=166.36, stdev=48.67 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 171], | 70.00th=[ 186], 80.00th=[ 205], 90.00th=[ 222], 95.00th=[ 247], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 174, max= 616, per=24.94%, avg=380.89, stdev=89.98, samples=53 iops : min= 43, max= 154, avg=94.91, stdev=22.59, samples=53 lat (msec) : 20=0.04%, 50=0.27%, 100=3.71%, 250=92.11%, 500=3.87% cpu : usr=0.18%, sys=0.97%, ctx=1805, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179295: Tue Sep 1 14:46:50 2026 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26817msec); 0 zone resets slat (usec): min=26, max=13005, avg=137.71, stdev=434.97 clat (msec): min=33, max=406, avg=166.55, stdev=48.80 lat (msec): min=33, max=406, avg=166.69, stdev=48.79 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 105], 10.00th=[ 115], 20.00th=[ 128], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 171], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 226], 95.00th=[ 251], | 99.00th=[ 334], 99.50th=[ 393], 99.90th=[ 401], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 127, max= 576, per=24.88%, avg=379.85, stdev=87.60, samples=53 iops : min= 31, max= 144, avg=94.68, stdev=21.96, samples=53 lat (msec) : 50=0.47%, 100=2.70%, 250=91.76%, 500=5.08% cpu : usr=0.22%, sys=1.00%, ctx=2788, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1527KiB/s (1564kB/s), 382KiB/s-386KiB/s (391kB/s-395kB/s), io=40.0MiB (41.9MB), run=26517-26817msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179309: Tue Sep 1 14:47:35 2026 read: IOPS=36, BW=144KiB/s (148kB/s)(5048KiB/34984msec) slat (usec): min=2947, max=57967, avg=15751.86, stdev=8998.40 clat (msec): min=39, max=439, avg=206.36, stdev=58.72 lat (msec): min=59, max=488, avg=222.12, stdev=61.46 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 116], 10.00th=[ 132], 20.00th=[ 153], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 205], 60.00th=[ 222], | 70.00th=[ 239], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 363], 99.50th=[ 405], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 53, max= 278, per=25.02%, avg=143.35, stdev=42.26, samples=69 iops : min= 13, max= 69, avg=35.38, stdev=10.62, samples=69 write: IOPS=37, BW=148KiB/s (152kB/s)(5192KiB/34984msec); 0 zone resets slat (usec): min=29, max=28263, avg=252.73, stdev=1222.29 clat (msec): min=31, max=460, avg=212.15, stdev=60.65 lat (msec): min=31, max=460, avg=212.40, stdev=60.75 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 123], 10.00th=[ 134], 20.00th=[ 157], | 30.00th=[ 178], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 241], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 380], 99.50th=[ 409], 99.90th=[ 443], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 45, max= 264, per=24.72%, avg=147.58, stdev=43.02, samples=69 iops : min= 11, max= 66, avg=36.43, stdev=10.74, samples=69 lat (msec) : 50=0.08%, 100=0.98%, 250=74.88%, 500=24.06% cpu : usr=0.24%, sys=1.66%, ctx=3901, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179310: Tue Sep 1 14:47:35 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(4976KiB/34908msec) slat (usec): min=2316, max=77895, avg=15827.58, stdev=9495.91 clat (msec): min=34, max=424, avg=205.52, stdev=59.73 lat (msec): min=41, max=452, avg=221.35, stdev=62.52 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 127], 20.00th=[ 155], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 203], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 372], 99.50th=[ 397], 99.90th=[ 422], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 55, max= 294, per=24.74%, avg=141.74, stdev=42.45, samples=69 iops : min= 13, max= 73, avg=34.97, stdev=10.65, samples=69 write: IOPS=37, BW=151KiB/s (154kB/s)(5264KiB/34908msec); 0 zone resets slat (usec): min=26, max=15140, avg=295.18, stdev=1115.40 clat (msec): min=86, max=445, avg=212.78, stdev=59.85 lat (msec): min=86, max=445, avg=213.07, stdev=59.89 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 120], 10.00th=[ 136], 20.00th=[ 159], | 30.00th=[ 180], 40.00th=[ 199], 50.00th=[ 215], 60.00th=[ 230], | 70.00th=[ 243], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 401], 99.50th=[ 422], 99.90th=[ 439], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 62, max= 279, per=25.23%, avg=150.61, stdev=44.79, samples=69 iops : min= 15, max= 69, avg=37.20, stdev=11.23, samples=69 lat (msec) : 50=0.04%, 100=1.29%, 250=75.78%, 500=22.89% cpu : usr=0.21%, sys=1.76%, ctx=4052, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179311: Tue Sep 1 14:47:35 2026 read: IOPS=35, BW=142KiB/s (146kB/s)(4984KiB/34977msec) slat (usec): min=2583, max=73340, avg=15967.60, stdev=9448.63 clat (msec): min=81, max=486, avg=207.65, stdev=62.14 lat (msec): min=88, max=541, avg=223.63, stdev=65.20 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 114], 10.00th=[ 130], 20.00th=[ 155], | 30.00th=[ 174], 40.00th=[ 190], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 426], 99.50th=[ 456], 99.90th=[ 481], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 46, max= 310, per=24.79%, avg=142.07, stdev=40.83, samples=69 iops : min= 11, max= 77, avg=35.06, stdev=10.19, samples=69 write: IOPS=37, BW=150KiB/s (154kB/s)(5256KiB/34977msec); 0 zone resets slat (usec): min=30, max=55531, avg=260.81, stdev=1712.14 clat (msec): min=84, max=508, avg=211.03, stdev=62.09 lat (msec): min=85, max=508, avg=211.30, stdev=62.11 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 118], 10.00th=[ 134], 20.00th=[ 155], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 397], 99.50th=[ 435], 99.90th=[ 498], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 54, max= 279, per=25.02%, avg=149.36, stdev=47.60, samples=69 iops : min= 13, max= 69, avg=36.87, stdev=11.95, samples=69 lat (msec) : 100=1.09%, 250=76.21%, 500=22.66%, 750=0.04% cpu : usr=0.26%, sys=1.61%, ctx=3906, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179312: Tue Sep 1 14:47:35 2026 read: IOPS=36, BW=145KiB/s (149kB/s)(5060KiB/34811msec) slat (usec): min=20, max=67484, avg=15463.89, stdev=8921.54 clat (msec): min=19, max=475, avg=207.19, stdev=63.78 lat (msec): min=19, max=518, avg=222.66, stdev=66.37 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 114], 10.00th=[ 129], 20.00th=[ 155], | 30.00th=[ 171], 40.00th=[ 188], 50.00th=[ 205], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 405], 99.50th=[ 426], 99.90th=[ 472], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 71, max= 349, per=25.18%, avg=144.30, stdev=44.75, samples=69 iops : min= 17, max= 87, avg=35.62, stdev=11.23, samples=69 write: IOPS=37, BW=149KiB/s (152kB/s)(5180KiB/34811msec); 0 zone resets slat (usec): min=21, max=11571, avg=205.21, stdev=618.92 clat (msec): min=88, max=467, avg=211.79, stdev=59.68 lat (msec): min=88, max=467, avg=212.00, stdev=59.70 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 157], | 30.00th=[ 180], 40.00th=[ 197], 50.00th=[ 213], 60.00th=[ 228], | 70.00th=[ 241], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 397], 99.50th=[ 414], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 62, max= 247, per=24.77%, avg=147.88, stdev=39.81, samples=69 iops : min= 15, max= 61, avg=36.51, stdev=10.01, samples=69 lat (msec) : 20=0.12%, 50=0.12%, 100=1.37%, 250=76.09%, 500=22.30% cpu : usr=0.25%, sys=1.59%, ctx=3683, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=574KiB/s (587kB/s), 142KiB/s-145KiB/s (146kB/s-149kB/s), io=19.6MiB (20.5MB), run=34811-34984msec WRITE: bw=597KiB/s (612kB/s), 148KiB/s-151KiB/s (152kB/s-154kB/s), io=20.4MiB (21.4MB), run=34811-34984msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179322: Tue Sep 1 14:47:47 2026 read: IOPS=1, BW=77.4MiB/s (81.1MB/s)(40.0MiB/517msec) slat (nsec): min=43177k, max=43177k, avg=43177365.00, stdev= 0.00 clat (nsec): min=446647k, max=446647k, avg=446647062.00, stdev= 0.00 lat (nsec): min=489855k, max=489855k, avg=489855268.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.33%, ctx=11, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.4MiB/s (81.1MB/s), 77.4MiB/s-77.4MiB/s (81.1MB/s-81.1MB/s), io=40.0MiB (41.9MB), run=517-517msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout