== sanity test 398c: run fio to test AIO ================= 19:06:56 (1786835216) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.359202 s, 117 MB/s osc.lustre-OST0000-osc-ffff92b4eb1c6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145381: Sat Aug 15 19:08:52 2026 write: IOPS=23, BW=93.4KiB/s (95.6kB/s)(10.0MiB/109688msec); 0 zone resets slat (usec): min=36, max=17986, avg=281.73, stdev=733.11 clat (msec): min=69, max=1239, avg=684.32, stdev=185.90 lat (msec): min=73, max=1239, avg=684.60, stdev=185.88 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 401], 10.00th=[ 485], 20.00th=[ 550], | 30.00th=[ 600], 40.00th=[ 642], 50.00th=[ 684], 60.00th=[ 718], | 70.00th=[ 776], 80.00th=[ 835], 90.00th=[ 911], 95.00th=[ 978], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 313, per=24.82%, avg=92.58, stdev=35.79, samples=218 iops : min= 1, max= 78, avg=22.75, stdev= 8.96, samples=218 lat (msec) : 100=0.04%, 250=2.73%, 500=8.71%, 750=54.34%, 1000=30.23% lat (msec) : 2000=3.95% cpu : usr=0.06%, sys=0.38%, ctx=2730, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145382: Sat Aug 15 19:08:52 2026 write: IOPS=23, BW=93.4KiB/s (95.6kB/s)(10.0MiB/109636msec); 0 zone resets slat (usec): min=37, max=13323, avg=268.34, stdev=720.26 clat (msec): min=146, max=1243, avg=684.10, stdev=182.85 lat (msec): min=147, max=1243, avg=684.37, stdev=182.86 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 397], 10.00th=[ 481], 20.00th=[ 550], | 30.00th=[ 609], 40.00th=[ 642], 50.00th=[ 684], 60.00th=[ 726], | 70.00th=[ 768], 80.00th=[ 835], 90.00th=[ 911], 95.00th=[ 969], | 99.00th=[ 1099], 99.50th=[ 1150], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 325, per=24.82%, avg=92.57, stdev=35.00, samples=218 iops : min= 1, max= 81, avg=22.75, stdev= 8.78, samples=218 lat (msec) : 250=3.36%, 500=9.14%, 750=52.66%, 1000=31.21%, 2000=3.63% cpu : usr=0.04%, sys=0.38%, ctx=2799, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145383: Sat Aug 15 19:08:52 2026 write: IOPS=23, BW=93.4KiB/s (95.6kB/s)(10.0MiB/109634msec); 0 zone resets slat (usec): min=35, max=15948, avg=278.30, stdev=703.40 clat (msec): min=142, max=1397, avg=684.06, stdev=186.18 lat (msec): min=142, max=1398, avg=684.34, stdev=186.18 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 393], 10.00th=[ 481], 20.00th=[ 550], | 30.00th=[ 600], 40.00th=[ 642], 50.00th=[ 684], 60.00th=[ 726], | 70.00th=[ 776], 80.00th=[ 835], 90.00th=[ 911], 95.00th=[ 969], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1250], 99.95th=[ 1267], | 99.99th=[ 1401] bw ( KiB/s): min= 8, max= 326, per=24.94%, avg=93.01, stdev=36.64, samples=217 iops : min= 2, max= 81, avg=22.86, stdev= 9.19, samples=217 lat (msec) : 250=3.52%, 500=8.83%, 750=52.70%, 1000=31.05%, 2000=3.91% cpu : usr=0.06%, sys=0.38%, ctx=2704, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145384: Sat Aug 15 19:08:52 2026 write: IOPS=23, BW=93.4KiB/s (95.7kB/s)(10.0MiB/109583msec); 0 zone resets slat (usec): min=37, max=12598, avg=299.11, stdev=687.51 clat (msec): min=63, max=1491, avg=684.23, stdev=189.79 lat (msec): min=63, max=1491, avg=684.53, stdev=189.76 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 372], 10.00th=[ 477], 20.00th=[ 550], | 30.00th=[ 600], 40.00th=[ 642], 50.00th=[ 684], 60.00th=[ 726], | 70.00th=[ 776], 80.00th=[ 835], 90.00th=[ 911], 95.00th=[ 978], | 99.00th=[ 1133], 99.50th=[ 1200], 99.90th=[ 1301], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 353, per=24.81%, avg=92.55, stdev=36.88, samples=218 iops : min= 1, max= 88, avg=22.74, stdev= 9.25, samples=218 lat (msec) : 100=0.20%, 250=3.63%, 500=8.91%, 750=52.30%, 1000=31.21% lat (msec) : 2000=3.75% cpu : usr=0.06%, sys=0.40%, ctx=3074, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=373KiB/s (382kB/s), 93.4KiB/s-93.4KiB/s (95.6kB/s-95.7kB/s), io=40.0MiB (41.9MB), run=109583-109688msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145410: Sat Aug 15 19:10:15 2026 read: IOPS=16, BW=66.1KiB/s (67.7kB/s)(5048KiB/76355msec) slat (usec): min=58, max=41847, avg=10483.41, stdev=5682.80 clat (msec): min=30, max=561, avg=295.45, stdev=71.11 lat (msec): min=40, max=571, avg=305.93, stdev=71.97 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 197], 10.00th=[ 213], 20.00th=[ 236], | 30.00th=[ 255], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 326], 80.00th=[ 347], 90.00th=[ 388], 95.00th=[ 426], | 99.00th=[ 489], 99.50th=[ 510], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 23, max= 143, per=25.04%, avg=65.60, stdev=25.33, samples=151 iops : min= 5, max= 35, avg=15.98, stdev= 6.35, samples=151 write: IOPS=16, BW=67.0KiB/s (69.6kB/s)(5192KiB/76355msec); 0 zone resets slat (usec): min=38, max=15773, avg=349.07, stdev=806.79 clat (msec): min=121, max=1311, avg=642.81, stdev=174.56 lat (msec): min=121, max=1311, avg=643.16, stdev=174.57 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 372], 10.00th=[ 426], 20.00th=[ 506], | 30.00th=[ 558], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 768], 90.00th=[ 860], 95.00th=[ 978], | 99.00th=[ 1133], 99.50th=[ 1183], 99.90th=[ 1301], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 16, max= 120, per=24.78%, avg=67.40, stdev=20.52, samples=151 iops : min= 4, max= 30, avg=16.43, stdev= 5.24, samples=151 lat (msec) : 50=0.16%, 100=0.04%, 250=13.67%, 500=44.57%, 750=30.51% lat (msec) : 1000=9.02%, 2000=2.03% cpu : usr=0.11%, sys=0.90%, ctx=5163, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145411: Sat Aug 15 19:10:15 2026 read: IOPS=16, BW=65.0KiB/s (66.6kB/s)(4976KiB/76501msec) slat (usec): min=2564, max=44984, avg=10623.56, stdev=5638.60 clat (msec): min=48, max=563, avg=295.14, stdev=69.69 lat (msec): min=54, max=581, avg=305.77, stdev=70.33 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 199], 10.00th=[ 215], 20.00th=[ 239], | 30.00th=[ 255], 40.00th=[ 271], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 326], 80.00th=[ 347], 90.00th=[ 384], 95.00th=[ 414], | 99.00th=[ 498], 99.50th=[ 518], 99.90th=[ 558], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 8, max= 144, per=24.57%, avg=64.36, stdev=27.36, samples=152 iops : min= 2, max= 36, avg=15.66, stdev= 6.95, samples=152 write: IOPS=17, BW=68.8KiB/s (70.5kB/s)(5264KiB/76501msec); 0 zone resets slat (usec): min=37, max=9772, avg=322.94, stdev=733.14 clat (msec): min=120, max=1266, avg=639.34, stdev=180.63 lat (msec): min=120, max=1266, avg=639.66, stdev=180.61 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 351], 10.00th=[ 401], 20.00th=[ 493], | 30.00th=[ 558], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 760], 90.00th=[ 877], 95.00th=[ 978], | 99.00th=[ 1150], 99.50th=[ 1183], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 15, max= 135, per=25.09%, avg=68.25, stdev=23.58, samples=152 iops : min= 3, max= 33, avg=16.64, stdev= 5.90, samples=152 lat (msec) : 50=0.04%, 100=0.35%, 250=12.93%, 500=46.17%, 750=29.18% lat (msec) : 1000=8.98%, 2000=2.34% cpu : usr=0.12%, sys=0.90%, ctx=5002, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145412: Sat Aug 15 19:10:15 2026 read: IOPS=16, BW=65.1KiB/s (66.7kB/s)(4984KiB/76572msec) slat (usec): min=2171, max=39960, avg=10718.42, stdev=5790.01 clat (msec): min=9, max=555, avg=298.66, stdev=73.76 lat (msec): min=15, max=565, avg=309.38, stdev=74.47 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 194], 10.00th=[ 213], 20.00th=[ 241], | 30.00th=[ 262], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 313], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 388], 95.00th=[ 426], | 99.00th=[ 506], 99.50th=[ 514], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 15, max= 159, per=24.65%, avg=64.57, stdev=26.83, samples=152 iops : min= 3, max= 39, avg=15.72, stdev= 6.77, samples=152 write: IOPS=17, BW=68.6KiB/s (70.3kB/s)(5256KiB/76572msec); 0 zone resets slat (usec): min=52, max=15723, avg=328.52, stdev=764.12 clat (msec): min=120, max=1292, avg=636.98, stdev=179.17 lat (msec): min=120, max=1292, avg=637.32, stdev=179.17 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 351], 10.00th=[ 409], 20.00th=[ 493], | 30.00th=[ 558], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 701], 80.00th=[ 751], 90.00th=[ 860], 95.00th=[ 961], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1284], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 144, per=24.87%, avg=67.64, stdev=23.30, samples=152 iops : min= 1, max= 36, avg=16.49, stdev= 5.84, samples=152 lat (msec) : 10=0.04%, 20=0.12%, 50=0.08%, 100=0.16%, 250=12.30% lat (msec) : 500=46.09%, 750=31.17%, 1000=8.20%, 2000=1.84% cpu : usr=0.11%, sys=0.89%, ctx=5164, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145413: Sat Aug 15 19:10:15 2026 read: IOPS=16, BW=66.5KiB/s (68.1kB/s)(5060KiB/76134msec) slat (usec): min=2318, max=40047, avg=10556.60, stdev=5592.19 clat (msec): min=7, max=760, avg=300.18, stdev=80.73 lat (msec): min=14, max=776, avg=310.74, stdev=81.29 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 197], 10.00th=[ 215], 20.00th=[ 241], | 30.00th=[ 259], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 401], 95.00th=[ 439], | 99.00th=[ 550], 99.50th=[ 625], 99.90th=[ 718], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 7, max= 127, per=25.16%, avg=65.91, stdev=24.88, samples=151 iops : min= 1, max= 31, avg=16.06, stdev= 6.25, samples=151 write: IOPS=17, BW=68.0KiB/s (69.7kB/s)(5180KiB/76134msec); 0 zone resets slat (usec): min=39, max=13123, avg=336.27, stdev=781.14 clat (msec): min=149, max=1197, avg=635.31, stdev=171.65 lat (msec): min=149, max=1199, avg=635.65, stdev=171.63 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 372], 10.00th=[ 422], 20.00th=[ 502], | 30.00th=[ 550], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 751], 90.00th=[ 852], 95.00th=[ 953], | 99.00th=[ 1116], 99.50th=[ 1167], 99.90th=[ 1183], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 23, max= 136, per=24.83%, avg=67.54, stdev=22.69, samples=151 iops : min= 5, max= 34, avg=16.46, stdev= 5.74, samples=151 lat (msec) : 10=0.04%, 20=0.04%, 50=0.20%, 100=0.23%, 250=12.62% lat (msec) : 500=45.39%, 750=31.17%, 1000=8.52%, 2000=1.80% cpu : usr=0.10%, sys=0.95%, ctx=5116, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=262KiB/s (268kB/s), 65.0KiB/s-66.5KiB/s (66.6kB/s-68.1kB/s), io=19.6MiB (20.5MB), run=76134-76572msec WRITE: bw=273KiB/s (279kB/s), 67.0KiB/s-68.8KiB/s (69.6kB/s-70.5kB/s), io=20.4MiB (21.4MB), run=76134-76572msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145430: Sat Aug 15 19:10:23 2026 read: IOPS=2, BW=116MiB/s (122MB/s)(40.0MiB/345msec) slat (nsec): min=27777k, max=27777k, avg=27776758.00, stdev= 0.00 clat (nsec): min=297822k, max=297822k, avg=297821865.00, stdev= 0.00 lat (nsec): min=325603k, max=325603k, avg=325603479.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.94%, ctx=30, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=116MiB/s (122MB/s), 116MiB/s-116MiB/s (122MB/s-122MB/s), io=40.0MiB (41.9MB), run=345-345msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout