== sanity test 398c: run fio to test AIO ================= 21:36:27 (1787103387) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.155315 s, 270 MB/s osc.lustre-OST0000-osc-ffff90aec614d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=150612: Tue Aug 18 21:37:15 2026 write: IOPS=57, BW=229KiB/s (234kB/s)(10.0MiB/44717msec); 0 zone resets slat (usec): min=38, max=4112, avg=202.04, stdev=188.59 clat (msec): min=61, max=559, avg=279.08, stdev=76.71 lat (msec): min=61, max=559, avg=279.29, stdev=76.72 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 178], 10.00th=[ 194], 20.00th=[ 215], | 30.00th=[ 232], 40.00th=[ 249], 50.00th=[ 266], 60.00th=[ 288], | 70.00th=[ 313], 80.00th=[ 342], 90.00th=[ 388], 95.00th=[ 426], | 99.00th=[ 493], 99.50th=[ 506], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 104, max= 416, per=24.90%, avg=227.36, stdev=57.51, samples=89 iops : min= 26, max= 104, avg=56.67, stdev=14.42, samples=89 lat (msec) : 100=0.62%, 250=40.20%, 500=58.55%, 750=0.62% cpu : usr=0.12%, sys=0.89%, ctx=2826, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150613: Tue Aug 18 21:37:15 2026 write: IOPS=57, BW=228KiB/s (234kB/s)(10.0MiB/44815msec); 0 zone resets slat (usec): min=39, max=4091, avg=214.28, stdev=211.73 clat (msec): min=96, max=554, avg=279.45, stdev=74.12 lat (msec): min=97, max=554, avg=279.67, stdev=74.13 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 184], 10.00th=[ 197], 20.00th=[ 220], | 30.00th=[ 234], 40.00th=[ 249], 50.00th=[ 266], 60.00th=[ 284], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 388], 95.00th=[ 422], | 99.00th=[ 489], 99.50th=[ 506], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 558] bw ( KiB/s): min= 127, max= 344, per=24.93%, avg=227.63, stdev=52.64, samples=89 iops : min= 31, max= 86, avg=56.74, stdev=13.19, samples=89 lat (msec) : 100=0.04%, 250=40.62%, 500=58.55%, 750=0.78% cpu : usr=0.10%, sys=0.93%, ctx=2815, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150614: Tue Aug 18 21:37:15 2026 write: IOPS=57, BW=228KiB/s (234kB/s)(10.0MiB/44816msec); 0 zone resets slat (usec): min=35, max=7086, avg=211.21, stdev=240.42 clat (msec): min=81, max=542, avg=279.54, stdev=75.68 lat (msec): min=81, max=542, avg=279.75, stdev=75.69 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 182], 10.00th=[ 194], 20.00th=[ 220], | 30.00th=[ 234], 40.00th=[ 251], 50.00th=[ 268], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 338], 90.00th=[ 388], 95.00th=[ 430], | 99.00th=[ 489], 99.50th=[ 514], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 542] bw ( KiB/s): min= 127, max= 384, per=24.93%, avg=227.63, stdev=56.81, samples=89 iops : min= 31, max= 96, avg=56.74, stdev=14.24, samples=89 lat (msec) : 100=0.62%, 250=39.61%, 500=58.95%, 750=0.82% cpu : usr=0.08%, sys=0.91%, ctx=2745, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150615: Tue Aug 18 21:37:15 2026 write: IOPS=57, BW=228KiB/s (234kB/s)(10.0MiB/44815msec); 0 zone resets slat (usec): min=52, max=5205, avg=207.55, stdev=194.90 clat (msec): min=29, max=559, avg=279.47, stdev=75.33 lat (msec): min=29, max=559, avg=279.68, stdev=75.33 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 184], 10.00th=[ 197], 20.00th=[ 215], | 30.00th=[ 232], 40.00th=[ 249], 50.00th=[ 268], 60.00th=[ 288], | 70.00th=[ 309], 80.00th=[ 342], 90.00th=[ 388], 95.00th=[ 426], | 99.00th=[ 485], 99.50th=[ 514], 99.90th=[ 535], 99.95th=[ 542], | 99.99th=[ 558] bw ( KiB/s): min= 127, max= 352, per=24.93%, avg=227.57, stdev=55.22, samples=89 iops : min= 31, max= 88, avg=56.72, stdev=13.84, samples=89 lat (msec) : 50=0.12%, 100=0.16%, 250=40.74%, 500=58.28%, 750=0.70% cpu : usr=0.08%, sys=0.93%, ctx=2845, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=914KiB/s (936kB/s), 228KiB/s-229KiB/s (234kB/s-234kB/s), io=40.0MiB (41.9MB), run=44717-44816msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=150632: Tue Aug 18 21:37:48 2026 read: IOPS=42, BW=171KiB/s (175kB/s)(5048KiB/29512msec) slat (usec): min=34, max=17551, avg=3969.83, stdev=1938.00 clat (msec): min=15, max=294, avg=117.10, stdev=30.37 lat (msec): min=17, max=296, avg=121.07, stdev=30.90 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 75], 10.00th=[ 85], 20.00th=[ 95], | 30.00th=[ 102], 40.00th=[ 107], 50.00th=[ 113], 60.00th=[ 121], | 70.00th=[ 127], 80.00th=[ 136], 90.00th=[ 155], 95.00th=[ 174], | 99.00th=[ 205], 99.50th=[ 251], 99.90th=[ 275], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 72, max= 256, per=25.30%, avg=171.31, stdev=43.19, samples=58 iops : min= 18, max= 64, avg=42.69, stdev=10.84, samples=58 write: IOPS=43, BW=176KiB/s (180kB/s)(5192KiB/29512msec); 0 zone resets slat (usec): min=38, max=3539, avg=214.25, stdev=239.58 clat (msec): min=41, max=490, avg=245.56, stdev=62.48 lat (msec): min=41, max=490, avg=245.77, stdev=62.50 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 146], 10.00th=[ 169], 20.00th=[ 194], | 30.00th=[ 213], 40.00th=[ 228], 50.00th=[ 247], 60.00th=[ 262], | 70.00th=[ 279], 80.00th=[ 296], 90.00th=[ 317], 95.00th=[ 342], | 99.00th=[ 409], 99.50th=[ 439], 99.90th=[ 477], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 103, max= 264, per=24.91%, avg=175.60, stdev=31.62, samples=58 iops : min= 25, max= 66, avg=43.76, stdev= 7.90, samples=58 lat (msec) : 20=0.20%, 50=0.35%, 100=13.55%, 250=61.05%, 500=24.84% cpu : usr=0.19%, sys=1.63%, ctx=5015, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150633: Tue Aug 18 21:37:48 2026 read: IOPS=41, BW=168KiB/s (172kB/s)(4976KiB/29624msec) slat (usec): min=1376, max=13925, avg=3851.77, stdev=1802.02 clat (msec): min=3, max=234, avg=114.96, stdev=28.59 lat (msec): min=5, max=239, avg=118.82, stdev=29.04 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 78], 10.00th=[ 85], 20.00th=[ 94], | 30.00th=[ 101], 40.00th=[ 106], 50.00th=[ 112], 60.00th=[ 120], | 70.00th=[ 127], 80.00th=[ 134], 90.00th=[ 150], 95.00th=[ 169], | 99.00th=[ 201], 99.50th=[ 207], 99.90th=[ 222], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 48, max= 296, per=24.68%, avg=167.05, stdev=51.83, samples=58 iops : min= 12, max= 74, avg=41.62, stdev=13.02, samples=58 write: IOPS=44, BW=178KiB/s (182kB/s)(5264KiB/29624msec); 0 zone resets slat (usec): min=45, max=3211, avg=202.62, stdev=193.02 clat (msec): min=83, max=461, avg=247.02, stdev=61.83 lat (msec): min=83, max=461, avg=247.22, stdev=61.84 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 153], 10.00th=[ 178], 20.00th=[ 194], | 30.00th=[ 213], 40.00th=[ 228], 50.00th=[ 245], 60.00th=[ 262], | 70.00th=[ 279], 80.00th=[ 296], 90.00th=[ 326], 95.00th=[ 351], | 99.00th=[ 414], 99.50th=[ 426], 99.90th=[ 456], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 112, max= 272, per=25.01%, avg=176.31, stdev=33.97, samples=58 iops : min= 28, max= 68, avg=43.93, stdev= 8.53, samples=58 lat (msec) : 4=0.04%, 20=0.20%, 50=0.39%, 100=13.98%, 250=61.37% lat (msec) : 500=24.02% cpu : usr=0.19%, sys=1.57%, ctx=4952, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150634: Tue Aug 18 21:37:48 2026 read: IOPS=42, BW=168KiB/s (172kB/s)(4984KiB/29625msec) slat (usec): min=1335, max=14720, avg=3909.69, stdev=1789.03 clat (msec): min=3, max=254, avg=116.36, stdev=30.52 lat (msec): min=6, max=261, avg=120.27, stdev=30.96 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 79], 10.00th=[ 84], 20.00th=[ 93], | 30.00th=[ 101], 40.00th=[ 108], 50.00th=[ 113], 60.00th=[ 118], | 70.00th=[ 127], 80.00th=[ 136], 90.00th=[ 157], 95.00th=[ 174], | 99.00th=[ 211], 99.50th=[ 222], 99.90th=[ 253], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 56, max= 320, per=24.85%, avg=168.22, stdev=49.24, samples=58 iops : min= 14, max= 80, avg=41.91, stdev=12.32, samples=58 write: IOPS=44, BW=177KiB/s (182kB/s)(5256KiB/29625msec); 0 zone resets slat (usec): min=51, max=2587, avg=201.80, stdev=173.74 clat (msec): min=58, max=488, avg=245.90, stdev=61.23 lat (msec): min=59, max=488, avg=246.11, stdev=61.23 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 153], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 228], 50.00th=[ 243], 60.00th=[ 259], | 70.00th=[ 279], 80.00th=[ 300], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 409], 99.50th=[ 418], 99.90th=[ 481], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 80, max= 253, per=24.92%, avg=175.71, stdev=35.70, samples=58 iops : min= 20, max= 63, avg=43.79, stdev= 8.99, samples=58 lat (msec) : 4=0.08%, 10=0.04%, 20=0.16%, 50=0.12%, 100=14.73% lat (msec) : 250=61.41%, 500=23.48% cpu : usr=0.18%, sys=1.59%, ctx=5039, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150635: Tue Aug 18 21:37:48 2026 read: IOPS=42, BW=171KiB/s (175kB/s)(5060KiB/29559msec) slat (usec): min=1341, max=16139, avg=3950.59, stdev=1805.07 clat (msec): min=5, max=258, avg=115.45, stdev=30.24 lat (msec): min=9, max=263, avg=119.41, stdev=30.71 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 77], 10.00th=[ 85], 20.00th=[ 92], | 30.00th=[ 100], 40.00th=[ 107], 50.00th=[ 112], 60.00th=[ 120], | 70.00th=[ 127], 80.00th=[ 134], 90.00th=[ 155], 95.00th=[ 171], | 99.00th=[ 209], 99.50th=[ 222], 99.90th=[ 255], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 64, max= 254, per=25.33%, avg=171.47, stdev=45.08, samples=58 iops : min= 16, max= 63, avg=42.72, stdev=11.30, samples=58 write: IOPS=43, BW=175KiB/s (179kB/s)(5180KiB/29559msec); 0 zone resets slat (usec): min=55, max=7469, avg=208.88, stdev=255.93 clat (msec): min=64, max=479, avg=247.69, stdev=61.89 lat (msec): min=64, max=479, avg=247.90, stdev=61.89 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 153], 10.00th=[ 176], 20.00th=[ 197], | 30.00th=[ 213], 40.00th=[ 228], 50.00th=[ 247], 60.00th=[ 264], | 70.00th=[ 284], 80.00th=[ 300], 90.00th=[ 330], 95.00th=[ 351], | 99.00th=[ 397], 99.50th=[ 422], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 120, max= 264, per=24.71%, avg=174.19, stdev=32.23, samples=58 iops : min= 30, max= 66, avg=43.40, stdev= 8.08, samples=58 lat (msec) : 10=0.08%, 20=0.20%, 50=0.16%, 100=15.47%, 250=60.16% lat (msec) : 500=23.95% cpu : usr=0.15%, sys=1.67%, ctx=5157, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=677KiB/s (694kB/s), 168KiB/s-171KiB/s (172kB/s-175kB/s), io=19.6MiB (20.5MB), run=29512-29625msec WRITE: bw=705KiB/s (722kB/s), 175KiB/s-178KiB/s (179kB/s-182kB/s), io=20.4MiB (21.4MB), run=29512-29625msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=150643: Tue Aug 18 21:37:52 2026 read: IOPS=5, BW=226MiB/s (237MB/s)(40.0MiB/177msec) slat (nsec): min=15941k, max=15941k, avg=15940975.00, stdev= 0.00 clat (nsec): min=155903k, max=155903k, avg=155902590.00, stdev= 0.00 lat (nsec): min=171848k, max=171848k, avg=171848339.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.82%, ctx=39, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=40.0MiB (41.9MB), run=177-177msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout