== sanity test 398c: run fio to test AIO ================= 16:20:00 (1775852400) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.337806 s, 124 MB/s osc.lustre-OST0000-osc-ffff9e0ac7017800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=190180: Fri Apr 10 16:20:35 2026 write: IOPS=95, BW=381KiB/s (390kB/s)(10.0MiB/26906msec); 0 zone resets slat (usec): min=49, max=14062, avg=305.33, stdev=799.39 clat (msec): min=24, max=318, avg=166.98, stdev=35.89 lat (msec): min=24, max=318, avg=167.29, stdev=35.89 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 117], 10.00th=[ 125], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 197], 90.00th=[ 218], 95.00th=[ 232], | 99.00th=[ 275], 99.50th=[ 279], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 254, max= 611, per=24.99%, avg=380.09, stdev=67.80, samples=53 iops : min= 63, max= 152, avg=94.66, stdev=17.03, samples=53 lat (msec) : 50=0.12%, 100=1.09%, 250=96.52%, 500=2.27% cpu : usr=0.16%, sys=1.23%, ctx=2592, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=190181: Fri Apr 10 16:20:35 2026 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26793msec); 0 zone resets slat (usec): min=36, max=13424, avg=308.97, stdev=775.94 clat (msec): min=24, max=345, avg=166.90, stdev=35.70 lat (msec): min=24, max=345, avg=167.22, stdev=35.67 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 118], 10.00th=[ 127], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 292], 99.95th=[ 300], | 99.99th=[ 347] bw ( KiB/s): min= 230, max= 531, per=25.04%, avg=380.89, stdev=61.97, samples=53 iops : min= 57, max= 132, avg=94.81, stdev=15.56, samples=53 lat (msec) : 50=0.08%, 100=1.56%, 250=96.33%, 500=2.03% cpu : usr=0.16%, sys=1.33%, ctx=2623, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=190182: Fri Apr 10 16:20:35 2026 write: IOPS=95, BW=381KiB/s (390kB/s)(10.0MiB/26882msec); 0 zone resets slat (usec): min=51, max=22331, avg=352.69, stdev=997.46 clat (msec): min=22, max=313, avg=167.08, stdev=35.81 lat (msec): min=23, max=313, avg=167.43, stdev=35.81 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 116], 10.00th=[ 126], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 211], 95.00th=[ 228], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 313] bw ( KiB/s): min= 254, max= 531, per=24.91%, avg=378.92, stdev=68.44, samples=53 iops : min= 63, max= 132, avg=94.36, stdev=17.19, samples=53 lat (msec) : 50=0.16%, 100=1.33%, 250=96.29%, 500=2.23% cpu : usr=0.20%, sys=1.50%, ctx=3041, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=190183: Fri Apr 10 16:20:35 2026 write: IOPS=95, BW=380KiB/s (390kB/s)(10.0MiB/26917msec); 0 zone resets slat (usec): min=38, max=11864, avg=270.82, stdev=690.68 clat (msec): min=36, max=281, avg=167.27, stdev=34.40 lat (msec): min=36, max=281, avg=167.55, stdev=34.36 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 122], 10.00th=[ 128], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 211], 95.00th=[ 230], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 254, max= 523, per=24.95%, avg=379.45, stdev=70.58, samples=53 iops : min= 63, max= 130, avg=94.47, stdev=17.70, samples=53 lat (msec) : 50=0.08%, 100=1.02%, 250=96.88%, 500=2.03% cpu : usr=0.13%, sys=1.15%, ctx=2041, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1522KiB/s (1558kB/s), 380KiB/s-382KiB/s (390kB/s-391kB/s), io=40.0MiB (41.9MB), run=26793-26917msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=190197: Fri Apr 10 16:21:14 2026 read: IOPS=40, BW=161KiB/s (164kB/s)(5048KiB/31442msec) slat (usec): min=2610, max=51780, avg=13843.85, stdev=7042.13 clat (msec): min=79, max=374, avg=184.12, stdev=41.90 lat (msec): min=87, max=398, avg=197.98, stdev=43.48 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 125], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 259], | 99.00th=[ 309], 99.50th=[ 326], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 102, max= 256, per=25.04%, avg=159.76, stdev=36.31, samples=62 iops : min= 25, max= 64, avg=39.44, stdev= 9.10, samples=62 write: IOPS=41, BW=165KiB/s (169kB/s)(5192KiB/31442msec); 0 zone resets slat (usec): min=58, max=17990, avg=373.09, stdev=1059.10 clat (msec): min=76, max=368, avg=192.59, stdev=42.58 lat (msec): min=77, max=368, avg=192.97, stdev=42.54 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 199], | 70.00th=[ 209], 80.00th=[ 226], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 326], 99.50th=[ 330], 99.90th=[ 338], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 87, max= 253, per=24.77%, avg=164.50, stdev=34.06, samples=62 iops : min= 21, max= 63, avg=40.65, stdev= 8.52, samples=62 lat (msec) : 100=0.78%, 250=91.52%, 500=7.70% cpu : usr=0.27%, sys=2.18%, ctx=4734, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190198: Fri Apr 10 16:21:14 2026 read: IOPS=39, BW=158KiB/s (162kB/s)(4976KiB/31436msec) slat (usec): min=81, max=67779, avg=14035.92, stdev=7253.19 clat (msec): min=48, max=350, avg=187.36, stdev=45.06 lat (msec): min=57, max=383, avg=201.41, stdev=46.64 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 122], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 94, max= 247, per=24.66%, avg=157.32, stdev=33.72, samples=62 iops : min= 23, max= 61, avg=38.85, stdev= 8.44, samples=62 write: IOPS=41, BW=167KiB/s (171kB/s)(5264KiB/31436msec); 0 zone resets slat (usec): min=54, max=12693, avg=340.90, stdev=904.47 clat (msec): min=82, max=346, avg=190.75, stdev=41.03 lat (msec): min=83, max=346, avg=191.10, stdev=41.02 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 132], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 334], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 87, max= 277, per=25.14%, avg=166.92, stdev=39.26, samples=62 iops : min= 21, max= 69, avg=41.26, stdev= 9.86, samples=62 lat (msec) : 50=0.04%, 100=1.05%, 250=90.20%, 500=8.71% cpu : usr=0.26%, sys=2.08%, ctx=4348, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190199: Fri Apr 10 16:21:14 2026 read: IOPS=39, BW=159KiB/s (162kB/s)(4984KiB/31408msec) slat (usec): min=3113, max=69158, avg=13852.85, stdev=7263.17 clat (msec): min=23, max=341, avg=186.47, stdev=45.19 lat (msec): min=28, max=364, avg=200.33, stdev=47.15 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 123], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 222], 90.00th=[ 247], 95.00th=[ 268], | 99.00th=[ 326], 99.50th=[ 330], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 79, max= 263, per=24.83%, avg=158.44, stdev=35.19, samples=62 iops : min= 19, max= 65, avg=39.13, stdev= 8.77, samples=62 write: IOPS=41, BW=167KiB/s (171kB/s)(5256KiB/31408msec); 0 zone resets slat (usec): min=57, max=11328, avg=355.45, stdev=831.82 clat (msec): min=61, max=327, avg=189.82, stdev=40.65 lat (msec): min=61, max=327, avg=190.18, stdev=40.65 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 199], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 243], 95.00th=[ 262], | 99.00th=[ 300], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 70, max= 255, per=25.11%, avg=166.74, stdev=37.93, samples=62 iops : min= 17, max= 63, avg=41.21, stdev= 9.54, samples=62 lat (msec) : 50=0.08%, 100=0.94%, 250=90.94%, 500=8.05% cpu : usr=0.32%, sys=2.19%, ctx=4666, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190200: Fri Apr 10 16:21:14 2026 read: IOPS=40, BW=161KiB/s (165kB/s)(5060KiB/31447msec) slat (usec): min=3295, max=59945, avg=13809.47, stdev=6802.61 clat (msec): min=74, max=329, avg=185.29, stdev=42.73 lat (msec): min=86, max=351, avg=199.10, stdev=44.23 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 243], 95.00th=[ 264], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 87, max= 253, per=25.07%, avg=159.97, stdev=33.96, samples=62 iops : min= 21, max= 63, avg=39.52, stdev= 8.57, samples=62 write: IOPS=41, BW=165KiB/s (169kB/s)(5180KiB/31447msec); 0 zone resets slat (usec): min=45, max=23601, avg=371.24, stdev=1057.08 clat (msec): min=80, max=354, avg=192.10, stdev=43.45 lat (msec): min=80, max=354, avg=192.47, stdev=43.46 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 197], | 70.00th=[ 209], 80.00th=[ 228], 90.00th=[ 253], 95.00th=[ 271], | 99.00th=[ 309], 99.50th=[ 321], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 79, max= 255, per=24.62%, avg=163.47, stdev=33.66, samples=62 iops : min= 19, max= 63, avg=40.40, stdev= 8.38, samples=62 lat (msec) : 100=1.21%, 250=89.49%, 500=9.30% cpu : usr=0.30%, sys=2.05%, ctx=4564, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=638KiB/s (653kB/s), 158KiB/s-161KiB/s (162kB/s-165kB/s), io=19.6MiB (20.5MB), run=31408-31447msec WRITE: bw=664KiB/s (680kB/s), 165KiB/s-167KiB/s (169kB/s-171kB/s), io=20.4MiB (21.4MB), run=31408-31447msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=190209: Fri Apr 10 16:21:24 2026 read: IOPS=2, BW=82.5MiB/s (86.5MB/s)(40.0MiB/485msec) slat (nsec): min=39338k, max=39338k, avg=39338194.00, stdev= 0.00 clat (nsec): min=415798k, max=415798k, avg=415798155.00, stdev= 0.00 lat (nsec): min=455142k, max=455142k, avg=455142402.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 418], 10.00th=[ 418], 20.00th=[ 418], | 30.00th=[ 418], 40.00th=[ 418], 50.00th=[ 418], 60.00th=[ 418], | 70.00th=[ 418], 80.00th=[ 418], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.89%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=82.5MiB/s (86.5MB/s), 82.5MiB/s-82.5MiB/s (86.5MB/s-86.5MB/s), io=40.0MiB (41.9MB), run=485-485msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout