== sanity test 398c: run fio to test AIO ================= 19:43:23 (1787787803) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.54944 s, 76.3 MB/s osc.lustre-OST0000-osc-ffff9082906d0800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180395: Wed Aug 26 19:43:59 2026 write: IOPS=95, BW=381KiB/s (390kB/s)(10.0MiB/26881msec); 0 zone resets slat (usec): min=30, max=20562, avg=217.64, stdev=958.23 clat (msec): min=22, max=296, avg=166.36, stdev=36.03 lat (msec): min=23, max=296, avg=166.58, stdev=35.99 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 116], 10.00th=[ 127], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 243, max= 512, per=24.87%, avg=378.79, stdev=67.67, samples=53 iops : min= 60, max= 128, avg=94.36, stdev=17.00, samples=53 lat (msec) : 50=0.12%, 100=0.90%, 250=96.48%, 500=2.50% cpu : usr=0.20%, sys=0.96%, ctx=2747, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180396: Wed Aug 26 19:43:59 2026 write: IOPS=95, BW=383KiB/s (392kB/s)(10.0MiB/26769msec); 0 zone resets slat (usec): min=29, max=10924, avg=159.08, stdev=470.49 clat (msec): min=23, max=312, avg=166.28, stdev=37.38 lat (msec): min=24, max=312, avg=166.44, stdev=37.38 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 116], 10.00th=[ 127], 20.00th=[ 138], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 215], 95.00th=[ 234], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 254, max= 520, per=25.11%, avg=382.40, stdev=68.27, samples=53 iops : min= 63, max= 130, avg=95.26, stdev=17.13, samples=53 lat (msec) : 50=0.12%, 100=1.88%, 250=95.20%, 500=2.81% cpu : usr=0.13%, sys=0.84%, ctx=2581, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180397: Wed Aug 26 19:43:59 2026 write: IOPS=95, BW=383KiB/s (392kB/s)(10.0MiB/26756msec); 0 zone resets slat (usec): min=27, max=18131, avg=214.81, stdev=812.49 clat (msec): min=46, max=326, avg=166.65, stdev=36.67 lat (msec): min=46, max=326, avg=166.87, stdev=36.67 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 116], 10.00th=[ 126], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 218], 95.00th=[ 236], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 326] bw ( KiB/s): min= 253, max= 520, per=25.10%, avg=382.32, stdev=61.56, samples=53 iops : min= 63, max= 130, avg=95.25, stdev=15.45, samples=53 lat (msec) : 50=0.04%, 100=1.45%, 250=95.43%, 500=3.09% cpu : usr=0.22%, sys=0.92%, ctx=2624, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180398: Wed Aug 26 19:43:59 2026 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26825msec); 0 zone resets slat (usec): min=29, max=23305, avg=190.51, stdev=733.11 clat (msec): min=50, max=301, avg=166.16, stdev=35.45 lat (msec): min=50, max=301, avg=166.36, stdev=35.45 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 118], 10.00th=[ 128], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 230], | 99.00th=[ 268], 99.50th=[ 275], 99.90th=[ 284], 99.95th=[ 292], | 99.99th=[ 300] bw ( KiB/s): min= 237, max= 520, per=25.02%, avg=381.08, stdev=61.03, samples=53 iops : min= 59, max= 130, avg=94.94, stdev=15.29, samples=53 lat (msec) : 100=0.94%, 250=96.72%, 500=2.34% cpu : usr=0.13%, sys=0.84%, ctx=2633, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1524KiB/s (1560kB/s), 381KiB/s-383KiB/s (390kB/s-392kB/s), io=40.0MiB (41.9MB), run=26756-26881msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180413: Wed Aug 26 19:44:38 2026 read: IOPS=43, BW=175KiB/s (179kB/s)(5048KiB/28863msec) slat (usec): min=56, max=56125, avg=12452.67, stdev=7236.73 clat (msec): min=14, max=347, avg=171.03, stdev=40.81 lat (msec): min=27, max=371, avg=183.49, stdev=42.30 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 114], 10.00th=[ 126], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 111, max= 256, per=25.21%, avg=174.47, stdev=31.84, samples=57 iops : min= 27, max= 64, avg=43.25, stdev= 8.05, samples=57 write: IOPS=44, BW=180KiB/s (184kB/s)(5192KiB/28863msec); 0 zone resets slat (usec): min=35, max=13650, avg=183.15, stdev=631.95 clat (msec): min=70, max=315, avg=176.78, stdev=38.66 lat (msec): min=70, max=315, avg=176.96, stdev=38.66 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 121], 10.00th=[ 131], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 230], 95.00th=[ 249], | 99.00th=[ 292], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 111, max= 262, per=24.98%, avg=179.84, stdev=39.31, samples=57 iops : min= 27, max= 65, avg=44.60, stdev= 9.89, samples=57 lat (msec) : 20=0.04%, 50=0.16%, 100=1.25%, 250=94.65%, 500=3.91% cpu : usr=0.31%, sys=1.62%, ctx=4381, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180414: Wed Aug 26 19:44:38 2026 read: IOPS=42, BW=172KiB/s (176kB/s)(4976KiB/28986msec) slat (usec): min=139, max=57676, avg=12399.32, stdev=6684.09 clat (msec): min=23, max=330, avg=169.91, stdev=41.16 lat (msec): min=23, max=341, avg=182.31, stdev=42.69 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 110], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 292], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 87, max= 261, per=24.55%, avg=169.86, stdev=37.09, samples=57 iops : min= 21, max= 65, avg=42.09, stdev= 9.37, samples=57 write: IOPS=45, BW=182KiB/s (186kB/s)(5264KiB/28986msec); 0 zone resets slat (usec): min=34, max=17703, avg=193.08, stdev=752.68 clat (msec): min=23, max=329, avg=177.81, stdev=36.57 lat (msec): min=23, max=330, avg=178.00, stdev=36.55 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 123], 10.00th=[ 136], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 241], | 99.00th=[ 279], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 95, max= 271, per=25.12%, avg=180.88, stdev=36.91, samples=57 iops : min= 23, max= 67, avg=44.84, stdev= 9.27, samples=57 lat (msec) : 50=0.23%, 100=1.52%, 250=94.26%, 500=3.98% cpu : usr=0.26%, sys=1.89%, ctx=4632, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180415: Wed Aug 26 19:44:38 2026 read: IOPS=43, BW=172KiB/s (176kB/s)(4984KiB/28938msec) slat (usec): min=2603, max=47334, avg=12668.46, stdev=6784.19 clat (msec): min=24, max=293, avg=170.42, stdev=38.76 lat (msec): min=39, max=318, avg=183.10, stdev=40.14 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 224], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 88, max= 251, per=24.89%, avg=172.23, stdev=38.02, samples=57 iops : min= 22, max= 62, avg=42.67, stdev= 9.53, samples=57 write: IOPS=45, BW=182KiB/s (186kB/s)(5256KiB/28938msec); 0 zone resets slat (usec): min=32, max=10487, avg=182.25, stdev=564.57 clat (msec): min=89, max=334, avg=176.75, stdev=37.53 lat (msec): min=90, max=334, avg=176.94, stdev=37.53 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 123], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 230], 95.00th=[ 249], | 99.00th=[ 271], 99.50th=[ 296], 99.90th=[ 326], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 111, max= 269, per=25.13%, avg=180.95, stdev=37.03, samples=57 iops : min= 27, max= 67, avg=44.86, stdev= 9.31, samples=57 lat (msec) : 50=0.04%, 100=1.29%, 250=94.61%, 500=4.06% cpu : usr=0.19%, sys=1.61%, ctx=4289, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180416: Wed Aug 26 19:44:38 2026 read: IOPS=43, BW=175KiB/s (179kB/s)(5060KiB/28911msec) slat (usec): min=2568, max=44763, avg=12417.38, stdev=6556.19 clat (msec): min=82, max=305, avg=169.73, stdev=38.24 lat (msec): min=87, max=334, avg=182.15, stdev=39.70 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 222], 95.00th=[ 241], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 79, max= 264, per=25.31%, avg=175.18, stdev=36.36, samples=57 iops : min= 19, max= 66, avg=43.42, stdev= 9.13, samples=57 write: IOPS=44, BW=179KiB/s (183kB/s)(5180KiB/28911msec); 0 zone resets slat (usec): min=31, max=17026, avg=171.40, stdev=640.62 clat (msec): min=65, max=310, avg=177.09, stdev=37.44 lat (msec): min=65, max=310, avg=177.26, stdev=37.44 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 230], 95.00th=[ 251], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 309], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 111, max= 272, per=24.77%, avg=178.32, stdev=33.48, samples=57 iops : min= 27, max= 68, avg=44.21, stdev= 8.47, samples=57 lat (msec) : 100=1.02%, 250=94.53%, 500=4.45% cpu : usr=0.26%, sys=1.71%, ctx=4473, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=692KiB/s (709kB/s), 172KiB/s-175KiB/s (176kB/s-179kB/s), io=19.6MiB (20.5MB), run=28863-28986msec WRITE: bw=721KiB/s (738kB/s), 179KiB/s-182KiB/s (183kB/s-186kB/s), io=20.4MiB (21.4MB), run=28863-28986msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180425: Wed Aug 26 19:44:48 2026 read: IOPS=2, BW=81.0MiB/s (85.9MB/s)(40.0MiB/488msec) slat (nsec): min=42139k, max=42139k, avg=42138504.00, stdev= 0.00 clat (nsec): min=424823k, max=424823k, avg=424823236.00, stdev= 0.00 lat (nsec): min=466966k, max=466966k, avg=466966326.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] lat (msec) : 500=100.00% cpu : usr=0.21%, sys=2.46%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=81.0MiB/s (85.9MB/s), 81.0MiB/s-81.0MiB/s (85.9MB/s-85.9MB/s), io=40.0MiB (41.9MB), run=488-488msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout