== sanity test 398c: run fio to test AIO ================= 21:16:16 (1787447776) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.627077 s, 66.9 MB/s osc.lustre-OST0000-osc-ffff8ffd87939000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=178879: Sat Aug 22 21:16:54 2026 write: IOPS=90, BW=363KiB/s (372kB/s)(10.0MiB/28197msec); 0 zone resets slat (usec): min=22, max=24058, avg=203.58, stdev=910.80 clat (msec): min=77, max=363, avg=175.84, stdev=50.52 lat (msec): min=77, max=363, avg=176.04, stdev=50.53 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 97], 10.00th=[ 115], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 186], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 245], 95.00th=[ 266], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 252, max= 594, per=25.24%, avg=363.91, stdev=81.16, samples=55 iops : min= 63, max= 148, avg=90.60, stdev=20.32, samples=55 lat (msec) : 100=5.35%, 250=86.64%, 500=8.01% cpu : usr=0.18%, sys=0.84%, ctx=1458, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178880: Sat Aug 22 21:16:54 2026 write: IOPS=90, BW=361KiB/s (369kB/s)(10.0MiB/28386msec); 0 zone resets slat (usec): min=21, max=16729, avg=233.12, stdev=858.31 clat (msec): min=18, max=372, avg=176.06, stdev=50.80 lat (msec): min=18, max=372, avg=176.29, stdev=50.79 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 96], 10.00th=[ 115], 20.00th=[ 133], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 199], 80.00th=[ 218], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 237, max= 558, per=25.03%, avg=360.89, stdev=80.13, samples=56 iops : min= 59, max= 139, avg=89.84, stdev=20.00, samples=56 lat (msec) : 20=0.04%, 100=5.55%, 250=85.16%, 500=9.26% cpu : usr=0.22%, sys=1.03%, ctx=2320, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178881: Sat Aug 22 21:16:54 2026 write: IOPS=90, BW=361KiB/s (370kB/s)(10.0MiB/28354msec); 0 zone resets slat (usec): min=21, max=11406, avg=222.24, stdev=639.99 clat (msec): min=28, max=338, avg=175.73, stdev=50.02 lat (msec): min=28, max=338, avg=175.96, stdev=50.03 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 105], 10.00th=[ 115], 20.00th=[ 132], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 174], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 220], 90.00th=[ 241], 95.00th=[ 266], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 334], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 198, max= 542, per=25.02%, avg=360.86, stdev=83.08, samples=56 iops : min= 49, max= 135, avg=89.86, stdev=20.77, samples=56 lat (msec) : 50=0.16%, 100=4.30%, 250=87.58%, 500=7.97% cpu : usr=0.19%, sys=1.15%, ctx=2306, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178882: Sat Aug 22 21:16:54 2026 write: IOPS=90, BW=362KiB/s (370kB/s)(10.0MiB/28305msec); 0 zone resets slat (usec): min=23, max=13118, avg=276.14, stdev=872.43 clat (msec): min=17, max=428, avg=175.68, stdev=50.26 lat (msec): min=17, max=428, avg=175.96, stdev=50.24 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 99], 10.00th=[ 115], 20.00th=[ 132], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 174], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 245], 95.00th=[ 264], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 238, max= 590, per=25.03%, avg=360.95, stdev=80.86, samples=56 iops : min= 59, max= 147, avg=89.86, stdev=20.24, samples=56 lat (msec) : 20=0.04%, 50=0.16%, 100=4.96%, 250=86.84%, 500=8.01% cpu : usr=0.25%, sys=1.21%, ctx=2856, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1443KiB/s (1478kB/s), 361KiB/s-363KiB/s (369kB/s-372kB/s), io=40.0MiB (41.9MB), run=28197-28386msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=178897: Sat Aug 22 21:17:38 2026 read: IOPS=37, BW=150KiB/s (153kB/s)(5048KiB/33689msec) slat (msec): min=3, max=149, avg=15.76, stdev= 9.52 clat (msec): min=42, max=447, avg=196.79, stdev=51.26 lat (msec): min=47, max=472, avg=212.56, stdev=52.92 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 120], 10.00th=[ 132], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 182], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 69, max= 270, per=25.06%, avg=149.12, stdev=40.51, samples=66 iops : min= 17, max= 67, avg=36.79, stdev=10.18, samples=66 write: IOPS=38, BW=154KiB/s (158kB/s)(5192KiB/33689msec); 0 zone resets slat (usec): min=27, max=25770, avg=307.16, stdev=1151.57 clat (msec): min=89, max=418, avg=204.96, stdev=52.53 lat (msec): min=89, max=419, avg=205.27, stdev=52.50 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 388], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 48, max= 247, per=24.85%, avg=154.06, stdev=41.83, samples=66 iops : min= 12, max= 61, avg=38.02, stdev=10.49, samples=66 lat (msec) : 50=0.04%, 100=0.70%, 250=82.23%, 500=17.03% cpu : usr=0.31%, sys=1.91%, ctx=4126, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178898: Sat Aug 22 21:17:38 2026 read: IOPS=36, BW=148KiB/s (152kB/s)(4976KiB/33629msec) slat (usec): min=37, max=66456, avg=16066.37, stdev=8824.05 clat (msec): min=68, max=469, avg=198.99, stdev=51.70 lat (msec): min=86, max=483, avg=215.07, stdev=53.63 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 120], 10.00th=[ 134], 20.00th=[ 155], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 342], 99.50th=[ 355], 99.90th=[ 430], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 87, max= 280, per=24.81%, avg=147.61, stdev=39.41, samples=66 iops : min= 21, max= 70, avg=36.38, stdev= 9.94, samples=66 write: IOPS=39, BW=157KiB/s (160kB/s)(5264KiB/33629msec); 0 zone resets slat (usec): min=27, max=17649, avg=311.10, stdev=984.89 clat (msec): min=60, max=468, avg=204.76, stdev=50.93 lat (msec): min=60, max=468, avg=205.08, stdev=50.99 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 125], 10.00th=[ 142], 20.00th=[ 163], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 228], 80.00th=[ 243], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 351], 99.50th=[ 380], 99.90th=[ 393], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 79, max= 255, per=25.29%, avg=156.79, stdev=40.07, samples=66 iops : min= 19, max= 63, avg=38.70, stdev=10.03, samples=66 lat (msec) : 100=0.98%, 250=83.71%, 500=15.31% cpu : usr=0.27%, sys=1.81%, ctx=4178, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178899: Sat Aug 22 21:17:38 2026 read: IOPS=37, BW=148KiB/s (152kB/s)(4984KiB/33633msec) slat (usec): min=2959, max=70596, avg=15689.23, stdev=8851.78 clat (msec): min=24, max=443, avg=198.17, stdev=52.98 lat (msec): min=33, max=500, avg=213.87, stdev=55.07 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 117], 10.00th=[ 134], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 69, max= 256, per=24.96%, avg=148.52, stdev=39.93, samples=66 iops : min= 17, max= 64, avg=36.64, stdev=10.07, samples=66 write: IOPS=39, BW=156KiB/s (160kB/s)(5256KiB/33633msec); 0 zone resets slat (usec): min=28, max=15876, avg=313.47, stdev=996.60 clat (msec): min=34, max=438, avg=204.25, stdev=51.80 lat (msec): min=34, max=438, avg=204.56, stdev=51.82 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 125], 10.00th=[ 138], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 351], 99.50th=[ 393], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 39, max= 270, per=25.19%, avg=156.20, stdev=45.34, samples=66 iops : min= 9, max= 67, avg=38.56, stdev=11.36, samples=66 lat (msec) : 50=0.12%, 100=0.62%, 250=83.20%, 500=16.05% cpu : usr=0.31%, sys=1.72%, ctx=4102, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178900: Sat Aug 22 21:17:38 2026 read: IOPS=37, BW=150KiB/s (154kB/s)(5060KiB/33663msec) slat (msec): min=2, max=204, avg=15.83, stdev=10.95 clat (msec): min=61, max=480, avg=195.44, stdev=53.89 lat (msec): min=71, max=511, avg=211.28, stdev=56.16 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 118], 10.00th=[ 132], 20.00th=[ 150], | 30.00th=[ 165], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 355], 99.50th=[ 409], 99.90th=[ 472], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 63, max= 255, per=25.25%, avg=150.23, stdev=41.43, samples=66 iops : min= 15, max= 63, avg=37.05, stdev=10.41, samples=66 write: IOPS=38, BW=154KiB/s (158kB/s)(5180KiB/33663msec); 0 zone resets slat (usec): min=29, max=15526, avg=265.38, stdev=861.80 clat (msec): min=90, max=471, avg=206.63, stdev=53.52 lat (msec): min=90, max=471, avg=206.89, stdev=53.51 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 127], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 380], 99.50th=[ 405], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 80, max= 270, per=24.69%, avg=153.06, stdev=41.07, samples=66 iops : min= 20, max= 67, avg=37.76, stdev=10.33, samples=66 lat (msec) : 100=1.02%, 250=82.66%, 500=16.33% cpu : usr=0.29%, sys=1.84%, ctx=3770, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=596KiB/s (610kB/s), 148KiB/s-150KiB/s (152kB/s-154kB/s), io=19.6MiB (20.5MB), run=33629-33689msec WRITE: bw=620KiB/s (635kB/s), 154KiB/s-157KiB/s (158kB/s-160kB/s), io=20.4MiB (21.4MB), run=33629-33689msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=178909: Sat Aug 22 21:17:50 2026 read: IOPS=1, BW=63.5MiB/s (66.6MB/s)(40.0MiB/630msec) slat (nsec): min=25372k, max=25372k, avg=25371622.00, stdev= 0.00 clat (nsec): min=576338k, max=576338k, avg=576338208.00, stdev= 0.00 lat (nsec): min=601716k, max=601716k, avg=601716251.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 750=100.00% cpu : usr=0.16%, sys=2.07%, ctx=7, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.5MiB/s (66.6MB/s), 63.5MiB/s-63.5MiB/s (66.6MB/s-66.6MB/s), io=40.0MiB (41.9MB), run=630-630msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout