== sanity test 398c: run fio to test AIO ================= 12:23:18 (1790785398) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.403907 s, 104 MB/s osc.lustre-OST0000-osc-ffff8e426b22c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=151102: Wed Sep 30 12:23:55 2026 write: IOPS=92, BW=368KiB/s (377kB/s)(10.0MiB/27806msec); 0 zone resets slat (usec): min=28, max=15554, avg=210.82, stdev=735.24 clat (msec): min=19, max=311, avg=173.00, stdev=35.34 lat (msec): min=19, max=311, avg=173.22, stdev=35.31 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 115], 10.00th=[ 131], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 215], 95.00th=[ 224], | 99.00th=[ 247], 99.50th=[ 275], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 272, max= 547, per=24.93%, avg=366.78, stdev=55.38, samples=55 iops : min= 68, max= 136, avg=91.38, stdev=13.86, samples=55 lat (msec) : 20=0.04%, 50=0.86%, 100=1.68%, 250=96.68%, 500=0.74% cpu : usr=0.14%, sys=0.91%, ctx=2645, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151103: Wed Sep 30 12:23:55 2026 write: IOPS=92, BW=368KiB/s (377kB/s)(10.0MiB/27790msec); 0 zone resets slat (usec): min=29, max=10462, avg=181.57, stdev=561.76 clat (msec): min=24, max=314, avg=173.33, stdev=34.54 lat (msec): min=24, max=314, avg=173.52, stdev=34.54 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 115], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 247], 99.50th=[ 253], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 284, max= 659, per=24.96%, avg=367.09, stdev=58.52, samples=55 iops : min= 71, max= 164, avg=91.45, stdev=14.62, samples=55 lat (msec) : 50=0.04%, 100=3.12%, 250=96.21%, 500=0.62% cpu : usr=0.17%, sys=0.94%, ctx=2777, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151104: Wed Sep 30 12:23:55 2026 write: IOPS=91, BW=368KiB/s (377kB/s)(10.0MiB/27832msec); 0 zone resets slat (usec): min=29, max=13430, avg=175.98, stdev=509.03 clat (msec): min=8, max=267, avg=172.93, stdev=33.84 lat (msec): min=10, max=267, avg=173.11, stdev=33.80 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 113], 10.00th=[ 130], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 192], 80.00th=[ 201], 90.00th=[ 213], 95.00th=[ 222], | 99.00th=[ 243], 99.50th=[ 245], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 285, max= 509, per=24.78%, avg=364.55, stdev=50.64, samples=55 iops : min= 71, max= 127, avg=90.84, stdev=12.72, samples=55 lat (msec) : 10=0.04%, 20=0.16%, 50=0.12%, 100=1.88%, 250=97.38% lat (msec) : 500=0.43% cpu : usr=0.17%, sys=0.89%, ctx=2436, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151105: Wed Sep 30 12:23:55 2026 write: IOPS=91, BW=368KiB/s (377kB/s)(10.0MiB/27841msec); 0 zone resets slat (usec): min=29, max=14641, avg=206.53, stdev=732.25 clat (msec): min=41, max=272, avg=173.15, stdev=33.41 lat (msec): min=42, max=272, avg=173.36, stdev=33.40 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 114], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 184], | 70.00th=[ 192], 80.00th=[ 201], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 249], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 271] bw ( KiB/s): min= 255, max= 544, per=24.90%, avg=366.33, stdev=59.96, samples=55 iops : min= 63, max= 136, avg=91.29, stdev=15.00, samples=55 lat (msec) : 50=0.08%, 100=3.05%, 250=96.29%, 500=0.59% cpu : usr=0.15%, sys=0.91%, ctx=2938, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1471KiB/s (1507kB/s), 368KiB/s-368KiB/s (377kB/s-377kB/s), io=40.0MiB (41.9MB), run=27790-27841msec 22% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151121: Wed Sep 30 12:24:37 2026 read: IOPS=36, BW=145KiB/s (149kB/s)(5048KiB/34712msec) slat (usec): min=3049, max=68662, avg=14556.88, stdev=8524.83 clat (msec): min=77, max=453, avg=206.85, stdev=62.13 lat (msec): min=97, max=472, avg=221.42, stdev=64.46 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 209], | 70.00th=[ 232], 80.00th=[ 259], 90.00th=[ 292], 95.00th=[ 321], | 99.00th=[ 397], 99.50th=[ 414], 99.90th=[ 451], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 55, max= 223, per=25.16%, avg=145.15, stdev=38.64, samples=68 iops : min= 13, max= 55, avg=35.84, stdev= 9.67, samples=68 write: IOPS=37, BW=150KiB/s (153kB/s)(5192KiB/34712msec); 0 zone resets slat (usec): min=34, max=13751, avg=266.90, stdev=810.81 clat (msec): min=99, max=450, avg=210.06, stdev=61.77 lat (msec): min=99, max=450, avg=210.33, stdev=61.78 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 199], 60.00th=[ 213], | 70.00th=[ 232], 80.00th=[ 257], 90.00th=[ 300], 95.00th=[ 338], | 99.00th=[ 393], 99.50th=[ 405], 99.90th=[ 430], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 63, max= 295, per=24.90%, avg=149.65, stdev=47.47, samples=68 iops : min= 15, max= 73, avg=36.96, stdev=11.90, samples=68 lat (msec) : 100=0.47%, 250=76.80%, 500=22.73% cpu : usr=0.22%, sys=1.54%, ctx=4671, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151122: Wed Sep 30 12:24:37 2026 read: IOPS=35, BW=143KiB/s (147kB/s)(4976KiB/34736msec) slat (usec): min=2556, max=69265, avg=14389.26, stdev=8474.04 clat (msec): min=82, max=473, avg=205.48, stdev=62.51 lat (msec): min=89, max=499, avg=219.89, stdev=65.30 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 209], | 70.00th=[ 228], 80.00th=[ 255], 90.00th=[ 296], 95.00th=[ 326], | 99.00th=[ 401], 99.50th=[ 426], 99.90th=[ 464], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 55, max= 248, per=24.72%, avg=142.62, stdev=43.51, samples=68 iops : min= 13, max= 62, avg=35.19, stdev=10.94, samples=68 write: IOPS=37, BW=152KiB/s (155kB/s)(5264KiB/34736msec); 0 zone resets slat (usec): min=32, max=20541, avg=254.39, stdev=910.18 clat (msec): min=99, max=441, avg=212.22, stdev=62.08 lat (msec): min=99, max=441, avg=212.48, stdev=62.08 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 257], 90.00th=[ 305], 95.00th=[ 338], | 99.00th=[ 405], 99.50th=[ 418], 99.90th=[ 430], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 48, max= 287, per=25.26%, avg=151.84, stdev=44.70, samples=68 iops : min= 12, max= 71, avg=37.51, stdev=11.17, samples=68 lat (msec) : 100=0.23%, 250=77.97%, 500=21.80% cpu : usr=0.23%, sys=1.48%, ctx=4506, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151123: Wed Sep 30 12:24:37 2026 read: IOPS=36, BW=144KiB/s (148kB/s)(4984KiB/34555msec) slat (usec): min=3107, max=80024, avg=14345.31, stdev=8352.83 clat (msec): min=11, max=474, avg=204.89, stdev=63.95 lat (msec): min=25, max=492, avg=219.25, stdev=66.76 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 228], 80.00th=[ 255], 90.00th=[ 292], 95.00th=[ 330], | 99.00th=[ 388], 99.50th=[ 422], 99.90th=[ 472], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 56, max= 248, per=25.10%, avg=144.85, stdev=44.33, samples=68 iops : min= 14, max= 62, avg=35.75, stdev=11.14, samples=68 write: IOPS=38, BW=152KiB/s (156kB/s)(5256KiB/34555msec); 0 zone resets slat (usec): min=31, max=21890, avg=243.67, stdev=848.09 clat (msec): min=25, max=458, avg=210.28, stdev=61.54 lat (msec): min=25, max=458, avg=210.52, stdev=61.55 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 136], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 230], 80.00th=[ 255], 90.00th=[ 296], 95.00th=[ 334], | 99.00th=[ 405], 99.50th=[ 418], 99.90th=[ 456], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 55, max= 287, per=25.24%, avg=151.68, stdev=46.43, samples=68 iops : min= 13, max= 71, avg=37.47, stdev=11.64, samples=68 lat (msec) : 20=0.04%, 50=0.04%, 100=0.31%, 250=78.01%, 500=21.60% cpu : usr=0.23%, sys=1.48%, ctx=4607, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151124: Wed Sep 30 12:24:37 2026 read: IOPS=36, BW=148KiB/s (151kB/s)(5060KiB/34214msec) slat (usec): min=64, max=62089, avg=14351.57, stdev=8475.49 clat (msec): min=23, max=564, avg=205.84, stdev=65.28 lat (msec): min=23, max=579, avg=220.19, stdev=68.04 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 123], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 209], | 70.00th=[ 228], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 321], | 99.00th=[ 405], 99.50th=[ 430], 99.90th=[ 510], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 71, max= 232, per=25.77%, avg=148.70, stdev=36.23, samples=67 iops : min= 17, max= 58, avg=36.72, stdev= 9.09, samples=67 write: IOPS=37, BW=151KiB/s (155kB/s)(5180KiB/34214msec); 0 zone resets slat (usec): min=33, max=15883, avg=263.16, stdev=940.31 clat (msec): min=25, max=580, avg=206.81, stdev=64.45 lat (msec): min=25, max=580, avg=207.08, stdev=64.41 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 226], 80.00th=[ 253], 90.00th=[ 296], 95.00th=[ 330], | 99.00th=[ 397], 99.50th=[ 430], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 39, max= 295, per=25.17%, avg=151.27, stdev=50.79, samples=67 iops : min= 9, max= 73, avg=37.36, stdev=12.70, samples=67 lat (msec) : 50=0.23%, 100=0.43%, 250=77.66%, 500=21.48%, 750=0.20% cpu : usr=0.24%, sys=1.57%, ctx=4276, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=578KiB/s (592kB/s), 143KiB/s-148KiB/s (147kB/s-151kB/s), io=19.6MiB (20.5MB), run=34214-34736msec WRITE: bw=601KiB/s (616kB/s), 150KiB/s-152KiB/s (153kB/s-156kB/s), io=20.4MiB (21.4MB), run=34214-34736msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=151134: Wed Sep 30 12:24:50 2026 read: IOPS=1, BW=73.4MiB/s (76.0MB/s)(40.0MiB/545msec) slat (nsec): min=28533k, max=28533k, avg=28533169.00, stdev= 0.00 clat (nsec): min=484050k, max=484050k, avg=484050015.00, stdev= 0.00 lat (nsec): min=512589k, max=512589k, avg=512588924.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.76%, ctx=12, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=73.4MiB/s (76.0MB/s), 73.4MiB/s-73.4MiB/s (76.0MB/s-76.0MB/s), io=40.0MiB (41.9MB), run=545-545msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout