== sanity test 398c: run fio to test AIO ================= 22:46:32 (1781232392) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.618062 s, 67.9 MB/s osc.lustre-OST0000-osc-ffff9de748bb9000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180021: Thu Jun 11 22:47:16 2026 write: IOPS=78, BW=312KiB/s (320kB/s)(10.0MiB/32784msec); 0 zone resets slat (usec): min=52, max=28416, avg=436.49, stdev=1379.37 clat (msec): min=16, max=363, avg=204.12, stdev=48.11 lat (msec): min=16, max=363, avg=204.56, stdev=48.03 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 224], 80.00th=[ 245], 90.00th=[ 268], 95.00th=[ 284], | 99.00th=[ 342], 99.50th=[ 355], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 168, max= 501, per=24.96%, avg=310.54, stdev=65.09, samples=65 iops : min= 42, max= 125, avg=77.26, stdev=16.30, samples=65 lat (msec) : 20=0.04%, 50=0.12%, 100=0.66%, 250=82.23%, 500=16.95% cpu : usr=0.15%, sys=1.38%, ctx=2684, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180022: Thu Jun 11 22:47:16 2026 write: IOPS=77, BW=311KiB/s (319kB/s)(10.0MiB/32900msec); 0 zone resets slat (usec): min=52, max=18500, avg=432.19, stdev=1262.72 clat (msec): min=22, max=366, avg=203.99, stdev=47.09 lat (msec): min=22, max=367, avg=204.42, stdev=47.08 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 264], 95.00th=[ 288], | 99.00th=[ 321], 99.50th=[ 326], 99.90th=[ 342], 99.95th=[ 359], | 99.99th=[ 368] bw ( KiB/s): min= 160, max= 501, per=24.83%, avg=308.89, stdev=61.48, samples=65 iops : min= 40, max= 125, avg=76.86, stdev=15.37, samples=65 lat (msec) : 50=0.08%, 100=0.59%, 250=82.89%, 500=16.45% cpu : usr=0.17%, sys=1.09%, ctx=2413, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180023: Thu Jun 11 22:47:16 2026 write: IOPS=77, BW=311KiB/s (319kB/s)(10.0MiB/32882msec); 0 zone resets slat (usec): min=51, max=16851, avg=456.66, stdev=1225.98 clat (msec): min=30, max=363, avg=203.95, stdev=46.61 lat (msec): min=30, max=363, avg=204.41, stdev=46.56 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 321], 99.50th=[ 347], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 192, max= 462, per=24.96%, avg=310.46, stdev=54.92, samples=65 iops : min= 48, max= 115, avg=77.22, stdev=13.72, samples=65 lat (msec) : 50=0.16%, 100=0.78%, 250=83.63%, 500=15.43% cpu : usr=0.18%, sys=1.38%, ctx=2918, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180024: Thu Jun 11 22:47:16 2026 write: IOPS=78, BW=312KiB/s (320kB/s)(10.0MiB/32801msec); 0 zone resets slat (usec): min=55, max=17750, avg=441.27, stdev=1180.53 clat (msec): min=25, max=366, avg=203.48, stdev=46.92 lat (msec): min=27, max=367, avg=203.93, stdev=46.94 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 338], 99.95th=[ 347], | 99.99th=[ 368] bw ( KiB/s): min= 168, max= 485, per=24.98%, avg=310.71, stdev=62.33, samples=65 iops : min= 42, max= 121, avg=77.29, stdev=15.60, samples=65 lat (msec) : 50=0.12%, 100=1.45%, 250=81.45%, 500=16.99% cpu : usr=0.25%, sys=1.20%, ctx=2806, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1245KiB/s (1275kB/s), 311KiB/s-312KiB/s (319kB/s-320kB/s), io=40.0MiB (41.9MB), run=32784-32900msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180040: Thu Jun 11 22:48:04 2026 read: IOPS=33, BW=133KiB/s (136kB/s)(5048KiB/37998msec) slat (usec): min=3482, max=66933, avg=17597.22, stdev=9276.46 clat (msec): min=42, max=410, avg=225.98, stdev=49.51 lat (msec): min=56, max=441, avg=243.59, stdev=51.16 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 150], 10.00th=[ 169], 20.00th=[ 184], | 30.00th=[ 199], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 236], | 70.00th=[ 249], 80.00th=[ 271], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 47, max= 207, per=25.02%, avg=132.09, stdev=26.68, samples=75 iops : min= 11, max= 51, avg=32.57, stdev= 6.69, samples=75 write: IOPS=34, BW=137KiB/s (140kB/s)(5192KiB/37998msec); 0 zone resets slat (usec): min=57, max=15161, avg=418.97, stdev=1025.89 clat (msec): min=112, max=416, avg=228.00, stdev=47.42 lat (msec): min=112, max=417, avg=228.42, stdev=47.45 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 159], 10.00th=[ 171], 20.00th=[ 188], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 222], 60.00th=[ 236], | 70.00th=[ 251], 80.00th=[ 268], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 351], 99.50th=[ 359], 99.90th=[ 405], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 42, max= 234, per=24.81%, avg=136.20, stdev=34.85, samples=75 iops : min= 10, max= 58, avg=33.60, stdev= 8.75, samples=75 lat (msec) : 50=0.04%, 100=0.04%, 250=70.16%, 500=29.77% cpu : usr=0.23%, sys=1.92%, ctx=4426, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180041: Thu Jun 11 22:48:04 2026 read: IOPS=32, BW=131KiB/s (135kB/s)(4976KiB/37877msec) slat (usec): min=3235, max=86031, avg=17153.88, stdev=9607.50 clat (msec): min=13, max=453, avg=224.90, stdev=52.29 lat (msec): min=21, max=478, avg=242.06, stdev=53.98 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 232], | 70.00th=[ 249], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 376], 99.50th=[ 384], 99.90th=[ 439], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 63, max= 200, per=24.67%, avg=130.27, stdev=30.48, samples=75 iops : min= 15, max= 50, avg=32.11, stdev= 7.70, samples=75 write: IOPS=34, BW=139KiB/s (142kB/s)(5264KiB/37877msec); 0 zone resets slat (usec): min=59, max=16724, avg=443.83, stdev=1107.09 clat (msec): min=40, max=453, avg=229.34, stdev=48.25 lat (msec): min=40, max=453, avg=229.79, stdev=48.26 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 157], 10.00th=[ 174], 20.00th=[ 190], | 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 351], 99.50th=[ 372], 99.90th=[ 451], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 71, max= 232, per=25.21%, avg=138.39, stdev=34.05, samples=75 iops : min= 17, max= 58, avg=34.13, stdev= 8.48, samples=75 lat (msec) : 20=0.04%, 50=0.08%, 100=0.08%, 250=69.34%, 500=30.47% cpu : usr=0.25%, sys=1.99%, ctx=4475, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180042: Thu Jun 11 22:48:04 2026 read: IOPS=32, BW=132KiB/s (135kB/s)(4984KiB/37874msec) slat (usec): min=91, max=72974, avg=17332.57, stdev=9331.00 clat (msec): min=21, max=436, avg=225.97, stdev=53.74 lat (msec): min=39, max=448, avg=243.31, stdev=55.19 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 148], 10.00th=[ 161], 20.00th=[ 184], | 30.00th=[ 199], 40.00th=[ 209], 50.00th=[ 222], 60.00th=[ 234], | 70.00th=[ 249], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 368], 99.50th=[ 405], 99.90th=[ 426], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 79, max= 215, per=24.82%, avg=131.05, stdev=27.74, samples=75 iops : min= 19, max= 53, avg=32.31, stdev= 6.96, samples=75 write: IOPS=34, BW=139KiB/s (142kB/s)(5256KiB/37874msec); 0 zone resets slat (usec): min=56, max=16261, avg=443.38, stdev=1227.48 clat (msec): min=100, max=377, avg=229.38, stdev=46.03 lat (msec): min=101, max=378, avg=229.83, stdev=46.10 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 228], 60.00th=[ 239], | 70.00th=[ 251], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 355], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 55, max= 223, per=25.03%, avg=137.40, stdev=38.59, samples=75 iops : min= 13, max= 55, avg=33.89, stdev= 9.66, samples=75 lat (msec) : 50=0.16%, 100=0.23%, 250=69.92%, 500=29.69% cpu : usr=0.26%, sys=1.99%, ctx=4344, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180043: Thu Jun 11 22:48:04 2026 read: IOPS=33, BW=133KiB/s (136kB/s)(5060KiB/37982msec) slat (usec): min=3208, max=68306, avg=17011.27, stdev=9050.38 clat (msec): min=53, max=407, avg=223.32, stdev=49.35 lat (msec): min=81, max=417, avg=240.35, stdev=50.91 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 150], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 220], 60.00th=[ 232], | 70.00th=[ 247], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 397], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 49, max= 205, per=25.15%, avg=132.79, stdev=31.65, samples=75 iops : min= 12, max= 51, avg=32.76, stdev= 7.96, samples=75 write: IOPS=34, BW=136KiB/s (140kB/s)(5180KiB/37982msec); 0 zone resets slat (usec): min=54, max=9167, avg=419.31, stdev=943.76 clat (msec): min=41, max=377, avg=231.03, stdev=47.51 lat (msec): min=41, max=377, avg=231.45, stdev=47.52 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 161], 10.00th=[ 174], 20.00th=[ 188], | 30.00th=[ 201], 40.00th=[ 215], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 255], 80.00th=[ 275], 90.00th=[ 296], 95.00th=[ 313], | 99.00th=[ 342], 99.50th=[ 355], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 55, max= 242, per=24.76%, avg=135.92, stdev=34.09, samples=75 iops : min= 13, max= 60, avg=33.52, stdev= 8.54, samples=75 lat (msec) : 50=0.04%, 100=0.12%, 250=68.95%, 500=30.90% cpu : usr=0.23%, sys=1.89%, ctx=4619, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=528KiB/s (541kB/s), 131KiB/s-133KiB/s (135kB/s-136kB/s), io=19.6MiB (20.5MB), run=37874-37998msec WRITE: bw=550KiB/s (563kB/s), 136KiB/s-139KiB/s (140kB/s-142kB/s), io=20.4MiB (21.4MB), run=37874-37998msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180053: Thu Jun 11 22:48:16 2026 read: IOPS=1, BW=65.0MiB/s (68.2MB/s)(40.0MiB/615msec) slat (nsec): min=72361k, max=72361k, avg=72361017.00, stdev= 0.00 clat (nsec): min=492263k, max=492263k, avg=492262902.00, stdev= 0.00 lat (nsec): min=564629k, max=564629k, avg=564628607.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 493], 10.00th=[ 493], 20.00th=[ 493], | 30.00th=[ 493], 40.00th=[ 493], 50.00th=[ 493], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.95%, ctx=9, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.0MiB/s (68.2MB/s), 65.0MiB/s-65.0MiB/s (68.2MB/s-68.2MB/s), io=40.0MiB (41.9MB), run=615-615msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout