== sanity test 398c: run fio to test AIO ================= 15:03:51 (1783969431) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.835697 s, 50.2 MB/s osc.lustre-OST0000-osc-ffff976a31110000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180323: Mon Jul 13 15:04:37 2026 write: IOPS=77, BW=308KiB/s (316kB/s)(10.0MiB/33196msec); 0 zone resets slat (usec): min=35, max=16561, avg=278.11, stdev=837.59 clat (msec): min=29, max=467, avg=205.91, stdev=75.00 lat (msec): min=30, max=467, avg=206.19, stdev=75.00 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 106], 10.00th=[ 120], 20.00th=[ 138], | 30.00th=[ 155], 40.00th=[ 184], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 234], 80.00th=[ 259], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 430], 99.50th=[ 451], 99.90th=[ 460], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 127, max= 648, per=24.99%, avg=308.18, stdev=107.19, samples=65 iops : min= 31, max= 162, avg=76.69, stdev=26.84, samples=65 lat (msec) : 50=0.04%, 100=3.48%, 250=73.59%, 500=22.89% cpu : usr=0.14%, sys=0.99%, ctx=2209, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180324: Mon Jul 13 15:04:37 2026 write: IOPS=77, BW=310KiB/s (318kB/s)(10.0MiB/32984msec); 0 zone resets slat (usec): min=36, max=30519, avg=347.33, stdev=1285.74 clat (msec): min=25, max=532, avg=205.25, stdev=76.93 lat (msec): min=26, max=533, avg=205.60, stdev=76.96 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 104], 10.00th=[ 118], 20.00th=[ 138], | 30.00th=[ 157], 40.00th=[ 178], 50.00th=[ 197], 60.00th=[ 213], | 70.00th=[ 234], 80.00th=[ 264], 90.00th=[ 305], 95.00th=[ 347], | 99.00th=[ 451], 99.50th=[ 493], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 141, max= 648, per=25.21%, avg=310.82, stdev=100.75, samples=65 iops : min= 35, max= 162, avg=77.31, stdev=25.24, samples=65 lat (msec) : 50=0.12%, 100=4.41%, 250=71.13%, 500=23.95%, 750=0.39% cpu : usr=0.19%, sys=1.06%, ctx=2373, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180325: Mon Jul 13 15:04:37 2026 write: IOPS=77, BW=309KiB/s (316kB/s)(10.0MiB/33177msec); 0 zone resets slat (usec): min=36, max=23061, avg=289.66, stdev=1079.06 clat (msec): min=35, max=514, avg=205.89, stdev=75.15 lat (msec): min=35, max=514, avg=206.18, stdev=75.20 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 107], 10.00th=[ 120], 20.00th=[ 140], | 30.00th=[ 155], 40.00th=[ 178], 50.00th=[ 201], 60.00th=[ 218], | 70.00th=[ 234], 80.00th=[ 264], 90.00th=[ 309], 95.00th=[ 342], | 99.00th=[ 430], 99.50th=[ 447], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 124, max= 608, per=25.03%, avg=308.58, stdev=108.17, samples=65 iops : min= 31, max= 152, avg=76.77, stdev=27.08, samples=65 lat (msec) : 50=0.08%, 100=3.44%, 250=71.45%, 500=24.92%, 750=0.12% cpu : usr=0.15%, sys=1.10%, ctx=2314, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180326: Mon Jul 13 15:04:37 2026 write: IOPS=77, BW=309KiB/s (316kB/s)(10.0MiB/33166msec); 0 zone resets slat (usec): min=35, max=20004, avg=296.89, stdev=1042.45 clat (msec): min=14, max=538, avg=205.34, stdev=75.40 lat (msec): min=15, max=538, avg=205.64, stdev=75.43 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 105], 10.00th=[ 117], 20.00th=[ 140], | 30.00th=[ 159], 40.00th=[ 180], 50.00th=[ 199], 60.00th=[ 213], | 70.00th=[ 236], 80.00th=[ 268], 90.00th=[ 305], 95.00th=[ 342], | 99.00th=[ 426], 99.50th=[ 489], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 542] bw ( KiB/s): min= 126, max= 640, per=25.03%, avg=308.63, stdev=105.58, samples=65 iops : min= 31, max= 160, avg=76.74, stdev=26.47, samples=65 lat (msec) : 20=0.04%, 50=0.20%, 100=3.63%, 250=71.88%, 500=24.10% lat (msec) : 750=0.16% cpu : usr=0.17%, sys=0.97%, ctx=2454, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1234KiB/s (1263kB/s), 308KiB/s-310KiB/s (316kB/s-318kB/s), io=40.0MiB (41.9MB), run=32984-33196msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180341: Mon Jul 13 15:05:23 2026 read: IOPS=34, BW=139KiB/s (142kB/s)(5048KiB/36395msec) slat (usec): min=2279, max=71413, avg=16321.05, stdev=9143.48 clat (msec): min=52, max=404, avg=214.17, stdev=49.20 lat (msec): min=75, max=414, avg=230.50, stdev=51.52 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 136], 10.00th=[ 153], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 372], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 15, max= 238, per=24.94%, avg=137.44, stdev=38.94, samples=72 iops : min= 3, max= 59, avg=33.85, stdev= 9.78, samples=72 write: IOPS=35, BW=143KiB/s (146kB/s)(5192KiB/36395msec); 0 zone resets slat (usec): min=41, max=13516, avg=341.39, stdev=881.06 clat (msec): min=52, max=373, avg=219.77, stdev=47.83 lat (msec): min=52, max=373, avg=220.11, stdev=47.83 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 144], 10.00th=[ 161], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 245], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 359], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 71, max= 231, per=25.06%, avg=143.82, stdev=33.86, samples=71 iops : min= 17, max= 57, avg=35.44, stdev= 8.50, samples=71 lat (msec) : 100=0.43%, 250=75.08%, 500=24.49% cpu : usr=0.25%, sys=1.95%, ctx=4258, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180342: Mon Jul 13 15:05:23 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(4976KiB/36135msec) slat (usec): min=2871, max=82380, avg=16389.36, stdev=9182.37 clat (msec): min=37, max=447, avg=212.98, stdev=51.24 lat (msec): min=54, max=466, avg=229.37, stdev=53.11 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 380], 99.90th=[ 435], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 23, max= 222, per=24.91%, avg=137.27, stdev=35.96, samples=71 iops : min= 5, max= 55, avg=33.80, stdev= 9.05, samples=71 write: IOPS=36, BW=146KiB/s (149kB/s)(5264KiB/36135msec); 0 zone resets slat (usec): min=52, max=22373, avg=355.94, stdev=1181.98 clat (msec): min=35, max=474, avg=218.93, stdev=48.91 lat (msec): min=36, max=474, avg=219.29, stdev=48.95 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 142], 10.00th=[ 161], 20.00th=[ 178], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 243], 80.00th=[ 257], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 338], 99.50th=[ 355], 99.90th=[ 468], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 72, max= 255, per=25.45%, avg=146.07, stdev=38.70, samples=71 iops : min= 18, max= 63, avg=36.01, stdev= 9.65, samples=71 lat (msec) : 50=0.12%, 100=0.51%, 250=75.43%, 500=23.95% cpu : usr=0.27%, sys=1.84%, ctx=4365, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180343: Mon Jul 13 15:05:23 2026 read: IOPS=34, BW=137KiB/s (141kB/s)(4984KiB/36292msec) slat (usec): min=3777, max=72048, avg=16301.69, stdev=9345.07 clat (msec): min=27, max=401, avg=215.11, stdev=48.95 lat (msec): min=43, max=412, avg=231.43, stdev=50.52 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 215], 60.00th=[ 226], | 70.00th=[ 241], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 38, max= 222, per=24.80%, avg=136.62, stdev=35.95, samples=72 iops : min= 9, max= 55, avg=33.65, stdev= 9.03, samples=72 write: IOPS=36, BW=145KiB/s (148kB/s)(5256KiB/36292msec); 0 zone resets slat (usec): min=52, max=16840, avg=341.52, stdev=984.51 clat (msec): min=83, max=387, avg=219.01, stdev=50.00 lat (msec): min=83, max=387, avg=219.35, stdev=50.02 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 176], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 245], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 380], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 46, max= 270, per=25.04%, avg=143.71, stdev=41.53, samples=72 iops : min= 11, max= 67, avg=35.42, stdev=10.43, samples=72 lat (msec) : 50=0.08%, 100=0.47%, 250=74.84%, 500=24.61% cpu : usr=0.24%, sys=1.81%, ctx=4372, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180344: Mon Jul 13 15:05:23 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(5060KiB/35979msec) slat (usec): min=88, max=83452, avg=16042.85, stdev=8842.10 clat (msec): min=49, max=402, avg=213.37, stdev=51.58 lat (msec): min=49, max=425, avg=229.42, stdev=53.63 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 351], 99.50th=[ 372], 99.90th=[ 388], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 79, max= 255, per=25.49%, avg=140.44, stdev=35.84, samples=71 iops : min= 19, max= 63, avg=34.59, stdev= 8.96, samples=71 write: IOPS=35, BW=144KiB/s (147kB/s)(5180KiB/35979msec); 0 zone resets slat (usec): min=59, max=11325, avg=301.99, stdev=652.44 clat (msec): min=64, max=424, avg=219.35, stdev=51.56 lat (msec): min=64, max=424, avg=219.65, stdev=51.58 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 138], 10.00th=[ 157], 20.00th=[ 178], | 30.00th=[ 190], 40.00th=[ 205], 50.00th=[ 220], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 262], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 355], 99.50th=[ 393], 99.90th=[ 422], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 64, max= 238, per=25.06%, avg=143.86, stdev=35.40, samples=71 iops : min= 16, max= 59, avg=35.45, stdev= 8.88, samples=71 lat (msec) : 50=0.04%, 100=0.59%, 250=74.53%, 500=24.84% cpu : usr=0.26%, sys=1.73%, ctx=4331, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=551KiB/s (565kB/s), 137KiB/s-141KiB/s (141kB/s-144kB/s), io=19.6MiB (20.5MB), run=35979-36395msec WRITE: bw=574KiB/s (588kB/s), 143KiB/s-146KiB/s (146kB/s-149kB/s), io=20.4MiB (21.4MB), run=35979-36395msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180354: Mon Jul 13 15:05:35 2026 read: IOPS=1, BW=78.1MiB/s (81.9MB/s)(40.0MiB/512msec) slat (nsec): min=45320k, max=45320k, avg=45319760.00, stdev= 0.00 clat (nsec): min=436777k, max=436777k, avg=436777377.00, stdev= 0.00 lat (nsec): min=482102k, max=482102k, avg=482102244.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.74%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=78.1MiB/s (81.9MB/s), 78.1MiB/s-78.1MiB/s (81.9MB/s-81.9MB/s), io=40.0MiB (41.9MB), run=512-512msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout