== sanity test 398c: run fio to test AIO ================= 10:46:44 (1786373204) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.447343 s, 93.8 MB/s osc.lustre-OST0000-osc-ffff9da7c02e7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147254: Mon Aug 10 10:48:19 2026 write: IOPS=29, BW=116KiB/s (119kB/s)(10.0MiB/88161msec); 0 zone resets slat (usec): min=36, max=11853, avg=268.62, stdev=599.71 clat (msec): min=89, max=1161, avg=549.95, stdev=143.47 lat (msec): min=90, max=1161, avg=550.22, stdev=143.51 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 326], 10.00th=[ 388], 20.00th=[ 439], | 30.00th=[ 477], 40.00th=[ 514], 50.00th=[ 550], 60.00th=[ 575], | 70.00th=[ 609], 80.00th=[ 659], 90.00th=[ 735], 95.00th=[ 793], | 99.00th=[ 936], 99.50th=[ 1036], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 16, max= 256, per=24.91%, avg=115.58, stdev=34.42, samples=175 iops : min= 4, max= 64, avg=28.55, stdev= 8.65, samples=175 lat (msec) : 100=0.62%, 250=0.66%, 500=35.82%, 750=54.34%, 1000=7.81% lat (msec) : 2000=0.74% cpu : usr=0.06%, sys=0.46%, ctx=3208, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147255: Mon Aug 10 10:48:19 2026 write: IOPS=29, BW=116KiB/s (119kB/s)(10.0MiB/88191msec); 0 zone resets slat (usec): min=35, max=18362, avg=272.29, stdev=648.87 clat (msec): min=98, max=1108, avg=550.31, stdev=140.15 lat (msec): min=100, max=1108, avg=550.58, stdev=140.13 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 330], 10.00th=[ 393], 20.00th=[ 443], | 30.00th=[ 481], 40.00th=[ 510], 50.00th=[ 542], 60.00th=[ 575], | 70.00th=[ 609], 80.00th=[ 659], 90.00th=[ 735], 95.00th=[ 785], | 99.00th=[ 927], 99.50th=[ 1020], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 256, per=24.91%, avg=115.57, stdev=34.61, samples=175 iops : min= 1, max= 64, avg=28.55, stdev= 8.69, samples=175 lat (msec) : 100=0.04%, 250=1.33%, 500=35.04%, 750=55.90%, 1000=6.88% lat (msec) : 2000=0.82% cpu : usr=0.07%, sys=0.48%, ctx=2746, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147256: Mon Aug 10 10:48:19 2026 write: IOPS=29, BW=116KiB/s (119kB/s)(10.0MiB/88050msec); 0 zone resets slat (usec): min=37, max=10447, avg=285.13, stdev=564.81 clat (msec): min=56, max=1115, avg=549.72, stdev=143.67 lat (msec): min=56, max=1115, avg=550.01, stdev=143.67 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 326], 10.00th=[ 384], 20.00th=[ 443], | 30.00th=[ 477], 40.00th=[ 514], 50.00th=[ 542], 60.00th=[ 575], | 70.00th=[ 609], 80.00th=[ 659], 90.00th=[ 735], 95.00th=[ 793], | 99.00th=[ 953], 99.50th=[ 1011], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 32, max= 278, per=24.89%, avg=115.49, stdev=34.48, samples=175 iops : min= 8, max= 69, avg=28.53, stdev= 8.62, samples=175 lat (msec) : 100=0.35%, 250=1.45%, 500=35.08%, 750=54.77%, 1000=7.73% lat (msec) : 2000=0.62% cpu : usr=0.07%, sys=0.50%, ctx=3135, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147257: Mon Aug 10 10:48:19 2026 write: IOPS=29, BW=116KiB/s (119kB/s)(10.0MiB/88163msec); 0 zone resets slat (usec): min=38, max=14032, avg=272.32, stdev=551.05 clat (msec): min=98, max=1190, avg=550.01, stdev=143.19 lat (msec): min=99, max=1191, avg=550.28, stdev=143.17 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 330], 10.00th=[ 393], 20.00th=[ 439], | 30.00th=[ 477], 40.00th=[ 510], 50.00th=[ 542], 60.00th=[ 575], | 70.00th=[ 609], 80.00th=[ 659], 90.00th=[ 735], 95.00th=[ 793], | 99.00th=[ 961], 99.50th=[ 1045], 99.90th=[ 1133], 99.95th=[ 1167], | 99.99th=[ 1183] bw ( KiB/s): min= 31, max= 253, per=24.91%, avg=115.57, stdev=34.23, samples=175 iops : min= 7, max= 63, avg=28.55, stdev= 8.57, samples=175 lat (msec) : 100=0.04%, 250=1.60%, 500=36.17%, 750=54.18%, 1000=7.30% lat (msec) : 2000=0.70% cpu : usr=0.06%, sys=0.48%, ctx=3031, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=464KiB/s (476kB/s), 116KiB/s-116KiB/s (119kB/s-119kB/s), io=40.0MiB (41.9MB), run=88050-88191msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147280: Mon Aug 10 10:49:16 2026 read: IOPS=24, BW=96.5KiB/s (98.8kB/s)(5048KiB/52320msec) slat (usec): min=1996, max=31300, avg=7127.20, stdev=3574.82 clat (msec): min=21, max=381, avg=204.23, stdev=52.52 lat (msec): min=29, max=397, avg=211.36, stdev=53.30 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 334], 99.50th=[ 347], 99.90th=[ 380], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 23, max= 176, per=25.03%, avg=95.87, stdev=31.22, samples=104 iops : min= 5, max= 44, avg=23.62, stdev= 7.82, samples=104 write: IOPS=24, BW=99.2KiB/s (102kB/s)(5192KiB/52320msec); 0 zone resets slat (usec): min=41, max=5395, avg=272.29, stdev=436.24 clat (msec): min=187, max=781, avg=438.04, stdev=110.18 lat (msec): min=187, max=781, avg=438.31, stdev=110.19 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 342], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 430], 60.00th=[ 456], | 70.00th=[ 489], 80.00th=[ 535], 90.00th=[ 584], 95.00th=[ 634], | 99.00th=[ 709], 99.50th=[ 743], 99.90th=[ 776], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 47, max= 160, per=24.72%, avg=98.39, stdev=24.19, samples=104 iops : min= 11, max= 40, avg=24.25, stdev= 6.11, samples=104 lat (msec) : 50=0.20%, 100=0.31%, 250=41.09%, 500=44.45%, 750=13.75% lat (msec) : 1000=0.20% cpu : usr=0.14%, sys=1.10%, ctx=5301, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147281: Mon Aug 10 10:49:16 2026 read: IOPS=23, BW=94.0KiB/s (97.3kB/s)(4976KiB/52382msec) slat (usec): min=1537, max=25845, avg=7186.38, stdev=3566.19 clat (msec): min=4, max=415, avg=204.22, stdev=53.40 lat (msec): min=12, max=427, avg=211.41, stdev=54.13 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 338], 99.50th=[ 355], 99.90th=[ 380], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 24, max= 192, per=24.61%, avg=94.27, stdev=33.26, samples=104 iops : min= 6, max= 48, avg=23.22, stdev= 8.30, samples=104 write: IOPS=25, BW=100KiB/s (103kB/s)(5264KiB/52382msec); 0 zone resets slat (usec): min=36, max=14225, avg=283.85, stdev=670.27 clat (msec): min=64, max=807, avg=435.98, stdev=115.77 lat (msec): min=65, max=807, avg=436.26, stdev=115.78 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 262], 10.00th=[ 300], 20.00th=[ 342], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 542], 90.00th=[ 584], 95.00th=[ 634], | 99.00th=[ 726], 99.50th=[ 768], 99.90th=[ 802], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 47, max= 160, per=25.05%, avg=99.71, stdev=28.35, samples=104 iops : min= 11, max= 40, avg=24.58, stdev= 7.18, samples=104 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=0.62%, 250=40.12% lat (msec) : 500=44.49%, 750=14.18%, 1000=0.43% cpu : usr=0.15%, sys=1.05%, ctx=5180, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147282: Mon Aug 10 10:49:16 2026 read: IOPS=23, BW=95.4KiB/s (97.7kB/s)(4984KiB/52241msec) slat (usec): min=48, max=40887, avg=7126.75, stdev=3748.02 clat (msec): min=9, max=539, avg=203.31, stdev=54.56 lat (msec): min=14, max=549, avg=210.44, stdev=55.42 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 338], 99.50th=[ 363], 99.90th=[ 531], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 32, max= 208, per=24.97%, avg=95.63, stdev=35.00, samples=103 iops : min= 8, max= 52, avg=23.55, stdev= 8.75, samples=103 write: IOPS=25, BW=101KiB/s (103kB/s)(5256KiB/52241msec); 0 zone resets slat (usec): min=38, max=5472, avg=273.23, stdev=437.41 clat (msec): min=77, max=787, avg=435.84, stdev=114.16 lat (msec): min=77, max=788, avg=436.12, stdev=114.18 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 264], 10.00th=[ 296], 20.00th=[ 342], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 489], 80.00th=[ 535], 90.00th=[ 592], 95.00th=[ 625], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 768], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 39, max= 166, per=25.10%, avg=99.88, stdev=26.92, samples=103 iops : min= 9, max= 41, avg=24.62, stdev= 6.79, samples=103 lat (msec) : 10=0.08%, 20=0.12%, 50=0.12%, 100=0.31%, 250=40.98% lat (msec) : 500=44.18%, 750=13.91%, 1000=0.31% cpu : usr=0.13%, sys=1.10%, ctx=5122, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147283: Mon Aug 10 10:49:16 2026 read: IOPS=24, BW=96.7KiB/s (99.0kB/s)(5060KiB/52319msec) slat (usec): min=1817, max=33444, avg=7165.24, stdev=3704.03 clat (msec): min=18, max=392, avg=203.48, stdev=53.52 lat (msec): min=26, max=400, avg=210.65, stdev=54.38 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 124], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 380], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 7, max= 208, per=25.09%, avg=96.11, stdev=35.52, samples=104 iops : min= 1, max= 52, avg=23.68, stdev= 8.92, samples=104 write: IOPS=24, BW=99.0KiB/s (101kB/s)(5180KiB/52319msec); 0 zone resets slat (usec): min=55, max=10131, avg=278.93, stdev=505.58 clat (msec): min=122, max=776, avg=439.25, stdev=112.46 lat (msec): min=122, max=776, avg=439.54, stdev=112.49 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 531], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 743], 99.50th=[ 751], 99.90th=[ 768], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 48, max= 192, per=24.66%, avg=98.15, stdev=26.41, samples=104 iops : min= 12, max= 48, avg=24.19, stdev= 6.62, samples=104 lat (msec) : 20=0.04%, 50=0.23%, 100=0.27%, 250=41.64%, 500=43.52% lat (msec) : 750=13.98%, 1000=0.31% cpu : usr=0.12%, sys=1.14%, ctx=5278, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=383KiB/s (392kB/s), 94.0KiB/s-96.7KiB/s (97.3kB/s-99.0kB/s), io=19.6MiB (20.5MB), run=52241-52382msec WRITE: bw=399KiB/s (408kB/s), 99.0KiB/s-101KiB/s (101kB/s-103kB/s), io=20.4MiB (21.4MB), run=52241-52382msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147296: Mon Aug 10 10:49:21 2026 read: IOPS=3, BW=153MiB/s (161MB/s)(40.0MiB/261msec) slat (nsec): min=11032k, max=11032k, avg=11031960.00, stdev= 0.00 clat (nsec): min=230910k, max=230910k, avg=230909723.00, stdev= 0.00 lat (nsec): min=241945k, max=241945k, avg=241945298.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.08%, ctx=26, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=40.0MiB (41.9MB), run=261-261msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout