== sanity test 398c: run fio to test AIO ================= 22:11:12 (1772680272) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.269145 s, 156 MB/s osc.lustre-OST0000-osc-ffff99e7ebe10800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154743: Wed Mar 4 22:12:08 2026 write: IOPS=49, BW=199KiB/s (203kB/s)(10.0MiB/51583msec); 0 zone resets slat (usec): min=34, max=7504, avg=222.13, stdev=292.97 clat (msec): min=101, max=853, avg=321.65, stdev=130.37 lat (msec): min=101, max=854, avg=321.88, stdev=130.37 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 165], 10.00th=[ 184], 20.00th=[ 211], | 30.00th=[ 239], 40.00th=[ 264], 50.00th=[ 292], 60.00th=[ 326], | 70.00th=[ 368], 80.00th=[ 422], 90.00th=[ 502], 95.00th=[ 592], | 99.00th=[ 726], 99.50th=[ 760], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 852] bw ( KiB/s): min= 24, max= 432, per=24.74%, avg=196.47, stdev=73.89, samples=102 iops : min= 6, max= 108, avg=48.88, stdev=18.56, samples=102 lat (msec) : 250=35.94%, 500=53.83%, 750=9.53%, 1000=0.70% cpu : usr=0.10%, sys=0.80%, ctx=2835, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154744: Wed Mar 4 22:12:08 2026 write: IOPS=49, BW=199KiB/s (204kB/s)(10.0MiB/51493msec); 0 zone resets slat (usec): min=35, max=7503, avg=227.29, stdev=300.84 clat (msec): min=42, max=853, avg=321.47, stdev=132.06 lat (msec): min=43, max=853, avg=321.70, stdev=132.05 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 165], 10.00th=[ 186], 20.00th=[ 213], | 30.00th=[ 236], 40.00th=[ 262], 50.00th=[ 288], 60.00th=[ 321], | 70.00th=[ 368], 80.00th=[ 426], 90.00th=[ 506], 95.00th=[ 584], | 99.00th=[ 751], 99.50th=[ 776], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 48, max= 432, per=24.87%, avg=197.46, stdev=74.17, samples=102 iops : min= 12, max= 108, avg=49.13, stdev=18.63, samples=102 lat (msec) : 50=0.20%, 100=0.08%, 250=35.78%, 500=53.20%, 750=9.73% lat (msec) : 1000=1.02% cpu : usr=0.10%, sys=0.82%, ctx=2954, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154745: Wed Mar 4 22:12:08 2026 write: IOPS=49, BW=199KiB/s (204kB/s)(10.0MiB/51516msec); 0 zone resets slat (usec): min=34, max=7202, avg=217.64, stdev=278.50 clat (msec): min=105, max=853, avg=321.43, stdev=128.35 lat (msec): min=105, max=853, avg=321.65, stdev=128.36 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 169], 10.00th=[ 188], 20.00th=[ 213], | 30.00th=[ 239], 40.00th=[ 262], 50.00th=[ 292], 60.00th=[ 321], | 70.00th=[ 368], 80.00th=[ 422], 90.00th=[ 510], 95.00th=[ 575], | 99.00th=[ 701], 99.50th=[ 793], 99.90th=[ 844], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 47, max= 416, per=24.87%, avg=197.45, stdev=74.01, samples=102 iops : min= 11, max= 104, avg=49.13, stdev=18.60, samples=102 lat (msec) : 250=36.25%, 500=52.42%, 750=10.70%, 1000=0.62% cpu : usr=0.09%, sys=0.79%, ctx=2898, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154746: Wed Mar 4 22:12:08 2026 write: IOPS=49, BW=199KiB/s (203kB/s)(10.0MiB/51538msec); 0 zone resets slat (usec): min=34, max=5511, avg=223.64, stdev=271.27 clat (msec): min=85, max=853, avg=321.31, stdev=129.70 lat (msec): min=85, max=855, avg=321.54, stdev=129.71 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 165], 10.00th=[ 186], 20.00th=[ 213], | 30.00th=[ 239], 40.00th=[ 262], 50.00th=[ 288], 60.00th=[ 321], | 70.00th=[ 368], 80.00th=[ 426], 90.00th=[ 510], 95.00th=[ 575], | 99.00th=[ 718], 99.50th=[ 751], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 71, max= 456, per=24.84%, avg=197.26, stdev=73.30, samples=102 iops : min= 17, max= 114, avg=49.09, stdev=18.44, samples=102 lat (msec) : 100=0.04%, 250=35.66%, 500=53.16%, 750=10.70%, 1000=0.43% cpu : usr=0.07%, sys=0.80%, ctx=2952, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=794KiB/s (813kB/s), 199KiB/s-199KiB/s (203kB/s-204kB/s), io=40.0MiB (41.9MB), run=51493-51583msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154763: Wed Mar 4 22:12:35 2026 read: IOPS=51, BW=207KiB/s (212kB/s)(5048KiB/24367msec) slat (usec): min=1087, max=19700, avg=3176.99, stdev=1542.13 clat (msec): min=3, max=206, avg=95.37, stdev=28.00 lat (msec): min=7, max=214, avg=98.55, stdev=28.62 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 97], | 70.00th=[ 105], 80.00th=[ 113], 90.00th=[ 132], 95.00th=[ 150], | 99.00th=[ 188], 99.50th=[ 197], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 120, max= 304, per=25.28%, avg=207.79, stdev=50.77, samples=48 iops : min= 30, max= 76, avg=51.79, stdev=12.75, samples=48 write: IOPS=53, BW=213KiB/s (218kB/s)(5192KiB/24367msec); 0 zone resets slat (usec): min=42, max=5506, avg=200.37, stdev=225.96 clat (msec): min=71, max=413, avg=203.75, stdev=58.19 lat (msec): min=71, max=413, avg=203.95, stdev=58.19 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 121], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 226], 80.00th=[ 247], 90.00th=[ 284], 95.00th=[ 317], | 99.00th=[ 384], 99.50th=[ 393], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 104, max= 304, per=24.83%, avg=212.56, stdev=49.90, samples=48 iops : min= 26, max= 76, avg=52.98, stdev=12.53, samples=48 lat (msec) : 4=0.04%, 10=0.12%, 20=0.12%, 50=0.31%, 100=32.11% lat (msec) : 250=57.89%, 500=9.41% cpu : usr=0.20%, sys=1.94%, ctx=5087, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154764: Wed Mar 4 22:12:35 2026 read: IOPS=50, BW=204KiB/s (209kB/s)(4976KiB/24401msec) slat (usec): min=1096, max=18607, avg=3162.74, stdev=1485.23 clat (msec): min=6, max=243, avg=94.59, stdev=26.93 lat (msec): min=9, max=251, avg=97.76, stdev=27.50 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 96], | 70.00th=[ 104], 80.00th=[ 113], 90.00th=[ 131], 95.00th=[ 146], | 99.00th=[ 182], 99.50th=[ 192], 99.90th=[ 199], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 80, max= 320, per=24.64%, avg=202.56, stdev=55.35, samples=48 iops : min= 20, max= 80, avg=50.48, stdev=13.88, samples=48 write: IOPS=53, BW=216KiB/s (221kB/s)(5264KiB/24401msec); 0 zone resets slat (usec): min=38, max=2910, avg=194.80, stdev=129.56 clat (msec): min=19, max=411, avg=203.69, stdev=58.98 lat (msec): min=19, max=412, avg=203.89, stdev=58.99 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 116], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 247], 90.00th=[ 288], 95.00th=[ 321], | 99.00th=[ 368], 99.50th=[ 388], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 96, max= 328, per=25.13%, avg=215.10, stdev=48.19, samples=48 iops : min= 24, max= 82, avg=53.60, stdev=12.09, samples=48 lat (msec) : 10=0.04%, 20=0.23%, 50=0.62%, 100=32.58%, 250=56.95% lat (msec) : 500=9.57% cpu : usr=0.19%, sys=1.92%, ctx=4995, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154765: Wed Mar 4 22:12:35 2026 read: IOPS=51, BW=205KiB/s (210kB/s)(4984KiB/24339msec) slat (usec): min=36, max=15338, avg=3144.19, stdev=1549.40 clat (msec): min=4, max=338, avg=94.23, stdev=29.35 lat (msec): min=7, max=340, avg=97.37, stdev=29.96 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 60], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 129], 95.00th=[ 150], | 99.00th=[ 188], 99.50th=[ 197], 99.90th=[ 245], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 87, max= 312, per=25.04%, avg=205.79, stdev=53.67, samples=48 iops : min= 21, max= 78, avg=51.29, stdev=13.52, samples=48 write: IOPS=53, BW=216KiB/s (221kB/s)(5256KiB/24339msec); 0 zone resets slat (usec): min=37, max=2708, avg=199.25, stdev=146.56 clat (msec): min=34, max=618, avg=203.47, stdev=66.66 lat (msec): min=34, max=618, avg=203.67, stdev=66.67 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 114], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 245], 90.00th=[ 284], 95.00th=[ 326], | 99.00th=[ 414], 99.50th=[ 506], 99.90th=[ 600], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 80, max= 336, per=25.11%, avg=214.94, stdev=53.73, samples=48 iops : min= 20, max= 84, avg=53.56, stdev=13.46, samples=48 lat (msec) : 10=0.08%, 20=0.43%, 50=0.82%, 100=32.97%, 250=56.68% lat (msec) : 500=8.71%, 750=0.31% cpu : usr=0.23%, sys=1.91%, ctx=5115, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154766: Wed Mar 4 22:12:35 2026 read: IOPS=51, BW=208KiB/s (212kB/s)(5060KiB/24384msec) slat (usec): min=1149, max=15437, avg=3089.82, stdev=1378.84 clat (msec): min=3, max=204, avg=95.02, stdev=26.68 lat (msec): min=6, max=209, avg=98.11, stdev=27.22 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 81], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 113], 90.00th=[ 132], 95.00th=[ 146], | 99.00th=[ 182], 99.50th=[ 192], 99.90th=[ 203], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 95, max= 368, per=25.34%, avg=208.33, stdev=55.49, samples=48 iops : min= 23, max= 92, avg=51.92, stdev=13.94, samples=48 write: IOPS=53, BW=212KiB/s (218kB/s)(5180KiB/24384msec); 0 zone resets slat (usec): min=57, max=19379, avg=219.06, stdev=558.72 clat (msec): min=76, max=426, avg=204.64, stdev=60.05 lat (msec): min=76, max=426, avg=204.86, stdev=60.10 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 120], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 226], 80.00th=[ 247], 90.00th=[ 288], 95.00th=[ 326], | 99.00th=[ 372], 99.50th=[ 401], 99.90th=[ 418], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 128, max= 296, per=24.73%, avg=211.67, stdev=44.44, samples=48 iops : min= 32, max= 74, avg=52.75, stdev=11.18, samples=48 lat (msec) : 4=0.04%, 10=0.04%, 20=0.12%, 50=0.20%, 100=33.16% lat (msec) : 250=57.19%, 500=9.26% cpu : usr=0.17%, sys=1.96%, ctx=5124, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=822KiB/s (842kB/s), 204KiB/s-208KiB/s (209kB/s-212kB/s), io=19.6MiB (20.5MB), run=24339-24401msec WRITE: bw=856KiB/s (877kB/s), 212KiB/s-216KiB/s (218kB/s-221kB/s), io=20.4MiB (21.4MB), run=24339-24401msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154775: Wed Mar 4 22:12:39 2026 read: IOPS=6, BW=272MiB/s (285MB/s)(40.0MiB/147msec) slat (nsec): min=11339k, max=11339k, avg=11338988.00, stdev= 0.00 clat (nsec): min=123404k, max=123404k, avg=123403878.00, stdev= 0.00 lat (nsec): min=134747k, max=134747k, avg=134746867.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.85%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=40.0MiB (41.9MB), run=147-147msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout