== sanity test 398c: run fio to test AIO ================= 02:46:31 (1779691591) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.3607 s, 116 MB/s osc.lustre-OST0000-osc-ffff928d7112c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146721: Mon May 25 02:48:13 2026 write: IOPS=26, BW=107KiB/s (110kB/s)(10.0MiB/95562msec); 0 zone resets slat (usec): min=33, max=11694, avg=239.98, stdev=502.20 clat (msec): min=115, max=1157, avg=596.50, stdev=138.04 lat (msec): min=116, max=1157, avg=596.74, stdev=138.05 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 372], 10.00th=[ 443], 20.00th=[ 498], | 30.00th=[ 535], 40.00th=[ 567], 50.00th=[ 600], 60.00th=[ 634], | 70.00th=[ 659], 80.00th=[ 709], 90.00th=[ 768], 95.00th=[ 818], | 99.00th=[ 911], 99.50th=[ 961], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1150] bw ( KiB/s): min= 31, max= 310, per=24.82%, avg=106.22, stdev=33.18, samples=190 iops : min= 7, max= 77, avg=26.20, stdev= 8.32, samples=190 lat (msec) : 250=1.88%, 500=20.00%, 750=65.43%, 1000=12.30%, 2000=0.39% cpu : usr=0.05%, sys=0.40%, ctx=2931, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146722: Mon May 25 02:48:13 2026 write: IOPS=26, BW=107KiB/s (110kB/s)(10.0MiB/95454msec); 0 zone resets slat (usec): min=36, max=12333, avg=256.78, stdev=626.74 clat (msec): min=45, max=1103, avg=596.02, stdev=141.10 lat (msec): min=46, max=1103, avg=596.28, stdev=141.10 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 363], 10.00th=[ 435], 20.00th=[ 493], | 30.00th=[ 527], 40.00th=[ 567], 50.00th=[ 600], 60.00th=[ 634], | 70.00th=[ 667], 80.00th=[ 709], 90.00th=[ 768], 95.00th=[ 810], | 99.00th=[ 919], 99.50th=[ 995], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 290, per=24.96%, avg=106.81, stdev=33.34, samples=189 iops : min= 1, max= 72, avg=26.34, stdev= 8.38, samples=189 lat (msec) : 50=0.04%, 100=0.35%, 250=1.80%, 500=19.69%, 750=66.29% lat (msec) : 1000=11.37%, 2000=0.47% cpu : usr=0.06%, sys=0.44%, ctx=2787, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146723: Mon May 25 02:48:13 2026 write: IOPS=26, BW=107KiB/s (110kB/s)(10.0MiB/95498msec); 0 zone resets slat (usec): min=35, max=10047, avg=263.99, stdev=599.73 clat (msec): min=114, max=1157, avg=596.03, stdev=136.06 lat (msec): min=116, max=1158, avg=596.30, stdev=136.08 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 376], 10.00th=[ 443], 20.00th=[ 493], | 30.00th=[ 531], 40.00th=[ 567], 50.00th=[ 600], 60.00th=[ 634], | 70.00th=[ 667], 80.00th=[ 701], 90.00th=[ 760], 95.00th=[ 810], | 99.00th=[ 911], 99.50th=[ 927], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1167] bw ( KiB/s): min= 24, max= 266, per=24.83%, avg=106.27, stdev=32.20, samples=190 iops : min= 6, max= 66, avg=26.22, stdev= 8.07, samples=190 lat (msec) : 250=1.88%, 500=19.14%, 750=67.23%, 1000=11.64%, 2000=0.12% cpu : usr=0.06%, sys=0.43%, ctx=2949, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146724: Mon May 25 02:48:13 2026 write: IOPS=26, BW=107KiB/s (110kB/s)(10.0MiB/95498msec); 0 zone resets slat (usec): min=37, max=18807, avg=270.74, stdev=716.49 clat (msec): min=114, max=1115, avg=595.99, stdev=140.79 lat (msec): min=115, max=1115, avg=596.26, stdev=140.80 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 368], 10.00th=[ 435], 20.00th=[ 493], | 30.00th=[ 531], 40.00th=[ 567], 50.00th=[ 600], 60.00th=[ 634], | 70.00th=[ 659], 80.00th=[ 701], 90.00th=[ 768], 95.00th=[ 818], | 99.00th=[ 936], 99.50th=[ 978], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 31, max= 311, per=24.82%, avg=106.22, stdev=34.10, samples=190 iops : min= 7, max= 77, avg=26.20, stdev= 8.54, samples=190 lat (msec) : 250=1.88%, 500=19.92%, 750=65.74%, 1000=12.11%, 2000=0.35% cpu : usr=0.06%, sys=0.44%, ctx=3009, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=429KiB/s (439kB/s), 107KiB/s-107KiB/s (110kB/s-110kB/s), io=40.0MiB (41.9MB), run=95454-95562msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146748: Mon May 25 02:49:28 2026 read: IOPS=18, BW=74.2KiB/s (75.9kB/s)(5048KiB/68065msec) slat (usec): min=2425, max=38347, avg=9917.33, stdev=5215.27 clat (msec): min=15, max=489, avg=268.54, stdev=64.81 lat (msec): min=22, max=500, avg=278.47, stdev=65.46 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 169], 10.00th=[ 194], 20.00th=[ 215], | 30.00th=[ 234], 40.00th=[ 249], 50.00th=[ 266], 60.00th=[ 279], | 70.00th=[ 300], 80.00th=[ 321], 90.00th=[ 355], 95.00th=[ 384], | 99.00th=[ 426], 99.50th=[ 430], 99.90th=[ 468], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 15, max= 140, per=25.04%, avg=73.63, stdev=26.47, samples=135 iops : min= 3, max= 35, avg=17.96, stdev= 6.64, samples=135 write: IOPS=19, BW=76.3KiB/s (78.1kB/s)(5192KiB/68065msec); 0 zone resets slat (usec): min=37, max=10128, avg=328.41, stdev=726.27 clat (msec): min=205, max=1029, avg=566.35, stdev=134.22 lat (msec): min=206, max=1029, avg=566.69, stdev=134.25 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 363], 10.00th=[ 397], 20.00th=[ 447], | 30.00th=[ 493], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 634], 80.00th=[ 676], 90.00th=[ 743], 95.00th=[ 793], | 99.00th=[ 869], 99.50th=[ 902], 99.90th=[ 986], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 31, max= 143, per=24.72%, avg=75.64, stdev=23.54, samples=135 iops : min= 7, max= 35, avg=18.46, stdev= 5.94, samples=135 lat (msec) : 20=0.08%, 50=0.20%, 100=0.08%, 250=20.27%, 500=45.00% lat (msec) : 750=29.77%, 1000=4.57%, 2000=0.04% cpu : usr=0.08%, sys=0.96%, ctx=5213, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146749: Mon May 25 02:49:28 2026 read: IOPS=18, BW=73.2KiB/s (74.9kB/s)(4976KiB/68007msec) slat (usec): min=63, max=42673, avg=9812.73, stdev=5352.89 clat (msec): min=17, max=495, avg=268.05, stdev=67.46 lat (msec): min=22, max=512, avg=277.87, stdev=68.22 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 169], 10.00th=[ 186], 20.00th=[ 213], | 30.00th=[ 232], 40.00th=[ 249], 50.00th=[ 264], 60.00th=[ 279], | 70.00th=[ 300], 80.00th=[ 321], 90.00th=[ 359], 95.00th=[ 388], | 99.00th=[ 430], 99.50th=[ 464], 99.90th=[ 493], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 23, max= 167, per=24.65%, avg=72.46, stdev=29.21, samples=135 iops : min= 5, max= 41, avg=17.67, stdev= 7.36, samples=135 write: IOPS=19, BW=77.4KiB/s (79.3kB/s)(5264KiB/68007msec); 0 zone resets slat (usec): min=39, max=9528, avg=306.57, stdev=640.13 clat (msec): min=114, max=1039, avg=563.47, stdev=138.93 lat (msec): min=114, max=1039, avg=563.78, stdev=138.96 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 351], 10.00th=[ 384], 20.00th=[ 447], | 30.00th=[ 489], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 634], 80.00th=[ 676], 90.00th=[ 743], 95.00th=[ 802], | 99.00th=[ 869], 99.50th=[ 902], 99.90th=[ 995], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 31, max= 134, per=25.07%, avg=76.70, stdev=20.52, samples=135 iops : min= 7, max= 33, avg=18.73, stdev= 5.19, samples=135 lat (msec) : 20=0.04%, 50=0.23%, 100=0.12%, 250=20.59%, 500=44.26% lat (msec) : 750=29.84%, 1000=4.88%, 2000=0.04% cpu : usr=0.13%, sys=0.95%, ctx=5145, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146750: Mon May 25 02:49:28 2026 read: IOPS=18, BW=73.2KiB/s (74.9kB/s)(4984KiB/68112msec) slat (usec): min=2221, max=35211, avg=9738.20, stdev=5016.89 clat (msec): min=15, max=523, avg=270.39, stdev=67.04 lat (msec): min=21, max=530, avg=280.13, stdev=67.61 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 174], 10.00th=[ 192], 20.00th=[ 215], | 30.00th=[ 234], 40.00th=[ 251], 50.00th=[ 268], 60.00th=[ 284], | 70.00th=[ 300], 80.00th=[ 326], 90.00th=[ 359], 95.00th=[ 393], | 99.00th=[ 435], 99.50th=[ 456], 99.90th=[ 472], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 15, max= 160, per=24.67%, avg=72.54, stdev=27.08, samples=135 iops : min= 3, max= 40, avg=17.69, stdev= 6.82, samples=135 write: IOPS=19, BW=77.2KiB/s (79.0kB/s)(5256KiB/68112msec); 0 zone resets slat (usec): min=40, max=11281, avg=334.81, stdev=704.85 clat (msec): min=104, max=961, avg=562.09, stdev=142.61 lat (msec): min=104, max=961, avg=562.43, stdev=142.61 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 338], 10.00th=[ 384], 20.00th=[ 447], | 30.00th=[ 485], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 634], 80.00th=[ 676], 90.00th=[ 743], 95.00th=[ 802], | 99.00th=[ 877], 99.50th=[ 911], 99.90th=[ 936], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 23, max= 128, per=24.78%, avg=75.83, stdev=21.17, samples=135 iops : min= 5, max= 32, avg=18.51, stdev= 5.35, samples=135 lat (msec) : 20=0.04%, 50=0.20%, 100=0.04%, 250=20.08%, 500=45.27% lat (msec) : 750=29.41%, 1000=4.96% cpu : usr=0.09%, sys=1.05%, ctx=5310, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146751: Mon May 25 02:49:28 2026 read: IOPS=18, BW=74.5KiB/s (76.3kB/s)(5060KiB/67889msec) slat (usec): min=1915, max=39974, avg=9732.99, stdev=5309.77 clat (msec): min=21, max=491, avg=272.25, stdev=67.05 lat (msec): min=28, max=502, avg=281.99, stdev=67.72 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 171], 10.00th=[ 190], 20.00th=[ 220], | 30.00th=[ 239], 40.00th=[ 253], 50.00th=[ 268], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 368], 95.00th=[ 393], | 99.00th=[ 447], 99.50th=[ 472], 99.90th=[ 489], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 15, max= 159, per=25.14%, avg=73.92, stdev=25.55, samples=135 iops : min= 3, max= 39, avg=18.04, stdev= 6.43, samples=135 write: IOPS=19, BW=76.3KiB/s (78.1kB/s)(5180KiB/67889msec); 0 zone resets slat (usec): min=38, max=9393, avg=309.76, stdev=651.98 clat (msec): min=147, max=994, avg=561.70, stdev=135.54 lat (msec): min=148, max=994, avg=562.02, stdev=135.51 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 355], 10.00th=[ 388], 20.00th=[ 439], | 30.00th=[ 485], 40.00th=[ 531], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 634], 80.00th=[ 676], 90.00th=[ 735], 95.00th=[ 793], | 99.00th=[ 885], 99.50th=[ 919], 99.90th=[ 978], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 23, max= 144, per=24.60%, avg=75.27, stdev=21.58, samples=135 iops : min= 5, max= 36, avg=18.37, stdev= 5.47, samples=135 lat (msec) : 50=0.08%, 100=0.23%, 250=19.34%, 500=46.56%, 750=29.45% lat (msec) : 1000=4.34% cpu : usr=0.15%, sys=1.00%, ctx=5298, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=295KiB/s (302kB/s), 73.2KiB/s-74.5KiB/s (74.9kB/s-76.3kB/s), io=19.6MiB (20.5MB), run=67889-68112msec WRITE: bw=307KiB/s (314kB/s), 76.3KiB/s-77.4KiB/s (78.1kB/s-79.3kB/s), io=20.4MiB (21.4MB), run=67889-68112msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146765: Mon May 25 02:49:34 2026 read: IOPS=3, BW=130MiB/s (136MB/s)(40.0MiB/308msec) slat (nsec): min=17885k, max=17885k, avg=17884870.00, stdev= 0.00 clat (nsec): min=267921k, max=267921k, avg=267920571.00, stdev= 0.00 lat (nsec): min=285810k, max=285810k, avg=285809607.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.23%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=40.0MiB (41.9MB), run=308-308msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout