== sanity test 398c: run fio to test AIO ================= 01:23:54 (1786080234) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.847931 s, 49.5 MB/s osc.lustre-OST0000-osc-ffff8fde4976a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144443: Fri Aug 7 01:27:09 2026 write: IOPS=13, BW=54.9KiB/s (56.2kB/s)(10.0MiB/186497msec); 0 zone resets slat (usec): min=50, max=14449, avg=441.21, stdev=924.85 clat (msec): min=260, max=2875, avg=1163.55, stdev=328.16 lat (msec): min=261, max=2875, avg=1164.00, stdev=328.15 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 701], 10.00th=[ 810], 20.00th=[ 911], | 30.00th=[ 986], 40.00th=[ 1062], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1318], 80.00th=[ 1418], 90.00th=[ 1569], 95.00th=[ 1703], | 99.00th=[ 2089], 99.50th=[ 2299], 99.90th=[ 2601], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 128, per=25.49%, avg=55.82, stdev=29.83, samples=361 iops : min= 1, max= 32, avg=13.59, stdev= 7.49, samples=361 lat (msec) : 500=1.91%, 750=5.59%, 1000=24.22%, 2000=66.60%, >=2000=1.68% cpu : usr=0.05%, sys=0.30%, ctx=2145, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144444: Fri Aug 7 01:27:09 2026 write: IOPS=13, BW=54.9KiB/s (56.2kB/s)(10.0MiB/186611msec); 0 zone resets slat (usec): min=49, max=16216, avg=450.33, stdev=1015.27 clat (msec): min=274, max=2200, avg=1164.74, stdev=320.94 lat (msec): min=276, max=2200, avg=1165.20, stdev=320.98 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 693], 10.00th=[ 810], 20.00th=[ 911], | 30.00th=[ 1003], 40.00th=[ 1083], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1284], 80.00th=[ 1418], 90.00th=[ 1586], 95.00th=[ 1754], | 99.00th=[ 2005], 99.50th=[ 2089], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 128, per=25.91%, avg=56.74, stdev=29.26, samples=355 iops : min= 1, max= 32, avg=13.82, stdev= 7.34, samples=355 lat (msec) : 500=1.91%, 750=5.12%, 1000=23.01%, 2000=68.87%, >=2000=1.09% cpu : usr=0.05%, sys=0.33%, ctx=2267, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144445: Fri Aug 7 01:27:09 2026 write: IOPS=13, BW=54.9KiB/s (56.2kB/s)(10.0MiB/186485msec); 0 zone resets slat (usec): min=49, max=12898, avg=485.26, stdev=1010.49 clat (msec): min=231, max=2862, avg=1163.38, stdev=323.46 lat (msec): min=231, max=2862, avg=1163.88, stdev=323.49 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 709], 10.00th=[ 810], 20.00th=[ 919], | 30.00th=[ 995], 40.00th=[ 1062], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1435], 90.00th=[ 1603], 95.00th=[ 1720], | 99.00th=[ 2005], 99.50th=[ 2140], 99.90th=[ 2333], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 128, per=25.27%, avg=55.34, stdev=28.65, samples=364 iops : min= 1, max= 32, avg=13.47, stdev= 7.18, samples=364 lat (msec) : 250=0.62%, 500=1.25%, 750=4.53%, 1000=24.61%, 2000=67.46% lat (msec) : >=2000=1.52% cpu : usr=0.05%, sys=0.35%, ctx=2443, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144446: Fri Aug 7 01:27:09 2026 write: IOPS=13, BW=54.0KiB/s (56.3kB/s)(10.0MiB/186182msec); 0 zone resets slat (usec): min=49, max=14653, avg=462.90, stdev=996.34 clat (msec): min=136, max=3522, avg=1162.28, stdev=342.69 lat (msec): min=136, max=3523, avg=1162.75, stdev=342.69 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 693], 10.00th=[ 793], 20.00th=[ 911], | 30.00th=[ 986], 40.00th=[ 1062], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1401], 90.00th=[ 1569], 95.00th=[ 1737], | 99.00th=[ 2140], 99.50th=[ 2333], 99.90th=[ 3104], 99.95th=[ 3205], | 99.99th=[ 3507] bw ( KiB/s): min= 7, max= 192, per=26.36%, avg=57.72, stdev=29.17, samples=349 iops : min= 1, max= 48, avg=14.07, stdev= 7.33, samples=349 lat (msec) : 250=0.74%, 500=1.95%, 750=4.77%, 1000=24.30%, 2000=66.64% lat (msec) : >=2000=1.60% cpu : usr=0.06%, sys=0.31%, ctx=2451, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=219KiB/s (225kB/s), 54.9KiB/s-54.0KiB/s (56.2kB/s-56.3kB/s), io=40.0MiB (41.9MB), run=186182-186611msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144482: Fri Aug 7 01:29:33 2026 read: IOPS=9, BW=37.2KiB/s (38.1kB/s)(5048KiB/135814msec) slat (usec): min=64, max=62728, avg=19577.98, stdev=9185.34 clat (msec): min=15, max=1506, avg=531.44, stdev=149.94 lat (msec): min=22, max=1532, avg=551.03, stdev=152.13 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 359], 10.00th=[ 388], 20.00th=[ 422], | 30.00th=[ 456], 40.00th=[ 485], 50.00th=[ 514], 60.00th=[ 542], | 70.00th=[ 575], 80.00th=[ 625], 90.00th=[ 709], 95.00th=[ 793], | 99.00th=[ 986], 99.50th=[ 1099], 99.90th=[ 1452], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 102, per=25.92%, avg=37.84, stdev=18.73, samples=261 iops : min= 1, max= 25, avg= 9.00, stdev= 4.75, samples=261 write: IOPS=9, BW=38.2KiB/s (39.1kB/s)(5192KiB/135814msec); 0 zone resets slat (usec): min=52, max=26380, avg=678.48, stdev=1728.41 clat (msec): min=225, max=2545, avg=1136.15, stdev=319.51 lat (msec): min=231, max=2546, avg=1136.83, stdev=319.48 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 693], 10.00th=[ 785], 20.00th=[ 885], | 30.00th=[ 961], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1368], 90.00th=[ 1519], 95.00th=[ 1720], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 104, per=25.34%, avg=38.51, stdev=17.91, samples=263 iops : min= 1, max= 26, avg= 9.17, stdev= 4.51, samples=263 lat (msec) : 20=0.04%, 50=0.23%, 100=0.31%, 250=0.23%, 500=21.91% lat (msec) : 750=27.03%, 1000=17.73%, 2000=31.60%, >=2000=0.90% cpu : usr=0.10%, sys=0.83%, ctx=5007, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144483: Fri Aug 7 01:29:33 2026 read: IOPS=9, BW=36.4KiB/s (37.3kB/s)(4976KiB/136703msec) slat (usec): min=4834, max=66149, avg=20134.77, stdev=9524.56 clat (msec): min=10, max=992, avg=531.33, stdev=130.97 lat (msec): min=33, max=1035, avg=551.48, stdev=133.34 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 363], 10.00th=[ 384], 20.00th=[ 422], | 30.00th=[ 460], 40.00th=[ 489], 50.00th=[ 514], 60.00th=[ 542], | 70.00th=[ 584], 80.00th=[ 634], 90.00th=[ 718], 95.00th=[ 776], | 99.00th=[ 919], 99.50th=[ 953], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 103, per=24.96%, avg=36.44, stdev=18.71, samples=266 iops : min= 1, max= 25, avg= 8.66, stdev= 4.70, samples=266 write: IOPS=9, BW=38.5KiB/s (39.4kB/s)(5264KiB/136703msec); 0 zone resets slat (usec): min=53, max=16276, avg=574.63, stdev=1221.85 clat (msec): min=434, max=2519, avg=1136.63, stdev=318.07 lat (msec): min=434, max=2519, avg=1137.21, stdev=318.13 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 693], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 953], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1368], 90.00th=[ 1536], 95.00th=[ 1754], | 99.00th=[ 2198], 99.50th=[ 2366], 99.90th=[ 2400], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 103, per=25.42%, avg=38.63, stdev=18.96, samples=267 iops : min= 1, max= 25, avg= 9.20, stdev= 4.80, samples=267 lat (msec) : 20=0.04%, 100=0.04%, 250=0.23%, 500=21.45%, 750=27.73% lat (msec) : 1000=18.48%, 2000=31.25%, >=2000=0.78% cpu : usr=0.08%, sys=0.82%, ctx=4970, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144484: Fri Aug 7 01:29:33 2026 read: IOPS=9, BW=36.5KiB/s (37.3kB/s)(4984KiB/136649msec) slat (usec): min=3642, max=67871, avg=19799.69, stdev=9006.07 clat (msec): min=24, max=1128, avg=532.88, stdev=130.65 lat (msec): min=41, max=1148, avg=552.69, stdev=132.00 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 363], 10.00th=[ 393], 20.00th=[ 426], | 30.00th=[ 460], 40.00th=[ 485], 50.00th=[ 510], 60.00th=[ 542], | 70.00th=[ 584], 80.00th=[ 634], 90.00th=[ 709], 95.00th=[ 768], | 99.00th=[ 911], 99.50th=[ 961], 99.90th=[ 1099], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 111, per=25.32%, avg=36.96, stdev=18.26, samples=264 iops : min= 1, max= 27, avg= 8.78, stdev= 4.56, samples=264 write: IOPS=9, BW=38.5KiB/s (39.4kB/s)(5256KiB/136649msec); 0 zone resets slat (usec): min=53, max=14186, avg=560.14, stdev=1134.14 clat (msec): min=388, max=2449, avg=1133.96, stdev=307.60 lat (msec): min=389, max=2449, avg=1134.52, stdev=307.65 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 701], 10.00th=[ 776], 20.00th=[ 894], | 30.00th=[ 961], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1351], 90.00th=[ 1552], 95.00th=[ 1720], | 99.00th=[ 2089], 99.50th=[ 2232], 99.90th=[ 2333], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 104, per=25.16%, avg=38.24, stdev=17.39, samples=268 iops : min= 1, max= 26, avg= 9.11, stdev= 4.44, samples=268 lat (msec) : 50=0.04%, 100=0.12%, 250=0.08%, 500=21.99%, 750=27.42% lat (msec) : 1000=17.70%, 2000=32.07%, >=2000=0.59% cpu : usr=0.12%, sys=0.84%, ctx=5116, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144485: Fri Aug 7 01:29:33 2026 read: IOPS=9, BW=37.3KiB/s (38.2kB/s)(5060KiB/135645msec) slat (usec): min=3853, max=69240, avg=19143.59, stdev=8988.65 clat (msec): min=39, max=1606, avg=533.00, stdev=137.20 lat (msec): min=66, max=1626, avg=552.16, stdev=139.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 363], 10.00th=[ 393], 20.00th=[ 430], | 30.00th=[ 460], 40.00th=[ 489], 50.00th=[ 514], 60.00th=[ 542], | 70.00th=[ 584], 80.00th=[ 617], 90.00th=[ 701], 95.00th=[ 776], | 99.00th=[ 911], 99.50th=[ 961], 99.90th=[ 1536], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 88, per=26.23%, avg=38.29, stdev=17.65, samples=259 iops : min= 1, max= 22, avg= 9.13, stdev= 4.43, samples=259 write: IOPS=9, BW=38.2KiB/s (39.1kB/s)(5180KiB/135645msec); 0 zone resets slat (usec): min=55, max=15385, avg=663.88, stdev=1459.15 clat (msec): min=447, max=2508, avg=1130.43, stdev=322.00 lat (msec): min=449, max=2509, avg=1131.10, stdev=322.09 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 701], 10.00th=[ 793], 20.00th=[ 885], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1368], 90.00th=[ 1519], 95.00th=[ 1754], | 99.00th=[ 2198], 99.50th=[ 2232], 99.90th=[ 2333], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 104, per=25.36%, avg=38.54, stdev=17.99, samples=262 iops : min= 1, max= 26, avg= 9.18, stdev= 4.55, samples=262 lat (msec) : 50=0.04%, 100=0.20%, 250=0.08%, 500=21.80%, 750=27.85% lat (msec) : 1000=18.83%, 2000=29.92%, >=2000=1.29% cpu : usr=0.10%, sys=0.91%, ctx=5355, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=147KiB/s (150kB/s), 36.4KiB/s-37.3KiB/s (37.3kB/s-38.2kB/s), io=19.6MiB (20.5MB), run=135645-136703msec WRITE: bw=153KiB/s (156kB/s), 38.2KiB/s-38.5KiB/s (39.1kB/s-39.4kB/s), io=20.4MiB (21.4MB), run=135645-136703msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144508: Fri Aug 7 01:29:43 2026 read: IOPS=1, BW=51.9MiB/s (54.5MB/s)(40.0MiB/770msec) slat (nsec): min=28368k, max=28368k, avg=28367982.00, stdev= 0.00 clat (nsec): min=725579k, max=725579k, avg=725578930.00, stdev= 0.00 lat (nsec): min=753952k, max=753952k, avg=753951794.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 726], 5.00th=[ 726], 10.00th=[ 726], 20.00th=[ 726], | 30.00th=[ 726], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.82%, ctx=16, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.9MiB/s (54.5MB/s), 51.9MiB/s-51.9MiB/s (54.5MB/s-54.5MB/s), io=40.0MiB (41.9MB), run=770-770msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout