== sanity test 398c: run fio to test AIO ================= 02:44:01 (1788590641) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.452824 s, 92.6 MB/s osc.lustre-OST0000-osc-ffff8bc7d14a6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146358: Sat Sep 5 02:47:06 2026 write: IOPS=14, BW=58.6KiB/s (60.0kB/s)(10.0MiB/174646msec); 0 zone resets slat (usec): min=21, max=21129, avg=180.42, stdev=758.79 clat (msec): min=318, max=2713, avg=1089.97, stdev=331.29 lat (msec): min=318, max=2713, avg=1090.15, stdev=331.30 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 609], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 986], 50.00th=[ 1053], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1536], 95.00th=[ 1687], | 99.00th=[ 2056], 99.50th=[ 2198], 99.90th=[ 2702], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 143, per=25.53%, avg=59.75, stdev=29.57, samples=337 iops : min= 1, max= 35, avg=14.50, stdev= 7.43, samples=337 lat (msec) : 500=2.19%, 750=9.96%, 1000=30.59%, 2000=56.13%, >=2000=1.13% cpu : usr=0.04%, sys=0.18%, ctx=1416, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146359: Sat Sep 5 02:47:06 2026 write: IOPS=14, BW=58.7KiB/s (60.1kB/s)(10.0MiB/174514msec); 0 zone resets slat (usec): min=22, max=19284, avg=199.81, stdev=882.01 clat (msec): min=180, max=2762, avg=1089.97, stdev=337.56 lat (msec): min=180, max=2762, avg=1090.18, stdev=337.56 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 617], 10.00th=[ 743], 20.00th=[ 818], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1351], 90.00th=[ 1502], 95.00th=[ 1670], | 99.00th=[ 2072], 99.50th=[ 2265], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 255, per=25.38%, avg=59.38, stdev=32.80, samples=339 iops : min= 1, max= 63, avg=14.40, stdev= 8.23, samples=339 lat (msec) : 250=0.62%, 500=2.03%, 750=8.20%, 1000=31.68%, 2000=56.25% lat (msec) : >=2000=1.21% cpu : usr=0.04%, sys=0.20%, ctx=1500, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146360: Sat Sep 5 02:47:06 2026 write: IOPS=14, BW=58.7KiB/s (60.1kB/s)(10.0MiB/174585msec); 0 zone resets slat (usec): min=20, max=18849, avg=207.77, stdev=900.27 clat (msec): min=122, max=2757, avg=1089.74, stdev=335.23 lat (msec): min=122, max=2757, avg=1089.95, stdev=335.26 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 625], 10.00th=[ 751], 20.00th=[ 827], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1351], 90.00th=[ 1502], 95.00th=[ 1670], | 99.00th=[ 2123], 99.50th=[ 2366], 99.90th=[ 2702], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 255, per=25.61%, avg=59.92, stdev=31.31, samples=336 iops : min= 1, max= 63, avg=14.54, stdev= 7.86, samples=336 lat (msec) : 250=0.66%, 500=2.03%, 750=7.42%, 1000=32.46%, 2000=56.29% lat (msec) : >=2000=1.13% cpu : usr=0.04%, sys=0.21%, ctx=1611, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146361: Sat Sep 5 02:47:06 2026 write: IOPS=14, BW=58.6KiB/s (60.0kB/s)(10.0MiB/174691msec); 0 zone resets slat (usec): min=23, max=24344, avg=204.92, stdev=1016.58 clat (msec): min=146, max=2739, avg=1090.47, stdev=333.65 lat (msec): min=146, max=2739, avg=1090.68, stdev=333.63 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 617], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1519], 95.00th=[ 1670], | 99.00th=[ 2056], 99.50th=[ 2333], 99.90th=[ 2635], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min= 7, max= 159, per=24.98%, avg=58.46, stdev=28.95, samples=343 iops : min= 1, max= 39, avg=14.17, stdev= 7.28, samples=343 lat (msec) : 250=0.31%, 500=2.46%, 750=8.87%, 1000=31.29%, 2000=55.86% lat (msec) : >=2000=1.21% cpu : usr=0.04%, sys=0.20%, ctx=1684, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=234KiB/s (240kB/s), 58.6KiB/s-58.7KiB/s (60.0kB/s-60.1kB/s), io=40.0MiB (41.9MB), run=174514-174691msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146396: Sat Sep 5 02:49:06 2026 read: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5048KiB/110299msec) slat (usec): min=53, max=50870, avg=13981.02, stdev=8127.48 clat (msec): min=26, max=1011, avg=435.21, stdev=112.29 lat (msec): min=26, max=1018, avg=449.20, stdev=113.22 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 288], 10.00th=[ 309], 20.00th=[ 342], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 481], 80.00th=[ 523], 90.00th=[ 575], 95.00th=[ 625], | 99.00th=[ 768], 99.50th=[ 793], 99.90th=[ 1003], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 7, max= 103, per=25.01%, avg=45.27, stdev=20.34, samples=219 iops : min= 1, max= 25, avg=10.84, stdev= 5.07, samples=219 write: IOPS=11, BW=47.1KiB/s (48.2kB/s)(5192KiB/110299msec); 0 zone resets slat (usec): min=23, max=19841, avg=282.68, stdev=1044.32 clat (msec): min=187, max=1829, avg=921.69, stdev=252.84 lat (msec): min=187, max=1829, avg=921.97, stdev=252.80 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 701], | 30.00th=[ 768], 40.00th=[ 844], 50.00th=[ 902], 60.00th=[ 978], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1603], 99.50th=[ 1620], 99.90th=[ 1770], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 110, per=24.64%, avg=46.32, stdev=21.65, samples=219 iops : min= 1, max= 27, avg=11.11, stdev= 5.47, samples=219 lat (msec) : 50=0.16%, 100=0.16%, 250=0.86%, 500=37.07%, 750=24.18% lat (msec) : 1000=19.06%, 2000=18.52% cpu : usr=0.09%, sys=0.61%, ctx=4382, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146397: Sat Sep 5 02:49:06 2026 read: IOPS=11, BW=45.1KiB/s (46.1kB/s)(4976KiB/110419msec) slat (usec): min=2825, max=55440, avg=14062.16, stdev=7683.35 clat (msec): min=10, max=878, avg=429.49, stdev=108.83 lat (msec): min=19, max=918, avg=443.56, stdev=109.75 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 477], 80.00th=[ 510], 90.00th=[ 558], 95.00th=[ 617], | 99.00th=[ 751], 99.50th=[ 793], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 112, per=25.14%, avg=45.50, stdev=20.39, samples=214 iops : min= 1, max= 28, avg=10.90, stdev= 5.10, samples=214 write: IOPS=11, BW=47.7KiB/s (48.8kB/s)(5264KiB/110419msec); 0 zone resets slat (usec): min=33, max=20777, avg=221.31, stdev=724.82 clat (msec): min=289, max=1708, avg=921.00, stdev=251.48 lat (msec): min=289, max=1708, avg=921.23, stdev=251.51 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 531], 10.00th=[ 609], 20.00th=[ 693], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 911], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1519], 99.50th=[ 1586], 99.90th=[ 1687], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 104, per=24.97%, avg=46.95, stdev=21.18, samples=219 iops : min= 1, max= 26, avg=11.26, stdev= 5.35, samples=219 lat (msec) : 20=0.16%, 50=0.08%, 100=0.23%, 250=0.82%, 500=37.81% lat (msec) : 750=22.70%, 1000=18.79%, 2000=19.41% cpu : usr=0.09%, sys=0.56%, ctx=4257, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146398: Sat Sep 5 02:49:06 2026 read: IOPS=11, BW=45.1KiB/s (46.2kB/s)(4984KiB/110561msec) slat (usec): min=2631, max=62418, avg=14609.17, stdev=8260.47 clat (msec): min=16, max=876, avg=433.85, stdev=110.93 lat (msec): min=33, max=913, avg=448.46, stdev=111.85 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 485], 80.00th=[ 518], 90.00th=[ 575], 95.00th=[ 634], | 99.00th=[ 735], 99.50th=[ 760], 99.90th=[ 810], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 126, per=24.60%, avg=44.52, stdev=20.91, samples=220 iops : min= 1, max= 31, avg=10.65, stdev= 5.25, samples=220 write: IOPS=11, BW=47.5KiB/s (48.7kB/s)(5256KiB/110561msec); 0 zone resets slat (usec): min=29, max=12825, avg=287.60, stdev=905.39 clat (msec): min=311, max=1774, avg=917.55, stdev=241.13 lat (msec): min=315, max=1774, avg=917.84, stdev=241.16 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 701], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 911], 60.00th=[ 986], | 70.00th=[ 1045], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1318], | 99.00th=[ 1536], 99.50th=[ 1569], 99.90th=[ 1687], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 96, per=24.94%, avg=46.89, stdev=19.76, samples=219 iops : min= 1, max= 24, avg=11.25, stdev= 4.97, samples=219 lat (msec) : 20=0.04%, 50=0.12%, 100=0.31%, 250=0.90%, 500=36.29% lat (msec) : 750=24.69%, 1000=18.63%, 2000=19.02% cpu : usr=0.09%, sys=0.59%, ctx=4438, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146399: Sat Sep 5 02:49:06 2026 read: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5060KiB/110561msec) slat (usec): min=2600, max=58180, avg=13928.98, stdev=7777.37 clat (msec): min=9, max=879, avg=429.77, stdev=105.45 lat (msec): min=21, max=917, avg=443.71, stdev=106.32 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 477], 80.00th=[ 514], 90.00th=[ 558], 95.00th=[ 617], | 99.00th=[ 684], 99.50th=[ 735], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 112, per=25.59%, avg=46.32, stdev=21.22, samples=214 iops : min= 1, max= 28, avg=11.10, stdev= 5.31, samples=214 write: IOPS=11, BW=46.9KiB/s (47.0kB/s)(5180KiB/110561msec); 0 zone resets slat (usec): min=25, max=21506, avg=295.92, stdev=1097.76 clat (msec): min=310, max=1809, avg=928.63, stdev=241.28 lat (msec): min=310, max=1809, avg=928.93, stdev=241.24 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1569], 99.50th=[ 1586], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 103, per=24.90%, avg=46.82, stdev=18.60, samples=217 iops : min= 1, max= 25, avg=11.23, stdev= 4.67, samples=217 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.12%, 250=0.70% lat (msec) : 500=38.12%, 750=22.97%, 1000=18.48%, 2000=19.41% cpu : usr=0.10%, sys=0.61%, ctx=4540, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=182KiB/s (186kB/s), 45.1KiB/s-45.8KiB/s (46.1kB/s-46.9kB/s), io=19.6MiB (20.5MB), run=110299-110561msec WRITE: bw=189KiB/s (193kB/s), 46.9KiB/s-47.7KiB/s (47.0kB/s-48.8kB/s), io=20.4MiB (21.4MB), run=110299-110561msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146423: Sat Sep 5 02:49:17 2026 read: IOPS=2, BW=81.0MiB/s (85.9MB/s)(40.0MiB/488msec) slat (nsec): min=22922k, max=22922k, avg=22921594.00, stdev= 0.00 clat (nsec): min=453493k, max=453493k, avg=453492688.00, stdev= 0.00 lat (nsec): min=476419k, max=476419k, avg=476419403.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.87%, ctx=20, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=81.0MiB/s (85.9MB/s), 81.0MiB/s-81.0MiB/s (85.9MB/s-85.9MB/s), io=40.0MiB (41.9MB), run=488-488msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout