== sanity test 398c: run fio to test AIO ================= 17:55:10 (1788645310) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.583282 s, 71.9 MB/s osc.lustre-OST0000-osc-ffff8dcbe586d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146811: Sat Sep 5 17:58:24 2026 write: IOPS=13, BW=55.5KiB/s (56.9kB/s)(10.0MiB/184374msec); 0 zone resets slat (usec): min=27, max=20157, avg=256.71, stdev=1012.59 clat (msec): min=311, max=2676, avg=1151.10, stdev=340.75 lat (msec): min=311, max=2676, avg=1151.36, stdev=340.75 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 609], 10.00th=[ 760], 20.00th=[ 877], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1586], 95.00th=[ 1770], | 99.00th=[ 2089], 99.50th=[ 2198], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 128, per=25.37%, avg=56.33, stdev=33.06, samples=357 iops : min= 1, max= 32, avg=13.60, stdev= 8.28, samples=357 lat (msec) : 500=1.91%, 750=7.34%, 1000=23.87%, 2000=64.88%, >=2000=1.99% cpu : usr=0.04%, sys=0.21%, ctx=1765, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146812: Sat Sep 5 17:58:24 2026 write: IOPS=13, BW=55.7KiB/s (57.0kB/s)(10.0MiB/183858msec); 0 zone resets slat (usec): min=23, max=16658, avg=298.34, stdev=1083.75 clat (msec): min=203, max=2731, avg=1147.94, stdev=343.63 lat (msec): min=203, max=2732, avg=1148.24, stdev=343.64 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 617], 10.00th=[ 751], 20.00th=[ 877], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1401], 90.00th=[ 1603], 95.00th=[ 1787], | 99.00th=[ 2039], 99.50th=[ 2165], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2735] bw ( KiB/s): min= 7, max= 128, per=25.52%, avg=56.65, stdev=29.69, samples=355 iops : min= 1, max= 32, avg=13.68, stdev= 7.45, samples=355 lat (msec) : 250=0.62%, 500=1.13%, 750=8.28%, 1000=24.34%, 2000=63.87% lat (msec) : >=2000=1.76% cpu : usr=0.04%, sys=0.23%, ctx=1851, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146813: Sat Sep 5 17:58:24 2026 write: IOPS=13, BW=55.5KiB/s (56.9kB/s)(10.0MiB/184374msec); 0 zone resets slat (usec): min=23, max=20767, avg=283.72, stdev=1100.26 clat (msec): min=429, max=2471, avg=1150.94, stdev=331.25 lat (msec): min=429, max=2471, avg=1151.23, stdev=331.24 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 634], 10.00th=[ 785], 20.00th=[ 894], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1418], 90.00th=[ 1620], 95.00th=[ 1754], | 99.00th=[ 2039], 99.50th=[ 2106], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 128, per=25.90%, avg=57.51, stdev=29.17, samples=350 iops : min= 1, max= 32, avg=13.89, stdev= 7.29, samples=350 lat (msec) : 500=1.13%, 750=7.58%, 1000=25.39%, 2000=64.41%, >=2000=1.48% cpu : usr=0.04%, sys=0.23%, ctx=1864, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146814: Sat Sep 5 17:58:24 2026 write: IOPS=13, BW=55.6KiB/s (56.0kB/s)(10.0MiB/184075msec); 0 zone resets slat (usec): min=24, max=23852, avg=290.20, stdev=1180.60 clat (msec): min=86, max=2693, avg=1148.73, stdev=340.72 lat (msec): min=87, max=2694, avg=1149.03, stdev=340.70 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 625], 10.00th=[ 768], 20.00th=[ 877], | 30.00th=[ 969], 40.00th=[ 1028], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1603], 95.00th=[ 1754], | 99.00th=[ 2165], 99.50th=[ 2265], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 136, per=25.16%, avg=55.86, stdev=29.64, samples=360 iops : min= 1, max= 34, avg=13.48, stdev= 7.41, samples=360 lat (msec) : 100=0.04%, 250=0.04%, 500=1.91%, 750=7.15%, 1000=26.02% lat (msec) : 2000=63.28%, >=2000=1.56% cpu : usr=0.05%, sys=0.22%, ctx=1792, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=222KiB/s (227kB/s), 55.5KiB/s-55.7KiB/s (56.9kB/s-57.0kB/s), io=40.0MiB (41.9MB), run=183858-184374msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146852: Sat Sep 5 18:00:29 2026 read: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5048KiB/111810msec) slat (msec): min=2, max=410, avg=15.33, stdev=13.84 clat (msec): min=19, max=1019, avg=438.09, stdev=129.30 lat (msec): min=36, max=1047, avg=453.43, stdev=132.02 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 334], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 489], 80.00th=[ 535], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 793], 99.50th=[ 827], 99.90th=[ 1003], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 103, per=25.10%, avg=44.93, stdev=20.14, samples=220 iops : min= 1, max= 25, avg=10.74, stdev= 5.06, samples=220 write: IOPS=11, BW=46.4KiB/s (47.6kB/s)(5192KiB/111810msec); 0 zone resets slat (usec): min=25, max=19951, avg=348.27, stdev=959.75 clat (msec): min=384, max=1846, avg=933.23, stdev=255.84 lat (msec): min=384, max=1846, avg=933.58, stdev=255.82 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 531], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1062], 80.00th=[ 1167], 90.00th=[ 1284], 95.00th=[ 1368], | 99.00th=[ 1586], 99.50th=[ 1687], 99.90th=[ 1770], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 104, per=24.99%, avg=46.47, stdev=20.26, samples=219 iops : min= 1, max= 26, avg=11.13, stdev= 5.07, samples=219 lat (msec) : 20=0.04%, 50=0.04%, 100=0.35%, 250=1.45%, 500=35.98% lat (msec) : 750=23.09%, 1000=20.35%, 2000=18.71% cpu : usr=0.13%, sys=0.70%, ctx=4808, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146853: Sat Sep 5 18:00:29 2026 read: IOPS=11, BW=44.5KiB/s (45.5kB/s)(4976KiB/111908msec) slat (usec): min=3431, max=65632, avg=15059.58, stdev=8188.92 clat (msec): min=45, max=1020, avg=431.92, stdev=123.11 lat (msec): min=67, max=1049, avg=446.99, stdev=124.88 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 257], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 485], 80.00th=[ 527], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 768], 99.50th=[ 835], 99.90th=[ 927], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 104, per=24.44%, avg=43.75, stdev=21.79, samples=222 iops : min= 1, max= 26, avg=10.45, stdev= 5.43, samples=222 write: IOPS=11, BW=47.0KiB/s (48.2kB/s)(5264KiB/111908msec); 0 zone resets slat (usec): min=32, max=16955, avg=294.98, stdev=822.16 clat (msec): min=308, max=1955, avg=935.50, stdev=269.66 lat (msec): min=308, max=1955, avg=935.79, stdev=269.64 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 527], 10.00th=[ 617], 20.00th=[ 701], | 30.00th=[ 776], 40.00th=[ 844], 50.00th=[ 911], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1670], 99.50th=[ 1821], 99.90th=[ 1888], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 119, per=25.12%, avg=46.72, stdev=22.71, samples=221 iops : min= 1, max= 29, avg=11.19, stdev= 5.70, samples=221 lat (msec) : 50=0.04%, 100=0.16%, 250=1.72%, 500=36.09%, 750=23.36% lat (msec) : 1000=19.02%, 2000=19.61% cpu : usr=0.11%, sys=0.72%, ctx=4806, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146854: Sat Sep 5 18:00:29 2026 read: IOPS=11, BW=44.6KiB/s (45.7kB/s)(4984KiB/111740msec) slat (msec): min=2, max=662, avg=15.66, stdev=22.70 clat (msec): min=18, max=1287, avg=435.67, stdev=130.43 lat (msec): min=26, max=1305, avg=451.34, stdev=132.90 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 259], 10.00th=[ 288], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 451], | 70.00th=[ 481], 80.00th=[ 527], 90.00th=[ 609], 95.00th=[ 667], | 99.00th=[ 793], 99.50th=[ 844], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 7, max= 120, per=24.87%, avg=44.52, stdev=22.37, samples=220 iops : min= 1, max= 30, avg=10.64, stdev= 5.63, samples=220 write: IOPS=11, BW=47.0KiB/s (48.2kB/s)(5256KiB/111740msec); 0 zone resets slat (usec): min=33, max=43321, avg=342.99, stdev=1387.18 clat (msec): min=326, max=1909, avg=929.52, stdev=263.02 lat (msec): min=327, max=1909, avg=929.87, stdev=263.01 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 531], 10.00th=[ 609], 20.00th=[ 693], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 911], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1670], 99.50th=[ 1720], 99.90th=[ 1888], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 111, per=25.20%, avg=46.88, stdev=21.71, samples=219 iops : min= 1, max= 27, avg=11.23, stdev= 5.43, samples=219 lat (msec) : 20=0.08%, 50=0.12%, 100=0.20%, 250=1.52%, 500=35.78% lat (msec) : 750=23.52%, 1000=19.26%, 2000=19.53% cpu : usr=0.10%, sys=0.70%, ctx=4759, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146855: Sat Sep 5 18:00:29 2026 read: IOPS=11, BW=45.6KiB/s (46.7kB/s)(5060KiB/110943msec) slat (usec): min=57, max=64900, avg=15185.60, stdev=8156.09 clat (msec): min=25, max=1076, avg=436.99, stdev=134.44 lat (msec): min=32, max=1091, avg=452.19, stdev=136.09 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 255], 10.00th=[ 284], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 397], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 489], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 835], 99.50th=[ 911], 99.90th=[ 944], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 119, per=25.60%, avg=45.82, stdev=20.49, samples=217 iops : min= 1, max= 29, avg=10.97, stdev= 5.17, samples=217 write: IOPS=11, BW=46.7KiB/s (47.8kB/s)(5180KiB/110943msec); 0 zone resets slat (usec): min=36, max=28754, avg=348.28, stdev=1183.54 clat (msec): min=130, max=1860, avg=927.55, stdev=261.67 lat (msec): min=130, max=1860, avg=927.90, stdev=261.69 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 550], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 835], 50.00th=[ 911], 60.00th=[ 986], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1620], 99.50th=[ 1687], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 103, per=24.82%, avg=46.16, stdev=21.38, samples=219 iops : min= 1, max= 25, avg=11.05, stdev= 5.35, samples=219 lat (msec) : 50=0.23%, 100=0.20%, 250=2.19%, 500=34.61%, 750=24.18% lat (msec) : 1000=19.57%, 2000=19.02% cpu : usr=0.12%, sys=0.72%, ctx=4584, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=179KiB/s (184kB/s), 44.5KiB/s-45.6KiB/s (45.5kB/s-46.7kB/s), io=19.6MiB (20.5MB), run=110943-111908msec WRITE: bw=187KiB/s (191kB/s), 46.4KiB/s-47.0KiB/s (47.6kB/s-48.2kB/s), io=20.4MiB (21.4MB), run=110943-111908msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146875: Sat Sep 5 18:00:40 2026 read: IOPS=2, BW=83.3MiB/s (87.4MB/s)(40.0MiB/480msec) slat (nsec): min=45311k, max=45311k, avg=45311123.00, stdev= 0.00 clat (nsec): min=405484k, max=405484k, avg=405483896.00, stdev= 0.00 lat (nsec): min=450799k, max=450799k, avg=450799034.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.13%, ctx=44, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.3MiB/s (87.4MB/s), 83.3MiB/s-83.3MiB/s (87.4MB/s-87.4MB/s), io=40.0MiB (41.9MB), run=480-480msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout