== sanity test 398c: run fio to test AIO ================= 12:41:37 (1789058497) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.292076 s, 144 MB/s osc.lustre-OST0000-osc-ffff9bbe83dcc000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145444: Thu Sep 10 12:44:59 2026 write: IOPS=13, BW=54.0KiB/s (55.3kB/s)(10.0MiB/189581msec); 0 zone resets slat (usec): min=28, max=18517, avg=231.47, stdev=880.32 clat (msec): min=70, max=2537, avg=1183.37, stdev=295.16 lat (msec): min=70, max=2538, avg=1183.60, stdev=295.15 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 760], 10.00th=[ 844], 20.00th=[ 953], | 30.00th=[ 1036], 40.00th=[ 1116], 50.00th=[ 1167], 60.00th=[ 1234], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1552], 95.00th=[ 1670], | 99.00th=[ 2022], 99.50th=[ 2165], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 191, per=25.18%, avg=54.13, stdev=23.73, samples=371 iops : min= 1, max= 47, avg=13.09, stdev= 5.98, samples=371 lat (msec) : 100=0.04%, 250=0.31%, 500=1.25%, 750=2.89%, 1000=20.31% lat (msec) : 2000=74.02%, >=2000=1.17% cpu : usr=0.04%, sys=0.21%, ctx=2098, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145445: Thu Sep 10 12:44:59 2026 write: IOPS=13, BW=53.0KiB/s (55.3kB/s)(10.0MiB/189664msec); 0 zone resets slat (usec): min=27, max=17838, avg=205.29, stdev=824.16 clat (msec): min=270, max=2332, avg=1183.84, stdev=285.91 lat (msec): min=270, max=2332, avg=1184.05, stdev=285.92 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 751], 10.00th=[ 860], 20.00th=[ 953], | 30.00th=[ 1045], 40.00th=[ 1116], 50.00th=[ 1183], 60.00th=[ 1234], | 70.00th=[ 1318], 80.00th=[ 1401], 90.00th=[ 1536], 95.00th=[ 1636], | 99.00th=[ 1989], 99.50th=[ 2089], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 128, per=25.46%, avg=54.73, stdev=26.04, samples=367 iops : min= 1, max= 32, avg=13.23, stdev= 6.55, samples=367 lat (msec) : 500=1.76%, 750=3.20%, 1000=19.57%, 2000=74.49%, >=2000=0.98% cpu : usr=0.04%, sys=0.17%, ctx=1618, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145446: Thu Sep 10 12:44:59 2026 write: IOPS=13, BW=54.1KiB/s (55.4kB/s)(10.0MiB/189262msec); 0 zone resets slat (usec): min=28, max=26372, avg=255.55, stdev=1149.07 clat (msec): min=259, max=2361, avg=1181.95, stdev=290.12 lat (msec): min=259, max=2361, avg=1182.21, stdev=290.13 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 760], 10.00th=[ 844], 20.00th=[ 953], | 30.00th=[ 1036], 40.00th=[ 1099], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1318], 80.00th=[ 1401], 90.00th=[ 1519], 95.00th=[ 1636], | 99.00th=[ 2005], 99.50th=[ 2089], 99.90th=[ 2232], 99.95th=[ 2333], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 255, per=25.74%, avg=55.35, stdev=29.87, samples=363 iops : min= 1, max= 63, avg=13.39, stdev= 7.50, samples=363 lat (msec) : 500=1.25%, 750=3.71%, 1000=20.70%, 2000=73.16%, >=2000=1.17% cpu : usr=0.04%, sys=0.21%, ctx=1596, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145447: Thu Sep 10 12:44:59 2026 write: IOPS=13, BW=53.0KiB/s (55.3kB/s)(10.0MiB/189702msec); 0 zone resets slat (usec): min=29, max=32958, avg=260.05, stdev=1170.99 clat (msec): min=264, max=2247, avg=1184.00, stdev=280.04 lat (msec): min=264, max=2247, avg=1184.26, stdev=280.05 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 768], 10.00th=[ 860], 20.00th=[ 953], | 30.00th=[ 1036], 40.00th=[ 1116], 50.00th=[ 1183], 60.00th=[ 1234], | 70.00th=[ 1318], 80.00th=[ 1401], 90.00th=[ 1536], 95.00th=[ 1636], | 99.00th=[ 1905], 99.50th=[ 2039], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 128, per=25.11%, avg=53.99, stdev=23.68, samples=372 iops : min= 1, max= 32, avg=13.06, stdev= 5.96, samples=372 lat (msec) : 500=1.25%, 750=3.09%, 1000=21.02%, 2000=73.98%, >=2000=0.66% cpu : usr=0.04%, sys=0.22%, ctx=1804, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=216KiB/s (221kB/s), 53.0KiB/s-54.1KiB/s (55.3kB/s-55.4kB/s), io=40.0MiB (41.9MB), run=189262-189702msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145484: Thu Sep 10 12:47:16 2026 read: IOPS=9, BW=39.8KiB/s (40.7kB/s)(5048KiB/126895msec) slat (usec): min=30, max=68280, avg=14974.90, stdev=8878.57 clat (msec): min=10, max=1346, avg=494.96, stdev=158.99 lat (msec): min=19, max=1356, avg=509.94, stdev=160.90 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 313], 10.00th=[ 347], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 468], 60.00th=[ 506], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 693], 95.00th=[ 776], | 99.00th=[ 1011], 99.50th=[ 1083], 99.90th=[ 1284], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 104, per=25.16%, avg=39.50, stdev=21.11, samples=249 iops : min= 1, max= 26, avg= 9.38, stdev= 5.30, samples=249 write: IOPS=10, BW=40.9KiB/s (41.9kB/s)(5192KiB/126895msec); 0 zone resets slat (usec): min=29, max=12487, avg=280.50, stdev=971.10 clat (msec): min=246, max=2572, avg=1066.82, stdev=350.34 lat (msec): min=246, max=2572, avg=1067.10, stdev=350.39 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 642], 10.00th=[ 709], 20.00th=[ 785], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1183], 80.00th=[ 1301], 90.00th=[ 1502], 95.00th=[ 1754], | 99.00th=[ 2265], 99.50th=[ 2333], 99.90th=[ 2500], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 94, per=24.99%, avg=40.73, stdev=18.64, samples=249 iops : min= 1, max= 23, avg= 9.69, stdev= 4.70, samples=249 lat (msec) : 20=0.08%, 50=0.16%, 100=0.27%, 250=0.47%, 500=28.28% lat (msec) : 750=24.18%, 1000=20.39%, 2000=24.69%, >=2000=1.48% cpu : usr=0.09%, sys=0.48%, ctx=4420, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145485: Thu Sep 10 12:47:16 2026 read: IOPS=9, BW=38.9KiB/s (39.9kB/s)(4976KiB/127773msec) slat (usec): min=2860, max=72740, avg=15411.43, stdev=8665.88 clat (msec): min=17, max=1163, avg=493.39, stdev=153.65 lat (msec): min=25, max=1184, avg=508.81, stdev=155.77 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 313], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 502], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 693], 95.00th=[ 793], | 99.00th=[ 1020], 99.50th=[ 1053], 99.90th=[ 1150], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 5, max= 103, per=24.54%, avg=38.52, stdev=20.83, samples=250 iops : min= 1, max= 25, avg= 9.14, stdev= 5.23, samples=250 write: IOPS=10, BW=41.2KiB/s (42.2kB/s)(5264KiB/127773msec); 0 zone resets slat (usec): min=31, max=21248, avg=260.11, stdev=1113.13 clat (msec): min=133, max=2572, avg=1069.82, stdev=355.75 lat (msec): min=134, max=2572, avg=1070.08, stdev=355.76 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1770], | 99.00th=[ 2333], 99.50th=[ 2467], 99.90th=[ 2500], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 110, per=25.11%, avg=40.93, stdev=19.07, samples=250 iops : min= 1, max= 27, avg= 9.74, stdev= 4.81, samples=250 lat (msec) : 20=0.04%, 50=0.04%, 100=0.04%, 250=0.74%, 500=29.06% lat (msec) : 750=22.97%, 1000=20.90%, 2000=24.53%, >=2000=1.68% cpu : usr=0.07%, sys=0.48%, ctx=4118, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145486: Thu Sep 10 12:47:16 2026 read: IOPS=9, BW=39.0KiB/s (39.0kB/s)(4984KiB/127634msec) slat (usec): min=3224, max=70064, avg=15360.84, stdev=9157.56 clat (msec): min=19, max=1210, avg=493.45, stdev=152.68 lat (msec): min=35, max=1226, avg=508.81, stdev=155.01 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 313], 10.00th=[ 338], 20.00th=[ 380], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 684], 95.00th=[ 776], | 99.00th=[ 1003], 99.50th=[ 1053], 99.90th=[ 1150], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 112, per=24.96%, avg=39.19, stdev=19.74, samples=249 iops : min= 1, max= 28, avg= 9.31, stdev= 4.94, samples=249 write: IOPS=10, BW=41.2KiB/s (42.2kB/s)(5256KiB/127634msec); 0 zone resets slat (usec): min=31, max=11914, avg=265.51, stdev=876.47 clat (msec): min=303, max=2571, avg=1067.46, stdev=347.09 lat (msec): min=303, max=2571, avg=1067.72, stdev=347.06 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 651], 10.00th=[ 718], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1502], 95.00th=[ 1737], | 99.00th=[ 2299], 99.50th=[ 2400], 99.90th=[ 2534], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 118, per=25.40%, avg=41.40, stdev=19.65, samples=247 iops : min= 1, max= 29, avg= 9.87, stdev= 4.97, samples=247 lat (msec) : 20=0.04%, 50=0.16%, 100=0.20%, 250=0.27%, 500=28.44% lat (msec) : 750=23.59%, 1000=21.88%, 2000=24.02%, >=2000=1.41% cpu : usr=0.07%, sys=0.46%, ctx=3825, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145487: Thu Sep 10 12:47:16 2026 read: IOPS=9, BW=39.9KiB/s (40.8kB/s)(5060KiB/126893msec) slat (usec): min=2991, max=63159, avg=15054.71, stdev=8530.12 clat (msec): min=24, max=1289, avg=496.58, stdev=158.83 lat (msec): min=32, max=1295, avg=511.64, stdev=160.74 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 372], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 502], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 701], 95.00th=[ 802], | 99.00th=[ 1045], 99.50th=[ 1133], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 7, max= 111, per=25.41%, avg=39.89, stdev=20.44, samples=248 iops : min= 1, max= 27, avg= 9.47, stdev= 5.14, samples=248 write: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5180KiB/126893msec); 0 zone resets slat (usec): min=32, max=12932, avg=193.06, stdev=522.20 clat (msec): min=447, max=2547, avg=1064.76, stdev=341.09 lat (msec): min=447, max=2547, avg=1064.96, stdev=341.11 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 651], 10.00th=[ 709], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 927], 50.00th=[ 995], 60.00th=[ 1070], | 70.00th=[ 1183], 80.00th=[ 1318], 90.00th=[ 1502], 95.00th=[ 1737], | 99.00th=[ 2165], 99.50th=[ 2265], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 95, per=25.03%, avg=40.80, stdev=18.85, samples=248 iops : min= 1, max= 23, avg= 9.71, stdev= 4.75, samples=248 lat (msec) : 50=0.04%, 100=0.04%, 250=0.51%, 500=29.06%, 750=23.79% lat (msec) : 1000=21.02%, 2000=24.38%, >=2000=1.17% cpu : usr=0.07%, sys=0.49%, ctx=4237, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=157KiB/s (161kB/s), 38.9KiB/s-39.9KiB/s (39.9kB/s-40.8kB/s), io=19.6MiB (20.5MB), run=126893-127773msec WRITE: bw=164KiB/s (167kB/s), 40.8KiB/s-41.2KiB/s (41.8kB/s-42.2kB/s), io=20.4MiB (21.4MB), run=126893-127773msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145509: Thu Sep 10 12:47:26 2026 read: IOPS=2, BW=96.9MiB/s (102MB/s)(40.0MiB/413msec) slat (nsec): min=33123k, max=33123k, avg=33123081.00, stdev= 0.00 clat (nsec): min=356839k, max=356839k, avg=356839273.00, stdev= 0.00 lat (nsec): min=389966k, max=389966k, avg=389966472.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] lat (msec) : 500=100.00% cpu : usr=0.24%, sys=2.67%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=96.9MiB/s (102MB/s), 96.9MiB/s-96.9MiB/s (102MB/s-102MB/s), io=40.0MiB (41.9MB), run=413-413msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout