== sanity test 398c: run fio to test AIO ================= 09:46:29 (1788529589) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.465515 s, 90.1 MB/s osc.lustre-OST0000-osc-ffff8e79c05d4800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146512: Fri Sep 4 09:49:02 2026 write: IOPS=17, BW=70.7KiB/s (72.4kB/s)(10.0MiB/144854msec); 0 zone resets slat (usec): min=29, max=14841, avg=181.82, stdev=685.15 clat (msec): min=164, max=2326, avg=904.34, stdev=280.31 lat (msec): min=164, max=2326, avg=904.53, stdev=280.30 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 514], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1217], 95.00th=[ 1418], | 99.00th=[ 1787], 99.50th=[ 2005], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 384, per=25.01%, avg=70.53, stdev=38.71, samples=286 iops : min= 1, max= 96, avg=17.17, stdev= 9.73, samples=286 lat (msec) : 250=1.25%, 500=2.15%, 750=21.80%, 1000=46.37%, 2000=27.89% lat (msec) : >=2000=0.55% cpu : usr=0.03%, sys=0.22%, ctx=1816, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146513: Fri Sep 4 09:49:02 2026 write: IOPS=17, BW=70.8KiB/s (72.5kB/s)(10.0MiB/144656msec); 0 zone resets slat (usec): min=27, max=13627, avg=153.89, stdev=526.28 clat (msec): min=137, max=2324, avg=903.54, stdev=285.33 lat (msec): min=137, max=2328, avg=903.69, stdev=285.35 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 518], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1062], 90.00th=[ 1234], 95.00th=[ 1435], | 99.00th=[ 1821], 99.50th=[ 2005], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 256, per=25.35%, avg=71.48, stdev=35.87, samples=282 iops : min= 1, max= 64, avg=17.41, stdev= 9.00, samples=282 lat (msec) : 250=1.88%, 500=2.11%, 750=21.45%, 1000=46.48%, 2000=27.42% lat (msec) : >=2000=0.66% cpu : usr=0.03%, sys=0.20%, ctx=1427, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146514: Fri Sep 4 09:49:02 2026 write: IOPS=17, BW=70.7KiB/s (72.4kB/s)(10.0MiB/144879msec); 0 zone resets slat (usec): min=27, max=17700, avg=202.98, stdev=797.04 clat (msec): min=159, max=2386, avg=904.68, stdev=284.50 lat (msec): min=160, max=2386, avg=904.88, stdev=284.49 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 506], 10.00th=[ 600], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 986], 80.00th=[ 1062], 90.00th=[ 1217], 95.00th=[ 1435], | 99.00th=[ 1854], 99.50th=[ 1989], 99.90th=[ 2333], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 328, per=25.01%, avg=70.52, stdev=34.43, samples=286 iops : min= 1, max= 82, avg=17.17, stdev= 8.64, samples=286 lat (msec) : 250=1.17%, 500=3.59%, 750=20.47%, 1000=46.64%, 2000=27.70% lat (msec) : >=2000=0.43% cpu : usr=0.05%, sys=0.21%, ctx=2076, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146515: Fri Sep 4 09:49:02 2026 write: IOPS=17, BW=70.8KiB/s (72.5kB/s)(10.0MiB/144655msec); 0 zone resets slat (usec): min=24, max=21374, avg=222.95, stdev=914.20 clat (msec): min=96, max=2265, avg=903.49, stdev=281.92 lat (msec): min=97, max=2265, avg=903.71, stdev=281.91 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 506], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1083], 90.00th=[ 1250], 95.00th=[ 1418], | 99.00th=[ 1821], 99.50th=[ 2089], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 256, per=25.36%, avg=71.52, stdev=33.25, samples=282 iops : min= 1, max= 64, avg=17.42, stdev= 8.35, samples=282 lat (msec) : 100=0.04%, 250=1.33%, 500=3.20%, 750=20.00%, 1000=47.97% lat (msec) : 2000=26.72%, >=2000=0.74% cpu : usr=0.04%, sys=0.22%, ctx=2156, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=283KiB/s (290kB/s), 70.7KiB/s-70.8KiB/s (72.4kB/s-72.5kB/s), io=40.0MiB (41.9MB), run=144655-144879msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146547: Fri Sep 4 09:50:55 2026 read: IOPS=11, BW=46.9KiB/s (48.0kB/s)(5048KiB/107689msec) slat (usec): min=2668, max=56537, avg=13419.31, stdev=7561.02 clat (msec): min=7, max=968, avg=413.05, stdev=128.02 lat (msec): min=18, max=988, avg=426.47, stdev=129.63 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 257], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 414], | 70.00th=[ 447], 80.00th=[ 510], 90.00th=[ 592], 95.00th=[ 651], | 99.00th=[ 844], 99.50th=[ 902], 99.90th=[ 953], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 6, max= 112, per=25.04%, avg=46.58, stdev=22.54, samples=213 iops : min= 1, max= 28, avg=11.20, stdev= 5.69, samples=213 write: IOPS=12, BW=48.2KiB/s (49.4kB/s)(5192KiB/107689msec); 0 zone resets slat (usec): min=31, max=9852, avg=219.47, stdev=620.59 clat (msec): min=333, max=2317, avg=910.31, stdev=316.72 lat (msec): min=333, max=2317, avg=910.53, stdev=316.70 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 502], 10.00th=[ 558], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 1003], 80.00th=[ 1116], 90.00th=[ 1318], 95.00th=[ 1519], | 99.00th=[ 2165], 99.50th=[ 2232], 99.90th=[ 2265], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 103, per=24.84%, avg=48.20, stdev=20.54, samples=211 iops : min= 1, max= 25, avg=11.60, stdev= 5.16, samples=211 lat (msec) : 10=0.04%, 50=0.08%, 100=0.08%, 250=1.91%, 500=39.02% lat (msec) : 750=23.63%, 1000=19.96%, 2000=14.73%, >=2000=0.55% cpu : usr=0.09%, sys=0.53%, ctx=4663, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146548: Fri Sep 4 09:50:55 2026 read: IOPS=11, BW=46.2KiB/s (47.3kB/s)(4976KiB/107684msec) slat (usec): min=37, max=419231, avg=13158.74, stdev=13864.29 clat (msec): min=31, max=988, avg=416.82, stdev=131.11 lat (msec): min=31, max=1003, avg=429.99, stdev=132.03 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 251], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 393], 60.00th=[ 426], | 70.00th=[ 456], 80.00th=[ 510], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 835], 99.50th=[ 860], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 119, per=24.65%, avg=45.85, stdev=23.21, samples=213 iops : min= 1, max= 29, avg=11.01, stdev= 5.88, samples=213 write: IOPS=12, BW=48.9KiB/s (50.1kB/s)(5264KiB/107684msec); 0 zone resets slat (usec): min=31, max=14211, avg=229.75, stdev=752.53 clat (msec): min=164, max=2298, avg=901.69, stdev=319.08 lat (msec): min=164, max=2298, avg=901.92, stdev=319.06 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 485], 10.00th=[ 550], 20.00th=[ 667], | 30.00th=[ 743], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 911], | 70.00th=[ 995], 80.00th=[ 1116], 90.00th=[ 1318], 95.00th=[ 1502], | 99.00th=[ 2005], 99.50th=[ 2165], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 135, per=25.06%, avg=48.61, stdev=22.90, samples=212 iops : min= 1, max= 33, avg=11.71, stdev= 5.76, samples=212 lat (msec) : 50=0.35%, 100=0.08%, 250=2.46%, 500=38.67%, 750=22.54% lat (msec) : 1000=20.62%, 2000=14.77%, >=2000=0.51% cpu : usr=0.09%, sys=0.53%, ctx=4183, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146549: Fri Sep 4 09:50:55 2026 read: IOPS=11, BW=46.3KiB/s (47.4kB/s)(4984KiB/107689msec) slat (usec): min=2276, max=66097, avg=13092.97, stdev=7590.97 clat (msec): min=10, max=995, avg=414.17, stdev=130.26 lat (msec): min=25, max=1005, avg=427.26, stdev=131.72 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 257], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 414], | 70.00th=[ 451], 80.00th=[ 506], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 827], 99.50th=[ 877], 99.90th=[ 986], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 112, per=24.91%, avg=46.33, stdev=22.27, samples=212 iops : min= 1, max= 28, avg=11.14, stdev= 5.61, samples=212 write: IOPS=12, BW=48.8KiB/s (49.0kB/s)(5256KiB/107689msec); 0 zone resets slat (usec): min=31, max=14298, avg=237.31, stdev=826.06 clat (msec): min=268, max=2304, avg=903.08, stdev=312.54 lat (msec): min=269, max=2305, avg=903.32, stdev=312.51 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 510], 10.00th=[ 575], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 902], | 70.00th=[ 986], 80.00th=[ 1099], 90.00th=[ 1301], 95.00th=[ 1552], | 99.00th=[ 1905], 99.50th=[ 2123], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 127, per=24.96%, avg=48.42, stdev=21.06, samples=212 iops : min= 1, max= 31, avg=11.65, stdev= 5.30, samples=212 lat (msec) : 20=0.08%, 50=0.12%, 100=0.04%, 250=1.84%, 500=38.63% lat (msec) : 750=24.65%, 1000=19.88%, 2000=14.30%, >=2000=0.47% cpu : usr=0.10%, sys=0.57%, ctx=4527, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146550: Fri Sep 4 09:50:55 2026 read: IOPS=11, BW=46.0KiB/s (48.1kB/s)(5060KiB/107688msec) slat (usec): min=2213, max=68804, avg=12810.10, stdev=7302.03 clat (msec): min=25, max=987, avg=418.07, stdev=131.44 lat (msec): min=38, max=1000, avg=430.88, stdev=132.96 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 264], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 414], | 70.00th=[ 451], 80.00th=[ 506], 90.00th=[ 609], 95.00th=[ 684], | 99.00th=[ 827], 99.50th=[ 869], 99.90th=[ 969], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 119, per=25.38%, avg=47.21, stdev=22.21, samples=211 iops : min= 1, max= 29, avg=11.36, stdev= 5.63, samples=211 write: IOPS=12, BW=48.1KiB/s (49.3kB/s)(5180KiB/107688msec); 0 zone resets slat (usec): min=31, max=12022, avg=199.46, stdev=578.08 clat (msec): min=136, max=2322, avg=907.30, stdev=315.18 lat (msec): min=136, max=2322, avg=907.51, stdev=315.18 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 514], 10.00th=[ 567], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 911], | 70.00th=[ 995], 80.00th=[ 1116], 90.00th=[ 1334], 95.00th=[ 1519], | 99.00th=[ 2005], 99.50th=[ 2198], 99.90th=[ 2299], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 135, per=24.62%, avg=47.76, stdev=22.58, samples=212 iops : min= 1, max= 33, avg=11.49, stdev= 5.67, samples=212 lat (msec) : 50=0.08%, 100=0.16%, 250=1.60%, 500=39.26%, 750=24.41% lat (msec) : 1000=19.57%, 2000=14.41%, >=2000=0.51% cpu : usr=0.10%, sys=0.50%, ctx=4550, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=186KiB/s (191kB/s), 46.2KiB/s-46.0KiB/s (47.3kB/s-48.1kB/s), io=19.6MiB (20.5MB), run=107684-107689msec WRITE: bw=194KiB/s (199kB/s), 48.1KiB/s-48.9KiB/s (49.3kB/s-50.1kB/s), io=20.4MiB (21.4MB), run=107684-107689msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146569: Fri Sep 4 09:51:05 2026 read: IOPS=1, BW=77.5MiB/s (81.3MB/s)(40.0MiB/516msec) slat (nsec): min=51907k, max=51907k, avg=51906614.00, stdev= 0.00 clat (nsec): min=448563k, max=448563k, avg=448563318.00, stdev= 0.00 lat (nsec): min=500474k, max=500474k, avg=500474314.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.14%, ctx=42, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.5MiB/s (81.3MB/s), 77.5MiB/s-77.5MiB/s (81.3MB/s-81.3MB/s), io=40.0MiB (41.9MB), run=516-516msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout