== sanity test 398c: run fio to test AIO ================= 03:34:14 (1785224054) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.360286 s, 116 MB/s osc.lustre-OST0000-osc-ffff99d68b279000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145333: Tue Jul 28 03:36:35 2026 write: IOPS=19, BW=77.0KiB/s (78.9kB/s)(10.0MiB/132953msec); 0 zone resets slat (usec): min=35, max=17190, avg=329.52, stdev=909.67 clat (msec): min=167, max=1430, avg=829.70, stdev=199.65 lat (msec): min=168, max=1430, avg=830.03, stdev=199.65 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 877], | 70.00th=[ 927], 80.00th=[ 995], 90.00th=[ 1083], 95.00th=[ 1167], | 99.00th=[ 1267], 99.50th=[ 1318], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 256, per=24.81%, avg=76.40, stdev=30.08, samples=263 iops : min= 1, max= 64, avg=18.60, stdev= 7.55, samples=263 lat (msec) : 250=0.90%, 500=3.40%, 750=29.41%, 1000=47.07%, 2000=19.22% cpu : usr=0.06%, sys=0.37%, ctx=2315, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145334: Tue Jul 28 03:36:35 2026 write: IOPS=19, BW=77.1KiB/s (78.0kB/s)(10.0MiB/132774msec); 0 zone resets slat (usec): min=37, max=24927, avg=360.47, stdev=1161.60 clat (msec): min=97, max=1442, avg=828.64, stdev=200.07 lat (msec): min=97, max=1442, avg=829.01, stdev=200.07 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 506], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 835], 60.00th=[ 877], | 70.00th=[ 927], 80.00th=[ 986], 90.00th=[ 1083], 95.00th=[ 1150], | 99.00th=[ 1284], 99.50th=[ 1318], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1435] bw ( KiB/s): min= 15, max= 248, per=24.87%, avg=76.61, stdev=28.56, samples=263 iops : min= 3, max= 62, avg=18.65, stdev= 7.18, samples=263 lat (msec) : 100=0.04%, 250=0.98%, 500=3.83%, 750=26.91%, 1000=50.66% lat (msec) : 2000=17.58% cpu : usr=0.05%, sys=0.38%, ctx=2176, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145335: Tue Jul 28 03:36:35 2026 write: IOPS=19, BW=77.2KiB/s (79.0kB/s)(10.0MiB/132724msec); 0 zone resets slat (usec): min=37, max=19404, avg=330.19, stdev=1031.12 clat (msec): min=188, max=1436, avg=828.79, stdev=199.44 lat (msec): min=188, max=1438, avg=829.12, stdev=199.46 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 514], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 927], 80.00th=[ 986], 90.00th=[ 1070], 95.00th=[ 1150], | 99.00th=[ 1284], 99.50th=[ 1318], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 256, per=24.77%, avg=76.29, stdev=29.97, samples=264 iops : min= 1, max= 64, avg=18.57, stdev= 7.52, samples=264 lat (msec) : 250=1.25%, 500=3.52%, 750=27.62%, 1000=49.34%, 2000=18.28% cpu : usr=0.06%, sys=0.41%, ctx=2298, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145336: Tue Jul 28 03:36:35 2026 write: IOPS=19, BW=77.1KiB/s (78.0kB/s)(10.0MiB/132777msec); 0 zone resets slat (usec): min=36, max=15958, avg=315.70, stdev=859.83 clat (msec): min=188, max=1430, avg=828.56, stdev=201.93 lat (msec): min=188, max=1430, avg=828.88, stdev=201.91 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 493], 10.00th=[ 584], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 877], | 70.00th=[ 927], 80.00th=[ 986], 90.00th=[ 1083], 95.00th=[ 1167], | 99.00th=[ 1301], 99.50th=[ 1318], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 240, per=25.07%, avg=77.20, stdev=30.37, samples=261 iops : min= 1, max= 60, avg=18.80, stdev= 7.63, samples=261 lat (msec) : 250=1.02%, 500=4.10%, 750=27.77%, 1000=49.14%, 2000=17.97% cpu : usr=0.07%, sys=0.37%, ctx=2404, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=308KiB/s (315kB/s), 77.0KiB/s-77.2KiB/s (78.9kB/s-79.0kB/s), io=40.0MiB (41.9MB), run=132724-132953msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145364: Tue Jul 28 03:38:09 2026 read: IOPS=14, BW=58.6KiB/s (60.0kB/s)(5048KiB/86117msec) slat (usec): min=88, max=49562, avg=12582.29, stdev=6715.83 clat (msec): min=22, max=650, avg=331.15, stdev=87.71 lat (msec): min=22, max=676, avg=343.74, stdev=89.04 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 201], 10.00th=[ 236], 20.00th=[ 264], | 30.00th=[ 288], 40.00th=[ 305], 50.00th=[ 321], 60.00th=[ 342], | 70.00th=[ 372], 80.00th=[ 405], 90.00th=[ 447], 95.00th=[ 481], | 99.00th=[ 567], 99.50th=[ 600], 99.90th=[ 634], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 7, max= 174, per=25.10%, avg=57.98, stdev=26.80, samples=171 iops : min= 1, max= 43, avg=13.97, stdev= 6.72, samples=171 write: IOPS=15, BW=60.3KiB/s (61.7kB/s)(5192KiB/86117msec); 0 zone resets slat (usec): min=41, max=8852, avg=401.31, stdev=791.15 clat (msec): min=103, max=1451, avg=726.17, stdev=210.60 lat (msec): min=103, max=1452, avg=726.58, stdev=210.59 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 430], 10.00th=[ 481], 20.00th=[ 550], | 30.00th=[ 600], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 902], 90.00th=[ 1011], 95.00th=[ 1099], | 99.00th=[ 1368], 99.50th=[ 1401], 99.90th=[ 1435], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 8, max= 119, per=24.68%, avg=59.47, stdev=22.75, samples=171 iops : min= 2, max= 29, avg=14.33, stdev= 5.72, samples=171 lat (msec) : 50=0.31%, 100=0.20%, 250=7.23%, 500=45.74%, 750=26.09% lat (msec) : 1000=15.04%, 2000=5.39% cpu : usr=0.12%, sys=0.97%, ctx=4952, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145365: Tue Jul 28 03:38:09 2026 read: IOPS=14, BW=57.6KiB/s (58.0kB/s)(4976KiB/86373msec) slat (usec): min=2240, max=50707, avg=12683.53, stdev=6603.50 clat (msec): min=12, max=700, avg=332.85, stdev=87.93 lat (msec): min=31, max=708, avg=345.54, stdev=88.79 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 207], 10.00th=[ 232], 20.00th=[ 264], | 30.00th=[ 288], 40.00th=[ 305], 50.00th=[ 330], 60.00th=[ 347], | 70.00th=[ 372], 80.00th=[ 401], 90.00th=[ 439], 95.00th=[ 493], | 99.00th=[ 575], 99.50th=[ 617], 99.90th=[ 676], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 7, max= 135, per=24.95%, avg=57.63, stdev=24.38, samples=169 iops : min= 1, max= 33, avg=13.87, stdev= 6.09, samples=169 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(5264KiB/86373msec); 0 zone resets slat (usec): min=47, max=23877, avg=431.04, stdev=1203.06 clat (msec): min=205, max=1542, avg=720.53, stdev=205.43 lat (msec): min=206, max=1542, avg=720.97, stdev=205.45 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 430], 10.00th=[ 477], 20.00th=[ 542], | 30.00th=[ 600], 40.00th=[ 651], 50.00th=[ 701], 60.00th=[ 743], | 70.00th=[ 810], 80.00th=[ 902], 90.00th=[ 1003], 95.00th=[ 1099], | 99.00th=[ 1301], 99.50th=[ 1418], 99.90th=[ 1452], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 15, max= 119, per=25.22%, avg=60.77, stdev=23.11, samples=170 iops : min= 3, max= 29, avg=14.66, stdev= 5.81, samples=170 lat (msec) : 20=0.04%, 50=0.20%, 100=0.12%, 250=6.91%, 500=46.25% lat (msec) : 750=26.64%, 1000=14.69%, 2000=5.16% cpu : usr=0.12%, sys=0.97%, ctx=4871, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145366: Tue Jul 28 03:38:09 2026 read: IOPS=14, BW=57.6KiB/s (58.9kB/s)(4984KiB/86591msec) slat (usec): min=2468, max=56403, avg=12779.76, stdev=6887.05 clat (msec): min=13, max=671, avg=334.94, stdev=90.41 lat (msec): min=24, max=694, avg=347.73, stdev=91.73 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 201], 10.00th=[ 234], 20.00th=[ 262], | 30.00th=[ 284], 40.00th=[ 305], 50.00th=[ 330], 60.00th=[ 351], | 70.00th=[ 372], 80.00th=[ 405], 90.00th=[ 447], 95.00th=[ 502], | 99.00th=[ 600], 99.50th=[ 617], 99.90th=[ 659], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 8, max= 136, per=24.83%, avg=57.35, stdev=25.19, samples=171 iops : min= 2, max= 34, avg=13.81, stdev= 6.30, samples=171 write: IOPS=15, BW=60.7KiB/s (62.2kB/s)(5256KiB/86591msec); 0 zone resets slat (usec): min=41, max=23022, avg=421.70, stdev=1033.43 clat (msec): min=181, max=1629, avg=721.78, stdev=208.87 lat (msec): min=181, max=1630, avg=722.20, stdev=208.87 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 426], 10.00th=[ 464], 20.00th=[ 535], | 30.00th=[ 600], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 760], | 70.00th=[ 818], 80.00th=[ 894], 90.00th=[ 1003], 95.00th=[ 1083], | 99.00th=[ 1284], 99.50th=[ 1435], 99.90th=[ 1603], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 111, per=24.84%, avg=59.85, stdev=22.43, samples=172 iops : min= 1, max= 27, avg=14.44, stdev= 5.66, samples=172 lat (msec) : 20=0.04%, 50=0.20%, 100=0.12%, 250=7.19%, 500=45.59% lat (msec) : 750=25.90%, 1000=15.90%, 2000=5.08% cpu : usr=0.13%, sys=0.90%, ctx=5267, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145367: Tue Jul 28 03:38:09 2026 read: IOPS=14, BW=58.4KiB/s (59.8kB/s)(5060KiB/86585msec) slat (usec): min=2679, max=53263, avg=12505.40, stdev=6628.59 clat (msec): min=57, max=662, avg=338.28, stdev=88.66 lat (msec): min=68, max=690, avg=350.79, stdev=89.60 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 205], 10.00th=[ 234], 20.00th=[ 264], | 30.00th=[ 288], 40.00th=[ 313], 50.00th=[ 334], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 405], 90.00th=[ 447], 95.00th=[ 502], | 99.00th=[ 592], 99.50th=[ 617], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 7, max= 151, per=25.29%, avg=58.41, stdev=24.67, samples=170 iops : min= 1, max= 37, avg=14.07, stdev= 6.17, samples=170 write: IOPS=14, BW=59.8KiB/s (61.3kB/s)(5180KiB/86585msec); 0 zone resets slat (usec): min=44, max=11708, avg=442.84, stdev=962.41 clat (msec): min=224, max=1473, avg=723.84, stdev=210.14 lat (msec): min=224, max=1474, avg=724.29, stdev=210.15 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 422], 10.00th=[ 477], 20.00th=[ 542], | 30.00th=[ 592], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 902], 90.00th=[ 995], 95.00th=[ 1116], | 99.00th=[ 1318], 99.50th=[ 1351], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 15, max= 119, per=24.69%, avg=59.50, stdev=21.75, samples=171 iops : min= 3, max= 29, avg=14.35, stdev= 5.48, samples=171 lat (msec) : 100=0.23%, 250=6.95%, 500=45.94%, 750=26.21%, 1000=15.86% lat (msec) : 2000=4.80% cpu : usr=0.14%, sys=0.95%, ctx=4885, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=232KiB/s (237kB/s), 57.6KiB/s-58.6KiB/s (58.9kB/s-60.0kB/s), io=19.6MiB (20.5MB), run=86117-86591msec WRITE: bw=241KiB/s (247kB/s), 59.8KiB/s-60.9KiB/s (61.3kB/s-62.4kB/s), io=20.4MiB (21.4MB), run=86117-86591msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145385: Tue Jul 28 03:38:16 2026 read: IOPS=2, BW=116MiB/s (121MB/s)(40.0MiB/346msec) slat (nsec): min=18325k, max=18325k, avg=18325211.00, stdev= 0.00 clat (nsec): min=303254k, max=303254k, avg=303253962.00, stdev= 0.00 lat (nsec): min=321584k, max=321584k, avg=321584383.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.77%, ctx=5, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=116MiB/s (121MB/s), 116MiB/s-116MiB/s (121MB/s-121MB/s), io=40.0MiB (41.9MB), run=346-346msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout