== sanity test 398c: run fio to test AIO ================= 00:27:41 (1762493261) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.396399 s, 106 MB/s osc.lustre-OST0000-osc-ffff90235089e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149184: Fri Nov 7 00:30:16 2025 write: IOPS=17, BW=69.2KiB/s (70.9kB/s)(10.0MiB/147902msec); 0 zone resets slat (usec): min=27, max=5719, avg=149.96, stdev=355.46 clat (msec): min=70, max=1744, avg=923.94, stdev=217.40 lat (msec): min=71, max=1745, avg=924.09, stdev=217.40 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 978], | 70.00th=[ 1028], 80.00th=[ 1083], 90.00th=[ 1183], 95.00th=[ 1250], | 99.00th=[ 1452], 99.50th=[ 1569], 99.90th=[ 1670], 99.95th=[ 1703], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 182, per=24.92%, avg=68.78, stdev=28.57, samples=293 iops : min= 1, max= 45, avg=16.75, stdev= 7.18, samples=293 lat (msec) : 100=0.04%, 250=0.47%, 500=3.79%, 750=14.30%, 1000=46.02% lat (msec) : 2000=35.39% cpu : usr=0.05%, sys=0.24%, ctx=1649, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149185: Fri Nov 7 00:30:16 2025 write: IOPS=17, BW=69.1KiB/s (70.7kB/s)(10.0MiB/148257msec); 0 zone resets slat (usec): min=27, max=5814, avg=151.81, stdev=393.04 clat (msec): min=232, max=1744, avg=925.45, stdev=212.74 lat (msec): min=232, max=1744, avg=925.61, stdev=212.76 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 936], 60.00th=[ 978], | 70.00th=[ 1028], 80.00th=[ 1083], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1502], 99.50th=[ 1620], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 230, per=25.54%, avg=70.50, stdev=29.43, samples=286 iops : min= 1, max= 57, avg=17.18, stdev= 7.39, samples=286 lat (msec) : 250=0.12%, 500=3.71%, 750=14.14%, 1000=46.76%, 2000=35.27% cpu : usr=0.04%, sys=0.23%, ctx=1563, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149186: Fri Nov 7 00:30:16 2025 write: IOPS=17, BW=69.1KiB/s (70.8kB/s)(10.0MiB/148179msec); 0 zone resets slat (usec): min=26, max=21006, avg=170.53, stdev=621.29 clat (msec): min=306, max=1732, avg=925.13, stdev=212.72 lat (msec): min=306, max=1732, avg=925.30, stdev=212.76 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 927], 60.00th=[ 978], | 70.00th=[ 1020], 80.00th=[ 1083], 90.00th=[ 1183], 95.00th=[ 1250], | 99.00th=[ 1485], 99.50th=[ 1620], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 255, per=25.27%, avg=69.75, stdev=30.46, samples=289 iops : min= 1, max= 63, avg=16.99, stdev= 7.63, samples=289 lat (msec) : 500=3.83%, 750=14.30%, 1000=47.23%, 2000=34.65% cpu : usr=0.05%, sys=0.23%, ctx=1578, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149187: Fri Nov 7 00:30:16 2025 write: IOPS=17, BW=69.0KiB/s (70.7kB/s)(10.0MiB/148316msec); 0 zone resets slat (usec): min=27, max=6071, avg=159.04, stdev=426.36 clat (msec): min=303, max=1709, avg=925.93, stdev=214.31 lat (msec): min=303, max=1709, avg=926.10, stdev=214.32 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 969], | 70.00th=[ 1020], 80.00th=[ 1083], 90.00th=[ 1183], 95.00th=[ 1267], | 99.00th=[ 1452], 99.50th=[ 1620], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 15, max= 166, per=24.76%, avg=68.34, stdev=29.04, samples=295 iops : min= 3, max= 41, avg=16.63, stdev= 7.26, samples=295 lat (msec) : 500=3.71%, 750=15.00%, 1000=47.07%, 2000=34.22% cpu : usr=0.04%, sys=0.24%, ctx=1544, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=276KiB/s (283kB/s), 69.0KiB/s-69.2KiB/s (70.7kB/s-70.9kB/s), io=40.0MiB (41.9MB), run=147902-148316msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149218: Fri Nov 7 00:32:06 2025 read: IOPS=12, BW=48.4KiB/s (49.6kB/s)(5048KiB/104223msec) slat (usec): min=2863, max=56016, avg=12300.94, stdev=6556.43 clat (msec): min=9, max=1223, avg=420.31, stdev=121.58 lat (msec): min=17, max=1237, avg=432.62, stdev=122.94 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 506], 90.00th=[ 567], 95.00th=[ 625], | 99.00th=[ 793], 99.50th=[ 852], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 135, per=25.14%, avg=48.01, stdev=20.67, samples=206 iops : min= 1, max= 33, avg=11.54, stdev= 5.20, samples=206 write: IOPS=12, BW=49.8KiB/s (51.0kB/s)(5192KiB/104223msec); 0 zone resets slat (usec): min=29, max=7708, avg=222.71, stdev=570.35 clat (msec): min=263, max=1687, avg=861.36, stdev=237.10 lat (msec): min=263, max=1687, avg=861.59, stdev=237.11 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 481], 10.00th=[ 550], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 802], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1284], | 99.00th=[ 1502], 99.50th=[ 1552], 99.90th=[ 1636], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 111, per=24.94%, avg=49.63, stdev=20.92, samples=205 iops : min= 1, max= 27, avg=11.94, stdev= 5.23, samples=205 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.23%, 250=1.29% lat (msec) : 500=40.08%, 750=23.09%, 1000=21.52%, 2000=13.59% cpu : usr=0.08%, sys=0.58%, ctx=4681, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149219: Fri Nov 7 00:32:06 2025 read: IOPS=11, BW=47.6KiB/s (48.7kB/s)(4976KiB/104580msec) slat (usec): min=2735, max=61067, avg=12035.68, stdev=6172.92 clat (msec): min=8, max=874, avg=415.63, stdev=109.20 lat (msec): min=16, max=885, avg=427.67, stdev=110.45 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 268], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 498], 90.00th=[ 558], 95.00th=[ 600], | 99.00th=[ 735], 99.50th=[ 768], 99.90th=[ 852], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 143, per=24.75%, avg=47.27, stdev=23.45, samples=205 iops : min= 1, max= 35, avg=11.36, stdev= 5.87, samples=205 write: IOPS=12, BW=50.3KiB/s (51.5kB/s)(5264KiB/104580msec); 0 zone resets slat (usec): min=29, max=9136, avg=204.54, stdev=534.33 clat (msec): min=187, max=1625, avg=865.22, stdev=242.01 lat (msec): min=188, max=1625, avg=865.42, stdev=241.99 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 464], 10.00th=[ 558], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 802], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1036], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1536], 99.50th=[ 1552], 99.90th=[ 1603], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 104, per=24.91%, avg=49.57, stdev=20.64, samples=207 iops : min= 1, max= 26, avg=11.93, stdev= 5.21, samples=207 lat (msec) : 10=0.08%, 50=0.12%, 100=0.08%, 250=1.64%, 500=40.16% lat (msec) : 750=22.19%, 1000=21.99%, 2000=13.75% cpu : usr=0.08%, sys=0.57%, ctx=4246, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149220: Fri Nov 7 00:32:06 2025 read: IOPS=11, BW=47.8KiB/s (48.0kB/s)(4984KiB/104227msec) slat (usec): min=35, max=60225, avg=12051.32, stdev=6847.32 clat (msec): min=15, max=978, avg=413.92, stdev=115.68 lat (msec): min=25, max=992, avg=425.98, stdev=117.37 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 255], 10.00th=[ 288], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 439], | 70.00th=[ 460], 80.00th=[ 502], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 735], 99.50th=[ 818], 99.90th=[ 927], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 136, per=24.87%, avg=47.50, stdev=22.49, samples=206 iops : min= 1, max= 34, avg=11.42, stdev= 5.65, samples=206 write: IOPS=12, BW=50.4KiB/s (51.6kB/s)(5256KiB/104227msec); 0 zone resets slat (usec): min=28, max=15390, avg=240.37, stdev=754.39 clat (msec): min=145, max=1680, avg=864.08, stdev=243.76 lat (msec): min=145, max=1680, avg=864.32, stdev=243.76 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 481], 10.00th=[ 567], 20.00th=[ 667], | 30.00th=[ 743], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1536], 99.50th=[ 1569], 99.90th=[ 1586], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 132, per=25.01%, avg=49.76, stdev=20.19, samples=206 iops : min= 1, max= 33, avg=11.98, stdev= 5.06, samples=206 lat (msec) : 20=0.04%, 50=0.20%, 100=0.39%, 250=1.84%, 500=39.41% lat (msec) : 750=22.03%, 1000=22.93%, 2000=13.16% cpu : usr=0.09%, sys=0.57%, ctx=4620, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149221: Fri Nov 7 00:32:06 2025 read: IOPS=12, BW=48.5KiB/s (49.6kB/s)(5060KiB/104418msec) slat (usec): min=2345, max=57586, avg=11917.25, stdev=6343.70 clat (msec): min=47, max=872, avg=420.01, stdev=110.51 lat (msec): min=58, max=892, avg=431.94, stdev=111.91 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 268], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 439], | 70.00th=[ 464], 80.00th=[ 510], 90.00th=[ 567], 95.00th=[ 609], | 99.00th=[ 743], 99.50th=[ 768], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 7, max= 120, per=25.26%, avg=48.25, stdev=22.20, samples=206 iops : min= 1, max= 30, avg=11.61, stdev= 5.58, samples=206 write: IOPS=12, BW=49.6KiB/s (50.8kB/s)(5180KiB/104418msec); 0 zone resets slat (usec): min=30, max=13064, avg=218.79, stdev=641.00 clat (msec): min=113, max=1672, avg=865.95, stdev=235.72 lat (msec): min=113, max=1672, avg=866.17, stdev=235.72 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 493], 10.00th=[ 558], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 969], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1485], 99.50th=[ 1519], 99.90th=[ 1603], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 96, per=24.81%, avg=49.38, stdev=19.89, samples=205 iops : min= 1, max= 24, avg=11.88, stdev= 4.97, samples=205 lat (msec) : 50=0.08%, 100=0.16%, 250=1.64%, 500=39.38%, 750=23.67% lat (msec) : 1000=21.64%, 2000=13.44% cpu : usr=0.08%, sys=0.60%, ctx=4271, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=192KiB/s (196kB/s), 47.6KiB/s-48.5KiB/s (48.7kB/s-49.6kB/s), io=19.6MiB (20.5MB), run=104223-104580msec WRITE: bw=200KiB/s (205kB/s), 49.6KiB/s-50.4KiB/s (50.8kB/s-51.6kB/s), io=20.4MiB (21.4MB), run=104223-104580msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149241: Fri Nov 7 00:32:18 2025 read: IOPS=1, BW=78.3MiB/s (82.1MB/s)(40.0MiB/511msec) slat (nsec): min=66776k, max=66776k, avg=66775885.00, stdev= 0.00 clat (nsec): min=414521k, max=414521k, avg=414520516.00, stdev= 0.00 lat (nsec): min=481302k, max=481302k, avg=481302347.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] lat (msec) : 500=100.00% cpu : usr=1.37%, sys=5.29%, ctx=33, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=78.3MiB/s (82.1MB/s), 78.3MiB/s-78.3MiB/s (82.1MB/s-82.1MB/s), io=40.0MiB (41.9MB), run=511-511msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout