== sanity test 398c: run fio to test AIO ================= 17:56:00 (1778536560) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.249787 s, 168 MB/s osc.lustre-OST0000-osc-ffff99a64a22a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=124391: Mon May 11 17:58:18 2026 write: IOPS=19, BW=78.3KiB/s (80.2kB/s)(10.0MiB/130783msec); 0 zone resets slat (usec): min=25, max=21941, avg=284.85, stdev=1081.53 clat (msec): min=69, max=1626, avg=816.70, stdev=192.65 lat (msec): min=69, max=1626, avg=816.99, stdev=192.65 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 523], 10.00th=[ 609], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 785], 50.00th=[ 827], 60.00th=[ 860], | 70.00th=[ 902], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1116], | 99.00th=[ 1318], 99.50th=[ 1401], 99.90th=[ 1552], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 15, max= 247, per=24.95%, avg=77.85, stdev=30.69, samples=259 iops : min= 3, max= 61, avg=19.01, stdev= 7.69, samples=259 lat (msec) : 100=0.12%, 250=0.90%, 500=3.52%, 750=27.85%, 1000=54.02% lat (msec) : 2000=13.59% cpu : usr=0.05%, sys=0.31%, ctx=1760, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=124392: Mon May 11 17:58:18 2026 write: IOPS=19, BW=78.2KiB/s (80.0kB/s)(10.0MiB/131028msec); 0 zone resets slat (usec): min=21, max=13760, avg=288.97, stdev=933.38 clat (msec): min=153, max=1488, avg=818.04, stdev=190.24 lat (msec): min=153, max=1498, avg=818.33, stdev=190.28 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 776], 50.00th=[ 827], 60.00th=[ 869], | 70.00th=[ 911], 80.00th=[ 961], 90.00th=[ 1036], 95.00th=[ 1116], | 99.00th=[ 1284], 99.50th=[ 1368], 99.90th=[ 1469], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 256, per=24.96%, avg=77.86, stdev=29.18, samples=259 iops : min= 1, max= 64, avg=19.02, stdev= 7.32, samples=259 lat (msec) : 250=1.25%, 500=2.93%, 750=29.22%, 1000=53.44%, 2000=13.16% cpu : usr=0.07%, sys=0.27%, ctx=1776, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=124393: Mon May 11 17:58:18 2026 write: IOPS=19, BW=78.1KiB/s (80.0kB/s)(10.0MiB/131030msec); 0 zone resets slat (usec): min=20, max=10824, avg=270.06, stdev=813.75 clat (msec): min=164, max=1662, avg=818.08, stdev=184.46 lat (msec): min=165, max=1662, avg=818.35, stdev=184.50 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 693], | 30.00th=[ 743], 40.00th=[ 776], 50.00th=[ 818], 60.00th=[ 860], | 70.00th=[ 902], 80.00th=[ 953], 90.00th=[ 1028], 95.00th=[ 1099], | 99.00th=[ 1301], 99.50th=[ 1368], 99.90th=[ 1552], 99.95th=[ 1569], | 99.99th=[ 1670] bw ( KiB/s): min= 15, max= 226, per=24.95%, avg=77.85, stdev=30.69, samples=259 iops : min= 3, max= 56, avg=19.02, stdev= 7.69, samples=259 lat (msec) : 250=0.70%, 500=3.28%, 750=28.24%, 1000=55.55%, 2000=12.23% cpu : usr=0.06%, sys=0.31%, ctx=1754, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=124394: Mon May 11 17:58:18 2026 write: IOPS=19, BW=78.2KiB/s (80.1kB/s)(10.0MiB/130920msec); 0 zone resets slat (usec): min=22, max=12231, avg=265.80, stdev=828.40 clat (msec): min=158, max=1571, avg=817.44, stdev=185.81 lat (msec): min=158, max=1572, avg=817.70, stdev=185.81 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 531], 10.00th=[ 609], 20.00th=[ 684], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 827], 60.00th=[ 860], | 70.00th=[ 902], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1083], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1502], 99.95th=[ 1552], | 99.99th=[ 1569] bw ( KiB/s): min= 8, max= 227, per=24.86%, avg=77.57, stdev=26.93, samples=260 iops : min= 2, max= 56, avg=18.94, stdev= 6.75, samples=260 lat (msec) : 250=0.86%, 500=3.55%, 750=28.52%, 1000=53.52%, 2000=13.55% cpu : usr=0.05%, sys=0.31%, ctx=2002, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=313KiB/s (320kB/s), 78.1KiB/s-78.3KiB/s (80.0kB/s-80.2kB/s), io=40.0MiB (41.9MB), run=130783-131030msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=124423: Mon May 11 18:00:08 2026 read: IOPS=12, BW=48.6KiB/s (49.7kB/s)(5048KiB/103903msec) slat (usec): min=2379, max=52855, avg=11692.35, stdev=6801.30 clat (msec): min=13, max=929, avg=409.59, stdev=102.92 lat (msec): min=17, max=943, avg=421.29, stdev=103.80 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 481], 90.00th=[ 535], 95.00th=[ 600], | 99.00th=[ 735], 99.50th=[ 818], 99.90th=[ 911], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 7, max= 120, per=25.20%, avg=48.39, stdev=21.56, samples=204 iops : min= 1, max= 30, avg=11.59, stdev= 5.40, samples=204 write: IOPS=12, BW=49.0KiB/s (51.2kB/s)(5192KiB/103903msec); 0 zone resets slat (usec): min=25, max=9137, avg=277.74, stdev=791.72 clat (msec): min=392, max=1777, avg=868.62, stdev=232.07 lat (msec): min=392, max=1777, avg=868.90, stdev=232.10 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 531], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1569], 99.50th=[ 1687], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 6, max= 104, per=24.72%, avg=49.43, stdev=19.45, samples=206 iops : min= 1, max= 26, avg=11.85, stdev= 4.90, samples=206 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=1.41%, 500=41.02% lat (msec) : 750=23.16%, 1000=20.62%, 2000=13.52% cpu : usr=0.11%, sys=0.70%, ctx=3442, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=124424: Mon May 11 18:00:08 2026 read: IOPS=11, BW=47.8KiB/s (48.0kB/s)(4976KiB/104013msec) slat (usec): min=36, max=53657, avg=11390.87, stdev=6845.46 clat (msec): min=28, max=940, avg=411.50, stdev=110.63 lat (msec): min=29, max=958, avg=422.90, stdev=111.39 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 401], 60.00th=[ 422], | 70.00th=[ 451], 80.00th=[ 489], 90.00th=[ 542], 95.00th=[ 600], | 99.00th=[ 760], 99.50th=[ 860], 99.90th=[ 936], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 128, per=24.82%, avg=47.65, stdev=23.46, samples=204 iops : min= 1, max= 32, avg=11.42, stdev= 5.88, samples=204 write: IOPS=12, BW=50.6KiB/s (51.8kB/s)(5264KiB/104013msec); 0 zone resets slat (usec): min=29, max=12889, avg=318.97, stdev=1033.35 clat (msec): min=137, max=1831, avg=863.84, stdev=236.71 lat (msec): min=137, max=1832, avg=864.16, stdev=236.74 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 514], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 986], 80.00th=[ 1045], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1603], 99.50th=[ 1720], 99.90th=[ 1787], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 111, per=25.17%, avg=50.35, stdev=21.56, samples=205 iops : min= 1, max= 27, avg=12.08, stdev= 5.43, samples=205 lat (msec) : 50=0.16%, 100=0.31%, 250=1.64%, 500=39.84%, 750=23.44% lat (msec) : 1000=20.47%, 2000=14.14% cpu : usr=0.12%, sys=0.64%, ctx=3272, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=124425: Mon May 11 18:00:08 2026 read: IOPS=11, BW=47.9KiB/s (49.0kB/s)(4984KiB/104074msec) slat (usec): min=2079, max=53397, avg=11692.63, stdev=7178.00 clat (msec): min=7, max=906, avg=413.48, stdev=108.69 lat (msec): min=17, max=925, avg=425.18, stdev=109.80 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 380], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 447], 80.00th=[ 481], 90.00th=[ 550], 95.00th=[ 600], | 99.00th=[ 785], 99.50th=[ 860], 99.90th=[ 894], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 7, max= 120, per=24.74%, avg=47.50, stdev=23.20, samples=206 iops : min= 1, max= 30, avg=11.37, stdev= 5.84, samples=206 write: IOPS=12, BW=50.5KiB/s (51.7kB/s)(5256KiB/104074msec); 0 zone resets slat (usec): min=28, max=12543, avg=329.65, stdev=1061.36 clat (msec): min=106, max=1741, avg=861.65, stdev=235.32 lat (msec): min=106, max=1749, avg=861.98, stdev=235.33 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 518], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1045], 90.00th=[ 1150], 95.00th=[ 1267], | 99.00th=[ 1552], 99.50th=[ 1636], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 119, per=24.79%, avg=49.59, stdev=20.13, samples=207 iops : min= 1, max= 29, avg=11.89, stdev= 5.04, samples=207 lat (msec) : 10=0.04%, 50=0.04%, 100=0.08%, 250=1.76%, 500=40.43% lat (msec) : 750=22.62%, 1000=21.13%, 2000=13.91% cpu : usr=0.10%, sys=0.66%, ctx=3385, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=124426: Mon May 11 18:00:08 2026 read: IOPS=12, BW=48.9KiB/s (50.0kB/s)(5060KiB/103555msec) slat (usec): min=2201, max=52594, avg=11254.50, stdev=6767.03 clat (msec): min=31, max=961, avg=410.85, stdev=104.68 lat (msec): min=40, max=974, avg=422.11, stdev=105.32 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 418], | 70.00th=[ 443], 80.00th=[ 481], 90.00th=[ 542], 95.00th=[ 584], | 99.00th=[ 751], 99.50th=[ 869], 99.90th=[ 953], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 119, per=25.62%, avg=49.19, stdev=19.85, samples=202 iops : min= 1, max= 29, avg=11.80, stdev= 4.97, samples=202 write: IOPS=12, BW=50.0KiB/s (51.2kB/s)(5180KiB/103555msec); 0 zone resets slat (usec): min=25, max=18370, avg=312.53, stdev=1078.45 clat (msec): min=349, max=1806, avg=864.94, stdev=235.61 lat (msec): min=349, max=1806, avg=865.25, stdev=235.56 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 523], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1053], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1569], 99.50th=[ 1670], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 104, per=24.68%, avg=49.35, stdev=18.75, samples=205 iops : min= 1, max= 26, avg=11.83, stdev= 4.74, samples=205 lat (msec) : 50=0.04%, 100=0.20%, 250=1.21%, 500=42.03%, 750=22.50% lat (msec) : 1000=20.47%, 2000=13.55% cpu : usr=0.11%, sys=0.64%, ctx=3186, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=193KiB/s (197kB/s), 47.8KiB/s-48.9KiB/s (48.0kB/s-50.0kB/s), io=19.6MiB (20.5MB), run=103555-104074msec WRITE: bw=201KiB/s (206kB/s), 49.0KiB/s-50.6KiB/s (51.2kB/s-51.8kB/s), io=20.4MiB (21.4MB), run=103555-104074msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=124445: Mon May 11 18:00:18 2026 read: IOPS=2, BW=83.7MiB/s (87.7MB/s)(40.0MiB/478msec) slat (nsec): min=47746k, max=47746k, avg=47745997.00, stdev= 0.00 clat (nsec): min=400927k, max=400927k, avg=400926568.00, stdev= 0.00 lat (nsec): min=448677k, max=448677k, avg=448677467.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.35%, ctx=3, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.7MiB/s (87.7MB/s), 83.7MiB/s-83.7MiB/s (87.7MB/s-87.7MB/s), io=40.0MiB (41.9MB), run=478-478msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck