== sanity test 398c: run fio to test AIO ================= 17:00:44 (1787864444) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.401312 s, 105 MB/s osc.lustre-OST0000-osc-ffff9e9248d4b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146552: Thu Aug 27 17:03:21 2026 write: IOPS=17, BW=68.6KiB/s (70.2kB/s)(10.0MiB/149300msec); 0 zone resets slat (usec): min=28, max=13718, avg=214.49, stdev=768.07 clat (msec): min=116, max=1954, avg=932.42, stdev=241.90 lat (msec): min=117, max=1954, avg=932.64, stdev=241.90 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 584], 10.00th=[ 659], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1053], 80.00th=[ 1116], 90.00th=[ 1217], 95.00th=[ 1318], | 99.00th=[ 1586], 99.50th=[ 1754], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 219, per=25.36%, avg=69.49, stdev=32.01, samples=290 iops : min= 1, max= 54, avg=16.95, stdev= 8.04, samples=290 lat (msec) : 250=0.62%, 500=3.16%, 750=18.20%, 1000=40.12%, 2000=37.89% cpu : usr=0.04%, sys=0.24%, ctx=1702, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146553: Thu Aug 27 17:03:21 2026 write: IOPS=17, BW=68.6KiB/s (70.3kB/s)(10.0MiB/149206msec); 0 zone resets slat (usec): min=29, max=22076, avg=244.80, stdev=982.57 clat (msec): min=129, max=1928, avg=931.40, stdev=244.17 lat (msec): min=129, max=1929, avg=931.64, stdev=244.17 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 584], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1045], 80.00th=[ 1116], 90.00th=[ 1217], 95.00th=[ 1318], | 99.00th=[ 1586], 99.50th=[ 1737], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 220, per=25.10%, avg=68.77, stdev=30.33, samples=293 iops : min= 1, max= 55, avg=16.77, stdev= 7.62, samples=293 lat (msec) : 250=0.62%, 500=3.12%, 750=17.54%, 1000=41.64%, 2000=37.07% cpu : usr=0.05%, sys=0.21%, ctx=1685, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146554: Thu Aug 27 17:03:21 2026 write: IOPS=17, BW=68.6KiB/s (70.2kB/s)(10.0MiB/149301msec); 0 zone resets slat (usec): min=28, max=16153, avg=224.11, stdev=829.29 clat (msec): min=51, max=1879, avg=932.01, stdev=237.85 lat (msec): min=52, max=1879, avg=932.23, stdev=237.81 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 911], 60.00th=[ 969], | 70.00th=[ 1036], 80.00th=[ 1116], 90.00th=[ 1217], 95.00th=[ 1334], | 99.00th=[ 1569], 99.50th=[ 1636], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 216, per=24.95%, avg=68.37, stdev=32.76, samples=295 iops : min= 1, max= 54, avg=16.67, stdev= 8.21, samples=295 lat (msec) : 100=0.08%, 250=0.62%, 500=1.99%, 750=17.42%, 1000=43.79% lat (msec) : 2000=36.09% cpu : usr=0.04%, sys=0.25%, ctx=2020, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146555: Thu Aug 27 17:03:21 2026 write: IOPS=17, BW=68.6KiB/s (70.2kB/s)(10.0MiB/149300msec); 0 zone resets slat (usec): min=29, max=23579, avg=202.22, stdev=828.54 clat (msec): min=196, max=1880, avg=931.95, stdev=236.20 lat (msec): min=196, max=1880, avg=932.16, stdev=236.21 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1045], 80.00th=[ 1116], 90.00th=[ 1217], 95.00th=[ 1351], | 99.00th=[ 1552], 99.50th=[ 1586], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1888] bw ( KiB/s): min= 8, max= 256, per=24.78%, avg=67.91, stdev=30.96, samples=297 iops : min= 2, max= 64, avg=16.55, stdev= 7.80, samples=297 lat (msec) : 250=0.62%, 500=2.03%, 750=17.97%, 1000=42.15%, 2000=37.23% cpu : usr=0.05%, sys=0.21%, ctx=1750, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=274KiB/s (281kB/s), 68.6KiB/s-68.6KiB/s (70.2kB/s-70.3kB/s), io=40.0MiB (41.9MB), run=149206-149301msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146598: Thu Aug 27 17:05:18 2026 read: IOPS=11, BW=46.9KiB/s (47.0kB/s)(5048KiB/107718msec) slat (msec): min=2, max=128, avg=14.40, stdev= 8.92 clat (msec): min=22, max=1153, avg=421.10, stdev=122.93 lat (msec): min=30, max=1180, avg=435.50, stdev=124.66 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 271], 10.00th=[ 292], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 401], 60.00th=[ 430], | 70.00th=[ 464], 80.00th=[ 514], 90.00th=[ 584], 95.00th=[ 659], | 99.00th=[ 776], 99.50th=[ 818], 99.90th=[ 1028], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 127, per=25.14%, avg=46.51, stdev=21.32, samples=213 iops : min= 1, max= 31, avg=11.14, stdev= 5.38, samples=213 write: IOPS=12, BW=48.2KiB/s (49.4kB/s)(5192KiB/107718msec); 0 zone resets slat (usec): min=32, max=18532, avg=313.87, stdev=1022.11 clat (msec): min=364, max=1639, avg=901.81, stdev=229.49 lat (msec): min=364, max=1639, avg=902.12, stdev=229.47 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 518], 10.00th=[ 584], 20.00th=[ 693], | 30.00th=[ 785], 40.00th=[ 844], 50.00th=[ 902], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1183], 95.00th=[ 1284], | 99.00th=[ 1469], 99.50th=[ 1552], 99.90th=[ 1586], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 111, per=24.82%, avg=47.90, stdev=20.80, samples=212 iops : min= 1, max= 27, avg=11.48, stdev= 5.25, samples=212 lat (msec) : 50=0.08%, 100=0.08%, 250=1.13%, 500=38.63%, 750=21.84% lat (msec) : 1000=20.66%, 2000=17.58% cpu : usr=0.08%, sys=0.62%, ctx=4530, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146599: Thu Aug 27 17:05:18 2026 read: IOPS=11, BW=46.2KiB/s (47.3kB/s)(4976KiB/107820msec) slat (usec): min=2909, max=67898, avg=14309.09, stdev=8352.79 clat (msec): min=19, max=1112, avg=422.86, stdev=125.67 lat (msec): min=34, max=1120, avg=437.17, stdev=127.32 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 257], 10.00th=[ 284], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 468], 80.00th=[ 518], 90.00th=[ 592], 95.00th=[ 667], | 99.00th=[ 776], 99.50th=[ 818], 99.90th=[ 1036], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 112, per=24.69%, avg=45.69, stdev=21.35, samples=213 iops : min= 1, max= 28, avg=10.92, stdev= 5.41, samples=213 write: IOPS=12, BW=48.8KiB/s (49.0kB/s)(5264KiB/107820msec); 0 zone resets slat (usec): min=32, max=14766, avg=288.56, stdev=947.01 clat (msec): min=216, max=1677, avg=895.54, stdev=234.81 lat (msec): min=216, max=1677, avg=895.83, stdev=234.82 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 506], 10.00th=[ 567], 20.00th=[ 667], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 902], 60.00th=[ 953], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1385], 99.50th=[ 1452], 99.90th=[ 1586], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 128, per=25.00%, avg=48.26, stdev=20.97, samples=214 iops : min= 1, max= 32, avg=11.57, stdev= 5.28, samples=214 lat (msec) : 20=0.04%, 50=0.04%, 100=0.08%, 250=1.76%, 500=37.81% lat (msec) : 750=22.42%, 1000=20.16%, 2000=17.70% cpu : usr=0.09%, sys=0.59%, ctx=4721, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146600: Thu Aug 27 17:05:18 2026 read: IOPS=11, BW=46.1KiB/s (47.3kB/s)(4984KiB/108002msec) slat (msec): min=2, max=433, avg=15.18, stdev=20.06 clat (msec): min=20, max=1158, avg=421.26, stdev=130.20 lat (msec): min=24, max=1179, avg=436.44, stdev=131.31 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 264], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 430], | 70.00th=[ 464], 80.00th=[ 510], 90.00th=[ 592], 95.00th=[ 667], | 99.00th=[ 785], 99.50th=[ 978], 99.90th=[ 1150], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 111, per=25.43%, avg=47.04, stdev=22.39, samples=206 iops : min= 1, max= 27, avg=11.27, stdev= 5.63, samples=206 write: IOPS=12, BW=48.7KiB/s (49.8kB/s)(5256KiB/108002msec); 0 zone resets slat (usec): min=34, max=14259, avg=278.68, stdev=790.98 clat (msec): min=94, max=1629, avg=898.93, stdev=250.64 lat (msec): min=95, max=1629, avg=899.21, stdev=250.62 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 498], 10.00th=[ 567], 20.00th=[ 684], | 30.00th=[ 793], 40.00th=[ 844], 50.00th=[ 902], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1116], 90.00th=[ 1217], 95.00th=[ 1301], | 99.00th=[ 1452], 99.50th=[ 1552], 99.90th=[ 1603], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 128, per=24.60%, avg=47.48, stdev=22.05, samples=214 iops : min= 1, max= 32, avg=11.38, stdev= 5.58, samples=214 lat (msec) : 50=0.04%, 100=0.16%, 250=2.07%, 500=38.48%, 750=20.23% lat (msec) : 1000=21.09%, 2000=17.93% cpu : usr=0.08%, sys=0.61%, ctx=4677, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146601: Thu Aug 27 17:05:18 2026 read: IOPS=11, BW=47.2KiB/s (48.3kB/s)(5060KiB/107263msec) slat (usec): min=37, max=72343, avg=14049.84, stdev=8613.30 clat (msec): min=22, max=1099, avg=419.36, stdev=122.90 lat (msec): min=37, max=1112, avg=433.42, stdev=124.49 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 259], 10.00th=[ 288], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 430], | 70.00th=[ 468], 80.00th=[ 510], 90.00th=[ 575], 95.00th=[ 634], | 99.00th=[ 768], 99.50th=[ 869], 99.90th=[ 1070], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 120, per=25.33%, avg=46.86, stdev=22.20, samples=212 iops : min= 1, max= 30, avg=11.22, stdev= 5.61, samples=212 write: IOPS=12, BW=48.3KiB/s (49.5kB/s)(5180KiB/107263msec); 0 zone resets slat (usec): min=32, max=10036, avg=269.90, stdev=687.97 clat (msec): min=162, max=1641, avg=901.08, stdev=234.17 lat (msec): min=164, max=1641, avg=901.36, stdev=234.17 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 502], 10.00th=[ 584], 20.00th=[ 701], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1183], 95.00th=[ 1284], | 99.00th=[ 1469], 99.50th=[ 1519], 99.90th=[ 1603], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 104, per=24.62%, avg=47.51, stdev=19.98, samples=213 iops : min= 1, max= 26, avg=11.38, stdev= 5.04, samples=213 lat (msec) : 50=0.04%, 100=0.31%, 250=1.68%, 500=38.67%, 750=20.62% lat (msec) : 1000=21.52%, 2000=17.15% cpu : usr=0.10%, sys=0.55%, ctx=4587, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=186KiB/s (190kB/s), 46.1KiB/s-47.2KiB/s (47.3kB/s-48.3kB/s), io=19.6MiB (20.5MB), run=107263-108002msec WRITE: bw=193KiB/s (198kB/s), 48.2KiB/s-48.8KiB/s (49.4kB/s-49.0kB/s), io=20.4MiB (21.4MB), run=107263-108002msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146620: Thu Aug 27 17:05:25 2026 read: IOPS=2, BW=109MiB/s (114MB/s)(40.0MiB/368msec) slat (nsec): min=33674k, max=33674k, avg=33673863.00, stdev= 0.00 clat (nsec): min=305526k, max=305526k, avg=305526364.00, stdev= 0.00 lat (nsec): min=339206k, max=339206k, avg=339206409.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.81%, ctx=35, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=40.0MiB (41.9MB), run=368-368msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout