== sanity test 398c: run fio to test AIO ================= 00:17:20 (1773202640) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.789019 s, 53.2 MB/s osc.lustre-OST0000-osc-ffff9d57ad598800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148288: Wed Mar 11 00:20:27 2026 write: IOPS=14, BW=58.8KiB/s (60.3kB/s)(10.0MiB/174000msec); 0 zone resets slat (usec): min=35, max=11515, avg=202.70, stdev=581.01 clat (msec): min=145, max=2684, avg=1086.69, stdev=369.80 lat (msec): min=145, max=2684, avg=1086.89, stdev=369.81 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 542], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1116], | 70.00th=[ 1234], 80.00th=[ 1401], 90.00th=[ 1603], 95.00th=[ 1754], | 99.00th=[ 2198], 99.50th=[ 2265], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 136, per=25.79%, avg=60.62, stdev=32.54, samples=332 iops : min= 1, max= 34, avg=14.67, stdev= 8.18, samples=332 lat (msec) : 250=0.08%, 500=3.83%, 750=10.90%, 1000=33.05%, 2000=50.20% lat (msec) : >=2000=1.95% cpu : usr=0.04%, sys=0.21%, ctx=1660, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148289: Wed Mar 11 00:20:27 2026 write: IOPS=14, BW=58.8KiB/s (60.2kB/s)(10.0MiB/174269msec); 0 zone resets slat (usec): min=38, max=21964, avg=219.75, stdev=853.44 clat (msec): min=223, max=2684, avg=1088.08, stdev=374.00 lat (msec): min=223, max=2684, avg=1088.30, stdev=374.04 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 567], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1234], 80.00th=[ 1401], 90.00th=[ 1603], 95.00th=[ 1754], | 99.00th=[ 2198], 99.50th=[ 2467], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 240, per=24.97%, avg=58.69, stdev=32.45, samples=343 iops : min= 1, max= 60, avg=14.19, stdev= 8.17, samples=343 lat (msec) : 250=0.08%, 500=2.46%, 750=12.89%, 1000=31.60%, 2000=51.09% lat (msec) : >=2000=1.88% cpu : usr=0.04%, sys=0.22%, ctx=1735, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148290: Wed Mar 11 00:20:27 2026 write: IOPS=14, BW=58.8KiB/s (60.2kB/s)(10.0MiB/174271msec); 0 zone resets slat (usec): min=35, max=25947, avg=225.55, stdev=820.68 clat (msec): min=272, max=2560, avg=1088.08, stdev=374.51 lat (msec): min=272, max=2560, avg=1088.30, stdev=374.51 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1217], 80.00th=[ 1418], 90.00th=[ 1620], 95.00th=[ 1787], | 99.00th=[ 2198], 99.50th=[ 2333], 99.90th=[ 2467], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 256, per=26.05%, avg=61.22, stdev=33.33, samples=329 iops : min= 1, max= 64, avg=14.83, stdev= 8.37, samples=329 lat (msec) : 500=2.81%, 750=12.66%, 1000=32.77%, 2000=49.77%, >=2000=1.99% cpu : usr=0.04%, sys=0.23%, ctx=1883, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148291: Wed Mar 11 00:20:27 2026 write: IOPS=14, BW=58.8KiB/s (60.2kB/s)(10.0MiB/174270msec); 0 zone resets slat (usec): min=35, max=19148, avg=204.09, stdev=689.23 clat (msec): min=153, max=2304, avg=1088.68, stdev=370.97 lat (msec): min=153, max=2304, avg=1088.89, stdev=371.02 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1234], 80.00th=[ 1418], 90.00th=[ 1603], 95.00th=[ 1770], | 99.00th=[ 2165], 99.50th=[ 2232], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 175, per=27.12%, avg=63.74, stdev=33.15, samples=316 iops : min= 1, max= 43, avg=15.46, stdev= 8.31, samples=316 lat (msec) : 250=0.62%, 500=2.15%, 750=11.72%, 1000=34.30%, 2000=48.67% lat (msec) : >=2000=2.54% cpu : usr=0.03%, sys=0.21%, ctx=1633, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=235KiB/s (241kB/s), 58.8KiB/s-58.8KiB/s (60.2kB/s-60.3kB/s), io=40.0MiB (41.9MB), run=174000-174271msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148327: Wed Mar 11 00:22:26 2026 read: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5048KiB/110253msec) slat (usec): min=3283, max=60713, avg=15297.98, stdev=8477.78 clat (msec): min=18, max=902, avg=431.02, stdev=115.46 lat (msec): min=25, max=929, avg=446.32, stdev=116.89 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 472], 80.00th=[ 510], 90.00th=[ 584], 95.00th=[ 651], | 99.00th=[ 768], 99.50th=[ 827], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 111, per=24.78%, avg=45.10, stdev=21.96, samples=219 iops : min= 1, max= 27, avg=10.79, stdev= 5.53, samples=219 write: IOPS=11, BW=47.1KiB/s (48.2kB/s)(5192KiB/110253msec); 0 zone resets slat (usec): min=44, max=15773, avg=307.67, stdev=922.39 clat (msec): min=87, max=1793, avg=921.01, stdev=250.07 lat (msec): min=87, max=1795, avg=921.32, stdev=250.09 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 535], 10.00th=[ 634], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 844], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 1020], 80.00th=[ 1099], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1620], 99.50th=[ 1703], 99.90th=[ 1770], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 110, per=24.56%, avg=46.43, stdev=19.21, samples=218 iops : min= 1, max= 27, avg=11.13, stdev= 4.81, samples=218 lat (msec) : 20=0.08%, 50=0.04%, 100=0.16%, 250=1.25%, 500=38.55% lat (msec) : 750=19.80%, 1000=23.87%, 2000=16.25% cpu : usr=0.10%, sys=0.67%, ctx=4779, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148328: Wed Mar 11 00:22:26 2026 read: IOPS=11, BW=45.1KiB/s (46.2kB/s)(4976KiB/110221msec) slat (usec): min=2983, max=77477, avg=15280.65, stdev=8652.85 clat (msec): min=11, max=926, avg=431.68, stdev=118.06 lat (msec): min=20, max=937, avg=446.97, stdev=119.76 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 477], 80.00th=[ 514], 90.00th=[ 584], 95.00th=[ 659], | 99.00th=[ 776], 99.50th=[ 810], 99.90th=[ 827], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 7, max= 136, per=24.61%, avg=44.78, stdev=20.30, samples=218 iops : min= 1, max= 34, avg=10.72, stdev= 5.11, samples=218 write: IOPS=11, BW=47.8KiB/s (48.9kB/s)(5264KiB/110221msec); 0 zone resets slat (usec): min=56, max=16220, avg=314.38, stdev=938.54 clat (msec): min=266, max=1798, avg=914.62, stdev=235.72 lat (msec): min=266, max=1798, avg=914.93, stdev=235.73 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 550], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 793], 40.00th=[ 844], 50.00th=[ 885], 60.00th=[ 944], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1368], | 99.00th=[ 1519], 99.50th=[ 1620], 99.90th=[ 1720], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 119, per=24.87%, avg=47.00, stdev=20.61, samples=219 iops : min= 1, max= 29, avg=11.27, stdev= 5.11, samples=219 lat (msec) : 20=0.08%, 50=0.16%, 100=0.27%, 250=0.86%, 500=37.50% lat (msec) : 750=20.35%, 1000=24.61%, 2000=16.17% cpu : usr=0.10%, sys=0.63%, ctx=4251, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148329: Wed Mar 11 00:22:26 2026 read: IOPS=11, BW=45.3KiB/s (46.4kB/s)(4984KiB/110046msec) slat (usec): min=92, max=67179, avg=15350.73, stdev=8946.95 clat (msec): min=21, max=910, avg=434.95, stdev=116.42 lat (msec): min=21, max=916, avg=450.30, stdev=118.05 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 518], 90.00th=[ 584], 95.00th=[ 659], | 99.00th=[ 776], 99.50th=[ 802], 99.90th=[ 885], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 7, max= 111, per=24.83%, avg=45.19, stdev=20.10, samples=217 iops : min= 1, max= 27, avg=10.82, stdev= 5.03, samples=217 write: IOPS=11, BW=47.8KiB/s (48.9kB/s)(5256KiB/110046msec); 0 zone resets slat (usec): min=40, max=19552, avg=328.59, stdev=1007.80 clat (msec): min=181, max=1820, avg=911.29, stdev=248.14 lat (msec): min=181, max=1820, avg=911.63, stdev=248.12 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 531], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1620], 99.50th=[ 1703], 99.90th=[ 1787], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 127, per=24.87%, avg=47.00, stdev=20.60, samples=218 iops : min= 1, max= 31, avg=11.27, stdev= 5.17, samples=218 lat (msec) : 50=0.23%, 100=0.16%, 250=0.70%, 500=37.93%, 750=20.86% lat (msec) : 1000=24.57%, 2000=15.55% cpu : usr=0.10%, sys=0.70%, ctx=4500, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148330: Wed Mar 11 00:22:26 2026 read: IOPS=11, BW=45.0KiB/s (47.1kB/s)(5060KiB/110110msec) slat (usec): min=2727, max=53148, avg=15123.95, stdev=8387.91 clat (msec): min=23, max=940, avg=437.39, stdev=119.65 lat (msec): min=31, max=955, avg=452.52, stdev=121.45 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 477], 80.00th=[ 518], 90.00th=[ 600], 95.00th=[ 676], | 99.00th=[ 802], 99.50th=[ 818], 99.90th=[ 844], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 104, per=25.13%, avg=45.73, stdev=20.17, samples=217 iops : min= 1, max= 26, avg=10.96, stdev= 5.10, samples=217 write: IOPS=11, BW=47.0KiB/s (48.2kB/s)(5180KiB/110110msec); 0 zone resets slat (usec): min=43, max=25725, avg=316.55, stdev=1069.55 clat (msec): min=279, max=1830, avg=914.77, stdev=236.46 lat (msec): min=279, max=1830, avg=915.08, stdev=236.45 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 550], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 793], 40.00th=[ 844], 50.00th=[ 885], 60.00th=[ 927], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1368], | 99.00th=[ 1586], 99.50th=[ 1620], 99.90th=[ 1770], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 103, per=24.72%, avg=46.73, stdev=19.48, samples=217 iops : min= 1, max= 25, avg=11.21, stdev= 4.87, samples=217 lat (msec) : 50=0.12%, 100=0.08%, 250=1.13%, 500=37.54%, 750=21.13% lat (msec) : 1000=24.38%, 2000=15.62% cpu : usr=0.08%, sys=0.66%, ctx=4659, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=182KiB/s (186kB/s), 45.1KiB/s-45.0KiB/s (46.2kB/s-47.1kB/s), io=19.6MiB (20.5MB), run=110046-110253msec WRITE: bw=189KiB/s (194kB/s), 47.0KiB/s-47.8KiB/s (48.2kB/s-48.9kB/s), io=20.4MiB (21.4MB), run=110046-110253msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148352: Wed Mar 11 00:22:36 2026 read: IOPS=1, BW=65.1MiB/s (68.3MB/s)(40.0MiB/614msec) slat (nsec): min=53973k, max=53973k, avg=53973213.00, stdev= 0.00 clat (nsec): min=533386k, max=533386k, avg=533386283.00, stdev= 0.00 lat (nsec): min=587365k, max=587365k, avg=587364988.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 535], 10.00th=[ 535], 20.00th=[ 535], | 30.00th=[ 535], 40.00th=[ 535], 50.00th=[ 535], 60.00th=[ 535], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.96%, ctx=43, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.1MiB/s (68.3MB/s), 65.1MiB/s-65.1MiB/s (68.3MB/s-68.3MB/s), io=40.0MiB (41.9MB), run=614-614msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout