== sanity test 398c: run fio to test AIO ================= 21:20:58 (1775870458) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.400732 s, 105 MB/s osc.lustre-OST0000-osc-ffff965ac6d7b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=151222: Fri Apr 10 21:23:49 2026 write: IOPS=15, BW=63.1KiB/s (64.6kB/s)(10.0MiB/162305msec); 0 zone resets slat (usec): min=42, max=13392, avg=285.59, stdev=846.20 clat (msec): min=139, max=1965, avg=1013.28, stdev=277.58 lat (msec): min=140, max=1965, avg=1013.56, stdev=277.57 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 584], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1720], 99.50th=[ 1770], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 256, per=24.93%, avg=62.82, stdev=30.79, samples=320 iops : min= 1, max= 64, avg=15.21, stdev= 7.67, samples=320 lat (msec) : 250=0.62%, 500=1.45%, 750=11.56%, 1000=38.87%, 2000=47.50% cpu : usr=0.04%, sys=0.25%, ctx=2255, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151223: Fri Apr 10 21:23:49 2026 write: IOPS=15, BW=63.1KiB/s (64.6kB/s)(10.0MiB/162238msec); 0 zone resets slat (usec): min=44, max=14949, avg=295.87, stdev=852.20 clat (msec): min=97, max=2005, avg=1012.76, stdev=274.07 lat (msec): min=99, max=2006, avg=1013.06, stdev=274.06 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 617], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 986], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1720], 99.50th=[ 1787], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 145, per=24.79%, avg=62.47, stdev=27.45, samples=322 iops : min= 1, max= 36, avg=15.12, stdev= 6.87, samples=322 lat (msec) : 100=0.04%, 250=0.27%, 500=2.42%, 750=10.51%, 1000=38.79% lat (msec) : 2000=47.93%, >=2000=0.04% cpu : usr=0.04%, sys=0.27%, ctx=2395, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151224: Fri Apr 10 21:23:49 2026 write: IOPS=15, BW=63.3KiB/s (64.9kB/s)(10.0MiB/161662msec); 0 zone resets slat (usec): min=43, max=15908, avg=287.66, stdev=866.55 clat (msec): min=125, max=1964, avg=1009.67, stdev=277.37 lat (msec): min=125, max=1964, avg=1009.96, stdev=277.39 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 617], 10.00th=[ 684], 20.00th=[ 818], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1519], | 99.00th=[ 1720], 99.50th=[ 1804], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 210, per=25.33%, avg=63.83, stdev=28.51, samples=315 iops : min= 1, max= 52, avg=15.47, stdev= 7.17, samples=315 lat (msec) : 250=0.62%, 500=2.62%, 750=11.48%, 1000=37.07%, 2000=48.20% cpu : usr=0.04%, sys=0.26%, ctx=2284, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151225: Fri Apr 10 21:23:49 2026 write: IOPS=15, BW=63.1KiB/s (64.6kB/s)(10.0MiB/162303msec); 0 zone resets slat (usec): min=38, max=13176, avg=278.61, stdev=733.58 clat (msec): min=226, max=1976, avg=1013.24, stdev=273.36 lat (msec): min=226, max=1976, avg=1013.52, stdev=273.34 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 986], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1737], 99.50th=[ 1787], 99.90th=[ 1955], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 136, per=24.88%, avg=62.69, stdev=27.84, samples=321 iops : min= 1, max= 34, avg=15.18, stdev= 6.96, samples=321 lat (msec) : 250=0.04%, 500=2.54%, 750=10.86%, 1000=39.57%, 2000=46.99% cpu : usr=0.04%, sys=0.26%, ctx=2308, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=252KiB/s (258kB/s), 63.1KiB/s-63.3KiB/s (64.6kB/s-64.9kB/s), io=40.0MiB (41.9MB), run=161662-162305msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151258: Fri Apr 10 21:25:41 2026 read: IOPS=12, BW=48.6KiB/s (49.8kB/s)(5048KiB/103774msec) slat (usec): min=56, max=80803, avg=12759.87, stdev=6819.70 clat (msec): min=12, max=1069, avg=396.12, stdev=91.64 lat (msec): min=15, max=1077, avg=408.89, stdev=92.70 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 275], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 468], 90.00th=[ 506], 95.00th=[ 542], | 99.00th=[ 625], 99.50th=[ 667], 99.90th=[ 902], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7, max= 104, per=25.17%, avg=48.33, stdev=20.85, samples=205 iops : min= 1, max= 26, avg=11.62, stdev= 5.21, samples=205 write: IOPS=12, BW=50.0KiB/s (51.2kB/s)(5192KiB/103774msec); 0 zone resets slat (usec): min=52, max=18950, avg=402.88, stdev=1037.67 clat (msec): min=131, max=1612, avg=880.20, stdev=231.70 lat (msec): min=133, max=1612, avg=880.61, stdev=231.72 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 527], 10.00th=[ 592], 20.00th=[ 693], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 969], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1301], | 99.00th=[ 1469], 99.50th=[ 1502], 99.90th=[ 1586], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 104, per=24.69%, avg=49.39, stdev=18.54, samples=206 iops : min= 1, max= 26, avg=11.89, stdev= 4.61, samples=206 lat (msec) : 20=0.08%, 50=0.08%, 100=0.12%, 250=1.13%, 500=43.95% lat (msec) : 750=18.01%, 1000=23.01%, 2000=13.63% cpu : usr=0.10%, sys=0.65%, ctx=4961, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151259: Fri Apr 10 21:25:41 2026 read: IOPS=11, BW=47.7KiB/s (48.8kB/s)(4976KiB/104357msec) slat (usec): min=2800, max=42954, avg=12557.39, stdev=6369.98 clat (msec): min=16, max=706, avg=393.55, stdev=85.39 lat (msec): min=27, max=715, avg=406.11, stdev=86.44 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 326], | 30.00th=[ 347], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 460], 90.00th=[ 510], 95.00th=[ 542], | 99.00th=[ 609], 99.50th=[ 634], 99.90th=[ 693], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7, max= 136, per=24.44%, avg=46.92, stdev=22.70, samples=207 iops : min= 1, max= 34, avg=11.27, stdev= 5.66, samples=207 write: IOPS=12, BW=50.4KiB/s (51.7kB/s)(5264KiB/104357msec); 0 zone resets slat (usec): min=37, max=11794, avg=363.26, stdev=811.45 clat (msec): min=120, max=1655, avg=882.79, stdev=229.07 lat (msec): min=120, max=1656, avg=883.16, stdev=229.09 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 531], 10.00th=[ 600], 20.00th=[ 693], | 30.00th=[ 768], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 919], | 70.00th=[ 969], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1452], 99.50th=[ 1519], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 103, per=24.85%, avg=49.71, stdev=19.72, samples=207 iops : min= 1, max= 25, avg=11.97, stdev= 4.92, samples=207 lat (msec) : 20=0.04%, 50=0.12%, 100=0.16%, 250=0.74%, 500=43.87% lat (msec) : 750=17.54%, 1000=23.75%, 2000=13.79% cpu : usr=0.07%, sys=0.66%, ctx=4856, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151260: Fri Apr 10 21:25:41 2026 read: IOPS=11, BW=47.8KiB/s (48.9kB/s)(4984KiB/104278msec) slat (usec): min=2769, max=47848, avg=12712.65, stdev=6445.31 clat (msec): min=13, max=713, avg=393.81, stdev=90.48 lat (msec): min=25, max=735, avg=406.53, stdev=91.59 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 271], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 468], 90.00th=[ 510], 95.00th=[ 542], | 99.00th=[ 609], 99.50th=[ 642], 99.90th=[ 701], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 7, max= 119, per=24.66%, avg=47.35, stdev=21.26, samples=207 iops : min= 1, max= 29, avg=11.38, stdev= 5.31, samples=207 write: IOPS=12, BW=50.4KiB/s (51.6kB/s)(5256KiB/104278msec); 0 zone resets slat (usec): min=47, max=11715, avg=358.54, stdev=754.12 clat (msec): min=246, max=1644, avg=881.77, stdev=224.97 lat (msec): min=246, max=1644, avg=882.13, stdev=224.99 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 518], 10.00th=[ 600], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1452], 99.50th=[ 1502], 99.90th=[ 1586], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 15, max= 103, per=24.79%, avg=49.57, stdev=18.26, samples=207 iops : min= 3, max= 25, avg=11.94, stdev= 4.57, samples=207 lat (msec) : 20=0.08%, 50=0.08%, 100=0.47%, 250=0.98%, 500=43.98% lat (msec) : 750=16.84%, 1000=23.83%, 2000=13.75% cpu : usr=0.07%, sys=0.67%, ctx=4822, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151261: Fri Apr 10 21:25:41 2026 read: IOPS=12, BW=48.5KiB/s (49.7kB/s)(5060KiB/104231msec) slat (usec): min=2727, max=76865, avg=12530.57, stdev=6390.84 clat (msec): min=25, max=686, avg=401.16, stdev=84.02 lat (msec): min=40, max=692, avg=413.69, stdev=84.55 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 334], | 30.00th=[ 355], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 472], 90.00th=[ 518], 95.00th=[ 550], | 99.00th=[ 609], 99.50th=[ 634], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 7, max= 111, per=25.15%, avg=48.28, stdev=19.97, samples=206 iops : min= 1, max= 27, avg=11.61, stdev= 4.96, samples=206 write: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5180KiB/104231msec); 0 zone resets slat (usec): min=53, max=10629, avg=391.83, stdev=814.73 clat (msec): min=344, max=1613, avg=880.83, stdev=220.52 lat (msec): min=345, max=1613, avg=881.23, stdev=220.54 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 518], 10.00th=[ 592], 20.00th=[ 701], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1062], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1469], 99.50th=[ 1552], 99.90th=[ 1586], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 8, max= 103, per=24.50%, avg=49.01, stdev=17.68, samples=207 iops : min= 2, max= 25, avg=11.79, stdev= 4.46, samples=207 lat (msec) : 50=0.04%, 100=0.12%, 250=0.66%, 500=44.06%, 750=18.09% lat (msec) : 1000=23.48%, 2000=13.55% cpu : usr=0.09%, sys=0.66%, ctx=5106, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=192KiB/s (197kB/s), 47.7KiB/s-48.6KiB/s (48.8kB/s-49.8kB/s), io=19.6MiB (20.5MB), run=103774-104357msec WRITE: bw=200KiB/s (205kB/s), 49.7KiB/s-50.4KiB/s (50.9kB/s-51.7kB/s), io=20.4MiB (21.4MB), run=103774-104357msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=151280: Fri Apr 10 21:25:50 2026 read: IOPS=1, BW=77.5MiB/s (81.3MB/s)(40.0MiB/516msec) slat (nsec): min=32882k, max=32882k, avg=32882090.00, stdev= 0.00 clat (nsec): min=460065k, max=460065k, avg=460064559.00, stdev= 0.00 lat (nsec): min=492952k, max=492952k, avg=492952250.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.14%, ctx=4, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.5MiB/s (81.3MB/s), 77.5MiB/s-77.5MiB/s (81.3MB/s-81.3MB/s), io=40.0MiB (41.9MB), run=516-516msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout