== sanity test 398c: run fio to test AIO ================= 11:54:12 (1772470452) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.408698 s, 103 MB/s osc.lustre-OST0000-osc-ffff96aded086000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147567: Mon Mar 2 11:56:55 2026 write: IOPS=16, BW=67.4KiB/s (69.0kB/s)(10.0MiB/151964msec); 0 zone resets slat (usec): min=22, max=23415, avg=194.42, stdev=936.40 clat (msec): min=56, max=2257, avg=948.73, stdev=295.56 lat (msec): min=57, max=2257, avg=948.93, stdev=295.56 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 502], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 902], 60.00th=[ 961], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1318], 95.00th=[ 1569], | 99.00th=[ 1938], 99.50th=[ 2056], 99.90th=[ 2165], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 223, per=24.87%, avg=66.90, stdev=31.53, samples=301 iops : min= 1, max= 55, avg=16.29, stdev= 7.90, samples=301 lat (msec) : 100=0.08%, 250=0.08%, 500=4.80%, 750=16.33%, 1000=44.61% lat (msec) : 2000=33.40%, >=2000=0.70% cpu : usr=0.05%, sys=0.21%, ctx=1847, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147568: Mon Mar 2 11:56:55 2026 write: IOPS=16, BW=67.5KiB/s (69.1kB/s)(10.0MiB/151721msec); 0 zone resets slat (usec): min=21, max=13781, avg=177.49, stdev=695.89 clat (msec): min=203, max=2188, avg=947.66, stdev=298.69 lat (msec): min=203, max=2188, avg=947.83, stdev=298.72 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 518], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 852], 50.00th=[ 902], 60.00th=[ 961], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1318], 95.00th=[ 1552], | 99.00th=[ 1938], 99.50th=[ 2039], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 175, per=24.98%, avg=67.20, stdev=31.23, samples=299 iops : min= 1, max= 43, avg=16.36, stdev= 7.84, samples=299 lat (msec) : 250=0.62%, 500=3.79%, 750=16.99%, 1000=44.57%, 2000=33.44% lat (msec) : >=2000=0.59% cpu : usr=0.04%, sys=0.22%, ctx=1843, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147569: Mon Mar 2 11:56:55 2026 write: IOPS=16, BW=67.4KiB/s (69.1kB/s)(10.0MiB/151848msec); 0 zone resets slat (usec): min=21, max=22192, avg=188.87, stdev=804.65 clat (msec): min=215, max=2249, avg=947.66, stdev=292.02 lat (msec): min=215, max=2249, avg=947.85, stdev=292.01 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 527], 10.00th=[ 667], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 953], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1318], 95.00th=[ 1569], | 99.00th=[ 1888], 99.50th=[ 2005], 99.90th=[ 2165], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 254, per=25.12%, avg=67.56, stdev=31.42, samples=298 iops : min= 1, max= 63, avg=16.45, stdev= 7.87, samples=298 lat (msec) : 250=0.04%, 500=4.65%, 750=16.52%, 1000=44.61%, 2000=33.71% lat (msec) : >=2000=0.47% cpu : usr=0.05%, sys=0.19%, ctx=1794, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147570: Mon Mar 2 11:56:55 2026 write: IOPS=16, BW=67.4KiB/s (69.1kB/s)(10.0MiB/151850msec); 0 zone resets slat (usec): min=22, max=22807, avg=213.26, stdev=933.53 clat (msec): min=185, max=2252, avg=947.80, stdev=289.20 lat (msec): min=185, max=2252, avg=948.02, stdev=289.21 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 514], 10.00th=[ 667], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 852], 50.00th=[ 902], 60.00th=[ 953], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1318], 95.00th=[ 1536], | 99.00th=[ 1838], 99.50th=[ 1989], 99.90th=[ 2140], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 232, per=25.11%, avg=67.55, stdev=31.55, samples=298 iops : min= 1, max= 58, avg=16.45, stdev= 7.92, samples=298 lat (msec) : 250=0.04%, 500=4.84%, 750=15.51%, 1000=46.25%, 2000=33.01% lat (msec) : >=2000=0.35% cpu : usr=0.06%, sys=0.22%, ctx=2022, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=270KiB/s (276kB/s), 67.4KiB/s-67.5KiB/s (69.0kB/s-69.1kB/s), io=40.0MiB (41.9MB), run=151721-151964msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147601: Mon Mar 2 11:58:51 2026 read: IOPS=11, BW=47.2KiB/s (48.3kB/s)(5048KiB/107015msec) slat (usec): min=2501, max=63010, avg=14155.15, stdev=7885.16 clat (msec): min=18, max=1355, avg=405.50, stdev=129.92 lat (msec): min=27, max=1370, avg=419.66, stdev=131.26 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 247], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 414], | 70.00th=[ 443], 80.00th=[ 485], 90.00th=[ 550], 95.00th=[ 625], | 99.00th=[ 835], 99.50th=[ 1011], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 119, per=25.35%, avg=47.16, stdev=22.07, samples=210 iops : min= 1, max= 29, avg=11.30, stdev= 5.59, samples=210 write: IOPS=12, BW=48.5KiB/s (49.7kB/s)(5192KiB/107015msec); 0 zone resets slat (usec): min=29, max=17114, avg=246.83, stdev=848.46 clat (msec): min=282, max=2083, avg=907.70, stdev=262.10 lat (msec): min=282, max=2083, avg=907.95, stdev=262.11 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1083], 90.00th=[ 1267], 95.00th=[ 1452], | 99.00th=[ 1720], 99.50th=[ 1871], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 103, per=24.94%, avg=48.39, stdev=18.52, samples=210 iops : min= 1, max= 25, avg=11.61, stdev= 4.68, samples=210 lat (msec) : 20=0.04%, 50=0.20%, 100=0.20%, 250=2.15%, 500=39.26% lat (msec) : 750=21.29%, 1000=22.19%, 2000=14.57%, >=2000=0.12% cpu : usr=0.10%, sys=0.66%, ctx=4174, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147602: Mon Mar 2 11:58:51 2026 read: IOPS=11, BW=46.3KiB/s (47.5kB/s)(4976KiB/107379msec) slat (msec): min=2, max=492, avg=14.62, stdev=15.71 clat (msec): min=31, max=1213, avg=410.30, stdev=123.05 lat (msec): min=42, max=1226, avg=424.92, stdev=123.92 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 251], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 451], 80.00th=[ 489], 90.00th=[ 558], 95.00th=[ 634], | 99.00th=[ 776], 99.50th=[ 902], 99.90th=[ 1053], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 112, per=24.83%, avg=46.18, stdev=21.67, samples=211 iops : min= 1, max= 28, avg=11.06, stdev= 5.45, samples=211 write: IOPS=12, BW=49.0KiB/s (50.2kB/s)(5264KiB/107379msec); 0 zone resets slat (usec): min=28, max=22472, avg=239.94, stdev=890.49 clat (msec): min=247, max=2089, avg=901.30, stdev=263.08 lat (msec): min=248, max=2090, avg=901.54, stdev=263.09 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1070], 90.00th=[ 1234], 95.00th=[ 1452], | 99.00th=[ 1703], 99.50th=[ 1838], 99.90th=[ 2072], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 111, per=24.99%, avg=48.47, stdev=20.82, samples=212 iops : min= 1, max= 27, avg=11.64, stdev= 5.23, samples=212 lat (msec) : 50=0.23%, 100=0.16%, 250=2.07%, 500=38.59%, 750=22.15% lat (msec) : 1000=22.70%, 2000=13.98%, >=2000=0.12% cpu : usr=0.09%, sys=0.62%, ctx=4086, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147603: Mon Mar 2 11:58:51 2026 read: IOPS=11, BW=46.4KiB/s (47.6kB/s)(4984KiB/107325msec) slat (usec): min=50, max=59311, avg=14031.93, stdev=7979.29 clat (msec): min=17, max=1027, avg=402.12, stdev=109.65 lat (msec): min=23, max=1060, avg=416.16, stdev=111.00 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 249], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 443], 80.00th=[ 481], 90.00th=[ 542], 95.00th=[ 600], | 99.00th=[ 718], 99.50th=[ 760], 99.90th=[ 936], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 128, per=24.99%, avg=46.49, stdev=23.62, samples=211 iops : min= 1, max= 32, avg=11.14, stdev= 5.94, samples=211 write: IOPS=12, BW=48.0KiB/s (50.1kB/s)(5256KiB/107325msec); 0 zone resets slat (usec): min=30, max=20352, avg=278.21, stdev=1194.28 clat (msec): min=136, max=2104, avg=911.11, stdev=273.39 lat (msec): min=137, max=2105, avg=911.39, stdev=273.39 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1267], 95.00th=[ 1485], | 99.00th=[ 1804], 99.50th=[ 1921], 99.90th=[ 2039], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 103, per=24.79%, avg=48.08, stdev=20.37, samples=213 iops : min= 1, max= 25, avg=11.54, stdev= 5.14, samples=213 lat (msec) : 20=0.08%, 50=0.08%, 100=0.20%, 250=2.23%, 500=39.22% lat (msec) : 750=21.72%, 1000=21.05%, 2000=15.31%, >=2000=0.12% cpu : usr=0.09%, sys=0.61%, ctx=4066, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147604: Mon Mar 2 11:58:51 2026 read: IOPS=11, BW=47.2KiB/s (48.4kB/s)(5060KiB/107112msec) slat (usec): min=3047, max=62039, avg=14380.80, stdev=8169.49 clat (msec): min=25, max=1198, avg=409.98, stdev=122.82 lat (msec): min=40, max=1218, avg=424.37, stdev=124.69 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 253], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 368], 50.00th=[ 393], 60.00th=[ 422], | 70.00th=[ 451], 80.00th=[ 493], 90.00th=[ 567], 95.00th=[ 625], | 99.00th=[ 802], 99.50th=[ 902], 99.90th=[ 1028], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 128, per=25.35%, avg=47.15, stdev=22.27, samples=211 iops : min= 1, max= 32, avg=11.30, stdev= 5.65, samples=211 write: IOPS=12, BW=48.4KiB/s (49.5kB/s)(5180KiB/107112msec); 0 zone resets slat (usec): min=25, max=12262, avg=231.78, stdev=878.47 clat (msec): min=285, max=2050, avg=905.65, stdev=258.14 lat (msec): min=285, max=2050, avg=905.88, stdev=258.16 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 575], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1070], 90.00th=[ 1234], 95.00th=[ 1452], | 99.00th=[ 1754], 99.50th=[ 1821], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 103, per=24.48%, avg=47.49, stdev=20.72, samples=213 iops : min= 1, max= 25, avg=11.39, stdev= 5.24, samples=213 lat (msec) : 50=0.08%, 100=0.20%, 250=1.99%, 500=39.10%, 750=21.56% lat (msec) : 1000=22.42%, 2000=14.57%, >=2000=0.08% cpu : usr=0.11%, sys=0.65%, ctx=4172, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=187KiB/s (191kB/s), 46.3KiB/s-47.2KiB/s (47.5kB/s-48.4kB/s), io=19.6MiB (20.5MB), run=107015-107379msec WRITE: bw=195KiB/s (199kB/s), 48.4KiB/s-49.0KiB/s (49.5kB/s-50.2kB/s), io=20.4MiB (21.4MB), run=107015-107379msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147625: Mon Mar 2 11:59:04 2026 read: IOPS=1, BW=58.9MiB/s (61.8MB/s)(40.0MiB/679msec) slat (nsec): min=55816k, max=55816k, avg=55815913.00, stdev= 0.00 clat (nsec): min=597633k, max=597633k, avg=597632616.00, stdev= 0.00 lat (nsec): min=653453k, max=653453k, avg=653452682.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 600], 10.00th=[ 600], 20.00th=[ 600], | 30.00th=[ 600], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 600], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.47%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=58.9MiB/s (61.8MB/s), 58.9MiB/s-58.9MiB/s (61.8MB/s-61.8MB/s), io=40.0MiB (41.9MB), run=679-679msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout