== sanity test 398c: run fio to test AIO ================= 02:15:51 (1787120151) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.506902 s, 82.7 MB/s osc.lustre-OST0000-osc-ffff9341d88e9000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145546: Wed Aug 19 02:18:34 2026 write: IOPS=16, BW=66.3KiB/s (67.9kB/s)(10.0MiB/154459msec); 0 zone resets slat (usec): min=43, max=11874, avg=358.52, stdev=828.57 clat (msec): min=85, max=1807, avg=963.87, stdev=243.96 lat (msec): min=85, max=1807, avg=964.23, stdev=243.95 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 600], 10.00th=[ 718], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 953], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1620], 99.50th=[ 1720], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 247, per=24.89%, avg=65.70, stdev=31.19, samples=306 iops : min= 1, max= 61, avg=15.88, stdev= 7.82, samples=306 lat (msec) : 100=0.04%, 250=0.62%, 500=3.20%, 750=9.57%, 1000=46.17% lat (msec) : 2000=40.39% cpu : usr=0.05%, sys=0.37%, ctx=2115, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145547: Wed Aug 19 02:18:34 2026 write: IOPS=16, BW=66.2KiB/s (67.8kB/s)(10.0MiB/154581msec); 0 zone resets slat (usec): min=38, max=17822, avg=453.47, stdev=1182.93 clat (msec): min=238, max=1790, avg=964.39, stdev=241.76 lat (msec): min=238, max=1790, avg=964.85, stdev=241.78 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1636], 99.50th=[ 1703], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 255, per=25.06%, avg=66.15, stdev=29.23, samples=304 iops : min= 1, max= 63, avg=15.99, stdev= 7.30, samples=304 lat (msec) : 250=0.62%, 500=3.12%, 750=10.23%, 1000=45.35%, 2000=40.66% cpu : usr=0.07%, sys=0.40%, ctx=2513, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145548: Wed Aug 19 02:18:34 2026 write: IOPS=16, BW=66.4KiB/s (68.0kB/s)(10.0MiB/154174msec); 0 zone resets slat (usec): min=37, max=11267, avg=413.92, stdev=989.48 clat (msec): min=181, max=2037, avg=962.58, stdev=246.22 lat (msec): min=182, max=2038, avg=963.00, stdev=246.22 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 567], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 902], 50.00th=[ 944], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1670], 99.50th=[ 1754], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 252, per=25.12%, avg=66.30, stdev=30.13, samples=303 iops : min= 1, max= 63, avg=16.03, stdev= 7.57, samples=303 lat (msec) : 250=0.62%, 500=3.44%, 750=9.96%, 1000=46.56%, 2000=39.38% lat (msec) : >=2000=0.04% cpu : usr=0.07%, sys=0.35%, ctx=2144, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145549: Wed Aug 19 02:18:34 2026 write: IOPS=16, BW=66.2KiB/s (67.8kB/s)(10.0MiB/154587msec); 0 zone resets slat (usec): min=41, max=10907, avg=375.58, stdev=860.94 clat (msec): min=257, max=1894, avg=964.65, stdev=243.12 lat (msec): min=257, max=1895, avg=965.03, stdev=243.11 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1670], 99.50th=[ 1754], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 191, per=25.38%, avg=67.00, stdev=30.52, samples=300 iops : min= 1, max= 47, avg=16.20, stdev= 7.64, samples=300 lat (msec) : 500=4.06%, 750=9.45%, 1000=45.70%, 2000=40.78% cpu : usr=0.05%, sys=0.35%, ctx=2120, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=265KiB/s (271kB/s), 66.2KiB/s-66.4KiB/s (67.8kB/s-68.0kB/s), io=40.0MiB (41.9MB), run=154174-154587msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145581: Wed Aug 19 02:20:35 2026 read: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5048KiB/110185msec) slat (usec): min=92, max=63389, avg=15035.56, stdev=8144.26 clat (msec): min=20, max=1071, avg=428.24, stdev=116.27 lat (msec): min=29, max=1085, avg=443.29, stdev=117.28 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 338], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 439], | 70.00th=[ 468], 80.00th=[ 510], 90.00th=[ 567], 95.00th=[ 634], | 99.00th=[ 776], 99.50th=[ 860], 99.90th=[ 1062], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7, max= 127, per=25.48%, avg=45.87, stdev=20.61, samples=215 iops : min= 1, max= 31, avg=10.93, stdev= 5.19, samples=215 write: IOPS=11, BW=47.1KiB/s (48.3kB/s)(5192KiB/110185msec); 0 zone resets slat (usec): min=55, max=19780, avg=526.05, stdev=1436.10 clat (msec): min=108, max=1723, avg=925.61, stdev=244.67 lat (msec): min=110, max=1723, avg=926.14, stdev=244.66 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 550], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1045], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1519], 99.50th=[ 1586], 99.90th=[ 1687], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 115, per=24.94%, avg=46.65, stdev=20.27, samples=217 iops : min= 1, max= 28, avg=11.12, stdev= 5.10, samples=217 lat (msec) : 50=0.27%, 100=0.27%, 250=1.05%, 500=38.71%, 750=20.23% lat (msec) : 1000=20.27%, 2000=19.18% cpu : usr=0.11%, sys=0.83%, ctx=5011, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145582: Wed Aug 19 02:20:35 2026 read: IOPS=11, BW=44.7KiB/s (45.8kB/s)(4976KiB/111240msec) slat (usec): min=3415, max=64358, avg=15170.05, stdev=7931.28 clat (msec): min=21, max=1111, avg=430.26, stdev=112.74 lat (msec): min=29, max=1131, avg=445.44, stdev=113.85 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 443], | 70.00th=[ 477], 80.00th=[ 506], 90.00th=[ 575], 95.00th=[ 625], | 99.00th=[ 760], 99.50th=[ 802], 99.90th=[ 953], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 111, per=24.57%, avg=44.22, stdev=21.77, samples=220 iops : min= 1, max= 27, avg=10.51, stdev= 5.47, samples=220 write: IOPS=11, BW=47.3KiB/s (48.5kB/s)(5264KiB/111240msec); 0 zone resets slat (usec): min=54, max=16368, avg=457.98, stdev=1162.96 clat (msec): min=210, max=1701, avg=928.31, stdev=242.55 lat (msec): min=210, max=1701, avg=928.77, stdev=242.55 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 550], 10.00th=[ 617], 20.00th=[ 726], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1045], 80.00th=[ 1116], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1502], 99.50th=[ 1603], 99.90th=[ 1653], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 104, per=24.96%, avg=46.68, stdev=19.50, samples=220 iops : min= 1, max= 26, avg=11.13, stdev= 4.87, samples=220 lat (msec) : 50=0.04%, 100=0.23%, 250=1.25%, 500=37.85%, 750=20.62% lat (msec) : 1000=20.86%, 2000=19.14% cpu : usr=0.09%, sys=0.73%, ctx=4656, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145583: Wed Aug 19 02:20:35 2026 read: IOPS=11, BW=44.8KiB/s (45.9kB/s)(4984KiB/111276msec) slat (usec): min=3241, max=82257, avg=15233.81, stdev=8345.18 clat (msec): min=62, max=878, avg=431.03, stdev=107.02 lat (msec): min=75, max=918, avg=446.28, stdev=107.98 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 288], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 477], 80.00th=[ 510], 90.00th=[ 567], 95.00th=[ 609], | 99.00th=[ 751], 99.50th=[ 810], 99.90th=[ 835], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 127, per=24.58%, avg=44.25, stdev=20.69, samples=221 iops : min= 1, max= 31, avg=10.52, stdev= 5.19, samples=221 write: IOPS=11, BW=47.2KiB/s (48.4kB/s)(5256KiB/111276msec); 0 zone resets slat (usec): min=54, max=33934, avg=532.31, stdev=1461.43 clat (msec): min=261, max=1705, avg=927.99, stdev=234.44 lat (msec): min=261, max=1705, avg=928.52, stdev=234.43 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1045], 80.00th=[ 1116], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1485], 99.50th=[ 1586], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 104, per=24.75%, avg=46.28, stdev=19.10, samples=221 iops : min= 1, max= 26, avg=11.02, stdev= 4.82, samples=221 lat (msec) : 100=0.31%, 250=0.74%, 500=37.23%, 750=21.29%, 1000=21.09% lat (msec) : 2000=19.34% cpu : usr=0.12%, sys=0.82%, ctx=5179, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145584: Wed Aug 19 02:20:35 2026 read: IOPS=11, BW=45.4KiB/s (46.5kB/s)(5060KiB/111346msec) slat (usec): min=3167, max=66311, avg=15335.97, stdev=8141.83 clat (msec): min=15, max=880, avg=428.15, stdev=112.46 lat (msec): min=24, max=893, avg=443.49, stdev=113.76 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 284], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 443], | 70.00th=[ 472], 80.00th=[ 510], 90.00th=[ 575], 95.00th=[ 634], | 99.00th=[ 726], 99.50th=[ 760], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 119, per=25.24%, avg=45.43, stdev=20.95, samples=218 iops : min= 1, max= 29, avg=10.81, stdev= 5.26, samples=218 write: IOPS=11, BW=46.5KiB/s (47.6kB/s)(5180KiB/111346msec); 0 zone resets slat (usec): min=58, max=13710, avg=431.09, stdev=1031.31 clat (msec): min=311, max=1722, avg=938.80, stdev=237.39 lat (msec): min=311, max=1722, avg=939.24, stdev=237.38 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 575], 10.00th=[ 634], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 995], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1536], 99.50th=[ 1586], 99.90th=[ 1703], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 96, per=24.46%, avg=45.73, stdev=18.21, samples=221 iops : min= 1, max= 24, avg=10.88, stdev= 4.60, samples=221 lat (msec) : 20=0.04%, 50=0.08%, 100=0.39%, 250=0.82%, 500=37.58% lat (msec) : 750=21.88%, 1000=19.73%, 2000=19.49% cpu : usr=0.10%, sys=0.72%, ctx=4583, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=180KiB/s (185kB/s), 44.7KiB/s-45.8KiB/s (45.8kB/s-46.9kB/s), io=19.6MiB (20.5MB), run=110185-111346msec WRITE: bw=188KiB/s (192kB/s), 46.5KiB/s-47.3KiB/s (47.6kB/s-48.5kB/s), io=20.4MiB (21.4MB), run=110185-111346msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145604: Wed Aug 19 02:20:46 2026 read: IOPS=1, BW=69.1MiB/s (72.4MB/s)(40.0MiB/579msec) slat (nsec): min=43607k, max=43607k, avg=43607158.00, stdev= 0.00 clat (nsec): min=507468k, max=507468k, avg=507468448.00, stdev= 0.00 lat (nsec): min=551080k, max=551080k, avg=551080439.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.25%, ctx=22, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=69.1MiB/s (72.4MB/s), 69.1MiB/s-69.1MiB/s (72.4MB/s-72.4MB/s), io=40.0MiB (41.9MB), run=579-579msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout