== sanity test 398c: run fio to test AIO ================= 03:18:18 (1786173498) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.896762 s, 46.8 MB/s osc.lustre-OST0000-osc-ffff9fa91d8d7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=143934: Sat Aug 8 03:20:48 2026 write: IOPS=18, BW=72.1KiB/s (73.8kB/s)(10.0MiB/142079msec); 0 zone resets slat (usec): min=42, max=69382, avg=740.37, stdev=3260.20 clat (msec): min=167, max=1613, avg=885.93, stdev=228.94 lat (msec): min=167, max=1613, avg=886.68, stdev=228.86 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 531], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1435], 99.50th=[ 1452], 99.90th=[ 1519], 99.95th=[ 1552], | 99.99th=[ 1620] bw ( KiB/s): min= 8, max= 254, per=24.76%, avg=71.30, stdev=30.96, samples=279 iops : min= 2, max= 63, avg=17.36, stdev= 7.70, samples=279 lat (msec) : 250=0.86%, 500=3.83%, 750=20.08%, 1000=45.98%, 2000=29.26% cpu : usr=0.07%, sys=0.47%, ctx=2305, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=143935: Sat Aug 8 03:20:48 2026 write: IOPS=18, BW=72.1KiB/s (73.8kB/s)(10.0MiB/142004msec); 0 zone resets slat (usec): min=49, max=60690, avg=756.11, stdev=2848.99 clat (msec): min=154, max=2178, avg=885.06, stdev=231.10 lat (msec): min=154, max=2178, avg=885.82, stdev=231.02 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 523], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 1003], 80.00th=[ 1070], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1418], 99.50th=[ 1469], 99.90th=[ 2106], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 262, per=24.98%, avg=71.95, stdev=30.17, samples=276 iops : min= 1, max= 65, avg=17.53, stdev= 7.54, samples=276 lat (msec) : 250=1.25%, 500=3.44%, 750=19.41%, 1000=45.55%, 2000=30.23% lat (msec) : >=2000=0.12% cpu : usr=0.07%, sys=0.52%, ctx=2535, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=143936: Sat Aug 8 03:20:48 2026 write: IOPS=18, BW=72.1KiB/s (73.8kB/s)(10.0MiB/142049msec); 0 zone resets slat (usec): min=50, max=96593, avg=762.75, stdev=3922.09 clat (msec): min=239, max=1584, avg=885.46, stdev=222.70 lat (msec): min=239, max=1584, avg=886.24, stdev=222.60 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 527], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1070], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1385], 99.50th=[ 1435], 99.90th=[ 1485], 99.95th=[ 1502], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 254, per=24.84%, avg=71.54, stdev=29.81, samples=278 iops : min= 1, max= 63, avg=17.41, stdev= 7.44, samples=278 lat (msec) : 250=0.62%, 500=3.98%, 750=19.41%, 1000=46.41%, 2000=29.57% cpu : usr=0.08%, sys=0.48%, ctx=2283, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=143937: Sat Aug 8 03:20:48 2026 write: IOPS=18, BW=72.1KiB/s (73.8kB/s)(10.0MiB/142053msec); 0 zone resets slat (usec): min=49, max=203548, avg=834.33, stdev=4903.56 clat (msec): min=103, max=2287, avg=886.32, stdev=232.35 lat (msec): min=104, max=2287, avg=887.17, stdev=232.31 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 514], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1070], 90.00th=[ 1167], 95.00th=[ 1234], | 99.00th=[ 1435], 99.50th=[ 1502], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 270, per=24.90%, avg=71.72, stdev=29.11, samples=277 iops : min= 1, max= 67, avg=17.46, stdev= 7.28, samples=277 lat (msec) : 250=1.13%, 500=3.67%, 750=19.30%, 1000=46.41%, 2000=29.26% lat (msec) : >=2000=0.23% cpu : usr=0.10%, sys=0.68%, ctx=2724, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=288KiB/s (295kB/s), 72.1KiB/s-72.1KiB/s (73.8kB/s-73.8kB/s), io=40.0MiB (41.9MB), run=142004-142079msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=143967: Sat Aug 8 03:22:42 2026 read: IOPS=11, BW=47.6KiB/s (48.8kB/s)(5048KiB/106000msec) slat (msec): min=3, max=161, avg=21.34, stdev=18.74 clat (msec): min=11, max=915, avg=424.11, stdev=115.35 lat (msec): min=22, max=937, avg=445.46, stdev=115.80 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 262], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 439], | 70.00th=[ 472], 80.00th=[ 510], 90.00th=[ 567], 95.00th=[ 625], | 99.00th=[ 751], 99.50th=[ 810], 99.90th=[ 911], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 7, max= 104, per=25.35%, avg=47.65, stdev=21.29, samples=206 iops : min= 1, max= 26, avg=11.45, stdev= 5.35, samples=206 write: IOPS=12, BW=48.0KiB/s (50.2kB/s)(5192KiB/106000msec); 0 zone resets slat (usec): min=49, max=53114, avg=692.52, stdev=2809.85 clat (msec): min=225, max=1806, avg=865.93, stdev=237.22 lat (msec): min=225, max=1807, avg=866.63, stdev=237.17 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 502], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1053], 90.00th=[ 1167], 95.00th=[ 1284], | 99.00th=[ 1519], 99.50th=[ 1620], 99.90th=[ 1787], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 111, per=24.69%, avg=48.40, stdev=21.34, samples=207 iops : min= 1, max= 27, avg=11.63, stdev= 5.35, samples=207 lat (msec) : 20=0.04%, 50=0.04%, 100=0.16%, 250=1.52%, 500=39.06% lat (msec) : 750=24.77%, 1000=21.21%, 2000=13.20% cpu : usr=0.10%, sys=0.92%, ctx=4510, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143968: Sat Aug 8 03:22:42 2026 read: IOPS=11, BW=46.7KiB/s (47.8kB/s)(4976KiB/106532msec) slat (msec): min=3, max=178, avg=21.38, stdev=18.48 clat (msec): min=21, max=903, avg=429.49, stdev=116.36 lat (msec): min=25, max=925, avg=450.88, stdev=116.62 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 255], 10.00th=[ 296], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 481], 80.00th=[ 518], 90.00th=[ 575], 95.00th=[ 625], | 99.00th=[ 718], 99.50th=[ 802], 99.90th=[ 869], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 112, per=24.83%, avg=46.67, stdev=22.11, samples=206 iops : min= 1, max= 28, avg=11.21, stdev= 5.56, samples=206 write: IOPS=12, BW=49.4KiB/s (50.6kB/s)(5264KiB/106532msec); 0 zone resets slat (usec): min=49, max=63864, avg=541.95, stdev=2306.86 clat (msec): min=170, max=1801, avg=862.53, stdev=245.39 lat (msec): min=170, max=1801, avg=863.08, stdev=245.31 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 481], 10.00th=[ 558], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 793], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1053], 90.00th=[ 1183], 95.00th=[ 1284], | 99.00th=[ 1569], 99.50th=[ 1670], 99.90th=[ 1787], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 108, per=25.00%, avg=49.01, stdev=21.33, samples=208 iops : min= 1, max= 27, avg=11.80, stdev= 5.35, samples=208 lat (msec) : 50=0.12%, 100=0.16%, 250=2.11%, 500=37.54%, 750=26.13% lat (msec) : 1000=20.31%, 2000=13.63% cpu : usr=0.12%, sys=0.89%, ctx=4356, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143969: Sat Aug 8 03:22:42 2026 read: IOPS=11, BW=46.8KiB/s (47.9kB/s)(4984KiB/106490msec) slat (msec): min=3, max=183, avg=21.52, stdev=17.94 clat (msec): min=23, max=904, avg=434.03, stdev=119.83 lat (msec): min=29, max=933, avg=455.56, stdev=119.75 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 264], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 422], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 531], 90.00th=[ 584], 95.00th=[ 642], | 99.00th=[ 785], 99.50th=[ 818], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 120, per=24.84%, avg=46.70, stdev=22.02, samples=207 iops : min= 1, max= 30, avg=11.22, stdev= 5.55, samples=207 write: IOPS=12, BW=49.4KiB/s (50.5kB/s)(5256KiB/106490msec); 0 zone resets slat (usec): min=53, max=48681, avg=638.29, stdev=1991.21 clat (msec): min=146, max=1828, avg=857.91, stdev=236.23 lat (msec): min=147, max=1828, avg=858.56, stdev=236.16 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 489], 10.00th=[ 567], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1267], | 99.00th=[ 1469], 99.50th=[ 1603], 99.90th=[ 1770], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 8, max= 106, per=24.89%, avg=48.78, stdev=18.96, samples=208 iops : min= 2, max= 26, avg=11.74, stdev= 4.74, samples=208 lat (msec) : 50=0.04%, 100=0.23%, 250=1.84%, 500=35.62%, 750=27.46% lat (msec) : 1000=22.54%, 2000=12.27% cpu : usr=0.11%, sys=0.94%, ctx=4597, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143970: Sat Aug 8 03:22:42 2026 read: IOPS=12, BW=48.1KiB/s (49.3kB/s)(5060KiB/105177msec) slat (usec): min=126, max=151087, avg=20231.35, stdev=16978.28 clat (msec): min=21, max=1058, avg=432.37, stdev=127.47 lat (msec): min=27, max=1065, avg=452.61, stdev=127.53 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 247], 10.00th=[ 288], 20.00th=[ 334], | 30.00th=[ 368], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 456], | 70.00th=[ 485], 80.00th=[ 527], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 760], 99.50th=[ 919], 99.90th=[ 1045], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 8, max= 95, per=25.69%, avg=48.30, stdev=19.72, samples=203 iops : min= 2, max= 23, avg=11.62, stdev= 4.94, samples=203 write: IOPS=12, BW=49.2KiB/s (50.4kB/s)(5180KiB/105177msec); 0 zone resets slat (usec): min=46, max=43897, avg=703.47, stdev=2482.00 clat (msec): min=146, max=2040, avg=854.83, stdev=249.17 lat (msec): min=147, max=2040, avg=855.54, stdev=249.00 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 477], 10.00th=[ 550], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1267], | 99.00th=[ 1636], 99.50th=[ 1770], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 116, per=25.09%, avg=49.17, stdev=20.20, samples=203 iops : min= 1, max= 29, avg=11.84, stdev= 5.11, samples=203 lat (msec) : 50=0.23%, 100=0.31%, 250=2.27%, 500=36.91%, 750=26.60% lat (msec) : 1000=20.74%, 2000=12.81%, >=2000=0.12% cpu : usr=0.08%, sys=0.85%, ctx=4496, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=188KiB/s (193kB/s), 46.7KiB/s-48.1KiB/s (47.8kB/s-49.3kB/s), io=19.6MiB (20.5MB), run=105177-106532msec WRITE: bw=196KiB/s (201kB/s), 48.0KiB/s-49.4KiB/s (50.2kB/s-50.6kB/s), io=20.4MiB (21.4MB), run=105177-106532msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=143991: Sat Aug 8 03:22:50 2026 read: IOPS=1, BW=62.4MiB/s (65.4MB/s)(40.0MiB/641msec) slat (nsec): min=63268k, max=63268k, avg=63268306.00, stdev= 0.00 clat (nsec): min=522977k, max=522977k, avg=522976875.00, stdev= 0.00 lat (nsec): min=586250k, max=586250k, avg=586249749.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[ 523], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 523], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] lat (msec) : 750=100.00% cpu : usr=0.31%, sys=3.28%, ctx=34, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.4MiB/s (65.4MB/s), 62.4MiB/s-62.4MiB/s (65.4MB/s-65.4MB/s), io=40.0MiB (41.9MB), run=641-641msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout