== sanity test 398c: run fio to test AIO ================= 12:36:38 (1787848598) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.488554 s, 85.9 MB/s osc.lustre-OST0000-osc-ffff89e650b85000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146857: Thu Aug 27 12:39:22 2026 write: IOPS=16, BW=66.8KiB/s (68.4kB/s)(10.0MiB/153359msec); 0 zone resets slat (usec): min=23, max=12323, avg=205.51, stdev=728.82 clat (msec): min=95, max=2090, avg=957.78, stdev=237.97 lat (msec): min=96, max=2090, avg=957.99, stdev=238.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1620], 99.50th=[ 1670], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 175, per=24.94%, avg=66.35, stdev=26.72, samples=304 iops : min= 1, max= 43, avg=16.15, stdev= 6.69, samples=304 lat (msec) : 100=0.12%, 250=0.47%, 500=1.80%, 750=15.12%, 1000=41.88% lat (msec) : 2000=40.59%, >=2000=0.04% cpu : usr=0.03%, sys=0.21%, ctx=2051, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146858: Thu Aug 27 12:39:22 2026 write: IOPS=16, BW=66.6KiB/s (68.2kB/s)(10.0MiB/153783msec); 0 zone resets slat (usec): min=23, max=19760, avg=211.28, stdev=933.32 clat (msec): min=207, max=2094, avg=959.99, stdev=229.88 lat (msec): min=207, max=2094, avg=960.21, stdev=229.89 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 609], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1070], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1536], 99.50th=[ 1653], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=24.87%, avg=66.14, stdev=26.12, samples=305 iops : min= 1, max= 32, avg=16.10, stdev= 6.54, samples=305 lat (msec) : 250=0.62%, 500=1.02%, 750=14.73%, 1000=42.30%, 2000=41.29% lat (msec) : >=2000=0.04% cpu : usr=0.04%, sys=0.20%, ctx=1907, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146859: Thu Aug 27 12:39:22 2026 write: IOPS=16, BW=66.7KiB/s (68.3kB/s)(10.0MiB/153603msec); 0 zone resets slat (usec): min=26, max=15925, avg=181.82, stdev=710.60 clat (msec): min=200, max=2034, avg=958.79, stdev=234.38 lat (msec): min=200, max=2036, avg=958.97, stdev=234.41 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 617], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1070], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1620], 99.50th=[ 1636], 99.90th=[ 1921], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 127, per=24.86%, avg=66.14, stdev=25.48, samples=305 iops : min= 1, max= 31, avg=16.10, stdev= 6.35, samples=305 lat (msec) : 250=0.62%, 500=1.91%, 750=15.04%, 1000=40.98%, 2000=41.37% lat (msec) : >=2000=0.08% cpu : usr=0.05%, sys=0.19%, ctx=1845, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146860: Thu Aug 27 12:39:22 2026 write: IOPS=16, BW=66.6KiB/s (68.2kB/s)(10.0MiB/153860msec); 0 zone resets slat (usec): min=28, max=29443, avg=222.42, stdev=1037.81 clat (msec): min=83, max=1842, avg=960.29, stdev=239.54 lat (msec): min=84, max=1842, avg=960.52, stdev=239.57 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 584], 10.00th=[ 684], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1586], 99.50th=[ 1670], 99.90th=[ 1770], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 143, per=25.03%, avg=66.57, stdev=29.22, samples=303 iops : min= 1, max= 35, avg=16.21, stdev= 7.32, samples=303 lat (msec) : 100=0.04%, 250=0.62%, 500=1.84%, 750=15.08%, 1000=39.10% lat (msec) : 2000=43.32% cpu : usr=0.04%, sys=0.21%, ctx=2033, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=266KiB/s (273kB/s), 66.6KiB/s-66.8KiB/s (68.2kB/s-68.4kB/s), io=40.0MiB (41.9MB), run=153359-153860msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146892: Thu Aug 27 12:41:18 2026 read: IOPS=11, BW=47.1KiB/s (48.2kB/s)(5048KiB/107205msec) slat (usec): min=2558, max=74756, avg=13818.04, stdev=8333.15 clat (msec): min=32, max=1136, avg=427.52, stdev=133.50 lat (msec): min=45, max=1146, avg=441.35, stdev=135.61 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 253], 10.00th=[ 284], 20.00th=[ 321], | 30.00th=[ 355], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 468], 80.00th=[ 514], 90.00th=[ 592], 95.00th=[ 676], | 99.00th=[ 885], 99.50th=[ 1003], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 135, per=25.22%, avg=46.91, stdev=22.13, samples=211 iops : min= 1, max= 33, avg=11.27, stdev= 5.58, samples=211 write: IOPS=12, BW=48.4KiB/s (49.6kB/s)(5192KiB/107205msec); 0 zone resets slat (usec): min=31, max=28394, avg=271.34, stdev=1125.06 clat (msec): min=290, max=1918, avg=889.77, stdev=293.39 lat (msec): min=290, max=1919, avg=890.04, stdev=293.36 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 485], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 768], 50.00th=[ 844], 60.00th=[ 919], | 70.00th=[ 1003], 80.00th=[ 1116], 90.00th=[ 1301], 95.00th=[ 1469], | 99.00th=[ 1754], 99.50th=[ 1821], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 104, per=24.72%, avg=47.96, stdev=19.85, samples=212 iops : min= 1, max= 26, avg=11.52, stdev= 5.05, samples=212 lat (msec) : 50=0.04%, 100=0.16%, 250=1.99%, 500=38.71%, 750=25.94% lat (msec) : 1000=17.73%, 2000=15.43% cpu : usr=0.08%, sys=0.53%, ctx=4736, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146893: Thu Aug 27 12:41:18 2026 read: IOPS=11, BW=46.3KiB/s (47.4kB/s)(4976KiB/107433msec) slat (usec): min=2696, max=61506, avg=13830.38, stdev=8252.72 clat (msec): min=35, max=1154, avg=427.38, stdev=142.11 lat (msec): min=39, max=1159, avg=441.22, stdev=143.91 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 247], 10.00th=[ 275], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 380], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 514], 90.00th=[ 617], 95.00th=[ 709], | 99.00th=[ 860], 99.50th=[ 986], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 128, per=24.56%, avg=45.68, stdev=23.36, samples=212 iops : min= 1, max= 32, avg=10.96, stdev= 5.95, samples=212 write: IOPS=12, BW=48.0KiB/s (50.2kB/s)(5264KiB/107433msec); 0 zone resets slat (usec): min=30, max=19504, avg=275.50, stdev=1081.54 clat (msec): min=41, max=1959, avg=887.24, stdev=290.15 lat (msec): min=41, max=1959, avg=887.52, stdev=290.13 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 481], 10.00th=[ 567], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 995], 80.00th=[ 1099], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1687], 99.50th=[ 1787], 99.90th=[ 1905], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 104, per=24.90%, avg=48.31, stdev=21.80, samples=212 iops : min= 1, max= 26, avg=11.61, stdev= 5.51, samples=212 lat (msec) : 50=0.16%, 100=0.20%, 250=2.66%, 500=37.97%, 750=23.63% lat (msec) : 1000=20.20%, 2000=15.20% cpu : usr=0.07%, sys=0.51%, ctx=4292, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146894: Thu Aug 27 12:41:18 2026 read: IOPS=11, BW=46.4KiB/s (47.6kB/s)(4984KiB/107305msec) slat (usec): min=2249, max=57079, avg=14062.02, stdev=8394.39 clat (msec): min=10, max=986, avg=424.18, stdev=130.29 lat (msec): min=17, max=999, avg=438.26, stdev=132.43 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 257], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 430], | 70.00th=[ 460], 80.00th=[ 514], 90.00th=[ 592], 95.00th=[ 684], | 99.00th=[ 844], 99.50th=[ 902], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 120, per=24.99%, avg=46.48, stdev=22.12, samples=211 iops : min= 1, max= 30, avg=11.15, stdev= 5.61, samples=211 write: IOPS=12, BW=48.0KiB/s (50.2kB/s)(5256KiB/107305msec); 0 zone resets slat (usec): min=29, max=21065, avg=281.20, stdev=1032.93 clat (msec): min=231, max=1961, avg=888.77, stdev=292.96 lat (msec): min=231, max=1961, avg=889.05, stdev=292.95 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 489], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 835], 60.00th=[ 902], | 70.00th=[ 986], 80.00th=[ 1116], 90.00th=[ 1318], 95.00th=[ 1469], | 99.00th=[ 1754], 99.50th=[ 1838], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 112, per=24.95%, avg=48.40, stdev=19.53, samples=212 iops : min= 1, max= 28, avg=11.63, stdev= 4.94, samples=212 lat (msec) : 20=0.04%, 100=0.20%, 250=1.80%, 500=39.02%, 750=24.49% lat (msec) : 1000=19.73%, 2000=14.73% cpu : usr=0.08%, sys=0.55%, ctx=4761, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146895: Thu Aug 27 12:41:18 2026 read: IOPS=11, BW=47.5KiB/s (48.6kB/s)(5060KiB/106545msec) slat (usec): min=69, max=60780, avg=13905.17, stdev=8379.98 clat (msec): min=6, max=1145, avg=424.11, stdev=138.88 lat (msec): min=10, max=1159, avg=438.02, stdev=141.25 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 241], 10.00th=[ 275], 20.00th=[ 317], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 472], 80.00th=[ 518], 90.00th=[ 600], 95.00th=[ 684], | 99.00th=[ 852], 99.50th=[ 927], 99.90th=[ 1028], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 126, per=25.54%, avg=47.50, stdev=21.71, samples=209 iops : min= 1, max= 31, avg=11.41, stdev= 5.47, samples=209 write: IOPS=12, BW=48.6KiB/s (49.8kB/s)(5180KiB/106545msec); 0 zone resets slat (usec): min=32, max=20643, avg=262.89, stdev=950.36 clat (msec): min=64, max=1988, avg=887.25, stdev=302.28 lat (msec): min=64, max=1988, avg=887.52, stdev=302.32 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 468], 10.00th=[ 550], 20.00th=[ 642], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 835], 60.00th=[ 911], | 70.00th=[ 1011], 80.00th=[ 1133], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1754], 99.50th=[ 1854], 99.90th=[ 1938], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 136, per=24.99%, avg=48.48, stdev=22.03, samples=209 iops : min= 1, max= 34, avg=11.64, stdev= 5.59, samples=209 lat (msec) : 10=0.04%, 20=0.20%, 50=0.12%, 100=0.20%, 250=3.09% lat (msec) : 500=37.62%, 750=25.59%, 1000=17.58%, 2000=15.59% cpu : usr=0.09%, sys=0.54%, ctx=4717, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=187KiB/s (191kB/s), 46.3KiB/s-47.5KiB/s (47.4kB/s-48.6kB/s), io=19.6MiB (20.5MB), run=106545-107433msec WRITE: bw=194KiB/s (199kB/s), 48.4KiB/s-48.0KiB/s (49.6kB/s-50.2kB/s), io=20.4MiB (21.4MB), run=106545-107433msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146914: Thu Aug 27 12:41:27 2026 read: IOPS=2, BW=92.4MiB/s (96.9MB/s)(40.0MiB/433msec) slat (nsec): min=25345k, max=25345k, avg=25345096.00, stdev= 0.00 clat (nsec): min=381627k, max=381627k, avg=381626912.00, stdev= 0.00 lat (nsec): min=406985k, max=406985k, avg=406985065.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.78%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=92.4MiB/s (96.9MB/s), 92.4MiB/s-92.4MiB/s (96.9MB/s-96.9MB/s), io=40.0MiB (41.9MB), run=433-433msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout