== sanity test 398c: run fio to test AIO ================= 11:26:18 (1754407578) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.145836 s, 288 MB/s osc.lustre-OST0000-osc-ffff8a69c9231800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148146: Tue Aug 5 11:27:06 2025 write: IOPS=58, BW=234KiB/s (240kB/s)(10.0MiB/43698msec); 0 zone resets slat (usec): min=28, max=1634, avg=162.74, stdev=99.85 clat (msec): min=56, max=573, avg=272.65, stdev=68.66 lat (msec): min=56, max=573, avg=272.82, stdev=68.67 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 161], 10.00th=[ 190], 20.00th=[ 218], | 30.00th=[ 236], 40.00th=[ 253], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 359], 95.00th=[ 388], | 99.00th=[ 477], 99.50th=[ 506], 99.90th=[ 531], 99.95th=[ 535], | 99.99th=[ 575] bw ( KiB/s): min= 136, max= 412, per=24.87%, avg=232.79, stdev=54.37, samples=87 iops : min= 34, max= 103, avg=58.14, stdev=13.65, samples=87 lat (msec) : 100=0.08%, 250=38.59%, 500=60.66%, 750=0.66% cpu : usr=0.14%, sys=0.89%, ctx=1984, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148147: Tue Aug 5 11:27:06 2025 write: IOPS=58, BW=234KiB/s (240kB/s)(10.0MiB/43719msec); 0 zone resets slat (usec): min=28, max=1968, avg=160.04, stdev=109.12 clat (msec): min=89, max=601, avg=272.78, stdev=68.32 lat (msec): min=89, max=603, avg=272.94, stdev=68.34 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 163], 10.00th=[ 190], 20.00th=[ 220], | 30.00th=[ 236], 40.00th=[ 253], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 359], 95.00th=[ 388], | 99.00th=[ 481], 99.50th=[ 514], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 600] bw ( KiB/s): min= 128, max= 412, per=24.87%, avg=232.79, stdev=54.87, samples=87 iops : min= 32, max= 103, avg=58.14, stdev=13.77, samples=87 lat (msec) : 100=0.04%, 250=38.01%, 500=61.29%, 750=0.66% cpu : usr=0.13%, sys=0.87%, ctx=1894, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148148: Tue Aug 5 11:27:06 2025 write: IOPS=58, BW=235KiB/s (241kB/s)(10.0MiB/43490msec); 0 zone resets slat (usec): min=26, max=3370, avg=166.09, stdev=116.29 clat (msec): min=15, max=574, avg=271.49, stdev=71.09 lat (msec): min=15, max=574, avg=271.66, stdev=71.11 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 157], 10.00th=[ 188], 20.00th=[ 215], | 30.00th=[ 234], 40.00th=[ 253], 50.00th=[ 268], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 359], 95.00th=[ 388], | 99.00th=[ 485], 99.50th=[ 527], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 575] bw ( KiB/s): min= 136, max= 489, per=25.11%, avg=235.01, stdev=58.41, samples=86 iops : min= 34, max= 122, avg=58.70, stdev=14.64, samples=86 lat (msec) : 20=0.04%, 50=0.04%, 100=0.51%, 250=38.36%, 500=60.39% lat (msec) : 750=0.66% cpu : usr=0.14%, sys=0.92%, ctx=2064, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148149: Tue Aug 5 11:27:06 2025 write: IOPS=58, BW=234KiB/s (240kB/s)(10.0MiB/43719msec); 0 zone resets slat (usec): min=26, max=2481, avg=154.15, stdev=108.78 clat (msec): min=84, max=600, avg=272.84, stdev=69.02 lat (msec): min=84, max=600, avg=272.99, stdev=69.03 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 159], 10.00th=[ 188], 20.00th=[ 220], | 30.00th=[ 239], 40.00th=[ 255], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 393], | 99.00th=[ 477], 99.50th=[ 498], 99.90th=[ 535], 99.95th=[ 542], | 99.99th=[ 600] bw ( KiB/s): min= 144, max= 412, per=24.87%, avg=232.79, stdev=56.61, samples=87 iops : min= 36, max= 103, avg=58.14, stdev=14.22, samples=87 lat (msec) : 100=0.20%, 250=37.46%, 500=61.95%, 750=0.39% cpu : usr=0.15%, sys=0.81%, ctx=1791, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=937KiB/s (959kB/s), 234KiB/s-235KiB/s (240kB/s-241kB/s), io=40.0MiB (41.9MB), run=43490-43719msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148165: Tue Aug 5 11:27:40 2025 read: IOPS=41, BW=168KiB/s (172kB/s)(5048KiB/30120msec) slat (usec): min=1271, max=10214, avg=3630.74, stdev=1222.58 clat (msec): min=19, max=359, avg=123.80, stdev=27.93 lat (msec): min=21, max=362, avg=127.43, stdev=28.08 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 88], 10.00th=[ 94], 20.00th=[ 104], | 30.00th=[ 110], 40.00th=[ 115], 50.00th=[ 122], 60.00th=[ 128], | 70.00th=[ 136], 80.00th=[ 144], 90.00th=[ 155], 95.00th=[ 165], | 99.00th=[ 201], 99.50th=[ 259], 99.90th=[ 334], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 96, max= 264, per=25.30%, avg=168.22, stdev=34.59, samples=59 iops : min= 24, max= 66, avg=41.93, stdev= 8.64, samples=59 write: IOPS=43, BW=172KiB/s (177kB/s)(5192KiB/30120msec); 0 zone resets slat (usec): min=35, max=1721, avg=153.82, stdev=100.09 clat (msec): min=104, max=468, avg=245.84, stdev=56.15 lat (msec): min=104, max=468, avg=246.00, stdev=56.15 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 197], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 243], 60.00th=[ 259], | 70.00th=[ 279], 80.00th=[ 296], 90.00th=[ 321], 95.00th=[ 338], | 99.00th=[ 372], 99.50th=[ 393], 99.90th=[ 456], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 104, max= 248, per=24.93%, avg=172.51, stdev=33.68, samples=59 iops : min= 26, max= 62, avg=43.00, stdev= 8.47, samples=59 lat (msec) : 20=0.04%, 50=0.27%, 100=7.54%, 250=68.91%, 500=23.24% cpu : usr=0.28%, sys=1.75%, ctx=2894, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148166: Tue Aug 5 11:27:40 2025 read: IOPS=41, BW=165KiB/s (169kB/s)(4976KiB/30172msec) slat (usec): min=1372, max=10439, avg=3588.07, stdev=1188.54 clat (msec): min=5, max=274, avg=122.59, stdev=26.37 lat (msec): min=9, max=277, avg=126.18, stdev=26.52 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 87], 10.00th=[ 94], 20.00th=[ 104], | 30.00th=[ 110], 40.00th=[ 115], 50.00th=[ 122], 60.00th=[ 128], | 70.00th=[ 134], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 165], | 99.00th=[ 199], 99.50th=[ 205], 99.90th=[ 230], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 96, max= 272, per=24.72%, avg=164.38, stdev=42.53, samples=60 iops : min= 24, max= 68, avg=40.98, stdev=10.61, samples=60 write: IOPS=43, BW=174KiB/s (179kB/s)(5264KiB/30172msec); 0 zone resets slat (usec): min=27, max=2080, avg=157.95, stdev=117.82 clat (msec): min=101, max=468, avg=246.48, stdev=57.99 lat (msec): min=102, max=468, avg=246.64, stdev=58.00 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 194], | 30.00th=[ 213], 40.00th=[ 230], 50.00th=[ 245], 60.00th=[ 262], | 70.00th=[ 279], 80.00th=[ 300], 90.00th=[ 326], 95.00th=[ 338], | 99.00th=[ 384], 99.50th=[ 418], 99.90th=[ 443], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 88, max= 256, per=25.07%, avg=173.47, stdev=32.74, samples=60 iops : min= 22, max= 64, avg=43.25, stdev= 8.23, samples=60 lat (msec) : 10=0.12%, 50=0.23%, 100=8.16%, 250=67.66%, 500=23.83% cpu : usr=0.25%, sys=1.76%, ctx=2944, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148167: Tue Aug 5 11:27:40 2025 read: IOPS=41, BW=165KiB/s (169kB/s)(4984KiB/30173msec) slat (usec): min=30, max=8953, avg=3564.60, stdev=1222.35 clat (msec): min=9, max=207, avg=123.22, stdev=24.85 lat (msec): min=14, max=212, avg=126.79, stdev=25.08 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 88], 10.00th=[ 94], 20.00th=[ 104], | 30.00th=[ 109], 40.00th=[ 116], 50.00th=[ 122], 60.00th=[ 128], | 70.00th=[ 134], 80.00th=[ 144], 90.00th=[ 155], 95.00th=[ 165], | 99.00th=[ 197], 99.50th=[ 201], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 72, max= 328, per=24.85%, avg=165.23, stdev=46.46, samples=60 iops : min= 18, max= 82, avg=41.18, stdev=11.64, samples=60 write: IOPS=43, BW=174KiB/s (178kB/s)(5256KiB/30173msec); 0 zone resets slat (usec): min=25, max=1852, avg=155.63, stdev=119.16 clat (msec): min=69, max=468, avg=246.64, stdev=58.23 lat (msec): min=69, max=468, avg=246.80, stdev=58.23 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 150], 10.00th=[ 171], 20.00th=[ 197], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 247], 60.00th=[ 262], | 70.00th=[ 279], 80.00th=[ 300], 90.00th=[ 321], 95.00th=[ 338], | 99.00th=[ 376], 99.50th=[ 397], 99.90th=[ 439], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 104, max= 240, per=24.92%, avg=172.47, stdev=33.45, samples=60 iops : min= 26, max= 60, avg=43.00, stdev= 8.35, samples=60 lat (msec) : 10=0.04%, 50=0.12%, 100=8.01%, 250=66.99%, 500=24.84% cpu : usr=0.28%, sys=1.69%, ctx=2910, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148168: Tue Aug 5 11:27:40 2025 read: IOPS=42, BW=168KiB/s (172kB/s)(5060KiB/30040msec) slat (usec): min=1406, max=9690, avg=3578.65, stdev=1161.87 clat (msec): min=14, max=371, avg=123.48, stdev=31.10 lat (msec): min=18, max=375, avg=127.07, stdev=31.22 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 85], 10.00th=[ 92], 20.00th=[ 103], | 30.00th=[ 110], 40.00th=[ 116], 50.00th=[ 122], 60.00th=[ 128], | 70.00th=[ 136], 80.00th=[ 144], 90.00th=[ 155], 95.00th=[ 167], | 99.00th=[ 205], 99.50th=[ 317], 99.90th=[ 342], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 72, max= 272, per=25.51%, avg=169.63, stdev=41.50, samples=59 iops : min= 18, max= 68, avg=42.29, stdev=10.38, samples=59 write: IOPS=43, BW=172KiB/s (177kB/s)(5180KiB/30040msec); 0 zone resets slat (usec): min=31, max=1751, avg=155.20, stdev=110.91 clat (msec): min=74, max=448, avg=245.87, stdev=57.70 lat (msec): min=74, max=448, avg=246.03, stdev=57.70 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 155], 10.00th=[ 171], 20.00th=[ 197], | 30.00th=[ 213], 40.00th=[ 228], 50.00th=[ 245], 60.00th=[ 259], | 70.00th=[ 279], 80.00th=[ 300], 90.00th=[ 321], 95.00th=[ 338], | 99.00th=[ 380], 99.50th=[ 397], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 96, max= 280, per=24.85%, avg=171.93, stdev=34.10, samples=59 iops : min= 24, max= 70, avg=42.86, stdev= 8.57, samples=59 lat (msec) : 20=0.04%, 50=0.43%, 100=8.71%, 250=67.77%, 500=23.05% cpu : usr=0.33%, sys=1.72%, ctx=3004, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=665KiB/s (681kB/s), 165KiB/s-168KiB/s (169kB/s-172kB/s), io=19.6MiB (20.5MB), run=30040-30173msec WRITE: bw=692KiB/s (709kB/s), 172KiB/s-174KiB/s (177kB/s-179kB/s), io=20.4MiB (21.4MB), run=30040-30173msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148175: Tue Aug 5 11:27:44 2025 read: IOPS=10, BW=421MiB/s (442MB/s)(40.0MiB/95msec) slat (nsec): min=14809k, max=14809k, avg=14809055.00, stdev= 0.00 clat (nsec): min=74701k, max=74701k, avg=74701486.00, stdev= 0.00 lat (nsec): min=89515k, max=89515k, avg=89515007.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[74974], 5.00th=[74974], 10.00th=[74974], 20.00th=[74974], | 30.00th=[74974], 40.00th=[74974], 50.00th=[74974], 60.00th=[74974], | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=15.96%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=421MiB/s (442MB/s), 421MiB/s-421MiB/s (442MB/s-442MB/s), io=40.0MiB (41.9MB), run=95-95msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout