== sanity test 398c: run fio to test AIO ================= 11:53:52 (1746633232) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.134047 s, 313 MB/s osc.lustre-OST0000-osc-ffff88012b4f3800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=9498: Wed May 7 11:54:54 2025 write: IOPS=42, BW=172KiB/s (176kB/s)(10.0MiB/59599msec) slat (usec): min=16, max=616, avg=97.70, stdev=54.08 clat (msec): min=129, max=631, avg=372.00, stdev=67.53 lat (msec): min=129, max=631, avg=372.10, stdev=67.53 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 359], 50.00th=[ 376], 60.00th=[ 388], | 70.00th=[ 409], 80.00th=[ 426], 90.00th=[ 451], 95.00th=[ 472], | 99.00th=[ 542], 99.50th=[ 558], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 634] bw ( KiB/s): min= 88, max= 256, per=24.88%, avg=170.96, stdev=31.51, samples=119 iops : min= 22, max= 64, avg=42.64, stdev= 7.86, samples=119 lat (msec) : 250=3.36%, 500=94.49%, 750=2.15% cpu : usr=0.10%, sys=0.53%, ctx=1759, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=9499: Wed May 7 11:54:54 2025 write: IOPS=42, BW=172KiB/s (176kB/s)(10.0MiB/59549msec) slat (usec): min=18, max=730, avg=101.34, stdev=53.85 clat (msec): min=118, max=629, avg=371.67, stdev=66.35 lat (msec): min=118, max=629, avg=371.78, stdev=66.35 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 264], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 359], 50.00th=[ 372], 60.00th=[ 388], | 70.00th=[ 409], 80.00th=[ 426], 90.00th=[ 451], 95.00th=[ 472], | 99.00th=[ 523], 99.50th=[ 550], 99.90th=[ 609], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 88, max= 256, per=24.89%, avg=170.99, stdev=29.95, samples=119 iops : min= 22, max= 64, avg=42.67, stdev= 7.49, samples=119 lat (msec) : 250=3.05%, 500=95.16%, 750=1.80% cpu : usr=0.12%, sys=0.53%, ctx=1866, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=9500: Wed May 7 11:54:54 2025 write: IOPS=43, BW=172KiB/s (176kB/s)(10.0MiB/59471msec) slat (usec): min=20, max=604, avg=100.83, stdev=54.78 clat (msec): min=19, max=589, avg=371.30, stdev=69.25 lat (msec): min=19, max=589, avg=371.40, stdev=69.25 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 359], 50.00th=[ 376], 60.00th=[ 393], | 70.00th=[ 409], 80.00th=[ 426], 90.00th=[ 456], 95.00th=[ 472], | 99.00th=[ 527], 99.50th=[ 550], 99.90th=[ 567], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 128, max= 320, per=25.01%, avg=171.84, stdev=32.08, samples=118 iops : min= 32, max= 80, avg=42.89, stdev= 8.02, samples=118 lat (msec) : 20=0.04%, 50=0.12%, 100=0.20%, 250=3.36%, 500=93.98% lat (msec) : 750=2.30% cpu : usr=0.09%, sys=0.55%, ctx=1907, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=9501: Wed May 7 11:54:54 2025 write: IOPS=42, BW=172KiB/s (176kB/s)(10.0MiB/59599msec) slat (usec): min=15, max=697, avg=97.98, stdev=54.39 clat (msec): min=148, max=568, avg=372.06, stdev=65.40 lat (msec): min=148, max=569, avg=372.16, stdev=65.40 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 342], 40.00th=[ 359], 50.00th=[ 372], 60.00th=[ 393], | 70.00th=[ 409], 80.00th=[ 426], 90.00th=[ 451], 95.00th=[ 477], | 99.00th=[ 523], 99.50th=[ 542], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 128, max= 256, per=24.89%, avg=170.99, stdev=30.25, samples=119 iops : min= 32, max= 64, avg=42.67, stdev= 7.60, samples=119 lat (msec) : 250=2.93%, 500=95.43%, 750=1.64% cpu : usr=0.10%, sys=0.53%, ctx=1726, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=687KiB/s (704kB/s), 172KiB/s-172KiB/s (176kB/s-176kB/s), io=40.0MiB (41.9MB), run=59471-59599msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9559: Wed May 7 11:55:22 2025 read: IOPS=45, BW=184KiB/s (188kB/s)(5048KiB/27500msec) slat (usec): min=644, max=4871, avg=1768.97, stdev=564.85 clat (usec): min=1933, max=219998, avg=114604.84, stdev=31835.11 lat (msec): min=3, max=222, avg=116.38, stdev=32.13 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 71], 10.00th=[ 78], 20.00th=[ 88], | 30.00th=[ 96], 40.00th=[ 104], 50.00th=[ 113], 60.00th=[ 122], | 70.00th=[ 131], 80.00th=[ 140], 90.00th=[ 157], 95.00th=[ 171], | 99.00th=[ 205], 99.50th=[ 213], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 88, max= 312, per=25.21%, avg=183.28, stdev=54.61, samples=54 iops : min= 22, max= 78, avg=45.72, stdev=13.64, samples=54 write: IOPS=47, BW=189KiB/s (193kB/s)(5192KiB/27500msec) slat (usec): min=14, max=625, avg=82.01, stdev=44.45 clat (msec): min=92, max=430, avg=225.15, stdev=63.49 lat (msec): min=92, max=430, avg=225.23, stdev=63.50 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 218], 60.00th=[ 234], | 70.00th=[ 259], 80.00th=[ 284], 90.00th=[ 317], 95.00th=[ 338], | 99.00th=[ 376], 99.50th=[ 397], 99.90th=[ 409], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 104, max= 360, per=24.86%, avg=188.17, stdev=49.80, samples=54 iops : min= 26, max= 90, avg=46.94, stdev=12.48, samples=54 lat (msec) : 2=0.04%, 4=0.04%, 20=0.12%, 50=0.16%, 100=17.07% lat (msec) : 250=65.35%, 500=17.23% cpu : usr=0.21%, sys=1.29%, ctx=3464, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9560: Wed May 7 11:55:22 2025 read: IOPS=45, BW=181KiB/s (186kB/s)(4976KiB/27436msec) slat (usec): min=66, max=4916, avg=1761.69, stdev=546.78 clat (msec): min=3, max=222, avg=115.28, stdev=31.88 lat (msec): min=3, max=224, avg=117.04, stdev=32.17 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 89], | 30.00th=[ 97], 40.00th=[ 107], 50.00th=[ 114], 60.00th=[ 123], | 70.00th=[ 132], 80.00th=[ 142], 90.00th=[ 157], 95.00th=[ 169], | 99.00th=[ 201], 99.50th=[ 207], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 72, max= 344, per=24.85%, avg=180.63, stdev=55.33, samples=54 iops : min= 18, max= 86, avg=45.07, stdev=13.84, samples=54 write: IOPS=47, BW=192KiB/s (196kB/s)(5264KiB/27436msec) slat (usec): min=14, max=403, avg=83.25, stdev=44.47 clat (msec): min=35, max=414, avg=222.67, stdev=63.40 lat (msec): min=35, max=415, avg=222.75, stdev=63.41 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 130], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 184], 40.00th=[ 199], 50.00th=[ 213], 60.00th=[ 232], | 70.00th=[ 257], 80.00th=[ 279], 90.00th=[ 313], 95.00th=[ 330], | 99.00th=[ 388], 99.50th=[ 397], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 104, max= 328, per=25.27%, avg=191.30, stdev=50.93, samples=54 iops : min= 26, max= 82, avg=47.74, stdev=12.76, samples=54 lat (msec) : 4=0.08%, 10=0.08%, 20=0.08%, 50=0.51%, 100=16.80% lat (msec) : 250=65.70%, 500=16.76% cpu : usr=0.17%, sys=1.36%, ctx=3342, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9561: Wed May 7 11:55:22 2025 read: IOPS=45, BW=181KiB/s (185kB/s)(4984KiB/27574msec) slat (usec): min=764, max=4474, avg=1791.03, stdev=555.49 clat (msec): min=12, max=213, avg=115.19, stdev=31.15 lat (msec): min=13, max=215, avg=116.98, stdev=31.42 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 71], 10.00th=[ 78], 20.00th=[ 87], | 30.00th=[ 96], 40.00th=[ 105], 50.00th=[ 113], 60.00th=[ 123], | 70.00th=[ 132], 80.00th=[ 142], 90.00th=[ 157], 95.00th=[ 169], | 99.00th=[ 190], 99.50th=[ 201], 99.90th=[ 209], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 80, max= 344, per=24.76%, avg=180.00, stdev=48.91, samples=55 iops : min= 20, max= 86, avg=44.95, stdev=12.25, samples=55 write: IOPS=47, BW=191KiB/s (195kB/s)(5256KiB/27574msec) slat (usec): min=16, max=2571, avg=86.18, stdev=83.16 clat (msec): min=52, max=430, avg=224.31, stdev=63.03 lat (msec): min=53, max=430, avg=224.39, stdev=63.05 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 134], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 218], 60.00th=[ 234], | 70.00th=[ 255], 80.00th=[ 284], 90.00th=[ 317], 95.00th=[ 330], | 99.00th=[ 376], 99.50th=[ 393], 99.90th=[ 409], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 112, max= 304, per=24.95%, avg=188.87, stdev=51.07, samples=55 iops : min= 28, max= 76, avg=47.16, stdev=12.80, samples=55 lat (msec) : 20=0.20%, 50=0.16%, 100=17.19%, 250=66.13%, 500=16.33% cpu : usr=0.17%, sys=1.34%, ctx=3618, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9562: Wed May 7 11:55:22 2025 read: IOPS=46, BW=185KiB/s (189kB/s)(5060KiB/27391msec) slat (usec): min=707, max=3690, avg=1746.39, stdev=520.74 clat (usec): min=1318, max=314217, avg=114210.13, stdev=32534.56 lat (msec): min=2, max=316, avg=115.96, stdev=32.82 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 69], 10.00th=[ 77], 20.00th=[ 86], | 30.00th=[ 95], 40.00th=[ 104], 50.00th=[ 113], 60.00th=[ 122], | 70.00th=[ 130], 80.00th=[ 142], 90.00th=[ 157], 95.00th=[ 169], | 99.00th=[ 201], 99.50th=[ 209], 99.90th=[ 241], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 88, max= 368, per=25.59%, avg=186.02, stdev=54.63, samples=54 iops : min= 22, max= 92, avg=46.46, stdev=13.68, samples=54 write: IOPS=47, BW=189KiB/s (194kB/s)(5180KiB/27391msec) slat (usec): min=14, max=356, avg=84.15, stdev=44.53 clat (msec): min=92, max=424, avg=224.59, stdev=62.42 lat (msec): min=92, max=424, avg=224.68, stdev=62.43 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 132], 10.00th=[ 150], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 255], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 334], | 99.00th=[ 388], 99.50th=[ 405], 99.90th=[ 422], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 112, max= 288, per=24.91%, avg=188.54, stdev=45.93, samples=54 iops : min= 28, max= 72, avg=47.09, stdev=11.50, samples=54 lat (msec) : 2=0.04%, 20=0.16%, 50=0.23%, 100=17.70%, 250=65.35% lat (msec) : 500=16.52% cpu : usr=0.16%, sys=1.35%, ctx=3718, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=728KiB/s (745kB/s), 181KiB/s-185KiB/s (185kB/s-189kB/s), io=19.6MiB (20.5MB), run=27391-27574msec WRITE: bw=758KiB/s (776kB/s), 189KiB/s-192KiB/s (193kB/s-196kB/s), io=20.4MiB (21.4MB), run=27391-27574msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=9587: Wed May 7 11:55:24 2025 read: IOPS=6, BW=268MiB/s (281MB/s)(40.0MiB/149msec) slat (nsec): min=6267.3k, max=6267.3k, avg=6267326.00, stdev= 0.00 clat (nsec): min=136287k, max=136287k, avg=136287152.00, stdev= 0.00 lat (nsec): min=142564k, max=142564k, avg=142563736.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.05%, ctx=6, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=40.0MiB (41.9MB), run=149-149msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout