== sanity test 398c: run fio to test AIO ================= 13:24:48 (1750181088) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.187936 s, 223 MB/s osc.lustre-OST0000-osc-ffff8800a68b0000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=4442: Tue Jun 17 13:26:42 2025 write: IOPS=23, BW=92.1KiB/s (94.3kB/s)(10.0MiB/111173msec) slat (usec): min=12, max=6855, avg=174.20, stdev=429.47 clat (msec): min=88, max=1454, avg=693.72, stdev=150.60 lat (msec): min=89, max=1454, avg=693.89, stdev=150.60 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 472], 10.00th=[ 527], 20.00th=[ 584], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 684], 60.00th=[ 726], | 70.00th=[ 760], 80.00th=[ 802], 90.00th=[ 869], 95.00th=[ 953], | 99.00th=[ 1083], 99.50th=[ 1133], 99.90th=[ 1368], 99.95th=[ 1418], | 99.99th=[ 1452] bw ( KiB/s): min= 7, max= 256, per=25.02%, avg=91.81, stdev=31.29, samples=221 iops : min= 1, max= 64, avg=22.76, stdev= 7.84, samples=221 lat (msec) : 100=0.62%, 250=0.08%, 500=5.78%, 750=60.62%, 1000=29.80% cpu : usr=0.07%, sys=0.54%, ctx=1373, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=4443: Tue Jun 17 13:26:42 2025 write: IOPS=22, BW=91.9KiB/s (94.1kB/s)(10.0MiB/111386msec) slat (usec): min=13, max=7295, avg=162.41, stdev=412.12 clat (msec): min=87, max=1317, avg=694.96, stdev=146.44 lat (msec): min=87, max=1317, avg=695.13, stdev=146.45 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 485], 10.00th=[ 527], 20.00th=[ 584], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 693], 60.00th=[ 726], | 70.00th=[ 760], 80.00th=[ 802], 90.00th=[ 869], 95.00th=[ 944], | 99.00th=[ 1116], 99.50th=[ 1167], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1318] bw ( KiB/s): min= 24, max= 264, per=25.13%, avg=92.24, stdev=27.18, samples=220 iops : min= 6, max= 66, avg=22.85, stdev= 6.83, samples=220 lat (msec) : 100=0.04%, 250=0.62%, 500=5.55%, 750=61.21%, 1000=29.69% cpu : usr=0.11%, sys=0.52%, ctx=1413, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=4444: Tue Jun 17 13:26:42 2025 write: IOPS=22, BW=91.9KiB/s (94.1kB/s)(10.0MiB/111387msec) slat (usec): min=15, max=44944, avg=192.02, stdev=1005.02 clat (msec): min=175, max=1244, avg=694.97, stdev=143.90 lat (msec): min=175, max=1244, avg=695.17, stdev=143.92 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 485], 10.00th=[ 531], 20.00th=[ 584], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 684], 60.00th=[ 718], | 70.00th=[ 751], 80.00th=[ 810], 90.00th=[ 885], 95.00th=[ 953], | 99.00th=[ 1062], 99.50th=[ 1133], 99.90th=[ 1200], 99.95th=[ 1234], | 99.99th=[ 1250] bw ( KiB/s): min= 16, max= 168, per=24.88%, avg=91.32, stdev=25.21, samples=222 iops : min= 4, max= 42, avg=22.55, stdev= 6.34, samples=222 lat (msec) : 250=0.62%, 500=5.23%, 750=62.19%, 1000=29.10% cpu : usr=0.07%, sys=0.61%, ctx=1457, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=4445: Tue Jun 17 13:26:42 2025 write: IOPS=22, BW=91.9KiB/s (94.1kB/s)(10.0MiB/111385msec) slat (usec): min=15, max=8182, avg=182.88, stdev=476.34 clat (msec): min=70, max=1203, avg=695.02, stdev=143.55 lat (msec): min=71, max=1203, avg=695.21, stdev=143.56 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 477], 10.00th=[ 527], 20.00th=[ 584], | 30.00th=[ 625], 40.00th=[ 659], 50.00th=[ 684], 60.00th=[ 718], | 70.00th=[ 760], 80.00th=[ 810], 90.00th=[ 877], 95.00th=[ 953], | 99.00th=[ 1053], 99.50th=[ 1083], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 160, per=24.88%, avg=91.32, stdev=30.49, samples=222 iops : min= 1, max= 40, avg=22.60, stdev= 7.62, samples=222 lat (msec) : 100=0.23%, 250=0.12%, 500=6.25%, 750=60.62%, 1000=30.86% cpu : usr=0.10%, sys=0.55%, ctx=1464, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=368KiB/s (377kB/s), 91.9KiB/s-92.1KiB/s (94.1kB/s-94.3kB/s), io=40.0MiB (41.9MB), run=111173-111387msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4544: Tue Jun 17 13:27:58 2025 read: IOPS=16, BW=67.5KiB/s (69.1kB/s)(5048KiB/74757msec) slat (usec): min=1725, max=24071, avg=6791.72, stdev=3233.30 clat (msec): min=4, max=621, avg=300.14, stdev=67.02 lat (msec): min=9, max=631, avg=306.94, stdev=67.68 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 211], 10.00th=[ 228], 20.00th=[ 249], | 30.00th=[ 266], 40.00th=[ 284], 50.00th=[ 296], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 351], 90.00th=[ 384], 95.00th=[ 414], | 99.00th=[ 514], 99.50th=[ 531], 99.90th=[ 567], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8, max= 160, per=25.03%, avg=67.08, stdev=25.41, samples=149 iops : min= 2, max= 40, avg=16.46, stdev= 6.34, samples=149 write: IOPS=17, BW=69.5KiB/s (71.1kB/s)(5192KiB/74757msec) slat (usec): min=16, max=6395, avg=222.93, stdev=513.40 clat (msec): min=203, max=1071, avg=621.17, stdev=155.71 lat (msec): min=203, max=1071, avg=621.39, stdev=155.72 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 372], 10.00th=[ 422], 20.00th=[ 485], | 30.00th=[ 527], 40.00th=[ 584], 50.00th=[ 625], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 760], 90.00th=[ 827], 95.00th=[ 869], | 99.00th=[ 1003], 99.50th=[ 1020], 99.90th=[ 1062], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 23, max= 128, per=24.64%, avg=68.74, stdev=22.70, samples=149 iops : min= 5, max= 32, avg=16.88, stdev= 5.67, samples=149 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.12%, 250=10.04% lat (msec) : 500=50.90%, 750=28.16%, 1000=10.16% cpu : usr=0.15%, sys=1.54%, ctx=2787, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4545: Tue Jun 17 13:27:58 2025 read: IOPS=16, BW=66.6KiB/s (68.2kB/s)(4976KiB/74757msec) slat (usec): min=1664, max=31387, avg=6852.24, stdev=3226.20 clat (msec): min=8, max=630, avg=300.57, stdev=66.17 lat (msec): min=15, max=639, avg=307.43, stdev=66.80 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 211], 10.00th=[ 226], 20.00th=[ 247], | 30.00th=[ 268], 40.00th=[ 284], 50.00th=[ 296], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 351], 90.00th=[ 388], 95.00th=[ 405], | 99.00th=[ 485], 99.50th=[ 510], 99.90th=[ 617], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 15, max= 144, per=24.63%, avg=66.01, stdev=27.51, samples=149 iops : min= 3, max= 36, avg=16.28, stdev= 6.93, samples=149 write: IOPS=17, BW=70.4KiB/s (72.1kB/s)(5264KiB/74757msec) slat (usec): min=16, max=6790, avg=220.67, stdev=500.71 clat (msec): min=116, max=1064, avg=617.23, stdev=156.12 lat (msec): min=116, max=1064, avg=617.45, stdev=156.14 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 372], 10.00th=[ 418], 20.00th=[ 481], | 30.00th=[ 523], 40.00th=[ 567], 50.00th=[ 617], 60.00th=[ 659], | 70.00th=[ 701], 80.00th=[ 760], 90.00th=[ 827], 95.00th=[ 877], | 99.00th=[ 969], 99.50th=[ 1003], 99.90th=[ 1036], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 16, max= 128, per=25.06%, avg=69.93, stdev=22.99, samples=149 iops : min= 4, max= 32, avg=17.26, stdev= 5.72, samples=149 lat (msec) : 10=0.04%, 50=0.08%, 100=0.27%, 250=10.27%, 500=50.39% lat (msec) : 750=28.09%, 1000=10.59% cpu : usr=0.26%, sys=1.49%, ctx=2780, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4546: Tue Jun 17 13:27:58 2025 read: IOPS=16, BW=66.6KiB/s (68.2kB/s)(4984KiB/74798msec) slat (usec): min=1697, max=79355, avg=6904.04, stdev=3752.72 clat (msec): min=5, max=632, avg=299.73, stdev=70.68 lat (msec): min=12, max=642, avg=306.64, stdev=71.25 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 207], 10.00th=[ 222], 20.00th=[ 249], | 30.00th=[ 266], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 313], | 70.00th=[ 330], 80.00th=[ 347], 90.00th=[ 388], 95.00th=[ 418], | 99.00th=[ 510], 99.50th=[ 531], 99.90th=[ 625], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8, max= 136, per=24.80%, avg=66.46, stdev=26.24, samples=149 iops : min= 2, max= 34, avg=16.40, stdev= 6.59, samples=149 write: IOPS=17, BW=70.3KiB/s (71.0kB/s)(5256KiB/74798msec) slat (usec): min=15, max=4356, avg=221.29, stdev=453.76 clat (msec): min=230, max=1124, avg=618.57, stdev=154.05 lat (msec): min=230, max=1124, avg=618.80, stdev=154.07 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 376], 10.00th=[ 414], 20.00th=[ 481], | 30.00th=[ 527], 40.00th=[ 575], 50.00th=[ 617], 60.00th=[ 659], | 70.00th=[ 709], 80.00th=[ 760], 90.00th=[ 827], 95.00th=[ 869], | 99.00th=[ 953], 99.50th=[ 1020], 99.90th=[ 1116], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 24, max= 127, per=24.95%, avg=69.62, stdev=21.48, samples=149 iops : min= 6, max= 31, avg=17.19, stdev= 5.37, samples=149 lat (msec) : 10=0.12%, 20=0.08%, 50=0.08%, 100=0.16%, 250=10.12% lat (msec) : 500=50.31%, 750=28.28%, 1000=10.59% cpu : usr=0.32%, sys=1.45%, ctx=2826, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4547: Tue Jun 17 13:27:58 2025 read: IOPS=16, BW=67.7KiB/s (69.4kB/s)(5060KiB/74694msec) slat (usec): min=16, max=28272, avg=6760.78, stdev=3290.20 clat (msec): min=4, max=630, avg=300.10, stdev=73.90 lat (msec): min=9, max=641, avg=306.87, stdev=74.53 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 201], 10.00th=[ 224], 20.00th=[ 245], | 30.00th=[ 266], 40.00th=[ 284], 50.00th=[ 296], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 351], 90.00th=[ 388], 95.00th=[ 418], | 99.00th=[ 518], 99.50th=[ 535], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 15, max= 152, per=25.14%, avg=67.36, stdev=26.43, samples=149 iops : min= 3, max= 38, avg=16.58, stdev= 6.67, samples=149 write: IOPS=17, BW=69.3KiB/s (71.0kB/s)(5180KiB/74694msec) slat (usec): min=14, max=7467, avg=212.89, stdev=483.79 clat (msec): min=110, max=1132, avg=622.27, stdev=160.72 lat (msec): min=110, max=1132, avg=622.49, stdev=160.75 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 376], 10.00th=[ 418], 20.00th=[ 477], | 30.00th=[ 535], 40.00th=[ 584], 50.00th=[ 625], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 768], 90.00th=[ 835], 95.00th=[ 877], | 99.00th=[ 969], 99.50th=[ 1003], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 16, max= 160, per=24.59%, avg=68.60, stdev=23.99, samples=149 iops : min= 4, max= 40, avg=16.89, stdev= 6.02, samples=149 lat (msec) : 10=0.04%, 20=0.08%, 50=0.35%, 100=0.16%, 250=10.98% lat (msec) : 500=49.30%, 750=27.73%, 1000=11.09% cpu : usr=0.23%, sys=1.52%, ctx=2796, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=268KiB/s (275kB/s), 66.6KiB/s-67.7KiB/s (68.2kB/s-69.4kB/s), io=19.6MiB (20.5MB), run=74694-74798msec WRITE: bw=279KiB/s (286kB/s), 69.3KiB/s-70.4KiB/s (71.0kB/s-72.1kB/s), io=20.4MiB (21.4MB), run=74694-74798msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=4611: Tue Jun 17 13:28:00 2025 read: IOPS=1, BW=70.7MiB/s (74.1MB/s)(40.0MiB/566msec) slat (nsec): min=15261k, max=15261k, avg=15260650.00, stdev= 0.00 clat (nsec): min=536906k, max=536906k, avg=536905703.00, stdev= 0.00 lat (nsec): min=552213k, max=552213k, avg=552213093.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 750=100.00% cpu : usr=0.53%, sys=1.23%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.7MiB/s (74.1MB/s), 70.7MiB/s-70.7MiB/s (74.1MB/s-74.1MB/s), io=40.0MiB (41.9MB), run=566-566msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout