== sanity test 398c: run fio to test AIO ================= 14:19:04 (1752689944) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.134149 s, 313 MB/s osc.lustre-OST0000-osc-ffff956fc8368000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=156170: Wed Jul 16 14:19:42 2025 write: IOPS=72, BW=292KiB/s (299kB/s)(10.0MiB/35071msec); 0 zone resets slat (usec): min=31, max=503, avg=125.95, stdev=61.57 clat (msec): min=49, max=383, avg=218.85, stdev=43.66 lat (msec): min=49, max=384, avg=218.97, stdev=43.67 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 144], 10.00th=[ 171], 20.00th=[ 190], | 30.00th=[ 201], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 334], 99.50th=[ 351], 99.90th=[ 380], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 208, max= 510, per=24.88%, avg=290.41, stdev=46.68, samples=69 iops : min= 52, max= 127, avg=72.59, stdev=11.63, samples=69 lat (msec) : 50=0.16%, 100=0.59%, 250=79.18%, 500=20.08% cpu : usr=0.16%, sys=0.89%, ctx=1966, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=156171: Wed Jul 16 14:19:42 2025 write: IOPS=73, BW=292KiB/s (299kB/s)(10.0MiB/35034msec); 0 zone resets slat (usec): min=27, max=589, avg=124.16, stdev=63.47 clat (msec): min=30, max=383, avg=218.71, stdev=44.42 lat (msec): min=31, max=383, avg=218.83, stdev=44.42 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 144], 10.00th=[ 171], 20.00th=[ 190], | 30.00th=[ 199], 40.00th=[ 211], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 334], 99.50th=[ 351], 99.90th=[ 368], 99.95th=[ 376], | 99.99th=[ 384] bw ( KiB/s): min= 200, max= 542, per=24.92%, avg=290.87, stdev=49.07, samples=69 iops : min= 50, max= 135, avg=72.71, stdev=12.23, samples=69 lat (msec) : 50=0.20%, 100=1.45%, 250=77.70%, 500=20.66% cpu : usr=0.17%, sys=0.85%, ctx=1897, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=156172: Wed Jul 16 14:19:42 2025 write: IOPS=73, BW=292KiB/s (299kB/s)(10.0MiB/35062msec); 0 zone resets slat (usec): min=26, max=548, avg=126.49, stdev=61.49 clat (msec): min=74, max=375, avg=218.74, stdev=43.08 lat (msec): min=74, max=375, avg=218.87, stdev=43.09 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 148], 10.00th=[ 169], 20.00th=[ 190], | 30.00th=[ 199], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 376] bw ( KiB/s): min= 200, max= 510, per=24.89%, avg=290.52, stdev=47.16, samples=69 iops : min= 50, max= 127, avg=72.62, stdev=11.76, samples=69 lat (msec) : 100=1.37%, 250=77.70%, 500=20.94% cpu : usr=0.13%, sys=0.93%, ctx=1962, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=156173: Wed Jul 16 14:19:42 2025 write: IOPS=73, BW=292KiB/s (299kB/s)(10.0MiB/35062msec); 0 zone resets slat (usec): min=25, max=512, avg=123.45, stdev=62.43 clat (msec): min=49, max=383, avg=218.78, stdev=42.97 lat (msec): min=49, max=383, avg=218.90, stdev=42.98 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 146], 10.00th=[ 169], 20.00th=[ 190], | 30.00th=[ 201], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 384] bw ( KiB/s): min= 200, max= 510, per=24.86%, avg=290.17, stdev=47.49, samples=69 iops : min= 50, max= 127, avg=72.54, stdev=11.84, samples=69 lat (msec) : 50=0.16%, 100=1.17%, 250=78.36%, 500=20.31% cpu : usr=0.15%, sys=0.88%, ctx=1885, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1168KiB/s (1196kB/s), 292KiB/s-292KiB/s (299kB/s-299kB/s), io=40.0MiB (41.9MB), run=35034-35071msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=156187: Wed Jul 16 14:20:09 2025 read: IOPS=51, BW=208KiB/s (213kB/s)(5048KiB/24290msec) slat (usec): min=1051, max=7442, avg=2924.98, stdev=1081.43 clat (msec): min=4, max=250, avg=97.29, stdev=21.06 lat (msec): min=6, max=253, avg=100.22, stdev=21.30 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 67], 10.00th=[ 72], 20.00th=[ 81], | 30.00th=[ 88], 40.00th=[ 93], 50.00th=[ 99], 60.00th=[ 103], | 70.00th=[ 107], 80.00th=[ 113], 90.00th=[ 122], 95.00th=[ 128], | 99.00th=[ 144], 99.50th=[ 153], 99.90th=[ 249], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 120, max= 328, per=25.30%, avg=208.50, stdev=44.33, samples=48 iops : min= 30, max= 82, avg=52.10, stdev=11.10, samples=48 write: IOPS=53, BW=214KiB/s (219kB/s)(5192KiB/24290msec); 0 zone resets slat (usec): min=27, max=813, avg=122.41, stdev=66.41 clat (msec): min=86, max=375, avg=201.05, stdev=43.73 lat (msec): min=86, max=375, avg=201.18, stdev=43.73 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 239], 90.00th=[ 257], 95.00th=[ 271], | 99.00th=[ 296], 99.50th=[ 317], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 152, max= 272, per=24.91%, avg=213.75, stdev=30.57, samples=48 iops : min= 38, max= 68, avg=53.42, stdev= 7.67, samples=48 lat (msec) : 10=0.04%, 20=0.16%, 50=0.16%, 100=26.29%, 250=66.68% lat (msec) : 500=6.68% cpu : usr=0.28%, sys=1.74%, ctx=3099, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=156188: Wed Jul 16 14:20:09 2025 read: IOPS=51, BW=205KiB/s (209kB/s)(4976KiB/24330msec) slat (usec): min=1104, max=8616, avg=2894.18, stdev=1082.84 clat (msec): min=13, max=155, avg=97.11, stdev=19.34 lat (msec): min=16, max=159, avg=100.00, stdev=19.58 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 82], | 30.00th=[ 88], 40.00th=[ 93], 50.00th=[ 97], 60.00th=[ 103], | 70.00th=[ 108], 80.00th=[ 114], 90.00th=[ 122], 95.00th=[ 127], | 99.00th=[ 142], 99.50th=[ 146], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 104, max= 320, per=24.83%, avg=204.58, stdev=43.85, samples=48 iops : min= 26, max= 80, avg=51.13, stdev=10.99, samples=48 write: IOPS=54, BW=216KiB/s (222kB/s)(5264KiB/24330msec); 0 zone resets slat (usec): min=28, max=1244, avg=121.91, stdev=72.39 clat (msec): min=60, max=328, avg=200.68, stdev=44.54 lat (msec): min=60, max=328, avg=200.81, stdev=44.54 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 124], 10.00th=[ 142], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 226], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 271], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 313], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 152, max= 328, per=25.15%, avg=215.75, stdev=39.02, samples=48 iops : min= 38, max= 82, avg=53.92, stdev= 9.76, samples=48 lat (msec) : 20=0.20%, 50=0.23%, 100=26.25%, 250=66.37%, 500=6.95% cpu : usr=0.21%, sys=1.80%, ctx=3070, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=156189: Wed Jul 16 14:20:09 2025 read: IOPS=51, BW=205KiB/s (210kB/s)(4984KiB/24290msec) slat (usec): min=35, max=7213, avg=2898.01, stdev=1075.00 clat (msec): min=4, max=231, avg=96.73, stdev=20.96 lat (msec): min=5, max=236, avg=99.63, stdev=21.27 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 87], 40.00th=[ 93], 50.00th=[ 99], 60.00th=[ 103], | 70.00th=[ 108], 80.00th=[ 113], 90.00th=[ 122], 95.00th=[ 129], | 99.00th=[ 144], 99.50th=[ 146], 99.90th=[ 157], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 120, max= 336, per=25.03%, avg=206.25, stdev=50.20, samples=48 iops : min= 30, max= 84, avg=51.54, stdev=12.55, samples=48 write: IOPS=54, BW=216KiB/s (222kB/s)(5256KiB/24290msec); 0 zone resets slat (usec): min=31, max=550, avg=120.54, stdev=63.81 clat (msec): min=28, max=393, avg=200.90, stdev=46.44 lat (msec): min=28, max=393, avg=201.03, stdev=46.45 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 123], 10.00th=[ 142], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 226], 80.00th=[ 241], 90.00th=[ 257], 95.00th=[ 271], | 99.00th=[ 296], 99.50th=[ 317], 99.90th=[ 376], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 152, max= 312, per=25.10%, avg=215.35, stdev=34.21, samples=48 iops : min= 38, max= 78, avg=53.81, stdev= 8.57, samples=48 lat (msec) : 10=0.12%, 20=0.31%, 50=0.43%, 100=26.60%, 250=65.16% lat (msec) : 500=7.38% cpu : usr=0.21%, sys=1.79%, ctx=3020, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=156190: Wed Jul 16 14:20:09 2025 read: IOPS=52, BW=208KiB/s (213kB/s)(5060KiB/24312msec) slat (usec): min=1019, max=7131, avg=2882.78, stdev=1060.78 clat (msec): min=4, max=252, avg=96.89, stdev=20.27 lat (msec): min=6, max=257, avg=99.78, stdev=20.54 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 67], 10.00th=[ 74], 20.00th=[ 81], | 30.00th=[ 87], 40.00th=[ 93], 50.00th=[ 99], 60.00th=[ 103], | 70.00th=[ 107], 80.00th=[ 113], 90.00th=[ 121], 95.00th=[ 128], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 224], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 120, max= 312, per=25.39%, avg=209.19, stdev=44.15, samples=48 iops : min= 30, max= 78, avg=52.27, stdev=11.06, samples=48 write: IOPS=53, BW=213KiB/s (218kB/s)(5180KiB/24312msec); 0 zone resets slat (usec): min=33, max=514, avg=121.40, stdev=63.92 clat (msec): min=78, max=397, avg=202.22, stdev=44.85 lat (msec): min=78, max=397, avg=202.34, stdev=44.86 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 215], | 70.00th=[ 226], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 271], | 99.00th=[ 296], 99.50th=[ 338], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 160, max= 296, per=24.83%, avg=213.02, stdev=29.93, samples=48 iops : min= 40, max= 74, avg=53.23, stdev= 7.50, samples=48 lat (msec) : 10=0.08%, 20=0.23%, 50=0.20%, 100=27.34%, 250=65.39% lat (msec) : 500=6.76% cpu : usr=0.19%, sys=1.79%, ctx=2932, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=825KiB/s (845kB/s), 205KiB/s-208KiB/s (209kB/s-213kB/s), io=19.6MiB (20.5MB), run=24290-24330msec WRITE: bw=859KiB/s (879kB/s), 213KiB/s-216KiB/s (218kB/s-222kB/s), io=20.4MiB (21.4MB), run=24290-24330msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=156197: Wed Jul 16 14:20:13 2025 read: IOPS=9, BW=377MiB/s (396MB/s)(40.0MiB/106msec) slat (nsec): min=9970.8k, max=9970.8k, avg=9970754.00, stdev= 0.00 clat (nsec): min=90580k, max=90580k, avg=90580473.00, stdev= 0.00 lat (nsec): min=100555k, max=100555k, avg=100554647.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[90702], 5.00th=[90702], 10.00th=[90702], 20.00th=[90702], | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702], | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.52%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=40.0MiB (41.9MB), run=106-106msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout