== sanity test 398c: run fio to test AIO ================= 01:49:46 (1744177786) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.124799 s, 336 MB/s osc.lustre-OST0000-osc-ffff880136b06000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=11192: Wed Apr 9 01:50:16 2025 write: IOPS=89, BW=358KiB/s (366kB/s)(10.0MiB/28638msec) slat (usec): min=13, max=469, avg=64.11, stdev=40.02 clat (msec): min=20, max=348, avg=178.73, stdev=36.13 lat (msec): min=20, max=348, avg=178.79, stdev=36.13 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 275], 99.50th=[ 313], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 351] bw ( KiB/s): min= 248, max= 560, per=25.02%, avg=357.07, stdev=57.23, samples=57 iops : min= 62, max= 140, avg=89.18, stdev=14.31, samples=57 lat (msec) : 50=0.23%, 100=0.35%, 250=96.48%, 500=2.93% cpu : usr=0.11%, sys=0.72%, ctx=2140, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=11193: Wed Apr 9 01:50:16 2025 write: IOPS=89, BW=357KiB/s (366kB/s)(10.0MiB/28651msec) slat (usec): min=12, max=401, avg=62.10, stdev=39.44 clat (msec): min=97, max=329, avg=178.70, stdev=34.97 lat (msec): min=97, max=329, avg=178.77, stdev=34.97 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 125], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 205], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 271], 99.50th=[ 292], 99.90th=[ 317], 99.95th=[ 326], | 99.99th=[ 330] bw ( KiB/s): min= 248, max= 632, per=25.00%, avg=356.74, stdev=62.96, samples=57 iops : min= 62, max= 158, avg=89.05, stdev=15.72, samples=57 lat (msec) : 100=0.78%, 250=96.21%, 500=3.01% cpu : usr=0.12%, sys=0.66%, ctx=2055, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=11194: Wed Apr 9 01:50:16 2025 write: IOPS=89, BW=357KiB/s (365kB/s)(10.0MiB/28700msec) slat (usec): min=13, max=293, avg=62.93, stdev=38.41 clat (msec): min=51, max=341, avg=178.94, stdev=34.66 lat (msec): min=51, max=341, avg=179.00, stdev=34.66 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 125], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 271], 99.50th=[ 288], 99.90th=[ 313], 99.95th=[ 321], | 99.99th=[ 342] bw ( KiB/s): min= 248, max= 512, per=24.88%, avg=355.07, stdev=55.05, samples=57 iops : min= 62, max= 128, avg=88.65, stdev=13.73, samples=57 lat (msec) : 100=0.20%, 250=96.72%, 500=3.09% cpu : usr=0.13%, sys=0.67%, ctx=2065, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=11195: Wed Apr 9 01:50:16 2025 write: IOPS=89, BW=357KiB/s (365kB/s)(10.0MiB/28700msec) slat (usec): min=13, max=456, avg=63.59, stdev=39.14 clat (msec): min=61, max=341, avg=178.94, stdev=34.90 lat (msec): min=61, max=341, avg=179.00, stdev=34.90 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 126], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 271], 99.50th=[ 296], 99.90th=[ 326], 99.95th=[ 338], | 99.99th=[ 342] bw ( KiB/s): min= 256, max= 512, per=24.95%, avg=356.04, stdev=51.75, samples=57 iops : min= 64, max= 128, avg=88.98, stdev=12.91, samples=57 lat (msec) : 100=0.16%, 250=96.76%, 500=3.09% cpu : usr=0.16%, sys=0.67%, ctx=2066, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1427KiB/s (1461kB/s), 357KiB/s-358KiB/s (365kB/s-366kB/s), io=40.0MiB (41.9MB), run=28638-28700msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11228: Wed Apr 9 01:50:34 2025 read: IOPS=74, BW=297KiB/s (304kB/s)(5048KiB/17014msec) slat (usec): min=714, max=3338, avg=1203.66, stdev=283.99 clat (usec): min=1715, max=176915, avg=69248.91, stdev=13291.88 lat (msec): min=2, max=178, avg=70.45, stdev=13.30 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 60], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 84], 95.00th=[ 88], | 99.00th=[ 97], 99.50th=[ 105], 99.90th=[ 167], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 184, max= 408, per=25.05%, avg=295.15, stdev=54.27, samples=34 iops : min= 46, max= 102, avg=73.68, stdev=13.59, samples=34 write: IOPS=76, BW=305KiB/s (312kB/s)(5192KiB/17014msec) slat (usec): min=14, max=193, avg=55.09, stdev=27.01 clat (msec): min=50, max=233, avg=140.84, stdev=32.36 lat (msec): min=50, max=233, avg=140.89, stdev=32.37 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 90], 10.00th=[ 97], 20.00th=[ 112], | 30.00th=[ 124], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 220], 99.50th=[ 222], 99.90th=[ 228], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 248, max= 415, per=24.73%, avg=303.38, stdev=36.82, samples=34 iops : min= 62, max= 103, avg=75.74, stdev= 9.08, samples=34 lat (msec) : 2=0.04%, 4=0.04%, 20=0.12%, 50=2.62%, 100=52.07% lat (msec) : 250=45.12% cpu : usr=0.16%, sys=1.49%, ctx=3368, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11229: Wed Apr 9 01:50:34 2025 read: IOPS=73, BW=292KiB/s (299kB/s)(4976KiB/17024msec) slat (usec): min=663, max=3286, avg=1212.65, stdev=284.59 clat (usec): min=1031, max=105869, avg=69173.90, stdev=13264.17 lat (usec): min=1971, max=107230, avg=70388.40, stdev=13283.67 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 60], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 97], 99.50th=[ 100], 99.90th=[ 103], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 192, max= 376, per=24.73%, avg=291.26, stdev=52.53, samples=34 iops : min= 48, max= 94, avg=72.79, stdev=13.14, samples=34 write: IOPS=77, BW=309KiB/s (317kB/s)(5264KiB/17024msec) slat (usec): min=13, max=286, avg=55.99, stdev=30.62 clat (msec): min=19, max=228, avg=140.10, stdev=32.62 lat (msec): min=19, max=228, avg=140.16, stdev=32.62 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 88], 10.00th=[ 100], 20.00th=[ 112], | 30.00th=[ 123], 40.00th=[ 132], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 180], 95.00th=[ 194], | 99.00th=[ 215], 99.50th=[ 220], 99.90th=[ 228], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 216, max= 400, per=25.06%, avg=307.50, stdev=44.27, samples=34 iops : min= 54, max= 100, avg=76.85, stdev=11.07, samples=34 lat (msec) : 2=0.12%, 4=0.08%, 20=0.27%, 50=2.70%, 100=51.09% lat (msec) : 250=45.74% cpu : usr=0.14%, sys=1.51%, ctx=3307, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11230: Wed Apr 9 01:50:34 2025 read: IOPS=73, BW=294KiB/s (301kB/s)(4984KiB/16931msec) slat (usec): min=12, max=2821, avg=1212.53, stdev=292.55 clat (msec): min=2, max=177, avg=68.78, stdev=14.09 lat (msec): min=2, max=180, avg=69.99, stdev=14.11 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 48], 10.00th=[ 54], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 91], | 99.00th=[ 97], 99.50th=[ 99], 99.90th=[ 131], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 208, max= 408, per=25.04%, avg=295.00, stdev=51.57, samples=33 iops : min= 52, max= 102, avg=73.73, stdev=12.92, samples=33 write: IOPS=77, BW=310KiB/s (318kB/s)(5256KiB/16931msec) slat (usec): min=12, max=227, avg=55.64, stdev=26.67 clat (msec): min=15, max=230, avg=139.59, stdev=33.61 lat (msec): min=15, max=230, avg=139.65, stdev=33.61 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 85], 10.00th=[ 94], 20.00th=[ 111], | 30.00th=[ 122], 40.00th=[ 131], 50.00th=[ 140], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 182], 95.00th=[ 194], | 99.00th=[ 215], 99.50th=[ 226], 99.90th=[ 228], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 240, max= 368, per=25.07%, avg=307.61, stdev=34.40, samples=33 iops : min= 60, max= 92, avg=76.88, stdev= 8.59, samples=33 lat (msec) : 4=0.08%, 10=0.12%, 20=0.55%, 50=2.58%, 100=51.60% lat (msec) : 250=45.08% cpu : usr=0.17%, sys=1.52%, ctx=3101, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11231: Wed Apr 9 01:50:34 2025 read: IOPS=74, BW=297KiB/s (304kB/s)(5060KiB/17024msec) slat (usec): min=697, max=3054, avg=1208.73, stdev=271.32 clat (usec): min=1610, max=102677, avg=69767.34, stdev=12734.06 lat (msec): min=2, max=103, avg=70.98, stdev=12.75 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 96], 99.50th=[ 99], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 224, max= 376, per=25.21%, avg=297.03, stdev=39.37, samples=34 iops : min= 56, max= 94, avg=74.15, stdev= 9.79, samples=34 write: IOPS=76, BW=304KiB/s (312kB/s)(5180KiB/17024msec) slat (usec): min=13, max=306, avg=57.11, stdev=29.95 clat (msec): min=55, max=228, avg=140.69, stdev=32.03 lat (msec): min=55, max=228, avg=140.75, stdev=32.03 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 113], | 30.00th=[ 123], 40.00th=[ 132], 50.00th=[ 140], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 169], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 213], 99.50th=[ 220], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 216, max= 416, per=24.57%, avg=301.50, stdev=40.76, samples=34 iops : min= 54, max= 104, avg=75.26, stdev=10.15, samples=34 lat (msec) : 2=0.04%, 4=0.16%, 10=0.04%, 20=0.16%, 50=2.07% lat (msec) : 100=52.15%, 250=45.39% cpu : usr=0.24%, sys=1.40%, ctx=3632, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1179KiB/s (1207kB/s), 292KiB/s-297KiB/s (299kB/s-304kB/s), io=19.6MiB (20.5MB), run=16931-17024msec WRITE: bw=1227KiB/s (1257kB/s), 304KiB/s-310KiB/s (312kB/s-318kB/s), io=20.4MiB (21.4MB), run=16931-17024msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=11248: Wed Apr 9 01:50:36 2025 read: IOPS=8, BW=331MiB/s (347MB/s)(40.0MiB/121msec) slat (nsec): min=6422.2k, max=6422.2k, avg=6422220.00, stdev= 0.00 clat (nsec): min=108225k, max=108225k, avg=108225231.00, stdev= 0.00 lat (nsec): min=114658k, max=114658k, avg=114657989.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 109], 50.00th=[ 109], 60.00th=[ 109], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 109], 95.00th=[ 109], | 99.00th=[ 109], 99.50th=[ 109], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.00%, ctx=17, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=40.0MiB (41.9MB), run=121-121msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout