== sanity test 398c: run fio to test AIO ================= 20:00:38 (1736989238) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0863055 s, 486 MB/s osc.lustre-OST0000-osc-ffff8800a1b37000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24492: Wed Jan 15 20:01:03 2025 write: IOPS=108, BW=435KiB/s (446kB/s)(10.0MiB/23536msec) slat (usec): min=13, max=461, avg=66.98, stdev=38.41 clat (msec): min=60, max=233, avg=146.94, stdev=22.62 lat (msec): min=60, max=233, avg=147.01, stdev=22.62 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 112], 10.00th=[ 118], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 201], 99.50th=[ 207], 99.90th=[ 215], 99.95th=[ 224], | 99.99th=[ 234] bw ( KiB/s): min= 368, max= 528, per=24.89%, avg=433.00, stdev=41.10, samples=47 iops : min= 92, max= 132, avg=108.13, stdev=10.20, samples=47 lat (msec) : 100=1.68%, 250=98.32% cpu : usr=0.17%, sys=0.85%, ctx=2427, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24493: Wed Jan 15 20:01:03 2025 write: IOPS=108, BW=435KiB/s (446kB/s)(10.0MiB/23527msec) slat (usec): min=12, max=517, avg=67.23, stdev=40.93 clat (msec): min=60, max=225, avg=146.89, stdev=22.91 lat (msec): min=60, max=225, avg=146.96, stdev=22.91 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 112], 10.00th=[ 118], 20.00th=[ 127], | 30.00th=[ 134], 40.00th=[ 140], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 165], 90.00th=[ 178], 95.00th=[ 186], | 99.00th=[ 205], 99.50th=[ 209], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 226] bw ( KiB/s): min= 376, max= 528, per=24.89%, avg=433.15, stdev=41.64, samples=47 iops : min= 94, max= 132, avg=108.26, stdev=10.39, samples=47 lat (msec) : 100=1.60%, 250=98.40% cpu : usr=0.14%, sys=0.90%, ctx=2301, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24494: Wed Jan 15 20:01:03 2025 write: IOPS=109, BW=437KiB/s (447kB/s)(10.0MiB/23450msec) slat (usec): min=13, max=546, avg=68.64, stdev=42.60 clat (msec): min=21, max=257, avg=146.40, stdev=23.75 lat (msec): min=21, max=257, avg=146.47, stdev=23.75 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 111], 10.00th=[ 117], 20.00th=[ 126], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 205], 99.50th=[ 209], 99.90th=[ 249], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 376, max= 576, per=25.03%, avg=435.61, stdev=43.15, samples=46 iops : min= 94, max= 144, avg=108.78, stdev=10.72, samples=46 lat (msec) : 50=0.16%, 100=2.07%, 250=97.70%, 500=0.08% cpu : usr=0.18%, sys=0.85%, ctx=2533, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24495: Wed Jan 15 20:01:03 2025 write: IOPS=108, BW=436KiB/s (446kB/s)(10.0MiB/23510msec) slat (usec): min=13, max=468, avg=67.54, stdev=39.04 clat (msec): min=71, max=245, avg=146.76, stdev=23.05 lat (msec): min=71, max=245, avg=146.83, stdev=23.05 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 111], 10.00th=[ 117], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 165], 90.00th=[ 178], 95.00th=[ 186], | 99.00th=[ 209], 99.50th=[ 211], 99.90th=[ 239], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 360, max= 568, per=24.89%, avg=433.00, stdev=46.88, samples=47 iops : min= 90, max= 142, avg=108.13, stdev=11.64, samples=47 lat (msec) : 100=1.37%, 250=98.63% cpu : usr=0.19%, sys=0.85%, ctx=2421, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1740KiB/s (1782kB/s), 435KiB/s-437KiB/s (446kB/s-447kB/s), io=40.0MiB (41.9MB), run=23450-23536msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24524: Wed Jan 15 20:01:19 2025 read: IOPS=88, BW=354KiB/s (362kB/s)(5048KiB/14269msec) slat (usec): min=455, max=2410, avg=771.40, stdev=183.18 clat (usec): min=845, max=158175, avg=60224.35, stdev=12145.35 lat (usec): min=1514, max=158944, avg=60997.58, stdev=12147.81 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 73], 95.00th=[ 78], | 99.00th=[ 86], 99.50th=[ 116], 99.90th=[ 150], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 232, max= 456, per=25.17%, avg=353.07, stdev=65.37, samples=28 iops : min= 58, max= 114, avg=88.21, stdev=16.39, samples=28 write: IOPS=90, BW=364KiB/s (373kB/s)(5192KiB/14269msec) slat (usec): min=14, max=889, avg=53.91, stdev=35.32 clat (msec): min=49, max=195, avg=116.26, stdev=27.06 lat (msec): min=49, max=195, avg=116.32, stdev=27.06 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 74], 10.00th=[ 80], 20.00th=[ 91], | 30.00th=[ 101], 40.00th=[ 109], 50.00th=[ 116], 60.00th=[ 124], | 70.00th=[ 132], 80.00th=[ 140], 90.00th=[ 153], 95.00th=[ 163], | 99.00th=[ 182], 99.50th=[ 184], 99.90th=[ 188], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 312, max= 440, per=24.89%, avg=363.64, stdev=31.53, samples=28 iops : min= 78, max= 110, avg=90.86, stdev= 7.90, samples=28 lat (usec) : 1000=0.04% lat (msec) : 2=0.04%, 10=0.04%, 20=0.39%, 50=7.30%, 100=56.25% lat (msec) : 250=35.94% cpu : usr=0.27%, sys=1.59%, ctx=3426, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24525: Wed Jan 15 20:01:19 2025 read: IOPS=87, BW=348KiB/s (357kB/s)(4976KiB/14286msec) slat (usec): min=46, max=2551, avg=770.89, stdev=202.46 clat (usec): min=1079, max=112651, avg=59813.72, stdev=11197.13 lat (usec): min=1325, max=113856, avg=60586.47, stdev=11205.83 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 45], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 78], | 99.00th=[ 84], 99.50th=[ 89], 99.90th=[ 112], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 255, max= 504, per=24.74%, avg=347.07, stdev=60.45, samples=28 iops : min= 63, max= 126, avg=86.71, stdev=15.13, samples=28 write: IOPS=92, BW=368KiB/s (377kB/s)(5264KiB/14286msec) slat (usec): min=13, max=238, avg=54.18, stdev=28.80 clat (msec): min=23, max=191, avg=116.25, stdev=28.14 lat (msec): min=23, max=191, avg=116.30, stdev=28.14 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 72], 10.00th=[ 80], 20.00th=[ 92], | 30.00th=[ 101], 40.00th=[ 109], 50.00th=[ 117], 60.00th=[ 125], | 70.00th=[ 133], 80.00th=[ 140], 90.00th=[ 155], 95.00th=[ 163], | 99.00th=[ 178], 99.50th=[ 184], 99.90th=[ 190], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 288, max= 440, per=25.03%, avg=365.64, stdev=41.42, samples=28 iops : min= 72, max= 110, avg=91.36, stdev=10.36, samples=28 lat (msec) : 2=0.12%, 4=0.04%, 10=0.16%, 20=0.08%, 50=7.66% lat (msec) : 100=55.98%, 250=35.98% cpu : usr=0.29%, sys=1.57%, ctx=3557, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24526: Wed Jan 15 20:01:19 2025 read: IOPS=87, BW=349KiB/s (357kB/s)(4984KiB/14295msec) slat (usec): min=466, max=2363, avg=780.97, stdev=188.78 clat (usec): min=775, max=150205, avg=60183.09, stdev=11010.10 lat (usec): min=1540, max=150946, avg=60965.87, stdev=11016.27 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 74], 95.00th=[ 78], | 99.00th=[ 83], 99.50th=[ 87], 99.90th=[ 113], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 240, max= 496, per=24.96%, avg=350.21, stdev=76.26, samples=28 iops : min= 60, max= 124, avg=87.50, stdev=19.07, samples=28 write: IOPS=91, BW=368KiB/s (377kB/s)(5256KiB/14295msec) slat (usec): min=14, max=243, avg=52.35, stdev=26.86 clat (msec): min=51, max=193, avg=116.00, stdev=26.67 lat (msec): min=51, max=194, avg=116.05, stdev=26.67 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 74], 10.00th=[ 82], 20.00th=[ 93], | 30.00th=[ 101], 40.00th=[ 108], 50.00th=[ 114], 60.00th=[ 123], | 70.00th=[ 131], 80.00th=[ 140], 90.00th=[ 153], 95.00th=[ 163], | 99.00th=[ 178], 99.50th=[ 182], 99.90th=[ 186], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 287, max= 448, per=25.03%, avg=365.64, stdev=37.04, samples=28 iops : min= 71, max= 112, avg=91.36, stdev= 9.29, samples=28 lat (usec) : 1000=0.04% lat (msec) : 10=0.12%, 20=0.12%, 50=7.62%, 100=55.90%, 250=36.21% cpu : usr=0.24%, sys=1.61%, ctx=3392, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24527: Wed Jan 15 20:01:19 2025 read: IOPS=88, BW=354KiB/s (362kB/s)(5060KiB/14295msec) slat (usec): min=425, max=3182, avg=775.92, stdev=215.97 clat (usec): min=1114, max=141303, avg=59924.99, stdev=12037.67 lat (msec): min=2, max=141, avg=60.70, stdev=12.04 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 74], 95.00th=[ 78], | 99.00th=[ 83], 99.50th=[ 86], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 224, max= 488, per=25.27%, avg=354.50, stdev=62.16, samples=28 iops : min= 56, max= 122, avg=88.57, stdev=15.52, samples=28 write: IOPS=90, BW=362KiB/s (371kB/s)(5180KiB/14295msec) slat (usec): min=13, max=810, avg=56.14, stdev=37.62 clat (msec): min=52, max=202, avg=117.02, stdev=27.64 lat (msec): min=52, max=202, avg=117.07, stdev=27.64 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 92], | 30.00th=[ 102], 40.00th=[ 110], 50.00th=[ 117], 60.00th=[ 124], | 70.00th=[ 132], 80.00th=[ 140], 90.00th=[ 155], 95.00th=[ 165], | 99.00th=[ 182], 99.50th=[ 186], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 271, max= 424, per=24.75%, avg=361.64, stdev=35.07, samples=28 iops : min= 67, max= 106, avg=90.36, stdev= 8.82, samples=28 lat (msec) : 2=0.04%, 4=0.04%, 10=0.20%, 20=0.31%, 50=7.54% lat (msec) : 100=55.08%, 250=36.80% cpu : usr=0.32%, sys=1.55%, ctx=3586, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1404KiB/s (1438kB/s), 348KiB/s-354KiB/s (357kB/s-362kB/s), io=19.6MiB (20.5MB), run=14269-14295msec WRITE: bw=1461KiB/s (1497kB/s), 362KiB/s-368KiB/s (371kB/s-377kB/s), io=20.4MiB (21.4MB), run=14269-14295msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24542: Wed Jan 15 20:01:20 2025 read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec) slat (nsec): min=7279.2k, max=7279.2k, avg=7279171.00, stdev= 0.00 clat (nsec): min=64837k, max=64837k, avg=64837153.00, stdev= 0.00 lat (nsec): min=72128k, max=72128k, avg=72127606.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[64750], 5.00th=[64750], 10.00th=[64750], 20.00th=[64750], | 30.00th=[64750], 40.00th=[64750], 50.00th=[64750], 60.00th=[64750], | 70.00th=[64750], 80.00th=[64750], 90.00th=[64750], 95.00th=[64750], | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.33%, ctx=24, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck