== sanity test 398c: run fio to test AIO ================= 00:30:20 (1746160220) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.276146 s, 152 MB/s osc.lustre-OST0000-osc-ffff8800b5819000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=26680: Fri May 2 00:30:43 2025 write: IOPS=130, BW=521KiB/s (534kB/s)(10.0MiB/19636msec) slat (usec): min=11, max=8188, avg=134.52, stdev=432.88 clat (msec): min=53, max=224, avg=122.02, stdev=27.18 lat (msec): min=54, max=224, avg=122.16, stdev=27.17 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 85], 10.00th=[ 92], 20.00th=[ 99], | 30.00th=[ 106], 40.00th=[ 112], 50.00th=[ 120], 60.00th=[ 127], | 70.00th=[ 133], 80.00th=[ 144], 90.00th=[ 161], 95.00th=[ 174], | 99.00th=[ 201], 99.50th=[ 209], 99.90th=[ 222], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 366, max= 680, per=24.92%, avg=519.23, stdev=78.71, samples=39 iops : min= 91, max= 170, avg=129.54, stdev=19.79, samples=39 lat (msec) : 100=22.19%, 250=77.81% cpu : usr=0.33%, sys=1.93%, ctx=842, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26681: Fri May 2 00:30:43 2025 write: IOPS=130, BW=523KiB/s (535kB/s)(10.0MiB/19590msec) slat (usec): min=11, max=9040, avg=152.86, stdev=492.36 clat (msec): min=20, max=236, avg=121.93, stdev=27.91 lat (msec): min=20, max=237, avg=122.08, stdev=27.90 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 86], 10.00th=[ 90], 20.00th=[ 100], | 30.00th=[ 106], 40.00th=[ 113], 50.00th=[ 120], 60.00th=[ 126], | 70.00th=[ 134], 80.00th=[ 144], 90.00th=[ 157], 95.00th=[ 174], | 99.00th=[ 199], 99.50th=[ 224], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 383, max= 648, per=24.98%, avg=520.64, stdev=71.94, samples=39 iops : min= 95, max= 162, avg=129.97, stdev=18.09, samples=39 lat (msec) : 50=0.70%, 100=21.56%, 250=77.73% cpu : usr=0.42%, sys=2.02%, ctx=1053, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26682: Fri May 2 00:30:43 2025 write: IOPS=130, BW=522KiB/s (535kB/s)(10.0MiB/19614msec) slat (usec): min=10, max=9193, avg=137.23, stdev=443.30 clat (msec): min=31, max=234, avg=121.98, stdev=26.46 lat (msec): min=31, max=234, avg=122.12, stdev=26.47 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 87], 10.00th=[ 91], 20.00th=[ 100], | 30.00th=[ 107], 40.00th=[ 113], 50.00th=[ 121], 60.00th=[ 127], | 70.00th=[ 134], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 167], | 99.00th=[ 201], 99.50th=[ 209], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 382, max= 648, per=24.98%, avg=520.49, stdev=80.04, samples=39 iops : min= 95, max= 162, avg=129.87, stdev=20.08, samples=39 lat (msec) : 50=0.04%, 100=20.55%, 250=79.41% cpu : usr=0.28%, sys=1.95%, ctx=860, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26683: Fri May 2 00:30:43 2025 write: IOPS=130, BW=521KiB/s (534kB/s)(10.0MiB/19649msec) slat (usec): min=10, max=7657, avg=142.29, stdev=441.55 clat (msec): min=22, max=236, avg=122.04, stdev=27.62 lat (msec): min=22, max=236, avg=122.19, stdev=27.63 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 85], 10.00th=[ 91], 20.00th=[ 101], | 30.00th=[ 106], 40.00th=[ 111], 50.00th=[ 118], 60.00th=[ 126], | 70.00th=[ 133], 80.00th=[ 146], 90.00th=[ 159], 95.00th=[ 174], | 99.00th=[ 205], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 236] bw ( KiB/s): min= 382, max= 640, per=24.80%, avg=516.77, stdev=71.46, samples=39 iops : min= 95, max= 160, avg=128.95, stdev=17.94, samples=39 lat (msec) : 50=0.23%, 100=19.96%, 250=79.80% cpu : usr=0.51%, sys=1.83%, ctx=900, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2085KiB/s (2135kB/s), 521KiB/s-523KiB/s (534kB/s-535kB/s), io=40.0MiB (41.9MB), run=19590-19649msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26709: Fri May 2 00:31:03 2025 read: IOPS=72, BW=290KiB/s (297kB/s)(5048KiB/17417msec) slat (usec): min=1775, max=35009, avg=7304.89, stdev=3322.70 clat (msec): min=10, max=200, avg=101.96, stdev=24.52 lat (msec): min=15, max=212, avg=109.27, stdev=25.56 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 81], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 102], 60.00th=[ 107], | 70.00th=[ 113], 80.00th=[ 123], 90.00th=[ 136], 95.00th=[ 146], | 99.00th=[ 167], 99.50th=[ 171], 99.90th=[ 192], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 112, max= 416, per=24.92%, avg=287.09, stdev=70.15, samples=34 iops : min= 28, max= 104, avg=71.44, stdev=17.55, samples=34 write: IOPS=74, BW=298KiB/s (305kB/s)(5192KiB/17417msec) slat (usec): min=12, max=10480, avg=166.74, stdev=464.38 clat (msec): min=10, max=178, avg=106.55, stdev=23.38 lat (msec): min=10, max=178, avg=106.71, stdev=23.38 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 79], 20.00th=[ 87], | 30.00th=[ 93], 40.00th=[ 100], 50.00th=[ 105], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 126], 90.00th=[ 140], 95.00th=[ 148], | 99.00th=[ 165], 99.50th=[ 169], 99.90th=[ 174], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 144, max= 424, per=24.61%, avg=295.09, stdev=56.15, samples=34 iops : min= 36, max= 106, avg=73.44, stdev=14.03, samples=34 lat (msec) : 20=0.20%, 50=0.31%, 100=44.96%, 250=54.53% cpu : usr=1.21%, sys=4.35%, ctx=2424, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26710: Fri May 2 00:31:03 2025 read: IOPS=71, BW=287KiB/s (294kB/s)(4976KiB/17342msec) slat (usec): min=2046, max=29379, avg=7304.81, stdev=3230.10 clat (msec): min=14, max=202, avg=102.70, stdev=23.84 lat (msec): min=18, max=206, avg=110.01, stdev=24.77 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 82], | 30.00th=[ 88], 40.00th=[ 96], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 113], 80.00th=[ 123], 90.00th=[ 134], 95.00th=[ 144], | 99.00th=[ 167], 99.50th=[ 178], 99.90th=[ 194], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 160, max= 408, per=24.85%, avg=286.26, stdev=57.30, samples=34 iops : min= 40, max= 102, avg=71.38, stdev=14.33, samples=34 write: IOPS=75, BW=304KiB/s (311kB/s)(5264KiB/17342msec) slat (usec): min=14, max=4358, avg=146.52, stdev=337.93 clat (msec): min=31, max=176, avg=105.75, stdev=23.35 lat (msec): min=31, max=176, avg=105.90, stdev=23.34 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 87], | 30.00th=[ 92], 40.00th=[ 99], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 117], 80.00th=[ 126], 90.00th=[ 138], 95.00th=[ 150], | 99.00th=[ 165], 99.50th=[ 169], 99.90th=[ 176], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 199, max= 447, per=25.31%, avg=303.44, stdev=63.35, samples=34 iops : min= 49, max= 111, avg=75.68, stdev=15.87, samples=34 lat (msec) : 20=0.04%, 50=0.12%, 100=44.69%, 250=55.16% cpu : usr=0.71%, sys=4.72%, ctx=2452, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26711: Fri May 2 00:31:03 2025 read: IOPS=71, BW=287KiB/s (294kB/s)(4984KiB/17341msec) slat (usec): min=42, max=33996, avg=7293.95, stdev=3385.22 clat (msec): min=10, max=205, avg=103.22, stdev=25.43 lat (msec): min=10, max=209, avg=110.52, stdev=26.38 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 82], | 30.00th=[ 88], 40.00th=[ 96], 50.00th=[ 103], 60.00th=[ 109], | 70.00th=[ 115], 80.00th=[ 124], 90.00th=[ 138], 95.00th=[ 150], | 99.00th=[ 169], 99.50th=[ 182], 99.90th=[ 203], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 199, max= 400, per=24.98%, avg=287.74, stdev=50.05, samples=34 iops : min= 49, max= 100, avg=71.62, stdev=12.49, samples=34 write: IOPS=75, BW=303KiB/s (310kB/s)(5256KiB/17341msec) slat (usec): min=14, max=7871, avg=165.92, stdev=476.30 clat (msec): min=29, max=193, avg=105.79, stdev=22.92 lat (msec): min=29, max=193, avg=105.95, stdev=22.91 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 78], 20.00th=[ 86], | 30.00th=[ 92], 40.00th=[ 99], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 117], 80.00th=[ 126], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 163], 99.50th=[ 169], 99.90th=[ 188], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 159, max= 424, per=25.16%, avg=301.62, stdev=60.42, samples=34 iops : min= 39, max= 106, avg=75.09, stdev=15.16, samples=34 lat (msec) : 20=0.16%, 50=0.16%, 100=44.22%, 250=55.47% cpu : usr=0.77%, sys=4.76%, ctx=2350, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26712: Fri May 2 00:31:03 2025 read: IOPS=72, BW=291KiB/s (298kB/s)(5060KiB/17387msec) slat (usec): min=1629, max=34872, avg=7218.74, stdev=3238.43 clat (msec): min=44, max=183, avg=103.14, stdev=23.23 lat (msec): min=49, max=201, avg=110.37, stdev=24.17 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 74], 20.00th=[ 82], | 30.00th=[ 90], 40.00th=[ 97], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 114], 80.00th=[ 122], 90.00th=[ 136], 95.00th=[ 146], | 99.00th=[ 163], 99.50th=[ 169], 99.90th=[ 178], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 144, max= 407, per=25.20%, avg=290.32, stdev=62.51, samples=34 iops : min= 36, max= 101, avg=72.38, stdev=15.58, samples=34 write: IOPS=74, BW=298KiB/s (305kB/s)(5180KiB/17387msec) slat (usec): min=11, max=7613, avg=157.76, stdev=424.84 clat (msec): min=55, max=174, avg=105.51, stdev=22.32 lat (msec): min=55, max=176, avg=105.67, stdev=22.32 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 86], | 30.00th=[ 93], 40.00th=[ 100], 50.00th=[ 105], 60.00th=[ 110], | 70.00th=[ 116], 80.00th=[ 126], 90.00th=[ 138], 95.00th=[ 146], | 99.00th=[ 159], 99.50th=[ 167], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 168, max= 440, per=24.71%, avg=296.24, stdev=57.88, samples=34 iops : min= 42, max= 110, avg=73.85, stdev=14.45, samples=34 lat (msec) : 50=0.12%, 100=44.69%, 250=55.20% cpu : usr=0.93%, sys=4.45%, ctx=2470, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1152KiB/s (1180kB/s), 287KiB/s-291KiB/s (294kB/s-298kB/s), io=19.6MiB (20.5MB), run=17341-17417msec WRITE: bw=1200KiB/s (1228kB/s), 298KiB/s-304KiB/s (305kB/s-311kB/s), io=20.4MiB (21.4MB), run=17341-17417msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=26730: Fri May 2 00:31:06 2025 read: IOPS=2, BW=83.7MiB/s (87.7MB/s)(40.0MiB/478msec) slat (nsec): min=3681.6k, max=3681.6k, avg=3681585.00, stdev= 0.00 clat (nsec): min=455097k, max=455097k, avg=455097483.00, stdev= 0.00 lat (nsec): min=458799k, max=458799k, avg=458798943.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.63%, ctx=6, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.7MiB/s (87.7MB/s), 83.7MiB/s-83.7MiB/s (87.7MB/s-87.7MB/s), io=40.0MiB (41.9MB), run=478-478msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout