== sanity test 398c: run fio to test AIO ================= 04:08:29 (1738487309) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0963735 s, 435 MB/s osc.lustre-OST0000-osc-ffff88007c0a3800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=6120: Sun Feb 2 04:09:05 2025 write: IOPS=74, BW=297KiB/s (304kB/s)(10.0MiB/34483msec) slat (usec): min=13, max=568, avg=76.32, stdev=45.66 clat (msec): min=62, max=351, avg=215.02, stdev=49.32 lat (msec): min=62, max=351, avg=215.10, stdev=49.33 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 262], 90.00th=[ 279], 95.00th=[ 292], | 99.00th=[ 321], 99.50th=[ 326], 99.90th=[ 338], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 200, max= 447, per=24.92%, avg=295.76, stdev=57.58, samples=68 iops : min= 50, max= 111, avg=73.85, stdev=14.42, samples=68 lat (msec) : 100=0.66%, 250=72.27%, 500=27.07% cpu : usr=0.15%, sys=0.65%, ctx=2160, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=6121: Sun Feb 2 04:09:05 2025 write: IOPS=74, BW=297KiB/s (304kB/s)(10.0MiB/34483msec) slat (usec): min=13, max=670, avg=77.51, stdev=48.40 clat (msec): min=96, max=352, avg=215.18, stdev=49.02 lat (msec): min=96, max=352, avg=215.26, stdev=49.03 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 133], 10.00th=[ 148], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 213], 60.00th=[ 228], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 279], 95.00th=[ 292], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 351], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 216, max= 456, per=24.91%, avg=295.72, stdev=60.11, samples=68 iops : min= 54, max= 114, avg=73.90, stdev=15.02, samples=68 lat (msec) : 100=0.04%, 250=73.05%, 500=26.91% cpu : usr=0.11%, sys=0.68%, ctx=2233, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=6122: Sun Feb 2 04:09:05 2025 write: IOPS=74, BW=297KiB/s (304kB/s)(10.0MiB/34483msec) slat (usec): min=13, max=394, avg=79.79, stdev=45.00 clat (msec): min=88, max=350, avg=215.04, stdev=49.48 lat (msec): min=88, max=350, avg=215.12, stdev=49.49 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 321], 99.50th=[ 330], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 351] bw ( KiB/s): min= 192, max= 440, per=24.89%, avg=295.47, stdev=59.40, samples=68 iops : min= 48, max= 110, avg=73.82, stdev=14.86, samples=68 lat (msec) : 100=0.62%, 250=71.64%, 500=27.73% cpu : usr=0.12%, sys=0.68%, ctx=2386, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=6123: Sun Feb 2 04:09:05 2025 write: IOPS=74, BW=298KiB/s (305kB/s)(10.0MiB/34417msec) slat (usec): min=14, max=702, avg=78.26, stdev=49.71 clat (msec): min=41, max=374, avg=214.79, stdev=50.62 lat (msec): min=41, max=374, avg=214.87, stdev=50.63 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 133], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 215], 60.00th=[ 230], | 70.00th=[ 247], 80.00th=[ 264], 90.00th=[ 279], 95.00th=[ 292], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99.95th=[ 359], | 99.99th=[ 376] bw ( KiB/s): min= 176, max= 456, per=24.98%, avg=296.53, stdev=62.92, samples=68 iops : min= 44, max= 114, avg=74.09, stdev=15.72, samples=68 lat (msec) : 50=0.31%, 100=0.35%, 250=71.60%, 500=27.73% cpu : usr=0.10%, sys=0.72%, ctx=2225, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1188KiB/s (1216kB/s), 297KiB/s-298KiB/s (304kB/s-305kB/s), io=40.0MiB (41.9MB), run=34417-34483msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6160: Sun Feb 2 04:09:24 2025 read: IOPS=69, BW=280KiB/s (286kB/s)(5048KiB/18057msec) slat (usec): min=26, max=3511, avg=1269.76, stdev=408.97 clat (msec): min=3, max=195, avg=74.05, stdev=19.48 lat (msec): min=3, max=196, avg=75.32, stdev=19.66 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 79], 80.00th=[ 85], 90.00th=[ 99], 95.00th=[ 111], | 99.00th=[ 138], 99.50th=[ 150], 99.90th=[ 184], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 144, max= 448, per=25.25%, avg=279.06, stdev=70.62, samples=36 iops : min= 36, max= 112, avg=69.72, stdev=17.63, samples=36 write: IOPS=71, BW=288KiB/s (294kB/s)(5192KiB/18057msec) slat (usec): min=13, max=242, avg=57.39, stdev=30.04 clat (msec): min=23, max=362, avg=149.09, stdev=46.92 lat (msec): min=23, max=362, avg=149.14, stdev=46.93 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 85], 10.00th=[ 97], 20.00th=[ 112], | 30.00th=[ 124], 40.00th=[ 134], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 167], 80.00th=[ 182], 90.00th=[ 203], 95.00th=[ 239], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 351], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 168, max= 384, per=24.90%, avg=286.39, stdev=53.02, samples=36 iops : min= 42, max= 96, avg=71.56, stdev=13.26, samples=36 lat (msec) : 4=0.20%, 10=0.12%, 20=0.08%, 50=1.56%, 100=49.10% lat (msec) : 250=46.91%, 500=2.03% cpu : usr=0.17%, sys=1.48%, ctx=3381, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6161: Sun Feb 2 04:09:24 2025 read: IOPS=68, BW=275KiB/s (281kB/s)(4976KiB/18119msec) slat (usec): min=697, max=3584, avg=1282.21, stdev=373.69 clat (msec): min=4, max=157, avg=74.07, stdev=18.25 lat (msec): min=5, max=159, avg=75.36, stdev=18.44 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 99], 95.00th=[ 109], | 99.00th=[ 131], 99.50th=[ 136], 99.90th=[ 153], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 144, max= 407, per=24.87%, avg=274.83, stdev=65.81, samples=36 iops : min= 36, max= 101, avg=68.67, stdev=16.41, samples=36 write: IOPS=72, BW=291KiB/s (297kB/s)(5264KiB/18119msec) slat (usec): min=13, max=564, avg=59.44, stdev=36.54 clat (msec): min=65, max=368, avg=148.65, stdev=45.75 lat (msec): min=65, max=368, avg=148.71, stdev=45.75 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 87], 10.00th=[ 95], 20.00th=[ 109], | 30.00th=[ 123], 40.00th=[ 134], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 167], 80.00th=[ 180], 90.00th=[ 203], 95.00th=[ 232], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 160, max= 464, per=25.19%, avg=289.72, stdev=68.97, samples=36 iops : min= 40, max= 116, avg=72.39, stdev=17.24, samples=36 lat (msec) : 10=0.12%, 20=0.31%, 50=1.25%, 100=49.65%, 250=46.88% lat (msec) : 500=1.80% cpu : usr=0.16%, sys=1.52%, ctx=3542, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6162: Sun Feb 2 04:09:24 2025 read: IOPS=68, BW=275KiB/s (281kB/s)(4984KiB/18152msec) slat (usec): min=626, max=4487, avg=1290.42, stdev=406.17 clat (msec): min=2, max=155, avg=73.82, stdev=17.94 lat (msec): min=3, max=157, avg=75.11, stdev=18.14 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 97], 95.00th=[ 110], | 99.00th=[ 130], 99.50th=[ 136], 99.90th=[ 150], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 144, max= 440, per=24.75%, avg=273.50, stdev=72.80, samples=36 iops : min= 36, max= 110, avg=68.33, stdev=18.19, samples=36 write: IOPS=72, BW=290KiB/s (297kB/s)(5256KiB/18152msec) slat (usec): min=14, max=349, avg=56.82, stdev=32.53 clat (msec): min=20, max=364, avg=149.35, stdev=46.13 lat (msec): min=21, max=364, avg=149.41, stdev=46.14 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 84], 10.00th=[ 96], 20.00th=[ 112], | 30.00th=[ 126], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 157], | 70.00th=[ 167], 80.00th=[ 178], 90.00th=[ 203], 95.00th=[ 236], | 99.00th=[ 309], 99.50th=[ 326], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 176, max= 392, per=24.81%, avg=285.28, stdev=53.14, samples=36 iops : min= 44, max= 98, avg=71.28, stdev=13.27, samples=36 lat (msec) : 4=0.04%, 10=0.12%, 20=0.23%, 50=2.19%, 100=48.20% lat (msec) : 250=47.46%, 500=1.76% cpu : usr=0.24%, sys=1.42%, ctx=3345, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6163: Sun Feb 2 04:09:24 2025 read: IOPS=69, BW=279KiB/s (286kB/s)(5060KiB/18131msec) slat (usec): min=631, max=3142, avg=1270.36, stdev=375.20 clat (msec): min=8, max=154, avg=73.91, stdev=17.14 lat (msec): min=9, max=157, avg=75.19, stdev=17.33 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 94], 95.00th=[ 107], | 99.00th=[ 128], 99.50th=[ 132], 99.90th=[ 146], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 88, max= 424, per=25.28%, avg=279.31, stdev=71.13, samples=36 iops : min= 22, max= 106, avg=69.81, stdev=17.79, samples=36 write: IOPS=71, BW=286KiB/s (293kB/s)(5180KiB/18131msec) slat (usec): min=14, max=1274, avg=58.97, stdev=46.13 clat (msec): min=60, max=362, avg=150.01, stdev=46.00 lat (msec): min=60, max=362, avg=150.07, stdev=46.01 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 86], 10.00th=[ 96], 20.00th=[ 113], | 30.00th=[ 126], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 157], | 70.00th=[ 167], 80.00th=[ 182], 90.00th=[ 205], 95.00th=[ 230], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[ 338], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 176, max= 384, per=24.67%, avg=283.75, stdev=55.00, samples=36 iops : min= 44, max= 96, avg=70.92, stdev=13.74, samples=36 lat (msec) : 10=0.08%, 20=0.20%, 50=1.25%, 100=50.62%, 250=46.21% lat (msec) : 500=1.64% cpu : usr=0.18%, sys=1.48%, ctx=3334, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1106KiB/s (1132kB/s), 275KiB/s-280KiB/s (281kB/s-286kB/s), io=19.6MiB (20.5MB), run=18057-18152msec WRITE: bw=1151KiB/s (1179kB/s), 286KiB/s-291KiB/s (293kB/s-297kB/s), io=20.4MiB (21.4MB), run=18057-18152msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=6181: Sun Feb 2 04:09:25 2025 read: IOPS=6, BW=280MiB/s (293MB/s)(40.0MiB/143msec) slat (nsec): min=5231.0k, max=5231.0k, avg=5231957.00, stdev= 0.00 clat (nsec): min=131874k, max=131874k, avg=131873993.00, stdev= 0.00 lat (nsec): min=137115k, max=137115k, avg=137114865.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.23%, ctx=4, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=280MiB/s (293MB/s), 280MiB/s-280MiB/s (293MB/s-293MB/s), io=40.0MiB (41.9MB), run=143-143msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout