== sanity test 398c: run fio to test AIO ================= 22:27:25 (1733801245) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.154018 s, 272 MB/s osc.lustre-OST0000-osc-ffff8800afe26800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=2624: Mon Dec 9 22:27:53 2024 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26811msec) slat (usec): min=13, max=695, avg=70.25, stdev=44.34 clat (msec): min=77, max=278, avg=167.26, stdev=30.92 lat (msec): min=77, max=278, avg=167.34, stdev=30.92 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 241], 99.50th=[ 247], 99.90th=[ 268], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 296, max= 512, per=24.99%, avg=381.53, stdev=42.12, samples=53 iops : min= 74, max= 128, avg=95.34, stdev=10.55, samples=53 lat (msec) : 100=1.33%, 250=98.20%, 500=0.47% cpu : usr=0.18%, sys=0.68%, ctx=2437, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2625: Mon Dec 9 22:27:53 2024 write: IOPS=95, BW=382KiB/s (392kB/s)(10.0MiB/26779msec) slat (usec): min=13, max=530, avg=71.76, stdev=42.04 clat (msec): min=23, max=352, avg=167.19, stdev=32.38 lat (msec): min=23, max=352, avg=167.26, stdev=32.38 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 118], 10.00th=[ 130], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 239], 99.50th=[ 251], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 355] bw ( KiB/s): min= 272, max= 592, per=25.07%, avg=382.83, stdev=48.28, samples=53 iops : min= 68, max= 148, avg=95.62, stdev=12.11, samples=53 lat (msec) : 50=0.16%, 100=1.60%, 250=97.73%, 500=0.51% cpu : usr=0.16%, sys=0.74%, ctx=2407, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2626: Mon Dec 9 22:27:53 2024 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26811msec) slat (usec): min=13, max=483, avg=71.27, stdev=44.24 clat (msec): min=54, max=305, avg=167.29, stdev=30.79 lat (msec): min=54, max=305, avg=167.36, stdev=30.79 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 241], 99.50th=[ 247], 99.90th=[ 268], 99.95th=[ 288], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 512, per=24.98%, avg=381.47, stdev=44.82, samples=53 iops : min= 64, max= 128, avg=95.28, stdev=11.24, samples=53 lat (msec) : 100=0.86%, 250=98.83%, 500=0.31% cpu : usr=0.16%, sys=0.69%, ctx=2504, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2627: Mon Dec 9 22:27:53 2024 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26799msec) slat (usec): min=14, max=445, avg=71.97, stdev=42.13 clat (msec): min=31, max=373, avg=167.25, stdev=31.66 lat (msec): min=32, max=373, avg=167.32, stdev=31.66 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 211], 95.00th=[ 224], | 99.00th=[ 239], 99.50th=[ 247], 99.90th=[ 338], 99.95th=[ 351], | 99.99th=[ 372] bw ( KiB/s): min= 288, max= 512, per=25.04%, avg=382.38, stdev=41.42, samples=53 iops : min= 72, max= 128, avg=95.51, stdev=10.38, samples=53 lat (msec) : 50=0.04%, 100=0.86%, 250=98.75%, 500=0.35% cpu : usr=0.15%, sys=0.75%, ctx=2497, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1528KiB/s (1564kB/s), 382KiB/s-382KiB/s (391kB/s-392kB/s), io=40.0MiB (41.9MB), run=26779-26811msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2658: Mon Dec 9 22:28:10 2024 read: IOPS=80, BW=320KiB/s (328kB/s)(5048KiB/15760msec) slat (usec): min=15, max=3257, avg=1189.48, stdev=299.70 clat (usec): min=1811, max=151733, avg=64211.69, stdev=12437.62 lat (usec): min=1969, max=152635, avg=65403.30, stdev=12487.68 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 58], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 70], 80.00th=[ 74], 90.00th=[ 79], 95.00th=[ 83], | 99.00th=[ 88], 99.50th=[ 92], 99.90th=[ 123], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 216, max= 424, per=25.29%, avg=320.23, stdev=47.71, samples=31 iops : min= 54, max= 106, avg=80.03, stdev=11.93, samples=31 write: IOPS=82, BW=329KiB/s (337kB/s)(5192KiB/15760msec) slat (usec): min=12, max=1075, avg=58.20, stdev=40.27 clat (msec): min=16, max=235, avg=130.47, stdev=30.96 lat (msec): min=16, max=235, avg=130.53, stdev=30.96 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 82], 10.00th=[ 93], 20.00th=[ 105], | 30.00th=[ 115], 40.00th=[ 123], 50.00th=[ 130], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 184], | 99.00th=[ 199], 99.50th=[ 205], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 248, max= 408, per=24.94%, avg=328.74, stdev=40.54, samples=31 iops : min= 62, max= 102, avg=82.16, stdev=10.13, samples=31 lat (msec) : 2=0.08%, 4=0.12%, 10=0.20%, 20=0.27%, 50=4.02% lat (msec) : 100=52.38%, 250=42.93% cpu : usr=0.22%, sys=1.59%, ctx=3134, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2659: Mon Dec 9 22:28:10 2024 read: IOPS=78, BW=314KiB/s (322kB/s)(4976KiB/15822msec) slat (usec): min=707, max=3156, avg=1191.55, stdev=294.50 clat (usec): min=1247, max=94786, avg=63954.98, stdev=11747.54 lat (usec): min=2327, max=96347, avg=65148.50, stdev=11767.55 clat percentiles (usec): | 1.00th=[29754], 5.00th=[46400], 10.00th=[50594], 20.00th=[55313], | 30.00th=[57934], 40.00th=[61604], 50.00th=[64750], 60.00th=[67634], | 70.00th=[70779], 80.00th=[73925], 90.00th=[78119], 95.00th=[81265], | 99.00th=[87557], 99.50th=[89654], 99.90th=[91751], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 216, max= 456, per=24.70%, avg=312.68, stdev=60.03, samples=31 iops : min= 54, max= 114, avg=78.10, stdev=14.99, samples=31 write: IOPS=83, BW=333KiB/s (341kB/s)(5264KiB/15822msec) slat (usec): min=13, max=374, avg=55.83, stdev=28.75 clat (msec): min=32, max=213, avg=130.44, stdev=29.98 lat (msec): min=32, max=213, avg=130.50, stdev=29.98 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 84], 10.00th=[ 93], 20.00th=[ 105], | 30.00th=[ 115], 40.00th=[ 123], 50.00th=[ 130], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 157], 90.00th=[ 169], 95.00th=[ 182], | 99.00th=[ 199], 99.50th=[ 203], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 216, max= 416, per=25.02%, avg=329.71, stdev=42.15, samples=31 iops : min= 54, max= 104, avg=82.35, stdev=10.60, samples=31 lat (msec) : 2=0.04%, 4=0.12%, 20=0.27%, 50=3.87%, 100=52.30% lat (msec) : 250=43.40% cpu : usr=0.30%, sys=1.48%, ctx=3080, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2660: Mon Dec 9 22:28:10 2024 read: IOPS=78, BW=314KiB/s (322kB/s)(4984KiB/15850msec) slat (usec): min=709, max=3422, avg=1199.71, stdev=300.07 clat (usec): min=1197, max=94905, avg=63846.61, stdev=11093.17 lat (usec): min=2396, max=96072, avg=65048.31, stdev=11118.17 clat percentiles (usec): | 1.00th=[34866], 5.00th=[45876], 10.00th=[51119], 20.00th=[55313], | 30.00th=[57934], 40.00th=[61080], 50.00th=[64226], 60.00th=[66847], | 70.00th=[69731], 80.00th=[72877], 90.00th=[78119], 95.00th=[81265], | 99.00th=[85459], 99.50th=[87557], 99.90th=[89654], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 208, max= 456, per=24.70%, avg=312.68, stdev=59.69, samples=31 iops : min= 52, max= 114, avg=78.10, stdev=14.97, samples=31 write: IOPS=82, BW=332KiB/s (340kB/s)(5256KiB/15850msec) slat (usec): min=13, max=1103, avg=56.81, stdev=40.71 clat (msec): min=27, max=213, avg=131.00, stdev=31.28 lat (msec): min=27, max=213, avg=131.06, stdev=31.28 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 81], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 184], | 99.00th=[ 205], 99.50th=[ 207], 99.90th=[ 211], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 272, max= 376, per=24.72%, avg=325.84, stdev=29.60, samples=31 iops : min= 68, max= 94, avg=81.39, stdev= 7.48, samples=31 lat (msec) : 2=0.04%, 10=0.04%, 20=0.16%, 50=4.34%, 100=52.19% lat (msec) : 250=43.24% cpu : usr=0.21%, sys=1.58%, ctx=3089, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2661: Mon Dec 9 22:28:10 2024 read: IOPS=80, BW=321KiB/s (329kB/s)(5060KiB/15761msec) slat (usec): min=624, max=3497, avg=1187.02, stdev=290.21 clat (msec): min=10, max=102, avg=63.91, stdev=11.40 lat (msec): min=11, max=103, avg=65.10, stdev=11.42 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 67], | 70.00th=[ 70], 80.00th=[ 73], 90.00th=[ 79], 95.00th=[ 82], | 99.00th=[ 88], 99.50th=[ 90], 99.90th=[ 95], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 232, max= 416, per=25.39%, avg=321.45, stdev=52.52, samples=31 iops : min= 58, max= 104, avg=80.29, stdev=13.16, samples=31 write: IOPS=82, BW=329KiB/s (337kB/s)(5180KiB/15761msec) slat (usec): min=13, max=298, avg=58.82, stdev=31.35 clat (msec): min=26, max=214, avg=130.85, stdev=29.15 lat (msec): min=26, max=214, avg=130.91, stdev=29.15 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 84], 10.00th=[ 94], 20.00th=[ 107], | 30.00th=[ 116], 40.00th=[ 124], 50.00th=[ 130], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 167], 95.00th=[ 182], | 99.00th=[ 207], 99.50th=[ 211], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 256, max= 400, per=24.84%, avg=327.39, stdev=32.94, samples=31 iops : min= 64, max= 100, avg=81.77, stdev= 8.26, samples=31 lat (msec) : 20=0.47%, 50=3.87%, 100=51.88%, 250=43.79% cpu : usr=0.23%, sys=1.57%, ctx=3180, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1266KiB/s (1297kB/s), 314KiB/s-321KiB/s (322kB/s-329kB/s), io=19.6MiB (20.5MB), run=15760-15850msec WRITE: bw=1318KiB/s (1350kB/s), 329KiB/s-333KiB/s (337kB/s-341kB/s), io=20.4MiB (21.4MB), run=15760-15850msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=2678: Mon Dec 9 22:28:11 2024 read: IOPS=14, BW=571MiB/s (599MB/s)(40.0MiB/70msec) slat (nsec): min=5823.2k, max=5823.2k, avg=5823175.00, stdev= 0.00 clat (nsec): min=60611k, max=60611k, avg=60610853.00, stdev= 0.00 lat (nsec): min=66444k, max=66444k, avg=66444137.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[60556], 5.00th=[60556], 10.00th=[60556], 20.00th=[60556], | 30.00th=[60556], 40.00th=[60556], 50.00th=[60556], 60.00th=[60556], | 70.00th=[60556], 80.00th=[60556], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.25%, ctx=16, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=571MiB/s (599MB/s), 571MiB/s-571MiB/s (599MB/s-599MB/s), io=40.0MiB (41.9MB), run=70-70msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout