== sanity test 398c: run fio to test AIO ================= 22:09:09 (1749607749) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.14784 s, 284 MB/s osc.lustre-OST0000-osc-ffff8800b38f2000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=23036: Tue Jun 10 22:09:20 2025 write: IOPS=281, BW=1125KiB/s (1152kB/s)(10.0MiB/9103msec) slat (usec): min=11, max=2697, avg=88.09, stdev=138.28 clat (msec): min=9, max=108, avg=56.63, stdev=11.28 lat (msec): min=10, max=108, avg=56.72, stdev=11.28 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 90], 99.50th=[ 99], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 896, max= 1400, per=25.13%, avg=1126.17, stdev=126.17, samples=18 iops : min= 224, max= 350, avg=281.50, stdev=31.53, samples=18 lat (msec) : 10=0.04%, 20=0.16%, 50=27.93%, 100=71.64%, 250=0.23% cpu : usr=1.37%, sys=2.76%, ctx=1091, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23037: Tue Jun 10 22:09:20 2025 write: IOPS=280, BW=1122KiB/s (1149kB/s)(10.0MiB/9129msec) slat (usec): min=11, max=1756, avg=97.05, stdev=157.55 clat (msec): min=14, max=120, avg=56.75, stdev=10.68 lat (msec): min=14, max=120, avg=56.85, stdev=10.68 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 58], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 75], | 99.00th=[ 86], 99.50th=[ 92], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 896, max= 1368, per=25.04%, avg=1121.89, stdev=124.21, samples=18 iops : min= 224, max= 342, avg=280.44, stdev=31.04, samples=18 lat (msec) : 20=0.04%, 50=27.19%, 100=72.58%, 250=0.20% cpu : usr=0.55%, sys=3.68%, ctx=1294, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23038: Tue Jun 10 22:09:20 2025 write: IOPS=280, BW=1121KiB/s (1148kB/s)(10.0MiB/9134msec) slat (usec): min=12, max=2830, avg=95.93, stdev=169.64 clat (msec): min=5, max=105, avg=56.74, stdev=10.82 lat (msec): min=5, max=105, avg=56.83, stdev=10.82 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 87], 99.50th=[ 88], 99.90th=[ 102], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 904, max= 1320, per=25.04%, avg=1122.11, stdev=124.37, samples=18 iops : min= 226, max= 330, avg=280.44, stdev=31.13, samples=18 lat (msec) : 10=0.08%, 20=0.16%, 50=26.95%, 100=72.54%, 250=0.27% cpu : usr=0.00%, sys=4.07%, ctx=1165, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23039: Tue Jun 10 22:09:20 2025 write: IOPS=280, BW=1120KiB/s (1147kB/s)(10.0MiB/9140msec) slat (usec): min=10, max=1625, avg=88.08, stdev=140.51 clat (usec): min=10531, max=90652, avg=56747.35, stdev=10564.81 lat (usec): min=10693, max=90692, avg=56837.43, stdev=10564.16 clat percentiles (usec): | 1.00th=[34341], 5.00th=[41681], 10.00th=[44827], 20.00th=[47973], | 30.00th=[50594], 40.00th=[53216], 50.00th=[55837], 60.00th=[58459], | 70.00th=[61080], 80.00th=[64750], 90.00th=[70779], 95.00th=[76022], | 99.00th=[85459], 99.50th=[87557], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 896, max= 1384, per=24.92%, avg=1116.44, stdev=128.59, samples=18 iops : min= 224, max= 346, avg=279.11, stdev=32.15, samples=18 lat (msec) : 20=0.12%, 50=26.99%, 100=72.89% cpu : usr=1.51%, sys=2.28%, ctx=1073, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4481KiB/s (4589kB/s), 1120KiB/s-1125KiB/s (1147kB/s-1152kB/s), io=40.0MiB (41.9MB), run=9103-9140msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23056: Tue Jun 10 22:09:27 2025 read: IOPS=197, BW=789KiB/s (808kB/s)(5048KiB/6399msec) slat (usec): min=1019, max=9498, avg=2473.79, stdev=840.24 clat (usec): min=2869, max=83221, avg=38038.92, stdev=7079.41 lat (usec): min=4948, max=85676, avg=40516.88, stdev=7268.26 clat percentiles (usec): | 1.00th=[25297], 5.00th=[27919], 10.00th=[29754], 20.00th=[32375], | 30.00th=[33817], 40.00th=[35390], 50.00th=[37487], 60.00th=[39060], | 70.00th=[41157], 80.00th=[43254], 90.00th=[46924], 95.00th=[50070], | 99.00th=[58459], 99.50th=[61080], 99.90th=[80217], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 696, max= 886, per=25.25%, avg=787.83, stdev=61.44, samples=12 iops : min= 174, max= 221, avg=196.92, stdev=15.29, samples=12 write: IOPS=202, BW=811KiB/s (831kB/s)(5192KiB/6399msec) slat (usec): min=13, max=2039, avg=81.60, stdev=121.62 clat (usec): min=7845, max=69402, avg=38807.13, stdev=6187.65 lat (usec): min=7917, max=69446, avg=38890.55, stdev=6191.09 clat percentiles (usec): | 1.00th=[26346], 5.00th=[29492], 10.00th=[31589], 20.00th=[33817], | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[40109], | 70.00th=[41681], 80.00th=[43779], 90.00th=[46924], 95.00th=[49546], | 99.00th=[55313], 99.50th=[57934], 99.90th=[61604], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 656, max= 936, per=25.05%, avg=813.83, stdev=86.66, samples=12 iops : min= 164, max= 234, avg=203.42, stdev=21.65, samples=12 lat (msec) : 4=0.04%, 10=0.04%, 50=95.00%, 100=4.92% cpu : usr=0.81%, sys=7.46%, ctx=2351, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23057: Tue Jun 10 22:09:27 2025 read: IOPS=195, BW=781KiB/s (800kB/s)(4976KiB/6369msec) slat (usec): min=25, max=8268, avg=2474.07, stdev=873.22 clat (usec): min=4522, max=85834, avg=38116.64, stdev=7601.13 lat (usec): min=4578, max=88939, avg=40594.19, stdev=7777.04 clat percentiles (usec): | 1.00th=[25035], 5.00th=[28181], 10.00th=[29754], 20.00th=[32113], | 30.00th=[33817], 40.00th=[35390], 50.00th=[37487], 60.00th=[39060], | 70.00th=[41157], 80.00th=[43254], 90.00th=[47449], 95.00th=[51119], | 99.00th=[61080], 99.50th=[73925], 99.90th=[81265], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 670, max= 872, per=24.95%, avg=778.33, stdev=69.71, samples=12 iops : min= 167, max= 218, avg=194.50, stdev=17.50, samples=12 write: IOPS=206, BW=827KiB/s (846kB/s)(5264KiB/6369msec) slat (usec): min=14, max=858, avg=74.06, stdev=81.65 clat (usec): min=11073, max=85752, avg=38796.51, stdev=6660.69 lat (usec): min=11099, max=85815, avg=38872.17, stdev=6659.79 clat percentiles (usec): | 1.00th=[25560], 5.00th=[29492], 10.00th=[31065], 20.00th=[33424], | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[40109], | 70.00th=[41681], 80.00th=[43779], 90.00th=[46924], 95.00th=[49021], | 99.00th=[56886], 99.50th=[63701], 99.90th=[78119], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 712, max= 1048, per=25.72%, avg=835.67, stdev=88.10, samples=12 iops : min= 178, max= 262, avg=208.83, stdev=22.00, samples=12 lat (msec) : 10=0.04%, 20=0.23%, 50=94.92%, 100=4.80% cpu : usr=0.46%, sys=7.65%, ctx=2288, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23058: Tue Jun 10 22:09:27 2025 read: IOPS=193, BW=775KiB/s (794kB/s)(4984KiB/6430msec) slat (usec): min=1053, max=7643, avg=2503.07, stdev=856.06 clat (usec): min=7241, max=69710, avg=38009.65, stdev=6986.83 lat (usec): min=9711, max=73069, avg=40516.74, stdev=7198.47 clat percentiles (usec): | 1.00th=[25297], 5.00th=[28443], 10.00th=[29754], 20.00th=[32113], | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060], | 70.00th=[41157], 80.00th=[43779], 90.00th=[46400], 95.00th=[50070], | 99.00th=[58983], 99.50th=[64226], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 616, max= 920, per=25.02%, avg=780.67, stdev=95.79, samples=12 iops : min= 154, max= 230, avg=195.17, stdev=23.95, samples=12 write: IOPS=204, BW=817KiB/s (837kB/s)(5256KiB/6430msec) slat (usec): min=12, max=1337, avg=80.60, stdev=107.26 clat (usec): min=22710, max=66997, avg=39209.19, stdev=6048.47 lat (usec): min=22793, max=67040, avg=39291.15, stdev=6045.62 clat percentiles (usec): | 1.00th=[27132], 5.00th=[30278], 10.00th=[31589], 20.00th=[33817], | 30.00th=[35914], 40.00th=[37487], 50.00th=[38536], 60.00th=[40109], | 70.00th=[42206], 80.00th=[44303], 90.00th=[46924], 95.00th=[49546], | 99.00th=[54789], 99.50th=[58983], 99.90th=[64750], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 608, max= 928, per=25.03%, avg=813.33, stdev=93.69, samples=12 iops : min= 152, max= 232, avg=203.33, stdev=23.42, samples=12 lat (msec) : 10=0.04%, 50=95.47%, 100=4.49% cpu : usr=2.26%, sys=5.65%, ctx=2307, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23059: Tue Jun 10 22:09:27 2025 read: IOPS=196, BW=787KiB/s (806kB/s)(5060KiB/6430msec) slat (usec): min=1133, max=7006, avg=2490.05, stdev=821.60 clat (usec): min=4311, max=68872, avg=37897.16, stdev=6734.37 lat (usec): min=6438, max=73283, avg=40390.70, stdev=6921.07 clat percentiles (usec): | 1.00th=[24511], 5.00th=[28181], 10.00th=[30016], 20.00th=[32113], | 30.00th=[34341], 40.00th=[35390], 50.00th=[37487], 60.00th=[39060], | 70.00th=[41157], 80.00th=[43254], 90.00th=[46924], 95.00th=[49021], | 99.00th=[56361], 99.50th=[57410], 99.90th=[64226], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 696, max= 936, per=25.27%, avg=788.50, stdev=79.33, samples=12 iops : min= 174, max= 234, avg=197.08, stdev=19.81, samples=12 write: IOPS=201, BW=806KiB/s (825kB/s)(5180KiB/6430msec) slat (usec): min=12, max=4001, avg=85.48, stdev=186.91 clat (usec): min=24730, max=76212, avg=39438.60, stdev=6386.83 lat (usec): min=24777, max=76232, avg=39525.61, stdev=6398.51 clat percentiles (usec): | 1.00th=[26870], 5.00th=[29754], 10.00th=[32113], 20.00th=[34341], | 30.00th=[35914], 40.00th=[37487], 50.00th=[39060], 60.00th=[40633], | 70.00th=[42206], 80.00th=[44827], 90.00th=[47449], 95.00th=[49546], | 99.00th=[56361], 99.50th=[64226], 99.90th=[74974], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 704, max= 936, per=24.86%, avg=807.83, stdev=74.91, samples=12 iops : min= 176, max= 234, avg=201.92, stdev=18.73, samples=12 lat (msec) : 10=0.08%, 50=95.66%, 100=4.26% cpu : usr=0.39%, sys=7.64%, ctx=2370, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3121KiB/s (3196kB/s), 775KiB/s-789KiB/s (794kB/s-808kB/s), io=19.6MiB (20.5MB), run=6369-6430msec WRITE: bw=3249KiB/s (3327kB/s), 806KiB/s-827KiB/s (825kB/s-846kB/s), io=20.4MiB (21.4MB), run=6369-6430msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=23068: Tue Jun 10 22:09:29 2025 read: IOPS=6, BW=276MiB/s (289MB/s)(40.0MiB/145msec) slat (nsec): min=8980.2k, max=8980.2k, avg=8980225.00, stdev= 0.00 clat (nsec): min=129922k, max=129922k, avg=129922481.00, stdev= 0.00 lat (nsec): min=138929k, max=138929k, avg=138928601.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.56%, ctx=9, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=40.0MiB (41.9MB), run=145-145msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout