== sanity test 398c: run fio to test AIO ================= 16:27:42 (1738963662) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.124039 s, 338 MB/s osc.lustre-OST0000-osc-ffff880134a54800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=27170: Fri Feb 7 16:27:51 2025 write: IOPS=362, BW=1448KiB/s (1483kB/s)(10.0MiB/7070msec) slat (usec): min=14, max=487, avg=85.21, stdev=47.67 clat (usec): min=8429, max=80079, avg=43939.15, stdev=5931.85 lat (usec): min=8562, max=80203, avg=44026.54, stdev=5934.32 clat percentiles (usec): | 1.00th=[28705], 5.00th=[32113], 10.00th=[36439], 20.00th=[40109], | 30.00th=[41681], 40.00th=[43254], 50.00th=[44827], 60.00th=[45876], | 70.00th=[46924], 80.00th=[48497], 90.00th=[49546], 95.00th=[51643], | 99.00th=[57410], 99.50th=[57934], 99.90th=[70779], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 1277, max= 1788, per=24.97%, avg=1446.36, stdev=138.82, samples=14 iops : min= 319, max= 447, avg=361.57, stdev=34.73, samples=14 lat (msec) : 10=0.04%, 50=91.25%, 100=8.71% cpu : usr=0.66%, sys=3.52%, ctx=1495, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27171: Fri Feb 7 16:27:51 2025 write: IOPS=362, BW=1448KiB/s (1483kB/s)(10.0MiB/7071msec) slat (usec): min=21, max=381, avg=78.56, stdev=42.30 clat (usec): min=7364, max=69955, avg=43872.98, stdev=5898.15 lat (usec): min=7504, max=70036, avg=43953.55, stdev=5905.65 clat percentiles (usec): | 1.00th=[28705], 5.00th=[31065], 10.00th=[36439], 20.00th=[39584], | 30.00th=[42206], 40.00th=[43779], 50.00th=[44827], 60.00th=[45876], | 70.00th=[46924], 80.00th=[48497], 90.00th=[50070], 95.00th=[51119], | 99.00th=[55837], 99.50th=[57410], 99.90th=[64750], 99.95th=[64750], | 99.99th=[69731] bw ( KiB/s): min= 1288, max= 1800, per=24.96%, avg=1445.50, stdev=143.63, samples=14 iops : min= 322, max= 450, avg=361.36, stdev=35.92, samples=14 lat (msec) : 10=0.08%, 50=90.27%, 100=9.65% cpu : usr=0.47%, sys=3.54%, ctx=983, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27172: Fri Feb 7 16:27:51 2025 write: IOPS=363, BW=1454KiB/s (1489kB/s)(10.0MiB/7043msec) slat (usec): min=14, max=479, avg=87.21, stdev=45.43 clat (usec): min=5176, max=93337, avg=43778.62, stdev=7017.24 lat (usec): min=5289, max=93474, avg=43867.72, stdev=7019.44 clat percentiles (usec): | 1.00th=[22152], 5.00th=[30802], 10.00th=[35390], 20.00th=[40109], | 30.00th=[42206], 40.00th=[43254], 50.00th=[44827], 60.00th=[45876], | 70.00th=[46924], 80.00th=[48497], 90.00th=[50070], 95.00th=[51643], | 99.00th=[57410], 99.50th=[63701], 99.90th=[92799], 99.95th=[92799], | 99.99th=[93848] bw ( KiB/s): min= 1285, max= 1884, per=25.10%, avg=1453.57, stdev=163.30, samples=14 iops : min= 321, max= 471, avg=363.36, stdev=40.84, samples=14 lat (msec) : 10=0.20%, 20=0.39%, 50=89.61%, 100=9.80% cpu : usr=0.57%, sys=3.88%, ctx=1437, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27173: Fri Feb 7 16:27:51 2025 write: IOPS=362, BW=1449KiB/s (1484kB/s)(10.0MiB/7066msec) slat (usec): min=13, max=611, avg=86.64, stdev=47.09 clat (usec): min=10665, max=71238, avg=43873.86, stdev=5910.30 lat (usec): min=10815, max=71384, avg=43962.52, stdev=5913.41 clat percentiles (usec): | 1.00th=[28443], 5.00th=[30802], 10.00th=[35914], 20.00th=[40109], | 30.00th=[42206], 40.00th=[43254], 50.00th=[44827], 60.00th=[45876], | 70.00th=[46924], 80.00th=[48497], 90.00th=[49546], 95.00th=[51643], | 99.00th=[56361], 99.50th=[57934], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 1296, max= 1792, per=24.98%, avg=1446.57, stdev=134.54, samples=14 iops : min= 324, max= 448, avg=361.64, stdev=33.64, samples=14 lat (msec) : 20=0.04%, 50=91.91%, 100=8.05% cpu : usr=0.45%, sys=3.72%, ctx=1543, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=5793KiB/s (5932kB/s), 1448KiB/s-1454KiB/s (1483kB/s-1489kB/s), io=40.0MiB (41.9MB), run=7043-7071msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27189: Fri Feb 7 16:27:55 2025 read: IOPS=337, BW=1349KiB/s (1381kB/s)(5048KiB/3743msec) slat (usec): min=574, max=4395, avg=1534.13, stdev=593.84 clat (usec): min=5939, max=44919, avg=22078.58, stdev=8234.26 lat (usec): min=6698, max=46921, avg=23615.45, stdev=8703.84 clat percentiles (usec): | 1.00th=[11207], 5.00th=[12256], 10.00th=[12911], 20.00th=[14222], | 30.00th=[15270], 40.00th=[16450], 50.00th=[19006], 60.00th=[25035], | 70.00th=[27919], 80.00th=[30802], 90.00th=[33817], 95.00th=[35390], | 99.00th=[39584], 99.50th=[41681], 99.90th=[43254], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 928, max= 2080, per=25.47%, avg=1365.29, stdev=455.81, samples=7 iops : min= 232, max= 520, avg=341.29, stdev=113.95, samples=7 write: IOPS=346, BW=1387KiB/s (1420kB/s)(5192KiB/3743msec) slat (usec): min=13, max=283, avg=62.53, stdev=36.17 clat (usec): min=11838, max=43431, avg=22568.38, stdev=8112.47 lat (usec): min=11864, max=43493, avg=22632.32, stdev=8128.80 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13304], 10.00th=[13960], 20.00th=[14746], | 30.00th=[15533], 40.00th=[16319], 50.00th=[19006], 60.00th=[26346], | 70.00th=[29492], 80.00th=[31327], 90.00th=[33817], 95.00th=[35390], | 99.00th=[39060], 99.50th=[39584], 99.90th=[41157], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 936, max= 2184, per=25.26%, avg=1409.86, stdev=542.17, samples=7 iops : min= 234, max= 546, avg=352.43, stdev=135.54, samples=7 lat (msec) : 10=0.08%, 20=51.09%, 50=48.83% cpu : usr=0.99%, sys=8.12%, ctx=2726, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27190: Fri Feb 7 16:27:55 2025 read: IOPS=332, BW=1330KiB/s (1362kB/s)(4976KiB/3740msec) slat (usec): min=634, max=3840, avg=1532.49, stdev=593.28 clat (usec): min=10059, max=42372, avg=22088.88, stdev=8172.81 lat (usec): min=11094, max=44445, avg=23624.13, stdev=8657.98 clat percentiles (usec): | 1.00th=[11207], 5.00th=[12518], 10.00th=[13173], 20.00th=[14353], | 30.00th=[15270], 40.00th=[16319], 50.00th=[19006], 60.00th=[25035], | 70.00th=[28181], 80.00th=[30802], 90.00th=[33817], 95.00th=[35390], | 99.00th=[39584], 99.50th=[40109], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 960, max= 2088, per=25.02%, avg=1341.29, stdev=473.92, samples=7 iops : min= 240, max= 522, avg=335.29, stdev=118.48, samples=7 write: IOPS=351, BW=1407KiB/s (1441kB/s)(5264KiB/3740msec) slat (usec): min=12, max=623, avg=65.57, stdev=45.04 clat (usec): min=11367, max=40949, avg=22561.08, stdev=7975.57 lat (usec): min=11402, max=41008, avg=22628.44, stdev=7992.66 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13435], 10.00th=[13960], 20.00th=[14877], | 30.00th=[15533], 40.00th=[16319], 50.00th=[19530], 60.00th=[26084], | 70.00th=[29492], 80.00th=[31327], 90.00th=[33162], 95.00th=[34866], | 99.00th=[38011], 99.50th=[39060], 99.90th=[40109], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 944, max= 2120, per=25.67%, avg=1432.57, stdev=505.10, samples=7 iops : min= 236, max= 530, avg=358.14, stdev=126.28, samples=7 lat (msec) : 20=51.02%, 50=48.98% cpu : usr=0.86%, sys=8.21%, ctx=2829, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27191: Fri Feb 7 16:27:55 2025 read: IOPS=334, BW=1339KiB/s (1371kB/s)(4984KiB/3722msec) slat (usec): min=647, max=4152, avg=1510.17, stdev=584.60 clat (usec): min=7619, max=49973, avg=21659.58, stdev=8038.34 lat (usec): min=9521, max=52775, avg=23172.37, stdev=8508.72 clat percentiles (usec): | 1.00th=[10552], 5.00th=[12387], 10.00th=[13042], 20.00th=[14222], | 30.00th=[15270], 40.00th=[16319], 50.00th=[17957], 60.00th=[24249], | 70.00th=[27132], 80.00th=[30278], 90.00th=[33162], 95.00th=[35390], | 99.00th=[39060], 99.50th=[40109], 99.90th=[42206], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 848, max= 2032, per=25.47%, avg=1365.29, stdev=492.77, samples=7 iops : min= 212, max= 508, avg=341.29, stdev=123.18, samples=7 write: IOPS=353, BW=1412KiB/s (1446kB/s)(5256KiB/3722msec) slat (usec): min=14, max=1797, avg=66.53, stdev=64.00 clat (usec): min=7316, max=49789, avg=22934.52, stdev=8225.01 lat (usec): min=7457, max=49828, avg=23002.54, stdev=8245.39 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13566], 10.00th=[14091], 20.00th=[14877], | 30.00th=[15533], 40.00th=[16450], 50.00th=[20579], 60.00th=[27395], | 70.00th=[30016], 80.00th=[31851], 90.00th=[33817], 95.00th=[35390], | 99.00th=[38011], 99.50th=[39584], 99.90th=[47973], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 928, max= 2232, per=25.53%, avg=1424.71, stdev=494.27, samples=7 iops : min= 232, max= 558, avg=356.14, stdev=123.57, samples=7 lat (msec) : 10=0.23%, 20=51.25%, 50=48.52% cpu : usr=1.02%, sys=8.09%, ctx=2705, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27192: Fri Feb 7 16:27:55 2025 read: IOPS=342, BW=1370KiB/s (1403kB/s)(5060KiB/3693msec) slat (usec): min=67, max=4476, avg=1518.88, stdev=595.96 clat (usec): min=3327, max=47854, avg=21806.47, stdev=8232.90 lat (usec): min=3587, max=50164, avg=23328.19, stdev=8705.91 clat percentiles (usec): | 1.00th=[10683], 5.00th=[12125], 10.00th=[12911], 20.00th=[13960], | 30.00th=[15139], 40.00th=[16188], 50.00th=[18482], 60.00th=[24773], | 70.00th=[28181], 80.00th=[30540], 90.00th=[33162], 95.00th=[35390], | 99.00th=[39584], 99.50th=[42206], 99.90th=[45876], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 920, max= 2024, per=25.89%, avg=1388.14, stdev=461.84, samples=7 iops : min= 230, max= 506, avg=347.00, stdev=115.45, samples=7 write: IOPS=350, BW=1403KiB/s (1436kB/s)(5180KiB/3693msec) slat (usec): min=14, max=1708, avg=68.01, stdev=68.03 clat (usec): min=9185, max=56003, avg=22642.00, stdev=8160.07 lat (usec): min=9330, max=56077, avg=22711.51, stdev=8180.73 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13566], 10.00th=[14091], 20.00th=[14877], | 30.00th=[15664], 40.00th=[16450], 50.00th=[19530], 60.00th=[26084], | 70.00th=[29492], 80.00th=[31589], 90.00th=[33424], 95.00th=[34866], | 99.00th=[38536], 99.50th=[40109], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 976, max= 2224, per=25.47%, avg=1421.29, stdev=518.09, samples=7 iops : min= 244, max= 556, avg=355.29, stdev=129.53, samples=7 lat (msec) : 4=0.08%, 10=0.27%, 20=51.21%, 50=48.32%, 100=0.12% cpu : usr=1.11%, sys=8.18%, ctx=2725, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5361KiB/s (5490kB/s), 1330KiB/s-1370KiB/s (1362kB/s-1403kB/s), io=19.6MiB (20.5MB), run=3693-3743msec WRITE: bw=5582KiB/s (5716kB/s), 1387KiB/s-1412KiB/s (1420kB/s-1446kB/s), io=20.4MiB (21.4MB), run=3693-3743msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=27199: Fri Feb 7 16:27:57 2025 read: IOPS=5, BW=237MiB/s (248MB/s)(40.0MiB/169msec) slat (nsec): min=5574.6k, max=5574.6k, avg=5574618.00, stdev= 0.00 clat (nsec): min=156438k, max=156438k, avg=156438209.00, stdev= 0.00 lat (nsec): min=162075k, max=162075k, avg=162074930.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=2.98%, ctx=9, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=237MiB/s (248MB/s), 237MiB/s-237MiB/s (248MB/s-248MB/s), io=40.0MiB (41.9MB), run=169-169msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout