== sanity test 398c: run fio to test AIO ================= 07:40:21 (1711107621) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.187382 s, 224 MB/s osc.lustre-OST0000-osc-ffff880099b0e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=8923: Fri Mar 22 07:40:37 2024 write: IOPS=188, BW=753KiB/s (771kB/s)(10.0MiB/13593msec) slat (usec): min=13, max=2864, avg=105.75, stdev=206.19 clat (msec): min=14, max=135, avg=84.59, stdev=14.33 lat (msec): min=14, max=135, avg=84.69, stdev=14.34 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 87], | 70.00th=[ 91], 80.00th=[ 95], 90.00th=[ 104], 95.00th=[ 112], | 99.00th=[ 125], 99.50th=[ 128], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 590, max= 880, per=24.92%, avg=750.48, stdev=75.77, samples=27 iops : min= 147, max= 220, avg=187.48, stdev=18.95, samples=27 lat (msec) : 20=0.12%, 50=0.04%, 100=87.42%, 250=12.42% cpu : usr=0.96%, sys=2.49%, ctx=1082, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8924: Fri Mar 22 07:40:37 2024 write: IOPS=188, BW=754KiB/s (773kB/s)(10.0MiB/13572msec) slat (usec): min=11, max=3756, avg=105.50, stdev=214.68 clat (msec): min=9, max=148, avg=84.47, stdev=15.42 lat (msec): min=9, max=148, avg=84.58, stdev=15.42 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 87], | 70.00th=[ 90], 80.00th=[ 96], 90.00th=[ 107], 95.00th=[ 113], | 99.00th=[ 123], 99.50th=[ 127], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 590, max= 888, per=24.97%, avg=751.89, stdev=79.10, samples=27 iops : min= 147, max= 222, avg=187.74, stdev=19.82, samples=27 lat (msec) : 10=0.08%, 20=0.20%, 50=0.59%, 100=84.06%, 250=15.08% cpu : usr=0.46%, sys=2.77%, ctx=1030, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8925: Fri Mar 22 07:40:37 2024 write: IOPS=188, BW=755KiB/s (773kB/s)(10.0MiB/13565msec) slat (usec): min=13, max=2719, avg=104.93, stdev=197.01 clat (msec): min=19, max=136, avg=84.43, stdev=15.19 lat (msec): min=19, max=136, avg=84.53, stdev=15.18 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 77], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 87], | 70.00th=[ 91], 80.00th=[ 96], 90.00th=[ 105], 95.00th=[ 111], | 99.00th=[ 126], 99.50th=[ 131], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 632, max= 870, per=25.00%, avg=752.74, stdev=72.99, samples=27 iops : min= 158, max= 217, avg=188.00, stdev=18.21, samples=27 lat (msec) : 20=0.62%, 50=0.27%, 100=85.66%, 250=13.44% cpu : usr=0.63%, sys=2.69%, ctx=1118, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8926: Fri Mar 22 07:40:37 2024 write: IOPS=188, BW=753KiB/s (771kB/s)(10.0MiB/13603msec) slat (usec): min=11, max=2401, avg=101.33, stdev=207.79 clat (msec): min=46, max=148, avg=84.67, stdev=14.49 lat (msec): min=46, max=149, avg=84.78, stdev=14.49 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 77], 40.00th=[ 80], 50.00th=[ 83], 60.00th=[ 88], | 70.00th=[ 91], 80.00th=[ 97], 90.00th=[ 105], 95.00th=[ 111], | 99.00th=[ 126], 99.50th=[ 134], 99.90th=[ 136], 99.95th=[ 148], | 99.99th=[ 150] bw ( KiB/s): min= 630, max= 896, per=24.87%, avg=748.96, stdev=68.11, samples=27 iops : min= 157, max= 224, avg=187.00, stdev=17.06, samples=27 lat (msec) : 50=0.04%, 100=86.91%, 250=13.05% cpu : usr=0.13%, sys=2.82%, ctx=907, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3011KiB/s (3083kB/s), 753KiB/s-755KiB/s (771kB/s-773kB/s), io=40.0MiB (41.9MB), run=13565-13603msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8947: Fri Mar 22 07:40:48 2024 read: IOPS=133, BW=535KiB/s (548kB/s)(5048KiB/9436msec) slat (usec): min=15, max=13641, avg=3741.96, stdev=1424.29 clat (usec): min=23569, max=91805, avg=55573.15, stdev=10906.01 lat (usec): min=27240, max=94128, avg=59321.13, stdev=11118.52 clat percentiles (usec): | 1.00th=[32113], 5.00th=[39060], 10.00th=[42730], 20.00th=[46400], | 30.00th=[49021], 40.00th=[51643], 50.00th=[54789], 60.00th=[57410], | 70.00th=[60556], 80.00th=[64750], 90.00th=[70779], 95.00th=[74974], | 99.00th=[82314], 99.50th=[83362], 99.90th=[89654], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 431, max= 656, per=25.20%, avg=534.39, stdev=51.92, samples=18 iops : min= 107, max= 164, avg=133.50, stdev=13.06, samples=18 write: IOPS=137, BW=550KiB/s (563kB/s)(5192KiB/9436msec) slat (usec): min=14, max=4916, avg=100.81, stdev=243.92 clat (usec): min=9783, max=92520, avg=58422.18, stdev=10414.46 lat (usec): min=9954, max=92612, avg=58525.35, stdev=10426.09 clat percentiles (usec): | 1.00th=[34866], 5.00th=[42730], 10.00th=[46400], 20.00th=[50070], | 30.00th=[52691], 40.00th=[54789], 50.00th=[57934], 60.00th=[61080], | 70.00th=[63701], 80.00th=[67634], 90.00th=[71828], 95.00th=[76022], | 99.00th=[82314], 99.50th=[84411], 99.90th=[90702], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 448, max= 694, per=24.95%, avg=550.89, stdev=69.19, samples=18 iops : min= 112, max= 173, avg=137.61, stdev=17.33, samples=18 lat (msec) : 10=0.04%, 20=0.08%, 50=26.13%, 100=73.75% cpu : usr=0.69%, sys=6.47%, ctx=2304, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8948: Fri Mar 22 07:40:48 2024 read: IOPS=131, BW=527KiB/s (540kB/s)(4976KiB/9443msec) slat (usec): min=22, max=11642, avg=3863.68, stdev=1469.20 clat (usec): min=11365, max=93239, avg=55845.28, stdev=11608.82 lat (usec): min=12359, max=98285, avg=59713.74, stdev=11941.73 clat percentiles (usec): | 1.00th=[29230], 5.00th=[38536], 10.00th=[42206], 20.00th=[46400], | 30.00th=[49546], 40.00th=[52691], 50.00th=[55313], 60.00th=[58459], | 70.00th=[61080], 80.00th=[64750], 90.00th=[71828], 95.00th=[77071], | 99.00th=[83362], 99.50th=[86508], 99.90th=[90702], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 424, max= 616, per=24.72%, avg=524.39, stdev=56.80, samples=18 iops : min= 106, max= 154, avg=130.94, stdev=14.22, samples=18 write: IOPS=139, BW=557KiB/s (571kB/s)(5264KiB/9443msec) slat (usec): min=13, max=4753, avg=93.51, stdev=189.27 clat (msec): min=20, max=102, avg=58.18, stdev=10.58 lat (msec): min=20, max=102, avg=58.28, stdev=10.58 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 87], 99.50th=[ 88], 99.90th=[ 99], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 415, max= 640, per=25.28%, avg=558.22, stdev=58.81, samples=18 iops : min= 103, max= 160, avg=139.39, stdev=14.87, samples=18 lat (msec) : 20=0.27%, 50=25.23%, 100=74.45%, 250=0.04% cpu : usr=0.00%, sys=7.30%, ctx=2289, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8949: Fri Mar 22 07:40:48 2024 read: IOPS=132, BW=529KiB/s (541kB/s)(4984KiB/9429msec) slat (usec): min=14, max=13911, avg=3846.08, stdev=1451.01 clat (usec): min=23641, max=91799, avg=55583.19, stdev=10891.89 lat (usec): min=26085, max=96344, avg=59434.76, stdev=11185.32 clat percentiles (usec): | 1.00th=[32113], 5.00th=[39060], 10.00th=[42206], 20.00th=[45876], | 30.00th=[49021], 40.00th=[52167], 50.00th=[55313], 60.00th=[58459], | 70.00th=[61080], 80.00th=[64226], 90.00th=[69731], 95.00th=[73925], | 99.00th=[84411], 99.50th=[87557], 99.90th=[90702], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 416, max= 686, per=25.07%, avg=531.72, stdev=70.59, samples=18 iops : min= 104, max= 171, avg=132.72, stdev=17.59, samples=18 write: IOPS=139, BW=557KiB/s (571kB/s)(5256KiB/9429msec) slat (usec): min=13, max=2228, avg=92.73, stdev=157.39 clat (usec): min=25019, max=92898, avg=58249.48, stdev=10438.45 lat (usec): min=25037, max=92924, avg=58343.94, stdev=10433.29 clat percentiles (usec): | 1.00th=[35390], 5.00th=[42206], 10.00th=[45876], 20.00th=[49546], | 30.00th=[52167], 40.00th=[54789], 50.00th=[57410], 60.00th=[60031], | 70.00th=[63177], 80.00th=[66847], 90.00th=[71828], 95.00th=[77071], | 99.00th=[84411], 99.50th=[87557], 99.90th=[89654], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 464, max= 664, per=25.07%, avg=553.44, stdev=50.24, samples=18 iops : min= 116, max= 166, avg=138.17, stdev=12.53, samples=18 lat (msec) : 50=26.91%, 100=73.09% cpu : usr=0.00%, sys=7.09%, ctx=2295, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8950: Fri Mar 22 07:40:48 2024 read: IOPS=133, BW=535KiB/s (548kB/s)(5060KiB/9461msec) slat (usec): min=26, max=12068, avg=3806.71, stdev=1488.11 clat (usec): min=10845, max=94005, avg=55776.02, stdev=11855.17 lat (usec): min=13992, max=99913, avg=59588.85, stdev=12093.10 clat percentiles (usec): | 1.00th=[25822], 5.00th=[38536], 10.00th=[41681], 20.00th=[45876], | 30.00th=[49021], 40.00th=[52691], 50.00th=[55837], 60.00th=[58459], | 70.00th=[61604], 80.00th=[65799], 90.00th=[70779], 95.00th=[74974], | 99.00th=[84411], 99.50th=[85459], 99.90th=[90702], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 424, max= 622, per=24.98%, avg=529.89, stdev=66.34, samples=18 iops : min= 106, max= 155, avg=132.28, stdev=16.50, samples=18 write: IOPS=136, BW=548KiB/s (561kB/s)(5180KiB/9461msec) slat (usec): min=11, max=2072, avg=87.45, stdev=127.42 clat (usec): min=14482, max=94752, avg=58501.36, stdev=10763.71 lat (usec): min=14545, max=95147, avg=58590.26, stdev=10762.39 clat percentiles (usec): | 1.00th=[35390], 5.00th=[42730], 10.00th=[45876], 20.00th=[49546], | 30.00th=[52691], 40.00th=[55313], 50.00th=[57934], 60.00th=[60556], | 70.00th=[63701], 80.00th=[67634], 90.00th=[72877], 95.00th=[76022], | 99.00th=[85459], 99.50th=[90702], 99.90th=[91751], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 456, max= 710, per=24.75%, avg=546.44, stdev=68.12, samples=18 iops : min= 114, max= 177, avg=136.39, stdev=17.00, samples=18 lat (msec) : 20=0.55%, 50=26.37%, 100=73.09% cpu : usr=0.63%, sys=6.64%, ctx=2278, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2121KiB/s (2172kB/s), 527KiB/s-535KiB/s (540kB/s-548kB/s), io=19.6MiB (20.5MB), run=9429-9461msec WRITE: bw=2208KiB/s (2261kB/s), 548KiB/s-557KiB/s (561kB/s-571kB/s), io=20.4MiB (21.4MB), run=9429-9461msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=8962: Fri Mar 22 07:40:50 2024 read: IOPS=4, BW=163MiB/s (171MB/s)(40.0MiB/245msec) slat (nsec): min=8321.4k, max=8321.4k, avg=8321380.00, stdev= 0.00 clat (nsec): min=236205k, max=236205k, avg=236204907.00, stdev= 0.00 lat (nsec): min=244550k, max=244550k, avg=244549738.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.28%, ctx=3, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=40.0MiB (41.9MB), run=245-245msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout