== sanity test 398c: run fio to test AIO ================= 23:38:33 (1746157113) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.15842 s, 265 MB/s osc.lustre-OST0000-osc-ffff8800a83bf000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=23122: Thu May 1 23:38:46 2025 write: IOPS=226, BW=906KiB/s (927kB/s)(10.0MiB/11308msec) slat (usec): min=11, max=3358, avg=98.24, stdev=213.16 clat (msec): min=18, max=121, avg=70.33, stdev=13.41 lat (msec): min=18, max=122, avg=70.43, stdev=13.42 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 60], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 69], 60.00th=[ 72], | 70.00th=[ 77], 80.00th=[ 82], 90.00th=[ 88], 95.00th=[ 93], | 99.00th=[ 111], 99.50th=[ 120], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 768, max= 1064, per=25.00%, avg=905.18, stdev=89.40, samples=22 iops : min= 192, max= 266, avg=226.23, stdev=22.35, samples=22 lat (msec) : 20=0.04%, 50=4.30%, 100=92.73%, 250=2.93% cpu : usr=0.44%, sys=2.94%, ctx=1162, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23123: Thu May 1 23:38:46 2025 write: IOPS=226, BW=906KiB/s (928kB/s)(10.0MiB/11301msec) slat (usec): min=10, max=4087, avg=80.52, stdev=187.61 clat (msec): min=7, max=127, avg=70.25, stdev=13.54 lat (msec): min=7, max=127, avg=70.33, stdev=13.54 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 69], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 87], 95.00th=[ 94], | 99.00th=[ 116], 99.50th=[ 121], 99.90th=[ 127], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 720, max= 1048, per=24.96%, avg=903.68, stdev=83.81, samples=22 iops : min= 180, max= 262, avg=225.77, stdev=20.97, samples=22 lat (msec) : 10=0.04%, 50=3.98%, 100=92.58%, 250=3.40% cpu : usr=0.86%, sys=2.04%, ctx=863, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23124: Thu May 1 23:38:46 2025 write: IOPS=227, BW=909KiB/s (930kB/s)(10.0MiB/11269msec) slat (usec): min=10, max=3152, avg=97.34, stdev=204.15 clat (msec): min=11, max=125, avg=70.10, stdev=14.21 lat (msec): min=11, max=125, avg=70.20, stdev=14.22 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 95], | 99.00th=[ 116], 99.50th=[ 121], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 768, max= 1064, per=25.11%, avg=909.18, stdev=89.50, samples=22 iops : min= 192, max= 266, avg=227.23, stdev=22.38, samples=22 lat (msec) : 20=0.31%, 50=3.52%, 100=91.95%, 250=4.22% cpu : usr=0.00%, sys=3.43%, ctx=1187, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23125: Thu May 1 23:38:46 2025 write: IOPS=226, BW=905KiB/s (927kB/s)(10.0MiB/11311msec) slat (usec): min=11, max=6386, avg=108.95, stdev=266.71 clat (msec): min=15, max=129, avg=70.23, stdev=13.29 lat (msec): min=15, max=129, avg=70.34, stdev=13.30 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 60], | 30.00th=[ 64], 40.00th=[ 66], 50.00th=[ 69], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 87], 95.00th=[ 94], | 99.00th=[ 111], 99.50th=[ 121], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 760, max= 1053, per=24.94%, avg=902.95, stdev=87.50, samples=22 iops : min= 190, max= 263, avg=225.64, stdev=21.86, samples=22 lat (msec) : 20=0.08%, 50=3.12%, 100=93.40%, 250=3.40% cpu : usr=0.82%, sys=2.91%, ctx=1334, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3621KiB/s (3708kB/s), 905KiB/s-909KiB/s (927kB/s-930kB/s), io=40.0MiB (41.9MB), run=11269-11311msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23144: Thu May 1 23:38:58 2025 read: IOPS=121, BW=484KiB/s (496kB/s)(5048KiB/10423msec) slat (usec): min=1243, max=19470, avg=4317.32, stdev=1871.33 clat (msec): min=20, max=114, avg=61.50, stdev=14.18 lat (msec): min=27, max=120, avg=65.82, stdev=14.71 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 40], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 81], 95.00th=[ 86], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 360, max= 624, per=24.97%, avg=480.45, stdev=71.13, samples=20 iops : min= 90, max= 156, avg=119.90, stdev=17.83, samples=20 write: IOPS=124, BW=498KiB/s (510kB/s)(5192KiB/10423msec) slat (usec): min=13, max=7442, avg=106.38, stdev=288.72 clat (msec): min=20, max=111, avg=63.39, stdev=13.53 lat (msec): min=20, max=111, avg=63.50, stdev=13.54 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 52], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 81], 95.00th=[ 88], | 99.00th=[ 101], 99.50th=[ 104], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 391, max= 664, per=24.73%, avg=495.30, stdev=89.48, samples=20 iops : min= 97, max= 166, avg=123.60, stdev=22.49, samples=20 lat (msec) : 50=18.75%, 100=80.39%, 250=0.86% cpu : usr=0.00%, sys=6.88%, ctx=2283, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23145: Thu May 1 23:38:58 2025 read: IOPS=119, BW=479KiB/s (491kB/s)(4976KiB/10378msec) slat (usec): min=1190, max=15536, avg=4307.04, stdev=1845.69 clat (msec): min=14, max=113, avg=61.33, stdev=14.13 lat (msec): min=18, max=115, avg=65.64, stdev=14.68 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 81], 95.00th=[ 87], | 99.00th=[ 101], 99.50th=[ 105], 99.90th=[ 110], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 383, max= 630, per=24.82%, avg=477.45, stdev=71.95, samples=20 iops : min= 95, max= 157, avg=119.05, stdev=17.98, samples=20 write: IOPS=126, BW=507KiB/s (519kB/s)(5264KiB/10378msec) slat (usec): min=13, max=2209, avg=96.59, stdev=191.96 clat (msec): min=20, max=128, avg=63.45, stdev=14.29 lat (msec): min=21, max=128, avg=63.55, stdev=14.30 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 89], | 99.00th=[ 102], 99.50th=[ 106], 99.90th=[ 116], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 392, max= 680, per=25.37%, avg=508.25, stdev=81.75, samples=20 iops : min= 98, max= 170, avg=126.75, stdev=20.54, samples=20 lat (msec) : 20=0.04%, 50=20.39%, 100=78.48%, 250=1.09% cpu : usr=0.00%, sys=6.89%, ctx=2272, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23146: Thu May 1 23:38:58 2025 read: IOPS=119, BW=478KiB/s (489kB/s)(4984KiB/10427msec) slat (usec): min=1304, max=14914, avg=4323.34, stdev=1815.26 clat (msec): min=24, max=104, avg=61.63, stdev=14.10 lat (msec): min=28, max=111, avg=65.96, stdev=14.64 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 81], 95.00th=[ 88], | 99.00th=[ 97], 99.50th=[ 101], 99.90th=[ 104], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 344, max= 616, per=24.82%, avg=477.50, stdev=75.42, samples=20 iops : min= 86, max= 154, avg=119.20, stdev=18.93, samples=20 write: IOPS=126, BW=504KiB/s (516kB/s)(5256KiB/10427msec) slat (usec): min=12, max=6018, avg=99.68, stdev=267.14 clat (msec): min=18, max=114, avg=63.30, stdev=14.06 lat (msec): min=18, max=114, avg=63.40, stdev=14.05 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 88], | 99.00th=[ 101], 99.50th=[ 106], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 368, max= 712, per=24.86%, avg=498.00, stdev=91.10, samples=20 iops : min= 92, max= 178, avg=124.30, stdev=22.93, samples=20 lat (msec) : 20=0.04%, 50=18.87%, 100=80.27%, 250=0.82% cpu : usr=0.74%, sys=5.99%, ctx=2252, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23147: Thu May 1 23:38:58 2025 read: IOPS=121, BW=487KiB/s (499kB/s)(5060KiB/10390msec) slat (usec): min=16, max=16342, avg=4290.60, stdev=1864.54 clat (msec): min=7, max=110, avg=61.75, stdev=14.10 lat (msec): min=9, max=113, avg=66.04, stdev=14.59 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 80], 95.00th=[ 88], | 99.00th=[ 100], 99.50th=[ 102], 99.90th=[ 110], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 352, max= 662, per=25.23%, avg=485.40, stdev=75.37, samples=20 iops : min= 88, max= 165, avg=121.15, stdev=18.80, samples=20 write: IOPS=124, BW=499KiB/s (511kB/s)(5180KiB/10390msec) slat (usec): min=15, max=4270, avg=101.40, stdev=227.12 clat (msec): min=14, max=112, avg=63.52, stdev=13.65 lat (msec): min=17, max=112, avg=63.62, stdev=13.64 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 81], 95.00th=[ 86], | 99.00th=[ 99], 99.50th=[ 101], 99.90th=[ 108], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 392, max= 624, per=24.81%, avg=497.00, stdev=70.32, samples=20 iops : min= 98, max= 156, avg=124.05, stdev=17.61, samples=20 lat (msec) : 10=0.12%, 20=0.12%, 50=18.87%, 100=80.27%, 250=0.62% cpu : usr=0.63%, sys=6.14%, ctx=2272, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1925KiB/s (1971kB/s), 478KiB/s-487KiB/s (489kB/s-499kB/s), io=19.6MiB (20.5MB), run=10378-10427msec WRITE: bw=2004KiB/s (2052kB/s), 498KiB/s-507KiB/s (510kB/s-519kB/s), io=20.4MiB (21.4MB), run=10378-10427msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=23159: Thu May 1 23:39:00 2025 read: IOPS=4, BW=178MiB/s (186MB/s)(40.0MiB/225msec) slat (nsec): min=2585.3k, max=2585.3k, avg=2585265.00, stdev= 0.00 clat (nsec): min=208650k, max=208650k, avg=208650465.00, stdev= 0.00 lat (nsec): min=211925k, max=211925k, avg=211924821.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] lat (msec) : 250=100.00% cpu : usr=0.45%, sys=0.89%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=178MiB/s (186MB/s), 178MiB/s-178MiB/s (186MB/s-186MB/s), io=40.0MiB (41.9MB), run=225-225msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout