== sanity test 398c: run fio to test AIO ================= 04:29:34 (1752481774) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.164876 s, 254 MB/s osc.lustre-OST0000-osc-ffff94f4c9c13000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=194448: Mon Jul 14 04:29:47 2025 write: IOPS=304, BW=1218KiB/s (1247kB/s)(10.0MiB/8410msec); 0 zone resets slat (usec): min=23, max=1335, avg=103.98, stdev=81.17 clat (msec): min=5, max=100, avg=52.36, stdev=11.23 lat (msec): min=5, max=100, avg=52.47, stdev=11.23 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 73], | 99.00th=[ 86], 99.50th=[ 96], 99.90th=[ 100], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 904, max= 1536, per=25.06%, avg=1214.38, stdev=189.49, samples=16 iops : min= 226, max= 384, avg=303.50, stdev=47.40, samples=16 lat (msec) : 10=0.16%, 20=0.04%, 50=49.41%, 100=50.31%, 250=0.08% cpu : usr=0.40%, sys=3.08%, ctx=1276, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=194449: Mon Jul 14 04:29:47 2025 write: IOPS=303, BW=1214KiB/s (1243kB/s)(10.0MiB/8434msec); 0 zone resets slat (usec): min=22, max=29100, avg=116.83, stdev=580.66 clat (msec): min=4, max=100, avg=52.20, stdev=11.28 lat (msec): min=4, max=100, avg=52.32, stdev=11.26 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 57], 80.00th=[ 62], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 90], 99.50th=[ 97], 99.90th=[ 99], 99.95th=[ 101], | 99.99th=[ 102] bw ( KiB/s): min= 928, max= 1536, per=24.97%, avg=1210.25, stdev=186.28, samples=16 iops : min= 232, max= 384, avg=302.50, stdev=46.63, samples=16 lat (msec) : 10=0.04%, 20=0.12%, 50=50.00%, 100=49.80%, 250=0.04% cpu : usr=0.56%, sys=3.27%, ctx=1299, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=194450: Mon Jul 14 04:29:47 2025 write: IOPS=302, BW=1212KiB/s (1241kB/s)(10.0MiB/8451msec); 0 zone resets slat (usec): min=24, max=2477, avg=102.93, stdev=96.62 clat (usec): min=10424, max=99290, avg=52270.91, stdev=11196.52 lat (usec): min=10591, max=99477, avg=52375.12, stdev=11199.31 clat percentiles (usec): | 1.00th=[32900], 5.00th=[38011], 10.00th=[40109], 20.00th=[43254], | 30.00th=[45876], 40.00th=[47973], 50.00th=[50070], 60.00th=[52691], | 70.00th=[56361], 80.00th=[60556], 90.00th=[66847], 95.00th=[73925], | 99.00th=[83362], 99.50th=[96994], 99.90th=[98042], 99.95th=[98042], | 99.99th=[99091] bw ( KiB/s): min= 894, max= 1536, per=24.89%, avg=1205.94, stdev=185.11, samples=16 iops : min= 223, max= 384, avg=301.44, stdev=46.35, samples=16 lat (msec) : 20=0.04%, 50=49.77%, 100=50.20% cpu : usr=0.49%, sys=2.85%, ctx=1199, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=194451: Mon Jul 14 04:29:47 2025 write: IOPS=302, BW=1212KiB/s (1241kB/s)(10.0MiB/8451msec); 0 zone resets slat (usec): min=22, max=1408, avg=101.19, stdev=87.99 clat (msec): min=29, max=100, avg=52.29, stdev=11.13 lat (msec): min=29, max=101, avg=52.39, stdev=11.14 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 73], | 99.00th=[ 94], 99.50th=[ 96], 99.90th=[ 99], 99.95th=[ 100], | 99.99th=[ 102] bw ( KiB/s): min= 910, max= 1536, per=24.94%, avg=1208.44, stdev=182.44, samples=16 iops : min= 227, max= 384, avg=302.06, stdev=45.68, samples=16 lat (msec) : 50=50.00%, 100=49.96%, 250=0.04% cpu : usr=0.51%, sys=3.07%, ctx=1237, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4847KiB/s (4963kB/s), 1212KiB/s-1218KiB/s (1241kB/s-1247kB/s), io=40.0MiB (41.9MB), run=8410-8451msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=194462: Mon Jul 14 04:30:00 2025 read: IOPS=134, BW=537KiB/s (550kB/s)(5048KiB/9393msec) slat (usec): min=1456, max=12364, avg=3887.77, stdev=1409.80 clat (msec): min=5, max=123, avg=54.91, stdev=11.88 lat (msec): min=10, max=126, avg=58.80, stdev=12.22 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 70], 95.00th=[ 74], | 99.00th=[ 89], 99.50th=[ 103], 99.90th=[ 115], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 408, max= 632, per=25.09%, avg=535.89, stdev=67.03, samples=18 iops : min= 102, max= 158, avg=133.83, stdev=16.71, samples=18 write: IOPS=138, BW=553KiB/s (566kB/s)(5192KiB/9393msec); 0 zone resets slat (usec): min=26, max=1248, avg=103.74, stdev=90.80 clat (msec): min=28, max=122, avg=57.61, stdev=11.57 lat (msec): min=28, max=122, avg=57.71, stdev=11.57 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 63], 80.00th=[ 66], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 94], 99.50th=[ 106], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 440, max= 680, per=24.66%, avg=548.39, stdev=74.88, samples=18 iops : min= 110, max= 170, avg=136.94, stdev=18.71, samples=18 lat (msec) : 10=0.04%, 20=0.08%, 50=30.59%, 100=68.71%, 250=0.59% cpu : usr=0.59%, sys=3.74%, ctx=2421, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194463: Mon Jul 14 04:30:00 2025 read: IOPS=132, BW=530KiB/s (543kB/s)(4976KiB/9381msec) slat (usec): min=1405, max=9800, avg=3814.76, stdev=1340.83 clat (msec): min=14, max=116, avg=54.76, stdev=11.67 lat (msec): min=17, max=121, avg=58.58, stdev=12.04 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 56], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 69], 95.00th=[ 75], | 99.00th=[ 92], 99.50th=[ 100], 99.90th=[ 114], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 376, max= 640, per=24.63%, avg=526.06, stdev=86.80, samples=18 iops : min= 94, max= 160, avg=131.33, stdev=21.75, samples=18 write: IOPS=140, BW=561KiB/s (575kB/s)(5264KiB/9381msec); 0 zone resets slat (usec): min=25, max=1205, avg=106.64, stdev=98.17 clat (msec): min=30, max=113, avg=57.77, stdev=11.31 lat (msec): min=30, max=113, avg=57.88, stdev=11.31 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 49], | 30.00th=[ 52], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 92], 99.50th=[ 99], 99.90th=[ 110], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 472, max= 672, per=25.18%, avg=559.94, stdev=57.12, samples=18 iops : min= 118, max= 168, avg=139.83, stdev=14.21, samples=18 lat (msec) : 20=0.04%, 50=31.21%, 100=68.36%, 250=0.39% cpu : usr=0.47%, sys=3.90%, ctx=2339, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194464: Mon Jul 14 04:30:00 2025 read: IOPS=133, BW=535KiB/s (547kB/s)(4984KiB/9324msec) slat (usec): min=40, max=13148, avg=3859.39, stdev=1430.82 clat (msec): min=9, max=119, avg=55.12, stdev=12.05 lat (msec): min=11, max=130, avg=58.98, stdev=12.44 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 70], 95.00th=[ 77], | 99.00th=[ 89], 99.50th=[ 99], 99.90th=[ 115], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 432, max= 656, per=25.10%, avg=536.06, stdev=57.76, samples=18 iops : min= 108, max= 164, avg=133.83, stdev=14.48, samples=18 write: IOPS=140, BW=564KiB/s (577kB/s)(5256KiB/9324msec); 0 zone resets slat (usec): min=27, max=2114, avg=115.16, stdev=140.87 clat (msec): min=15, max=114, avg=57.31, stdev=11.21 lat (msec): min=16, max=114, avg=57.42, stdev=11.22 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 63], 80.00th=[ 66], 90.00th=[ 71], 95.00th=[ 79], | 99.00th=[ 93], 99.50th=[ 97], 99.90th=[ 106], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 360, max= 694, per=25.21%, avg=560.61, stdev=81.49, samples=18 iops : min= 90, max= 173, avg=140.00, stdev=20.32, samples=18 lat (msec) : 10=0.04%, 20=0.35%, 50=30.70%, 100=68.52%, 250=0.39% cpu : usr=0.44%, sys=3.95%, ctx=2398, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194465: Mon Jul 14 04:30:00 2025 read: IOPS=134, BW=539KiB/s (552kB/s)(5060KiB/9393msec) slat (usec): min=1451, max=11360, avg=3829.57, stdev=1394.38 clat (msec): min=30, max=110, avg=55.03, stdev=11.55 lat (msec): min=32, max=116, avg=58.86, stdev=11.91 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 70], 95.00th=[ 77], | 99.00th=[ 91], 99.50th=[ 105], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 427, max= 616, per=25.12%, avg=536.56, stdev=59.40, samples=18 iops : min= 106, max= 154, avg=133.94, stdev=14.91, samples=18 write: IOPS=137, BW=551KiB/s (565kB/s)(5180KiB/9393msec); 0 zone resets slat (usec): min=27, max=2680, avg=111.03, stdev=126.58 clat (msec): min=6, max=117, avg=57.72, stdev=11.45 lat (msec): min=7, max=117, avg=57.83, stdev=11.46 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 59], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 95], 99.50th=[ 104], 99.90th=[ 112], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 392, max= 640, per=24.60%, avg=547.11, stdev=73.21, samples=18 iops : min= 98, max= 160, avg=136.61, stdev=18.30, samples=18 lat (msec) : 10=0.04%, 20=0.04%, 50=31.41%, 100=67.89%, 250=0.62% cpu : usr=0.43%, sys=4.01%, ctx=2435, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2136KiB/s (2188kB/s), 530KiB/s-539KiB/s (543kB/s-552kB/s), io=19.6MiB (20.5MB), run=9324-9393msec WRITE: bw=2224KiB/s (2278kB/s), 551KiB/s-564KiB/s (565kB/s-577kB/s), io=20.4MiB (21.4MB), run=9324-9393msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=194470: Mon Jul 14 04:30:04 2025 read: IOPS=5, BW=211MiB/s (221MB/s)(40.0MiB/190msec) slat (nsec): min=16229k, max=16229k, avg=16228899.00, stdev= 0.00 clat (nsec): min=157968k, max=157968k, avg=157968302.00, stdev= 0.00 lat (nsec): min=174203k, max=174203k, avg=174202648.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.47%, ctx=2, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=40.0MiB (41.9MB), run=190-190msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout