== sanity test 398c: run fio to test AIO ================= 03:39:36 (1753429176) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.137526 s, 305 MB/s osc.lustre-OST0000-osc-ffff97fa17e1e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=194052: Fri Jul 25 03:39:48 2025 write: IOPS=310, BW=1241KiB/s (1270kB/s)(10.0MiB/8254msec); 0 zone resets slat (usec): min=23, max=1619, avg=115.81, stdev=93.48 clat (msec): min=6, max=103, avg=51.40, stdev=12.32 lat (msec): min=6, max=103, avg=51.52, stdev=12.32 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 34], 10.00th=[ 37], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 54], | 70.00th=[ 58], 80.00th=[ 63], 90.00th=[ 68], 95.00th=[ 72], | 99.00th=[ 87], 99.50th=[ 91], 99.90th=[ 94], 99.95th=[ 96], | 99.99th=[ 104] bw ( KiB/s): min= 894, max= 1704, per=24.86%, avg=1229.88, stdev=217.11, samples=16 iops : min= 223, max= 426, avg=307.38, stdev=54.28, samples=16 lat (msec) : 10=0.04%, 20=0.16%, 50=51.21%, 100=48.55%, 250=0.04% cpu : usr=0.53%, sys=3.22%, ctx=1288, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=194053: Fri Jul 25 03:39:48 2025 write: IOPS=309, BW=1237KiB/s (1267kB/s)(10.0MiB/8279msec); 0 zone resets slat (usec): min=29, max=1203, avg=117.03, stdev=89.28 clat (usec): min=4312, max=99939, avg=51232.08, stdev=12558.21 lat (usec): min=4426, max=99981, avg=51350.16, stdev=12559.03 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 34], 10.00th=[ 37], 20.00th=[ 41], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 54], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 74], | 99.00th=[ 89], 99.50th=[ 91], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 894, max= 1720, per=24.71%, avg=1222.44, stdev=211.42, samples=16 iops : min= 223, max= 430, avg=305.56, stdev=52.92, samples=16 lat (msec) : 10=0.16%, 50=51.72%, 100=48.12% cpu : usr=0.54%, sys=3.19%, ctx=1354, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=194054: Fri Jul 25 03:39:48 2025 write: IOPS=309, BW=1237KiB/s (1267kB/s)(10.0MiB/8279msec); 0 zone resets slat (usec): min=23, max=28846, avg=119.80, stdev=573.26 clat (msec): min=12, max=102, avg=51.30, stdev=12.50 lat (msec): min=24, max=103, avg=51.42, stdev=12.48 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 34], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 89], 99.50th=[ 92], 99.90th=[ 101], 99.95th=[ 103], | 99.99th=[ 104] bw ( KiB/s): min= 862, max= 1760, per=24.73%, avg=1223.50, stdev=221.02, samples=16 iops : min= 215, max= 440, avg=305.81, stdev=55.31, samples=16 lat (msec) : 20=0.04%, 50=51.95%, 100=47.85%, 250=0.16% cpu : usr=0.42%, sys=3.54%, ctx=1220, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=194055: Fri Jul 25 03:39:48 2025 write: IOPS=309, BW=1239KiB/s (1269kB/s)(10.0MiB/8266msec); 0 zone resets slat (usec): min=27, max=1139, avg=122.35, stdev=93.60 clat (msec): min=5, max=102, avg=51.17, stdev=12.50 lat (msec): min=5, max=102, avg=51.30, stdev=12.50 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 34], 10.00th=[ 37], 20.00th=[ 41], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 89], 99.50th=[ 92], 99.90th=[ 100], 99.95th=[ 100], | 99.99th=[ 104] bw ( KiB/s): min= 886, max= 1728, per=24.78%, avg=1225.87, stdev=208.02, samples=16 iops : min= 221, max= 432, avg=306.38, stdev=52.04, samples=16 lat (msec) : 10=0.08%, 20=0.12%, 50=52.89%, 100=46.88%, 250=0.04% cpu : usr=0.54%, sys=3.73%, ctx=1431, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4947KiB/s (5066kB/s), 1237KiB/s-1241KiB/s (1267kB/s-1270kB/s), io=40.0MiB (41.9MB), run=8254-8279msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=194066: Fri Jul 25 03:40:01 2025 read: IOPS=138, BW=554KiB/s (568kB/s)(5048KiB/9106msec) slat (usec): min=1262, max=12678, avg=3931.72, stdev=1317.42 clat (usec): min=29550, max=92284, avg=54359.53, stdev=9934.63 lat (usec): min=33074, max=98322, avg=58294.82, stdev=10221.01 clat percentiles (usec): | 1.00th=[34341], 5.00th=[39584], 10.00th=[42206], 20.00th=[46400], | 30.00th=[48497], 40.00th=[51119], 50.00th=[53216], 60.00th=[55837], | 70.00th=[58983], 80.00th=[62653], 90.00th=[67634], 95.00th=[72877], | 99.00th=[80217], 99.50th=[86508], 99.90th=[91751], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 424, max= 696, per=25.14%, avg=553.65, stdev=68.37, samples=17 iops : min= 106, max= 174, avg=138.29, stdev=17.05, samples=17 write: IOPS=142, BW=570KiB/s (584kB/s)(5192KiB/9106msec); 0 zone resets slat (usec): min=26, max=3259, avg=107.27, stdev=117.50 clat (usec): min=15063, max=95188, avg=54766.86, stdev=9281.97 lat (usec): min=15144, max=95227, avg=54875.23, stdev=9285.54 clat percentiles (usec): | 1.00th=[36963], 5.00th=[41157], 10.00th=[43254], 20.00th=[46400], | 30.00th=[49546], 40.00th=[51643], 50.00th=[54264], 60.00th=[56361], | 70.00th=[58983], 80.00th=[62129], 90.00th=[66847], 95.00th=[70779], | 99.00th=[81265], 99.50th=[87557], 99.90th=[89654], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 415, max= 768, per=24.80%, avg=568.71, stdev=86.42, samples=17 iops : min= 103, max= 192, avg=142.06, stdev=21.66, samples=17 lat (msec) : 20=0.04%, 50=33.59%, 100=66.37% cpu : usr=0.49%, sys=4.27%, ctx=2364, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194067: Fri Jul 25 03:40:01 2025 read: IOPS=137, BW=550KiB/s (564kB/s)(4976KiB/9042msec) slat (usec): min=40, max=12653, avg=3945.03, stdev=1300.88 clat (msec): min=6, max=109, avg=54.02, stdev=10.24 lat (msec): min=6, max=113, avg=57.97, stdev=10.51 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 82], 99.50th=[ 84], 99.90th=[ 95], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 400, max= 656, per=24.94%, avg=549.12, stdev=66.22, samples=17 iops : min= 100, max= 164, avg=137.18, stdev=16.59, samples=17 write: IOPS=145, BW=582KiB/s (596kB/s)(5264KiB/9042msec); 0 zone resets slat (usec): min=29, max=2666, avg=115.19, stdev=136.46 clat (msec): min=12, max=117, avg=54.83, stdev= 9.98 lat (msec): min=12, max=117, avg=54.94, stdev= 9.98 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 42], 10.00th=[ 44], 20.00th=[ 47], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 59], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 72], | 99.00th=[ 85], 99.50th=[ 91], 99.90th=[ 113], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 424, max= 736, per=25.44%, avg=583.41, stdev=86.77, samples=17 iops : min= 106, max= 184, avg=145.71, stdev=21.66, samples=17 lat (msec) : 10=0.04%, 20=0.04%, 50=33.95%, 100=65.86%, 250=0.12% cpu : usr=0.69%, sys=4.09%, ctx=2418, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194068: Fri Jul 25 03:40:01 2025 read: IOPS=136, BW=548KiB/s (561kB/s)(4984KiB/9100msec) slat (usec): min=1331, max=16002, avg=3874.05, stdev=1288.60 clat (usec): min=3820, max=93096, avg=54084.38, stdev=10009.35 lat (usec): min=6832, max=98583, avg=57961.40, stdev=10236.23 clat percentiles (usec): | 1.00th=[34341], 5.00th=[39060], 10.00th=[41681], 20.00th=[45351], | 30.00th=[48497], 40.00th=[50594], 50.00th=[53216], 60.00th=[56361], | 70.00th=[58983], 80.00th=[62129], 90.00th=[67634], 95.00th=[71828], | 99.00th=[80217], 99.50th=[82314], 99.90th=[85459], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 432, max= 664, per=24.95%, avg=549.29, stdev=73.97, samples=17 iops : min= 108, max= 166, avg=137.18, stdev=18.48, samples=17 write: IOPS=144, BW=578KiB/s (591kB/s)(5256KiB/9100msec); 0 zone resets slat (usec): min=28, max=2079, avg=113.58, stdev=113.62 clat (usec): min=34130, max=95907, avg=55201.05, stdev=9500.95 lat (usec): min=34219, max=95983, avg=55315.65, stdev=9497.56 clat percentiles (usec): | 1.00th=[38011], 5.00th=[41681], 10.00th=[43779], 20.00th=[46924], | 30.00th=[49546], 40.00th=[52167], 50.00th=[54264], 60.00th=[56886], | 70.00th=[59507], 80.00th=[63177], 90.00th=[67634], 95.00th=[71828], | 99.00th=[82314], 99.50th=[85459], 99.90th=[92799], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 440, max= 688, per=24.99%, avg=573.12, stdev=83.84, samples=17 iops : min= 110, max= 172, avg=143.18, stdev=21.00, samples=17 lat (msec) : 4=0.04%, 50=34.10%, 100=65.86% cpu : usr=0.47%, sys=4.28%, ctx=2415, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194069: Fri Jul 25 03:40:01 2025 read: IOPS=138, BW=555KiB/s (569kB/s)(5060KiB/9110msec) slat (usec): min=1379, max=12529, avg=3854.99, stdev=1249.32 clat (msec): min=6, max=102, avg=53.58, stdev= 9.89 lat (msec): min=9, max=104, avg=57.44, stdev=10.13 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 67], 95.00th=[ 71], | 99.00th=[ 80], 99.50th=[ 83], 99.90th=[ 95], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 392, max= 704, per=25.10%, avg=552.76, stdev=82.49, samples=17 iops : min= 98, max= 176, avg=138.06, stdev=20.64, samples=17 write: IOPS=142, BW=569KiB/s (582kB/s)(5180KiB/9110msec); 0 zone resets slat (usec): min=32, max=3351, avg=108.13, stdev=117.13 clat (usec): min=16821, max=92807, avg=55691.32, stdev=10033.09 lat (usec): min=17073, max=92861, avg=55800.76, stdev=10036.58 clat percentiles (usec): | 1.00th=[36439], 5.00th=[41681], 10.00th=[44303], 20.00th=[47449], | 30.00th=[50070], 40.00th=[52167], 50.00th=[54789], 60.00th=[57410], | 70.00th=[60031], 80.00th=[63177], 90.00th=[68682], 95.00th=[72877], | 99.00th=[87557], 99.50th=[88605], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 440, max= 648, per=24.78%, avg=568.24, stdev=56.38, samples=17 iops : min= 110, max= 162, avg=141.94, stdev=14.10, samples=17 lat (msec) : 10=0.04%, 20=0.08%, 50=33.75%, 100=66.09%, 250=0.04% cpu : usr=0.53%, sys=4.24%, ctx=2422, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2203KiB/s (2256kB/s), 548KiB/s-555KiB/s (561kB/s-569kB/s), io=19.6MiB (20.5MB), run=9042-9110msec WRITE: bw=2293KiB/s (2348kB/s), 569KiB/s-582KiB/s (582kB/s-596kB/s), io=20.4MiB (21.4MB), run=9042-9110msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=194073: Fri Jul 25 03:40:04 2025 read: IOPS=6, BW=267MiB/s (280MB/s)(40.0MiB/150msec) slat (nsec): min=13689k, max=13689k, avg=13688506.00, stdev= 0.00 clat (nsec): min=128600k, max=128600k, avg=128600094.00, stdev= 0.00 lat (nsec): min=142293k, max=142293k, avg=142292873.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 129], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.72%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=40.0MiB (41.9MB), run=150-150msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout