== sanity test 398c: run fio to test AIO ================= 23:55:44 (1764737744) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.178025 s, 236 MB/s osc.lustre-OST0000-osc-ffff99c2c7a9a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182471: Tue Dec 2 23:55:58 2025 write: IOPS=265, BW=1061KiB/s (1087kB/s)(10.0MiB/9649msec); 0 zone resets slat (usec): min=33, max=844, avg=141.25, stdev=97.03 clat (msec): min=13, max=105, avg=59.75, stdev=11.46 lat (msec): min=13, max=105, avg=59.90, stdev=11.46 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 74], 95.00th=[ 79], | 99.00th=[ 92], 99.50th=[ 96], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 856, max= 1408, per=24.91%, avg=1057.21, stdev=131.01, samples=19 iops : min= 214, max= 352, avg=264.26, stdev=32.75, samples=19 lat (msec) : 20=0.04%, 50=20.00%, 100=79.53%, 250=0.43% cpu : usr=0.51%, sys=3.44%, ctx=2634, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182472: Tue Dec 2 23:55:58 2025 write: IOPS=267, BW=1069KiB/s (1094kB/s)(10.0MiB/9583msec); 0 zone resets slat (usec): min=24, max=1320, avg=137.37, stdev=94.31 clat (msec): min=12, max=129, avg=59.68, stdev=11.90 lat (msec): min=12, max=129, avg=59.82, stdev=11.90 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 95], 99.50th=[ 103], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 958, max= 1328, per=25.46%, avg=1080.39, stdev=109.44, samples=18 iops : min= 239, max= 332, avg=270.06, stdev=27.39, samples=18 lat (msec) : 20=0.04%, 50=22.70%, 100=76.72%, 250=0.55% cpu : usr=0.43%, sys=3.16%, ctx=2502, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182473: Tue Dec 2 23:55:58 2025 write: IOPS=265, BW=1063KiB/s (1088kB/s)(10.0MiB/9635msec); 0 zone resets slat (usec): min=30, max=26885, avg=165.22, stdev=538.11 clat (msec): min=11, max=123, avg=59.68, stdev=11.84 lat (msec): min=11, max=123, avg=59.85, stdev=11.81 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 95], 99.50th=[ 103], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 124] bw ( KiB/s): min= 744, max= 1344, per=24.91%, avg=1057.21, stdev=127.02, samples=19 iops : min= 186, max= 336, avg=264.26, stdev=31.75, samples=19 lat (msec) : 20=0.12%, 50=21.05%, 100=78.01%, 250=0.82% cpu : usr=0.59%, sys=3.77%, ctx=2542, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182474: Tue Dec 2 23:55:58 2025 write: IOPS=266, BW=1065KiB/s (1091kB/s)(10.0MiB/9614msec); 0 zone resets slat (usec): min=31, max=31134, avg=171.33, stdev=622.03 clat (msec): min=7, max=114, avg=59.60, stdev=11.94 lat (msec): min=7, max=114, avg=59.77, stdev=11.91 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 94], 99.50th=[ 102], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 114] bw ( KiB/s): min= 936, max= 1384, per=25.34%, avg=1075.33, stdev=116.71, samples=18 iops : min= 234, max= 346, avg=268.78, stdev=29.16, samples=18 lat (msec) : 10=0.16%, 20=0.16%, 50=19.77%, 100=79.34%, 250=0.59% cpu : usr=0.56%, sys=4.00%, ctx=2614, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4245KiB/s (4347kB/s), 1061KiB/s-1069KiB/s (1087kB/s-1094kB/s), io=40.0MiB (41.9MB), run=9583-9649msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182485: Tue Dec 2 23:56:12 2025 read: IOPS=130, BW=522KiB/s (535kB/s)(5048KiB/9666msec) slat (usec): min=43, max=12648, avg=4274.80, stdev=1492.30 clat (msec): min=11, max=101, avg=57.53, stdev=12.28 lat (msec): min=12, max=108, avg=61.82, stdev=12.74 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 60], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 89], 99.50th=[ 91], 99.90th=[ 97], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 381, max= 640, per=25.20%, avg=521.05, stdev=70.52, samples=19 iops : min= 95, max= 160, avg=130.11, stdev=17.66, samples=19 write: IOPS=134, BW=537KiB/s (550kB/s)(5192KiB/9666msec); 0 zone resets slat (usec): min=37, max=1449, avg=169.65, stdev=133.00 clat (msec): min=19, max=107, avg=58.64, stdev=11.11 lat (msec): min=19, max=108, avg=58.82, stdev=11.11 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 79], | 99.00th=[ 90], 99.50th=[ 93], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 421, max= 680, per=24.96%, avg=537.21, stdev=64.51, samples=19 iops : min= 105, max= 170, avg=134.16, stdev=16.17, samples=19 lat (msec) : 20=0.23%, 50=23.87%, 100=75.70%, 250=0.20% cpu : usr=0.69%, sys=5.61%, ctx=4486, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182486: Tue Dec 2 23:56:12 2025 read: IOPS=128, BW=513KiB/s (525kB/s)(4976KiB/9699msec) slat (usec): min=1687, max=12116, avg=4203.52, stdev=1385.56 clat (msec): min=14, max=106, avg=57.52, stdev=11.65 lat (msec): min=18, max=109, avg=61.72, stdev=12.05 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 80], | 99.00th=[ 89], 99.50th=[ 96], 99.90th=[ 104], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 373, max= 632, per=24.69%, avg=510.63, stdev=65.12, samples=19 iops : min= 93, max= 158, avg=127.47, stdev=16.32, samples=19 write: IOPS=135, BW=543KiB/s (556kB/s)(5264KiB/9699msec); 0 zone resets slat (usec): min=35, max=2190, avg=168.55, stdev=132.29 clat (msec): min=31, max=107, avg=58.66, stdev=10.67 lat (msec): min=32, max=107, avg=58.83, stdev=10.67 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 87], 99.50th=[ 92], 99.90th=[ 104], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 413, max= 680, per=25.24%, avg=543.11, stdev=67.73, samples=19 iops : min= 103, max= 170, avg=135.58, stdev=17.07, samples=19 lat (msec) : 20=0.04%, 50=25.04%, 100=74.69%, 250=0.23% cpu : usr=0.75%, sys=5.53%, ctx=4460, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182487: Tue Dec 2 23:56:12 2025 read: IOPS=128, BW=514KiB/s (526kB/s)(4984KiB/9704msec) slat (usec): min=1636, max=10808, avg=4226.06, stdev=1343.97 clat (usec): min=7499, max=99961, avg=57250.61, stdev=11458.94 lat (msec): min=11, max=102, avg=61.48, stdev=11.84 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 80], | 99.00th=[ 88], 99.50th=[ 91], 99.90th=[ 94], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 380, max= 608, per=24.86%, avg=514.11, stdev=71.84, samples=19 iops : min= 95, max= 152, avg=128.37, stdev=18.00, samples=19 write: IOPS=135, BW=542KiB/s (555kB/s)(5256KiB/9704msec); 0 zone resets slat (usec): min=30, max=1444, avg=178.45, stdev=148.05 clat (msec): min=9, max=104, avg=58.90, stdev=11.41 lat (msec): min=9, max=104, avg=59.08, stdev=11.42 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 95], 99.50th=[ 99], 99.90th=[ 103], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 415, max= 704, per=25.01%, avg=538.26, stdev=77.59, samples=19 iops : min= 103, max= 176, avg=134.37, stdev=19.45, samples=19 lat (msec) : 10=0.08%, 20=0.04%, 50=25.00%, 100=74.69%, 250=0.20% cpu : usr=0.91%, sys=5.48%, ctx=4502, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182488: Tue Dec 2 23:56:12 2025 read: IOPS=130, BW=521KiB/s (534kB/s)(5060KiB/9704msec) slat (usec): min=1511, max=12338, avg=4296.00, stdev=1485.03 clat (usec): min=9387, max=96717, avg=57368.55, stdev=11804.88 lat (msec): min=13, max=103, avg=61.67, stdev=12.26 clat percentiles (usec): | 1.00th=[34341], 5.00th=[40109], 10.00th=[42730], 20.00th=[47449], | 30.00th=[50594], 40.00th=[53216], 50.00th=[56361], 60.00th=[59507], | 70.00th=[62653], 80.00th=[66847], 90.00th=[72877], 95.00th=[79168], | 99.00th=[88605], 99.50th=[91751], 99.90th=[93848], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 365, max= 632, per=25.17%, avg=520.47, stdev=67.38, samples=19 iops : min= 91, max= 158, avg=129.95, stdev=16.89, samples=19 write: IOPS=133, BW=534KiB/s (547kB/s)(5180KiB/9704msec); 0 zone resets slat (usec): min=36, max=3111, avg=164.98, stdev=152.80 clat (msec): min=11, max=102, avg=58.75, stdev=10.84 lat (msec): min=11, max=102, avg=58.92, stdev=10.84 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 79], | 99.00th=[ 88], 99.50th=[ 92], 99.90th=[ 97], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 440, max= 648, per=24.69%, avg=531.26, stdev=58.56, samples=19 iops : min= 110, max= 162, avg=132.63, stdev=14.71, samples=19 lat (msec) : 10=0.04%, 20=0.08%, 50=24.41%, 100=75.43%, 250=0.04% cpu : usr=0.87%, sys=5.46%, ctx=4516, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2068KiB/s (2118kB/s), 513KiB/s-522KiB/s (525kB/s-535kB/s), io=19.6MiB (20.5MB), run=9666-9704msec WRITE: bw=2153KiB/s (2205kB/s), 534KiB/s-543KiB/s (547kB/s-556kB/s), io=20.4MiB (21.4MB), run=9666-9704msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182493: Tue Dec 2 23:56:16 2025 read: IOPS=5, BW=234MiB/s (245MB/s)(40.0MiB/171msec) slat (nsec): min=18456k, max=18456k, avg=18455864.00, stdev= 0.00 clat (nsec): min=143587k, max=143587k, avg=143587295.00, stdev= 0.00 lat (nsec): min=162052k, max=162052k, avg=162051510.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=9.41%, ctx=12, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=40.0MiB (41.9MB), run=171-171msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout