== sanity test 398c: run fio to test AIO ================= 10:12:02 (1766416322) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.172126 s, 244 MB/s osc.lustre-OST0000-osc-ffff9f9b5fe58800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147666: Mon Dec 22 10:12:53 2025 write: IOPS=54, BW=220KiB/s (225kB/s)(10.0MiB/46609msec); 0 zone resets slat (usec): min=27, max=3587, avg=139.12, stdev=126.18 clat (msec): min=60, max=631, avg=290.94, stdev=79.32 lat (msec): min=60, max=631, avg=291.08, stdev=79.33 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 199], 10.00th=[ 211], 20.00th=[ 232], | 30.00th=[ 249], 40.00th=[ 264], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 342], 90.00th=[ 397], 95.00th=[ 447], | 99.00th=[ 535], 99.50th=[ 584], 99.90th=[ 625], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 87, max= 414, per=24.98%, avg=218.82, stdev=55.97, samples=92 iops : min= 21, max= 103, avg=54.49, stdev=14.08, samples=92 lat (msec) : 100=0.66%, 250=31.41%, 500=65.55%, 750=2.38% cpu : usr=0.13%, sys=0.62%, ctx=2896, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147667: Mon Dec 22 10:12:53 2025 write: IOPS=54, BW=219KiB/s (224kB/s)(10.0MiB/46728msec); 0 zone resets slat (usec): min=22, max=1956, avg=136.01, stdev=103.63 clat (msec): min=61, max=592, avg=291.53, stdev=79.36 lat (msec): min=61, max=592, avg=291.66, stdev=79.37 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 194], 10.00th=[ 211], 20.00th=[ 232], | 30.00th=[ 249], 40.00th=[ 264], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 313], 80.00th=[ 342], 90.00th=[ 405], 95.00th=[ 447], | 99.00th=[ 550], 99.50th=[ 567], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 104, max= 328, per=24.87%, avg=217.84, stdev=51.69, samples=93 iops : min= 26, max= 82, avg=54.25, stdev=12.98, samples=93 lat (msec) : 100=0.66%, 250=30.82%, 500=65.90%, 750=2.62% cpu : usr=0.12%, sys=0.65%, ctx=2835, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147668: Mon Dec 22 10:12:53 2025 write: IOPS=54, BW=219KiB/s (225kB/s)(10.0MiB/46686msec); 0 zone resets slat (usec): min=30, max=1949, avg=135.17, stdev=99.46 clat (msec): min=70, max=592, avg=291.33, stdev=78.28 lat (msec): min=70, max=592, avg=291.47, stdev=78.28 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 194], 10.00th=[ 213], 20.00th=[ 232], | 30.00th=[ 249], 40.00th=[ 264], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 313], 80.00th=[ 347], 90.00th=[ 405], 95.00th=[ 451], | 99.00th=[ 518], 99.50th=[ 531], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 592] bw ( KiB/s): min= 104, max= 383, per=24.95%, avg=218.58, stdev=53.46, samples=92 iops : min= 26, max= 95, avg=54.42, stdev=13.43, samples=92 lat (msec) : 100=0.62%, 250=30.74%, 500=66.80%, 750=1.84% cpu : usr=0.12%, sys=0.65%, ctx=2733, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147669: Mon Dec 22 10:12:53 2025 write: IOPS=54, BW=219KiB/s (224kB/s)(10.0MiB/46727msec); 0 zone resets slat (usec): min=29, max=1568, avg=141.45, stdev=108.38 clat (msec): min=35, max=630, avg=291.50, stdev=78.76 lat (msec): min=35, max=631, avg=291.64, stdev=78.77 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 197], 10.00th=[ 211], 20.00th=[ 234], | 30.00th=[ 249], 40.00th=[ 264], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 401], 95.00th=[ 456], | 99.00th=[ 550], 99.50th=[ 575], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 634] bw ( KiB/s): min= 72, max= 336, per=24.86%, avg=217.81, stdev=51.22, samples=93 iops : min= 18, max= 84, avg=54.25, stdev=12.89, samples=93 lat (msec) : 50=0.08%, 100=0.12%, 250=31.05%, 500=66.21%, 750=2.54% cpu : usr=0.13%, sys=0.65%, ctx=2854, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=877KiB/s (898kB/s), 219KiB/s-220KiB/s (224kB/s-225kB/s), io=40.0MiB (41.9MB), run=46609-46728msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147685: Mon Dec 22 10:13:27 2025 read: IOPS=41, BW=167KiB/s (171kB/s)(5048KiB/30270msec) slat (usec): min=38, max=12523, avg=3524.14, stdev=1279.28 clat (msec): min=7, max=247, avg=118.23, stdev=29.66 lat (msec): min=11, max=251, avg=121.76, stdev=30.06 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 75], 10.00th=[ 84], 20.00th=[ 95], | 30.00th=[ 103], 40.00th=[ 110], 50.00th=[ 117], 60.00th=[ 124], | 70.00th=[ 131], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 169], | 99.00th=[ 199], 99.50th=[ 203], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 88, max= 272, per=25.31%, avg=166.57, stdev=40.44, samples=60 iops : min= 22, max= 68, avg=41.52, stdev=10.19, samples=60 write: IOPS=42, BW=172KiB/s (176kB/s)(5192KiB/30270msec); 0 zone resets slat (usec): min=36, max=2153, avg=133.31, stdev=100.06 clat (msec): min=37, max=463, avg=254.31, stdev=68.52 lat (msec): min=37, max=463, avg=254.44, stdev=68.53 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 144], 10.00th=[ 167], 20.00th=[ 194], | 30.00th=[ 222], 40.00th=[ 236], 50.00th=[ 249], 60.00th=[ 266], | 70.00th=[ 284], 80.00th=[ 313], 90.00th=[ 351], 95.00th=[ 376], | 99.00th=[ 414], 99.50th=[ 418], 99.90th=[ 460], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 88, max= 288, per=24.95%, avg=170.93, stdev=39.14, samples=60 iops : min= 22, max= 72, avg=42.60, stdev= 9.81, samples=60 lat (msec) : 10=0.08%, 20=0.08%, 50=0.31%, 100=12.77%, 250=61.76% lat (msec) : 500=25.00% cpu : usr=0.17%, sys=1.46%, ctx=5074, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147686: Mon Dec 22 10:13:27 2025 read: IOPS=40, BW=163KiB/s (167kB/s)(4976KiB/30481msec) slat (usec): min=1301, max=9198, avg=3512.02, stdev=1219.43 clat (msec): min=4, max=244, avg=117.55, stdev=29.51 lat (msec): min=9, max=248, avg=121.06, stdev=29.86 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 77], 10.00th=[ 86], 20.00th=[ 94], | 30.00th=[ 102], 40.00th=[ 109], 50.00th=[ 116], 60.00th=[ 123], | 70.00th=[ 132], 80.00th=[ 142], 90.00th=[ 157], 95.00th=[ 167], | 99.00th=[ 190], 99.50th=[ 205], 99.90th=[ 213], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 87, max= 304, per=24.71%, avg=162.58, stdev=48.97, samples=60 iops : min= 21, max= 76, avg=40.52, stdev=12.27, samples=60 write: IOPS=43, BW=173KiB/s (177kB/s)(5264KiB/30481msec); 0 zone resets slat (usec): min=23, max=1926, avg=136.70, stdev=96.71 clat (msec): min=97, max=460, avg=255.35, stdev=66.10 lat (msec): min=97, max=460, avg=255.48, stdev=66.11 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 146], 10.00th=[ 169], 20.00th=[ 203], | 30.00th=[ 222], 40.00th=[ 236], 50.00th=[ 251], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 313], 90.00th=[ 342], 95.00th=[ 376], | 99.00th=[ 418], 99.50th=[ 422], 99.90th=[ 435], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 72, max= 280, per=25.12%, avg=172.08, stdev=35.73, samples=60 iops : min= 18, max= 70, avg=42.88, stdev= 9.00, samples=60 lat (msec) : 10=0.12%, 20=0.16%, 50=0.35%, 100=12.81%, 250=60.74% lat (msec) : 500=25.82% cpu : usr=0.24%, sys=1.40%, ctx=4906, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147687: Mon Dec 22 10:13:27 2025 read: IOPS=40, BW=164KiB/s (167kB/s)(4984KiB/30481msec) slat (usec): min=1297, max=9708, avg=3565.06, stdev=1276.27 clat (msec): min=3, max=277, avg=117.68, stdev=29.38 lat (msec): min=6, max=280, avg=121.25, stdev=29.76 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 75], 10.00th=[ 84], 20.00th=[ 93], | 30.00th=[ 102], 40.00th=[ 110], 50.00th=[ 116], 60.00th=[ 124], | 70.00th=[ 131], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 169], | 99.00th=[ 199], 99.50th=[ 209], 99.90th=[ 236], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 79, max= 336, per=24.89%, avg=163.75, stdev=57.03, samples=60 iops : min= 19, max= 84, avg=40.80, stdev=14.33, samples=60 write: IOPS=43, BW=172KiB/s (177kB/s)(5256KiB/30481msec); 0 zone resets slat (usec): min=36, max=1294, avg=136.25, stdev=95.57 clat (msec): min=62, max=463, avg=255.39, stdev=67.45 lat (msec): min=63, max=464, avg=255.53, stdev=67.45 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 148], 10.00th=[ 174], 20.00th=[ 197], | 30.00th=[ 222], 40.00th=[ 236], 50.00th=[ 249], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 309], 90.00th=[ 351], 95.00th=[ 380], | 99.00th=[ 414], 99.50th=[ 422], 99.90th=[ 447], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 96, max= 256, per=25.01%, avg=171.33, stdev=35.97, samples=60 iops : min= 24, max= 64, avg=42.70, stdev= 9.04, samples=60 lat (msec) : 4=0.04%, 10=0.04%, 20=0.08%, 50=0.12%, 100=13.59% lat (msec) : 250=61.09%, 500=25.04% cpu : usr=0.21%, sys=1.41%, ctx=5114, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147688: Mon Dec 22 10:13:27 2025 read: IOPS=41, BW=166KiB/s (170kB/s)(5060KiB/30438msec) slat (usec): min=1162, max=11018, avg=3551.29, stdev=1204.59 clat (msec): min=6, max=226, avg=118.11, stdev=28.23 lat (msec): min=9, max=230, avg=121.67, stdev=28.56 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 78], 10.00th=[ 87], 20.00th=[ 95], | 30.00th=[ 103], 40.00th=[ 109], 50.00th=[ 116], 60.00th=[ 123], | 70.00th=[ 131], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 165], | 99.00th=[ 199], 99.50th=[ 207], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 87, max= 296, per=25.34%, avg=166.77, stdev=42.99, samples=60 iops : min= 21, max= 74, avg=41.57, stdev=10.85, samples=60 write: IOPS=42, BW=170KiB/s (174kB/s)(5180KiB/30438msec); 0 zone resets slat (usec): min=25, max=8072, avg=149.75, stdev=285.73 clat (msec): min=83, max=441, avg=256.40, stdev=67.12 lat (msec): min=83, max=441, avg=256.56, stdev=67.10 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 146], 10.00th=[ 169], 20.00th=[ 203], | 30.00th=[ 222], 40.00th=[ 236], 50.00th=[ 251], 60.00th=[ 271], | 70.00th=[ 292], 80.00th=[ 313], 90.00th=[ 347], 95.00th=[ 380], | 99.00th=[ 414], 99.50th=[ 418], 99.90th=[ 430], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 104, max= 280, per=24.74%, avg=169.48, stdev=36.36, samples=60 iops : min= 26, max= 70, avg=42.23, stdev= 9.11, samples=60 lat (msec) : 10=0.04%, 20=0.04%, 50=0.16%, 100=13.55%, 250=60.51% lat (msec) : 500=25.70% cpu : usr=0.22%, sys=1.41%, ctx=5109, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=658KiB/s (674kB/s), 163KiB/s-167KiB/s (167kB/s-171kB/s), io=19.6MiB (20.5MB), run=30270-30481msec WRITE: bw=685KiB/s (702kB/s), 170KiB/s-173KiB/s (174kB/s-177kB/s), io=20.4MiB (21.4MB), run=30270-30481msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147696: Mon Dec 22 10:13:31 2025 read: IOPS=7, BW=313MiB/s (328MB/s)(40.0MiB/128msec) slat (nsec): min=14940k, max=14940k, avg=14939826.00, stdev= 0.00 clat (nsec): min=104591k, max=104591k, avg=104590986.00, stdev= 0.00 lat (nsec): min=119537k, max=119537k, avg=119537264.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=10.24%, ctx=43, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=313MiB/s (328MB/s), 313MiB/s-313MiB/s (328MB/s-328MB/s), io=40.0MiB (41.9MB), run=128-128msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout