== sanity test 398c: run fio to test AIO ================= 13:16:42 (1772475402) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.166195 s, 252 MB/s osc.lustre-OST0000-osc-ffff9041787b0800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147963: Mon Mar 2 13:17:30 2026 write: IOPS=58, BW=232KiB/s (238kB/s)(10.0MiB/44112msec); 0 zone resets slat (usec): min=37, max=2591, avg=224.20, stdev=153.75 clat (msec): min=60, max=603, avg=275.21, stdev=73.87 lat (msec): min=61, max=603, avg=275.44, stdev=73.88 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 163], 10.00th=[ 192], 20.00th=[ 220], | 30.00th=[ 236], 40.00th=[ 253], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 300], 80.00th=[ 330], 90.00th=[ 372], 95.00th=[ 401], | 99.00th=[ 506], 99.50th=[ 535], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 126, max= 414, per=24.99%, avg=231.91, stdev=56.53, samples=87 iops : min= 31, max= 103, avg=57.80, stdev=14.20, samples=87 lat (msec) : 100=0.66%, 250=37.23%, 500=61.02%, 750=1.09% cpu : usr=0.14%, sys=1.02%, ctx=2870, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147964: Mon Mar 2 13:17:30 2026 write: IOPS=58, BW=233KiB/s (238kB/s)(10.0MiB/43972msec); 0 zone resets slat (usec): min=36, max=3606, avg=231.35, stdev=197.66 clat (msec): min=28, max=713, avg=274.39, stdev=76.01 lat (msec): min=28, max=713, avg=274.62, stdev=76.03 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 157], 10.00th=[ 192], 20.00th=[ 218], | 30.00th=[ 234], 40.00th=[ 251], 50.00th=[ 271], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 368], 95.00th=[ 405], | 99.00th=[ 506], 99.50th=[ 535], 99.90th=[ 617], 99.95th=[ 693], | 99.99th=[ 718] bw ( KiB/s): min= 126, max= 517, per=25.03%, avg=232.26, stdev=60.66, samples=87 iops : min= 31, max= 129, avg=57.90, stdev=15.23, samples=87 lat (msec) : 50=0.62%, 250=38.12%, 500=60.23%, 750=1.02% cpu : usr=0.13%, sys=1.01%, ctx=2788, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147965: Mon Mar 2 13:17:30 2026 write: IOPS=58, BW=233KiB/s (238kB/s)(10.0MiB/43987msec); 0 zone resets slat (usec): min=44, max=3389, avg=230.82, stdev=177.00 clat (msec): min=32, max=585, avg=274.44, stdev=73.72 lat (msec): min=33, max=585, avg=274.67, stdev=73.73 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 161], 10.00th=[ 190], 20.00th=[ 218], | 30.00th=[ 236], 40.00th=[ 253], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 368], 95.00th=[ 401], | 99.00th=[ 498], 99.50th=[ 523], 99.90th=[ 550], 99.95th=[ 567], | 99.99th=[ 584] bw ( KiB/s): min= 126, max= 420, per=25.06%, avg=232.53, stdev=56.54, samples=87 iops : min= 31, max= 105, avg=57.97, stdev=14.22, samples=87 lat (msec) : 50=0.04%, 100=0.59%, 250=37.66%, 500=60.82%, 750=0.90% cpu : usr=0.12%, sys=1.03%, ctx=2885, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147966: Mon Mar 2 13:17:30 2026 write: IOPS=58, BW=232KiB/s (238kB/s)(10.0MiB/44111msec); 0 zone resets slat (usec): min=39, max=4863, avg=235.64, stdev=218.31 clat (msec): min=18, max=564, avg=275.20, stdev=72.83 lat (msec): min=19, max=564, avg=275.44, stdev=72.84 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 163], 10.00th=[ 197], 20.00th=[ 218], | 30.00th=[ 236], 40.00th=[ 253], 50.00th=[ 271], 60.00th=[ 284], | 70.00th=[ 300], 80.00th=[ 334], 90.00th=[ 372], 95.00th=[ 401], | 99.00th=[ 498], 99.50th=[ 523], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 567] bw ( KiB/s): min= 87, max= 420, per=24.95%, avg=231.53, stdev=58.06, samples=87 iops : min= 21, max= 105, avg=57.71, stdev=14.61, samples=87 lat (msec) : 20=0.04%, 100=0.23%, 250=38.40%, 500=60.39%, 750=0.94% cpu : usr=0.10%, sys=1.03%, ctx=2894, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=929KiB/s (951kB/s), 232KiB/s-233KiB/s (238kB/s-238kB/s), io=40.0MiB (41.9MB), run=43972-44112msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147983: Mon Mar 2 13:18:06 2026 read: IOPS=38, BW=155KiB/s (158kB/s)(5048KiB/32655msec) slat (usec): min=87, max=15327, avg=4129.15, stdev=1996.18 clat (msec): min=5, max=386, avg=128.87, stdev=37.98 lat (msec): min=5, max=393, avg=133.01, stdev=38.62 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 84], 10.00th=[ 93], 20.00th=[ 103], | 30.00th=[ 110], 40.00th=[ 116], 50.00th=[ 123], 60.00th=[ 130], | 70.00th=[ 140], 80.00th=[ 155], 90.00th=[ 178], 95.00th=[ 199], | 99.00th=[ 241], 99.50th=[ 251], 99.90th=[ 368], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 63, max= 248, per=25.41%, avg=155.25, stdev=46.21, samples=64 iops : min= 15, max= 62, avg=38.56, stdev=11.61, samples=64 write: IOPS=39, BW=159KiB/s (163kB/s)(5192KiB/32655msec); 0 zone resets slat (usec): min=59, max=6022, avg=242.56, stdev=286.94 clat (msec): min=51, max=552, avg=272.59, stdev=83.32 lat (msec): min=51, max=552, avg=272.84, stdev=83.34 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 167], 10.00th=[ 186], 20.00th=[ 209], | 30.00th=[ 224], 40.00th=[ 239], 50.00th=[ 255], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 326], 90.00th=[ 405], 95.00th=[ 447], | 99.00th=[ 514], 99.50th=[ 531], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 71, max= 240, per=25.04%, avg=159.27, stdev=39.52, samples=64 iops : min= 17, max= 60, avg=39.56, stdev= 9.93, samples=64 lat (msec) : 10=0.20%, 20=0.16%, 50=0.27%, 100=8.24%, 250=63.75% lat (msec) : 500=26.52%, 750=0.86% cpu : usr=0.20%, sys=1.70%, ctx=4946, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147984: Mon Mar 2 13:18:06 2026 read: IOPS=37, BW=152KiB/s (155kB/s)(4976KiB/32830msec) slat (usec): min=1378, max=15895, avg=4188.10, stdev=1905.68 clat (msec): min=5, max=258, avg=129.07, stdev=35.73 lat (msec): min=9, max=262, avg=133.26, stdev=36.23 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 82], 10.00th=[ 91], 20.00th=[ 103], | 30.00th=[ 110], 40.00th=[ 117], 50.00th=[ 124], 60.00th=[ 133], | 70.00th=[ 142], 80.00th=[ 157], 90.00th=[ 178], 95.00th=[ 197], | 99.00th=[ 239], 99.50th=[ 247], 99.90th=[ 253], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 48, max= 272, per=24.73%, avg=151.12, stdev=44.61, samples=65 iops : min= 12, max= 68, avg=37.54, stdev=11.11, samples=65 write: IOPS=40, BW=160KiB/s (164kB/s)(5264KiB/32830msec); 0 zone resets slat (usec): min=35, max=3772, avg=235.53, stdev=222.52 clat (msec): min=59, max=565, avg=272.13, stdev=80.10 lat (msec): min=59, max=566, avg=272.37, stdev=80.11 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 171], 10.00th=[ 186], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 239], 50.00th=[ 257], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 326], 90.00th=[ 388], 95.00th=[ 443], | 99.00th=[ 506], 99.50th=[ 523], 99.90th=[ 535], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 71, max= 240, per=25.17%, avg=160.08, stdev=40.45, samples=65 iops : min= 17, max= 60, avg=39.77, stdev=10.20, samples=65 lat (msec) : 10=0.08%, 20=0.12%, 50=0.35%, 100=8.36%, 250=63.63% lat (msec) : 500=26.84%, 750=0.62% cpu : usr=0.22%, sys=1.67%, ctx=4854, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147985: Mon Mar 2 13:18:06 2026 read: IOPS=37, BW=152KiB/s (156kB/s)(4984KiB/32811msec) slat (usec): min=1203, max=15166, avg=4189.94, stdev=2001.59 clat (msec): min=5, max=285, avg=130.79, stdev=36.25 lat (msec): min=11, max=288, avg=134.98, stdev=36.80 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 83], 10.00th=[ 92], 20.00th=[ 103], | 30.00th=[ 111], 40.00th=[ 118], 50.00th=[ 125], 60.00th=[ 133], | 70.00th=[ 144], 80.00th=[ 159], 90.00th=[ 178], 95.00th=[ 199], | 99.00th=[ 249], 99.50th=[ 253], 99.90th=[ 259], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 79, max= 256, per=24.90%, avg=152.15, stdev=41.97, samples=65 iops : min= 19, max= 64, avg=37.78, stdev=10.57, samples=65 write: IOPS=40, BW=160KiB/s (164kB/s)(5256KiB/32811msec); 0 zone resets slat (usec): min=50, max=3067, avg=231.77, stdev=178.35 clat (msec): min=116, max=569, avg=270.40, stdev=79.96 lat (msec): min=117, max=569, avg=270.63, stdev=79.95 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 167], 10.00th=[ 186], 20.00th=[ 207], | 30.00th=[ 222], 40.00th=[ 239], 50.00th=[ 255], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 326], 90.00th=[ 384], 95.00th=[ 439], | 99.00th=[ 502], 99.50th=[ 518], 99.90th=[ 535], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 80, max= 256, per=25.09%, avg=159.58, stdev=42.70, samples=65 iops : min= 20, max= 64, avg=39.65, stdev=10.71, samples=65 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=8.40%, 250=63.91% lat (msec) : 500=26.99%, 750=0.51% cpu : usr=0.16%, sys=1.74%, ctx=4958, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147986: Mon Mar 2 13:18:06 2026 read: IOPS=38, BW=154KiB/s (158kB/s)(5060KiB/32813msec) slat (usec): min=1493, max=13933, avg=4141.91, stdev=1814.77 clat (msec): min=3, max=306, avg=129.77, stdev=36.11 lat (msec): min=8, max=309, avg=133.92, stdev=36.61 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 83], 10.00th=[ 92], 20.00th=[ 103], | 30.00th=[ 111], 40.00th=[ 117], 50.00th=[ 124], 60.00th=[ 132], | 70.00th=[ 142], 80.00th=[ 155], 90.00th=[ 176], 95.00th=[ 199], | 99.00th=[ 243], 99.50th=[ 249], 99.90th=[ 305], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 16, max= 271, per=25.21%, avg=154.03, stdev=53.82, samples=65 iops : min= 4, max= 67, avg=38.26, stdev=13.49, samples=65 write: IOPS=39, BW=158KiB/s (162kB/s)(5180KiB/32813msec); 0 zone resets slat (usec): min=42, max=4316, avg=242.38, stdev=235.18 clat (msec): min=114, max=529, avg=273.26, stdev=80.64 lat (msec): min=114, max=529, avg=273.51, stdev=80.64 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 167], 10.00th=[ 184], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 243], 50.00th=[ 262], 60.00th=[ 279], | 70.00th=[ 300], 80.00th=[ 330], 90.00th=[ 397], 95.00th=[ 443], | 99.00th=[ 502], 99.50th=[ 506], 99.90th=[ 527], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 72, max= 256, per=24.82%, avg=157.83, stdev=39.98, samples=65 iops : min= 18, max= 64, avg=39.22, stdev=10.03, samples=65 lat (msec) : 4=0.04%, 20=0.08%, 50=0.20%, 100=8.16%, 250=63.28% lat (msec) : 500=27.73%, 750=0.51% cpu : usr=0.18%, sys=1.71%, ctx=4886, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=611KiB/s (626kB/s), 152KiB/s-155KiB/s (155kB/s-158kB/s), io=19.6MiB (20.5MB), run=32655-32830msec WRITE: bw=636KiB/s (652kB/s), 158KiB/s-160KiB/s (162kB/s-164kB/s), io=20.4MiB (21.4MB), run=32655-32830msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147994: Mon Mar 2 13:18:10 2026 read: IOPS=6, BW=263MiB/s (276MB/s)(40.0MiB/152msec) slat (nsec): min=13375k, max=13375k, avg=13375265.00, stdev= 0.00 clat (nsec): min=129248k, max=129248k, avg=129247658.00, stdev= 0.00 lat (nsec): min=142628k, max=142628k, avg=142628402.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.95%, ctx=35, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=263MiB/s (276MB/s), 263MiB/s-263MiB/s (276MB/s-276MB/s), io=40.0MiB (41.9MB), run=152-152msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout