== sanity test 398c: run fio to test AIO ================= 18:20:41 (1781216441) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.424507 s, 98.8 MB/s osc.lustre-OST0000-osc-ffff9992c6bb5000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180107: Thu Jun 11 18:21:25 2026 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34284msec); 0 zone resets slat (usec): min=37, max=15455, avg=261.56, stdev=809.70 clat (msec): min=65, max=441, avg=213.82, stdev=50.26 lat (msec): min=66, max=441, avg=214.08, stdev=50.31 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 132], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 330], 99.50th=[ 334], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 443] bw ( KiB/s): min= 125, max= 510, per=24.89%, avg=296.69, stdev=74.09, samples=68 iops : min= 31, max= 127, avg=73.75, stdev=18.51, samples=68 lat (msec) : 100=0.66%, 250=75.51%, 500=23.83% cpu : usr=0.09%, sys=0.83%, ctx=1705, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180108: Thu Jun 11 18:21:25 2026 write: IOPS=74, BW=298KiB/s (305kB/s)(10.0MiB/34360msec); 0 zone resets slat (usec): min=37, max=15607, avg=286.04, stdev=873.44 clat (msec): min=28, max=362, avg=213.20, stdev=50.84 lat (msec): min=28, max=362, avg=213.49, stdev=50.86 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 142], 10.00th=[ 153], 20.00th=[ 165], | 30.00th=[ 184], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 347], 99.95th=[ 359], | 99.99th=[ 363] bw ( KiB/s): min= 160, max= 461, per=24.89%, avg=296.65, stdev=62.71, samples=68 iops : min= 40, max= 115, avg=73.74, stdev=15.68, samples=68 lat (msec) : 50=0.20%, 100=0.20%, 250=76.56%, 500=23.05% cpu : usr=0.18%, sys=1.08%, ctx=2397, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180109: Thu Jun 11 18:21:25 2026 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34267msec); 0 zone resets slat (usec): min=52, max=19237, avg=321.61, stdev=1038.78 clat (msec): min=20, max=368, avg=212.70, stdev=52.83 lat (msec): min=20, max=368, avg=213.03, stdev=52.81 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 241], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 368] bw ( KiB/s): min= 159, max= 470, per=24.95%, avg=297.35, stdev=63.29, samples=68 iops : min= 39, max= 117, avg=73.93, stdev=15.80, samples=68 lat (msec) : 50=0.20%, 100=0.43%, 250=74.10%, 500=25.27% cpu : usr=0.14%, sys=1.06%, ctx=2563, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180110: Thu Jun 11 18:21:25 2026 write: IOPS=74, BW=298KiB/s (306kB/s)(10.0MiB/34315msec); 0 zone resets slat (usec): min=38, max=18656, avg=317.54, stdev=950.42 clat (msec): min=29, max=368, avg=213.10, stdev=51.54 lat (msec): min=29, max=368, avg=213.42, stdev=51.57 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 142], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 128, max= 478, per=24.94%, avg=297.24, stdev=62.29, samples=68 iops : min= 32, max= 119, avg=73.90, stdev=15.53, samples=68 lat (msec) : 50=0.08%, 100=0.31%, 250=74.88%, 500=24.73% cpu : usr=0.17%, sys=1.09%, ctx=2819, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1192KiB/s (1221kB/s), 298KiB/s-299KiB/s (305kB/s-306kB/s), io=40.0MiB (41.9MB), run=34267-34360msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180125: Thu Jun 11 18:22:12 2026 read: IOPS=33, BW=133KiB/s (136kB/s)(5048KiB/37922msec) slat (usec): min=2463, max=68135, avg=16767.02, stdev=9103.21 clat (msec): min=19, max=417, avg=222.03, stdev=51.17 lat (msec): min=25, max=446, avg=238.80, stdev=52.55 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 146], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 220], 60.00th=[ 232], | 70.00th=[ 247], 80.00th=[ 264], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 393], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 62, max= 212, per=25.00%, avg=132.25, stdev=31.71, samples=75 iops : min= 15, max= 53, avg=32.57, stdev= 7.98, samples=75 write: IOPS=34, BW=137KiB/s (140kB/s)(5192KiB/37922msec); 0 zone resets slat (usec): min=41, max=37433, avg=452.25, stdev=1605.40 clat (msec): min=40, max=461, avg=230.76, stdev=51.18 lat (msec): min=41, max=461, avg=231.22, stdev=51.13 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 188], | 30.00th=[ 203], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 376], 99.50th=[ 388], 99.90th=[ 456], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 41, max= 223, per=24.85%, avg=136.68, stdev=32.62, samples=75 iops : min= 10, max= 55, avg=33.68, stdev= 8.18, samples=75 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=69.30%, 500=30.43% cpu : usr=0.25%, sys=1.98%, ctx=4353, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180126: Thu Jun 11 18:22:12 2026 read: IOPS=32, BW=131KiB/s (134kB/s)(4976KiB/37926msec) slat (usec): min=3226, max=56373, avg=16308.20, stdev=8681.49 clat (msec): min=112, max=407, avg=221.79, stdev=51.28 lat (msec): min=120, max=415, avg=238.10, stdev=52.37 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 142], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 247], 80.00th=[ 266], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 363], 99.50th=[ 372], 99.90th=[ 388], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 34, max= 198, per=24.64%, avg=130.33, stdev=32.39, samples=75 iops : min= 8, max= 49, avg=32.08, stdev= 8.12, samples=75 write: IOPS=34, BW=139KiB/s (142kB/s)(5264KiB/37926msec); 0 zone resets slat (usec): min=54, max=18811, avg=371.07, stdev=1146.83 clat (msec): min=80, max=419, avg=231.75, stdev=50.14 lat (msec): min=80, max=419, avg=232.12, stdev=50.14 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 159], 10.00th=[ 171], 20.00th=[ 188], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 321], | 99.00th=[ 372], 99.50th=[ 393], 99.90th=[ 418], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 76, max= 200, per=25.21%, avg=138.67, stdev=33.99, samples=75 iops : min= 19, max= 50, avg=34.17, stdev= 8.55, samples=75 lat (msec) : 100=0.04%, 250=69.49%, 500=30.47% cpu : usr=0.24%, sys=2.01%, ctx=4487, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180127: Thu Jun 11 18:22:12 2026 read: IOPS=32, BW=132KiB/s (135kB/s)(4984KiB/37852msec) slat (usec): min=3056, max=59388, avg=16734.17, stdev=8868.14 clat (msec): min=39, max=458, avg=221.61, stdev=52.82 lat (msec): min=53, max=476, avg=238.35, stdev=54.11 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 249], 80.00th=[ 264], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 372], 99.50th=[ 388], 99.90th=[ 443], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 48, max= 223, per=24.80%, avg=131.20, stdev=32.82, samples=75 iops : min= 12, max= 55, avg=32.32, stdev= 8.15, samples=75 write: IOPS=34, BW=139KiB/s (142kB/s)(5256KiB/37852msec); 0 zone resets slat (usec): min=56, max=18327, avg=382.29, stdev=1094.70 clat (msec): min=102, max=420, avg=231.01, stdev=49.31 lat (msec): min=102, max=420, avg=231.39, stdev=49.25 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 157], 10.00th=[ 176], 20.00th=[ 188], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 363], 99.50th=[ 405], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 55, max= 239, per=25.04%, avg=137.69, stdev=37.34, samples=75 iops : min= 13, max= 59, avg=33.92, stdev= 9.43, samples=75 lat (msec) : 50=0.04%, 100=0.08%, 250=69.57%, 500=30.31% cpu : usr=0.23%, sys=1.85%, ctx=4381, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180128: Thu Jun 11 18:22:12 2026 read: IOPS=33, BW=135KiB/s (138kB/s)(5060KiB/37529msec) slat (usec): min=90, max=69769, avg=16816.82, stdev=9475.85 clat (msec): min=22, max=413, avg=221.61, stdev=51.80 lat (msec): min=23, max=461, avg=238.43, stdev=53.37 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 144], 10.00th=[ 161], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 207], 50.00th=[ 220], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 397], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 78, max= 216, per=25.42%, avg=134.49, stdev=29.45, samples=74 iops : min= 19, max= 54, avg=33.14, stdev= 7.42, samples=74 write: IOPS=34, BW=138KiB/s (141kB/s)(5180KiB/37529msec); 0 zone resets slat (usec): min=37, max=16845, avg=362.40, stdev=1065.59 clat (msec): min=98, max=437, avg=229.61, stdev=50.89 lat (msec): min=99, max=437, avg=229.97, stdev=50.89 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 155], 10.00th=[ 167], 20.00th=[ 186], | 30.00th=[ 199], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 241], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 393], 99.50th=[ 405], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 55, max= 247, per=24.89%, avg=136.88, stdev=34.41, samples=74 iops : min= 13, max= 61, avg=33.74, stdev= 8.67, samples=74 lat (msec) : 50=0.20%, 100=0.16%, 250=69.73%, 500=29.92% cpu : usr=0.24%, sys=1.92%, ctx=4416, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=529KiB/s (542kB/s), 131KiB/s-135KiB/s (134kB/s-138kB/s), io=19.6MiB (20.5MB), run=37529-37926msec WRITE: bw=551KiB/s (564kB/s), 137KiB/s-139KiB/s (140kB/s-142kB/s), io=20.4MiB (21.4MB), run=37529-37926msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180138: Thu Jun 11 18:22:23 2026 read: IOPS=1, BW=59.5MiB/s (62.4MB/s)(40.0MiB/672msec) slat (nsec): min=24944k, max=24944k, avg=24943539.00, stdev= 0.00 clat (nsec): min=616131k, max=616131k, avg=616130907.00, stdev= 0.00 lat (nsec): min=641080k, max=641080k, avg=641080469.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 617], 5.00th=[ 617], 10.00th=[ 617], 20.00th=[ 617], | 30.00th=[ 617], 40.00th=[ 617], 50.00th=[ 617], 60.00th=[ 617], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.38%, ctx=4, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=59.5MiB/s (62.4MB/s), 59.5MiB/s-59.5MiB/s (62.4MB/s-62.4MB/s), io=40.0MiB (41.9MB), run=672-672msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout