== sanity test 398c: run fio to test AIO ================= 21:20:19 (1781227219) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.266732 s, 157 MB/s osc.lustre-OST0000-osc-ffffa08a71098800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154313: Thu Jun 11 21:21:23 2026 write: IOPS=43, BW=175KiB/s (179kB/s)(10.0MiB/58677msec); 0 zone resets slat (usec): min=36, max=6233, avg=267.94, stdev=282.19 clat (msec): min=117, max=658, avg=366.25, stdev=82.86 lat (msec): min=118, max=658, avg=366.52, stdev=82.86 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 243], 10.00th=[ 275], 20.00th=[ 305], | 30.00th=[ 326], 40.00th=[ 342], 50.00th=[ 359], 60.00th=[ 376], | 70.00th=[ 397], 80.00th=[ 426], 90.00th=[ 481], 95.00th=[ 518], | 99.00th=[ 600], 99.50th=[ 609], 99.90th=[ 642], 99.95th=[ 651], | 99.99th=[ 659] bw ( KiB/s): min= 87, max= 314, per=25.02%, avg=174.13, stdev=37.76, samples=116 iops : min= 21, max= 78, avg=43.23, stdev= 9.46, samples=116 lat (msec) : 250=5.90%, 500=87.54%, 750=6.56% cpu : usr=0.10%, sys=0.80%, ctx=3146, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154314: Thu Jun 11 21:21:23 2026 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58783msec); 0 zone resets slat (usec): min=36, max=7743, avg=253.34, stdev=276.70 clat (msec): min=22, max=659, avg=366.62, stdev=83.20 lat (msec): min=22, max=659, avg=366.87, stdev=83.20 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 247], 10.00th=[ 275], 20.00th=[ 305], | 30.00th=[ 321], 40.00th=[ 342], 50.00th=[ 363], 60.00th=[ 380], | 70.00th=[ 397], 80.00th=[ 426], 90.00th=[ 481], 95.00th=[ 527], | 99.00th=[ 584], 99.50th=[ 600], 99.90th=[ 617], 99.95th=[ 625], | 99.99th=[ 659] bw ( KiB/s): min= 88, max= 307, per=24.85%, avg=172.97, stdev=39.02, samples=117 iops : min= 22, max= 76, avg=42.94, stdev= 9.73, samples=117 lat (msec) : 50=0.08%, 100=0.04%, 250=5.04%, 500=87.42%, 750=7.42% cpu : usr=0.09%, sys=0.79%, ctx=3002, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154315: Thu Jun 11 21:21:23 2026 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58782msec); 0 zone resets slat (usec): min=39, max=5099, avg=268.17, stdev=271.93 clat (msec): min=150, max=657, avg=366.43, stdev=80.13 lat (msec): min=151, max=657, avg=366.71, stdev=80.13 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 251], 10.00th=[ 275], 20.00th=[ 300], | 30.00th=[ 321], 40.00th=[ 342], 50.00th=[ 359], 60.00th=[ 376], | 70.00th=[ 397], 80.00th=[ 426], 90.00th=[ 481], 95.00th=[ 518], | 99.00th=[ 592], 99.50th=[ 617], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 659] bw ( KiB/s): min= 103, max= 256, per=24.88%, avg=173.15, stdev=34.56, samples=117 iops : min= 25, max= 64, avg=42.99, stdev= 8.65, samples=117 lat (msec) : 250=4.84%, 500=88.71%, 750=6.45% cpu : usr=0.11%, sys=0.80%, ctx=3143, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154316: Thu Jun 11 21:21:23 2026 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58816msec); 0 zone resets slat (usec): min=38, max=7290, avg=264.50, stdev=295.80 clat (msec): min=128, max=658, avg=366.74, stdev=79.73 lat (msec): min=129, max=658, avg=367.01, stdev=79.72 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 253], 10.00th=[ 275], 20.00th=[ 305], | 30.00th=[ 326], 40.00th=[ 342], 50.00th=[ 363], 60.00th=[ 376], | 70.00th=[ 397], 80.00th=[ 426], 90.00th=[ 472], 95.00th=[ 518], | 99.00th=[ 592], 99.50th=[ 609], 99.90th=[ 651], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 71, max= 256, per=24.88%, avg=173.16, stdev=39.41, samples=117 iops : min= 17, max= 64, avg=42.99, stdev= 9.85, samples=117 lat (msec) : 250=4.80%, 500=88.98%, 750=6.21% cpu : usr=0.09%, sys=0.77%, ctx=3019, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=696KiB/s (713kB/s), 174KiB/s-175KiB/s (178kB/s-179kB/s), io=40.0MiB (41.9MB), run=58677-58816msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154334: Thu Jun 11 21:22:03 2026 read: IOPS=34, BW=139KiB/s (143kB/s)(5048KiB/36220msec) slat (usec): min=1767, max=15240, avg=4703.62, stdev=1815.35 clat (msec): min=14, max=346, avg=143.25, stdev=35.51 lat (msec): min=18, max=351, avg=147.96, stdev=35.89 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 116], | 30.00th=[ 125], 40.00th=[ 133], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 205], | 99.00th=[ 243], 99.50th=[ 262], 99.90th=[ 284], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 63, max= 208, per=25.11%, avg=138.60, stdev=33.56, samples=72 iops : min= 15, max= 52, avg=34.39, stdev= 8.37, samples=72 write: IOPS=35, BW=143KiB/s (147kB/s)(5192KiB/36220msec); 0 zone resets slat (usec): min=56, max=3639, avg=274.53, stdev=237.75 clat (msec): min=72, max=572, avg=301.40, stdev=76.55 lat (msec): min=72, max=572, avg=301.67, stdev=76.56 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 176], 10.00th=[ 201], 20.00th=[ 239], | 30.00th=[ 264], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 338], 80.00th=[ 368], 90.00th=[ 401], 95.00th=[ 435], | 99.00th=[ 514], 99.50th=[ 535], 99.90th=[ 550], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 72, max= 208, per=24.75%, avg=142.31, stdev=31.40, samples=72 iops : min= 18, max= 52, avg=35.31, stdev= 7.86, samples=72 lat (msec) : 20=0.04%, 50=0.35%, 100=3.79%, 250=56.99%, 500=38.16% lat (msec) : 750=0.66% cpu : usr=0.21%, sys=1.73%, ctx=5086, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154335: Thu Jun 11 21:22:03 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4976KiB/36330msec) slat (usec): min=59, max=15850, avg=4706.71, stdev=1788.57 clat (msec): min=4, max=273, avg=143.83, stdev=35.52 lat (msec): min=9, max=276, avg=148.54, stdev=35.85 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 96], 10.00th=[ 106], 20.00th=[ 118], | 30.00th=[ 126], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 190], 95.00th=[ 207], | 99.00th=[ 241], 99.50th=[ 247], 99.90th=[ 268], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 63, max= 256, per=24.71%, avg=136.38, stdev=43.65, samples=72 iops : min= 15, max= 64, avg=33.83, stdev=10.94, samples=72 write: IOPS=36, BW=145KiB/s (148kB/s)(5264KiB/36330msec); 0 zone resets slat (usec): min=56, max=3528, avg=267.66, stdev=221.25 clat (msec): min=44, max=580, avg=300.73, stdev=80.66 lat (msec): min=44, max=580, avg=301.00, stdev=80.67 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 176], 10.00th=[ 199], 20.00th=[ 236], | 30.00th=[ 259], 40.00th=[ 279], 50.00th=[ 300], 60.00th=[ 321], | 70.00th=[ 342], 80.00th=[ 363], 90.00th=[ 401], 95.00th=[ 435], | 99.00th=[ 514], 99.50th=[ 542], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 87, max= 233, per=25.08%, avg=144.22, stdev=31.79, samples=72 iops : min= 21, max= 58, avg=35.79, stdev= 7.92, samples=72 lat (msec) : 10=0.08%, 20=0.23%, 50=0.43%, 100=2.93%, 250=57.81% lat (msec) : 500=37.93%, 750=0.59% cpu : usr=0.18%, sys=1.69%, ctx=5005, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154336: Thu Jun 11 21:22:03 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4984KiB/36330msec) slat (usec): min=1823, max=15749, avg=4739.43, stdev=1797.53 clat (msec): min=6, max=281, avg=142.79, stdev=33.69 lat (msec): min=10, max=291, avg=147.53, stdev=34.04 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 95], 10.00th=[ 104], 20.00th=[ 117], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 186], 95.00th=[ 203], | 99.00th=[ 243], 99.50th=[ 255], 99.90th=[ 268], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 31, max= 288, per=24.88%, avg=137.35, stdev=46.64, samples=72 iops : min= 7, max= 72, avg=34.07, stdev=11.72, samples=72 write: IOPS=36, BW=145KiB/s (148kB/s)(5256KiB/36330msec); 0 zone resets slat (usec): min=59, max=3746, avg=278.53, stdev=244.65 clat (msec): min=79, max=560, avg=301.47, stdev=76.08 lat (msec): min=79, max=560, avg=301.75, stdev=76.09 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 180], 10.00th=[ 207], 20.00th=[ 241], | 30.00th=[ 259], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 368], 90.00th=[ 401], 95.00th=[ 435], | 99.00th=[ 498], 99.50th=[ 531], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 88, max= 224, per=24.98%, avg=143.64, stdev=31.30, samples=72 iops : min= 22, max= 56, avg=35.64, stdev= 7.82, samples=72 lat (msec) : 10=0.08%, 20=0.04%, 50=0.16%, 100=3.95%, 250=56.99% lat (msec) : 500=38.28%, 750=0.51% cpu : usr=0.21%, sys=1.71%, ctx=4999, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154337: Thu Jun 11 21:22:03 2026 read: IOPS=35, BW=140KiB/s (144kB/s)(5060KiB/36104msec) slat (usec): min=1728, max=14749, avg=4744.20, stdev=1756.52 clat (msec): min=5, max=327, avg=143.66, stdev=34.95 lat (msec): min=10, max=335, avg=148.41, stdev=35.21 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 97], 10.00th=[ 107], 20.00th=[ 118], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 243], 99.50th=[ 257], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 64, max= 280, per=25.40%, avg=140.20, stdev=39.09, samples=71 iops : min= 16, max= 70, avg=34.79, stdev= 9.81, samples=71 write: IOPS=35, BW=143KiB/s (147kB/s)(5180KiB/36104msec); 0 zone resets slat (usec): min=59, max=3114, avg=280.28, stdev=254.43 clat (msec): min=48, max=536, avg=300.09, stdev=77.35 lat (msec): min=48, max=536, avg=300.37, stdev=77.36 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 178], 10.00th=[ 203], 20.00th=[ 236], | 30.00th=[ 259], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 338], 80.00th=[ 368], 90.00th=[ 397], 95.00th=[ 435], | 99.00th=[ 493], 99.50th=[ 514], 99.90th=[ 527], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 87, max= 232, per=24.82%, avg=142.69, stdev=31.10, samples=71 iops : min= 21, max= 58, avg=35.41, stdev= 7.81, samples=71 lat (msec) : 10=0.16%, 20=0.12%, 50=0.35%, 100=2.73%, 250=58.28% lat (msec) : 500=37.93%, 750=0.43% cpu : usr=0.19%, sys=1.75%, ctx=5042, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=552KiB/s (566kB/s), 137KiB/s-140KiB/s (140kB/s-144kB/s), io=19.6MiB (20.5MB), run=36104-36330msec WRITE: bw=575KiB/s (589kB/s), 143KiB/s-145KiB/s (147kB/s-148kB/s), io=20.4MiB (21.4MB), run=36104-36330msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154346: Thu Jun 11 21:22:07 2026 read: IOPS=4, BW=199MiB/s (209MB/s)(40.0MiB/201msec) slat (nsec): min=14372k, max=14372k, avg=14372463.00, stdev= 0.00 clat (nsec): min=175307k, max=175307k, avg=175307385.00, stdev= 0.00 lat (nsec): min=189684k, max=189684k, avg=189683675.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.50%, ctx=43, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=199MiB/s (209MB/s), 199MiB/s-199MiB/s (209MB/s-209MB/s), io=40.0MiB (41.9MB), run=201-201msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout