== sanity test 398c: run fio to test AIO ================= 10:52:03 (1785941523) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.636 s, 65.9 MB/s osc.lustre-OST0000-osc-ffff915c09d8f000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=178920: Wed Aug 5 10:52:55 2026 write: IOPS=66, BW=265KiB/s (271kB/s)(10.0MiB/38684msec); 0 zone resets slat (usec): min=27, max=31886, avg=144.25, stdev=874.68 clat (msec): min=100, max=1188, avg=240.41, stdev=117.78 lat (msec): min=100, max=1189, avg=240.56, stdev=117.86 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 126], 10.00th=[ 136], 20.00th=[ 163], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 239], | 70.00th=[ 264], 80.00th=[ 288], 90.00th=[ 342], 95.00th=[ 409], | 99.00th=[ 701], 99.50th=[ 1167], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 26, max= 512, per=25.08%, avg=265.37, stdev=103.60, samples=76 iops : min= 6, max= 128, avg=66.03, stdev=25.96, samples=76 lat (msec) : 250=63.98%, 500=32.85%, 750=2.54%, 2000=0.62% cpu : usr=0.09%, sys=0.56%, ctx=1698, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178921: Wed Aug 5 10:52:55 2026 write: IOPS=66, BW=265KiB/s (271kB/s)(10.0MiB/38659msec); 0 zone resets slat (usec): min=27, max=19639, avg=125.40, stdev=648.11 clat (msec): min=27, max=1212, avg=239.78, stdev=117.63 lat (msec): min=27, max=1212, avg=239.91, stdev=117.66 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 121], 10.00th=[ 140], 20.00th=[ 163], | 30.00th=[ 184], 40.00th=[ 201], 50.00th=[ 220], 60.00th=[ 239], | 70.00th=[ 266], 80.00th=[ 288], 90.00th=[ 342], 95.00th=[ 414], | 99.00th=[ 651], 99.50th=[ 1167], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 71, max= 510, per=25.05%, avg=265.04, stdev=96.83, samples=76 iops : min= 17, max= 127, avg=65.93, stdev=24.24, samples=76 lat (msec) : 50=0.04%, 100=0.78%, 250=63.16%, 500=33.67%, 750=1.72% lat (msec) : 2000=0.62% cpu : usr=0.06%, sys=0.50%, ctx=1124, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178922: Wed Aug 5 10:52:55 2026 write: IOPS=66, BW=265KiB/s (271kB/s)(10.0MiB/38694msec); 0 zone resets slat (usec): min=28, max=34732, avg=187.92, stdev=1244.34 clat (msec): min=27, max=1213, avg=240.44, stdev=118.21 lat (msec): min=28, max=1213, avg=240.63, stdev=118.29 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 123], 10.00th=[ 138], 20.00th=[ 163], | 30.00th=[ 184], 40.00th=[ 203], 50.00th=[ 222], 60.00th=[ 239], | 70.00th=[ 266], 80.00th=[ 288], 90.00th=[ 342], 95.00th=[ 409], | 99.00th=[ 642], 99.50th=[ 1167], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 526, per=25.06%, avg=265.09, stdev=96.89, samples=76 iops : min= 1, max= 131, avg=65.95, stdev=24.25, samples=76 lat (msec) : 50=0.04%, 100=0.62%, 250=64.06%, 500=31.80%, 750=2.85% lat (msec) : 2000=0.62% cpu : usr=0.15%, sys=0.70%, ctx=2474, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178923: Wed Aug 5 10:52:55 2026 write: IOPS=66, BW=266KiB/s (273kB/s)(10.0MiB/38442msec); 0 zone resets slat (usec): min=27, max=21216, avg=126.94, stdev=733.33 clat (msec): min=31, max=1186, avg=239.94, stdev=117.42 lat (msec): min=31, max=1186, avg=240.07, stdev=117.47 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 182], 40.00th=[ 201], 50.00th=[ 220], 60.00th=[ 241], | 70.00th=[ 264], 80.00th=[ 288], 90.00th=[ 342], 95.00th=[ 426], | 99.00th=[ 642], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 15, max= 550, per=25.24%, avg=266.99, stdev=99.30, samples=75 iops : min= 3, max= 137, avg=66.41, stdev=24.85, samples=75 lat (msec) : 50=0.08%, 100=1.05%, 250=61.91%, 500=34.53%, 750=1.80% lat (msec) : 2000=0.62% cpu : usr=0.10%, sys=0.50%, ctx=1786, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1059KiB/s (1084kB/s), 265KiB/s-266KiB/s (271kB/s-273kB/s), io=40.0MiB (41.9MB), run=38442-38694msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=178939: Wed Aug 5 10:53:54 2026 read: IOPS=27, BW=108KiB/s (111kB/s)(5048KiB/46726msec) slat (msec): min=2, max=161, avg=20.34, stdev=13.04 clat (msec): min=95, max=537, avg=274.74, stdev=67.01 lat (msec): min=102, max=559, avg=295.08, stdev=69.15 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 176], 10.00th=[ 197], 20.00th=[ 222], | 30.00th=[ 241], 40.00th=[ 255], 50.00th=[ 271], 60.00th=[ 284], | 70.00th=[ 300], 80.00th=[ 326], 90.00th=[ 363], 95.00th=[ 397], | 99.00th=[ 472], 99.50th=[ 502], 99.90th=[ 531], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 24, max= 189, per=24.88%, avg=106.75, stdev=30.23, samples=92 iops : min= 6, max= 47, avg=26.24, stdev= 7.57, samples=92 write: IOPS=27, BW=111KiB/s (114kB/s)(5192KiB/46726msec); 0 zone resets slat (usec): min=28, max=22081, avg=231.30, stdev=1270.85 clat (msec): min=56, max=548, avg=284.59, stdev=64.79 lat (msec): min=56, max=549, avg=284.82, stdev=64.76 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 188], 10.00th=[ 205], 20.00th=[ 232], | 30.00th=[ 251], 40.00th=[ 266], 50.00th=[ 279], 60.00th=[ 296], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 372], 95.00th=[ 393], | 99.00th=[ 477], 99.50th=[ 502], 99.90th=[ 535], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 23, max= 182, per=24.66%, avg=110.25, stdev=29.91, samples=92 iops : min= 5, max= 45, avg=27.10, stdev= 7.57, samples=92 lat (msec) : 100=0.12%, 250=33.52%, 500=65.82%, 750=0.55% cpu : usr=0.18%, sys=0.99%, ctx=3768, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178940: Wed Aug 5 10:53:54 2026 read: IOPS=26, BW=107KiB/s (109kB/s)(4976KiB/46563msec) slat (msec): min=2, max=147, avg=19.98, stdev=12.96 clat (msec): min=92, max=578, avg=274.78, stdev=67.21 lat (msec): min=121, max=632, avg=294.77, stdev=69.91 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 180], 10.00th=[ 197], 20.00th=[ 220], | 30.00th=[ 239], 40.00th=[ 251], 50.00th=[ 268], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 363], 95.00th=[ 388], | 99.00th=[ 464], 99.50th=[ 489], 99.90th=[ 550], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 39, max= 207, per=24.72%, avg=106.04, stdev=32.32, samples=92 iops : min= 9, max= 51, avg=26.05, stdev= 8.12, samples=92 write: IOPS=28, BW=113KiB/s (116kB/s)(5264KiB/46563msec); 0 zone resets slat (usec): min=28, max=14767, avg=159.91, stdev=719.77 clat (msec): min=115, max=556, avg=284.10, stdev=65.02 lat (msec): min=115, max=556, avg=284.26, stdev=65.00 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 190], 10.00th=[ 211], 20.00th=[ 236], | 30.00th=[ 253], 40.00th=[ 266], 50.00th=[ 279], 60.00th=[ 292], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 363], 95.00th=[ 401], | 99.00th=[ 502], 99.50th=[ 531], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 32, max= 192, per=25.22%, avg=112.75, stdev=29.84, samples=92 iops : min= 8, max= 48, avg=27.73, stdev= 7.49, samples=92 lat (msec) : 100=0.08%, 250=33.98%, 500=65.12%, 750=0.82% cpu : usr=0.17%, sys=1.11%, ctx=3852, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178941: Wed Aug 5 10:53:54 2026 read: IOPS=26, BW=107KiB/s (109kB/s)(4984KiB/46676msec) slat (usec): min=2247, max=98394, avg=20329.50, stdev=12928.19 clat (msec): min=15, max=531, avg=276.12, stdev=65.66 lat (msec): min=47, max=540, avg=296.46, stdev=67.54 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 182], 10.00th=[ 201], 20.00th=[ 224], | 30.00th=[ 239], 40.00th=[ 253], 50.00th=[ 275], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 321], 90.00th=[ 363], 95.00th=[ 397], | 99.00th=[ 468], 99.50th=[ 493], 99.90th=[ 514], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 39, max= 183, per=24.83%, avg=106.53, stdev=28.74, samples=92 iops : min= 9, max= 45, avg=26.15, stdev= 7.21, samples=92 write: IOPS=28, BW=113KiB/s (115kB/s)(5256KiB/46676msec); 0 zone resets slat (usec): min=30, max=21616, avg=198.16, stdev=1143.33 clat (msec): min=113, max=559, avg=282.60, stdev=63.24 lat (msec): min=113, max=559, avg=282.80, stdev=63.17 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 192], 10.00th=[ 209], 20.00th=[ 230], | 30.00th=[ 251], 40.00th=[ 266], 50.00th=[ 279], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 363], 95.00th=[ 393], | 99.00th=[ 468], 99.50th=[ 506], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 39, max= 205, per=25.05%, avg=111.98, stdev=30.25, samples=92 iops : min= 9, max= 51, avg=27.53, stdev= 7.60, samples=92 lat (msec) : 20=0.04%, 100=0.04%, 250=33.59%, 500=65.94%, 750=0.39% cpu : usr=0.15%, sys=0.94%, ctx=3619, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178942: Wed Aug 5 10:53:54 2026 read: IOPS=27, BW=109KiB/s (112kB/s)(5060KiB/46392msec) slat (usec): min=50, max=94627, avg=19868.89, stdev=13152.13 clat (msec): min=27, max=562, avg=275.37, stdev=67.67 lat (msec): min=30, max=582, avg=295.25, stdev=69.85 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 180], 10.00th=[ 199], 20.00th=[ 224], | 30.00th=[ 239], 40.00th=[ 255], 50.00th=[ 271], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 359], 95.00th=[ 393], | 99.00th=[ 481], 99.50th=[ 514], 99.90th=[ 542], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 54, max= 208, per=25.36%, avg=108.81, stdev=29.51, samples=91 iops : min= 13, max= 52, avg=26.74, stdev= 7.39, samples=91 write: IOPS=27, BW=112KiB/s (114kB/s)(5180KiB/46392msec); 0 zone resets slat (usec): min=28, max=50102, avg=201.98, stdev=1634.86 clat (msec): min=118, max=538, avg=284.08, stdev=63.88 lat (msec): min=118, max=538, avg=284.29, stdev=63.90 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 192], 10.00th=[ 211], 20.00th=[ 234], | 30.00th=[ 249], 40.00th=[ 264], 50.00th=[ 279], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 363], 95.00th=[ 401], | 99.00th=[ 489], 99.50th=[ 510], 99.90th=[ 531], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 55, max= 167, per=24.78%, avg=110.77, stdev=29.10, samples=91 iops : min= 13, max= 41, avg=27.24, stdev= 7.34, samples=91 lat (msec) : 50=0.04%, 100=0.16%, 250=33.55%, 500=65.51%, 750=0.74% cpu : usr=0.17%, sys=0.97%, ctx=3437, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=429KiB/s (440kB/s), 107KiB/s-109KiB/s (109kB/s-112kB/s), io=19.6MiB (20.5MB), run=46392-46726msec WRITE: bw=447KiB/s (458kB/s), 111KiB/s-113KiB/s (114kB/s-116kB/s), io=20.4MiB (21.4MB), run=46392-46726msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=178953: Wed Aug 5 10:54:09 2026 read: IOPS=1, BW=77.4MiB/s (81.1MB/s)(40.0MiB/517msec) slat (nsec): min=60318k, max=60318k, avg=60317801.00, stdev= 0.00 clat (nsec): min=425247k, max=425247k, avg=425246917.00, stdev= 0.00 lat (nsec): min=485569k, max=485569k, avg=485568563.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.94%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.4MiB/s (81.1MB/s), 77.4MiB/s-77.4MiB/s (81.1MB/s-81.1MB/s), io=40.0MiB (41.9MB), run=517-517msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout