== sanity test 398c: run fio to test AIO ================= 17:37:03 (1788817023) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.336901 s, 124 MB/s osc.lustre-OST0000-osc-ffff89074389a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145044: Mon Sep 7 17:38:41 2026 write: IOPS=27, BW=110KiB/s (113kB/s)(10.0MiB/92805msec); 0 zone resets slat (usec): min=21, max=13574, avg=218.99, stdev=626.37 clat (msec): min=101, max=1239, avg=579.11, stdev=138.58 lat (msec): min=101, max=1239, avg=579.33, stdev=138.60 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 363], 10.00th=[ 422], 20.00th=[ 477], | 30.00th=[ 514], 40.00th=[ 550], 50.00th=[ 584], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 684], 90.00th=[ 735], 95.00th=[ 793], | 99.00th=[ 936], 99.50th=[ 1133], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1234] bw ( KiB/s): min= 24, max= 255, per=24.90%, avg=109.79, stdev=31.15, samples=184 iops : min= 6, max= 63, avg=27.04, stdev= 7.80, samples=184 lat (msec) : 250=2.42%, 500=23.55%, 750=65.94%, 1000=7.38%, 2000=0.70% cpu : usr=0.08%, sys=0.38%, ctx=2675, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145045: Mon Sep 7 17:38:41 2026 write: IOPS=27, BW=110KiB/s (113kB/s)(10.0MiB/92677msec); 0 zone resets slat (usec): min=21, max=13679, avg=234.83, stdev=682.37 clat (msec): min=97, max=1239, avg=578.77, stdev=144.03 lat (msec): min=97, max=1239, avg=579.00, stdev=144.03 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 347], 10.00th=[ 422], 20.00th=[ 472], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 684], 90.00th=[ 743], 95.00th=[ 793], | 99.00th=[ 978], 99.50th=[ 1133], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1234] bw ( KiB/s): min= 32, max= 255, per=25.18%, avg=111.04, stdev=35.13, samples=182 iops : min= 8, max= 63, avg=27.35, stdev= 8.79, samples=182 lat (msec) : 100=0.62%, 250=1.29%, 500=24.80%, 750=63.71%, 1000=8.63% lat (msec) : 2000=0.94% cpu : usr=0.07%, sys=0.40%, ctx=2255, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145046: Mon Sep 7 17:38:41 2026 write: IOPS=27, BW=110KiB/s (113kB/s)(10.0MiB/92677msec); 0 zone resets slat (usec): min=21, max=20390, avg=210.94, stdev=698.37 clat (msec): min=77, max=1252, avg=578.37, stdev=142.79 lat (msec): min=77, max=1252, avg=578.58, stdev=142.80 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 355], 10.00th=[ 418], 20.00th=[ 477], | 30.00th=[ 514], 40.00th=[ 550], 50.00th=[ 584], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 676], 90.00th=[ 743], 95.00th=[ 785], | 99.00th=[ 995], 99.50th=[ 1116], 99.90th=[ 1200], 99.95th=[ 1234], | 99.99th=[ 1250] bw ( KiB/s): min= 32, max= 255, per=24.91%, avg=109.83, stdev=33.22, samples=184 iops : min= 8, max= 63, avg=27.05, stdev= 8.29, samples=184 lat (msec) : 100=0.62%, 250=1.88%, 500=23.32%, 750=65.27%, 1000=7.97% lat (msec) : 2000=0.94% cpu : usr=0.08%, sys=0.38%, ctx=2071, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145047: Mon Sep 7 17:38:41 2026 write: IOPS=27, BW=110KiB/s (113kB/s)(10.0MiB/92804msec); 0 zone resets slat (usec): min=21, max=10975, avg=213.68, stdev=557.70 clat (msec): min=38, max=1239, avg=579.08, stdev=140.50 lat (msec): min=38, max=1239, avg=579.30, stdev=140.48 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 372], 10.00th=[ 426], 20.00th=[ 481], | 30.00th=[ 514], 40.00th=[ 550], 50.00th=[ 584], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 676], 90.00th=[ 743], 95.00th=[ 802], | 99.00th=[ 995], 99.50th=[ 1045], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1234] bw ( KiB/s): min= 39, max= 263, per=24.90%, avg=109.81, stdev=33.63, samples=184 iops : min= 9, max= 65, avg=27.05, stdev= 8.41, samples=184 lat (msec) : 50=0.04%, 100=0.08%, 250=1.91%, 500=23.52%, 750=65.12% lat (msec) : 1000=8.52%, 2000=0.82% cpu : usr=0.08%, sys=0.38%, ctx=2229, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=441KiB/s (452kB/s), 110KiB/s-110KiB/s (113kB/s-113kB/s), io=40.0MiB (41.9MB), run=92677-92805msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145070: Mon Sep 7 17:39:38 2026 read: IOPS=24, BW=99.6KiB/s (102kB/s)(5048KiB/50685msec) slat (usec): min=1781, max=33004, avg=6799.36, stdev=3618.43 clat (msec): min=3, max=512, avg=197.32, stdev=62.90 lat (msec): min=8, max=524, avg=204.13, stdev=64.20 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 110], 10.00th=[ 124], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 176], 50.00th=[ 190], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 243], 90.00th=[ 279], 95.00th=[ 313], | 99.00th=[ 376], 99.50th=[ 414], 99.90th=[ 481], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 31, max= 239, per=25.15%, avg=99.08, stdev=38.54, samples=100 iops : min= 7, max= 59, avg=24.46, stdev= 9.64, samples=100 write: IOPS=25, BW=102KiB/s (105kB/s)(5192KiB/50685msec); 0 zone resets slat (usec): min=30, max=10643, avg=221.21, stdev=505.59 clat (msec): min=158, max=823, avg=424.60, stdev=124.27 lat (msec): min=158, max=824, avg=424.82, stdev=124.29 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 255], 10.00th=[ 279], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 372], 50.00th=[ 409], 60.00th=[ 443], | 70.00th=[ 485], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 743], 99.50th=[ 785], 99.90th=[ 818], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 39, max= 208, per=24.79%, avg=101.88, stdev=31.97, samples=100 iops : min= 9, max= 52, avg=25.16, stdev= 8.06, samples=100 lat (msec) : 4=0.04%, 10=0.08%, 50=0.12%, 100=1.17%, 250=41.17% lat (msec) : 500=43.52%, 750=13.48%, 1000=0.43% cpu : usr=0.16%, sys=0.99%, ctx=5134, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145071: Mon Sep 7 17:39:38 2026 read: IOPS=24, BW=98.2KiB/s (101kB/s)(4976KiB/50649msec) slat (usec): min=38, max=208390, avg=6942.24, stdev=6804.07 clat (msec): min=5, max=471, avg=196.41, stdev=62.58 lat (msec): min=11, max=484, avg=203.36, stdev=63.74 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 110], 10.00th=[ 125], 20.00th=[ 144], | 30.00th=[ 161], 40.00th=[ 178], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 224], 80.00th=[ 243], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 376], 99.50th=[ 409], 99.90th=[ 468], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 24, max= 207, per=24.72%, avg=97.40, stdev=36.19, samples=100 iops : min= 6, max= 51, avg=24.04, stdev= 9.10, samples=100 write: IOPS=25, BW=104KiB/s (106kB/s)(5264KiB/50649msec); 0 zone resets slat (usec): min=30, max=11101, avg=222.03, stdev=633.47 clat (msec): min=119, max=820, avg=422.98, stdev=124.45 lat (msec): min=119, max=821, avg=423.21, stdev=124.47 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 249], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 439], | 70.00th=[ 481], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 642], | 99.00th=[ 735], 99.50th=[ 760], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 40, max= 191, per=25.18%, avg=103.49, stdev=30.48, samples=100 iops : min= 10, max= 47, avg=25.56, stdev= 7.65, samples=100 lat (msec) : 10=0.04%, 20=0.12%, 50=0.23%, 100=1.17%, 250=41.56% lat (msec) : 500=43.20%, 750=13.32%, 1000=0.35% cpu : usr=0.15%, sys=1.00%, ctx=5043, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145072: Mon Sep 7 17:39:38 2026 read: IOPS=24, BW=98.1KiB/s (100kB/s)(4984KiB/50791msec) slat (usec): min=1454, max=39354, avg=6970.26, stdev=3946.64 clat (msec): min=3, max=482, avg=194.84, stdev=62.83 lat (msec): min=7, max=493, avg=201.83, stdev=64.33 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 106], 10.00th=[ 122], 20.00th=[ 142], | 30.00th=[ 159], 40.00th=[ 176], 50.00th=[ 190], 60.00th=[ 207], | 70.00th=[ 226], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 368], 99.50th=[ 380], 99.90th=[ 477], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 24, max= 192, per=24.86%, avg=97.96, stdev=38.94, samples=101 iops : min= 6, max= 48, avg=24.17, stdev= 9.73, samples=101 write: IOPS=25, BW=103KiB/s (106kB/s)(5256KiB/50791msec); 0 zone resets slat (usec): min=34, max=6876, avg=212.96, stdev=453.97 clat (msec): min=131, max=811, avg=425.62, stdev=123.24 lat (msec): min=132, max=811, avg=425.83, stdev=123.28 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 251], 10.00th=[ 284], 20.00th=[ 321], | 30.00th=[ 342], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 443], | 70.00th=[ 481], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 642], | 99.00th=[ 735], 99.50th=[ 776], 99.90th=[ 802], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 48, max= 192, per=24.94%, avg=102.49, stdev=30.01, samples=101 iops : min= 12, max= 48, avg=25.31, stdev= 7.55, samples=101 lat (msec) : 4=0.04%, 10=0.08%, 20=0.12%, 50=0.27%, 100=1.33% lat (msec) : 250=40.51%, 500=44.10%, 750=13.20%, 1000=0.35% cpu : usr=0.16%, sys=0.99%, ctx=5112, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145073: Mon Sep 7 17:39:38 2026 read: IOPS=24, BW=99.6KiB/s (102kB/s)(5060KiB/50824msec) slat (usec): min=1576, max=323622, avg=7276.70, stdev=12258.08 clat (msec): min=22, max=746, avg=198.85, stdev=72.60 lat (msec): min=24, max=751, avg=206.13, stdev=74.56 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 107], 10.00th=[ 121], 20.00th=[ 144], | 30.00th=[ 159], 40.00th=[ 174], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 224], 80.00th=[ 247], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 401], 99.50th=[ 489], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 31, max= 192, per=25.12%, avg=98.97, stdev=35.72, samples=101 iops : min= 7, max= 48, avg=24.43, stdev= 8.96, samples=101 write: IOPS=25, BW=102KiB/s (104kB/s)(5180KiB/50824msec); 0 zone resets slat (usec): min=22, max=6680, avg=208.36, stdev=469.52 clat (msec): min=88, max=1054, avg=425.37, stdev=130.46 lat (msec): min=88, max=1055, avg=425.59, stdev=130.51 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 255], 10.00th=[ 288], 20.00th=[ 313], | 30.00th=[ 342], 40.00th=[ 372], 50.00th=[ 409], 60.00th=[ 443], | 70.00th=[ 481], 80.00th=[ 535], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 760], 99.50th=[ 869], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 39, max= 192, per=24.57%, avg=100.97, stdev=32.86, samples=101 iops : min= 9, max= 48, avg=24.92, stdev= 8.24, samples=101 lat (msec) : 50=0.27%, 100=1.64%, 250=40.55%, 500=44.18%, 750=12.85% lat (msec) : 1000=0.39%, 2000=0.12% cpu : usr=0.14%, sys=0.98%, ctx=5103, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=395KiB/s (404kB/s), 98.1KiB/s-99.6KiB/s (100kB/s-102kB/s), io=19.6MiB (20.5MB), run=50649-50824msec WRITE: bw=411KiB/s (421kB/s), 102KiB/s-104KiB/s (104kB/s-106kB/s), io=20.4MiB (21.4MB), run=50649-50824msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145085: Mon Sep 7 17:39:42 2026 read: IOPS=5, BW=225MiB/s (236MB/s)(40.0MiB/178msec) slat (nsec): min=12776k, max=12776k, avg=12775676.00, stdev= 0.00 clat (nsec): min=156489k, max=156489k, avg=156489489.00, stdev= 0.00 lat (nsec): min=169269k, max=169269k, avg=169268981.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.21%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=40.0MiB (41.9MB), run=178-178msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout