== sanity test 398c: run fio to test AIO ================= 13:10:40 (1789319440)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.129444 s, 324 MB/s
osc.lustre-OST0000-osc-ffff967e87820800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=190087: Sun Sep 13 13:10:48 2026
  write: IOPS=509, BW=2040KiB/s (2089kB/s)(10.0MiB/5020msec); 0 zone resets
    slat (usec): min=25, max=26539, avg=114.60, stdev=526.73
    clat (usec): min=4052, max=62036, avg=30945.91, stdev=3282.57
     lat (usec): min=4119, max=62121, avg=31061.64, stdev=3258.72
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[26346], 10.00th=[27657], 20.00th=[29230],
     | 30.00th=[29754], 40.00th=[30540], 50.00th=[31065], 60.00th=[31589],
     | 70.00th=[32113], 80.00th=[32637], 90.00th=[33817], 95.00th=[34866],
     | 99.00th=[39584], 99.50th=[44827], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[62129]
   bw (  KiB/s): min= 1928, max= 2168, per=25.24%, avg=2053.67, stdev=64.32, samples=9
   iops        : min=  482, max=  542, avg=513.11, stdev=16.03, samples=9
  lat (msec)   : 10=0.20%, 20=0.39%, 50=99.10%, 100=0.31%
  cpu          : usr=0.50%, sys=4.68%, ctx=2923, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190088: Sun Sep 13 13:10:48 2026
  write: IOPS=508, BW=2034KiB/s (2083kB/s)(10.0MiB/5034msec); 0 zone resets
    slat (usec): min=25, max=595, avg=105.91, stdev=69.16
    clat (usec): min=17786, max=49363, avg=30972.19, stdev=2578.79
     lat (usec): min=17839, max=49434, avg=31079.30, stdev=2579.50
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[26608], 10.00th=[27657], 20.00th=[29230],
     | 30.00th=[30016], 40.00th=[30540], 50.00th=[31065], 60.00th=[31589],
     | 70.00th=[32113], 80.00th=[32637], 90.00th=[33817], 95.00th=[34866],
     | 99.00th=[37487], 99.50th=[41157], 99.90th=[43254], 99.95th=[44827],
     | 99.99th=[49546]
   bw (  KiB/s): min= 1960, max= 2120, per=25.10%, avg=2042.11, stdev=51.28, samples=9
   iops        : min=  490, max=  530, avg=510.44, stdev=12.95, samples=9
  lat (msec)   : 20=0.08%, 50=99.92%
  cpu          : usr=0.62%, sys=4.11%, ctx=2848, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190089: Sun Sep 13 13:10:48 2026
  write: IOPS=512, BW=2048KiB/s (2098kB/s)(10.0MiB/4999msec); 0 zone resets
    slat (usec): min=24, max=556, avg=105.02, stdev=66.36
    clat (usec): min=6237, max=58359, avg=31084.09, stdev=3257.90
     lat (usec): min=6337, max=58406, avg=31190.32, stdev=3260.09
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26346], 10.00th=[27919], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30540], 50.00th=[31065], 60.00th=[31589],
     | 70.00th=[32113], 80.00th=[32637], 90.00th=[33817], 95.00th=[34866],
     | 99.00th=[44303], 99.50th=[50594], 99.90th=[52167], 99.95th=[53216],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1944, max= 2137, per=25.30%, avg=2058.78, stdev=60.16, samples=9
   iops        : min=  486, max=  534, avg=514.67, stdev=15.00, samples=9
  lat (msec)   : 10=0.04%, 20=0.20%, 50=99.22%, 100=0.55%
  cpu          : usr=0.50%, sys=4.22%, ctx=2888, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190090: Sun Sep 13 13:10:48 2026
  write: IOPS=508, BW=2034KiB/s (2083kB/s)(10.0MiB/5034msec); 0 zone resets
    slat (usec): min=29, max=550, avg=103.17, stdev=66.13
    clat (usec): min=17830, max=58739, avg=30986.17, stdev=2889.41
     lat (usec): min=17921, max=58798, avg=31090.57, stdev=2893.38
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26346], 10.00th=[27657], 20.00th=[29230],
     | 30.00th=[30016], 40.00th=[30540], 50.00th=[31065], 60.00th=[31589],
     | 70.00th=[32113], 80.00th=[32637], 90.00th=[33817], 95.00th=[34866],
     | 99.00th=[38011], 99.50th=[44303], 99.90th=[52691], 99.95th=[55837],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1928, max= 2128, per=25.14%, avg=2045.44, stdev=59.45, samples=9
   iops        : min=  482, max=  532, avg=511.33, stdev=14.90, samples=9
  lat (msec)   : 20=0.08%, 50=99.69%, 100=0.23%
  cpu          : usr=0.62%, sys=4.45%, ctx=2881, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8137KiB/s (8332kB/s), 2034KiB/s-2048KiB/s (2083kB/s-2098kB/s), io=40.0MiB (41.9MB), run=4999-5034msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190101: Sun Sep 13 13:10:57 2026
  read: IOPS=208, BW=835KiB/s (855kB/s)(5048KiB/6046msec)
    slat (usec): min=48, max=38075, avg=2601.48, stdev=1691.52
    clat (msec): min=3, max=131, avg=36.20, stdev= 9.22
     lat (msec): min=5, max=133, avg=38.80, stdev= 9.48
    clat percentiles (msec):
     |  1.00th=[   11],  5.00th=[   28], 10.00th=[   30], 20.00th=[   32],
     | 30.00th=[   34], 40.00th=[   35], 50.00th=[   36], 60.00th=[   37],
     | 70.00th=[   39], 80.00th=[   41], 90.00th=[   43], 95.00th=[   45],
     | 99.00th=[   59], 99.50th=[  113], 99.90th=[  126], 99.95th=[  132],
     | 99.99th=[  132]
   bw (  KiB/s): min=  629, max=  920, per=24.71%, avg=820.09, stdev=76.25, samples=11
   iops        : min=  157, max=  230, avg=205.00, stdev=19.13, samples=11
  write: IOPS=214, BW=859KiB/s (879kB/s)(5192KiB/6046msec); 0 zone resets
    slat (usec): min=36, max=569, avg=94.17, stdev=52.89
    clat (msec): min=3, max=131, avg=36.11, stdev= 7.08
     lat (msec): min=3, max=131, avg=36.21, stdev= 7.08
    clat percentiles (msec):
     |  1.00th=[   21],  5.00th=[   30], 10.00th=[   32], 20.00th=[   33],
     | 30.00th=[   34], 40.00th=[   35], 50.00th=[   36], 60.00th=[   37],
     | 70.00th=[   39], 80.00th=[   40], 90.00th=[   41], 95.00th=[   43],
     | 99.00th=[   50], 99.50th=[   52], 99.90th=[  132], 99.95th=[  132],
     | 99.99th=[  132]
   bw (  KiB/s): min=  648, max=  952, per=24.69%, avg=853.09, stdev=90.46, samples=11
   iops        : min=  162, max=  238, avg=213.27, stdev=22.61, samples=11
  lat (msec)   : 4=0.16%, 10=0.43%, 20=0.59%, 50=97.85%, 100=0.43%
  lat (msec)   : 250=0.55%
  cpu          : usr=0.46%, sys=4.33%, ctx=5035, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190102: Sun Sep 13 13:10:57 2026
  read: IOPS=208, BW=834KiB/s (854kB/s)(4976KiB/5969msec)
    slat (usec): min=35, max=7489, avg=2506.87, stdev=713.48
    clat (usec): min=4897, max=71602, avg=35748.55, stdev=5427.48
     lat (usec): min=5004, max=75128, avg=38257.25, stdev=5518.00
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[28443], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41681], 95.00th=[43254],
     | 99.00th=[47973], 99.50th=[54264], 99.90th=[68682], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  757, max=  888, per=24.97%, avg=828.82, stdev=46.86, samples=11
   iops        : min=  189, max=  222, avg=207.18, stdev=11.75, samples=11
  write: IOPS=220, BW=882KiB/s (903kB/s)(5264KiB/5969msec); 0 zone resets
    slat (usec): min=26, max=412, avg=93.36, stdev=49.91
    clat (usec): min=24322, max=60633, avg=36173.06, stdev=4031.47
     lat (usec): min=24381, max=60704, avg=36267.43, stdev=4029.85
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30278], 10.00th=[31589], 20.00th=[33162],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[41157], 95.00th=[42730],
     | 99.00th=[48497], 99.50th=[51643], 99.90th=[53216], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  752, max= 1008, per=25.72%, avg=888.55, stdev=65.14, samples=11
   iops        : min=  188, max=  252, avg=222.09, stdev=16.26, samples=11
  lat (msec)   : 10=0.31%, 50=98.83%, 100=0.86%
  cpu          : usr=0.42%, sys=4.46%, ctx=4820, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190103: Sun Sep 13 13:10:57 2026
  read: IOPS=207, BW=830KiB/s (850kB/s)(4984KiB/6006msec)
    slat (usec): min=1033, max=5595, avg=2523.61, stdev=722.98
    clat (usec): min=24141, max=59023, avg=35688.99, stdev=4436.53
     lat (usec): min=25929, max=63663, avg=38213.91, stdev=4522.91
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28443], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[41157], 95.00th=[42730],
     | 99.00th=[46924], 99.50th=[49021], 99.90th=[52167], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  731, max=  920, per=25.12%, avg=833.73, stdev=66.25, samples=11
   iops        : min=  182, max=  230, avg=208.36, stdev=16.68, samples=11
  write: IOPS=218, BW=875KiB/s (896kB/s)(5256KiB/6006msec); 0 zone resets
    slat (usec): min=28, max=485, avg=97.07, stdev=54.14
    clat (usec): min=24205, max=58002, avg=36063.32, stdev=4023.25
     lat (usec): min=24279, max=58069, avg=36161.41, stdev=4024.73
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[29754], 10.00th=[31327], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[55313], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  776, max= 1016, per=25.17%, avg=869.73, stdev=90.54, samples=11
   iops        : min=  194, max=  254, avg=217.36, stdev=22.71, samples=11
  lat (msec)   : 50=99.41%, 100=0.59%
  cpu          : usr=0.42%, sys=4.41%, ctx=5052, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190104: Sun Sep 13 13:10:57 2026
  read: IOPS=211, BW=846KiB/s (866kB/s)(5060KiB/5983msec)
    slat (usec): min=1101, max=10077, avg=2505.28, stdev=746.33
    clat (usec): min=21274, max=59892, avg=35768.66, stdev=4782.59
     lat (usec): min=22975, max=63833, avg=38275.30, stdev=4846.27
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28181], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41157], 95.00th=[42730],
     | 99.00th=[49021], 99.50th=[52167], 99.90th=[56886], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  760, max=  912, per=25.51%, avg=846.55, stdev=48.24, samples=11
   iops        : min=  190, max=  228, avg=211.64, stdev=12.06, samples=11
  write: IOPS=216, BW=866KiB/s (887kB/s)(5180KiB/5983msec); 0 zone resets
    slat (usec): min=36, max=433, avg=97.46, stdev=53.05
    clat (usec): min=24876, max=56782, avg=35949.98, stdev=3860.55
     lat (usec): min=24970, max=56874, avg=36048.70, stdev=3861.46
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[30278], 10.00th=[31327], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[48497], 99.50th=[50594], 99.90th=[55837], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  796, max=  944, per=25.17%, avg=869.45, stdev=43.02, samples=11
   iops        : min=  199, max=  236, avg=217.36, stdev=10.75, samples=11
  lat (msec)   : 50=99.22%, 100=0.78%
  cpu          : usr=0.30%, sys=4.60%, ctx=5003, majf=1, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3319KiB/s (3399kB/s), 830KiB/s-846KiB/s (850kB/s-866kB/s), io=19.6MiB (20.5MB), run=5969-6046msec
  WRITE: bw=3456KiB/s (3538kB/s), 859KiB/s-882KiB/s (879kB/s-903kB/s), io=20.4MiB (21.4MB), run=5969-6046msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190109: Sun Sep 13 13:11:01 2026
  read: IOPS=7, BW=317MiB/s (333MB/s)(40.0MiB/126msec)
    slat (nsec): min=8734.1k, max=8734.1k, avg=8734060.00, stdev= 0.00
    clat (nsec): min=112533k, max=112533k, avg=112532969.00, stdev= 0.00
     lat (nsec): min=121270k, max=121270k, avg=121269869.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  113],  5.00th=[  113], 10.00th=[  113], 20.00th=[  113],
     | 30.00th=[  113], 40.00th=[  113], 50.00th=[  113], 60.00th=[  113],
     | 70.00th=[  113], 80.00th=[  113], 90.00th=[  113], 95.00th=[  113],
     | 99.00th=[  113], 99.50th=[  113], 99.90th=[  113], 99.95th=[  113],
     | 99.99th=[  113]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.80%, sys=5.60%, ctx=7, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=317MiB/s (333MB/s), 317MiB/s-317MiB/s (333MB/s-333MB/s), io=40.0MiB (41.9MB), run=126-126msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout