== sanity test 398c: run fio to test AIO ================= 13:50:58 (1747849858)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.115899 s, 362 MB/s
osc.lustre-OST0000-osc-ffff8800a3c4b000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=11838: Wed May 21 13:51:06 2025
  write: IOPS=393, BW=1574KiB/s (1612kB/s)(10.0MiB/6505msec)
    slat (usec): min=12, max=1072, avg=75.35, stdev=84.71
    clat (usec): min=3881, max=68794, avg=40444.82, stdev=5958.69
     lat (usec): min=4144, max=68840, avg=40521.85, stdev=5958.93
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[32637], 10.00th=[34341], 20.00th=[35914],
     | 30.00th=[36963], 40.00th=[38536], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[47973], 95.00th=[51119],
     | 99.00th=[57934], 99.50th=[59507], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1272, max= 1760, per=24.87%, avg=1565.92, stdev=139.12, samples=13
   iops        : min=  318, max=  440, avg=391.46, stdev=34.80, samples=13
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.04%, 50=92.66%, 100=7.19%
  cpu          : usr=1.25%, sys=3.17%, ctx=1411, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11839: Wed May 21 13:51:06 2025
  write: IOPS=394, BW=1578KiB/s (1616kB/s)(10.0MiB/6490msec)
    slat (usec): min=12, max=1191, avg=77.27, stdev=86.39
    clat (msec): min=5, max=100, avg=40.35, stdev= 6.63
     lat (msec): min=5, max=100, avg=40.43, stdev= 6.63
    clat percentiles (msec):
     |  1.00th=[   28],  5.00th=[   33], 10.00th=[   34], 20.00th=[   36],
     | 30.00th=[   37], 40.00th=[   39], 50.00th=[   40], 60.00th=[   42],
     | 70.00th=[   43], 80.00th=[   45], 90.00th=[   48], 95.00th=[   52],
     | 99.00th=[   57], 99.50th=[   61], 99.90th=[   92], 99.95th=[  101],
     | 99.99th=[  101]
   bw (  KiB/s): min= 1413, max= 1736, per=25.39%, avg=1598.42, stdev=112.13, samples=12
   iops        : min=  353, max=  434, avg=399.58, stdev=28.07, samples=12
  lat (msec)   : 10=0.16%, 20=0.16%, 50=92.50%, 100=7.11%, 250=0.08%
  cpu          : usr=0.60%, sys=3.99%, ctx=1346, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11840: Wed May 21 13:51:06 2025
  write: IOPS=393, BW=1575KiB/s (1612kB/s)(10.0MiB/6503msec)
    slat (usec): min=13, max=1042, avg=72.68, stdev=80.24
    clat (msec): min=7, max=100, avg=40.36, stdev= 6.38
     lat (msec): min=7, max=100, avg=40.43, stdev= 6.38
    clat percentiles (msec):
     |  1.00th=[   29],  5.00th=[   33], 10.00th=[   34], 20.00th=[   36],
     | 30.00th=[   37], 40.00th=[   39], 50.00th=[   40], 60.00th=[   42],
     | 70.00th=[   43], 80.00th=[   45], 90.00th=[   48], 95.00th=[   52],
     | 99.00th=[   58], 99.50th=[   62], 99.90th=[  101], 99.95th=[  101],
     | 99.99th=[  102]
   bw (  KiB/s): min= 1232, max= 1752, per=24.87%, avg=1565.85, stdev=147.57, samples=13
   iops        : min=  308, max=  438, avg=391.46, stdev=36.89, samples=13
  lat (msec)   : 10=0.04%, 50=92.23%, 100=7.62%, 250=0.12%
  cpu          : usr=0.00%, sys=4.31%, ctx=1263, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11841: Wed May 21 13:51:06 2025
  write: IOPS=393, BW=1575KiB/s (1612kB/s)(10.0MiB/6503msec)
    slat (usec): min=13, max=877, avg=74.91, stdev=81.13
    clat (usec): min=5651, max=85423, avg=40387.78, stdev=6066.80
     lat (usec): min=5727, max=85570, avg=40464.49, stdev=6068.24
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[32375], 10.00th=[33817], 20.00th=[35914],
     | 30.00th=[36963], 40.00th=[38536], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[47973], 95.00th=[51643],
     | 99.00th=[57410], 99.50th=[60031], 99.90th=[79168], 99.95th=[85459],
     | 99.99th=[85459]
   bw (  KiB/s): min= 1264, max= 1768, per=24.87%, avg=1565.85, stdev=152.29, samples=13
   iops        : min=  316, max=  442, avg=391.46, stdev=38.07, samples=13
  lat (msec)   : 10=0.12%, 50=92.85%, 100=7.03%
  cpu          : usr=1.23%, sys=3.11%, ctx=1344, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6297KiB/s (6448kB/s), 1574KiB/s-1578KiB/s (1612kB/s-1616kB/s), io=40.0MiB (41.9MB), run=6490-6505msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=11856: Wed May 21 13:51:13 2025
   read: IOPS=224, BW=898KiB/s (920kB/s)(5048KiB/5619msec)
    slat (usec): min=1053, max=6919, avg=2150.77, stdev=627.09
    clat (usec): min=2491, max=61165, avg=33057.99, stdev=5687.69
     lat (usec): min=4480, max=62453, avg=35211.88, stdev=5846.13
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[24249], 10.00th=[26084], 20.00th=[27919],
     | 30.00th=[29754], 40.00th=[31327], 50.00th=[32900], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[38011], 90.00th=[40109], 95.00th=[42206],
     | 99.00th=[47449], 99.50th=[48497], 99.90th=[52691], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  736, max= 1016, per=25.00%, avg=892.73, stdev=75.12, samples=11
   iops        : min=  184, max=  254, avg=223.09, stdev=18.82, samples=11
  write: IOPS=231, BW=924KiB/s (946kB/s)(5192KiB/5619msec)
    slat (usec): min=13, max=839, avg=70.15, stdev=71.14
    clat (usec): min=6534, max=55208, avg=34238.91, stdev=5195.90
     lat (usec): min=6582, max=55243, avg=34310.75, stdev=5197.26
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26608], 10.00th=[28181], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[40633], 95.00th=[44303],
     | 99.00th=[48497], 99.50th=[50070], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  768, max= 1045, per=24.83%, avg=923.18, stdev=83.16, samples=11
   iops        : min=  192, max=  261, avg=230.73, stdev=20.82, samples=11
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.08%, 50=99.34%, 100=0.51%
  cpu          : usr=1.10%, sys=6.71%, ctx=2262, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11857: Wed May 21 13:51:13 2025
   read: IOPS=221, BW=886KiB/s (907kB/s)(4976KiB/5618msec)
    slat (usec): min=1021, max=9370, avg=2156.89, stdev=659.29
    clat (usec): min=3825, max=56440, avg=33281.75, stdev=6061.47
     lat (usec): min=7613, max=57747, avg=35441.96, stdev=6191.20
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[24773], 10.00th=[26084], 20.00th=[28181],
     | 30.00th=[30016], 40.00th=[31327], 50.00th=[32900], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[41157], 95.00th=[43779],
     | 99.00th=[49021], 99.50th=[51643], 99.90th=[55313], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=  720, max=  960, per=24.68%, avg=881.27, stdev=76.83, samples=11
   iops        : min=  180, max=  240, avg=220.27, stdev=19.24, samples=11
  write: IOPS=234, BW=937KiB/s (959kB/s)(5264KiB/5618msec)
    slat (usec): min=12, max=849, avg=69.63, stdev=72.01
    clat (usec): min=16360, max=58892, avg=34247.32, stdev=4944.14
     lat (usec): min=16448, max=58929, avg=34318.53, stdev=4945.38
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27132], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32375], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[40633], 95.00th=[43254],
     | 99.00th=[47449], 99.50th=[49546], 99.90th=[56361], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  816, max= 1069, per=25.09%, avg=932.82, stdev=80.70, samples=11
   iops        : min=  204, max=  267, avg=233.18, stdev=20.13, samples=11
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.35%, 50=98.98%, 100=0.55%
  cpu          : usr=2.31%, sys=5.41%, ctx=2234, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11858: Wed May 21 13:51:13 2025
   read: IOPS=223, BW=892KiB/s (914kB/s)(4984KiB/5586msec)
    slat (usec): min=17, max=6296, avg=2121.44, stdev=644.33
    clat (usec): min=5352, max=68209, avg=33073.56, stdev=5860.61
     lat (usec): min=7419, max=70116, avg=35198.43, stdev=5982.09
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[24511], 10.00th=[26084], 20.00th=[28181],
     | 30.00th=[29754], 40.00th=[31589], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[35390], 80.00th=[37487], 90.00th=[40109], 95.00th=[42730],
     | 99.00th=[47449], 99.50th=[49546], 99.90th=[65274], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  760, max= 1016, per=25.07%, avg=895.09, stdev=78.20, samples=11
   iops        : min=  190, max=  254, avg=223.73, stdev=19.58, samples=11
  write: IOPS=235, BW=941KiB/s (964kB/s)(5256KiB/5586msec)
    slat (usec): min=16, max=838, avg=69.21, stdev=65.32
    clat (usec): min=16924, max=73513, avg=34426.43, stdev=5686.11
     lat (usec): min=16952, max=73549, avg=34497.13, stdev=5686.44
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26346], 10.00th=[28181], 20.00th=[30016],
     | 30.00th=[31589], 40.00th=[32637], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[36439], 80.00th=[38536], 90.00th=[41157], 95.00th=[43779],
     | 99.00th=[48497], 99.50th=[56361], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  808, max= 1061, per=25.28%, avg=940.09, stdev=74.42, samples=11
   iops        : min=  202, max=  265, avg=235.00, stdev=18.56, samples=11
  lat (msec)   : 10=0.04%, 20=0.20%, 50=99.06%, 100=0.70%
  cpu          : usr=0.00%, sys=7.86%, ctx=2313, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11859: Wed May 21 13:51:13 2025
   read: IOPS=225, BW=901KiB/s (922kB/s)(5060KiB/5618msec)
    slat (usec): min=1111, max=6232, avg=2158.52, stdev=648.39
    clat (usec): min=15361, max=53370, avg=33089.14, stdev=5692.22
     lat (usec): min=19114, max=56882, avg=35251.37, stdev=5816.48
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[24511], 10.00th=[25822], 20.00th=[27919],
     | 30.00th=[29754], 40.00th=[31327], 50.00th=[32900], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[47973], 99.50th=[50070], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  752, max= 1008, per=25.23%, avg=900.91, stdev=76.88, samples=11
   iops        : min=  188, max=  252, avg=225.18, stdev=19.25, samples=11
  write: IOPS=230, BW=922KiB/s (944kB/s)(5180KiB/5618msec)
    slat (usec): min=16, max=1794, avg=70.11, stdev=83.21
    clat (usec): min=20353, max=54428, avg=34443.80, stdev=5158.69
     lat (usec): min=20444, max=54478, avg=34515.26, stdev=5154.81
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26608], 10.00th=[28443], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[32900], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[36439], 80.00th=[38536], 90.00th=[41157], 95.00th=[44303],
     | 99.00th=[49021], 99.50th=[50594], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  816, max= 1008, per=24.72%, avg=919.09, stdev=66.67, samples=11
   iops        : min=  204, max=  252, avg=229.73, stdev=16.66, samples=11
  lat (msec)   : 20=0.23%, 50=99.30%, 100=0.47%
  cpu          : usr=0.00%, sys=7.80%, ctx=2309, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3571KiB/s (3657kB/s), 886KiB/s-901KiB/s (907kB/s-922kB/s), io=19.6MiB (20.5MB), run=5586-5619msec
  WRITE: bw=3718KiB/s (3807kB/s), 922KiB/s-941KiB/s (944kB/s-964kB/s), io=20.4MiB (21.4MB), run=5586-5619msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=11867: Wed May 21 13:51:14 2025
   read: IOPS=5, BW=235MiB/s (247MB/s)(40.0MiB/170msec)
    slat (nsec): min=10468k, max=10468k, avg=10468246.00, stdev= 0.00
    clat (nsec): min=150357k, max=150357k, avg=150357282.00, stdev= 0.00
     lat (nsec): min=160850k, max=160850k, avg=160850006.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  150],  5.00th=[  150], 10.00th=[  150], 20.00th=[  150],
     | 30.00th=[  150], 40.00th=[  150], 50.00th=[  150], 60.00th=[  150],
     | 70.00th=[  150], 80.00th=[  150], 90.00th=[  150], 95.00th=[  150],
     | 99.00th=[  150], 99.50th=[  150], 99.90th=[  150], 99.95th=[  150],
     | 99.99th=[  150]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.59%, sys=4.73%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=235MiB/s (247MB/s), 235MiB/s-235MiB/s (247MB/s-247MB/s), io=40.0MiB (41.9MB), run=170-170msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout