== sanity test 398c: run fio to test AIO ================= 15:29:10 (1758742150)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.16468 s, 255 MB/s
osc.lustre-OST0000-osc-ffff95da204d4800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183219: Wed Sep 24 15:29:19 2025
  write: IOPS=438, BW=1752KiB/s (1794kB/s)(10.0MiB/5844msec); 0 zone resets
    slat (usec): min=28, max=1823, avg=105.23, stdev=69.80
    clat (usec): min=20936, max=84586, avg=36027.78, stdev=10092.27
     lat (usec): min=21028, max=84797, avg=36134.19, stdev=10108.08
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26346], 10.00th=[27395], 20.00th=[28705],
     | 30.00th=[30016], 40.00th=[31589], 50.00th=[33162], 60.00th=[34866],
     | 70.00th=[37487], 80.00th=[41157], 90.00th=[49546], 95.00th=[59507],
     | 99.00th=[72877], 99.50th=[78119], 99.90th=[83362], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1032, max= 2192, per=25.10%, avg=1759.00, stdev=378.68, samples=11
   iops        : min=  258, max=  548, avg=439.73, stdev=94.67, samples=11
  lat (msec)   : 50=90.59%, 100=9.41%
  cpu          : usr=0.51%, sys=4.47%, ctx=2348, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183220: Wed Sep 24 15:29:19 2025
  write: IOPS=441, BW=1768KiB/s (1810kB/s)(10.0MiB/5792msec); 0 zone resets
    slat (usec): min=21, max=1098, avg=106.04, stdev=61.65
    clat (usec): min=6322, max=84323, avg=36021.42, stdev=10325.76
     lat (usec): min=6388, max=84446, avg=36128.80, stdev=10341.85
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[25035], 10.00th=[27395], 20.00th=[28705],
     | 30.00th=[30278], 40.00th=[31589], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[36963], 80.00th=[41157], 90.00th=[49546], 95.00th=[60031],
     | 99.00th=[70779], 99.50th=[79168], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1056, max= 2184, per=25.31%, avg=1773.91, stdev=379.36, samples=11
   iops        : min=  264, max=  546, avg=443.45, stdev=94.83, samples=11
  lat (msec)   : 10=0.08%, 20=0.62%, 50=89.77%, 100=9.53%
  cpu          : usr=0.90%, sys=3.71%, ctx=2400, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183221: Wed Sep 24 15:29:19 2025
  write: IOPS=438, BW=1756KiB/s (1798kB/s)(10.0MiB/5833msec); 0 zone resets
    slat (usec): min=28, max=26038, avg=118.84, stdev=516.44
    clat (usec): min=7854, max=84530, avg=36014.78, stdev=10268.35
     lat (usec): min=7964, max=84874, avg=36134.73, stdev=10270.12
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[25560], 10.00th=[27395], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[31589], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[36963], 80.00th=[41157], 90.00th=[49546], 95.00th=[60031],
     | 99.00th=[70779], 99.50th=[80217], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1032, max= 2168, per=25.15%, avg=1762.18, stdev=376.10, samples=11
   iops        : min=  258, max=  542, avg=440.55, stdev=94.03, samples=11
  lat (msec)   : 10=0.08%, 20=0.43%, 50=89.80%, 100=9.69%
  cpu          : usr=0.53%, sys=4.66%, ctx=2406, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183222: Wed Sep 24 15:29:19 2025
  write: IOPS=438, BW=1752KiB/s (1794kB/s)(10.0MiB/5844msec); 0 zone resets
    slat (usec): min=20, max=730, avg=102.59, stdev=59.27
    clat (usec): min=20863, max=84117, avg=36012.04, stdev=10121.75
     lat (usec): min=20964, max=84178, avg=36115.86, stdev=10136.58
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[25822], 10.00th=[27132], 20.00th=[28705],
     | 30.00th=[30016], 40.00th=[31589], 50.00th=[33162], 60.00th=[34866],
     | 70.00th=[36963], 80.00th=[41157], 90.00th=[49021], 95.00th=[60031],
     | 99.00th=[68682], 99.50th=[80217], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1040, max= 2184, per=25.10%, avg=1758.73, stdev=377.58, samples=11
   iops        : min=  260, max=  546, avg=439.64, stdev=94.39, samples=11
  lat (msec)   : 50=90.39%, 100=9.61%
  cpu          : usr=0.72%, sys=3.68%, ctx=2361, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7009KiB/s (7177kB/s), 1752KiB/s-1768KiB/s (1794kB/s-1810kB/s), io=40.0MiB (41.9MB), run=5792-5844msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183232: Wed Sep 24 15:29:29 2025
  read: IOPS=213, BW=854KiB/s (874kB/s)(5048KiB/5912msec)
    slat (usec): min=1030, max=8578, avg=2732.12, stdev=913.79
    clat (usec): min=16085, max=58239, avg=35160.22, stdev=6022.52
     lat (usec): min=17694, max=61317, avg=37894.68, stdev=6163.39
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26346], 10.00th=[27919], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[33162], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[51119], 99.50th=[52691], 99.90th=[57934], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  753, max=  920, per=25.13%, avg=853.00, stdev=47.11, samples=11
   iops        : min=  188, max=  230, avg=213.18, stdev=11.82, samples=11
  write: IOPS=219, BW=878KiB/s (899kB/s)(5192KiB/5912msec); 0 zone resets
    slat (usec): min=33, max=3072, avg=99.57, stdev=99.53
    clat (usec): min=15033, max=54162, avg=35213.06, stdev=5341.68
     lat (usec): min=15108, max=54229, avg=35313.75, stdev=5342.16
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[27657], 10.00th=[28967], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39584], 90.00th=[42730], 95.00th=[45351],
     | 99.00th=[49021], 99.50th=[49546], 99.90th=[51119], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  720, max= 1104, per=24.88%, avg=878.91, stdev=111.57, samples=11
   iops        : min=  180, max=  276, avg=219.64, stdev=27.99, samples=11
  lat (msec)   : 20=0.20%, 50=98.71%, 100=1.09%
  cpu          : usr=0.54%, sys=5.06%, ctx=4376, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183233: Wed Sep 24 15:29:29 2025
  read: IOPS=211, BW=844KiB/s (864kB/s)(4976KiB/5895msec)
    slat (usec): min=1054, max=7989, avg=2734.63, stdev=911.38
    clat (usec): min=20513, max=65550, avg=35055.47, stdev=6042.37
     lat (usec): min=23340, max=67944, avg=37792.60, stdev=6233.45
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26608], 10.00th=[27657], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32900], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[51119], 99.50th=[53216], 99.90th=[62129], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  714, max=  952, per=24.94%, avg=846.55, stdev=80.41, samples=11
   iops        : min=  178, max=  238, avg=211.55, stdev=20.18, samples=11
  write: IOPS=223, BW=893KiB/s (914kB/s)(5264KiB/5895msec); 0 zone resets
    slat (usec): min=32, max=3268, avg=101.63, stdev=103.94
    clat (usec): min=22021, max=59022, avg=35386.33, stdev=5351.15
     lat (usec): min=22235, max=59118, avg=35488.97, stdev=5355.39
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[27919], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39584], 90.00th=[42730], 95.00th=[45351],
     | 99.00th=[51119], 99.50th=[51643], 99.90th=[53216], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  782, max= 1024, per=25.41%, avg=897.82, stdev=80.30, samples=11
   iops        : min=  195, max=  256, avg=224.36, stdev=20.19, samples=11
  lat (msec)   : 50=98.63%, 100=1.37%
  cpu          : usr=0.66%, sys=4.95%, ctx=4353, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183234: Wed Sep 24 15:29:29 2025
  read: IOPS=210, BW=844KiB/s (864kB/s)(4984KiB/5908msec)
    slat (usec): min=1107, max=8992, avg=2818.68, stdev=971.98
    clat (usec): min=6239, max=53552, avg=35312.50, stdev=5801.37
     lat (usec): min=8265, max=57827, avg=38133.66, stdev=6006.07
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26608], 10.00th=[28181], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[42730], 95.00th=[45351],
     | 99.00th=[49546], 99.50th=[51119], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  688, max= 1016, per=25.07%, avg=850.82, stdev=88.84, samples=11
   iops        : min=  172, max=  254, avg=212.64, stdev=22.20, samples=11
  write: IOPS=222, BW=890KiB/s (911kB/s)(5256KiB/5908msec); 0 zone resets
    slat (usec): min=33, max=2438, avg=98.79, stdev=84.07
    clat (usec): min=14649, max=55486, avg=35218.23, stdev=5190.65
     lat (usec): min=14745, max=55557, avg=35317.99, stdev=5191.66
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[27657], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39584], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[49546], 99.50th=[51643], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  750, max= 1016, per=25.10%, avg=886.91, stdev=81.54, samples=11
   iops        : min=  187, max=  254, avg=221.64, stdev=20.52, samples=11
  lat (msec)   : 10=0.04%, 20=0.04%, 50=98.98%, 100=0.94%
  cpu          : usr=0.76%, sys=4.93%, ctx=4337, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183235: Wed Sep 24 15:29:29 2025
  read: IOPS=215, BW=861KiB/s (882kB/s)(5060KiB/5875msec)
    slat (usec): min=45, max=6968, avg=2752.40, stdev=915.74
    clat (usec): min=4444, max=60250, avg=35324.80, stdev=6254.52
     lat (usec): min=5468, max=63089, avg=38080.06, stdev=6465.68
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26608], 10.00th=[28181], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33162], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40633], 90.00th=[43779], 95.00th=[46924],
     | 99.00th=[50070], 99.50th=[50594], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  800, max=  960, per=25.37%, avg=861.18, stdev=53.19, samples=11
   iops        : min=  200, max=  240, avg=215.18, stdev=13.38, samples=11
  write: IOPS=220, BW=882KiB/s (903kB/s)(5180KiB/5875msec); 0 zone resets
    slat (usec): min=31, max=1030, avg=98.42, stdev=60.53
    clat (usec): min=7769, max=61320, avg=35161.83, stdev=5457.38
     lat (usec): min=7911, max=61390, avg=35261.53, stdev=5457.04
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27657], 10.00th=[28967], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[42730], 95.00th=[45351],
     | 99.00th=[50070], 99.50th=[51119], 99.90th=[61080], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  768, max= 1000, per=25.03%, avg=884.45, stdev=59.44, samples=11
   iops        : min=  192, max=  250, avg=221.00, stdev=14.93, samples=11
  lat (msec)   : 10=0.23%, 20=0.16%, 50=98.59%, 100=1.02%
  cpu          : usr=0.54%, sys=5.14%, ctx=4316, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3394KiB/s (3476kB/s), 844KiB/s-861KiB/s (864kB/s-882kB/s), io=19.6MiB (20.5MB), run=5875-5912msec
  WRITE: bw=3534KiB/s (3619kB/s), 878KiB/s-893KiB/s (899kB/s-914kB/s), io=20.4MiB (21.4MB), run=5875-5912msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183240: Wed Sep 24 15:29:33 2025
  read: IOPS=9, BW=381MiB/s (399MB/s)(40.0MiB/105msec)
    slat (nsec): min=12020k, max=12020k, avg=12019960.00, stdev= 0.00
    clat (nsec): min=87114k, max=87114k, avg=87113581.00, stdev= 0.00
     lat (nsec): min=99139k, max=99139k, avg=99138937.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[87557],  5.00th=[87557], 10.00th=[87557], 20.00th=[87557],
     | 30.00th=[87557], 40.00th=[87557], 50.00th=[87557], 60.00th=[87557],
     | 70.00th=[87557], 80.00th=[87557], 90.00th=[87557], 95.00th=[87557],
     | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557],
     | 99.99th=[87557]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.62%, ctx=12, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=381MiB/s (399MB/s), 381MiB/s-381MiB/s (399MB/s-399MB/s), io=40.0MiB (41.9MB), run=105-105msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout