== sanity test 398c: run fio to test AIO ================= 17:51:51 (1751320311)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0993269 s, 422 MB/s
osc.lustre-OST0000-osc-ffff8800b5d3a000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30070: Mon Jun 30 17:51:57 2025
  write: IOPS=565, BW=2262KiB/s (2316kB/s)(10.0MiB/4527msec)
    slat (usec): min=11, max=28069, avg=71.42, stdev=554.95
    clat (usec): min=3879, max=48798, avg=28006.98, stdev=7767.88
     lat (usec): min=4437, max=48834, avg=28080.21, stdev=7765.37
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15401], 10.00th=[16450], 20.00th=[18744],
     | 30.00th=[22676], 40.00th=[26870], 50.00th=[30016], 60.00th=[31851],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36439], 95.00th=[38536],
     | 99.00th=[41157], 99.50th=[43254], 99.90th=[48497], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min= 1712, max= 3712, per=24.80%, avg=2243.56, stdev=626.93, samples=9
   iops        : min=  428, max=  928, avg=560.89, stdev=156.73, samples=9
  lat (msec)   : 4=0.04%, 10=0.27%, 20=23.32%, 50=76.37%
  cpu          : usr=0.62%, sys=4.82%, ctx=1117, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30071: Mon Jun 30 17:51:57 2025
  write: IOPS=567, BW=2271KiB/s (2325kB/s)(10.0MiB/4510msec)
    slat (usec): min=12, max=360, avg=62.91, stdev=40.25
    clat (usec): min=6584, max=44835, avg=28094.47, stdev=7650.46
     lat (usec): min=6609, max=44890, avg=28159.34, stdev=7663.17
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15401], 10.00th=[16450], 20.00th=[19006],
     | 30.00th=[23462], 40.00th=[27132], 50.00th=[29754], 60.00th=[31851],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36439], 95.00th=[38536],
     | 99.00th=[41157], 99.50th=[42730], 99.90th=[44827], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1736, max= 3696, per=25.01%, avg=2262.22, stdev=609.36, samples=9
   iops        : min=  434, max=  924, avg=565.56, stdev=152.34, samples=9
  lat (msec)   : 10=0.20%, 20=22.30%, 50=77.50%
  cpu          : usr=0.91%, sys=4.24%, ctx=1163, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30072: Mon Jun 30 17:51:57 2025
  write: IOPS=565, BW=2263KiB/s (2318kB/s)(10.0MiB/4524msec)
    slat (usec): min=11, max=27670, avg=76.81, stdev=547.32
    clat (usec): min=3268, max=44797, avg=27983.81, stdev=7704.89
     lat (usec): min=3320, max=44846, avg=28062.41, stdev=7713.41
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15401], 10.00th=[16450], 20.00th=[18744],
     | 30.00th=[22938], 40.00th=[26870], 50.00th=[29754], 60.00th=[32113],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36439], 95.00th=[38536],
     | 99.00th=[41681], 99.50th=[43779], 99.90th=[44827], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1720, max= 3712, per=24.89%, avg=2251.56, stdev=623.68, samples=9
   iops        : min=  430, max=  928, avg=562.89, stdev=155.92, samples=9
  lat (msec)   : 4=0.08%, 10=0.04%, 20=23.52%, 50=76.37%
  cpu          : usr=0.71%, sys=5.13%, ctx=1341, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30073: Mon Jun 30 17:51:57 2025
  write: IOPS=565, BW=2263KiB/s (2318kB/s)(10.0MiB/4524msec)
    slat (usec): min=11, max=26037, avg=75.23, stdev=514.92
    clat (usec): min=3703, max=48853, avg=27985.53, stdev=7768.25
     lat (usec): min=3843, max=48897, avg=28062.67, stdev=7769.85
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15270], 10.00th=[16450], 20.00th=[18744],
     | 30.00th=[23200], 40.00th=[27132], 50.00th=[29754], 60.00th=[31851],
     | 70.00th=[33424], 80.00th=[35390], 90.00th=[36439], 95.00th=[38536],
     | 99.00th=[41157], 99.50th=[42730], 99.90th=[49021], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min= 1760, max= 3720, per=24.93%, avg=2255.11, stdev=620.02, samples=9
   iops        : min=  440, max=  930, avg=563.78, stdev=155.00, samples=9
  lat (msec)   : 4=0.04%, 10=0.31%, 20=22.66%, 50=76.99%
  cpu          : usr=0.55%, sys=4.86%, ctx=1175, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9048KiB/s (9265kB/s), 2262KiB/s-2271KiB/s (2316kB/s-2325kB/s), io=40.0MiB (41.9MB), run=4510-4527msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30086: Mon Jun 30 17:52:00 2025
   read: IOPS=607, BW=2429KiB/s (2488kB/s)(5048KiB/2078msec)
    slat (usec): min=484, max=1636, avg=774.30, stdev=132.43
    clat (usec): min=8002, max=24955, avg=12341.45, stdev=1771.10
     lat (usec): min=8638, max=25638, avg=13117.35, stdev=1781.68
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12256], 60.00th=[12780],
     | 70.00th=[13173], 80.00th=[13698], 90.00th=[14353], 95.00th=[15270],
     | 99.00th=[16319], 99.50th=[17433], 99.90th=[23725], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 2344, max= 2592, per=25.29%, avg=2436.00, stdev=109.01, samples=4
   iops        : min=  586, max=  648, avg=609.00, stdev=27.25, samples=4
  write: IOPS=624, BW=2499KiB/s (2559kB/s)(5192KiB/2078msec)
    slat (usec): min=12, max=714, avg=39.62, stdev=34.05
    clat (usec): min=2746, max=23768, avg=12645.37, stdev=1410.95
     lat (usec): min=2786, max=23794, avg=12686.17, stdev=1409.13
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[10552], 10.00th=[10945], 20.00th=[11600],
     | 30.00th=[11863], 40.00th=[12256], 50.00th=[12649], 60.00th=[12911],
     | 70.00th=[13304], 80.00th=[13698], 90.00th=[14353], 95.00th=[14877],
     | 99.00th=[15926], 99.50th=[16319], 99.90th=[21627], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 2416, max= 2544, per=24.87%, avg=2494.00, stdev=55.57, samples=4
   iops        : min=  604, max=  636, avg=623.50, stdev=13.89, samples=4
  lat (msec)   : 4=0.04%, 10=5.35%, 20=94.34%, 50=0.27%
  cpu          : usr=0.96%, sys=9.39%, ctx=2277, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30087: Mon Jun 30 17:52:00 2025
   read: IOPS=597, BW=2389KiB/s (2446kB/s)(4976KiB/2083msec)
    slat (usec): min=500, max=1717, avg=776.59, stdev=132.10
    clat (usec): min=7069, max=17173, avg=12363.67, stdev=1664.18
     lat (usec): min=7655, max=18110, avg=13141.86, stdev=1673.38
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9765], 10.00th=[10290], 20.00th=[10814],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12387], 60.00th=[12780],
     | 70.00th=[13304], 80.00th=[13829], 90.00th=[14615], 95.00th=[15139],
     | 99.00th=[16319], 99.50th=[16712], 99.90th=[17171], 99.95th=[17171],
     | 99.99th=[17171]
   bw (  KiB/s): min= 2312, max= 2536, per=24.72%, avg=2382.00, stdev=103.56, samples=4
   iops        : min=  578, max=  634, avg=595.50, stdev=25.89, samples=4
  write: IOPS=631, BW=2527KiB/s (2588kB/s)(5264KiB/2083msec)
    slat (usec): min=12, max=289, avg=38.14, stdev=20.40
    clat (usec): min=7025, max=20659, avg=12730.46, stdev=1403.14
     lat (usec): min=7060, max=20706, avg=12769.81, stdev=1401.63
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10552], 10.00th=[11076], 20.00th=[11600],
     | 30.00th=[11994], 40.00th=[12387], 50.00th=[12649], 60.00th=[13042],
     | 70.00th=[13435], 80.00th=[13829], 90.00th=[14484], 95.00th=[15008],
     | 99.00th=[16188], 99.50th=[16450], 99.90th=[18220], 99.95th=[20579],
     | 99.99th=[20579]
   bw (  KiB/s): min= 2456, max= 2600, per=25.21%, avg=2528.00, stdev=59.15, samples=4
   iops        : min=  614, max=  650, avg=632.00, stdev=14.79, samples=4
  lat (msec)   : 10=4.49%, 20=95.47%, 50=0.04%
  cpu          : usr=1.54%, sys=8.50%, ctx=2265, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30088: Mon Jun 30 17:52:00 2025
   read: IOPS=598, BW=2395KiB/s (2452kB/s)(4984KiB/2081msec)
    slat (usec): min=438, max=1848, avg=780.64, stdev=137.90
    clat (usec): min=7236, max=18166, avg=12316.55, stdev=1645.13
     lat (usec): min=8079, max=18873, avg=13098.85, stdev=1656.09
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9765], 10.00th=[10290], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11731], 50.00th=[12256], 60.00th=[12780],
     | 70.00th=[13173], 80.00th=[13698], 90.00th=[14484], 95.00th=[15139],
     | 99.00th=[16188], 99.50th=[16581], 99.90th=[17433], 99.95th=[18220],
     | 99.99th=[18220]
   bw (  KiB/s): min= 2328, max= 2448, per=25.02%, avg=2410.00, stdev=55.57, samples=4
   iops        : min=  582, max=  612, avg=602.50, stdev=13.89, samples=4
  write: IOPS=631, BW=2526KiB/s (2586kB/s)(5256KiB/2081msec)
    slat (usec): min=12, max=424, avg=38.85, stdev=23.55
    clat (usec): min=8670, max=21800, avg=12723.87, stdev=1347.13
     lat (usec): min=8711, max=21831, avg=12763.92, stdev=1346.68
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10683], 10.00th=[11076], 20.00th=[11731],
     | 30.00th=[12125], 40.00th=[12387], 50.00th=[12649], 60.00th=[12911],
     | 70.00th=[13304], 80.00th=[13829], 90.00th=[14353], 95.00th=[14877],
     | 99.00th=[16057], 99.50th=[17171], 99.90th=[21103], 99.95th=[21890],
     | 99.99th=[21890]
   bw (  KiB/s): min= 2424, max= 2592, per=25.03%, avg=2510.00, stdev=94.74, samples=4
   iops        : min=  606, max=  648, avg=627.50, stdev=23.69, samples=4
  lat (msec)   : 10=4.34%, 20=95.55%, 50=0.12%
  cpu          : usr=1.59%, sys=8.70%, ctx=2307, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30089: Mon Jun 30 17:52:00 2025
   read: IOPS=609, BW=2437KiB/s (2496kB/s)(5060KiB/2076msec)
    slat (usec): min=31, max=2017, avg=775.01, stdev=149.33
    clat (usec): min=1295, max=23519, avg=12316.43, stdev=1862.64
     lat (usec): min=1367, max=24207, avg=13093.41, stdev=1884.04
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9634], 10.00th=[10028], 20.00th=[10683],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12256], 60.00th=[12780],
     | 70.00th=[13304], 80.00th=[13829], 90.00th=[14484], 95.00th=[15008],
     | 99.00th=[16909], 99.50th=[18482], 99.90th=[22676], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 2384, max= 2512, per=25.35%, avg=2442.00, stdev=57.46, samples=4
   iops        : min=  596, max=  628, avg=610.50, stdev=14.36, samples=4
  write: IOPS=623, BW=2495KiB/s (2555kB/s)(5180KiB/2076msec)
    slat (usec): min=12, max=714, avg=39.27, stdev=29.41
    clat (usec): min=3635, max=28884, avg=12738.03, stdev=1537.95
     lat (usec): min=3678, max=28952, avg=12778.52, stdev=1536.99
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[10683], 10.00th=[11076], 20.00th=[11600],
     | 30.00th=[11994], 40.00th=[12387], 50.00th=[12649], 60.00th=[13042],
     | 70.00th=[13435], 80.00th=[13698], 90.00th=[14353], 95.00th=[14877],
     | 99.00th=[16909], 99.50th=[17171], 99.90th=[26346], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 2416, max= 2584, per=24.89%, avg=2496.00, stdev=69.44, samples=4
   iops        : min=  604, max=  646, avg=624.00, stdev=17.36, samples=4
  lat (msec)   : 2=0.08%, 4=0.08%, 10=5.39%, 20=94.18%, 50=0.27%
  cpu          : usr=0.92%, sys=9.45%, ctx=2300, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9634KiB/s (9865kB/s), 2389KiB/s-2437KiB/s (2446kB/s-2496kB/s), io=19.6MiB (20.5MB), run=2076-2083msec
  WRITE: bw=9.79MiB/s (10.3MB/s), 2495KiB/s-2527KiB/s (2555kB/s-2588kB/s), io=20.4MiB (21.4MB), run=2076-2083msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30095: Mon Jun 30 17:52:02 2025
   read: IOPS=10, BW=426MiB/s (446MB/s)(40.0MiB/94msec)
    slat (nsec): min=6279.8k, max=6279.8k, avg=6279820.00, stdev= 0.00
    clat (nsec): min=83929k, max=83929k, avg=83928730.00, stdev= 0.00
     lat (nsec): min=90221k, max=90221k, avg=90220985.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[84411],  5.00th=[84411], 10.00th=[84411], 20.00th=[84411],
     | 30.00th=[84411], 40.00th=[84411], 50.00th=[84411], 60.00th=[84411],
     | 70.00th=[84411], 80.00th=[84411], 90.00th=[84411], 95.00th=[84411],
     | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[84411]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.45%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=40.0MiB (41.9MB), run=94-94msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck