== sanity test 398c: run fio to test AIO ================= 14:50:15 (1758826215)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.112035 s, 374 MB/s
osc.lustre-OST0000-osc-ffff9c2d2c377000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=192232: Thu Sep 25 14:50:23 2025
  write: IOPS=575, BW=2303KiB/s (2358kB/s)(10.0MiB/4446msec); 0 zone resets
    slat (usec): min=22, max=472, avg=85.91, stdev=42.55
    clat (usec): min=7333, max=48775, avg=27654.88, stdev=3246.37
     lat (usec): min=7454, max=48913, avg=27741.84, stdev=3250.95
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[22938], 10.00th=[24773], 20.00th=[25822],
     | 30.00th=[26346], 40.00th=[27132], 50.00th=[27657], 60.00th=[27919],
     | 70.00th=[28705], 80.00th=[29492], 90.00th=[30802], 95.00th=[33817],
     | 99.00th=[36439], 99.50th=[39060], 99.90th=[42730], 99.95th=[43254],
     | 99.99th=[49021]
   bw (  KiB/s): min= 2072, max= 2491, per=25.13%, avg=2293.38, stdev=133.33, samples=8
   iops        : min=  518, max=  622, avg=573.25, stdev=33.17, samples=8
  lat (msec)   : 10=0.08%, 20=1.80%, 50=98.12%
  cpu          : usr=0.36%, sys=4.18%, ctx=2565, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192233: Thu Sep 25 14:50:23 2025
  write: IOPS=570, BW=2282KiB/s (2336kB/s)(10.0MiB/4488msec); 0 zone resets
    slat (usec): min=20, max=385, avg=82.44, stdev=39.30
    clat (usec): min=16006, max=39939, avg=27659.50, stdev=2933.70
     lat (usec): min=16087, max=40095, avg=27742.95, stdev=2941.85
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[23200], 10.00th=[24773], 20.00th=[25822],
     | 30.00th=[26608], 40.00th=[27132], 50.00th=[27657], 60.00th=[27919],
     | 70.00th=[28705], 80.00th=[29230], 90.00th=[30540], 95.00th=[33817],
     | 99.00th=[35914], 99.50th=[38536], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2064, max= 2408, per=24.84%, avg=2266.62, stdev=103.91, samples=8
   iops        : min=  516, max=  602, avg=566.62, stdev=25.96, samples=8
  lat (msec)   : 20=0.82%, 50=99.18%
  cpu          : usr=0.22%, sys=4.15%, ctx=2542, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192234: Thu Sep 25 14:50:23 2025
  write: IOPS=570, BW=2283KiB/s (2337kB/s)(10.0MiB/4486msec); 0 zone resets
    slat (usec): min=20, max=316, avg=81.21, stdev=40.82
    clat (usec): min=15437, max=42550, avg=27677.50, stdev=3003.15
     lat (usec): min=15513, max=42623, avg=27759.73, stdev=3010.84
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[23200], 10.00th=[24773], 20.00th=[25822],
     | 30.00th=[26608], 40.00th=[27132], 50.00th=[27395], 60.00th=[27919],
     | 70.00th=[28705], 80.00th=[29492], 90.00th=[30802], 95.00th=[33162],
     | 99.00th=[36963], 99.50th=[38536], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[42730]
   bw (  KiB/s): min= 2072, max= 2384, per=24.85%, avg=2267.62, stdev=103.92, samples=8
   iops        : min=  518, max=  596, avg=566.88, stdev=25.94, samples=8
  lat (msec)   : 20=1.56%, 50=98.44%
  cpu          : usr=0.42%, sys=4.28%, ctx=2638, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192235: Thu Sep 25 14:50:23 2025
  write: IOPS=573, BW=2293KiB/s (2348kB/s)(10.0MiB/4466msec); 0 zone resets
    slat (usec): min=22, max=23221, avg=96.13, stdev=459.97
    clat (usec): min=3701, max=51953, avg=27604.72, stdev=3325.53
     lat (usec): min=3815, max=52048, avg=27701.88, stdev=3315.52
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[22938], 10.00th=[24773], 20.00th=[25822],
     | 30.00th=[26346], 40.00th=[27132], 50.00th=[27395], 60.00th=[27919],
     | 70.00th=[28443], 80.00th=[29230], 90.00th=[30802], 95.00th=[33817],
     | 99.00th=[36963], 99.50th=[39060], 99.90th=[46400], 99.95th=[49021],
     | 99.99th=[52167]
   bw (  KiB/s): min= 2064, max= 2453, per=25.00%, avg=2281.62, stdev=122.80, samples=8
   iops        : min=  516, max=  613, avg=570.38, stdev=30.65, samples=8
  lat (msec)   : 4=0.04%, 10=0.12%, 20=1.84%, 50=97.97%, 100=0.04%
  cpu          : usr=0.38%, sys=4.73%, ctx=2653, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9127KiB/s (9346kB/s), 2282KiB/s-2303KiB/s (2336kB/s-2358kB/s), io=40.0MiB (41.9MB), run=4446-4488msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=192246: Thu Sep 25 14:50:30 2025
  read: IOPS=257, BW=1029KiB/s (1054kB/s)(5048KiB/4904msec)
    slat (usec): min=959, max=8490, avg=2294.30, stdev=710.95
    clat (usec): min=4994, max=44153, avg=28948.88, stdev=4137.85
     lat (usec): min=7037, max=45702, avg=31244.56, stdev=4159.57
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[22676], 10.00th=[23725], 20.00th=[25560],
     | 30.00th=[26870], 40.00th=[27657], 50.00th=[28967], 60.00th=[30016],
     | 70.00th=[31065], 80.00th=[32113], 90.00th=[33817], 95.00th=[35914],
     | 99.00th=[40109], 99.50th=[42206], 99.90th=[42730], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min=  940, max= 1128, per=25.05%, avg=1024.44, stdev=65.19, samples=9
   iops        : min=  235, max=  282, avg=256.11, stdev=16.30, samples=9
  write: IOPS=264, BW=1059KiB/s (1084kB/s)(5192KiB/4904msec); 0 zone resets
    slat (usec): min=30, max=658, avg=86.41, stdev=51.55
    clat (usec): min=20908, max=45457, avg=29486.49, stdev=3316.49
     lat (usec): min=20972, max=45529, avg=29573.86, stdev=3316.25
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[24511], 10.00th=[25297], 20.00th=[26608],
     | 30.00th=[27657], 40.00th=[28443], 50.00th=[29230], 60.00th=[30278],
     | 70.00th=[31065], 80.00th=[32113], 90.00th=[33817], 95.00th=[35390],
     | 99.00th=[38011], 99.50th=[40109], 99.90th=[42730], 99.95th=[45351],
     | 99.99th=[45351]
   bw (  KiB/s): min=  901, max= 1200, per=24.92%, avg=1061.00, stdev=89.39, samples=9
   iops        : min=  225, max=  300, avg=265.22, stdev=22.40, samples=9
  lat (msec)   : 10=0.08%, 20=0.27%, 50=99.65%
  cpu          : usr=0.61%, sys=5.00%, ctx=4334, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192247: Thu Sep 25 14:50:30 2025
  read: IOPS=254, BW=1017KiB/s (1041kB/s)(4976KiB/4893msec)
    slat (usec): min=930, max=18352, avg=2327.16, stdev=817.64
    clat (usec): min=9742, max=47996, avg=29093.65, stdev=4055.30
     lat (usec): min=13036, max=50452, avg=31422.25, stdev=4050.89
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[22938], 10.00th=[23987], 20.00th=[25822],
     | 30.00th=[26870], 40.00th=[27919], 50.00th=[28967], 60.00th=[30016],
     | 70.00th=[30802], 80.00th=[32113], 90.00th=[33817], 95.00th=[35914],
     | 99.00th=[40109], 99.50th=[44303], 99.90th=[47973], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min=  876, max= 1088, per=24.66%, avg=1008.44, stdev=65.56, samples=9
   iops        : min=  219, max=  272, avg=252.11, stdev=16.39, samples=9
  write: IOPS=268, BW=1076KiB/s (1102kB/s)(5264KiB/4893msec); 0 zone resets
    slat (usec): min=23, max=1447, avg=88.41, stdev=63.36
    clat (usec): min=16026, max=45023, avg=29394.50, stdev=3318.89
     lat (usec): min=16090, max=45075, avg=29483.79, stdev=3320.47
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[24511], 10.00th=[25297], 20.00th=[26608],
     | 30.00th=[27657], 40.00th=[28443], 50.00th=[29230], 60.00th=[30278],
     | 70.00th=[31065], 80.00th=[32113], 90.00th=[33424], 95.00th=[34866],
     | 99.00th=[38011], 99.50th=[38536], 99.90th=[41157], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1040, max= 1200, per=25.43%, avg=1082.67, stdev=48.50, samples=9
   iops        : min=  260, max=  300, avg=270.67, stdev=12.12, samples=9
  lat (msec)   : 10=0.04%, 20=0.35%, 50=99.61%
  cpu          : usr=0.45%, sys=5.11%, ctx=4390, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192248: Thu Sep 25 14:50:30 2025
  read: IOPS=256, BW=1026KiB/s (1050kB/s)(4984KiB/4859msec)
    slat (usec): min=22, max=5981, avg=2286.62, stdev=666.21
    clat (usec): min=4427, max=50192, avg=28935.26, stdev=4517.20
     lat (usec): min=4459, max=53751, avg=31223.30, stdev=4612.46
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[22152], 10.00th=[23725], 20.00th=[25560],
     | 30.00th=[26870], 40.00th=[27919], 50.00th=[28967], 60.00th=[29754],
     | 70.00th=[31065], 80.00th=[32375], 90.00th=[34341], 95.00th=[35914],
     | 99.00th=[40633], 99.50th=[42206], 99.90th=[49021], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min=  888, max= 1144, per=25.16%, avg=1028.89, stdev=77.71, samples=9
   iops        : min=  222, max=  286, avg=257.22, stdev=19.43, samples=9
  write: IOPS=270, BW=1082KiB/s (1108kB/s)(5256KiB/4859msec); 0 zone resets
    slat (usec): min=23, max=1165, avg=89.51, stdev=60.73
    clat (usec): min=16774, max=50002, avg=29343.17, stdev=3454.37
     lat (usec): min=16852, max=50058, avg=29433.57, stdev=3451.91
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[24249], 10.00th=[25297], 20.00th=[26608],
     | 30.00th=[27657], 40.00th=[28443], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[30802], 80.00th=[31589], 90.00th=[33424], 95.00th=[34866],
     | 99.00th=[39060], 99.50th=[42206], 99.90th=[50070], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min=  985, max= 1192, per=25.35%, avg=1079.22, stdev=69.51, samples=9
   iops        : min=  246, max=  298, avg=269.78, stdev=17.42, samples=9
  lat (msec)   : 10=0.23%, 20=0.78%, 50=98.91%, 100=0.08%
  cpu          : usr=0.60%, sys=5.08%, ctx=4360, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192249: Thu Sep 25 14:50:30 2025
  read: IOPS=257, BW=1031KiB/s (1056kB/s)(5060KiB/4906msec)
    slat (usec): min=945, max=8137, avg=2304.57, stdev=705.34
    clat (usec): min=1537, max=43953, avg=29015.87, stdev=4343.00
     lat (usec): min=7477, max=45945, avg=31321.98, stdev=4338.89
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[22676], 10.00th=[23987], 20.00th=[25560],
     | 30.00th=[26870], 40.00th=[27919], 50.00th=[28967], 60.00th=[30016],
     | 70.00th=[31065], 80.00th=[32375], 90.00th=[34341], 95.00th=[35914],
     | 99.00th=[40633], 99.50th=[41681], 99.90th=[43254], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  914, max= 1144, per=25.06%, avg=1025.11, stdev=73.42, samples=9
   iops        : min=  228, max=  286, avg=256.22, stdev=18.45, samples=9
  write: IOPS=263, BW=1056KiB/s (1081kB/s)(5180KiB/4906msec); 0 zone resets
    slat (usec): min=34, max=2034, avg=86.86, stdev=72.36
    clat (usec): min=7483, max=44359, avg=29435.81, stdev=3489.09
     lat (usec): min=7539, max=44437, avg=29523.60, stdev=3489.19
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[24511], 10.00th=[25560], 20.00th=[26608],
     | 30.00th=[27657], 40.00th=[28443], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[31065], 80.00th=[32113], 90.00th=[33817], 95.00th=[35390],
     | 99.00th=[38536], 99.50th=[39584], 99.90th=[41157], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min=  895, max= 1176, per=24.71%, avg=1052.33, stdev=77.41, samples=9
   iops        : min=  223, max=  294, avg=263.00, stdev=19.54, samples=9
  lat (msec)   : 2=0.04%, 10=0.23%, 20=0.31%, 50=99.41%
  cpu          : usr=0.61%, sys=4.95%, ctx=4408, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4091KiB/s (4189kB/s), 1017KiB/s-1031KiB/s (1041kB/s-1056kB/s), io=19.6MiB (20.5MB), run=4859-4906msec
  WRITE: bw=4258KiB/s (4361kB/s), 1056KiB/s-1082KiB/s (1081kB/s-1108kB/s), io=20.4MiB (21.4MB), run=4859-4906msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=192253: Thu Sep 25 14:50:34 2025
  read: IOPS=9, BW=367MiB/s (385MB/s)(40.0MiB/109msec)
    slat (nsec): min=9197.3k, max=9197.3k, avg=9197330.00, stdev= 0.00
    clat (nsec): min=95270k, max=95270k, avg=95270371.00, stdev= 0.00
     lat (nsec): min=104471k, max=104471k, avg=104471256.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[94897],  5.00th=[94897], 10.00th=[94897], 20.00th=[94897],
     | 30.00th=[94897], 40.00th=[94897], 50.00th=[94897], 60.00th=[94897],
     | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897],
     | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897],
     | 99.99th=[94897]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.41%, ctx=3, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=367MiB/s (385MB/s), 367MiB/s-367MiB/s (385MB/s-385MB/s), io=40.0MiB (41.9MB), run=109-109msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout