== sanity test 398c: run fio to test AIO ================= 18:48:23 (1780526903)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.118766 s, 353 MB/s
osc.lustre-OST0000-osc-ffff9fd6e0f52000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=191986: Wed Jun  3 18:48:30 2026
  write: IOPS=560, BW=2242KiB/s (2295kB/s)(10.0MiB/4568msec); 0 zone resets
    slat (usec): min=49, max=724, avg=159.90, stdev=84.85
    clat (usec): min=18258, max=49812, avg=28088.53, stdev=2543.98
     lat (usec): min=18395, max=49952, avg=28249.35, stdev=2548.53
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[23987], 10.00th=[25035], 20.00th=[26608],
     | 30.00th=[27132], 40.00th=[27657], 50.00th=[27919], 60.00th=[28705],
     | 70.00th=[29230], 80.00th=[30016], 90.00th=[31065], 95.00th=[32113],
     | 99.00th=[34341], 99.50th=[34341], 99.90th=[47449], 99.95th=[49546],
     | 99.99th=[50070]
   bw (  KiB/s): min= 2072, max= 2344, per=24.97%, avg=2239.25, stdev=106.73, samples=8
   iops        : min=  518, max=  586, avg=559.75, stdev=26.63, samples=8
  lat (msec)   : 20=0.90%, 50=99.10%
  cpu          : usr=0.48%, sys=7.01%, ctx=2813, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=191987: Wed Jun  3 18:48:30 2026
  write: IOPS=562, BW=2249KiB/s (2303kB/s)(10.0MiB/4554msec); 0 zone resets
    slat (usec): min=43, max=24781, avg=164.42, stdev=494.37
    clat (usec): min=7365, max=58085, avg=28170.21, stdev=2751.60
     lat (usec): min=15268, max=58205, avg=28335.68, stdev=2725.90
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[23987], 10.00th=[25297], 20.00th=[26608],
     | 30.00th=[27132], 40.00th=[27657], 50.00th=[27919], 60.00th=[28705],
     | 70.00th=[29230], 80.00th=[30016], 90.00th=[31065], 95.00th=[32375],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[57934]
   bw (  KiB/s): min= 2056, max= 2356, per=25.10%, avg=2250.50, stdev=98.93, samples=8
   iops        : min=  514, max=  589, avg=562.62, stdev=24.73, samples=8
  lat (msec)   : 10=0.04%, 20=0.62%, 50=99.22%, 100=0.12%
  cpu          : usr=0.37%, sys=7.05%, ctx=3058, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=191988: Wed Jun  3 18:48:30 2026
  write: IOPS=564, BW=2259KiB/s (2314kB/s)(10.0MiB/4532msec); 0 zone resets
    slat (usec): min=49, max=671, avg=172.81, stdev=84.22
    clat (usec): min=6397, max=57835, avg=28099.99, stdev=2929.37
     lat (usec): min=6549, max=58173, avg=28273.79, stdev=2932.99
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[23725], 10.00th=[25035], 20.00th=[26346],
     | 30.00th=[27132], 40.00th=[27657], 50.00th=[27919], 60.00th=[28443],
     | 70.00th=[29230], 80.00th=[29754], 90.00th=[31065], 95.00th=[32375],
     | 99.00th=[35390], 99.50th=[36439], 99.90th=[57410], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min= 2064, max= 2445, per=25.22%, avg=2261.62, stdev=125.70, samples=8
   iops        : min=  516, max=  611, avg=565.38, stdev=31.37, samples=8
  lat (msec)   : 10=0.04%, 20=0.70%, 50=99.06%, 100=0.20%
  cpu          : usr=0.55%, sys=7.37%, ctx=3013, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=191989: Wed Jun  3 18:48:30 2026
  write: IOPS=560, BW=2243KiB/s (2297kB/s)(10.0MiB/4565msec); 0 zone resets
    slat (usec): min=44, max=23010, avg=184.76, stdev=459.97
    clat (usec): min=7354, max=50869, avg=28077.96, stdev=2679.06
     lat (usec): min=7892, max=51166, avg=28263.74, stdev=2678.37
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[23987], 10.00th=[25035], 20.00th=[26346],
     | 30.00th=[27132], 40.00th=[27395], 50.00th=[27919], 60.00th=[28705],
     | 70.00th=[29230], 80.00th=[29754], 90.00th=[31065], 95.00th=[32113],
     | 99.00th=[34341], 99.50th=[34866], 99.90th=[50594], 99.95th=[50594],
     | 99.99th=[51119]
   bw (  KiB/s): min= 2080, max= 2344, per=25.02%, avg=2243.00, stdev=95.71, samples=8
   iops        : min=  520, max=  586, avg=560.75, stdev=23.93, samples=8
  lat (msec)   : 10=0.04%, 20=0.51%, 50=99.34%, 100=0.12%
  cpu          : usr=0.57%, sys=7.76%, ctx=3001, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8967KiB/s (9182kB/s), 2242KiB/s-2259KiB/s (2295kB/s-2314kB/s), io=40.0MiB (41.9MB), run=4532-4568msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=191999: Wed Jun  3 18:48:38 2026
  read: IOPS=227, BW=909KiB/s (930kB/s)(5048KiB/5556msec)
    slat (usec): min=1017, max=9190, avg=2394.50, stdev=677.52
    clat (usec): min=7764, max=51896, avg=32818.50, stdev=4480.99
     lat (usec): min=10482, max=53425, avg=35214.28, stdev=4488.51
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[26346], 10.00th=[27657], 20.00th=[29492],
     | 30.00th=[30540], 40.00th=[31589], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[44827], 99.50th=[45876], 99.90th=[51119], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min=  815, max=  968, per=24.92%, avg=899.90, stdev=51.92, samples=10
   iops        : min=  203, max=  242, avg=224.90, stdev=13.12, samples=10
  write: IOPS=233, BW=934KiB/s (957kB/s)(5192KiB/5556msec); 0 zone resets
    slat (usec): min=45, max=1016, avg=157.61, stdev=82.23
    clat (usec): min=7749, max=46987, avg=33468.16, stdev=3653.26
     lat (usec): min=8019, max=47057, avg=33626.79, stdev=3652.98
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28181], 10.00th=[29230], 20.00th=[30540],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[43254], 99.50th=[45351], 99.90th=[46400], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  770, max= 1088, per=24.92%, avg=937.00, stdev=84.77, samples=10
   iops        : min=  192, max=  272, avg=234.20, stdev=21.30, samples=10
  lat (msec)   : 10=0.08%, 20=0.43%, 50=99.41%, 100=0.08%
  cpu          : usr=0.34%, sys=6.57%, ctx=4492, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192000: Wed Jun  3 18:48:38 2026
  read: IOPS=224, BW=897KiB/s (919kB/s)(4976KiB/5546msec)
    slat (usec): min=1000, max=8946, avg=2415.15, stdev=707.38
    clat (usec): min=10075, max=54612, avg=32701.65, stdev=4181.61
     lat (usec): min=12714, max=57816, avg=35118.04, stdev=4216.54
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26084], 10.00th=[27395], 20.00th=[29230],
     | 30.00th=[30540], 40.00th=[31589], 50.00th=[32637], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[42206], 99.50th=[43779], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min=  744, max=  992, per=24.66%, avg=890.40, stdev=66.08, samples=10
   iops        : min=  186, max=  248, avg=222.60, stdev=16.52, samples=10
  write: IOPS=237, BW=949KiB/s (972kB/s)(5264KiB/5546msec); 0 zone resets
    slat (usec): min=43, max=855, avg=156.60, stdev=82.82
    clat (usec): min=20019, max=50945, avg=33695.13, stdev=3449.06
     lat (usec): min=20127, max=51120, avg=33852.78, stdev=3451.79
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28443], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[48497], 99.95th=[51119],
     | 99.99th=[51119]
   bw (  KiB/s): min=  880, max= 1016, per=25.48%, avg=958.20, stdev=38.28, samples=10
   iops        : min=  220, max=  254, avg=239.50, stdev= 9.63, samples=10
  lat (msec)   : 20=0.08%, 50=99.80%, 100=0.12%
  cpu          : usr=0.38%, sys=6.53%, ctx=4612, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192001: Wed Jun  3 18:48:38 2026
  read: IOPS=224, BW=899KiB/s (921kB/s)(4984KiB/5541msec)
    slat (usec): min=1001, max=9723, avg=2422.97, stdev=710.54
    clat (usec): min=6585, max=49614, avg=32584.03, stdev=4237.40
     lat (usec): min=8673, max=53075, avg=35008.29, stdev=4282.79
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[25822], 10.00th=[27132], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31589], 50.00th=[32637], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[45876], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min=  821, max=  984, per=25.05%, avg=904.50, stdev=65.03, samples=10
   iops        : min=  205, max=  246, avg=226.10, stdev=16.29, samples=10
  write: IOPS=237, BW=949KiB/s (971kB/s)(5256KiB/5541msec); 0 zone resets
    slat (usec): min=48, max=866, avg=162.38, stdev=80.51
    clat (usec): min=21318, max=53806, avg=33600.62, stdev=3499.93
     lat (usec): min=21442, max=53942, avg=33763.95, stdev=3502.67
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28181], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[37487], 95.00th=[39584],
     | 99.00th=[43254], 99.50th=[44827], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  821, max= 1088, per=25.16%, avg=946.10, stdev=88.99, samples=10
   iops        : min=  205, max=  272, avg=236.50, stdev=22.29, samples=10
  lat (msec)   : 10=0.04%, 20=0.08%, 50=99.73%, 100=0.16%
  cpu          : usr=0.56%, sys=6.48%, ctx=4572, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192002: Wed Jun  3 18:48:38 2026
  read: IOPS=229, BW=916KiB/s (938kB/s)(5060KiB/5522msec)
    slat (usec): min=62, max=8682, avg=2375.93, stdev=738.16
    clat (usec): min=4609, max=45499, avg=32778.54, stdev=4292.31
     lat (usec): min=4761, max=47310, avg=35155.72, stdev=4351.86
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26346], 10.00th=[27657], 20.00th=[29230],
     | 30.00th=[30802], 40.00th=[31851], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[42206], 99.50th=[43779], 99.90th=[45351], 99.95th=[45351],
     | 99.99th=[45351]
   bw (  KiB/s): min=  856, max=  960, per=25.41%, avg=917.40, stdev=38.64, samples=10
   iops        : min=  214, max=  240, avg=229.30, stdev= 9.71, samples=10
  write: IOPS=234, BW=938KiB/s (961kB/s)(5180KiB/5522msec); 0 zone resets
    slat (usec): min=57, max=844, avg=159.81, stdev=83.39
    clat (usec): min=7891, max=55853, avg=33596.08, stdev=3717.90
     lat (usec): min=8033, max=56024, avg=33756.97, stdev=3718.48
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28181], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[43254], 99.50th=[45876], 99.90th=[55837], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min=  860, max= 1008, per=25.01%, avg=940.40, stdev=47.12, samples=10
   iops        : min=  215, max=  252, avg=235.10, stdev=11.78, samples=10
  lat (msec)   : 10=0.20%, 20=0.12%, 50=99.57%, 100=0.12%
  cpu          : usr=0.63%, sys=6.34%, ctx=4591, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3612KiB/s (3699kB/s), 897KiB/s-916KiB/s (919kB/s-938kB/s), io=19.6MiB (20.5MB), run=5522-5556msec
  WRITE: bw=3760KiB/s (3851kB/s), 934KiB/s-949KiB/s (957kB/s-972kB/s), io=20.4MiB (21.4MB), run=5522-5556msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=192007: Wed Jun  3 18:48:42 2026
  read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec)
    slat (nsec): min=7701.3k, max=7701.3k, avg=7701326.00, stdev= 0.00
    clat (nsec): min=77126k, max=77126k, avg=77125873.00, stdev= 0.00
     lat (nsec): min=84830k, max=84830k, avg=84829725.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[77071],  5.00th=[77071], 10.00th=[77071], 20.00th=[77071],
     | 30.00th=[77071], 40.00th=[77071], 50.00th=[77071], 60.00th=[77071],
     | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071],
     | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[77071]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.99%, ctx=7, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout