== sanity test 398c: run fio to test AIO ================= 01:19:53 (1763878793)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.121535 s, 345 MB/s
osc.lustre-OST0000-osc-ffff8c58478ad800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=194808: Sun Nov 23 01:20:03 2025
  write: IOPS=384, BW=1537KiB/s (1573kB/s)(10.0MiB/6664msec); 0 zone resets
    slat (usec): min=23, max=430, avg=97.16, stdev=48.55
    clat (usec): min=20081, max=74379, avg=41475.83, stdev=9694.41
     lat (usec): min=20171, max=74499, avg=41574.19, stdev=9695.42
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[28967], 10.00th=[30016], 20.00th=[31589],
     | 30.00th=[33817], 40.00th=[38011], 50.00th=[40633], 60.00th=[43254],
     | 70.00th=[46400], 80.00th=[49546], 90.00th=[55313], 95.00th=[60031],
     | 99.00th=[64750], 99.50th=[69731], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1208, max= 2120, per=25.19%, avg=1535.85, stdev=313.51, samples=13
   iops        : min=  302, max=  530, avg=383.92, stdev=78.38, samples=13
  lat (msec)   : 50=80.66%, 100=19.34%
  cpu          : usr=0.33%, sys=3.03%, ctx=2729, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194809: Sun Nov 23 01:20:03 2025
  write: IOPS=382, BW=1528KiB/s (1565kB/s)(10.0MiB/6700msec); 0 zone resets
    slat (usec): min=26, max=24006, avg=105.79, stdev=475.36
    clat (usec): min=7685, max=74566, avg=41404.42, stdev=9811.40
     lat (usec): min=7762, max=74684, avg=41511.41, stdev=9808.86
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[28967], 10.00th=[30016], 20.00th=[31589],
     | 30.00th=[33817], 40.00th=[37487], 50.00th=[40109], 60.00th=[42730],
     | 70.00th=[46400], 80.00th=[50070], 90.00th=[55837], 95.00th=[60031],
     | 99.00th=[64226], 99.50th=[68682], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1184, max= 2128, per=25.07%, avg=1528.23, stdev=314.64, samples=13
   iops        : min=  296, max=  532, avg=382.00, stdev=78.68, samples=13
  lat (msec)   : 10=0.08%, 20=0.08%, 50=80.20%, 100=19.65%
  cpu          : usr=0.34%, sys=3.30%, ctx=2795, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194810: Sun Nov 23 01:20:03 2025
  write: IOPS=381, BW=1525KiB/s (1562kB/s)(10.0MiB/6715msec); 0 zone resets
    slat (usec): min=25, max=536, avg=95.19, stdev=48.99
    clat (usec): min=22081, max=74722, avg=41352.40, stdev=9707.17
     lat (usec): min=22129, max=74849, avg=41448.78, stdev=9709.74
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[29230], 10.00th=[30016], 20.00th=[31589],
     | 30.00th=[33817], 40.00th=[37487], 50.00th=[40109], 60.00th=[42730],
     | 70.00th=[45876], 80.00th=[49546], 90.00th=[55313], 95.00th=[60031],
     | 99.00th=[64226], 99.50th=[68682], 99.90th=[73925], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1184, max= 2144, per=24.96%, avg=1522.08, stdev=328.15, samples=13
   iops        : min=  296, max=  536, avg=380.46, stdev=82.07, samples=13
  lat (msec)   : 50=80.59%, 100=19.41%
  cpu          : usr=0.36%, sys=2.90%, ctx=2667, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194811: Sun Nov 23 01:20:03 2025
  write: IOPS=381, BW=1524KiB/s (1561kB/s)(10.0MiB/6717msec); 0 zone resets
    slat (usec): min=26, max=416, avg=93.93, stdev=49.11
    clat (usec): min=25285, max=75063, avg=41450.17, stdev=9700.31
     lat (usec): min=25352, max=75105, avg=41545.20, stdev=9702.80
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[28967], 10.00th=[30016], 20.00th=[31589],
     | 30.00th=[33817], 40.00th=[38011], 50.00th=[40633], 60.00th=[42730],
     | 70.00th=[46400], 80.00th=[49546], 90.00th=[55313], 95.00th=[59507],
     | 99.00th=[64226], 99.50th=[68682], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1192, max= 2112, per=24.97%, avg=1522.54, stdev=310.83, samples=13
   iops        : min=  298, max=  528, avg=380.62, stdev=77.72, samples=13
  lat (msec)   : 50=80.16%, 100=19.84%
  cpu          : usr=0.39%, sys=3.10%, ctx=2777, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6098KiB/s (6244kB/s), 1524KiB/s-1537KiB/s (1561kB/s-1573kB/s), io=40.0MiB (41.9MB), run=6664-6717msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=194821: Sun Nov 23 01:20:11 2025
  read: IOPS=236, BW=946KiB/s (968kB/s)(5048KiB/5338msec)
    slat (usec): min=1039, max=9317, avg=2517.07, stdev=735.02
    clat (usec): min=4350, max=51063, avg=31746.14, stdev=4284.05
     lat (usec): min=6357, max=52732, avg=34264.45, stdev=4275.95
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[25035], 10.00th=[26346], 20.00th=[28181],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[35390], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[51119], 99.95th=[51119],
     | 99.99th=[51119]
   bw (  KiB/s): min=  832, max= 1024, per=25.04%, avg=941.10, stdev=60.95, samples=10
   iops        : min=  208, max=  256, avg=235.20, stdev=15.30, samples=10
  write: IOPS=243, BW=973KiB/s (996kB/s)(5192KiB/5338msec); 0 zone resets
    slat (usec): min=32, max=416, avg=89.45, stdev=44.03
    clat (usec): min=17428, max=42585, avg=31836.16, stdev=3188.66
     lat (usec): min=17481, max=42703, avg=31926.62, stdev=3188.60
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[26870], 10.00th=[27919], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[31065], 50.00th=[31851], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[40109], 99.50th=[40633], 99.90th=[42206], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=  861, max= 1104, per=24.81%, avg=970.90, stdev=70.58, samples=10
   iops        : min=  215, max=  276, avg=242.70, stdev=17.69, samples=10
  lat (msec)   : 10=0.08%, 20=0.16%, 50=99.69%, 100=0.08%
  cpu          : usr=0.37%, sys=4.91%, ctx=4668, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194822: Sun Nov 23 01:20:11 2025
  read: IOPS=233, BW=932KiB/s (954kB/s)(4976KiB/5339msec)
    slat (usec): min=1042, max=9207, avg=2536.21, stdev=744.26
    clat (usec): min=14285, max=47185, avg=31657.62, stdev=4210.87
     lat (usec): min=16242, max=49363, avg=34195.08, stdev=4220.95
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[24773], 10.00th=[26346], 20.00th=[28181],
     | 30.00th=[29492], 40.00th=[30540], 50.00th=[31589], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[42206], 99.50th=[43779], 99.90th=[45876], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  779, max= 1008, per=24.70%, avg=928.30, stdev=74.95, samples=10
   iops        : min=  194, max=  252, avg=232.00, stdev=18.90, samples=10
  write: IOPS=246, BW=986KiB/s (1010kB/s)(5264KiB/5339msec); 0 zone resets
    slat (usec): min=32, max=440, avg=87.16, stdev=42.03
    clat (usec): min=14208, max=45910, avg=31946.74, stdev=3335.15
     lat (usec): min=14277, max=46008, avg=32034.87, stdev=3332.18
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[26608], 10.00th=[27919], 20.00th=[29230],
     | 30.00th=[30016], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33424], 80.00th=[34341], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[40633], 99.50th=[41681], 99.90th=[43779], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min=  896, max= 1080, per=25.15%, avg=984.00, stdev=48.15, samples=10
   iops        : min=  224, max=  270, avg=246.00, stdev=12.04, samples=10
  lat (msec)   : 20=0.20%, 50=99.80%
  cpu          : usr=0.37%, sys=4.85%, ctx=4558, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194823: Sun Nov 23 01:20:11 2025
  read: IOPS=233, BW=936KiB/s (958kB/s)(4984KiB/5327msec)
    slat (usec): min=1045, max=7817, avg=2535.41, stdev=751.84
    clat (usec): min=21230, max=47715, avg=31733.48, stdev=4091.80
     lat (usec): min=23217, max=50836, avg=34270.28, stdev=4138.35
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25297], 10.00th=[26346], 20.00th=[28181],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[42730], 99.50th=[43779], 99.90th=[45351], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min=  856, max= 1056, per=24.94%, avg=937.10, stdev=63.44, samples=10
   iops        : min=  214, max=  264, avg=234.20, stdev=15.90, samples=10
  write: IOPS=246, BW=987KiB/s (1010kB/s)(5256KiB/5327msec); 0 zone resets
    slat (usec): min=27, max=410, avg=88.26, stdev=44.40
    clat (usec): min=22872, max=47436, avg=31853.26, stdev=3385.56
     lat (usec): min=22948, max=47501, avg=31942.60, stdev=3385.28
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[26870], 10.00th=[27657], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30802], 50.00th=[31851], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[34341], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[40633], 99.50th=[42206], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min=  869, max= 1152, per=25.06%, avg=980.50, stdev=91.16, samples=10
   iops        : min=  217, max=  288, avg=245.10, stdev=22.83, samples=10
  lat (msec)   : 50=100.00%
  cpu          : usr=0.51%, sys=4.75%, ctx=4474, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194824: Sun Nov 23 01:20:11 2025
  read: IOPS=238, BW=952KiB/s (975kB/s)(5060KiB/5314msec)
    slat (usec): min=57, max=9592, avg=2557.37, stdev=777.30
    clat (usec): min=4302, max=46217, avg=31667.23, stdev=4410.06
     lat (usec): min=4801, max=48500, avg=34225.86, stdev=4452.40
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[25035], 10.00th=[26346], 20.00th=[28443],
     | 30.00th=[29492], 40.00th=[30802], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[44303], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min=  864, max=  984, per=25.24%, avg=948.60, stdev=38.48, samples=10
   iops        : min=  216, max=  246, avg=237.10, stdev= 9.62, samples=10
  write: IOPS=243, BW=975KiB/s (998kB/s)(5180KiB/5314msec); 0 zone resets
    slat (usec): min=40, max=427, avg=90.13, stdev=43.87
    clat (usec): min=13987, max=57053, avg=31996.76, stdev=3450.92
     lat (usec): min=14162, max=57135, avg=32087.85, stdev=3450.31
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[26870], 10.00th=[27919], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[40633], 99.50th=[41157], 99.90th=[50594], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  912, max= 1040, per=24.85%, avg=972.30, stdev=44.26, samples=10
   iops        : min=  228, max=  260, avg=243.00, stdev=11.09, samples=10
  lat (msec)   : 10=0.23%, 20=0.20%, 50=99.45%, 100=0.12%
  cpu          : usr=0.30%, sys=5.01%, ctx=4583, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3759KiB/s (3849kB/s), 932KiB/s-952KiB/s (954kB/s-975kB/s), io=19.6MiB (20.5MB), run=5314-5339msec
  WRITE: bw=3913KiB/s (4007kB/s), 973KiB/s-987KiB/s (996kB/s-1010kB/s), io=20.4MiB (21.4MB), run=5314-5339msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=194829: Sun Nov 23 01:20:15 2025
  read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec)
    slat (nsec): min=13139k, max=13139k, avg=13139025.00, stdev= 0.00
    clat (nsec): min=71607k, max=71607k, avg=71607036.00, stdev= 0.00
     lat (nsec): min=84751k, max=84751k, avg=84750520.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[71828],  5.00th=[71828], 10.00th=[71828], 20.00th=[71828],
     | 30.00th=[71828], 40.00th=[71828], 50.00th=[71828], 60.00th=[71828],
     | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828],
     | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[71828]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=13.48%, ctx=7, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout