== sanity test 398c: run fio to test AIO ================= 23:28:54 (1770438534)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.172129 s, 244 MB/s
osc.lustre-OST0000-osc-ffff9531a08e8800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=196281: Fri Feb  6 23:29:05 2026
  write: IOPS=308, BW=1235KiB/s (1265kB/s)(10.0MiB/8291msec); 0 zone resets
    slat (usec): min=25, max=692, avg=154.39, stdev=92.09
    clat (usec): min=24207, max=91585, avg=51009.07, stdev=14289.60
     lat (usec): min=24472, max=91767, avg=51165.13, stdev=14326.97
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28181], 10.00th=[29754], 20.00th=[33162],
     | 30.00th=[38011], 40.00th=[53216], 50.00th=[57410], 60.00th=[59507],
     | 70.00th=[61080], 80.00th=[63177], 90.00th=[65274], 95.00th=[66847],
     | 99.00th=[72877], 99.50th=[87557], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
   bw (  KiB/s): min= 1008, max= 2064, per=24.97%, avg=1233.31, stdev=291.65, samples=16
   iops        : min=  252, max=  516, avg=308.31, stdev=72.90, samples=16
  lat (msec)   : 50=37.62%, 100=62.38%
  cpu          : usr=0.58%, sys=3.55%, ctx=2789, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=196282: Fri Feb  6 23:29:05 2026
  write: IOPS=309, BW=1239KiB/s (1269kB/s)(10.0MiB/8263msec); 0 zone resets
    slat (usec): min=27, max=842, avg=156.33, stdev=93.06
    clat (msec): min=24, max=105, avg=51.21, stdev=14.45
     lat (msec): min=24, max=106, avg=51.37, stdev=14.49
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   29], 10.00th=[   31], 20.00th=[   34],
     | 30.00th=[   39], 40.00th=[   53], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   62], 80.00th=[   64], 90.00th=[   66], 95.00th=[   67],
     | 99.00th=[   81], 99.50th=[   89], 99.90th=[  106], 99.95th=[  106],
     | 99.99th=[  107]
   bw (  KiB/s): min= 1000, max= 2088, per=25.07%, avg=1238.69, stdev=297.96, samples=16
   iops        : min=  250, max=  522, avg=309.63, stdev=74.44, samples=16
  lat (msec)   : 50=37.46%, 100=62.42%, 250=0.12%
  cpu          : usr=0.63%, sys=3.70%, ctx=2747, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=196283: Fri Feb  6 23:29:05 2026
  write: IOPS=309, BW=1236KiB/s (1266kB/s)(10.0MiB/8282msec); 0 zone resets
    slat (usec): min=37, max=930, avg=159.89, stdev=92.64
    clat (usec): min=12132, max=94164, avg=50995.49, stdev=14302.94
     lat (usec): min=12413, max=94296, avg=51157.26, stdev=14339.70
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28443], 10.00th=[29754], 20.00th=[33424],
     | 30.00th=[38536], 40.00th=[53216], 50.00th=[57410], 60.00th=[59507],
     | 70.00th=[61080], 80.00th=[63177], 90.00th=[65274], 95.00th=[66847],
     | 99.00th=[71828], 99.50th=[88605], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[93848]
   bw (  KiB/s): min=  968, max= 2072, per=25.01%, avg=1235.56, stdev=291.04, samples=16
   iops        : min=  242, max=  518, avg=308.88, stdev=72.75, samples=16
  lat (msec)   : 20=0.04%, 50=38.20%, 100=61.76%
  cpu          : usr=0.54%, sys=4.40%, ctx=2809, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=196284: Fri Feb  6 23:29:05 2026
  write: IOPS=310, BW=1242KiB/s (1272kB/s)(10.0MiB/8246msec); 0 zone resets
    slat (usec): min=25, max=741, avg=156.21, stdev=89.53
    clat (msec): min=23, max=121, avg=51.28, stdev=14.75
     lat (msec): min=23, max=121, avg=51.44, stdev=14.78
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   29], 10.00th=[   30], 20.00th=[   34],
     | 30.00th=[   39], 40.00th=[   53], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   62], 80.00th=[   64], 90.00th=[   66], 95.00th=[   67],
     | 99.00th=[   85], 99.50th=[   94], 99.90th=[  110], 99.95th=[  110],
     | 99.99th=[  123]
   bw (  KiB/s): min= 1008, max= 2104, per=25.14%, avg=1241.94, stdev=299.63, samples=16
   iops        : min=  252, max=  526, avg=310.44, stdev=74.86, samples=16
  lat (msec)   : 50=37.34%, 100=62.46%, 250=0.20%
  cpu          : usr=0.55%, sys=3.81%, ctx=2812, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=4940KiB/s (5059kB/s), 1235KiB/s-1242KiB/s (1265kB/s-1272kB/s), io=40.0MiB (41.9MB), run=8246-8291msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=196294: Fri Feb  6 23:29:14 2026
  read: IOPS=237, BW=949KiB/s (971kB/s)(5048KiB/5322msec)
    slat (usec): min=969, max=8046, avg=2468.69, stdev=704.59
    clat (usec): min=4269, max=47261, avg=31739.37, stdev=4196.26
     lat (usec): min=7465, max=49861, avg=34209.35, stdev=4205.93
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25560], 10.00th=[26608], 20.00th=[28443],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38536],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[46924], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min=  832, max= 1040, per=24.99%, avg=942.20, stdev=55.88, samples=10
   iops        : min=  208, max=  260, avg=235.50, stdev=14.03, samples=10
  write: IOPS=243, BW=976KiB/s (999kB/s)(5192KiB/5322msec); 0 zone resets
    slat (usec): min=26, max=2897, avg=87.28, stdev=97.39
    clat (usec): min=6706, max=45985, avg=31738.94, stdev=3567.25
     lat (usec): min=6866, max=46031, avg=31827.11, stdev=3565.82
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[26870], 10.00th=[27919], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30802], 50.00th=[31589], 60.00th=[32375],
     | 70.00th=[33162], 80.00th=[34341], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[41157], 99.50th=[41681], 99.90th=[44827], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min=  842, max= 1128, per=24.79%, avg=973.00, stdev=90.44, samples=10
   iops        : min=  210, max=  282, avg=243.20, stdev=22.69, samples=10
  lat (msec)   : 10=0.16%, 20=0.31%, 50=99.53%
  cpu          : usr=0.49%, sys=4.47%, ctx=4792, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=196295: Fri Feb  6 23:29:14 2026
  read: IOPS=235, BW=940KiB/s (963kB/s)(4976KiB/5291msec)
    slat (usec): min=30, max=8341, avg=2479.78, stdev=728.27
    clat (usec): min=21029, max=57537, avg=31766.89, stdev=3984.94
     lat (usec): min=22959, max=59997, avg=34247.98, stdev=3992.36
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25560], 10.00th=[26608], 20.00th=[28443],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31589], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[41157], 99.50th=[43254], 99.90th=[55837], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  822, max= 1072, per=24.93%, avg=939.80, stdev=69.83, samples=10
   iops        : min=  205, max=  268, avg=234.90, stdev=17.55, samples=10
  write: IOPS=248, BW=995KiB/s (1019kB/s)(5264KiB/5291msec); 0 zone resets
    slat (usec): min=29, max=445, avg=83.89, stdev=43.65
    clat (usec): min=17025, max=48370, avg=31605.63, stdev=3351.38
     lat (usec): min=17086, max=48443, avg=31690.44, stdev=3350.69
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[26608], 10.00th=[27657], 20.00th=[28967],
     | 30.00th=[29754], 40.00th=[30540], 50.00th=[31327], 60.00th=[32113],
     | 70.00th=[33162], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[40633], 99.50th=[42206], 99.90th=[44303], 99.95th=[48497],
     | 99.99th=[48497]
   bw (  KiB/s): min=  904, max= 1176, per=25.38%, avg=996.30, stdev=79.43, samples=10
   iops        : min=  226, max=  294, avg=249.00, stdev=19.83, samples=10
  lat (msec)   : 20=0.20%, 50=99.69%, 100=0.12%
  cpu          : usr=0.42%, sys=4.52%, ctx=4680, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=196296: Fri Feb  6 23:29:14 2026
  read: IOPS=235, BW=943KiB/s (965kB/s)(4984KiB/5286msec)
    slat (usec): min=23, max=6921, avg=2455.71, stdev=754.25
    clat (usec): min=12067, max=65684, avg=31756.97, stdev=4423.37
     lat (usec): min=12109, max=68923, avg=34213.91, stdev=4505.67
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[25297], 10.00th=[26608], 20.00th=[28443],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[41157], 99.50th=[42730], 99.90th=[62129], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  848, max= 1008, per=25.10%, avg=946.40, stdev=50.46, samples=10
   iops        : min=  212, max=  252, avg=236.60, stdev=12.62, samples=10
  write: IOPS=248, BW=994KiB/s (1018kB/s)(5256KiB/5286msec); 0 zone resets
    slat (usec): min=24, max=437, avg=83.02, stdev=43.82
    clat (usec): min=8410, max=61962, avg=31711.89, stdev=3857.48
     lat (usec): min=8551, max=62006, avg=31795.76, stdev=3857.64
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[26870], 10.00th=[27919], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[30802], 50.00th=[31589], 60.00th=[32375],
     | 70.00th=[33162], 80.00th=[34341], 90.00th=[35914], 95.00th=[36963],
     | 99.00th=[39060], 99.50th=[41681], 99.90th=[62129], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  912, max= 1112, per=25.32%, avg=993.70, stdev=68.08, samples=10
   iops        : min=  228, max=  278, avg=248.40, stdev=17.04, samples=10
  lat (msec)   : 10=0.12%, 20=0.82%, 50=98.79%, 100=0.27%
  cpu          : usr=0.32%, sys=4.64%, ctx=4683, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=196297: Fri Feb  6 23:29:14 2026
  read: IOPS=238, BW=952KiB/s (975kB/s)(5060KiB/5313msec)
    slat (usec): min=991, max=8897, avg=2472.00, stdev=708.20
    clat (usec): min=20665, max=43508, avg=31717.94, stdev=3779.39
     lat (usec): min=22891, max=46269, avg=34191.30, stdev=3787.92
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[25560], 10.00th=[26870], 20.00th=[28705],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[41157], 99.50th=[41681], 99.90th=[42730], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  880, max= 1048, per=25.24%, avg=951.70, stdev=50.55, samples=10
   iops        : min=  220, max=  262, avg=237.90, stdev=12.67, samples=10
  write: IOPS=243, BW=975KiB/s (998kB/s)(5180KiB/5313msec); 0 zone resets
    slat (usec): min=37, max=464, avg=84.27, stdev=43.30
    clat (usec): min=23792, max=47805, avg=31760.72, stdev=3174.59
     lat (usec): min=23841, max=47897, avg=31846.00, stdev=3174.73
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[27132], 10.00th=[27919], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30802], 50.00th=[31589], 60.00th=[32375],
     | 70.00th=[33162], 80.00th=[34341], 90.00th=[35914], 95.00th=[36963],
     | 99.00th=[40109], 99.50th=[41157], 99.90th=[42730], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min=  874, max= 1056, per=24.83%, avg=974.60, stdev=53.15, samples=10
   iops        : min=  218, max=  264, avg=243.60, stdev=13.39, samples=10
  lat (msec)   : 50=100.00%
  cpu          : usr=0.49%, sys=4.46%, ctx=4830, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3771KiB/s (3861kB/s), 940KiB/s-952KiB/s (963kB/s-975kB/s), io=19.6MiB (20.5MB), run=5286-5322msec
  WRITE: bw=3926KiB/s (4020kB/s), 975KiB/s-995KiB/s (998kB/s-1019kB/s), io=20.4MiB (21.4MB), run=5286-5322msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=196302: Fri Feb  6 23:29:17 2026
  read: IOPS=10, BW=435MiB/s (456MB/s)(40.0MiB/92msec)
    slat (nsec): min=7910.9k, max=7910.9k, avg=7910865.00, stdev= 0.00
    clat (nsec): min=79104k, max=79104k, avg=79104220.00, stdev= 0.00
     lat (nsec): min=87018k, max=87018k, avg=87018100.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[79168],  5.00th=[79168], 10.00th=[79168], 20.00th=[79168],
     | 30.00th=[79168], 40.00th=[79168], 50.00th=[79168], 60.00th=[79168],
     | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168],
     | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.79%, ctx=5, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=435MiB/s (456MB/s), 435MiB/s-435MiB/s (456MB/s-456MB/s), io=40.0MiB (41.9MB), run=92-92msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout