== sanity test 398c: run fio to test AIO ================= 14:43:45 (1767037425)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.136568 s, 307 MB/s
osc.lustre-OST0000-osc-ffff9c365294f800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=186654: Mon Dec 29 14:43:54 2025
  write: IOPS=465, BW=1860KiB/s (1905kB/s)(10.0MiB/5505msec); 0 zone resets
    slat (usec): min=31, max=28059, avg=112.27, stdev=554.47
    clat (usec): min=7841, max=61982, avg=33996.59, stdev=5233.29
     lat (usec): min=8049, max=62035, avg=34110.07, stdev=5213.37
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[26608], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[33162], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[39584], 95.00th=[42730],
     | 99.00th=[48497], 99.50th=[51119], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[62129]
   bw (  KiB/s): min= 1576, max= 2072, per=24.85%, avg=1844.00, stdev=147.77, samples=10
   iops        : min=  394, max=  518, avg=461.00, stdev=36.94, samples=10
  lat (msec)   : 10=0.12%, 20=2.03%, 50=97.15%, 100=0.70%
  cpu          : usr=0.44%, sys=4.60%, ctx=2498, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186655: Mon Dec 29 14:43:54 2025
  write: IOPS=463, BW=1856KiB/s (1900kB/s)(10.0MiB/5518msec); 0 zone resets
    slat (usec): min=25, max=666, avg=102.03, stdev=46.90
    clat (usec): min=6402, max=52082, avg=34020.94, stdev=4873.83
     lat (usec): min=6478, max=52162, avg=34124.20, stdev=4879.07
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[26870], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33162], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[47973], 99.50th=[50070], 99.90th=[52167], 99.95th=[52167],
     | 99.99th=[52167]
   bw (  KiB/s): min= 1568, max= 2104, per=24.71%, avg=1834.20, stdev=149.08, samples=10
   iops        : min=  392, max=  526, avg=458.50, stdev=37.24, samples=10
  lat (msec)   : 10=0.08%, 20=1.21%, 50=98.16%, 100=0.55%
  cpu          : usr=0.63%, sys=3.95%, ctx=2574, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186656: Mon Dec 29 14:43:54 2025
  write: IOPS=468, BW=1873KiB/s (1918kB/s)(10.0MiB/5468msec); 0 zone resets
    slat (usec): min=24, max=742, avg=101.91, stdev=49.04
    clat (usec): min=7126, max=55053, avg=34019.34, stdev=5183.47
     lat (usec): min=7220, max=55111, avg=34122.45, stdev=5187.07
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[26608], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33162], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[38011], 90.00th=[39584], 95.00th=[42206],
     | 99.00th=[49546], 99.50th=[51119], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1592, max= 2102, per=24.98%, avg=1854.20, stdev=159.91, samples=10
   iops        : min=  398, max=  525, avg=463.50, stdev=39.89, samples=10
  lat (msec)   : 10=0.08%, 20=2.07%, 50=96.99%, 100=0.86%
  cpu          : usr=0.64%, sys=3.93%, ctx=2602, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186657: Mon Dec 29 14:43:54 2025
  write: IOPS=464, BW=1856KiB/s (1901kB/s)(10.0MiB/5516msec); 0 zone resets
    slat (usec): min=29, max=431, avg=102.32, stdev=51.09
    clat (usec): min=5179, max=59631, avg=34041.24, stdev=4980.20
     lat (usec): min=5329, max=59680, avg=34144.79, stdev=4983.47
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[26870], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[33162], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[39584], 95.00th=[42206],
     | 99.00th=[47449], 99.50th=[49021], 99.90th=[57410], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min= 1552, max= 2104, per=24.80%, avg=1840.40, stdev=153.31, samples=10
   iops        : min=  388, max=  526, avg=460.10, stdev=38.33, samples=10
  lat (msec)   : 10=0.08%, 20=0.94%, 50=98.55%, 100=0.43%
  cpu          : usr=0.54%, sys=4.37%, ctx=2639, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7423KiB/s (7601kB/s), 1856KiB/s-1873KiB/s (1900kB/s-1918kB/s), io=40.0MiB (41.9MB), run=5468-5518msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=186667: Mon Dec 29 14:44:02 2025
  read: IOPS=240, BW=963KiB/s (986kB/s)(5048KiB/5240msec)
    slat (usec): min=872, max=7533, avg=2377.32, stdev=763.34
    clat (usec): min=8855, max=54351, avg=30965.93, stdev=5599.67
     lat (usec): min=10713, max=60859, avg=33344.80, stdev=5799.69
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[23200], 10.00th=[24511], 20.00th=[26608],
     | 30.00th=[27919], 40.00th=[29230], 50.00th=[30278], 60.00th=[31589],
     | 70.00th=[33162], 80.00th=[34866], 90.00th=[38011], 95.00th=[41157],
     | 99.00th=[49021], 99.50th=[51643], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  682, max= 1120, per=25.22%, avg=965.60, stdev=129.83, samples=10
   iops        : min=  170, max=  280, avg=241.30, stdev=32.62, samples=10
  write: IOPS=247, BW=991KiB/s (1015kB/s)(5192KiB/5240msec); 0 zone resets
    slat (usec): min=29, max=419, avg=82.61, stdev=42.79
    clat (usec): min=14583, max=56566, avg=31218.29, stdev=4631.04
     lat (usec): min=14685, max=56649, avg=31301.86, stdev=4634.22
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[25035], 10.00th=[26084], 20.00th=[27657],
     | 30.00th=[28443], 40.00th=[29492], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[32900], 80.00th=[34341], 90.00th=[37487], 95.00th=[40633],
     | 99.00th=[44827], 99.50th=[46400], 99.90th=[56361], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=  720, max= 1144, per=25.02%, avg=997.40, stdev=120.75, samples=10
   iops        : min=  180, max=  286, avg=249.30, stdev=30.22, samples=10
  lat (msec)   : 10=0.04%, 20=0.31%, 50=99.14%, 100=0.51%
  cpu          : usr=0.52%, sys=4.52%, ctx=4411, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186668: Mon Dec 29 14:44:02 2025
  read: IOPS=237, BW=950KiB/s (973kB/s)(4976KiB/5238msec)
    slat (usec): min=906, max=8475, avg=2377.65, stdev=783.23
    clat (usec): min=13825, max=59651, avg=30975.09, stdev=5555.39
     lat (usec): min=17601, max=65700, avg=33354.41, stdev=5771.66
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[23462], 10.00th=[25035], 20.00th=[26608],
     | 30.00th=[27657], 40.00th=[28967], 50.00th=[30278], 60.00th=[31327],
     | 70.00th=[32900], 80.00th=[34866], 90.00th=[38011], 95.00th=[40633],
     | 99.00th=[49546], 99.50th=[51119], 99.90th=[56886], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  695, max= 1144, per=24.94%, avg=954.90, stdev=134.14, samples=10
   iops        : min=  173, max=  286, avg=238.60, stdev=33.73, samples=10
  write: IOPS=251, BW=1005KiB/s (1029kB/s)(5264KiB/5238msec); 0 zone resets
    slat (usec): min=31, max=713, avg=84.86, stdev=50.19
    clat (usec): min=18639, max=53346, avg=31661.77, stdev=4910.74
     lat (usec): min=18675, max=53422, avg=31747.65, stdev=4910.99
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[25297], 10.00th=[26346], 20.00th=[27657],
     | 30.00th=[28705], 40.00th=[29754], 50.00th=[30802], 60.00th=[32113],
     | 70.00th=[33162], 80.00th=[34866], 90.00th=[38011], 95.00th=[41681],
     | 99.00th=[48497], 99.50th=[49021], 99.90th=[53216], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  803, max= 1168, per=25.30%, avg=1008.90, stdev=101.34, samples=10
   iops        : min=  200, max=  292, avg=252.10, stdev=25.54, samples=10
  lat (msec)   : 20=0.20%, 50=99.18%, 100=0.62%
  cpu          : usr=0.32%, sys=4.72%, ctx=4420, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186669: Mon Dec 29 14:44:02 2025
  read: IOPS=238, BW=954KiB/s (977kB/s)(4984KiB/5225msec)
    slat (usec): min=20, max=9829, avg=2361.97, stdev=832.95
    clat (usec): min=7790, max=61587, avg=31423.39, stdev=5729.09
     lat (usec): min=10176, max=68640, avg=33786.84, stdev=5881.97
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[23987], 10.00th=[25035], 20.00th=[26608],
     | 30.00th=[28181], 40.00th=[29492], 50.00th=[30802], 60.00th=[32113],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[49546], 99.50th=[51643], 99.90th=[56361], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  771, max= 1072, per=25.07%, avg=960.10, stdev=91.26, samples=10
   iops        : min=  192, max=  268, avg=239.90, stdev=23.00, samples=10
  write: IOPS=251, BW=1006KiB/s (1030kB/s)(5256KiB/5225msec); 0 zone resets
    slat (usec): min=21, max=3902, avg=83.72, stdev=113.47
    clat (usec): min=13164, max=57517, avg=31356.43, stdev=4998.71
     lat (usec): min=13379, max=57633, avg=31441.04, stdev=5001.75
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[24773], 10.00th=[26084], 20.00th=[27395],
     | 30.00th=[28443], 40.00th=[29754], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[32637], 80.00th=[34866], 90.00th=[38011], 95.00th=[41157],
     | 99.00th=[46924], 99.50th=[48497], 99.90th=[56361], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  803, max= 1200, per=25.28%, avg=1008.10, stdev=131.13, samples=10
   iops        : min=  200, max=  300, avg=251.90, stdev=32.98, samples=10
  lat (msec)   : 10=0.08%, 20=0.27%, 50=99.06%, 100=0.59%
  cpu          : usr=0.71%, sys=4.36%, ctx=4249, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186670: Mon Dec 29 14:44:02 2025
  read: IOPS=241, BW=966KiB/s (989kB/s)(5060KiB/5238msec)
    slat (usec): min=898, max=9974, avg=2400.30, stdev=809.81
    clat (usec): min=10752, max=56185, avg=31098.41, stdev=5643.75
     lat (usec): min=13045, max=61013, avg=33500.41, stdev=5837.84
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[23462], 10.00th=[24511], 20.00th=[26608],
     | 30.00th=[28181], 40.00th=[29230], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[33162], 80.00th=[34866], 90.00th=[38011], 95.00th=[41681],
     | 99.00th=[50594], 99.50th=[53216], 99.90th=[55837], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=  714, max= 1096, per=25.36%, avg=971.20, stdev=126.94, samples=10
   iops        : min=  178, max=  274, avg=242.70, stdev=31.90, samples=10
  write: IOPS=247, BW=989KiB/s (1013kB/s)(5180KiB/5238msec); 0 zone resets
    slat (usec): min=33, max=476, avg=81.03, stdev=39.96
    clat (usec): min=18558, max=50855, avg=31254.35, stdev=4598.08
     lat (usec): min=18603, max=50920, avg=31336.38, stdev=4601.06
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[24773], 10.00th=[26084], 20.00th=[27657],
     | 30.00th=[28705], 40.00th=[29754], 50.00th=[30802], 60.00th=[31589],
     | 70.00th=[32637], 80.00th=[34341], 90.00th=[37487], 95.00th=[40109],
     | 99.00th=[46400], 99.50th=[47449], 99.90th=[49021], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min=  746, max= 1088, per=24.84%, avg=990.40, stdev=101.62, samples=10
   iops        : min=  186, max=  272, avg=247.50, stdev=25.57, samples=10
  lat (msec)   : 20=0.39%, 50=98.98%, 100=0.62%
  cpu          : usr=0.42%, sys=4.62%, ctx=4394, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3830KiB/s (3922kB/s), 950KiB/s-966KiB/s (973kB/s-989kB/s), io=19.6MiB (20.5MB), run=5225-5240msec
  WRITE: bw=3987KiB/s (4083kB/s), 989KiB/s-1006KiB/s (1013kB/s-1030kB/s), io=20.4MiB (21.4MB), run=5225-5240msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=186675: Mon Dec 29 14:44:05 2025
  read: IOPS=12, BW=500MiB/s (524MB/s)(40.0MiB/80msec)
    slat (nsec): min=7498.0k, max=7498.0k, avg=7498012.00, stdev= 0.00
    clat (nsec): min=68512k, max=68512k, avg=68511504.00, stdev= 0.00
     lat (nsec): min=76013k, max=76013k, avg=76012526.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[68682],  5.00th=[68682], 10.00th=[68682], 20.00th=[68682],
     | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[68682],
     | 70.00th=[68682], 80.00th=[68682], 90.00th=[68682], 95.00th=[68682],
     | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.86%, ctx=8, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=40.0MiB (41.9MB), run=80-80msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout