== sanity test 398c: run fio to test AIO ================= 07:52:54 (1781610774)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.126624 s, 331 MB/s
osc.lustre-OST0000-osc-ffff88850971a800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=168376: Tue Jun 16 07:53:02 2026
  write: IOPS=535, BW=2142KiB/s (2194kB/s)(10.0MiB/4780msec); 0 zone resets
    slat (usec): min=42, max=851, avg=177.35, stdev=94.03
    clat (usec): min=9682, max=79238, avg=29639.51, stdev=4554.63
     lat (usec): min=9872, max=79466, avg=29817.90, stdev=4565.62
    clat percentiles (usec):
     |  1.00th=[15008],  5.00th=[24249], 10.00th=[26608], 20.00th=[27657],
     | 30.00th=[28443], 40.00th=[28967], 50.00th=[29492], 60.00th=[30016],
     | 70.00th=[30802], 80.00th=[31851], 90.00th=[33817], 95.00th=[34866],
     | 99.00th=[39060], 99.50th=[47973], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1936, max= 2443, per=25.66%, avg=2155.00, stdev=135.73, samples=9
   iops        : min=  484, max=  610, avg=538.67, stdev=33.73, samples=9
  lat (msec)   : 10=0.04%, 20=3.05%, 50=96.48%, 100=0.43%
  cpu          : usr=0.38%, sys=7.32%, ctx=2637, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=168377: Tue Jun 16 07:53:02 2026
  write: IOPS=527, BW=2111KiB/s (2162kB/s)(10.0MiB/4850msec); 0 zone resets
    slat (usec): min=44, max=25003, avg=203.51, stdev=500.52
    clat (msec): min=9, max=105, avg=29.90, stdev= 5.87
     lat (msec): min=9, max=105, avg=30.11, stdev= 5.87
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   25], 10.00th=[   27], 20.00th=[   28],
     | 30.00th=[   29], 40.00th=[   29], 50.00th=[   30], 60.00th=[   31],
     | 70.00th=[   31], 80.00th=[   32], 90.00th=[   34], 95.00th=[   35],
     | 99.00th=[   45], 99.50th=[   77], 99.90th=[  101], 99.95th=[  105],
     | 99.99th=[  106]
   bw (  KiB/s): min= 1976, max= 2315, per=25.52%, avg=2143.44, stdev=100.11, samples=9
   iops        : min=  494, max=  578, avg=535.78, stdev=24.87, samples=9
  lat (msec)   : 10=0.04%, 20=2.19%, 50=97.15%, 100=0.47%, 250=0.16%
  cpu          : usr=0.47%, sys=8.23%, ctx=2755, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=168378: Tue Jun 16 07:53:02 2026
  write: IOPS=525, BW=2103KiB/s (2153kB/s)(10.0MiB/4870msec); 0 zone resets
    slat (usec): min=51, max=748, avg=189.50, stdev=101.32
    clat (msec): min=11, max=100, avg=29.94, stdev= 5.77
     lat (msec): min=11, max=100, avg=30.13, stdev= 5.78
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   25], 10.00th=[   27], 20.00th=[   28],
     | 30.00th=[   29], 40.00th=[   29], 50.00th=[   30], 60.00th=[   31],
     | 70.00th=[   31], 80.00th=[   32], 90.00th=[   34], 95.00th=[   35],
     | 99.00th=[   47], 99.50th=[   62], 99.90th=[  101], 99.95th=[  101],
     | 99.99th=[  101]
   bw (  KiB/s): min= 2000, max= 2271, per=25.45%, avg=2137.67, stdev=92.92, samples=9
   iops        : min=  500, max=  567, avg=534.33, stdev=23.10, samples=9
  lat (msec)   : 20=2.23%, 50=96.95%, 100=0.70%, 250=0.12%
  cpu          : usr=0.41%, sys=8.03%, ctx=2829, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=168379: Tue Jun 16 07:53:02 2026
  write: IOPS=524, BW=2100KiB/s (2150kB/s)(10.0MiB/4877msec); 0 zone resets
    slat (usec): min=50, max=896, avg=186.39, stdev=100.31
    clat (usec): min=9503, max=90694, avg=29961.92, stdev=5029.94
     lat (usec): min=9721, max=90873, avg=30149.55, stdev=5042.55
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[24511], 10.00th=[26346], 20.00th=[27657],
     | 30.00th=[28443], 40.00th=[28967], 50.00th=[29492], 60.00th=[30278],
     | 70.00th=[30802], 80.00th=[31851], 90.00th=[33817], 95.00th=[35390],
     | 99.00th=[49546], 99.50th=[55837], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min= 1992, max= 2240, per=25.41%, avg=2134.22, stdev=87.25, samples=9
   iops        : min=  498, max=  560, avg=533.56, stdev=21.81, samples=9
  lat (msec)   : 10=0.04%, 20=1.84%, 50=97.19%, 100=0.94%
  cpu          : usr=0.59%, sys=7.24%, ctx=2693, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8399KiB/s (8600kB/s), 2100KiB/s-2142KiB/s (2150kB/s-2194kB/s), io=40.0MiB (41.9MB), run=4780-4877msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=168389: Tue Jun 16 07:53:11 2026
  read: IOPS=223, BW=895KiB/s (917kB/s)(5048KiB/5638msec)
    slat (usec): min=1024, max=6059, avg=2423.67, stdev=676.58
    clat (usec): min=18771, max=46787, avg=33427.84, stdev=4213.25
     lat (usec): min=21457, max=49009, avg=35852.84, stdev=4219.67
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26346], 10.00th=[27657], 20.00th=[29754],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[42730], 99.50th=[44303], 99.90th=[45876], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  776, max=  944, per=25.04%, avg=890.91, stdev=47.50, samples=11
   iops        : min=  194, max=  236, avg=222.73, stdev=11.88, samples=11
  write: IOPS=230, BW=921KiB/s (943kB/s)(5192KiB/5638msec); 0 zone resets
    slat (usec): min=51, max=1904, avg=168.78, stdev=94.54
    clat (usec): min=18644, max=43971, avg=33979.72, stdev=3173.42
     lat (usec): min=18762, max=44141, avg=34149.57, stdev=3172.54
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28967], 10.00th=[30016], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[41681], 99.50th=[42206], 99.90th=[43779], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  838, max= 1096, per=24.85%, avg=920.55, stdev=69.28, samples=11
   iops        : min=  209, max=  274, avg=230.09, stdev=17.38, samples=11
  lat (msec)   : 20=0.12%, 50=99.88%
  cpu          : usr=0.35%, sys=6.76%, ctx=4757, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=168390: Tue Jun 16 07:53:11 2026
  read: IOPS=220, BW=882KiB/s (904kB/s)(4976KiB/5639msec)
    slat (usec): min=61, max=8204, avg=2432.04, stdev=671.78
    clat (usec): min=6013, max=46432, avg=33647.74, stdev=4450.27
     lat (usec): min=8819, max=48400, avg=36081.20, stdev=4448.16
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26084], 10.00th=[27919], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[32637], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[39060], 95.00th=[40109],
     | 99.00th=[43254], 99.50th=[43779], 99.90th=[45351], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min=  808, max=  968, per=24.72%, avg=879.64, stdev=51.49, samples=11
   iops        : min=  202, max=  242, avg=219.91, stdev=12.87, samples=11
  write: IOPS=233, BW=933KiB/s (956kB/s)(5264KiB/5639msec); 0 zone resets
    slat (usec): min=52, max=694, avg=165.27, stdev=85.31
    clat (usec): min=15267, max=47247, avg=34027.82, stdev=3547.68
     lat (usec): min=15368, max=47338, avg=34194.16, stdev=3548.38
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28443], 10.00th=[29754], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[46400], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min=  824, max= 1008, per=25.10%, avg=929.82, stdev=52.26, samples=11
   iops        : min=  206, max=  252, avg=232.45, stdev=13.06, samples=11
  lat (msec)   : 10=0.12%, 20=0.27%, 50=99.61%
  cpu          : usr=0.46%, sys=6.51%, ctx=4799, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=168391: Tue Jun 16 07:53:11 2026
  read: IOPS=222, BW=889KiB/s (911kB/s)(4984KiB/5604msec)
    slat (usec): min=44, max=5622, avg=2411.98, stdev=729.31
    clat (usec): min=11902, max=66575, avg=33803.01, stdev=4490.24
     lat (usec): min=11979, max=69171, avg=36216.31, stdev=4562.05
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26870], 10.00th=[28443], 20.00th=[30540],
     | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39060], 95.00th=[40633],
     | 99.00th=[42730], 99.50th=[43779], 99.90th=[58459], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  760, max=  984, per=25.04%, avg=890.80, stdev=65.54, samples=10
   iops        : min=  190, max=  246, avg=222.70, stdev=16.38, samples=10
  write: IOPS=234, BW=938KiB/s (960kB/s)(5256KiB/5604msec); 0 zone resets
    slat (usec): min=43, max=2663, avg=169.30, stdev=112.15
    clat (usec): min=10817, max=65949, avg=33595.77, stdev=4165.83
     lat (usec): min=10908, max=66104, avg=33766.10, stdev=4171.25
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[28181], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[41681], 99.50th=[45351], 99.90th=[63177], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  848, max= 1040, per=25.36%, avg=939.30, stdev=57.55, samples=10
   iops        : min=  212, max=  260, avg=234.80, stdev=14.40, samples=10
  lat (msec)   : 20=0.70%, 50=98.95%, 100=0.35%
  cpu          : usr=0.41%, sys=6.60%, ctx=4748, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=168392: Tue Jun 16 07:53:11 2026
  read: IOPS=224, BW=899KiB/s (920kB/s)(5060KiB/5629msec)
    slat (usec): min=67, max=6251, avg=2435.65, stdev=697.55
    clat (usec): min=18831, max=46503, avg=33569.88, stdev=4030.29
     lat (usec): min=18988, max=49351, avg=36006.83, stdev=4032.69
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26608], 10.00th=[28443], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[32637], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[38536], 95.00th=[40109],
     | 99.00th=[43254], 99.50th=[44827], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min=  864, max=  928, per=25.27%, avg=899.20, stdev=23.00, samples=10
   iops        : min=  216, max=  232, avg=224.80, stdev= 5.75, samples=10
  write: IOPS=230, BW=920KiB/s (942kB/s)(5180KiB/5629msec); 0 zone resets
    slat (usec): min=47, max=2577, avg=168.30, stdev=108.31
    clat (usec): min=23985, max=53621, avg=33900.71, stdev=3397.43
     lat (usec): min=24216, max=53810, avg=34070.03, stdev=3402.11
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28705], 10.00th=[29754], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[53216], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  824, max=  984, per=24.75%, avg=916.60, stdev=57.52, samples=10
   iops        : min=  206, max=  246, avg=229.10, stdev=14.43, samples=10
  lat (msec)   : 20=0.04%, 50=99.88%, 100=0.08%
  cpu          : usr=0.43%, sys=6.63%, ctx=4847, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3559KiB/s (3644kB/s), 882KiB/s-899KiB/s (904kB/s-920kB/s), io=19.6MiB (20.5MB), run=5604-5639msec
  WRITE: bw=3705KiB/s (3794kB/s), 920KiB/s-938KiB/s (942kB/s-960kB/s), io=20.4MiB (21.4MB), run=5604-5639msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=168396: Tue Jun 16 07:53:14 2026
  read: IOPS=9, BW=396MiB/s (415MB/s)(40.0MiB/101msec)
    slat (nsec): min=11401k, max=11401k, avg=11401473.00, stdev= 0.00
    clat (nsec): min=85006k, max=85006k, avg=85006044.00, stdev= 0.00
     lat (nsec): min=96411k, max=96411k, avg=96411467.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[85459],  5.00th=[85459], 10.00th=[85459], 20.00th=[85459],
     | 30.00th=[85459], 40.00th=[85459], 50.00th=[85459], 60.00th=[85459],
     | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459],
     | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459],
     | 99.99th=[85459]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.00%, ctx=11, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=396MiB/s (415MB/s), 396MiB/s-396MiB/s (415MB/s-415MB/s), io=40.0MiB (41.9MB), run=101-101msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout