== sanity test 398c: run fio to test AIO ================= 21:09:58 (1770689398)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.130526 s, 321 MB/s
osc.lustre-OST0000-osc-ffff915e92034000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182006: Mon Feb  9 21:10:06 2026
  write: IOPS=537, BW=2149KiB/s (2201kB/s)(10.0MiB/4765msec); 0 zone resets
    slat (usec): min=21, max=404, avg=85.95, stdev=37.82
    clat (usec): min=10064, max=48917, avg=29652.19, stdev=3740.31
     lat (usec): min=10160, max=49000, avg=29739.22, stdev=3742.97
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25560], 10.00th=[26084], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28181], 50.00th=[28967], 60.00th=[29754],
     | 70.00th=[30802], 80.00th=[31851], 90.00th=[33817], 95.00th=[36963],
     | 99.00th=[42730], 99.50th=[44303], 99.90th=[45876], 99.95th=[45876],
     | 99.99th=[49021]
   bw (  KiB/s): min= 1936, max= 2368, per=25.07%, avg=2140.67, stdev=131.65, samples=9
   iops        : min=  484, max=  592, avg=535.11, stdev=32.87, samples=9
  lat (msec)   : 20=0.55%, 50=99.45%
  cpu          : usr=0.61%, sys=3.76%, ctx=2632, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182007: Mon Feb  9 21:10:06 2026
  write: IOPS=533, BW=2135KiB/s (2186kB/s)(10.0MiB/4796msec); 0 zone resets
    slat (usec): min=21, max=1035, avg=80.97, stdev=42.86
    clat (usec): min=3847, max=44604, avg=29571.48, stdev=3503.78
     lat (usec): min=3944, max=44802, avg=29653.50, stdev=3508.56
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25560], 10.00th=[26084], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28181], 50.00th=[28967], 60.00th=[30016],
     | 70.00th=[30540], 80.00th=[31851], 90.00th=[33424], 95.00th=[36439],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[44827], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1960, max= 2328, per=24.86%, avg=2123.44, stdev=116.90, samples=9
   iops        : min=  490, max=  582, avg=530.78, stdev=29.27, samples=9
  lat (msec)   : 4=0.04%, 20=0.12%, 50=99.84%
  cpu          : usr=0.50%, sys=3.57%, ctx=2476, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182008: Mon Feb  9 21:10:06 2026
  write: IOPS=534, BW=2136KiB/s (2188kB/s)(10.0MiB/4793msec); 0 zone resets
    slat (usec): min=25, max=469, avg=88.64, stdev=39.20
    clat (usec): min=12932, max=45079, avg=29586.57, stdev=3570.65
     lat (usec): min=13035, max=45194, avg=29676.25, stdev=3576.17
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25297], 10.00th=[26084], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28181], 50.00th=[28967], 60.00th=[29754],
     | 70.00th=[30802], 80.00th=[31851], 90.00th=[33817], 95.00th=[36963],
     | 99.00th=[42206], 99.50th=[43779], 99.90th=[44303], 99.95th=[44303],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1944, max= 2368, per=24.93%, avg=2129.00, stdev=128.86, samples=9
   iops        : min=  486, max=  592, avg=532.22, stdev=32.22, samples=9
  lat (msec)   : 20=0.39%, 50=99.61%
  cpu          : usr=0.67%, sys=3.80%, ctx=2700, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182009: Mon Feb  9 21:10:06 2026
  write: IOPS=534, BW=2140KiB/s (2191kB/s)(10.0MiB/4786msec); 0 zone resets
    slat (usec): min=26, max=24278, avg=98.26, stdev=480.55
    clat (usec): min=10030, max=48878, avg=29597.25, stdev=3593.90
     lat (usec): min=10243, max=48941, avg=29696.56, stdev=3578.56
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25560], 10.00th=[26084], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28181], 50.00th=[28967], 60.00th=[29754],
     | 70.00th=[30802], 80.00th=[31851], 90.00th=[33424], 95.00th=[36439],
     | 99.00th=[42206], 99.50th=[44303], 99.90th=[45351], 99.95th=[45351],
     | 99.99th=[49021]
   bw (  KiB/s): min= 1952, max= 2376, per=24.98%, avg=2133.00, stdev=128.62, samples=9
   iops        : min=  488, max=  594, avg=533.22, stdev=32.16, samples=9
  lat (msec)   : 20=0.39%, 50=99.61%
  cpu          : usr=0.63%, sys=4.33%, ctx=2663, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8540KiB/s (8745kB/s), 2135KiB/s-2149KiB/s (2186kB/s-2201kB/s), io=40.0MiB (41.9MB), run=4765-4796msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182019: Mon Feb  9 21:10:14 2026
  read: IOPS=224, BW=900KiB/s (921kB/s)(5048KiB/5612msec)
    slat (usec): min=1003, max=6684, avg=2445.67, stdev=777.84
    clat (usec): min=18346, max=56526, avg=33415.71, stdev=4965.14
     lat (usec): min=20014, max=58328, avg=35863.54, stdev=5072.91
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[25822], 10.00th=[27132], 20.00th=[29230],
     | 30.00th=[30802], 40.00th=[32113], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[37487], 90.00th=[40109], 95.00th=[42206],
     | 99.00th=[46400], 99.50th=[47449], 99.90th=[55837], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=  752, max=  984, per=25.16%, avg=899.30, stdev=76.22, samples=10
   iops        : min=  188, max=  246, avg=224.80, stdev=19.10, samples=10
  write: IOPS=231, BW=925KiB/s (947kB/s)(5192KiB/5612msec); 0 zone resets
    slat (usec): min=25, max=865, avg=85.71, stdev=50.95
    clat (usec): min=18238, max=49417, avg=33594.03, stdev=4299.94
     lat (usec): min=18337, max=49466, avg=33680.63, stdev=4300.12
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[26870], 10.00th=[28705], 20.00th=[30016],
     | 30.00th=[31065], 40.00th=[32113], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[39584], 95.00th=[41157],
     | 99.00th=[45351], 99.50th=[46400], 99.90th=[48497], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min=  741, max= 1056, per=25.10%, avg=934.10, stdev=99.90, samples=10
   iops        : min=  185, max=  264, avg=233.50, stdev=25.03, samples=10
  lat (msec)   : 20=0.08%, 50=99.84%, 100=0.08%
  cpu          : usr=0.41%, sys=4.51%, ctx=4563, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182020: Mon Feb  9 21:10:14 2026
  read: IOPS=222, BW=890KiB/s (911kB/s)(4976KiB/5594msec)
    slat (usec): min=979, max=8042, avg=2433.32, stdev=794.51
    clat (usec): min=9905, max=51909, avg=33199.53, stdev=5124.92
     lat (usec): min=12790, max=54505, avg=35634.81, stdev=5237.04
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25822], 10.00th=[27132], 20.00th=[29230],
     | 30.00th=[30540], 40.00th=[31589], 50.00th=[32637], 60.00th=[33817],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[40109], 95.00th=[42730],
     | 99.00th=[47449], 99.50th=[47973], 99.90th=[51119], 99.95th=[52167],
     | 99.99th=[52167]
   bw (  KiB/s): min=  747, max= 1016, per=24.89%, avg=889.90, stdev=84.41, samples=10
   iops        : min=  186, max=  254, avg=222.40, stdev=21.25, samples=10
  write: IOPS=235, BW=941KiB/s (964kB/s)(5264KiB/5594msec); 0 zone resets
    slat (usec): min=32, max=655, avg=83.70, stdev=44.64
    clat (usec): min=20006, max=57831, avg=33821.19, stdev=4729.39
     lat (usec): min=20077, max=57921, avg=33905.88, stdev=4732.71
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28705], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32375], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[40109], 95.00th=[42730],
     | 99.00th=[46924], 99.50th=[50594], 99.90th=[56886], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  856, max= 1032, per=25.69%, avg=956.00, stdev=56.35, samples=10
   iops        : min=  214, max=  258, avg=239.00, stdev=14.09, samples=10
  lat (msec)   : 10=0.04%, 20=0.16%, 50=99.38%, 100=0.43%
  cpu          : usr=0.50%, sys=4.43%, ctx=4516, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182021: Mon Feb  9 21:10:14 2026
  read: IOPS=223, BW=895KiB/s (916kB/s)(4984KiB/5570msec)
    slat (usec): min=35, max=7726, avg=2453.53, stdev=802.97
    clat (usec): min=5613, max=58096, avg=33512.55, stdev=5151.53
     lat (usec): min=8052, max=59918, avg=35968.12, stdev=5237.91
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25822], 10.00th=[27395], 20.00th=[29492],
     | 30.00th=[30802], 40.00th=[32113], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[37487], 90.00th=[40109], 95.00th=[42206],
     | 99.00th=[47449], 99.50th=[49546], 99.90th=[53216], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  848, max= 1024, per=25.31%, avg=905.00, stdev=67.22, samples=10
   iops        : min=  212, max=  256, avg=226.20, stdev=16.85, samples=10
  write: IOPS=235, BW=944KiB/s (966kB/s)(5256KiB/5570msec); 0 zone resets
    slat (usec): min=34, max=698, avg=86.06, stdev=51.48
    clat (usec): min=7271, max=57904, avg=33501.37, stdev=4732.16
     lat (usec): min=7337, max=57965, avg=33588.41, stdev=4731.61
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27132], 10.00th=[28443], 20.00th=[29754],
     | 30.00th=[31065], 40.00th=[32113], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[47449], 99.50th=[49021], 99.90th=[54789], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  808, max= 1120, per=25.42%, avg=946.10, stdev=113.33, samples=10
   iops        : min=  202, max=  280, avg=236.50, stdev=28.36, samples=10
  lat (msec)   : 10=0.08%, 20=0.23%, 50=99.34%, 100=0.35%
  cpu          : usr=0.45%, sys=4.47%, ctx=4492, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182022: Mon Feb  9 21:10:14 2026
  read: IOPS=225, BW=902KiB/s (924kB/s)(5060KiB/5607msec)
    slat (usec): min=920, max=7476, avg=2441.17, stdev=782.86
    clat (usec): min=14706, max=50877, avg=33440.04, stdev=4887.47
     lat (usec): min=18081, max=53835, avg=35883.42, stdev=5002.17
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25560], 10.00th=[27395], 20.00th=[29492],
     | 30.00th=[30802], 40.00th=[32113], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[46400], 99.50th=[47449], 99.90th=[50070], 99.95th=[51119],
     | 99.99th=[51119]
   bw (  KiB/s): min=  793, max= 1016, per=25.40%, avg=908.10, stdev=74.14, samples=10
   iops        : min=  198, max=  254, avg=227.00, stdev=18.58, samples=10
  write: IOPS=230, BW=924KiB/s (946kB/s)(5180KiB/5607msec); 0 zone resets
    slat (usec): min=33, max=877, avg=86.09, stdev=51.16
    clat (usec): min=23410, max=53695, avg=33728.83, stdev=4359.72
     lat (usec): min=23702, max=53773, avg=33815.95, stdev=4358.86
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[27657], 10.00th=[28705], 20.00th=[30016],
     | 30.00th=[31065], 40.00th=[32113], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[45876], 99.50th=[46924], 99.90th=[52167], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  816, max= 1000, per=24.94%, avg=928.30, stdev=71.44, samples=10
   iops        : min=  204, max=  250, avg=232.00, stdev=17.99, samples=10
  lat (msec)   : 20=0.08%, 50=99.65%, 100=0.27%
  cpu          : usr=0.32%, sys=4.62%, ctx=4600, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3576KiB/s (3662kB/s), 890KiB/s-902KiB/s (911kB/s-924kB/s), io=19.6MiB (20.5MB), run=5570-5612msec
  WRITE: bw=3723KiB/s (3812kB/s), 924KiB/s-944KiB/s (946kB/s-966kB/s), io=20.4MiB (21.4MB), run=5570-5612msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182026: Mon Feb  9 21:10:18 2026
  read: IOPS=10, BW=412MiB/s (432MB/s)(40.0MiB/97msec)
    slat (nsec): min=9480.7k, max=9480.7k, avg=9480664.00, stdev= 0.00
    clat (nsec): min=82769k, max=82769k, avg=82769388.00, stdev= 0.00
     lat (nsec): min=92253k, max=92253k, avg=92253164.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[82314],  5.00th=[82314], 10.00th=[82314], 20.00th=[82314],
     | 30.00th=[82314], 40.00th=[82314], 50.00th=[82314], 60.00th=[82314],
     | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314],
     | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[82314]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.33%, ctx=8, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=412MiB/s (432MB/s), 412MiB/s-412MiB/s (432MB/s-432MB/s), io=40.0MiB (41.9MB), run=97-97msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout