== sanity test 398c: run fio to test AIO ================= 22:22:50 (1772680970)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.140874 s, 298 MB/s
osc.lustre-OST0000-osc-ffff988374d4b000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=191321: Wed Mar  4 22:22:58 2026
  write: IOPS=471, BW=1888KiB/s (1933kB/s)(10.0MiB/5425msec); 0 zone resets
    slat (usec): min=41, max=752, avg=181.01, stdev=96.81
    clat (usec): min=5989, max=59975, avg=33665.30, stdev=4025.88
     lat (usec): min=6122, max=60167, avg=33847.45, stdev=4023.06
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28443], 10.00th=[29754], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38536], 95.00th=[40633],
     | 99.00th=[45876], 99.50th=[53216], 99.90th=[56361], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min= 1640, max= 2057, per=25.06%, avg=1878.50, stdev=138.39, samples=10
   iops        : min=  410, max=  514, avg=469.60, stdev=34.56, samples=10
  lat (msec)   : 10=0.04%, 20=0.12%, 50=99.26%, 100=0.59%
  cpu          : usr=0.42%, sys=6.54%, ctx=2640, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=191322: Wed Mar  4 22:22:58 2026
  write: IOPS=469, BW=1876KiB/s (1921kB/s)(10.0MiB/5458msec); 0 zone resets
    slat (usec): min=50, max=25767, avg=206.71, stdev=515.84
    clat (usec): min=5070, max=58511, avg=33649.45, stdev=3940.72
     lat (usec): min=5197, max=58606, avg=33857.27, stdev=3905.64
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28443], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[45876], 99.50th=[47449], 99.90th=[55837], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1624, max= 2024, per=24.93%, avg=1869.00, stdev=129.12, samples=10
   iops        : min=  406, max=  506, avg=467.20, stdev=32.25, samples=10
  lat (msec)   : 10=0.08%, 20=0.04%, 50=99.57%, 100=0.31%
  cpu          : usr=0.42%, sys=7.44%, ctx=2688, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=191323: Wed Mar  4 22:22:58 2026
  write: IOPS=468, BW=1875KiB/s (1920kB/s)(10.0MiB/5461msec); 0 zone resets
    slat (usec): min=53, max=24426, avg=197.64, stdev=489.20
    clat (usec): min=11443, max=57997, avg=33645.86, stdev=3929.18
     lat (usec): min=14755, max=58177, avg=33844.59, stdev=3909.92
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28443], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[45876], 99.50th=[50594], 99.90th=[55837], 99.95th=[55837],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1640, max= 2056, per=24.95%, avg=1870.10, stdev=135.04, samples=10
   iops        : min=  410, max=  514, avg=467.50, stdev=33.75, samples=10
  lat (msec)   : 20=0.12%, 50=99.30%, 100=0.59%
  cpu          : usr=0.35%, sys=7.22%, ctx=2699, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=191324: Wed Mar  4 22:22:58 2026
  write: IOPS=468, BW=1874KiB/s (1919kB/s)(10.0MiB/5464msec); 0 zone resets
    slat (usec): min=53, max=806, avg=189.32, stdev=104.95
    clat (usec): min=23863, max=55425, avg=33645.48, stdev=3758.42
     lat (usec): min=24047, max=55536, avg=33836.00, stdev=3766.30
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28705], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[45876], 99.50th=[47449], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1600, max= 2040, per=24.89%, avg=1866.00, stdev=142.15, samples=10
   iops        : min=  400, max=  510, avg=466.50, stdev=35.54, samples=10
  lat (msec)   : 50=99.69%, 100=0.31%
  cpu          : usr=0.60%, sys=6.81%, ctx=2756, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7496KiB/s (7676kB/s), 1874KiB/s-1888KiB/s (1919kB/s-1933kB/s), io=40.0MiB (41.9MB), run=5425-5464msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=191334: Wed Mar  4 22:23:07 2026
  read: IOPS=212, BW=850KiB/s (870kB/s)(5048KiB/5939msec)
    slat (usec): min=1075, max=5505, avg=2555.10, stdev=692.94
    clat (usec): min=22150, max=53994, avg=35330.71, stdev=4734.41
     lat (usec): min=23551, max=57906, avg=37888.21, stdev=4734.49
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27919], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41157], 95.00th=[43254],
     | 99.00th=[46924], 99.50th=[49546], 99.90th=[51643], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  736, max=  928, per=25.17%, avg=850.64, stdev=60.69, samples=11
   iops        : min=  184, max=  232, avg=212.64, stdev=15.19, samples=11
  write: IOPS=218, BW=874KiB/s (895kB/s)(5192KiB/5939msec); 0 zone resets
    slat (usec): min=48, max=852, avg=184.95, stdev=100.38
    clat (usec): min=9326, max=55574, avg=35573.06, stdev=3493.10
     lat (usec): min=9467, max=55718, avg=35759.06, stdev=3490.11
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30278], 10.00th=[31589], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[39584], 95.00th=[41157],
     | 99.00th=[45351], 99.50th=[46924], 99.90th=[51119], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min=  733, max= 1000, per=24.93%, avg=876.82, stdev=79.11, samples=11
   iops        : min=  183, max=  250, avg=219.18, stdev=19.82, samples=11
  lat (msec)   : 10=0.04%, 50=99.65%, 100=0.31%
  cpu          : usr=0.56%, sys=7.12%, ctx=4750, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=191335: Wed Mar  4 22:23:07 2026
  read: IOPS=211, BW=846KiB/s (866kB/s)(4976KiB/5881msec)
    slat (usec): min=42, max=8428, avg=2566.03, stdev=737.07
    clat (usec): min=5045, max=67869, avg=35066.14, stdev=5288.29
     lat (usec): min=5091, max=69303, avg=37634.64, stdev=5346.56
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[27132], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[41157], 95.00th=[43254],
     | 99.00th=[46400], 99.50th=[47449], 99.90th=[62653], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  771, max=  896, per=25.06%, avg=846.82, stdev=45.98, samples=11
   iops        : min=  192, max=  224, avg=211.64, stdev=11.62, samples=11
  write: IOPS=223, BW=895KiB/s (917kB/s)(5264KiB/5881msec); 0 zone resets
    slat (usec): min=40, max=836, avg=181.83, stdev=93.41
    clat (usec): min=13160, max=65182, avg=35610.97, stdev=3937.58
     lat (usec): min=13277, max=65588, avg=35793.90, stdev=3940.65
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[30016], 10.00th=[31327], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[41157],
     | 99.00th=[45351], 99.50th=[47973], 99.90th=[54264], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  808, max= 1016, per=25.49%, avg=896.55, stdev=64.67, samples=11
   iops        : min=  202, max=  254, avg=224.09, stdev=16.13, samples=11
  lat (msec)   : 10=0.23%, 20=0.31%, 50=99.14%, 100=0.31%
  cpu          : usr=0.36%, sys=7.33%, ctx=4656, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=191336: Wed Mar  4 22:23:07 2026
  read: IOPS=209, BW=839KiB/s (859kB/s)(4984KiB/5939msec)
    slat (usec): min=1127, max=10939, avg=2541.38, stdev=718.77
    clat (usec): min=6947, max=54329, avg=35297.52, stdev=4872.72
     lat (usec): min=9367, max=57655, avg=37840.39, stdev=4859.89
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27657], 10.00th=[29230], 20.00th=[31065],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[41157], 95.00th=[43254],
     | 99.00th=[47449], 99.50th=[49546], 99.90th=[52691], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  740, max=  912, per=24.87%, avg=840.36, stdev=69.98, samples=11
   iops        : min=  185, max=  228, avg=210.09, stdev=17.50, samples=11
  write: IOPS=221, BW=885KiB/s (906kB/s)(5256KiB/5939msec); 0 zone resets
    slat (usec): min=64, max=935, avg=184.57, stdev=101.30
    clat (usec): min=6335, max=50374, avg=35785.74, stdev=3860.83
     lat (usec): min=6850, max=50531, avg=35971.30, stdev=3855.82
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30278], 10.00th=[31327], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40633], 95.00th=[41681],
     | 99.00th=[44827], 99.50th=[45351], 99.90th=[49021], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min=  784, max= 1008, per=24.94%, avg=877.00, stdev=84.94, samples=11
   iops        : min=  196, max=  252, avg=219.18, stdev=21.31, samples=11
  lat (msec)   : 10=0.08%, 20=0.20%, 50=99.45%, 100=0.27%
  cpu          : usr=0.44%, sys=7.06%, ctx=4726, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=191337: Wed Mar  4 22:23:07 2026
  read: IOPS=212, BW=852KiB/s (872kB/s)(5060KiB/5939msec)
    slat (usec): min=1141, max=7955, avg=2546.82, stdev=693.28
    clat (usec): min=22289, max=53310, avg=35148.19, stdev=4501.65
     lat (usec): min=24880, max=55489, avg=37696.41, stdev=4538.04
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28181], 10.00th=[29230], 20.00th=[31327],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[45876], 99.50th=[47973], 99.90th=[50594], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  740, max=  928, per=25.19%, avg=851.27, stdev=44.43, samples=11
   iops        : min=  185, max=  232, avg=212.82, stdev=11.11, samples=11
  write: IOPS=218, BW=872KiB/s (893kB/s)(5180KiB/5939msec); 0 zone resets
    slat (usec): min=56, max=861, avg=185.41, stdev=100.01
    clat (usec): min=25998, max=57317, avg=35710.07, stdev=3472.75
     lat (usec): min=26119, max=57600, avg=35896.62, stdev=3469.33
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[30278], 10.00th=[31327], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[44827], 99.50th=[46400], 99.90th=[55837], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  771, max=  928, per=24.84%, avg=873.73, stdev=44.24, samples=11
   iops        : min=  192, max=  232, avg=218.36, stdev=11.24, samples=11
  lat (msec)   : 50=99.65%, 100=0.35%
  cpu          : usr=0.35%, sys=7.28%, ctx=4760, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3379KiB/s (3460kB/s), 839KiB/s-852KiB/s (859kB/s-872kB/s), io=19.6MiB (20.5MB), run=5881-5939msec
  WRITE: bw=3518KiB/s (3602kB/s), 872KiB/s-895KiB/s (893kB/s-917kB/s), io=20.4MiB (21.4MB), run=5881-5939msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=191341: Wed Mar  4 22:23:11 2026
  read: IOPS=8, BW=357MiB/s (374MB/s)(40.0MiB/112msec)
    slat (nsec): min=9648.1k, max=9648.1k, avg=9648062.00, stdev= 0.00
    clat (nsec): min=96861k, max=96861k, avg=96861123.00, stdev= 0.00
     lat (nsec): min=106513k, max=106513k, avg=106513232.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[96994],  5.00th=[96994], 10.00th=[96994], 20.00th=[96994],
     | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994],
     | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994],
     | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994],
     | 99.99th=[96994]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.21%, ctx=10, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=357MiB/s (374MB/s), 357MiB/s-357MiB/s (374MB/s-374MB/s), io=40.0MiB (41.9MB), run=112-112msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout