== sanity test 398c: run fio to test AIO ================= 07:45:34 (1756813534)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.139942 s, 300 MB/s
osc.lustre-OST0000-osc-ffff91aa9260b000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=187308: Tue Sep  2 07:45:42 2025
  write: IOPS=539, BW=2156KiB/s (2208kB/s)(10.0MiB/4749msec); 0 zone resets
    slat (usec): min=29, max=501, avg=88.48, stdev=39.24
    clat (usec): min=5067, max=42193, avg=29214.21, stdev=3105.39
     lat (usec): min=5138, max=42368, avg=29303.97, stdev=3107.11
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[24511], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27919], 40.00th=[28705], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[30540], 80.00th=[31327], 90.00th=[32637], 95.00th=[33817],
     | 99.00th=[37487], 99.50th=[41157], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2056, max= 2288, per=24.96%, avg=2152.89, stdev=88.22, samples=9
   iops        : min=  514, max=  572, avg=538.22, stdev=22.06, samples=9
  lat (msec)   : 10=0.08%, 20=0.31%, 50=99.61%
  cpu          : usr=0.67%, sys=4.49%, ctx=1217, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=187309: Tue Sep  2 07:45:42 2025
  write: IOPS=539, BW=2160KiB/s (2212kB/s)(10.0MiB/4741msec); 0 zone resets
    slat (usec): min=29, max=26235, avg=98.33, stdev=518.10
    clat (usec): min=3724, max=60013, avg=29224.54, stdev=3370.81
     lat (usec): min=3805, max=60067, avg=29324.05, stdev=3342.51
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[24249], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27919], 40.00th=[28443], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[30540], 80.00th=[31327], 90.00th=[32375], 95.00th=[33424],
     | 99.00th=[38536], 99.50th=[42206], 99.90th=[53216], 99.95th=[56886],
     | 99.99th=[60031]
   bw (  KiB/s): min= 2048, max= 2296, per=25.12%, avg=2166.56, stdev=89.60, samples=9
   iops        : min=  512, max=  574, avg=541.56, stdev=22.42, samples=9
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.31%, 50=99.30%, 100=0.27%
  cpu          : usr=0.49%, sys=5.30%, ctx=1333, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=187310: Tue Sep  2 07:45:42 2025
  write: IOPS=542, BW=2170KiB/s (2223kB/s)(10.0MiB/4718msec); 0 zone resets
    slat (usec): min=29, max=490, avg=88.92, stdev=42.55
    clat (usec): min=4783, max=59893, avg=29338.28, stdev=3602.34
     lat (usec): min=4850, max=59969, avg=29428.27, stdev=3599.81
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[24511], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27919], 40.00th=[28705], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[30540], 80.00th=[31327], 90.00th=[32637], 95.00th=[33817],
     | 99.00th=[41681], 99.50th=[46924], 99.90th=[53740], 99.95th=[56361],
     | 99.99th=[60031]
   bw (  KiB/s): min= 2048, max= 2296, per=25.20%, avg=2173.44, stdev=103.43, samples=9
   iops        : min=  512, max=  574, avg=543.33, stdev=25.85, samples=9
  lat (msec)   : 10=0.04%, 20=0.51%, 50=99.06%, 100=0.39%
  cpu          : usr=0.72%, sys=4.49%, ctx=1323, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=187311: Tue Sep  2 07:45:42 2025
  write: IOPS=539, BW=2157KiB/s (2209kB/s)(10.0MiB/4747msec); 0 zone resets
    slat (usec): min=28, max=337, avg=85.86, stdev=37.53
    clat (usec): min=3485, max=45960, avg=29240.41, stdev=3051.88
     lat (usec): min=3606, max=46016, avg=29327.36, stdev=3052.45
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[24511], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27919], 40.00th=[28705], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[30802], 80.00th=[31327], 90.00th=[32637], 95.00th=[33424],
     | 99.00th=[35914], 99.50th=[41157], 99.90th=[42206], 99.95th=[42730],
     | 99.99th=[45876]
   bw (  KiB/s): min= 2008, max= 2312, per=25.02%, avg=2158.00, stdev=101.31, samples=9
   iops        : min=  502, max=  578, avg=539.44, stdev=25.38, samples=9
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.43%, 50=99.49%
  cpu          : usr=0.48%, sys=5.01%, ctx=1137, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8625KiB/s (8832kB/s), 2156KiB/s-2170KiB/s (2208kB/s-2223kB/s), io=40.0MiB (41.9MB), run=4718-4749msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=187321: Tue Sep  2 07:45:51 2025
  read: IOPS=207, BW=831KiB/s (850kB/s)(5048KiB/6078msec)
    slat (usec): min=1137, max=9616, avg=2813.86, stdev=901.02
    clat (usec): min=18835, max=59578, avg=36141.46, stdev=5893.00
     lat (usec): min=21800, max=63608, avg=38957.90, stdev=6079.30
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27657], 10.00th=[29492], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43779], 95.00th=[47973],
     | 99.00th=[53740], 99.50th=[56361], 99.90th=[58983], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  704, max=  880, per=24.78%, avg=817.82, stdev=61.64, samples=11
   iops        : min=  176, max=  220, avg=204.45, stdev=15.41, samples=11
  write: IOPS=213, BW=854KiB/s (875kB/s)(5192KiB/6078msec); 0 zone resets
    slat (usec): min=32, max=471, avg=85.21, stdev=36.28
    clat (usec): min=8652, max=57171, avg=36325.32, stdev=4920.00
     lat (usec): min=8740, max=57242, avg=36411.52, stdev=4920.22
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29754], 10.00th=[31065], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43254], 95.00th=[45351],
     | 99.00th=[50594], 99.50th=[51643], 99.90th=[54789], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  680, max= 1056, per=24.85%, avg=853.73, stdev=112.96, samples=11
   iops        : min=  170, max=  264, avg=213.36, stdev=28.31, samples=11
  lat (msec)   : 10=0.04%, 20=0.04%, 50=97.81%, 100=2.11%
  cpu          : usr=0.41%, sys=5.12%, ctx=2329, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=187322: Tue Sep  2 07:45:51 2025
  read: IOPS=204, BW=818KiB/s (838kB/s)(4976KiB/6081msec)
    slat (usec): min=1131, max=7960, avg=2770.81, stdev=862.12
    clat (usec): min=8087, max=65176, avg=36007.23, stdev=6104.10
     lat (usec): min=10506, max=66747, avg=38779.86, stdev=6265.72
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[27395], 10.00th=[29230], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43779], 95.00th=[47973],
     | 99.00th=[53216], 99.50th=[54789], 99.90th=[61604], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  684, max=  904, per=24.43%, avg=806.18, stdev=77.26, samples=11
   iops        : min=  171, max=  226, avg=201.55, stdev=19.32, samples=11
  write: IOPS=216, BW=866KiB/s (886kB/s)(5264KiB/6081msec); 0 zone resets
    slat (usec): min=25, max=505, avg=86.56, stdev=38.04
    clat (usec): min=7990, max=54098, avg=36382.59, stdev=5095.66
     lat (usec): min=8075, max=54154, avg=36470.17, stdev=5096.62
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[30802], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43254], 95.00th=[46400],
     | 99.00th=[50594], 99.50th=[52691], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  640, max=  992, per=25.17%, avg=864.64, stdev=104.54, samples=11
   iops        : min=  160, max=  248, avg=216.09, stdev=26.20, samples=11
  lat (msec)   : 10=0.08%, 20=0.31%, 50=97.58%, 100=2.03%
  cpu          : usr=0.49%, sys=5.08%, ctx=2333, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=187323: Tue Sep  2 07:45:51 2025
  read: IOPS=207, BW=830KiB/s (850kB/s)(4984KiB/6005msec)
    slat (usec): min=38, max=7630, avg=2758.88, stdev=867.03
    clat (usec): min=5820, max=59191, avg=35880.46, stdev=6292.26
     lat (usec): min=5862, max=60725, avg=38640.88, stdev=6464.35
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[26870], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43779], 95.00th=[47449],
     | 99.00th=[52691], 99.50th=[54264], 99.90th=[57410], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  624, max=  944, per=25.18%, avg=830.82, stdev=100.82, samples=11
   iops        : min=  156, max=  236, avg=207.64, stdev=25.18, samples=11
  write: IOPS=218, BW=875KiB/s (896kB/s)(5256KiB/6005msec); 0 zone resets
    slat (usec): min=31, max=1622, avg=85.30, stdev=55.61
    clat (usec): min=18656, max=60546, avg=36259.66, stdev=5097.33
     lat (usec): min=18755, max=60590, avg=36346.17, stdev=5096.05
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28967], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[51119], 99.50th=[53216], 99.90th=[56886], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  680, max= 1040, per=25.25%, avg=867.45, stdev=99.41, samples=11
   iops        : min=  170, max=  260, avg=216.82, stdev=24.86, samples=11
  lat (msec)   : 10=0.16%, 20=0.39%, 50=97.27%, 100=2.19%
  cpu          : usr=0.67%, sys=4.85%, ctx=2367, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=187324: Tue Sep  2 07:45:51 2025
  read: IOPS=208, BW=833KiB/s (853kB/s)(5060KiB/6076msec)
    slat (usec): min=1150, max=7785, avg=2802.89, stdev=848.72
    clat (usec): min=7424, max=58324, avg=36231.48, stdev=5987.92
     lat (usec): min=9962, max=61706, avg=39035.96, stdev=6163.46
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27395], 10.00th=[29230], 20.00th=[31589],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43779], 95.00th=[47973],
     | 99.00th=[54264], 99.50th=[56886], 99.90th=[57934], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  680, max=  920, per=25.05%, avg=826.55, stdev=75.01, samples=11
   iops        : min=  170, max=  230, avg=206.64, stdev=18.75, samples=11
  write: IOPS=213, BW=853KiB/s (873kB/s)(5180KiB/6076msec); 0 zone resets
    slat (usec): min=33, max=1089, avg=86.68, stdev=50.02
    clat (usec): min=6903, max=66007, avg=36211.84, stdev=5452.46
     lat (usec): min=6994, max=66066, avg=36299.64, stdev=5455.34
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28181], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[53216], 99.50th=[57410], 99.90th=[64750], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  640, max=  968, per=24.62%, avg=845.55, stdev=97.52, samples=11
   iops        : min=  160, max=  242, avg=211.36, stdev=24.40, samples=11
  lat (msec)   : 10=0.08%, 20=0.04%, 50=97.58%, 100=2.30%
  cpu          : usr=0.56%, sys=4.94%, ctx=2365, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3300KiB/s (3379kB/s), 818KiB/s-833KiB/s (838kB/s-853kB/s), io=19.6MiB (20.5MB), run=6005-6081msec
  WRITE: bw=3436KiB/s (3518kB/s), 853KiB/s-875KiB/s (873kB/s-896kB/s), io=20.4MiB (21.4MB), run=6005-6081msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=187328: Tue Sep  2 07:45:54 2025
  read: IOPS=10, BW=426MiB/s (446MB/s)(40.0MiB/94msec)
    slat (nsec): min=13246k, max=13246k, avg=13245857.00, stdev= 0.00
    clat (nsec): min=75298k, max=75298k, avg=75298139.00, stdev= 0.00
     lat (nsec): min=88548k, max=88548k, avg=88548438.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[74974],  5.00th=[74974], 10.00th=[74974], 20.00th=[74974],
     | 30.00th=[74974], 40.00th=[74974], 50.00th=[74974], 60.00th=[74974],
     | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974],
     | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=13.98%, ctx=3, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=40.0MiB (41.9MB), run=94-94msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout