== sanity test 398c: run fio to test AIO ================= 15:05:29 (1755803129)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.1316 s, 319 MB/s
osc.lustre-OST0000-osc-ffffa0d760971800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=193955: Thu Aug 21 15:05:39 2025
  write: IOPS=419, BW=1676KiB/s (1717kB/s)(10.0MiB/6108msec); 0 zone resets
    slat (usec): min=43, max=729, avg=172.18, stdev=81.21
    clat (usec): min=4021, max=80182, avg=37932.98, stdev=6486.96
     lat (usec): min=4280, max=80301, avg=38107.01, stdev=6490.29
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[29230], 10.00th=[31327], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45351], 95.00th=[47449],
     | 99.00th=[58459], 99.50th=[66847], 99.90th=[76022], 99.95th=[78119],
     | 99.99th=[80217]
   bw (  KiB/s): min= 1424, max= 1912, per=25.54%, avg=1688.27, stdev=171.52, samples=11
   iops        : min=  356, max=  478, avg=422.00, stdev=42.79, samples=11
  lat (msec)   : 10=0.43%, 20=0.27%, 50=96.91%, 100=2.38%
  cpu          : usr=0.64%, sys=6.96%, ctx=1657, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=193956: Thu Aug 21 15:05:39 2025
  write: IOPS=413, BW=1653KiB/s (1693kB/s)(10.0MiB/6195msec); 0 zone resets
    slat (usec): min=36, max=728, avg=163.66, stdev=79.37
    clat (usec): min=5969, max=72895, avg=38076.86, stdev=5528.65
     lat (usec): min=6114, max=73011, avg=38242.34, stdev=5533.18
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30016], 10.00th=[31589], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[45351], 95.00th=[47449],
     | 99.00th=[51643], 99.50th=[56886], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1424, max= 1904, per=25.00%, avg=1652.58, stdev=153.20, samples=12
   iops        : min=  356, max=  476, avg=413.08, stdev=38.26, samples=12
  lat (msec)   : 10=0.08%, 50=98.59%, 100=1.33%
  cpu          : usr=0.76%, sys=7.06%, ctx=1354, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=193957: Thu Aug 21 15:05:39 2025
  write: IOPS=413, BW=1653KiB/s (1693kB/s)(10.0MiB/6195msec); 0 zone resets
    slat (usec): min=43, max=758, avg=159.71, stdev=77.22
    clat (usec): min=5427, max=72957, avg=38034.61, stdev=5609.05
     lat (usec): min=5491, max=73059, avg=38196.01, stdev=5618.13
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30278], 10.00th=[31589], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45351], 95.00th=[47449],
     | 99.00th=[53216], 99.50th=[57410], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1408, max= 1840, per=25.02%, avg=1653.92, stdev=150.43, samples=12
   iops        : min=  352, max=  460, avg=413.42, stdev=37.57, samples=12
  lat (msec)   : 10=0.16%, 50=98.05%, 100=1.80%
  cpu          : usr=0.65%, sys=6.46%, ctx=1408, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=193958: Thu Aug 21 15:05:39 2025
  write: IOPS=414, BW=1656KiB/s (1696kB/s)(10.0MiB/6182msec); 0 zone resets
    slat (usec): min=41, max=818, avg=162.93, stdev=80.00
    clat (usec): min=23600, max=60615, avg=37985.97, stdev=5394.73
     lat (usec): min=23796, max=60834, avg=38150.72, stdev=5399.06
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29754], 10.00th=[31589], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[42206], 90.00th=[45351], 95.00th=[47449],
     | 99.00th=[52167], 99.50th=[56886], 99.90th=[57410], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1408, max= 1872, per=25.00%, avg=1652.58, stdev=155.88, samples=12
   iops        : min=  352, max=  468, avg=413.08, stdev=38.93, samples=12
  lat (msec)   : 50=98.71%, 100=1.29%
  cpu          : usr=0.66%, sys=6.39%, ctx=1377, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6612KiB/s (6770kB/s), 1653KiB/s-1676KiB/s (1693kB/s-1717kB/s), io=40.0MiB (41.9MB), run=6108-6195msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=193968: Thu Aug 21 15:05:49 2025
  read: IOPS=179, BW=716KiB/s (733kB/s)(5048KiB/7048msec)
    slat (usec): min=1117, max=11693, avg=3569.46, stdev=1288.41
    clat (usec): min=4095, max=69547, avg=41758.54, stdev=6826.08
     lat (usec): min=7931, max=76069, avg=45329.43, stdev=6847.82
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[31589], 10.00th=[33817], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[46924], 90.00th=[50070], 95.00th=[52691],
     | 99.00th=[57934], 99.50th=[58459], 99.90th=[66323], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  642, max=  784, per=24.97%, avg=710.92, stdev=41.40, samples=13
   iops        : min=  160, max=  196, avg=177.69, stdev=10.42, samples=13
  write: IOPS=184, BW=737KiB/s (754kB/s)(5192KiB/7048msec); 0 zone resets
    slat (usec): min=27, max=4305, avg=77.92, stdev=129.57
    clat (usec): min=4768, max=64256, avg=41985.65, stdev=5211.56
     lat (usec): min=4824, max=64357, avg=42064.59, stdev=5209.37
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[34341], 10.00th=[35914], 20.00th=[38011],
     | 30.00th=[39584], 40.00th=[40633], 50.00th=[41681], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[45876], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[54789], 99.50th=[56886], 99.90th=[61604], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  648, max=  840, per=24.87%, avg=737.08, stdev=64.70, samples=13
   iops        : min=  162, max=  210, avg=184.23, stdev=16.23, samples=13
  lat (msec)   : 10=0.23%, 20=0.16%, 50=91.48%, 100=8.12%
  cpu          : usr=0.31%, sys=3.92%, ctx=2163, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=193969: Thu Aug 21 15:05:49 2025
  read: IOPS=176, BW=707KiB/s (724kB/s)(4976KiB/7038msec)
    slat (usec): min=1117, max=11681, avg=3580.14, stdev=1258.87
    clat (usec): min=13654, max=69442, avg=41574.14, stdev=5996.86
     lat (usec): min=17236, max=72875, avg=45155.71, stdev=6085.33
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[32637], 10.00th=[34341], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[58983], 99.50th=[61080], 99.90th=[65799], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  597, max=  784, per=24.70%, avg=703.15, stdev=61.57, samples=13
   iops        : min=  149, max=  196, avg=175.77, stdev=15.43, samples=13
  write: IOPS=186, BW=748KiB/s (766kB/s)(5264KiB/7038msec); 0 zone resets
    slat (usec): min=30, max=4224, avg=77.54, stdev=118.98
    clat (usec): min=23618, max=69341, avg=42016.74, stdev=4893.02
     lat (usec): min=23714, max=69432, avg=42095.22, stdev=4896.99
    clat percentiles (usec):
     |  1.00th=[32375],  5.00th=[34866], 10.00th=[35914], 20.00th=[38011],
     | 30.00th=[39060], 40.00th=[40633], 50.00th=[41681], 60.00th=[42730],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[48497], 95.00th=[50594],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[61080], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  686, max=  880, per=25.47%, avg=754.92, stdev=61.94, samples=13
   iops        : min=  171, max=  220, avg=188.69, stdev=15.53, samples=13
  lat (msec)   : 20=0.04%, 50=93.59%, 100=6.37%
  cpu          : usr=0.34%, sys=3.89%, ctx=2199, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=193970: Thu Aug 21 15:05:49 2025
  read: IOPS=177, BW=711KiB/s (728kB/s)(4984KiB/7007msec)
    slat (usec): min=999, max=12936, avg=3555.73, stdev=1256.27
    clat (usec): min=25012, max=67605, avg=41541.56, stdev=5962.59
     lat (usec): min=27073, max=69560, avg=45098.53, stdev=5986.46
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[32113], 10.00th=[33817], 20.00th=[36439],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[45876], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[56886], 99.50th=[59507], 99.90th=[63177], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  608, max=  808, per=25.12%, avg=715.15, stdev=74.68, samples=13
   iops        : min=  152, max=  202, avg=178.77, stdev=18.70, samples=13
  write: IOPS=187, BW=750KiB/s (768kB/s)(5256KiB/7007msec); 0 zone resets
    slat (usec): min=27, max=358, avg=76.30, stdev=33.43
    clat (usec): min=7241, max=63597, avg=41843.66, stdev=5338.59
     lat (usec): min=7333, max=63660, avg=41920.78, stdev=5338.08
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[33817], 10.00th=[35914], 20.00th=[38011],
     | 30.00th=[39060], 40.00th=[40633], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44303], 80.00th=[45876], 90.00th=[48497], 95.00th=[50594],
     | 99.00th=[55313], 99.50th=[58459], 99.90th=[61604], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  616, max=  880, per=25.09%, avg=743.77, stdev=71.20, samples=13
   iops        : min=  154, max=  220, avg=185.92, stdev=17.82, samples=13
  lat (msec)   : 10=0.04%, 20=0.16%, 50=92.62%, 100=7.19%
  cpu          : usr=0.41%, sys=3.85%, ctx=2193, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=193971: Thu Aug 21 15:05:49 2025
  read: IOPS=181, BW=727KiB/s (745kB/s)(5060KiB/6959msec)
    slat (usec): min=97, max=8951, avg=3472.65, stdev=1202.11
    clat (usec): min=6231, max=68386, avg=41077.64, stdev=6285.50
     lat (usec): min=6399, max=72410, avg=44551.77, stdev=6377.38
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[32113], 10.00th=[33817], 20.00th=[36439],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[45876], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[57410], 99.50th=[58983], 99.90th=[62129], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  632, max=  792, per=25.42%, avg=723.62, stdev=44.98, samples=13
   iops        : min=  158, max=  198, avg=180.85, stdev=11.25, samples=13
  write: IOPS=186, BW=744KiB/s (762kB/s)(5180KiB/6959msec); 0 zone resets
    slat (usec): min=26, max=507, avg=76.90, stdev=38.59
    clat (usec): min=20967, max=78505, avg=42207.77, stdev=5175.80
     lat (usec): min=21192, max=78597, avg=42285.56, stdev=5173.09
    clat percentiles (usec):
     |  1.00th=[31851],  5.00th=[34866], 10.00th=[35914], 20.00th=[38011],
     | 30.00th=[39584], 40.00th=[40633], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[46400], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  624, max=  848, per=25.23%, avg=747.92, stdev=60.02, samples=13
   iops        : min=  156, max=  212, avg=186.92, stdev=15.02, samples=13
  lat (msec)   : 10=0.16%, 20=0.12%, 50=92.97%, 100=6.76%
  cpu          : usr=0.27%, sys=4.05%, ctx=2259, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2847KiB/s (2916kB/s), 707KiB/s-727KiB/s (724kB/s-745kB/s), io=19.6MiB (20.5MB), run=6959-7048msec
  WRITE: bw=2964KiB/s (3035kB/s), 737KiB/s-750KiB/s (754kB/s-768kB/s), io=20.4MiB (21.4MB), run=6959-7048msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=193976: Thu Aug 21 15:05:52 2025
  read: IOPS=9, BW=385MiB/s (403MB/s)(40.0MiB/104msec)
    slat (nsec): min=7835.2k, max=7835.2k, avg=7835200.00, stdev= 0.00
    clat (nsec): min=90388k, max=90388k, avg=90387898.00, stdev= 0.00
     lat (nsec): min=98226k, max=98226k, avg=98225719.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[90702],  5.00th=[90702], 10.00th=[90702], 20.00th=[90702],
     | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702],
     | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702],
     | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.77%, ctx=3, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=385MiB/s (403MB/s), 385MiB/s-385MiB/s (403MB/s-403MB/s), io=40.0MiB (41.9MB), run=104-104msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout