== sanity test 398c: run fio to test AIO ================= 14:30:34 (1752690634)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.113107 s, 371 MB/s
osc.lustre-OST0000-osc-ffff89bb84fa0800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=194579: Wed Jul 16 14:30:42 2025
  write: IOPS=529, BW=2117KiB/s (2167kB/s)(10.0MiB/4838msec); 0 zone resets
    slat (usec): min=26, max=359, avg=83.40, stdev=35.55
    clat (usec): min=6017, max=48923, avg=29798.12, stdev=3889.52
     lat (usec): min=6107, max=49014, avg=29882.46, stdev=3889.96
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[24249], 10.00th=[25297], 20.00th=[26870],
     | 30.00th=[27919], 40.00th=[28443], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[31327], 80.00th=[33162], 90.00th=[34866], 95.00th=[36439],
     | 99.00th=[39060], 99.50th=[47973], 99.90th=[49021], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min= 1848, max= 2344, per=25.11%, avg=2125.56, stdev=150.50, samples=9
   iops        : min=  462, max=  586, avg=531.33, stdev=37.62, samples=9
  lat (msec)   : 10=0.04%, 50=99.96%
  cpu          : usr=0.45%, sys=4.36%, ctx=1305, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194580: Wed Jul 16 14:30:42 2025
  write: IOPS=529, BW=2119KiB/s (2170kB/s)(10.0MiB/4832msec); 0 zone resets
    slat (usec): min=28, max=420, avg=85.91, stdev=36.62
    clat (usec): min=3948, max=60668, avg=29790.50, stdev=4067.73
     lat (usec): min=4046, max=60773, avg=29877.56, stdev=4069.48
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[24249], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28443], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[31327], 80.00th=[32900], 90.00th=[34866], 95.00th=[36439],
     | 99.00th=[40109], 99.50th=[47973], 99.90th=[50594], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1856, max= 2336, per=25.18%, avg=2132.00, stdev=157.30, samples=9
   iops        : min=  464, max=  584, avg=533.00, stdev=39.33, samples=9
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.12%, 50=99.65%, 100=0.16%
  cpu          : usr=0.56%, sys=4.82%, ctx=1411, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194581: Wed Jul 16 14:30:42 2025
  write: IOPS=533, BW=2134KiB/s (2185kB/s)(10.0MiB/4798msec); 0 zone resets
    slat (usec): min=25, max=450, avg=86.30, stdev=36.36
    clat (usec): min=3622, max=58379, avg=29850.26, stdev=4272.63
     lat (usec): min=3692, max=58460, avg=29937.43, stdev=4272.82
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[23987], 10.00th=[25297], 20.00th=[26870],
     | 30.00th=[27919], 40.00th=[28443], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[31065], 80.00th=[33162], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[40633], 99.50th=[48497], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1872, max= 2336, per=25.32%, avg=2144.00, stdev=160.85, samples=9
   iops        : min=  468, max=  584, avg=536.00, stdev=40.21, samples=9
  lat (msec)   : 4=0.08%, 10=0.08%, 20=0.35%, 50=99.30%, 100=0.20%
  cpu          : usr=0.54%, sys=4.38%, ctx=1402, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194582: Wed Jul 16 14:30:42 2025
  write: IOPS=530, BW=2120KiB/s (2171kB/s)(10.0MiB/4830msec); 0 zone resets
    slat (usec): min=27, max=25137, avg=95.53, stdev=496.36
    clat (usec): min=6486, max=57817, avg=29800.95, stdev=4074.51
     lat (usec): min=6902, max=57901, avg=29897.45, stdev=4051.80
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[24249], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27919], 40.00th=[28443], 50.00th=[29230], 60.00th=[30016],
     | 70.00th=[31327], 80.00th=[33162], 90.00th=[34866], 95.00th=[36439],
     | 99.00th=[40109], 99.50th=[47449], 99.90th=[49021], 99.95th=[54264],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1840, max= 2320, per=25.17%, avg=2131.22, stdev=157.96, samples=9
   iops        : min=  460, max=  580, avg=532.78, stdev=39.48, samples=9
  lat (msec)   : 10=0.08%, 20=0.27%, 50=99.57%, 100=0.08%
  cpu          : usr=0.70%, sys=4.66%, ctx=1358, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8466KiB/s (8669kB/s), 2117KiB/s-2134KiB/s (2167kB/s-2185kB/s), io=40.0MiB (41.9MB), run=4798-4838msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=194592: Wed Jul 16 14:30:52 2025
  read: IOPS=173, BW=693KiB/s (710kB/s)(5048KiB/7284msec)
    slat (usec): min=1106, max=10224, avg=3572.04, stdev=1291.23
    clat (usec): min=3785, max=69512, avg=42935.60, stdev=7920.59
     lat (usec): min=8998, max=72496, avg=46509.48, stdev=8268.69
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[30016], 10.00th=[32375], 20.00th=[35914],
     | 30.00th=[38536], 40.00th=[41157], 50.00th=[43254], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[49021], 90.00th=[52691], 95.00th=[55313],
     | 99.00th=[61604], 99.50th=[64750], 99.90th=[68682], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  548, max=  976, per=25.10%, avg=690.57, stdev=120.20, samples=14
   iops        : min=  137, max=  244, avg=172.64, stdev=30.05, samples=14
  write: IOPS=178, BW=713KiB/s (730kB/s)(5192KiB/7284msec); 0 zone resets
    slat (usec): min=29, max=1072, avg=83.44, stdev=46.41
    clat (usec): min=8685, max=66275, avg=43727.10, stdev=7383.17
     lat (usec): min=8741, max=66448, avg=43811.51, stdev=7384.91
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[31065], 10.00th=[33424], 20.00th=[37487],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47973], 80.00th=[49546], 90.00th=[53216], 95.00th=[55313],
     | 99.00th=[61080], 99.50th=[62129], 99.90th=[65274], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  528, max=  888, per=24.84%, avg=711.50, stdev=98.49, samples=14
   iops        : min=  132, max=  222, avg=177.86, stdev=24.65, samples=14
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.04%, 50=81.60%, 100=18.28%
  cpu          : usr=0.43%, sys=4.01%, ctx=2412, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194593: Wed Jul 16 14:30:52 2025
  read: IOPS=171, BW=686KiB/s (703kB/s)(4976KiB/7252msec)
    slat (usec): min=45, max=10647, avg=3536.96, stdev=1342.66
    clat (usec): min=5573, max=69355, avg=43183.72, stdev=8119.93
     lat (usec): min=8636, max=74054, avg=46722.22, stdev=8485.70
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[29754], 10.00th=[32375], 20.00th=[36439],
     | 30.00th=[39060], 40.00th=[41157], 50.00th=[43254], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[50070], 90.00th=[53216], 95.00th=[56361],
     | 99.00th=[62653], 99.50th=[64226], 99.90th=[68682], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  544, max=  952, per=24.86%, avg=683.86, stdev=113.40, samples=14
   iops        : min=  136, max=  238, avg=170.93, stdev=28.39, samples=14
  write: IOPS=181, BW=726KiB/s (743kB/s)(5264KiB/7252msec); 0 zone resets
    slat (usec): min=33, max=875, avg=82.24, stdev=42.80
    clat (usec): min=6623, max=65020, avg=43752.71, stdev=7500.96
     lat (usec): min=6721, max=65119, avg=43836.05, stdev=7502.87
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30278], 10.00th=[32900], 20.00th=[37487],
     | 30.00th=[40633], 40.00th=[42206], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[47973], 80.00th=[50070], 90.00th=[53216], 95.00th=[55837],
     | 99.00th=[60031], 99.50th=[62129], 99.90th=[63701], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  616, max=  928, per=25.26%, avg=723.57, stdev=103.04, samples=14
   iops        : min=  154, max=  232, avg=180.86, stdev=25.80, samples=14
  lat (msec)   : 10=0.08%, 20=0.08%, 50=79.84%, 100=20.00%
  cpu          : usr=0.36%, sys=4.07%, ctx=2396, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194594: Wed Jul 16 14:30:52 2025
  read: IOPS=170, BW=683KiB/s (700kB/s)(4984KiB/7293msec)
    slat (usec): min=1113, max=10116, avg=3574.19, stdev=1307.56
    clat (usec): min=3162, max=67933, avg=43242.83, stdev=8370.15
     lat (usec): min=6518, max=72328, avg=46818.65, stdev=8704.54
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[29492], 10.00th=[32113], 20.00th=[36963],
     | 30.00th=[39584], 40.00th=[41681], 50.00th=[43254], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[50070], 90.00th=[53740], 95.00th=[56361],
     | 99.00th=[63177], 99.50th=[64750], 99.90th=[66847], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  510, max=  968, per=24.73%, avg=680.43, stdev=112.95, samples=14
   iops        : min=  127, max=  242, avg=170.07, stdev=28.30, samples=14
  write: IOPS=180, BW=721KiB/s (738kB/s)(5256KiB/7293msec); 0 zone resets
    slat (usec): min=27, max=812, avg=81.53, stdev=43.00
    clat (usec): min=9155, max=68397, avg=43472.84, stdev=7803.88
     lat (usec): min=9232, max=68458, avg=43555.38, stdev=7805.44
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[30016], 10.00th=[32637], 20.00th=[36963],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[49546], 90.00th=[53216], 95.00th=[55837],
     | 99.00th=[60556], 99.50th=[61604], 99.90th=[66323], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  584, max= 1016, per=24.76%, avg=709.07, stdev=113.80, samples=14
   iops        : min=  146, max=  254, avg=177.21, stdev=28.51, samples=14
  lat (msec)   : 4=0.08%, 10=0.08%, 20=0.31%, 50=80.62%, 100=18.91%
  cpu          : usr=0.33%, sys=4.02%, ctx=2328, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194595: Wed Jul 16 14:30:52 2025
  read: IOPS=173, BW=694KiB/s (711kB/s)(5060KiB/7290msec)
    slat (usec): min=1084, max=10605, avg=3553.39, stdev=1290.04
    clat (usec): min=6993, max=66321, avg=43200.78, stdev=7835.29
     lat (usec): min=10074, max=70693, avg=46755.65, stdev=8226.04
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29754], 10.00th=[32637], 20.00th=[36439],
     | 30.00th=[39584], 40.00th=[41681], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[49546], 90.00th=[52691], 95.00th=[55837],
     | 99.00th=[61080], 99.50th=[62653], 99.90th=[65274], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  542, max=  904, per=25.13%, avg=691.29, stdev=101.55, samples=14
   iops        : min=  135, max=  226, avg=172.79, stdev=25.44, samples=14
  write: IOPS=177, BW=711KiB/s (728kB/s)(5180KiB/7290msec); 0 zone resets
    slat (usec): min=31, max=333, avg=79.96, stdev=33.21
    clat (usec): min=8491, max=65310, avg=43538.16, stdev=7127.97
     lat (usec): min=8582, max=65387, avg=43619.09, stdev=7130.11
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[31327], 10.00th=[33817], 20.00th=[36963],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[49546], 90.00th=[52167], 95.00th=[54789],
     | 99.00th=[58983], 99.50th=[60556], 99.90th=[63701], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  535, max=  928, per=24.70%, avg=707.36, stdev=103.28, samples=14
   iops        : min=  133, max=  232, avg=176.79, stdev=25.92, samples=14
  lat (msec)   : 10=0.08%, 50=81.29%, 100=18.63%
  cpu          : usr=0.44%, sys=3.91%, ctx=2375, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2752KiB/s (2818kB/s), 683KiB/s-694KiB/s (700kB/s-711kB/s), io=19.6MiB (20.5MB), run=7252-7293msec
  WRITE: bw=2865KiB/s (2933kB/s), 711KiB/s-726KiB/s (728kB/s-743kB/s), io=20.4MiB (21.4MB), run=7252-7293msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=194599: Wed Jul 16 14:30:55 2025
  read: IOPS=9, BW=370MiB/s (388MB/s)(40.0MiB/108msec)
    slat (nsec): min=8543.9k, max=8543.9k, avg=8543855.00, stdev= 0.00
    clat (nsec): min=93773k, max=93773k, avg=93772511.00, stdev= 0.00
     lat (nsec): min=102320k, max=102320k, avg=102319756.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[93848],  5.00th=[93848], 10.00th=[93848], 20.00th=[93848],
     | 30.00th=[93848], 40.00th=[93848], 50.00th=[93848], 60.00th=[93848],
     | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848],
     | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848],
     | 99.99th=[93848]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.41%, ctx=2, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=370MiB/s (388MB/s), 370MiB/s-370MiB/s (388MB/s-388MB/s), io=40.0MiB (41.9MB), run=108-108msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout