== sanity test 398c: run fio to test AIO ================= 04:53:09 (1722243189)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.128059 s, 328 MB/s
osc.lustre-OST0000-osc-ffff8800b590b800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=19582: Mon Jul 29 04:53:18 2024
  write: IOPS=353, BW=1413KiB/s (1447kB/s)(10.0MiB/7248msec)
    slat (usec): min=11, max=4807, avg=74.20, stdev=153.57
    clat (usec): min=9422, max=76012, avg=44957.38, stdev=9812.16
     lat (usec): min=9544, max=76039, avg=45033.09, stdev=9816.19
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[30016], 10.00th=[32637], 20.00th=[36439],
     | 30.00th=[39584], 40.00th=[42206], 50.00th=[44303], 60.00th=[46400],
     | 70.00th=[50070], 80.00th=[52691], 90.00th=[58459], 95.00th=[61604],
     | 99.00th=[71828], 99.50th=[72877], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1240, max= 1856, per=24.67%, avg=1393.86, stdev=174.91, samples=14
   iops        : min=  310, max=  464, avg=348.43, stdev=43.75, samples=14
  lat (msec)   : 10=0.08%, 20=0.12%, 50=70.23%, 100=29.57%
  cpu          : usr=0.00%, sys=4.21%, ctx=1332, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19583: Mon Jul 29 04:53:18 2024
  write: IOPS=354, BW=1417KiB/s (1451kB/s)(10.0MiB/7226msec)
    slat (usec): min=10, max=2683, avg=75.65, stdev=132.93
    clat (usec): min=2966, max=76632, avg=45029.97, stdev=9803.44
     lat (usec): min=2998, max=76657, avg=45107.30, stdev=9806.11
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[30278], 10.00th=[32900], 20.00th=[36963],
     | 30.00th=[39584], 40.00th=[42206], 50.00th=[44303], 60.00th=[46924],
     | 70.00th=[50070], 80.00th=[53216], 90.00th=[57934], 95.00th=[62129],
     | 99.00th=[69731], 99.50th=[72877], 99.90th=[76022], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1229, max= 1888, per=24.81%, avg=1401.50, stdev=191.59, samples=14
   iops        : min=  307, max=  472, avg=350.29, stdev=47.94, samples=14
  lat (msec)   : 4=0.04%, 20=0.16%, 50=70.12%, 100=29.69%
  cpu          : usr=1.47%, sys=2.96%, ctx=1453, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19584: Mon Jul 29 04:53:18 2024
  write: IOPS=353, BW=1412KiB/s (1446kB/s)(10.0MiB/7250msec)
    slat (usec): min=10, max=20392, avg=81.16, stdev=421.62
    clat (usec): min=11890, max=75725, avg=44998.66, stdev=9894.64
     lat (usec): min=11995, max=75787, avg=45081.27, stdev=9891.68
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[30278], 10.00th=[32637], 20.00th=[36439],
     | 30.00th=[39584], 40.00th=[42206], 50.00th=[44303], 60.00th=[46924],
     | 70.00th=[50070], 80.00th=[52691], 90.00th=[57934], 95.00th=[62653],
     | 99.00th=[72877], 99.50th=[73925], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1224, max= 1884, per=24.64%, avg=1391.86, stdev=184.87, samples=14
   iops        : min=  306, max=  471, avg=347.93, stdev=46.25, samples=14
  lat (msec)   : 20=0.23%, 50=69.38%, 100=30.39%
  cpu          : usr=1.27%, sys=2.99%, ctx=1355, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19585: Mon Jul 29 04:53:18 2024
  write: IOPS=353, BW=1413KiB/s (1447kB/s)(10.0MiB/7247msec)
    slat (usec): min=11, max=19895, avg=82.08, stdev=405.90
    clat (usec): min=5240, max=77555, avg=44976.94, stdev=9839.63
     lat (usec): min=5296, max=77592, avg=45060.40, stdev=9829.07
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[30278], 10.00th=[32900], 20.00th=[36439],
     | 30.00th=[39584], 40.00th=[42206], 50.00th=[44303], 60.00th=[46924],
     | 70.00th=[50070], 80.00th=[52691], 90.00th=[57934], 95.00th=[61604],
     | 99.00th=[70779], 99.50th=[74974], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1192, max= 1864, per=24.68%, avg=1394.29, stdev=185.04, samples=14
   iops        : min=  298, max=  466, avg=348.57, stdev=46.26, samples=14
  lat (msec)   : 10=0.08%, 20=0.08%, 50=69.61%, 100=30.23%
  cpu          : usr=0.70%, sys=3.57%, ctx=1496, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5650KiB/s (5785kB/s), 1412KiB/s-1417KiB/s (1446kB/s-1451kB/s), io=40.0MiB (41.9MB), run=7226-7250msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=19601: Mon Jul 29 04:53:24 2024
   read: IOPS=232, BW=930KiB/s (952kB/s)(5048KiB/5430msec)
    slat (usec): min=18, max=6450, avg=1945.04, stdev=813.56
    clat (usec): min=7486, max=63666, avg=32093.73, stdev=7630.01
     lat (usec): min=9223, max=64820, avg=34041.90, stdev=7928.06
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[21627], 10.00th=[23462], 20.00th=[25297],
     | 30.00th=[27395], 40.00th=[29230], 50.00th=[31065], 60.00th=[33162],
     | 70.00th=[35390], 80.00th=[38011], 90.00th=[42730], 95.00th=[45876],
     | 99.00th=[52691], 99.50th=[56361], 99.90th=[61604], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  736, max= 1160, per=24.70%, avg=908.60, stdev=139.86, samples=10
   iops        : min=  184, max=  290, avg=227.10, stdev=34.96, samples=10
  write: IOPS=239, BW=956KiB/s (979kB/s)(5192KiB/5430msec)
    slat (usec): min=12, max=1361, avg=63.05, stdev=81.57
    clat (usec): min=18061, max=59740, avg=33594.13, stdev=7410.34
     lat (usec): min=18111, max=59792, avg=33658.64, stdev=7413.08
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[23200], 10.00th=[24511], 20.00th=[26608],
     | 30.00th=[28967], 40.00th=[31065], 50.00th=[32900], 60.00th=[34866],
     | 70.00th=[36963], 80.00th=[39584], 90.00th=[43254], 95.00th=[46924],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[57934], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  752, max= 1136, per=24.69%, avg=945.40, stdev=122.80, samples=10
   iops        : min=  188, max=  284, avg=236.30, stdev=30.70, samples=10
  lat (msec)   : 10=0.04%, 20=1.17%, 50=96.13%, 100=2.66%
  cpu          : usr=3.04%, sys=5.05%, ctx=2397, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19602: Mon Jul 29 04:53:24 2024
   read: IOPS=228, BW=915KiB/s (937kB/s)(4976KiB/5439msec)
    slat (usec): min=16, max=8211, avg=1977.37, stdev=847.71
    clat (usec): min=3286, max=61028, avg=32166.52, stdev=7646.47
     lat (usec): min=3473, max=63897, avg=34147.02, stdev=7911.33
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[21365], 10.00th=[23462], 20.00th=[25560],
     | 30.00th=[27395], 40.00th=[29230], 50.00th=[31065], 60.00th=[32900],
     | 70.00th=[35914], 80.00th=[39060], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[51119], 99.50th=[53216], 99.90th=[58459], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  704, max= 1160, per=24.31%, avg=894.00, stdev=145.38, samples=10
   iops        : min=  176, max=  290, avg=223.40, stdev=36.40, samples=10
  write: IOPS=241, BW=968KiB/s (991kB/s)(5264KiB/5439msec)
    slat (usec): min=13, max=2159, avg=68.55, stdev=110.17
    clat (usec): min=13385, max=60179, avg=33648.22, stdev=7530.47
     lat (usec): min=13482, max=60398, avg=33718.09, stdev=7535.86
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[23200], 10.00th=[24511], 20.00th=[26870],
     | 30.00th=[29230], 40.00th=[31065], 50.00th=[32900], 60.00th=[34866],
     | 70.00th=[36963], 80.00th=[39584], 90.00th=[43779], 95.00th=[46924],
     | 99.00th=[55837], 99.50th=[58459], 99.90th=[59507], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  776, max= 1152, per=25.00%, avg=957.20, stdev=127.90, samples=10
   iops        : min=  194, max=  288, avg=239.20, stdev=32.01, samples=10
  lat (msec)   : 4=0.04%, 20=1.45%, 50=95.98%, 100=2.54%
  cpu          : usr=1.80%, sys=6.36%, ctx=2491, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19603: Mon Jul 29 04:53:24 2024
   read: IOPS=228, BW=913KiB/s (935kB/s)(4984KiB/5456msec)
    slat (usec): min=627, max=7432, avg=2011.80, stdev=836.07
    clat (usec): min=13817, max=59175, avg=32303.33, stdev=7759.85
     lat (usec): min=14942, max=62424, avg=34318.08, stdev=8074.52
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[21627], 10.00th=[23462], 20.00th=[25560],
     | 30.00th=[27132], 40.00th=[29230], 50.00th=[31065], 60.00th=[33424],
     | 70.00th=[35390], 80.00th=[38536], 90.00th=[43254], 95.00th=[47449],
     | 99.00th=[53740], 99.50th=[55313], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  648, max= 1032, per=24.48%, avg=900.40, stdev=136.80, samples=10
   iops        : min=  162, max=  258, avg=225.00, stdev=34.25, samples=10
  write: IOPS=240, BW=963KiB/s (986kB/s)(5256KiB/5456msec)
    slat (usec): min=12, max=1195, avg=66.50, stdev=84.18
    clat (usec): min=20136, max=60718, avg=33408.38, stdev=7222.13
     lat (usec): min=20180, max=61249, avg=33476.87, stdev=7226.51
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[23462], 10.00th=[24773], 20.00th=[26870],
     | 30.00th=[28967], 40.00th=[30540], 50.00th=[32637], 60.00th=[34341],
     | 70.00th=[36439], 80.00th=[39060], 90.00th=[43254], 95.00th=[46924],
     | 99.00th=[52691], 99.50th=[56361], 99.90th=[58983], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  752, max= 1256, per=24.54%, avg=939.60, stdev=154.84, samples=10
   iops        : min=  188, max=  314, avg=234.80, stdev=38.74, samples=10
  lat (msec)   : 20=0.86%, 50=96.13%, 100=3.01%
  cpu          : usr=0.00%, sys=8.34%, ctx=2443, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19604: Mon Jul 29 04:53:24 2024
   read: IOPS=232, BW=929KiB/s (951kB/s)(5060KiB/5449msec)
    slat (usec): min=730, max=6904, avg=2023.60, stdev=851.65
    clat (usec): min=2693, max=64367, avg=32216.34, stdev=7938.61
     lat (usec): min=4650, max=65927, avg=34242.43, stdev=8239.00
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[21365], 10.00th=[23200], 20.00th=[25297],
     | 30.00th=[27132], 40.00th=[28967], 50.00th=[31065], 60.00th=[33162],
     | 70.00th=[35914], 80.00th=[38536], 90.00th=[42730], 95.00th=[47449],
     | 99.00th=[53216], 99.50th=[54789], 99.90th=[61604], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  696, max= 1120, per=24.88%, avg=915.00, stdev=140.45, samples=10
   iops        : min=  174, max=  280, avg=228.70, stdev=35.11, samples=10
  write: IOPS=237, BW=951KiB/s (973kB/s)(5180KiB/5449msec)
    slat (usec): min=11, max=3156, avg=70.76, stdev=137.65
    clat (usec): min=8412, max=61376, avg=33441.99, stdev=7384.56
     lat (usec): min=8458, max=61421, avg=33514.12, stdev=7392.46
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[23200], 10.00th=[24511], 20.00th=[26608],
     | 30.00th=[28967], 40.00th=[30802], 50.00th=[32637], 60.00th=[34866],
     | 70.00th=[36439], 80.00th=[39584], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[54264], 99.50th=[57410], 99.90th=[61080], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  704, max= 1192, per=24.25%, avg=928.60, stdev=144.30, samples=10
   iops        : min=  176, max=  298, avg=232.10, stdev=36.07, samples=10
  lat (msec)   : 4=0.04%, 10=0.04%, 20=1.05%, 50=96.37%, 100=2.50%
  cpu          : usr=1.47%, sys=6.68%, ctx=2500, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3678KiB/s (3766kB/s), 913KiB/s-930KiB/s (935kB/s-952kB/s), io=19.6MiB (20.5MB), run=5430-5456msec
  WRITE: bw=3829KiB/s (3921kB/s), 951KiB/s-968KiB/s (973kB/s-991kB/s), io=20.4MiB (21.4MB), run=5430-5456msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=19612: Mon Jul 29 04:53:26 2024
   read: IOPS=8, BW=325MiB/s (341MB/s)(40.0MiB/123msec)
    slat (nsec): min=9046.2k, max=9046.2k, avg=9046217.00, stdev= 0.00
    clat (nsec): min=109617k, max=109617k, avg=109617188.00, stdev= 0.00
     lat (nsec): min=118686k, max=118686k, avg=118685737.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  110],  5.00th=[  110], 10.00th=[  110], 20.00th=[  110],
     | 30.00th=[  110], 40.00th=[  110], 50.00th=[  110], 60.00th=[  110],
     | 70.00th=[  110], 80.00th=[  110], 90.00th=[  110], 95.00th=[  110],
     | 99.00th=[  110], 99.50th=[  110], 99.90th=[  110], 99.95th=[  110],
     | 99.99th=[  110]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=6.56%, ctx=6, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=40.0MiB (41.9MB), run=123-123msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck