== sanity test 398c: run fio to test AIO ================= 07:53:56 (1730811236)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.115095 s, 364 MB/s
osc.lustre-OST0000-osc-ffff8800b435f800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=11820: Tue Nov  5 07:54:05 2024
  write: IOPS=378, BW=1515KiB/s (1552kB/s)(10.0MiB/6757msec)
    slat (usec): min=20, max=598, avg=92.41, stdev=52.08
    clat (usec): min=4577, max=85491, avg=41963.76, stdev=4726.29
     lat (usec): min=4662, max=85531, avg=42058.49, stdev=4729.21
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[35914], 10.00th=[37487], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[42730], 60.00th=[43779],
     | 70.00th=[44303], 80.00th=[44827], 90.00th=[45351], 95.00th=[45876],
     | 99.00th=[49021], 99.50th=[53740], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[85459]
   bw (  KiB/s): min= 1440, max= 1624, per=25.16%, avg=1518.46, stdev=51.16, samples=13
   iops        : min=  360, max=  406, avg=379.62, stdev=12.79, samples=13
  lat (msec)   : 10=0.16%, 20=0.27%, 50=98.91%, 100=0.66%
  cpu          : usr=0.78%, sys=3.88%, ctx=1507, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11821: Tue Nov  5 07:54:05 2024
  write: IOPS=377, BW=1509KiB/s (1545kB/s)(10.0MiB/6786msec)
    slat (usec): min=24, max=647, avg=80.88, stdev=48.28
    clat (usec): min=24230, max=69612, avg=42074.57, stdev=3542.22
     lat (usec): min=24277, max=69680, avg=42157.54, stdev=3543.26
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[35914], 10.00th=[37487], 20.00th=[39584],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[42730], 60.00th=[43779],
     | 70.00th=[44303], 80.00th=[44827], 90.00th=[45351], 95.00th=[45876],
     | 99.00th=[49021], 99.50th=[49021], 99.90th=[56361], 99.95th=[65274],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1448, max= 1624, per=24.99%, avg=1508.00, stdev=53.47, samples=13
   iops        : min=  362, max=  406, avg=377.00, stdev=13.37, samples=13
  lat (msec)   : 50=99.77%, 100=0.23%
  cpu          : usr=0.71%, sys=3.39%, ctx=1037, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11822: Tue Nov  5 07:54:05 2024
  write: IOPS=377, BW=1510KiB/s (1546kB/s)(10.0MiB/6781msec)
    slat (usec): min=24, max=535, avg=87.40, stdev=50.11
    clat (usec): min=8878, max=87214, avg=42109.32, stdev=3909.62
     lat (usec): min=8991, max=87311, avg=42198.89, stdev=3909.71
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[35914], 10.00th=[37487], 20.00th=[39584],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[42730], 60.00th=[43779],
     | 70.00th=[44303], 80.00th=[44827], 90.00th=[45351], 95.00th=[45876],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[87557]
   bw (  KiB/s): min= 1424, max= 1588, per=25.05%, avg=1511.69, stdev=51.34, samples=13
   iops        : min=  356, max=  397, avg=377.92, stdev=12.84, samples=13
  lat (msec)   : 10=0.04%, 50=99.49%, 100=0.47%
  cpu          : usr=0.66%, sys=3.75%, ctx=1434, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11823: Tue Nov  5 07:54:05 2024
  write: IOPS=377, BW=1510KiB/s (1546kB/s)(10.0MiB/6781msec)
    slat (usec): min=24, max=640, avg=87.22, stdev=53.52
    clat (usec): min=9273, max=86924, avg=42071.78, stdev=3951.30
     lat (usec): min=9369, max=87161, avg=42161.27, stdev=3955.41
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[35914], 10.00th=[37487], 20.00th=[39584],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[42730], 60.00th=[43779],
     | 70.00th=[44303], 80.00th=[44827], 90.00th=[45351], 95.00th=[45876],
     | 99.00th=[48497], 99.50th=[49021], 99.90th=[74974], 99.95th=[79168],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1440, max= 1560, per=25.04%, avg=1511.08, stdev=42.27, samples=13
   iops        : min=  360, max=  390, avg=377.77, stdev=10.57, samples=13
  lat (msec)   : 10=0.04%, 20=0.04%, 50=99.57%, 100=0.35%
  cpu          : usr=0.66%, sys=3.60%, ctx=1393, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6036KiB/s (6181kB/s), 1509KiB/s-1515KiB/s (1545kB/s-1552kB/s), io=40.0MiB (41.9MB), run=6757-6786msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=11838: Tue Nov  5 07:54:08 2024
   read: IOPS=467, BW=1872KiB/s (1917kB/s)(5048KiB/2697msec)
    slat (usec): min=15, max=3705, avg=1088.99, stdev=342.91
    clat (usec): min=2268, max=41773, avg=15851.64, stdev=4523.93
     lat (usec): min=4463, max=44061, avg=16942.46, stdev=4722.27
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[11600], 10.00th=[12256], 20.00th=[13173],
     | 30.00th=[13829], 40.00th=[14484], 50.00th=[15008], 60.00th=[15664],
     | 70.00th=[16319], 80.00th=[17171], 90.00th=[19006], 95.00th=[24773],
     | 99.00th=[35914], 99.50th=[37487], 99.90th=[39060], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1280, max= 2120, per=25.19%, avg=1859.20, stdev=330.94, samples=5
   iops        : min=  320, max=  530, avg=464.80, stdev=82.74, samples=5
  write: IOPS=481, BW=1925KiB/s (1971kB/s)(5192KiB/2697msec)
    slat (usec): min=12, max=230, avg=42.37, stdev=22.50
    clat (usec): min=3586, max=42682, avg=16628.35, stdev=4336.63
     lat (usec): min=3635, max=42718, avg=16671.80, stdev=4345.82
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13042], 10.00th=[13566], 20.00th=[14222],
     | 30.00th=[14877], 40.00th=[15270], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17433], 90.00th=[19006], 95.00th=[25822],
     | 99.00th=[36439], 99.50th=[39584], 99.90th=[42730], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1296, max= 2128, per=25.14%, avg=1931.20, stdev=356.50, samples=5
   iops        : min=  324, max=  532, avg=482.80, stdev=89.12, samples=5
  lat (msec)   : 4=0.16%, 10=0.47%, 20=91.09%, 50=8.28%
  cpu          : usr=1.22%, sys=7.23%, ctx=2211, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11839: Tue Nov  5 07:54:08 2024
   read: IOPS=458, BW=1834KiB/s (1878kB/s)(4976KiB/2713msec)
    slat (usec): min=628, max=3397, avg=1100.32, stdev=353.08
    clat (usec): min=9551, max=41198, avg=15960.13, stdev=4226.52
     lat (usec): min=10522, max=43557, avg=17062.27, stdev=4463.72
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[11863], 10.00th=[12518], 20.00th=[13304],
     | 30.00th=[13960], 40.00th=[14615], 50.00th=[15270], 60.00th=[15795],
     | 70.00th=[16450], 80.00th=[17171], 90.00th=[18744], 95.00th=[24249],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[40109], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 1112, max= 2096, per=24.61%, avg=1816.00, stdev=405.17, samples=5
   iops        : min=  278, max=  524, avg=454.00, stdev=101.29, samples=5
  write: IOPS=485, BW=1940KiB/s (1987kB/s)(5264KiB/2713msec)
    slat (usec): min=14, max=376, avg=43.11, stdev=25.80
    clat (usec): min=10430, max=40762, avg=16541.59, stdev=4214.52
     lat (usec): min=10477, max=40820, avg=16585.70, stdev=4222.92
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[12911], 10.00th=[13566], 20.00th=[14222],
     | 30.00th=[14877], 40.00th=[15270], 50.00th=[15664], 60.00th=[16057],
     | 70.00th=[16712], 80.00th=[17433], 90.00th=[19006], 95.00th=[23200],
     | 99.00th=[36963], 99.50th=[39584], 99.90th=[40109], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1368, max= 2136, per=25.32%, avg=1945.60, stdev=325.28, samples=5
   iops        : min=  342, max=  534, avg=486.40, stdev=81.32, samples=5
  lat (msec)   : 10=0.23%, 20=92.30%, 50=7.46%
  cpu          : usr=1.14%, sys=7.30%, ctx=2208, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11840: Tue Nov  5 07:54:08 2024
   read: IOPS=458, BW=1833KiB/s (1877kB/s)(4984KiB/2719msec)
    slat (usec): min=597, max=3152, avg=1102.70, stdev=342.79
    clat (usec): min=1211, max=39209, avg=15833.75, stdev=4102.10
     lat (usec): min=2137, max=41379, avg=16938.27, stdev=4332.00
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[11863], 10.00th=[12387], 20.00th=[13304],
     | 30.00th=[13960], 40.00th=[14615], 50.00th=[15139], 60.00th=[15664],
     | 70.00th=[16319], 80.00th=[17171], 90.00th=[18482], 95.00th=[22152],
     | 99.00th=[34866], 99.50th=[36963], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1224, max= 2008, per=24.65%, avg=1819.20, stdev=333.74, samples=5
   iops        : min=  306, max=  502, avg=454.80, stdev=83.43, samples=5
  write: IOPS=483, BW=1933KiB/s (1979kB/s)(5256KiB/2719msec)
    slat (usec): min=13, max=269, avg=42.01, stdev=23.67
    clat (usec): min=7236, max=44006, avg=16619.39, stdev=4334.70
     lat (usec): min=7268, max=44197, avg=16662.45, stdev=4342.60
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12911], 10.00th=[13566], 20.00th=[14353],
     | 30.00th=[14877], 40.00th=[15270], 50.00th=[15664], 60.00th=[16188],
     | 70.00th=[16581], 80.00th=[17433], 90.00th=[19006], 95.00th=[26346],
     | 99.00th=[36439], 99.50th=[37487], 99.90th=[43779], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 1168, max= 2232, per=25.01%, avg=1921.60, stdev=433.46, samples=5
   iops        : min=  292, max=  558, avg=480.40, stdev=108.36, samples=5
  lat (msec)   : 2=0.04%, 10=0.16%, 20=92.19%, 50=7.62%
  cpu          : usr=0.74%, sys=7.69%, ctx=2205, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11841: Tue Nov  5 07:54:08 2024
   read: IOPS=465, BW=1861KiB/s (1906kB/s)(5060KiB/2719msec)
    slat (usec): min=602, max=3063, avg=1092.90, stdev=324.45
    clat (usec): min=6095, max=42677, avg=15816.28, stdev=4180.94
     lat (usec): min=7058, max=45375, avg=16910.91, stdev=4395.66
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[11863], 10.00th=[12387], 20.00th=[13304],
     | 30.00th=[13829], 40.00th=[14484], 50.00th=[15139], 60.00th=[15664],
     | 70.00th=[16319], 80.00th=[16909], 90.00th=[18744], 95.00th=[22414],
     | 99.00th=[34866], 99.50th=[37487], 99.90th=[40633], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1168, max= 2104, per=25.04%, avg=1848.00, stdev=384.37, samples=5
   iops        : min=  292, max=  526, avg=462.00, stdev=96.09, samples=5
  write: IOPS=476, BW=1905KiB/s (1951kB/s)(5180KiB/2719msec)
    slat (usec): min=13, max=706, avg=42.64, stdev=29.23
    clat (usec): min=3309, max=42032, avg=16547.55, stdev=4375.06
     lat (usec): min=3335, max=42076, avg=16591.26, stdev=4384.71
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12911], 10.00th=[13435], 20.00th=[14222],
     | 30.00th=[14877], 40.00th=[15270], 50.00th=[15664], 60.00th=[16188],
     | 70.00th=[16581], 80.00th=[17433], 90.00th=[19006], 95.00th=[24773],
     | 99.00th=[36963], 99.50th=[39584], 99.90th=[40109], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1168, max= 2128, per=24.53%, avg=1884.80, stdev=403.29, samples=5
   iops        : min=  292, max=  532, avg=471.20, stdev=100.82, samples=5
  lat (msec)   : 4=0.04%, 10=0.43%, 20=91.64%, 50=7.89%
  cpu          : usr=1.03%, sys=7.51%, ctx=2219, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7381KiB/s (7558kB/s), 1833KiB/s-1872KiB/s (1877kB/s-1917kB/s), io=19.6MiB (20.5MB), run=2697-2719msec
  WRITE: bw=7684KiB/s (7868kB/s), 1905KiB/s-1940KiB/s (1951kB/s-1987kB/s), io=20.4MiB (21.4MB), run=2697-2719msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=11847: Tue Nov  5 07:54:10 2024
   read: IOPS=11, BW=471MiB/s (493MB/s)(40.0MiB/85msec)
    slat (nsec): min=5845.6k, max=5845.6k, avg=5845612.00, stdev= 0.00
    clat (nsec): min=74496k, max=74496k, avg=74496065.00, stdev= 0.00
     lat (nsec): min=80351k, max=80351k, avg=80351450.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[74974],  5.00th=[74974], 10.00th=[74974], 20.00th=[74974],
     | 30.00th=[74974], 40.00th=[74974], 50.00th=[74974], 60.00th=[74974],
     | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974],
     | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.14%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=471MiB/s (493MB/s), 471MiB/s-471MiB/s (493MB/s-493MB/s), io=40.0MiB (41.9MB), run=85-85msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout