== sanity test 398c: run fio to test AIO ================= 23:52:08 (1763355128)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.169234 s, 248 MB/s
osc.lustre-OST0000-osc-ffff9a8ca0171000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=184329: Sun Nov 16 23:52:19 2025
  write: IOPS=341, BW=1368KiB/s (1400kB/s)(10.0MiB/7488msec); 0 zone resets
    slat (usec): min=32, max=27386, avg=137.27, stdev=546.83
    clat (usec): min=12230, max=92862, avg=46330.28, stdev=9992.58
     lat (usec): min=16872, max=92933, avg=46469.00, stdev=9973.52
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[31851], 10.00th=[33424], 20.00th=[36963],
     | 30.00th=[40633], 40.00th=[43779], 50.00th=[46400], 60.00th=[47973],
     | 70.00th=[50594], 80.00th=[54264], 90.00th=[60556], 95.00th=[64750],
     | 99.00th=[70779], 99.50th=[74974], 99.90th=[83362], 99.95th=[88605],
     | 99.99th=[92799]
   bw (  KiB/s): min= 1208, max= 1760, per=25.02%, avg=1368.43, stdev=172.16, samples=14
   iops        : min=  302, max=  440, avg=342.07, stdev=43.00, samples=14
  lat (msec)   : 20=0.12%, 50=68.40%, 100=31.48%
  cpu          : usr=0.52%, sys=3.89%, ctx=2670, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184330: Sun Nov 16 23:52:19 2025
  write: IOPS=343, BW=1373KiB/s (1406kB/s)(10.0MiB/7458msec); 0 zone resets
    slat (usec): min=30, max=1002, avg=119.88, stdev=81.09
    clat (usec): min=8022, max=90204, avg=46420.57, stdev=10259.17
     lat (usec): min=8160, max=90284, avg=46541.69, stdev=10260.74
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[31589], 10.00th=[33424], 20.00th=[36963],
     | 30.00th=[40633], 40.00th=[43254], 50.00th=[45876], 60.00th=[48497],
     | 70.00th=[50594], 80.00th=[54264], 90.00th=[60556], 95.00th=[64750],
     | 99.00th=[71828], 99.50th=[74974], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[90702]
   bw (  KiB/s): min= 1176, max= 1784, per=25.16%, avg=1376.14, stdev=193.09, samples=14
   iops        : min=  294, max=  446, avg=344.00, stdev=48.21, samples=14
  lat (msec)   : 10=0.04%, 20=0.23%, 50=65.98%, 100=33.75%
  cpu          : usr=0.58%, sys=3.34%, ctx=2610, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184331: Sun Nov 16 23:52:19 2025
  write: IOPS=341, BW=1368KiB/s (1400kB/s)(10.0MiB/7488msec); 0 zone resets
    slat (usec): min=23, max=1387, avg=122.48, stdev=87.62
    clat (usec): min=6541, max=76230, avg=46265.93, stdev=10006.66
     lat (usec): min=6648, max=76375, avg=46389.57, stdev=10011.12
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31851], 10.00th=[33817], 20.00th=[36963],
     | 30.00th=[40633], 40.00th=[43779], 50.00th=[45876], 60.00th=[47973],
     | 70.00th=[50594], 80.00th=[54264], 90.00th=[60556], 95.00th=[64750],
     | 99.00th=[69731], 99.50th=[73925], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1176, max= 1800, per=24.99%, avg=1366.93, stdev=189.10, samples=14
   iops        : min=  294, max=  450, avg=341.71, stdev=47.26, samples=14
  lat (msec)   : 10=0.16%, 20=0.16%, 50=66.91%, 100=32.77%
  cpu          : usr=0.55%, sys=3.71%, ctx=2730, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184332: Sun Nov 16 23:52:19 2025
  write: IOPS=341, BW=1368KiB/s (1400kB/s)(10.0MiB/7488msec); 0 zone resets
    slat (usec): min=32, max=24980, avg=135.73, stdev=498.45
    clat (usec): min=12917, max=88800, avg=46334.56, stdev=9997.11
     lat (usec): min=15549, max=88937, avg=46471.55, stdev=9980.55
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[31589], 10.00th=[33817], 20.00th=[36963],
     | 30.00th=[40633], 40.00th=[43779], 50.00th=[46400], 60.00th=[47973],
     | 70.00th=[50594], 80.00th=[54264], 90.00th=[60556], 95.00th=[64750],
     | 99.00th=[70779], 99.50th=[74974], 99.90th=[88605], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min= 1200, max= 1760, per=25.02%, avg=1368.79, stdev=179.74, samples=14
   iops        : min=  300, max=  440, avg=342.14, stdev=44.87, samples=14
  lat (msec)   : 20=0.20%, 50=66.91%, 100=32.89%
  cpu          : usr=0.48%, sys=3.89%, ctx=2625, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5470KiB/s (5601kB/s), 1368KiB/s-1373KiB/s (1400kB/s-1406kB/s), io=40.0MiB (41.9MB), run=7458-7488msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=184342: Sun Nov 16 23:52:29 2025
  read: IOPS=189, BW=758KiB/s (776kB/s)(5048KiB/6658msec)
    slat (usec): min=35, max=7266, avg=2926.46, stdev=968.73
    clat (usec): min=3551, max=84999, avg=39692.74, stdev=8188.98
     lat (usec): min=5875, max=89138, avg=42621.93, stdev=8443.46
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28967], 10.00th=[31065], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[38536], 60.00th=[40109],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[49546], 95.00th=[55313],
     | 99.00th=[63177], 99.50th=[71828], 99.90th=[82314], 99.95th=[85459],
     | 99.99th=[85459]
   bw (  KiB/s): min=  600, max=  936, per=25.20%, avg=758.46, stdev=98.33, samples=13
   iops        : min=  150, max=  234, avg=189.62, stdev=24.58, samples=13
  write: IOPS=194, BW=780KiB/s (799kB/s)(5192KiB/6658msec); 0 zone resets
    slat (usec): min=30, max=880, avg=106.94, stdev=70.81
    clat (usec): min=18787, max=82339, avg=40221.75, stdev=7752.10
     lat (usec): min=18860, max=82421, avg=40329.91, stdev=7755.73
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30540], 10.00th=[32113], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[38536], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[50594], 95.00th=[55313],
     | 99.00th=[66323], 99.50th=[71828], 99.90th=[78119], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min=  648, max=  920, per=24.95%, avg=782.00, stdev=90.65, samples=13
   iops        : min=  162, max=  230, avg=195.46, stdev=22.70, samples=13
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.08%, 50=90.12%, 100=9.73%
  cpu          : usr=0.56%, sys=4.90%, ctx=4628, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184343: Sun Nov 16 23:52:29 2025
  read: IOPS=187, BW=748KiB/s (766kB/s)(4976KiB/6651msec)
    slat (usec): min=36, max=35447, avg=2994.69, stdev=1633.54
    clat (msec): min=12, max=109, avg=39.67, stdev= 9.33
     lat (msec): min=12, max=111, avg=42.67, stdev= 9.65
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   29], 10.00th=[   31], 20.00th=[   33],
     | 30.00th=[   35], 40.00th=[   37], 50.00th=[   39], 60.00th=[   40],
     | 70.00th=[   43], 80.00th=[   46], 90.00th=[   51], 95.00th=[   54],
     | 99.00th=[   77], 99.50th=[   92], 99.90th=[  105], 99.95th=[  109],
     | 99.99th=[  109]
   bw (  KiB/s): min=  528, max=  856, per=24.86%, avg=748.15, stdev=102.39, samples=13
   iops        : min=  132, max=  214, avg=187.00, stdev=25.64, samples=13
  write: IOPS=197, BW=791KiB/s (810kB/s)(5264KiB/6651msec); 0 zone resets
    slat (usec): min=27, max=1304, avg=110.75, stdev=88.97
    clat (msec): min=7, max=106, avg=40.26, stdev= 9.29
     lat (msec): min=7, max=106, avg=40.37, stdev= 9.29
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[   31], 10.00th=[   33], 20.00th=[   34],
     | 30.00th=[   36], 40.00th=[   37], 50.00th=[   39], 60.00th=[   41],
     | 70.00th=[   43], 80.00th=[   46], 90.00th=[   52], 95.00th=[   56],
     | 99.00th=[   70], 99.50th=[   89], 99.90th=[  106], 99.95th=[  107],
     | 99.99th=[  107]
   bw (  KiB/s): min=  616, max=  928, per=25.30%, avg=793.00, stdev=93.77, samples=13
   iops        : min=  154, max=  232, avg=198.23, stdev=23.46, samples=13
  lat (msec)   : 10=0.04%, 20=0.78%, 50=88.44%, 100=10.51%, 250=0.23%
  cpu          : usr=0.56%, sys=4.78%, ctx=4509, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184344: Sun Nov 16 23:52:29 2025
  read: IOPS=187, BW=748KiB/s (766kB/s)(4984KiB/6660msec)
    slat (usec): min=1267, max=9520, avg=2967.34, stdev=946.77
    clat (usec): min=4968, max=79894, avg=39598.80, stdev=8045.06
     lat (usec): min=6325, max=83403, avg=42569.10, stdev=8293.34
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28967], 10.00th=[30802], 20.00th=[33162],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[38536], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[45876], 90.00th=[50070], 95.00th=[53216],
     | 99.00th=[66323], 99.50th=[69731], 99.90th=[74974], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  656, max=  896, per=24.97%, avg=751.62, stdev=80.80, samples=13
   iops        : min=  164, max=  224, avg=187.85, stdev=20.27, samples=13
  write: IOPS=197, BW=789KiB/s (808kB/s)(5256KiB/6660msec); 0 zone resets
    slat (usec): min=27, max=1168, avg=113.73, stdev=83.53
    clat (usec): min=24419, max=77926, avg=40151.95, stdev=7464.36
     lat (usec): min=24668, max=78030, avg=40266.76, stdev=7463.45
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[30802], 10.00th=[32375], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[38536], 60.00th=[40633],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[50594], 95.00th=[55313],
     | 99.00th=[62653], 99.50th=[68682], 99.90th=[74974], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  608, max=  960, per=25.19%, avg=789.31, stdev=122.54, samples=13
   iops        : min=  152, max=  240, avg=197.31, stdev=30.65, samples=13
  lat (msec)   : 10=0.04%, 20=0.04%, 50=89.69%, 100=10.23%
  cpu          : usr=0.57%, sys=4.79%, ctx=4604, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184345: Sun Nov 16 23:52:29 2025
  read: IOPS=189, BW=759KiB/s (777kB/s)(5060KiB/6666msec)
    slat (usec): min=1212, max=10180, avg=2947.91, stdev=964.64
    clat (usec): min=18903, max=71317, avg=39375.14, stdev=7301.86
     lat (usec): min=21520, max=75429, avg=42325.78, stdev=7578.08
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29230], 10.00th=[30802], 20.00th=[33162],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[40109],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[48497], 95.00th=[52691],
     | 99.00th=[62653], 99.50th=[65799], 99.90th=[69731], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  656, max=  864, per=25.30%, avg=761.46, stdev=80.66, samples=13
   iops        : min=  164, max=  216, avg=190.31, stdev=20.25, samples=13
  write: IOPS=194, BW=777KiB/s (796kB/s)(5180KiB/6666msec); 0 zone resets
    slat (usec): min=32, max=1858, avg=110.55, stdev=84.01
    clat (usec): min=21527, max=75761, avg=40357.86, stdev=7687.39
     lat (usec): min=21585, max=75878, avg=40469.51, stdev=7689.00
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30540], 10.00th=[32375], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[38536], 60.00th=[40109],
     | 70.00th=[43254], 80.00th=[46400], 90.00th=[51643], 95.00th=[55313],
     | 99.00th=[63701], 99.50th=[66323], 99.90th=[73925], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  584, max=  920, per=24.70%, avg=774.15, stdev=108.13, samples=13
   iops        : min=  146, max=  230, avg=193.54, stdev=27.03, samples=13
  lat (msec)   : 20=0.04%, 50=90.23%, 100=9.73%
  cpu          : usr=0.47%, sys=4.94%, ctx=4607, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3011KiB/s (3083kB/s), 748KiB/s-759KiB/s (766kB/s-777kB/s), io=19.6MiB (20.5MB), run=6651-6666msec
  WRITE: bw=3134KiB/s (3209kB/s), 777KiB/s-791KiB/s (796kB/s-810kB/s), io=20.4MiB (21.4MB), run=6651-6666msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=184350: Sun Nov 16 23:52:33 2025
  read: IOPS=8, BW=328MiB/s (344MB/s)(40.0MiB/122msec)
    slat (nsec): min=12753k, max=12753k, avg=12752628.00, stdev= 0.00
    clat (nsec): min=101995k, max=101995k, avg=101994895.00, stdev= 0.00
     lat (nsec): min=114836k, max=114836k, avg=114835950.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  103],  5.00th=[  103], 10.00th=[  103], 20.00th=[  103],
     | 30.00th=[  103], 40.00th=[  103], 50.00th=[  103], 60.00th=[  103],
     | 70.00th=[  103], 80.00th=[  103], 90.00th=[  103], 95.00th=[  103],
     | 99.00th=[  103], 99.50th=[  103], 99.90th=[  103], 99.95th=[  103],
     | 99.99th=[  103]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=8.20%, ctx=10, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=40.0MiB (41.9MB), run=122-122msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout