== sanity test 398c: run fio to test AIO ================= 03:16:02 (1769760962)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.117235 s, 358 MB/s
osc.lustre-OST0000-osc-ffff8e6cb88d9000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=181985: Fri Jan 30 03:16:10 2026
  write: IOPS=528, BW=2116KiB/s (2166kB/s)(10.0MiB/4840msec); 0 zone resets
    slat (usec): min=50, max=1004, avg=174.07, stdev=95.53
    clat (usec): min=4403, max=46516, avg=29763.64, stdev=3080.53
     lat (usec): min=4543, max=46626, avg=29938.68, stdev=3083.62
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[25297], 10.00th=[26608], 20.00th=[27657],
     | 30.00th=[28705], 40.00th=[29230], 50.00th=[30016], 60.00th=[30540],
     | 70.00th=[31327], 80.00th=[31851], 90.00th=[33162], 95.00th=[34341],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min= 1992, max= 2196, per=24.94%, avg=2110.67, stdev=62.61, samples=9
   iops        : min=  498, max=  549, avg=527.67, stdev=15.65, samples=9
  lat (msec)   : 10=0.08%, 20=1.48%, 50=98.44%
  cpu          : usr=0.62%, sys=7.42%, ctx=2631, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181986: Fri Jan 30 03:16:10 2026
  write: IOPS=529, BW=2119KiB/s (2170kB/s)(10.0MiB/4833msec); 0 zone resets
    slat (usec): min=40, max=23406, avg=189.44, stdev=469.92
    clat (usec): min=4431, max=50724, avg=29738.32, stdev=3419.61
     lat (usec): min=4621, max=50884, avg=29928.83, stdev=3403.01
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[24773], 10.00th=[26870], 20.00th=[27657],
     | 30.00th=[28443], 40.00th=[29230], 50.00th=[30016], 60.00th=[30540],
     | 70.00th=[31065], 80.00th=[32113], 90.00th=[33162], 95.00th=[34341],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[47449], 99.95th=[47973],
     | 99.99th=[50594]
   bw (  KiB/s): min= 1992, max= 2253, per=25.02%, avg=2117.00, stdev=76.69, samples=9
   iops        : min=  498, max=  563, avg=529.22, stdev=19.12, samples=9
  lat (msec)   : 10=0.16%, 20=1.48%, 50=98.32%, 100=0.04%
  cpu          : usr=0.46%, sys=7.88%, ctx=2670, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181987: Fri Jan 30 03:16:10 2026
  write: IOPS=528, BW=2116KiB/s (2166kB/s)(10.0MiB/4840msec); 0 zone resets
    slat (usec): min=36, max=1199, avg=175.16, stdev=102.79
    clat (usec): min=2527, max=42966, avg=29734.03, stdev=3002.11
     lat (usec): min=2939, max=43291, avg=29910.73, stdev=3009.73
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[24773], 10.00th=[26608], 20.00th=[27657],
     | 30.00th=[28443], 40.00th=[29230], 50.00th=[30016], 60.00th=[30540],
     | 70.00th=[31065], 80.00th=[31851], 90.00th=[33162], 95.00th=[34341],
     | 99.00th=[35914], 99.50th=[35914], 99.90th=[36963], 99.95th=[40109],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1944, max= 2168, per=24.89%, avg=2106.44, stdev=74.01, samples=9
   iops        : min=  486, max=  542, avg=526.56, stdev=18.46, samples=9
  lat (msec)   : 4=0.04%, 20=1.33%, 50=98.63%
  cpu          : usr=0.68%, sys=6.90%, ctx=2562, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181988: Fri Jan 30 03:16:10 2026
  write: IOPS=532, BW=2131KiB/s (2182kB/s)(10.0MiB/4805msec); 0 zone resets
    slat (usec): min=37, max=1769, avg=186.26, stdev=107.78
    clat (usec): min=5254, max=52080, avg=29790.00, stdev=3327.23
     lat (usec): min=5381, max=52230, avg=29977.32, stdev=3332.12
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[24773], 10.00th=[26870], 20.00th=[27657],
     | 30.00th=[28705], 40.00th=[29230], 50.00th=[30016], 60.00th=[30540],
     | 70.00th=[31327], 80.00th=[32113], 90.00th=[33424], 95.00th=[34341],
     | 99.00th=[36439], 99.50th=[39060], 99.90th=[44303], 99.95th=[44303],
     | 99.99th=[52167]
   bw (  KiB/s): min= 1976, max= 2323, per=25.11%, avg=2124.78, stdev=98.66, samples=9
   iops        : min=  494, max=  580, avg=531.11, stdev=24.48, samples=9
  lat (msec)   : 10=0.08%, 20=1.41%, 50=98.48%, 100=0.04%
  cpu          : usr=0.73%, sys=7.33%, ctx=2701, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8463KiB/s (8666kB/s), 2116KiB/s-2131KiB/s (2166kB/s-2182kB/s), io=40.0MiB (41.9MB), run=4805-4840msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=181998: Fri Jan 30 03:16:18 2026
  read: IOPS=247, BW=991KiB/s (1015kB/s)(5048KiB/5093msec)
    slat (usec): min=88, max=6057, avg=2326.30, stdev=677.23
    clat (usec): min=4672, max=50799, avg=30385.09, stdev=4207.37
     lat (usec): min=4772, max=53276, avg=32712.80, stdev=4275.67
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[23987], 10.00th=[25297], 20.00th=[26870],
     | 30.00th=[28443], 40.00th=[29754], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[32375], 80.00th=[33817], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[39584], 99.50th=[41681], 99.90th=[47973], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min=  856, max= 1088, per=25.10%, avg=984.33, stdev=67.23, samples=9
   iops        : min=  214, max=  272, avg=246.00, stdev=16.79, samples=9
  write: IOPS=254, BW=1019KiB/s (1044kB/s)(5192KiB/5093msec); 0 zone resets
    slat (usec): min=51, max=919, avg=144.87, stdev=72.00
    clat (usec): min=10702, max=47756, avg=30560.64, stdev=3336.68
     lat (usec): min=10893, max=47862, avg=30706.43, stdev=3335.77
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[25297], 10.00th=[26608], 20.00th=[27919],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31327],
     | 70.00th=[32113], 80.00th=[33424], 90.00th=[34341], 95.00th=[35914],
     | 99.00th=[38011], 99.50th=[40109], 99.90th=[45351], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min=  952, max= 1168, per=25.13%, avg=1025.89, stdev=66.69, samples=9
   iops        : min=  238, max=  292, avg=256.33, stdev=16.78, samples=9
  lat (msec)   : 10=0.16%, 20=0.39%, 50=99.41%, 100=0.04%
  cpu          : usr=0.57%, sys=6.42%, ctx=4715, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181999: Fri Jan 30 03:16:18 2026
  read: IOPS=244, BW=978KiB/s (1001kB/s)(4976KiB/5090msec)
    slat (usec): min=36, max=6522, avg=2316.91, stdev=664.08
    clat (usec): min=7675, max=49031, avg=30536.14, stdev=3900.67
     lat (usec): min=10796, max=51702, avg=32854.48, stdev=3914.15
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[24249], 10.00th=[25560], 20.00th=[27395],
     | 30.00th=[28705], 40.00th=[29754], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[32375], 80.00th=[33817], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[40109], 99.50th=[41157], 99.90th=[46400], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min=  856, max= 1072, per=24.83%, avg=973.78, stdev=72.79, samples=9
   iops        : min=  214, max=  268, avg=243.33, stdev=18.28, samples=9
  write: IOPS=258, BW=1034KiB/s (1059kB/s)(5264KiB/5090msec); 0 zone resets
    slat (usec): min=48, max=596, avg=142.62, stdev=72.55
    clat (usec): min=8966, max=58679, avg=30561.38, stdev=3475.64
     lat (usec): min=9061, max=58783, avg=30704.96, stdev=3476.99
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[25297], 10.00th=[26608], 20.00th=[28181],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31327],
     | 70.00th=[32113], 80.00th=[33162], 90.00th=[34341], 95.00th=[35914],
     | 99.00th=[38536], 99.50th=[39584], 99.90th=[51643], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  952, max= 1136, per=25.48%, avg=1040.00, stdev=69.49, samples=9
   iops        : min=  238, max=  284, avg=259.89, stdev=17.41, samples=9
  lat (msec)   : 10=0.12%, 20=0.35%, 50=99.45%, 100=0.08%
  cpu          : usr=0.39%, sys=6.48%, ctx=4684, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182000: Fri Jan 30 03:16:18 2026
  read: IOPS=243, BW=975KiB/s (998kB/s)(4984KiB/5112msec)
    slat (usec): min=962, max=5354, avg=2347.86, stdev=635.80
    clat (usec): min=3683, max=40549, avg=30522.12, stdev=3955.22
     lat (usec): min=5810, max=43346, avg=32871.22, stdev=3958.76
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[23987], 10.00th=[25297], 20.00th=[26870],
     | 30.00th=[28443], 40.00th=[29492], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[32637], 80.00th=[33817], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[39060], 99.50th=[39584], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min=  922, max= 1040, per=24.85%, avg=974.22, stdev=35.06, samples=9
   iops        : min=  230, max=  260, avg=243.44, stdev= 8.85, samples=9
  write: IOPS=257, BW=1028KiB/s (1053kB/s)(5256KiB/5112msec); 0 zone resets
    slat (usec): min=51, max=1840, avg=147.59, stdev=89.19
    clat (usec): min=7956, max=40265, avg=30426.25, stdev=3137.22
     lat (usec): min=8063, max=40414, avg=30574.85, stdev=3140.45
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25822], 10.00th=[26608], 20.00th=[27919],
     | 30.00th=[28967], 40.00th=[29492], 50.00th=[30278], 60.00th=[31065],
     | 70.00th=[31851], 80.00th=[33162], 90.00th=[34341], 95.00th=[35390],
     | 99.00th=[38011], 99.50th=[39584], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min=  877, max= 1144, per=25.07%, avg=1023.33, stdev=95.54, samples=9
   iops        : min=  219, max=  286, avg=255.78, stdev=23.93, samples=9
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.27%, 50=99.65%
  cpu          : usr=0.35%, sys=6.57%, ctx=4709, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182001: Fri Jan 30 03:16:18 2026
  read: IOPS=247, BW=989KiB/s (1013kB/s)(5060KiB/5117msec)
    slat (usec): min=943, max=13256, avg=2328.20, stdev=714.80
    clat (usec): min=9561, max=43256, avg=30474.54, stdev=3987.54
     lat (usec): min=11503, max=45381, avg=32803.95, stdev=4009.03
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[23987], 10.00th=[25560], 20.00th=[27132],
     | 30.00th=[28443], 40.00th=[29754], 50.00th=[30802], 60.00th=[31589],
     | 70.00th=[32637], 80.00th=[33817], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[39060], 99.50th=[41157], 99.90th=[41681], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  912, max= 1056, per=25.20%, avg=988.11, stdev=61.20, samples=9
   iops        : min=  228, max=  264, avg=246.89, stdev=15.47, samples=9
  write: IOPS=253, BW=1012KiB/s (1037kB/s)(5180KiB/5117msec); 0 zone resets
    slat (usec): min=40, max=648, avg=145.18, stdev=77.72
    clat (usec): min=11507, max=41223, avg=30569.15, stdev=3169.11
     lat (usec): min=11567, max=41699, avg=30715.23, stdev=3164.50
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25560], 10.00th=[26870], 20.00th=[28181],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31327],
     | 70.00th=[32113], 80.00th=[33162], 90.00th=[34341], 95.00th=[35390],
     | 99.00th=[38011], 99.50th=[39584], 99.90th=[41157], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min=  900, max= 1093, per=24.59%, avg=1003.67, stdev=57.58, samples=9
   iops        : min=  225, max=  273, avg=250.89, stdev=14.35, samples=9
  lat (msec)   : 10=0.04%, 20=0.55%, 50=99.41%
  cpu          : usr=0.47%, sys=6.43%, ctx=4669, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3922KiB/s (4016kB/s), 975KiB/s-991KiB/s (998kB/s-1015kB/s), io=19.6MiB (20.5MB), run=5090-5117msec
  WRITE: bw=4083KiB/s (4181kB/s), 1012KiB/s-1034KiB/s (1037kB/s-1059kB/s), io=20.4MiB (21.4MB), run=5090-5117msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182006: Fri Jan 30 03:16:21 2026
  read: IOPS=7, BW=313MiB/s (328MB/s)(40.0MiB/128msec)
    slat (nsec): min=9864.6k, max=9864.6k, avg=9864633.00, stdev= 0.00
    clat (nsec): min=112712k, max=112712k, avg=112712385.00, stdev= 0.00
     lat (nsec): min=122581k, max=122581k, avg=122580838.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  113],  5.00th=[  113], 10.00th=[  113], 20.00th=[  113],
     | 30.00th=[  113], 40.00th=[  113], 50.00th=[  113], 60.00th=[  113],
     | 70.00th=[  113], 80.00th=[  113], 90.00th=[  113], 95.00th=[  113],
     | 99.00th=[  113], 99.50th=[  113], 99.90th=[  113], 99.95th=[  113],
     | 99.99th=[  113]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.09%, ctx=6, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=313MiB/s (328MB/s), 313MiB/s-313MiB/s (328MB/s-328MB/s), io=40.0MiB (41.9MB), run=128-128msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout