== sanity test 398c: run fio to test AIO ================= 23:41:07 (1761104467)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.157305 s, 267 MB/s
osc.lustre-OST0000-osc-ffff8c330ccd6000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=195857: Tue Oct 21 23:41:15 2025
  write: IOPS=469, BW=1880KiB/s (1925kB/s)(10.0MiB/5447msec); 0 zone resets
    slat (usec): min=23, max=718, avg=109.42, stdev=67.13
    clat (usec): min=3925, max=85268, avg=33790.94, stdev=8067.26
     lat (usec): min=4094, max=85399, avg=33901.75, stdev=8087.14
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[25035], 10.00th=[26870], 20.00th=[28705],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[32113], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[37487], 90.00th=[44303], 95.00th=[49546],
     | 99.00th=[61604], 99.50th=[76022], 99.90th=[85459], 99.95th=[85459],
     | 99.99th=[85459]
   bw (  KiB/s): min= 1520, max= 2104, per=25.24%, avg=1885.50, stdev=175.14, samples=10
   iops        : min=  380, max=  526, avg=471.30, stdev=43.75, samples=10
  lat (msec)   : 4=0.04%, 10=0.16%, 20=0.35%, 50=94.73%, 100=4.73%
  cpu          : usr=0.55%, sys=3.76%, ctx=2808, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195858: Tue Oct 21 23:41:15 2025
  write: IOPS=469, BW=1880KiB/s (1925kB/s)(10.0MiB/5447msec); 0 zone resets
    slat (usec): min=23, max=775, avg=110.34, stdev=70.98
    clat (usec): min=4254, max=84932, avg=33875.81, stdev=8049.57
     lat (usec): min=4338, max=85055, avg=33987.41, stdev=8068.23
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[25297], 10.00th=[26870], 20.00th=[28705],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[32113], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[38011], 90.00th=[44303], 95.00th=[49546],
     | 99.00th=[63701], 99.50th=[65799], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1552, max= 2128, per=25.26%, avg=1886.70, stdev=168.15, samples=10
   iops        : min=  388, max=  532, avg=471.60, stdev=42.00, samples=10
  lat (msec)   : 10=0.20%, 20=0.43%, 50=94.80%, 100=4.57%
  cpu          : usr=0.37%, sys=3.87%, ctx=2975, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195859: Tue Oct 21 23:41:15 2025
  write: IOPS=466, BW=1867KiB/s (1912kB/s)(10.0MiB/5484msec); 0 zone resets
    slat (usec): min=30, max=775, avg=108.97, stdev=67.05
    clat (usec): min=3489, max=84952, avg=33776.52, stdev=7821.60
     lat (usec): min=3605, max=85118, avg=33886.59, stdev=7844.77
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[25560], 10.00th=[26870], 20.00th=[28705],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[32113], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[37487], 90.00th=[43779], 95.00th=[49546],
     | 99.00th=[61080], 99.50th=[72877], 99.90th=[82314], 99.95th=[84411],
     | 99.99th=[85459]
   bw (  KiB/s): min= 1568, max= 2088, per=25.03%, avg=1869.70, stdev=167.17, samples=10
   iops        : min=  392, max=  522, avg=467.40, stdev=41.80, samples=10
  lat (msec)   : 4=0.08%, 10=0.16%, 20=0.27%, 50=94.69%, 100=4.80%
  cpu          : usr=0.42%, sys=4.29%, ctx=2945, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195860: Tue Oct 21 23:41:15 2025
  write: IOPS=467, BW=1868KiB/s (1913kB/s)(10.0MiB/5481msec); 0 zone resets
    slat (usec): min=26, max=28176, avg=121.97, stdev=558.42
    clat (usec): min=6874, max=84773, avg=33782.74, stdev=7761.34
     lat (usec): min=6965, max=85094, avg=33905.99, stdev=7764.38
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25297], 10.00th=[26870], 20.00th=[28705],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[37487], 90.00th=[43779], 95.00th=[49546],
     | 99.00th=[61080], 99.50th=[65799], 99.90th=[80217], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1568, max= 2104, per=25.10%, avg=1874.60, stdev=159.44, samples=10
   iops        : min=  392, max=  526, avg=468.60, stdev=39.88, samples=10
  lat (msec)   : 10=0.08%, 20=0.27%, 50=95.00%, 100=4.65%
  cpu          : usr=0.38%, sys=4.38%, ctx=2972, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7469KiB/s (7648kB/s), 1867KiB/s-1880KiB/s (1912kB/s-1925kB/s), io=40.0MiB (41.9MB), run=5447-5484msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=195870: Tue Oct 21 23:41:24 2025
  read: IOPS=234, BW=939KiB/s (962kB/s)(5048KiB/5374msec)
    slat (usec): min=1006, max=6889, avg=2555.45, stdev=775.92
    clat (usec): min=6228, max=53282, avg=31840.23, stdev=4555.71
     lat (usec): min=8587, max=54864, avg=34396.99, stdev=4542.65
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[24773], 10.00th=[26346], 20.00th=[28181],
     | 30.00th=[29492], 40.00th=[30540], 50.00th=[31589], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[35390], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[43254], 99.50th=[45876], 99.90th=[52167], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  816, max= 1008, per=25.13%, avg=938.20, stdev=58.23, samples=10
   iops        : min=  204, max=  252, avg=234.50, stdev=14.60, samples=10
  write: IOPS=241, BW=966KiB/s (989kB/s)(5192KiB/5374msec); 0 zone resets
    slat (usec): min=36, max=365, avg=88.68, stdev=43.96
    clat (usec): min=6213, max=50170, avg=32078.58, stdev=3713.51
     lat (usec): min=6288, max=50247, avg=32168.11, stdev=3712.76
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[26346], 10.00th=[27657], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[41681], 99.50th=[43254], 99.90th=[47973], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min=  837, max= 1136, per=24.98%, avg=970.90, stdev=85.12, samples=10
   iops        : min=  209, max=  284, avg=242.70, stdev=21.32, samples=10
  lat (msec)   : 10=0.08%, 20=0.08%, 50=99.73%, 100=0.12%
  cpu          : usr=0.43%, sys=4.56%, ctx=4742, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195871: Tue Oct 21 23:41:24 2025
  read: IOPS=233, BW=935KiB/s (957kB/s)(4976KiB/5324msec)
    slat (usec): min=42, max=5885, avg=2574.91, stdev=802.43
    clat (usec): min=19216, max=55191, avg=31663.11, stdev=4386.04
     lat (usec): min=22073, max=58654, avg=34239.45, stdev=4422.95
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[24773], 10.00th=[26346], 20.00th=[27919],
     | 30.00th=[29230], 40.00th=[30540], 50.00th=[31589], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[42730], 99.50th=[47973], 99.90th=[51119], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  848, max= 1008, per=25.05%, avg=935.20, stdev=57.87, samples=10
   iops        : min=  212, max=  252, avg=233.80, stdev=14.47, samples=10
  write: IOPS=247, BW=989KiB/s (1012kB/s)(5264KiB/5324msec); 0 zone resets
    slat (usec): min=29, max=1153, avg=89.82, stdev=55.20
    clat (usec): min=3962, max=53019, avg=32140.43, stdev=3969.34
     lat (usec): min=4283, max=53086, avg=32231.15, stdev=3971.04
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26346], 10.00th=[27657], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31327], 50.00th=[32113], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[41681], 99.50th=[44303], 99.90th=[50070], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  872, max= 1080, per=25.49%, avg=990.70, stdev=67.34, samples=10
   iops        : min=  218, max=  270, avg=247.60, stdev=16.78, samples=10
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.20%, 50=99.53%, 100=0.16%
  cpu          : usr=0.47%, sys=4.57%, ctx=4746, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195872: Tue Oct 21 23:41:24 2025
  read: IOPS=231, BW=927KiB/s (950kB/s)(4984KiB/5375msec)
    slat (usec): min=1050, max=31612, avg=2622.86, stdev=1254.98
    clat (usec): min=2094, max=80746, avg=31951.33, stdev=5628.69
     lat (usec): min=3796, max=82835, avg=34575.50, stdev=5738.67
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[24773], 10.00th=[26084], 20.00th=[27919],
     | 30.00th=[29492], 40.00th=[30802], 50.00th=[31589], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[35390], 90.00th=[37487], 95.00th=[39584],
     | 99.00th=[46400], 99.50th=[64750], 99.90th=[74974], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min=  840, max= 1008, per=24.75%, avg=924.00, stdev=60.01, samples=10
   iops        : min=  210, max=  252, avg=231.00, stdev=15.00, samples=10
  write: IOPS=244, BW=978KiB/s (1001kB/s)(5256KiB/5375msec); 0 zone resets
    slat (usec): min=29, max=405, avg=87.86, stdev=45.85
    clat (usec): min=4888, max=76469, avg=32219.75, stdev=5068.26
     lat (usec): min=5063, max=76549, avg=32308.52, stdev=5067.33
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[26608], 10.00th=[27657], 20.00th=[29230],
     | 30.00th=[30016], 40.00th=[31065], 50.00th=[31851], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[45876], 99.50th=[69731], 99.90th=[73925], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  864, max= 1120, per=24.92%, avg=968.50, stdev=90.91, samples=10
   iops        : min=  216, max=  280, avg=242.10, stdev=22.76, samples=10
  lat (msec)   : 4=0.08%, 10=0.16%, 20=0.51%, 50=98.55%, 100=0.70%
  cpu          : usr=0.52%, sys=4.41%, ctx=4646, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195873: Tue Oct 21 23:41:24 2025
  read: IOPS=235, BW=943KiB/s (966kB/s)(5060KiB/5364msec)
    slat (usec): min=1028, max=9839, avg=2536.00, stdev=785.07
    clat (usec): min=18620, max=45769, avg=31665.49, stdev=4336.31
     lat (usec): min=21257, max=49222, avg=34202.80, stdev=4361.55
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[24773], 10.00th=[26084], 20.00th=[27919],
     | 30.00th=[29230], 40.00th=[30540], 50.00th=[31589], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[35390], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[43254], 99.50th=[44303], 99.90th=[45351], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min=  872, max= 1024, per=25.18%, avg=939.80, stdev=42.61, samples=10
   iops        : min=  218, max=  256, avg=234.90, stdev=10.71, samples=10
  write: IOPS=241, BW=966KiB/s (989kB/s)(5180KiB/5364msec); 0 zone resets
    slat (usec): min=24, max=528, avg=88.38, stdev=44.49
    clat (usec): min=21872, max=50100, avg=32309.63, stdev=3663.77
     lat (usec): min=22046, max=50195, avg=32398.93, stdev=3662.95
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[26608], 10.00th=[27919], 20.00th=[29230],
     | 30.00th=[30540], 40.00th=[31327], 50.00th=[32113], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[42206], 99.50th=[43779], 99.90th=[50070], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min=  869, max= 1024, per=24.94%, avg=969.30, stdev=53.26, samples=10
   iops        : min=  217, max=  256, avg=242.30, stdev=13.37, samples=10
  lat (msec)   : 20=0.04%, 50=99.88%, 100=0.08%
  cpu          : usr=0.48%, sys=4.55%, ctx=4709, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3734KiB/s (3823kB/s), 927KiB/s-943KiB/s (950kB/s-966kB/s), io=19.6MiB (20.5MB), run=5324-5375msec
  WRITE: bw=3887KiB/s (3980kB/s), 966KiB/s-989KiB/s (989kB/s-1012kB/s), io=20.4MiB (21.4MB), run=5324-5375msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=195877: Tue Oct 21 23:41:27 2025
  read: IOPS=9, BW=367MiB/s (385MB/s)(40.0MiB/109msec)
    slat (nsec): min=8837.0k, max=8837.0k, avg=8837972.00, stdev= 0.00
    clat (nsec): min=94300k, max=94300k, avg=94300273.00, stdev= 0.00
     lat (nsec): min=103141k, max=103141k, avg=103140882.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[93848],  5.00th=[93848], 10.00th=[93848], 20.00th=[93848],
     | 30.00th=[93848], 40.00th=[93848], 50.00th=[93848], 60.00th=[93848],
     | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848],
     | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848],
     | 99.99th=[93848]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.41%, ctx=9, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=367MiB/s (385MB/s), 367MiB/s-367MiB/s (385MB/s-385MB/s), io=40.0MiB (41.9MB), run=109-109msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout