== sanity test 398c: run fio to test AIO ================= 03:49:49 (1785397789) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.459852 s, 91.2 MB/s osc.lustre-OST0000-osc-ffff889e8cc6e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179563: Thu Jul 30 03:50:41 2026 write: IOPS=64, BW=259KiB/s (266kB/s)(10.0MiB/39466msec); 0 zone resets slat (usec): min=49, max=24116, avg=256.36, stdev=882.15 clat (msec): min=23, max=466, avg=245.64, stdev=56.04 lat (msec): min=23, max=466, avg=245.90, stdev=56.04 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 153], 10.00th=[ 178], 20.00th=[ 201], | 30.00th=[ 218], 40.00th=[ 232], 50.00th=[ 245], 60.00th=[ 257], | 70.00th=[ 271], 80.00th=[ 292], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 393], 99.50th=[ 422], 99.90th=[ 456], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 142, max= 479, per=24.95%, avg=257.99, stdev=52.99, samples=78 iops : min= 35, max= 119, avg=64.08, stdev=13.27, samples=78 lat (msec) : 50=0.20%, 100=0.12%, 250=54.14%, 500=45.55% cpu : usr=0.12%, sys=0.76%, ctx=2256, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179564: Thu Jul 30 03:50:41 2026 write: IOPS=65, BW=260KiB/s (267kB/s)(10.0MiB/39329msec); 0 zone resets slat (usec): min=50, max=18852, avg=301.03, stdev=1046.61 clat (msec): min=20, max=508, avg=245.28, stdev=58.02 lat (msec): min=20, max=509, avg=245.59, stdev=58.03 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 157], 10.00th=[ 180], 20.00th=[ 201], | 30.00th=[ 218], 40.00th=[ 232], 50.00th=[ 247], 60.00th=[ 259], | 70.00th=[ 271], 80.00th=[ 288], 90.00th=[ 317], 95.00th=[ 338], | 99.00th=[ 397], 99.50th=[ 430], 99.90th=[ 506], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 135, max= 471, per=24.99%, avg=258.36, stdev=54.32, samples=78 iops : min= 33, max= 117, avg=64.18, stdev=13.62, samples=78 lat (msec) : 50=0.12%, 100=1.33%, 250=52.27%, 500=46.17%, 750=0.12% cpu : usr=0.13%, sys=0.89%, ctx=2887, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179565: Thu Jul 30 03:50:41 2026 write: IOPS=64, BW=259KiB/s (265kB/s)(10.0MiB/39547msec); 0 zone resets slat (usec): min=48, max=15752, avg=198.43, stdev=666.05 clat (msec): min=38, max=399, avg=245.97, stdev=54.60 lat (msec): min=38, max=399, avg=246.17, stdev=54.64 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 161], 10.00th=[ 180], 20.00th=[ 199], | 30.00th=[ 220], 40.00th=[ 234], 50.00th=[ 245], 60.00th=[ 257], | 70.00th=[ 271], 80.00th=[ 288], 90.00th=[ 321], 95.00th=[ 347], | 99.00th=[ 388], 99.50th=[ 393], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 127, max= 439, per=24.88%, avg=257.22, stdev=57.62, samples=78 iops : min= 31, max= 109, avg=63.90, stdev=14.41, samples=78 lat (msec) : 50=0.08%, 250=53.24%, 500=46.68% cpu : usr=0.10%, sys=0.64%, ctx=1312, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179566: Thu Jul 30 03:50:41 2026 write: IOPS=64, BW=259KiB/s (265kB/s)(10.0MiB/39587msec); 0 zone resets slat (usec): min=48, max=11151, avg=217.04, stdev=664.15 clat (msec): min=69, max=439, avg=246.15, stdev=55.45 lat (msec): min=69, max=439, avg=246.37, stdev=55.45 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 153], 10.00th=[ 180], 20.00th=[ 201], | 30.00th=[ 218], 40.00th=[ 232], 50.00th=[ 245], 60.00th=[ 259], | 70.00th=[ 271], 80.00th=[ 292], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 388], 99.50th=[ 393], 99.90th=[ 401], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 126, max= 510, per=24.75%, avg=255.91, stdev=70.67, samples=78 iops : min= 31, max= 127, avg=63.59, stdev=17.69, samples=78 lat (msec) : 100=0.20%, 250=53.24%, 500=46.56% cpu : usr=0.09%, sys=0.69%, ctx=1707, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1035KiB/s (1060kB/s), 259KiB/s-260KiB/s (265kB/s-267kB/s), io=40.0MiB (41.9MB), run=39329-39587msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179582: Thu Jul 30 03:51:43 2026 read: IOPS=26, BW=106KiB/s (108kB/s)(5048KiB/47662msec) slat (usec): min=2331, max=84955, avg=22931.68, stdev=13497.92 clat (msec): min=65, max=629, avg=279.56, stdev=66.62 lat (msec): min=73, max=699, avg=302.50, stdev=68.96 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 174], 10.00th=[ 199], 20.00th=[ 228], | 30.00th=[ 247], 40.00th=[ 262], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 363], 95.00th=[ 388], | 99.00th=[ 477], 99.50th=[ 498], 99.90th=[ 592], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 23, max= 207, per=25.13%, avg=105.53, stdev=26.92, samples=94 iops : min= 5, max= 51, avg=25.93, stdev= 6.79, samples=94 write: IOPS=27, BW=109KiB/s (112kB/s)(5192KiB/47662msec); 0 zone resets slat (usec): min=52, max=12387, avg=298.75, stdev=926.55 clat (msec): min=31, max=593, avg=288.64, stdev=61.09 lat (msec): min=31, max=593, avg=288.94, stdev=61.11 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 194], 10.00th=[ 218], 20.00th=[ 241], | 30.00th=[ 259], 40.00th=[ 271], 50.00th=[ 284], 60.00th=[ 300], | 70.00th=[ 317], 80.00th=[ 338], 90.00th=[ 372], 95.00th=[ 393], | 99.00th=[ 443], 99.50th=[ 460], 99.90th=[ 567], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 7, max= 183, per=24.88%, avg=108.72, stdev=31.07, samples=94 iops : min= 1, max= 45, avg=26.72, stdev= 7.77, samples=94 lat (msec) : 50=0.04%, 100=0.04%, 250=28.75%, 500=70.78%, 750=0.39% cpu : usr=0.17%, sys=1.26%, ctx=3823, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179583: Thu Jul 30 03:51:43 2026 read: IOPS=26, BW=105KiB/s (107kB/s)(4976KiB/47550msec) slat (usec): min=92, max=85644, avg=22919.43, stdev=13552.02 clat (msec): min=13, max=791, avg=284.20, stdev=71.93 lat (msec): min=29, max=797, avg=307.13, stdev=74.16 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 174], 10.00th=[ 201], 20.00th=[ 230], | 30.00th=[ 249], 40.00th=[ 266], 50.00th=[ 279], 60.00th=[ 296], | 70.00th=[ 317], 80.00th=[ 342], 90.00th=[ 372], 95.00th=[ 397], | 99.00th=[ 447], 99.50th=[ 514], 99.90th=[ 718], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 39, max= 192, per=24.78%, avg=104.06, stdev=28.67, samples=94 iops : min= 9, max= 48, avg=25.56, stdev= 7.22, samples=94 write: IOPS=27, BW=111KiB/s (113kB/s)(5264KiB/47550msec); 0 zone resets slat (usec): min=51, max=33752, avg=349.30, stdev=1386.72 clat (msec): min=42, max=748, avg=286.33, stdev=67.69 lat (msec): min=42, max=754, avg=286.68, stdev=67.71 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 186], 10.00th=[ 207], 20.00th=[ 234], | 30.00th=[ 251], 40.00th=[ 266], 50.00th=[ 284], 60.00th=[ 300], | 70.00th=[ 317], 80.00th=[ 338], 90.00th=[ 372], 95.00th=[ 401], | 99.00th=[ 456], 99.50th=[ 498], 99.90th=[ 693], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 39, max= 215, per=25.29%, avg=110.52, stdev=33.22, samples=94 iops : min= 9, max= 53, avg=27.17, stdev= 8.34, samples=94 lat (msec) : 20=0.04%, 50=0.12%, 100=0.08%, 250=29.49%, 500=69.77% lat (msec) : 750=0.47%, 1000=0.04% cpu : usr=0.17%, sys=1.16%, ctx=3556, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179584: Thu Jul 30 03:51:43 2026 read: IOPS=26, BW=105KiB/s (107kB/s)(4984KiB/47671msec) slat (usec): min=2760, max=94592, avg=22814.64, stdev=13219.49 clat (msec): min=53, max=578, avg=280.90, stdev=66.75 lat (msec): min=67, max=603, avg=303.72, stdev=68.16 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 174], 10.00th=[ 199], 20.00th=[ 228], | 30.00th=[ 249], 40.00th=[ 264], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 313], 80.00th=[ 338], 90.00th=[ 372], 95.00th=[ 393], | 99.00th=[ 447], 99.50th=[ 464], 99.90th=[ 506], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 38, max= 207, per=24.82%, avg=104.22, stdev=30.32, samples=94 iops : min= 9, max= 51, avg=25.60, stdev= 7.66, samples=94 write: IOPS=27, BW=110KiB/s (113kB/s)(5256KiB/47671msec); 0 zone resets slat (usec): min=50, max=16947, avg=305.74, stdev=1035.31 clat (msec): min=111, max=540, avg=288.47, stdev=62.76 lat (msec): min=111, max=541, avg=288.78, stdev=62.77 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 192], 10.00th=[ 213], 20.00th=[ 236], | 30.00th=[ 255], 40.00th=[ 271], 50.00th=[ 284], 60.00th=[ 305], | 70.00th=[ 321], 80.00th=[ 338], 90.00th=[ 368], 95.00th=[ 393], | 99.00th=[ 456], 99.50th=[ 481], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 15, max= 188, per=25.06%, avg=109.53, stdev=32.38, samples=94 iops : min= 3, max= 47, avg=26.93, stdev= 8.10, samples=94 lat (msec) : 100=0.08%, 250=28.98%, 500=70.70%, 750=0.23% cpu : usr=0.15%, sys=1.29%, ctx=3833, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179585: Thu Jul 30 03:51:43 2026 read: IOPS=26, BW=106KiB/s (109kB/s)(5060KiB/47733msec) slat (msec): min=2, max=103, avg=22.96, stdev=13.44 clat (msec): min=55, max=510, avg=281.54, stdev=66.77 lat (msec): min=71, max=532, avg=304.50, stdev=68.59 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 182], 10.00th=[ 203], 20.00th=[ 228], | 30.00th=[ 245], 40.00th=[ 262], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 372], 95.00th=[ 405], | 99.00th=[ 451], 99.50th=[ 464], 99.90th=[ 502], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 46, max= 183, per=25.09%, avg=105.39, stdev=28.52, samples=94 iops : min= 11, max= 45, avg=25.89, stdev= 7.13, samples=94 write: IOPS=27, BW=109KiB/s (111kB/s)(5180KiB/47733msec); 0 zone resets slat (usec): min=50, max=42858, avg=328.96, stdev=1629.04 clat (msec): min=116, max=533, avg=288.79, stdev=64.95 lat (msec): min=116, max=533, avg=289.12, stdev=64.94 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 184], 10.00th=[ 209], 20.00th=[ 241], | 30.00th=[ 255], 40.00th=[ 271], 50.00th=[ 284], 60.00th=[ 300], | 70.00th=[ 317], 80.00th=[ 342], 90.00th=[ 372], 95.00th=[ 401], | 99.00th=[ 460], 99.50th=[ 481], 99.90th=[ 527], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 47, max= 207, per=24.67%, avg=107.82, stdev=30.55, samples=94 iops : min= 11, max= 51, avg=26.50, stdev= 7.67, samples=94 lat (msec) : 100=0.12%, 250=29.10%, 500=70.55%, 750=0.23% cpu : usr=0.24%, sys=1.42%, ctx=3672, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=420KiB/s (431kB/s), 105KiB/s-106KiB/s (107kB/s-109kB/s), io=19.6MiB (20.5MB), run=47550-47733msec WRITE: bw=438KiB/s (448kB/s), 109KiB/s-111KiB/s (111kB/s-113kB/s), io=20.4MiB (21.4MB), run=47550-47733msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179597: Thu Jul 30 03:51:59 2026 read: IOPS=1, BW=71.6MiB/s (75.0MB/s)(40.0MiB/559msec) slat (nsec): min=56828k, max=56828k, avg=56827923.00, stdev= 0.00 clat (nsec): min=485626k, max=485626k, avg=485625712.00, stdev= 0.00 lat (nsec): min=542458k, max=542458k, avg=542458029.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.61%, ctx=8, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.6MiB/s (75.0MB/s), 71.6MiB/s-71.6MiB/s (75.0MB/s-75.0MB/s), io=40.0MiB (41.9MB), run=559-559msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout