== sanity test 398c: run fio to test AIO ================= 19:20:48 (1785885648) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.841186 s, 49.9 MB/s osc.lustre-OST0000-osc-ffff94b5a008c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179968: Tue Aug 4 19:21:40 2026 write: IOPS=64, BW=258KiB/s (264kB/s)(10.0MiB/39669msec); 0 zone resets slat (usec): min=48, max=27141, avg=290.84, stdev=1059.62 clat (msec): min=51, max=503, avg=247.02, stdev=67.53 lat (msec): min=51, max=503, avg=247.31, stdev=67.49 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 144], 10.00th=[ 161], 20.00th=[ 188], | 30.00th=[ 209], 40.00th=[ 228], 50.00th=[ 245], 60.00th=[ 262], | 70.00th=[ 279], 80.00th=[ 305], 90.00th=[ 334], 95.00th=[ 363], | 99.00th=[ 418], 99.50th=[ 430], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 502] bw ( KiB/s): min= 136, max= 390, per=24.99%, avg=257.68, stdev=60.71, samples=78 iops : min= 34, max= 97, avg=63.96, stdev=15.22, samples=78 lat (msec) : 100=0.78%, 250=51.95%, 500=47.23%, 750=0.04% cpu : usr=0.15%, sys=0.78%, ctx=2513, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179969: Tue Aug 4 19:21:40 2026 write: IOPS=64, BW=259KiB/s (265kB/s)(10.0MiB/39581msec); 0 zone resets slat (usec): min=48, max=17587, avg=311.34, stdev=993.68 clat (msec): min=41, max=510, avg=246.85, stdev=69.21 lat (msec): min=41, max=510, avg=247.16, stdev=69.18 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 142], 10.00th=[ 161], 20.00th=[ 184], | 30.00th=[ 207], 40.00th=[ 228], 50.00th=[ 245], 60.00th=[ 264], | 70.00th=[ 279], 80.00th=[ 300], 90.00th=[ 334], 95.00th=[ 372], | 99.00th=[ 435], 99.50th=[ 468], 99.90th=[ 485], 99.95th=[ 506], | 99.99th=[ 510] bw ( KiB/s): min= 126, max= 393, per=25.06%, avg=258.36, stdev=66.14, samples=78 iops : min= 31, max= 98, avg=64.14, stdev=16.56, samples=78 lat (msec) : 50=0.04%, 100=0.47%, 250=52.85%, 500=46.56%, 750=0.08% cpu : usr=0.13%, sys=0.95%, ctx=2497, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179970: Tue Aug 4 19:21:40 2026 write: IOPS=64, BW=258KiB/s (264kB/s)(10.0MiB/39697msec); 0 zone resets slat (usec): min=48, max=17209, avg=259.74, stdev=821.37 clat (msec): min=43, max=502, avg=247.07, stdev=67.90 lat (msec): min=43, max=502, avg=247.34, stdev=67.88 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 140], 10.00th=[ 161], 20.00th=[ 190], | 30.00th=[ 209], 40.00th=[ 226], 50.00th=[ 241], 60.00th=[ 259], | 70.00th=[ 275], 80.00th=[ 300], 90.00th=[ 338], 95.00th=[ 368], | 99.00th=[ 426], 99.50th=[ 439], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 502] bw ( KiB/s): min= 127, max= 388, per=24.90%, avg=256.71, stdev=67.74, samples=78 iops : min= 31, max= 97, avg=63.74, stdev=16.96, samples=78 lat (msec) : 50=0.04%, 100=0.59%, 250=53.95%, 500=45.39%, 750=0.04% cpu : usr=0.12%, sys=0.70%, ctx=2005, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179971: Tue Aug 4 19:21:40 2026 write: IOPS=64, BW=259KiB/s (265kB/s)(10.0MiB/39543msec); 0 zone resets slat (usec): min=49, max=18558, avg=280.74, stdev=894.17 clat (msec): min=26, max=511, avg=246.27, stdev=68.80 lat (msec): min=34, max=512, avg=246.55, stdev=68.77 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 146], 10.00th=[ 161], 20.00th=[ 188], | 30.00th=[ 211], 40.00th=[ 226], 50.00th=[ 239], 60.00th=[ 262], | 70.00th=[ 275], 80.00th=[ 300], 90.00th=[ 334], 95.00th=[ 372], | 99.00th=[ 439], 99.50th=[ 447], 99.90th=[ 502], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 127, max= 486, per=25.10%, avg=258.81, stdev=65.40, samples=78 iops : min= 31, max= 121, avg=64.24, stdev=16.38, samples=78 lat (msec) : 50=0.08%, 100=0.66%, 250=54.02%, 500=45.16%, 750=0.08% cpu : usr=0.14%, sys=0.84%, ctx=2612, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1032KiB/s (1057kB/s), 258KiB/s-259KiB/s (264kB/s-265kB/s), io=40.0MiB (41.9MB), run=39543-39697msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179987: Tue Aug 4 19:22:34 2026 read: IOPS=29, BW=118KiB/s (121kB/s)(5048KiB/42625msec) slat (usec): min=54, max=67522, avg=19565.00, stdev=10868.67 clat (msec): min=37, max=519, avg=251.80, stdev=60.67 lat (msec): min=37, max=553, avg=271.37, stdev=62.56 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 163], 10.00th=[ 182], 20.00th=[ 203], | 30.00th=[ 222], 40.00th=[ 236], 50.00th=[ 251], 60.00th=[ 266], | 70.00th=[ 279], 80.00th=[ 296], 90.00th=[ 326], 95.00th=[ 347], | 99.00th=[ 435], 99.50th=[ 456], 99.90th=[ 514], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 47, max= 197, per=25.19%, avg=117.87, stdev=30.69, samples=84 iops : min= 11, max= 49, avg=29.04, stdev= 7.67, samples=84 write: IOPS=30, BW=122KiB/s (125kB/s)(5192KiB/42625msec); 0 zone resets slat (usec): min=54, max=13050, avg=279.82, stdev=836.98 clat (msec): min=71, max=512, avg=260.75, stdev=51.56 lat (msec): min=72, max=512, avg=261.03, stdev=51.64 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 180], 10.00th=[ 199], 20.00th=[ 220], | 30.00th=[ 236], 40.00th=[ 249], 50.00th=[ 262], 60.00th=[ 271], | 70.00th=[ 284], 80.00th=[ 296], 90.00th=[ 321], 95.00th=[ 347], | 99.00th=[ 401], 99.50th=[ 435], 99.90th=[ 502], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 55, max= 183, per=24.94%, avg=121.46, stdev=27.35, samples=84 iops : min= 13, max= 45, avg=29.94, stdev= 6.93, samples=84 lat (msec) : 50=0.20%, 100=0.31%, 250=44.84%, 500=54.49%, 750=0.16% cpu : usr=0.19%, sys=1.50%, ctx=4176, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179988: Tue Aug 4 19:22:34 2026 read: IOPS=29, BW=116KiB/s (119kB/s)(4976KiB/42833msec) slat (usec): min=2721, max=76123, avg=19569.61, stdev=10951.99 clat (msec): min=23, max=487, avg=254.34, stdev=58.11 lat (msec): min=31, max=538, avg=273.91, stdev=59.02 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 169], 10.00th=[ 184], 20.00th=[ 209], | 30.00th=[ 226], 40.00th=[ 239], 50.00th=[ 251], 60.00th=[ 264], | 70.00th=[ 279], 80.00th=[ 300], 90.00th=[ 330], 95.00th=[ 355], | 99.00th=[ 418], 99.50th=[ 435], 99.90th=[ 485], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 56, max= 200, per=24.69%, avg=115.55, stdev=27.50, samples=84 iops : min= 14, max= 50, avg=28.44, stdev= 6.93, samples=84 write: IOPS=30, BW=123KiB/s (126kB/s)(5264KiB/42833msec); 0 zone resets slat (usec): min=51, max=12188, avg=267.27, stdev=733.50 clat (msec): min=102, max=449, avg=260.39, stdev=52.42 lat (msec): min=102, max=449, avg=260.65, stdev=52.41 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 180], 10.00th=[ 201], 20.00th=[ 220], | 30.00th=[ 234], 40.00th=[ 245], 50.00th=[ 259], 60.00th=[ 271], | 70.00th=[ 284], 80.00th=[ 300], 90.00th=[ 321], 95.00th=[ 347], | 99.00th=[ 414], 99.50th=[ 430], 99.90th=[ 439], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 47, max= 183, per=25.14%, avg=122.45, stdev=29.08, samples=84 iops : min= 11, max= 45, avg=30.18, stdev= 7.33, samples=84 lat (msec) : 50=0.12%, 100=0.04%, 250=46.05%, 500=53.79% cpu : usr=0.14%, sys=1.55%, ctx=4230, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179989: Tue Aug 4 19:22:34 2026 read: IOPS=29, BW=116KiB/s (119kB/s)(4984KiB/42848msec) slat (usec): min=2716, max=98721, avg=19868.57, stdev=11671.84 clat (msec): min=67, max=563, avg=252.88, stdev=57.05 lat (msec): min=74, max=572, avg=272.75, stdev=58.54 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 165], 10.00th=[ 184], 20.00th=[ 209], | 30.00th=[ 224], 40.00th=[ 234], 50.00th=[ 249], 60.00th=[ 264], | 70.00th=[ 279], 80.00th=[ 296], 90.00th=[ 326], 95.00th=[ 351], | 99.00th=[ 409], 99.50th=[ 426], 99.90th=[ 531], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 39, max= 216, per=24.73%, avg=115.75, stdev=34.64, samples=84 iops : min= 9, max= 54, avg=28.52, stdev= 8.73, samples=84 write: IOPS=30, BW=123KiB/s (126kB/s)(5256KiB/42848msec); 0 zone resets slat (usec): min=54, max=11938, avg=257.28, stdev=705.34 clat (msec): min=76, max=498, avg=260.10, stdev=52.69 lat (msec): min=76, max=499, avg=260.36, stdev=52.71 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 178], 10.00th=[ 199], 20.00th=[ 218], | 30.00th=[ 234], 40.00th=[ 247], 50.00th=[ 259], 60.00th=[ 271], | 70.00th=[ 284], 80.00th=[ 300], 90.00th=[ 321], 95.00th=[ 347], | 99.00th=[ 414], 99.50th=[ 426], 99.90th=[ 489], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 60, max= 200, per=24.94%, avg=121.45, stdev=29.66, samples=84 iops : min= 15, max= 50, avg=29.95, stdev= 7.46, samples=84 lat (msec) : 100=0.16%, 250=46.52%, 500=53.20%, 750=0.12% cpu : usr=0.20%, sys=1.53%, ctx=4341, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179990: Tue Aug 4 19:22:34 2026 read: IOPS=29, BW=118KiB/s (121kB/s)(5060KiB/42826msec) slat (usec): min=2563, max=68005, avg=19851.41, stdev=10951.14 clat (msec): min=26, max=491, avg=251.19, stdev=56.73 lat (msec): min=54, max=517, avg=271.05, stdev=58.03 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 167], 10.00th=[ 184], 20.00th=[ 205], | 30.00th=[ 222], 40.00th=[ 236], 50.00th=[ 247], 60.00th=[ 262], | 70.00th=[ 275], 80.00th=[ 292], 90.00th=[ 321], 95.00th=[ 347], | 99.00th=[ 418], 99.50th=[ 447], 99.90th=[ 481], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 53, max= 224, per=25.16%, avg=117.73, stdev=27.58, samples=84 iops : min= 13, max= 56, avg=28.99, stdev= 6.94, samples=84 write: IOPS=30, BW=121KiB/s (124kB/s)(5180KiB/42826msec); 0 zone resets slat (usec): min=51, max=15242, avg=275.69, stdev=771.11 clat (msec): min=67, max=438, avg=261.79, stdev=52.86 lat (msec): min=69, max=439, avg=262.06, stdev=52.88 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 178], 10.00th=[ 199], 20.00th=[ 218], | 30.00th=[ 234], 40.00th=[ 247], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 288], 80.00th=[ 305], 90.00th=[ 326], 95.00th=[ 355], | 99.00th=[ 401], 99.50th=[ 418], 99.90th=[ 435], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 63, max= 192, per=24.67%, avg=120.13, stdev=28.69, samples=84 iops : min= 15, max= 48, avg=29.60, stdev= 7.23, samples=84 lat (msec) : 50=0.04%, 100=0.08%, 250=46.56%, 500=53.32% cpu : usr=0.19%, sys=1.50%, ctx=4354, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=468KiB/s (480kB/s), 116KiB/s-118KiB/s (119kB/s-121kB/s), io=19.6MiB (20.5MB), run=42625-42848msec WRITE: bw=488KiB/s (499kB/s), 121KiB/s-123KiB/s (124kB/s-126kB/s), io=20.4MiB (21.4MB), run=42625-42848msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180000: Tue Aug 4 19:22:46 2026 read: IOPS=1, BW=66.1MiB/s (69.3MB/s)(40.0MiB/605msec) slat (nsec): min=46853k, max=46853k, avg=46853065.00, stdev= 0.00 clat (nsec): min=507374k, max=507374k, avg=507373667.00, stdev= 0.00 lat (nsec): min=554232k, max=554232k, avg=554231922.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.99%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.1MiB/s (69.3MB/s), 66.1MiB/s-66.1MiB/s (69.3MB/s-69.3MB/s), io=40.0MiB (41.9MB), run=605-605msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout