== sanity test 398c: run fio to test AIO ================= 22:55:09 (1775876109) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.287782 s, 146 MB/s osc.lustre-OST0000-osc-ffffa09689394000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147867: Fri Apr 10 22:56:33 2026 write: IOPS=32, BW=129KiB/s (132kB/s)(10.0MiB/79294msec); 0 zone resets slat (usec): min=35, max=8681, avg=283.33, stdev=512.28 clat (msec): min=36, max=851, avg=495.04, stdev=121.21 lat (msec): min=36, max=851, avg=495.33, stdev=121.23 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 271], 10.00th=[ 351], 20.00th=[ 405], | 30.00th=[ 439], 40.00th=[ 468], 50.00th=[ 498], 60.00th=[ 523], | 70.00th=[ 558], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 693], | 99.00th=[ 768], 99.50th=[ 793], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 56, max= 319, per=24.92%, avg=128.34, stdev=34.95, samples=157 iops : min= 14, max= 79, avg=31.70, stdev= 8.77, samples=157 lat (msec) : 50=0.04%, 100=0.20%, 250=3.48%, 500=47.89%, 750=46.88% lat (msec) : 1000=1.52% cpu : usr=0.08%, sys=0.61%, ctx=3038, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147868: Fri Apr 10 22:56:33 2026 write: IOPS=32, BW=129KiB/s (132kB/s)(10.0MiB/79500msec); 0 zone resets slat (usec): min=37, max=12385, avg=279.47, stdev=521.92 clat (msec): min=182, max=867, avg=496.19, stdev=116.69 lat (msec): min=182, max=867, avg=496.47, stdev=116.71 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 296], 10.00th=[ 355], 20.00th=[ 401], | 30.00th=[ 439], 40.00th=[ 468], 50.00th=[ 493], 60.00th=[ 527], | 70.00th=[ 558], 80.00th=[ 592], 90.00th=[ 642], 95.00th=[ 693], | 99.00th=[ 760], 99.50th=[ 802], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 48, max= 256, per=24.86%, avg=128.03, stdev=33.49, samples=158 iops : min= 12, max= 64, avg=31.63, stdev= 8.39, samples=158 lat (msec) : 250=2.89%, 500=49.65%, 750=45.86%, 1000=1.60% cpu : usr=0.08%, sys=0.56%, ctx=3076, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147869: Fri Apr 10 22:56:33 2026 write: IOPS=32, BW=129KiB/s (132kB/s)(10.0MiB/79500msec); 0 zone resets slat (usec): min=38, max=8523, avg=277.55, stdev=444.04 clat (msec): min=185, max=841, avg=496.13, stdev=118.18 lat (msec): min=186, max=841, avg=496.41, stdev=118.18 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 284], 10.00th=[ 351], 20.00th=[ 397], | 30.00th=[ 439], 40.00th=[ 468], 50.00th=[ 498], 60.00th=[ 527], | 70.00th=[ 558], 80.00th=[ 592], 90.00th=[ 642], 95.00th=[ 693], | 99.00th=[ 776], 99.50th=[ 810], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 64, max= 256, per=24.86%, avg=128.03, stdev=32.42, samples=158 iops : min= 16, max= 64, avg=31.63, stdev= 8.14, samples=158 lat (msec) : 250=2.97%, 500=48.09%, 750=47.27%, 1000=1.68% cpu : usr=0.07%, sys=0.55%, ctx=3115, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147870: Fri Apr 10 22:56:33 2026 write: IOPS=32, BW=129KiB/s (132kB/s)(10.0MiB/79426msec); 0 zone resets slat (usec): min=35, max=7922, avg=275.83, stdev=480.50 clat (msec): min=34, max=865, avg=495.80, stdev=120.03 lat (msec): min=34, max=865, avg=496.08, stdev=120.03 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 271], 10.00th=[ 351], 20.00th=[ 401], | 30.00th=[ 435], 40.00th=[ 468], 50.00th=[ 498], 60.00th=[ 523], | 70.00th=[ 558], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 693], | 99.00th=[ 776], 99.50th=[ 844], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 869] bw ( KiB/s): min= 24, max= 280, per=24.85%, avg=127.98, stdev=36.92, samples=158 iops : min= 6, max= 70, avg=31.61, stdev= 9.26, samples=158 lat (msec) : 50=0.04%, 100=0.08%, 250=2.73%, 500=48.44%, 750=47.03% lat (msec) : 1000=1.68% cpu : usr=0.08%, sys=0.55%, ctx=2848, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=515KiB/s (528kB/s), 129KiB/s-129KiB/s (132kB/s-132kB/s), io=40.0MiB (41.9MB), run=79294-79500msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147891: Fri Apr 10 22:57:20 2026 read: IOPS=28, BW=115KiB/s (117kB/s)(5048KiB/44028msec) slat (usec): min=1656, max=20979, avg=6097.08, stdev=2892.33 clat (msec): min=4, max=354, avg=175.54, stdev=42.66 lat (msec): min=6, max=365, avg=181.64, stdev=43.26 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 207], 90.00th=[ 230], 95.00th=[ 247], | 99.00th=[ 275], 99.50th=[ 296], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 40, max= 208, per=25.16%, avg=114.46, stdev=35.85, samples=87 iops : min= 10, max= 52, avg=28.26, stdev= 9.02, samples=87 write: IOPS=29, BW=118KiB/s (121kB/s)(5192KiB/44028msec); 0 zone resets slat (usec): min=42, max=4560, avg=288.09, stdev=375.37 clat (msec): min=79, max=681, avg=364.28, stdev=99.41 lat (msec): min=79, max=681, avg=364.57, stdev=99.39 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 218], 10.00th=[ 245], 20.00th=[ 279], | 30.00th=[ 305], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 376], | 70.00th=[ 405], 80.00th=[ 447], 90.00th=[ 506], 95.00th=[ 542], | 99.00th=[ 617], 99.50th=[ 625], 99.90th=[ 642], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 64, max= 184, per=24.80%, avg=117.32, stdev=27.51, samples=87 iops : min= 16, max= 46, avg=28.99, stdev= 6.91, samples=87 lat (msec) : 10=0.12%, 20=0.04%, 50=0.27%, 100=0.86%, 250=51.52% lat (msec) : 500=41.68%, 750=5.51% cpu : usr=0.17%, sys=1.37%, ctx=5175, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147892: Fri Apr 10 22:57:20 2026 read: IOPS=28, BW=113KiB/s (116kB/s)(4976KiB/43970msec) slat (usec): min=58, max=27478, avg=6087.79, stdev=2949.76 clat (msec): min=9, max=479, avg=174.94, stdev=47.51 lat (msec): min=12, max=485, avg=181.03, stdev=48.15 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 111], 10.00th=[ 123], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 226], 95.00th=[ 253], | 99.00th=[ 300], 99.50th=[ 376], 99.90th=[ 477], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 40, max= 216, per=24.80%, avg=112.83, stdev=38.79, samples=87 iops : min= 10, max= 54, avg=27.86, stdev= 9.73, samples=87 write: IOPS=29, BW=120KiB/s (123kB/s)(5264KiB/43970msec); 0 zone resets slat (usec): min=51, max=4484, avg=275.85, stdev=361.69 clat (msec): min=97, max=680, avg=362.61, stdev=104.54 lat (msec): min=97, max=680, avg=362.89, stdev=104.55 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 199], 10.00th=[ 239], 20.00th=[ 279], | 30.00th=[ 300], 40.00th=[ 334], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 447], 90.00th=[ 510], 95.00th=[ 550], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 659], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 48, max= 182, per=25.22%, avg=119.28, stdev=28.33, samples=87 iops : min= 12, max= 45, avg=29.47, stdev= 7.10, samples=87 lat (msec) : 10=0.04%, 20=0.08%, 50=0.39%, 100=1.02%, 250=51.33% lat (msec) : 500=41.45%, 750=5.70% cpu : usr=0.17%, sys=1.33%, ctx=5099, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147893: Fri Apr 10 22:57:20 2026 read: IOPS=28, BW=113KiB/s (116kB/s)(4984KiB/44082msec) slat (usec): min=1662, max=20460, avg=6155.00, stdev=2908.00 clat (msec): min=6, max=340, avg=174.70, stdev=42.51 lat (msec): min=11, max=346, avg=180.86, stdev=43.06 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 113], 10.00th=[ 124], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 247], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 313], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 24, max= 240, per=24.77%, avg=112.69, stdev=39.71, samples=87 iops : min= 6, max= 60, avg=27.83, stdev= 9.95, samples=87 write: IOPS=29, BW=119KiB/s (122kB/s)(5256KiB/44082msec); 0 zone resets slat (usec): min=47, max=5092, avg=297.63, stdev=408.71 clat (msec): min=49, max=700, avg=363.57, stdev=102.54 lat (msec): min=49, max=700, avg=363.87, stdev=102.56 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 205], 10.00th=[ 239], 20.00th=[ 279], | 30.00th=[ 309], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 409], 80.00th=[ 447], 90.00th=[ 502], 95.00th=[ 558], | 99.00th=[ 617], 99.50th=[ 642], 99.90th=[ 659], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 55, max= 208, per=24.90%, avg=117.79, stdev=27.37, samples=87 iops : min= 13, max= 52, avg=29.10, stdev= 6.89, samples=87 lat (msec) : 10=0.16%, 20=0.08%, 50=0.31%, 100=0.98%, 250=51.60% lat (msec) : 500=41.52%, 750=5.35% cpu : usr=0.17%, sys=1.38%, ctx=5168, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147894: Fri Apr 10 22:57:20 2026 read: IOPS=28, BW=115KiB/s (118kB/s)(5060KiB/43938msec) slat (usec): min=1498, max=27078, avg=6071.34, stdev=2968.27 clat (msec): min=19, max=475, avg=178.42, stdev=43.29 lat (msec): min=25, max=480, avg=184.50, stdev=43.95 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 116], 10.00th=[ 128], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 292], 99.50th=[ 313], 99.90th=[ 447], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 7, max= 199, per=25.30%, avg=115.11, stdev=34.52, samples=87 iops : min= 1, max= 49, avg=28.43, stdev= 8.60, samples=87 write: IOPS=29, BW=118KiB/s (121kB/s)(5180KiB/43938msec); 0 zone resets slat (usec): min=52, max=5144, avg=272.11, stdev=375.63 clat (msec): min=96, max=757, avg=360.74, stdev=103.30 lat (msec): min=96, max=757, avg=361.02, stdev=103.31 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 211], 10.00th=[ 239], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 326], 50.00th=[ 347], 60.00th=[ 372], | 70.00th=[ 409], 80.00th=[ 443], 90.00th=[ 498], 95.00th=[ 550], | 99.00th=[ 634], 99.50th=[ 676], 99.90th=[ 693], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 48, max= 192, per=24.77%, avg=117.18, stdev=29.44, samples=87 iops : min= 12, max= 48, avg=28.95, stdev= 7.42, samples=87 lat (msec) : 20=0.04%, 50=0.08%, 100=0.66%, 250=52.93%, 500=41.29% lat (msec) : 750=4.96%, 1000=0.04% cpu : usr=0.16%, sys=1.37%, ctx=5082, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=455KiB/s (466kB/s), 113KiB/s-115KiB/s (116kB/s-118kB/s), io=19.6MiB (20.5MB), run=43938-44082msec WRITE: bw=474KiB/s (485kB/s), 118KiB/s-120KiB/s (121kB/s-123kB/s), io=20.4MiB (21.4MB), run=43938-44082msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147904: Fri Apr 10 22:57:25 2026 read: IOPS=3, BW=143MiB/s (150MB/s)(40.0MiB/280msec) slat (nsec): min=19780k, max=19780k, avg=19779616.00, stdev= 0.00 clat (nsec): min=248786k, max=248786k, avg=248785811.00, stdev= 0.00 lat (nsec): min=268594k, max=268594k, avg=268594200.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 249], 80.00th=[ 249], 90.00th=[ 249], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.94%, ctx=28, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=40.0MiB (41.9MB), run=280-280msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout