== sanity test 398c: run fio to test AIO ================= 14:16:23 (1788804983) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.708758 s, 59.2 MB/s osc.lustre-OST0000-osc-ffff9190c6296800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180125: Mon Sep 7 14:17:01 2026 write: IOPS=98, BW=393KiB/s (402kB/s)(10.0MiB/26084msec); 0 zone resets slat (usec): min=24, max=24162, avg=215.23, stdev=914.97 clat (msec): min=12, max=418, avg=162.15, stdev=49.19 lat (msec): min=13, max=418, avg=162.37, stdev=49.22 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 107], 10.00th=[ 114], 20.00th=[ 125], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 176], 80.00th=[ 194], 90.00th=[ 226], 95.00th=[ 264], | 99.00th=[ 309], 99.50th=[ 384], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 135, max= 558, per=25.15%, avg=393.82, stdev=89.29, samples=51 iops : min= 33, max= 139, avg=98.12, stdev=22.41, samples=51 lat (msec) : 20=0.12%, 50=0.43%, 100=2.50%, 250=89.84%, 500=7.11% cpu : usr=0.23%, sys=1.21%, ctx=3053, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180126: Mon Sep 7 14:17:01 2026 write: IOPS=98, BW=392KiB/s (401kB/s)(10.0MiB/26122msec); 0 zone resets slat (usec): min=21, max=10662, avg=144.85, stdev=585.22 clat (msec): min=56, max=384, avg=162.42, stdev=47.24 lat (msec): min=56, max=384, avg=162.56, stdev=47.26 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 109], 10.00th=[ 117], 20.00th=[ 127], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 161], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 222], 95.00th=[ 264], | 99.00th=[ 309], 99.50th=[ 355], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 384] bw ( KiB/s): min= 247, max= 512, per=24.99%, avg=391.29, stdev=97.12, samples=51 iops : min= 61, max= 128, avg=97.49, stdev=24.38, samples=51 lat (msec) : 100=2.58%, 250=91.72%, 500=5.70% cpu : usr=0.19%, sys=0.70%, ctx=948, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180127: Mon Sep 7 14:17:01 2026 write: IOPS=98, BW=396KiB/s (405kB/s)(10.0MiB/25861msec); 0 zone resets slat (usec): min=21, max=12981, avg=148.70, stdev=612.67 clat (msec): min=50, max=413, avg=161.33, stdev=50.29 lat (msec): min=50, max=413, avg=161.48, stdev=50.31 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 104], 10.00th=[ 114], 20.00th=[ 125], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 161], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 228], 95.00th=[ 275], | 99.00th=[ 330], 99.50th=[ 384], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 414] bw ( KiB/s): min= 135, max= 630, per=25.16%, avg=394.02, stdev=109.13, samples=51 iops : min= 33, max= 157, avg=98.16, stdev=27.37, samples=51 lat (msec) : 100=3.75%, 250=89.14%, 500=7.11% cpu : usr=0.19%, sys=0.80%, ctx=1283, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180128: Mon Sep 7 14:17:01 2026 write: IOPS=97, BW=392KiB/s (401kB/s)(10.0MiB/26147msec); 0 zone resets slat (usec): min=21, max=9665, avg=176.14, stdev=597.14 clat (msec): min=17, max=411, avg=162.44, stdev=47.82 lat (msec): min=17, max=411, avg=162.61, stdev=47.81 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 105], 10.00th=[ 115], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 161], | 70.00th=[ 178], 80.00th=[ 197], 90.00th=[ 226], 95.00th=[ 257], | 99.00th=[ 300], 99.50th=[ 342], 99.90th=[ 409], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 223, max= 621, per=24.92%, avg=390.18, stdev=88.06, samples=51 iops : min= 55, max= 155, avg=97.22, stdev=22.09, samples=51 lat (msec) : 20=0.04%, 50=0.20%, 100=2.30%, 250=92.23%, 500=5.23% cpu : usr=0.21%, sys=0.87%, ctx=2309, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1567KiB/s (1604kB/s), 392KiB/s-396KiB/s (401kB/s-405kB/s), io=40.0MiB (41.9MB), run=25861-26147msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180143: Mon Sep 7 14:17:43 2026 read: IOPS=38, BW=155KiB/s (159kB/s)(5048KiB/32484msec) slat (usec): min=48, max=87610, avg=14517.69, stdev=8045.22 clat (msec): min=28, max=389, avg=191.71, stdev=39.38 lat (msec): min=29, max=400, avg=206.23, stdev=40.57 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 199], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 245], 95.00th=[ 262], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 351], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 78, max= 248, per=25.07%, avg=154.45, stdev=33.17, samples=64 iops : min= 19, max= 62, avg=38.14, stdev= 8.37, samples=64 write: IOPS=39, BW=160KiB/s (164kB/s)(5192KiB/32484msec); 0 zone resets slat (usec): min=24, max=24728, avg=220.34, stdev=1105.62 clat (msec): min=64, max=393, avg=199.17, stdev=39.94 lat (msec): min=65, max=394, avg=199.39, stdev=39.95 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 207], | 70.00th=[ 218], 80.00th=[ 228], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 338], 99.90th=[ 363], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 78, max= 247, per=24.84%, avg=159.23, stdev=32.87, samples=64 iops : min= 19, max= 61, avg=39.36, stdev= 8.16, samples=64 lat (msec) : 50=0.08%, 100=0.31%, 250=90.86%, 500=8.75% cpu : usr=0.28%, sys=1.56%, ctx=4328, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180144: Mon Sep 7 14:17:43 2026 read: IOPS=38, BW=153KiB/s (157kB/s)(4976KiB/32543msec) slat (usec): min=2914, max=49939, avg=14192.31, stdev=7233.61 clat (msec): min=21, max=348, avg=192.41, stdev=38.82 lat (msec): min=50, max=354, avg=206.60, stdev=39.94 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 224], 90.00th=[ 243], 95.00th=[ 255], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 338], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 55, max= 230, per=24.58%, avg=151.41, stdev=32.80, samples=64 iops : min= 13, max= 57, avg=37.38, stdev= 8.24, samples=64 write: IOPS=40, BW=162KiB/s (166kB/s)(5264KiB/32543msec); 0 zone resets slat (usec): min=30, max=21500, avg=190.72, stdev=891.79 clat (msec): min=54, max=339, avg=197.32, stdev=39.15 lat (msec): min=54, max=339, avg=197.51, stdev=39.16 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 197], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 228], 90.00th=[ 249], 95.00th=[ 268], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 102, max= 240, per=25.19%, avg=161.47, stdev=32.90, samples=64 iops : min= 25, max= 60, avg=39.92, stdev= 8.25, samples=64 lat (msec) : 50=0.04%, 100=0.35%, 250=91.76%, 500=7.85% cpu : usr=0.31%, sys=1.59%, ctx=4317, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180145: Mon Sep 7 14:17:43 2026 read: IOPS=38, BW=153KiB/s (157kB/s)(4984KiB/32552msec) slat (usec): min=3012, max=49089, avg=14127.04, stdev=7307.67 clat (msec): min=85, max=362, avg=192.59, stdev=41.24 lat (msec): min=102, max=376, avg=206.72, stdev=42.35 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 133], 10.00th=[ 144], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 199], | 70.00th=[ 209], 80.00th=[ 226], 90.00th=[ 245], 95.00th=[ 264], | 99.00th=[ 317], 99.50th=[ 338], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 38, max= 247, per=24.70%, avg=152.13, stdev=35.50, samples=64 iops : min= 9, max= 61, avg=37.58, stdev= 8.89, samples=64 write: IOPS=40, BW=161KiB/s (165kB/s)(5256KiB/32552msec); 0 zone resets slat (usec): min=34, max=22425, avg=239.01, stdev=1107.01 clat (msec): min=72, max=389, avg=196.64, stdev=40.16 lat (msec): min=72, max=389, avg=196.88, stdev=40.17 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 228], 90.00th=[ 247], 95.00th=[ 271], | 99.00th=[ 300], 99.50th=[ 317], 99.90th=[ 359], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 46, max= 253, per=24.97%, avg=160.03, stdev=44.05, samples=64 iops : min= 11, max= 63, avg=39.56, stdev=11.05, samples=64 lat (msec) : 100=0.12%, 250=91.21%, 500=8.67% cpu : usr=0.25%, sys=1.72%, ctx=4316, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180146: Mon Sep 7 14:17:43 2026 read: IOPS=38, BW=156KiB/s (159kB/s)(5060KiB/32491msec) slat (usec): min=2930, max=46779, avg=14236.69, stdev=7247.86 clat (msec): min=30, max=341, avg=190.98, stdev=40.95 lat (msec): min=50, max=384, avg=205.22, stdev=41.88 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 199], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 243], 95.00th=[ 264], | 99.00th=[ 305], 99.50th=[ 330], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 39, max= 224, per=25.24%, avg=155.50, stdev=34.66, samples=64 iops : min= 9, max= 56, avg=38.42, stdev= 8.72, samples=64 write: IOPS=39, BW=159KiB/s (163kB/s)(5180KiB/32491msec); 0 zone resets slat (usec): min=30, max=16418, avg=193.27, stdev=706.41 clat (msec): min=104, max=334, avg=197.12, stdev=38.70 lat (msec): min=104, max=334, avg=197.32, stdev=38.70 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 228], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 70, max= 232, per=24.74%, avg=158.56, stdev=35.86, samples=64 iops : min= 17, max= 58, avg=39.19, stdev= 9.00, samples=64 lat (msec) : 50=0.04%, 100=0.27%, 250=91.33%, 500=8.36% cpu : usr=0.26%, sys=1.65%, ctx=4422, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=616KiB/s (631kB/s), 153KiB/s-156KiB/s (157kB/s-159kB/s), io=19.6MiB (20.5MB), run=32484-32552msec WRITE: bw=642KiB/s (657kB/s), 159KiB/s-162KiB/s (163kB/s-166kB/s), io=20.4MiB (21.4MB), run=32484-32552msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180155: Mon Sep 7 14:17:55 2026 read: IOPS=1, BW=70.4MiB/s (73.8MB/s)(40.0MiB/568msec) slat (nsec): min=32920k, max=32920k, avg=32919646.00, stdev= 0.00 clat (nsec): min=518705k, max=518705k, avg=518705158.00, stdev= 0.00 lat (nsec): min=551631k, max=551631k, avg=551630532.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 518], 10.00th=[ 518], 20.00th=[ 518], | 30.00th=[ 518], 40.00th=[ 518], 50.00th=[ 518], 60.00th=[ 518], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.12%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.4MiB/s (73.8MB/s), 70.4MiB/s-70.4MiB/s (73.8MB/s-73.8MB/s), io=40.0MiB (41.9MB), run=568-568msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout