== sanity test 398c: run fio to test AIO ================= 02:44:13 (1787121853) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.469085 s, 89.4 MB/s osc.lustre-OST0000-osc-ffff9e84391c5000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179996: Wed Aug 19 02:44:47 2026 write: IOPS=104, BW=419KiB/s (429kB/s)(10.0MiB/24453msec); 0 zone resets slat (usec): min=37, max=15633, avg=282.50, stdev=781.12 clat (msec): min=8, max=315, avg=152.36, stdev=38.48 lat (msec): min=9, max=317, avg=152.64, stdev=38.50 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 100], 10.00th=[ 109], 20.00th=[ 123], | 30.00th=[ 131], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 157], | 70.00th=[ 167], 80.00th=[ 184], 90.00th=[ 207], 95.00th=[ 222], | 99.00th=[ 266], 99.50th=[ 284], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 263, max= 549, per=25.23%, avg=419.31, stdev=78.76, samples=48 iops : min= 65, max= 137, avg=104.46, stdev=19.76, samples=48 lat (msec) : 10=0.04%, 50=0.16%, 100=4.84%, 250=93.63%, 500=1.33% cpu : usr=0.24%, sys=1.40%, ctx=2851, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179997: Wed Aug 19 02:44:47 2026 write: IOPS=103, BW=416KiB/s (426kB/s)(10.0MiB/24639msec); 0 zone resets slat (usec): min=37, max=21666, avg=287.15, stdev=892.74 clat (msec): min=43, max=263, avg=152.80, stdev=35.63 lat (msec): min=43, max=264, avg=153.09, stdev=35.68 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 103], 10.00th=[ 112], 20.00th=[ 125], | 30.00th=[ 131], 40.00th=[ 138], 50.00th=[ 148], 60.00th=[ 159], | 70.00th=[ 171], 80.00th=[ 184], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 245], 99.50th=[ 251], 99.90th=[ 262], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 263, max= 638, per=25.00%, avg=415.52, stdev=83.40, samples=48 iops : min= 65, max= 159, avg=103.50, stdev=20.94, samples=48 lat (msec) : 50=0.04%, 100=4.18%, 250=95.16%, 500=0.62% cpu : usr=0.19%, sys=1.29%, ctx=2604, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179998: Wed Aug 19 02:44:47 2026 write: IOPS=103, BW=416KiB/s (426kB/s)(10.0MiB/24624msec); 0 zone resets slat (usec): min=40, max=17787, avg=300.31, stdev=827.15 clat (msec): min=10, max=279, avg=152.71, stdev=35.98 lat (msec): min=11, max=279, avg=153.01, stdev=36.03 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 103], 10.00th=[ 111], 20.00th=[ 124], | 30.00th=[ 131], 40.00th=[ 138], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 167], 80.00th=[ 184], 90.00th=[ 201], 95.00th=[ 218], | 99.00th=[ 249], 99.50th=[ 257], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 256, max= 638, per=25.01%, avg=415.60, stdev=81.24, samples=48 iops : min= 64, max= 159, avg=103.50, stdev=20.38, samples=48 lat (msec) : 20=0.08%, 50=0.04%, 100=2.93%, 250=96.13%, 500=0.82% cpu : usr=0.24%, sys=1.35%, ctx=2551, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179999: Wed Aug 19 02:44:47 2026 write: IOPS=103, BW=416KiB/s (426kB/s)(10.0MiB/24621msec); 0 zone resets slat (usec): min=36, max=16621, avg=319.14, stdev=882.32 clat (msec): min=10, max=325, avg=152.82, stdev=37.74 lat (msec): min=11, max=325, avg=153.14, stdev=37.73 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 104], 10.00th=[ 110], 20.00th=[ 123], | 30.00th=[ 131], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 157], | 70.00th=[ 167], 80.00th=[ 182], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 253], 99.50th=[ 259], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 254, max= 582, per=25.06%, avg=416.52, stdev=75.96, samples=48 iops : min= 63, max= 145, avg=103.73, stdev=19.06, samples=48 lat (msec) : 20=0.04%, 50=0.31%, 100=3.32%, 250=94.61%, 500=1.72% cpu : usr=0.19%, sys=1.49%, ctx=2853, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1662KiB/s (1702kB/s), 416KiB/s-419KiB/s (426kB/s-429kB/s), io=40.0MiB (41.9MB), run=24453-24639msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180013: Wed Aug 19 02:45:26 2026 read: IOPS=40, BW=162KiB/s (166kB/s)(5048KiB/31081msec) slat (usec): min=2851, max=54619, avg=14181.38, stdev=8009.11 clat (msec): min=20, max=393, avg=182.50, stdev=48.40 lat (msec): min=32, max=400, avg=196.69, stdev=50.74 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 190], | 70.00th=[ 205], 80.00th=[ 222], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 305], 99.50th=[ 334], 99.90th=[ 376], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 71, max= 256, per=25.06%, avg=161.66, stdev=39.32, samples=61 iops : min= 17, max= 64, avg=40.00, stdev= 9.88, samples=61 write: IOPS=41, BW=167KiB/s (171kB/s)(5192KiB/31081msec); 0 zone resets slat (usec): min=40, max=25043, avg=366.57, stdev=1243.88 clat (msec): min=80, max=375, avg=187.91, stdev=46.23 lat (msec): min=80, max=375, avg=188.28, stdev=46.23 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 121], 10.00th=[ 131], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 197], | 70.00th=[ 209], 80.00th=[ 226], 90.00th=[ 245], 95.00th=[ 271], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 50, max= 282, per=24.83%, avg=166.87, stdev=48.34, samples=61 iops : min= 12, max= 70, avg=41.28, stdev=12.14, samples=61 lat (msec) : 50=0.04%, 100=1.09%, 250=89.65%, 500=9.22% cpu : usr=0.32%, sys=2.21%, ctx=4573, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180014: Wed Aug 19 02:45:26 2026 read: IOPS=40, BW=162KiB/s (166kB/s)(4976KiB/30744msec) slat (usec): min=94, max=84012, avg=13997.04, stdev=8069.38 clat (msec): min=20, max=430, avg=180.68, stdev=51.19 lat (msec): min=21, max=442, avg=194.69, stdev=53.18 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 112], 10.00th=[ 122], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 203], 80.00th=[ 224], 90.00th=[ 247], 95.00th=[ 264], | 99.00th=[ 334], 99.50th=[ 351], 99.90th=[ 393], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 87, max= 288, per=25.04%, avg=161.50, stdev=42.93, samples=60 iops : min= 21, max= 72, avg=39.93, stdev=10.76, samples=60 write: IOPS=42, BW=171KiB/s (175kB/s)(5264KiB/30744msec); 0 zone resets slat (usec): min=41, max=12411, avg=367.69, stdev=939.95 clat (msec): min=68, max=400, avg=188.67, stdev=48.62 lat (msec): min=68, max=400, avg=189.04, stdev=48.66 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 122], 10.00th=[ 132], 20.00th=[ 150], | 30.00th=[ 159], 40.00th=[ 171], 50.00th=[ 184], 60.00th=[ 197], | 70.00th=[ 207], 80.00th=[ 226], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 330], 99.50th=[ 368], 99.90th=[ 393], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 79, max= 256, per=25.59%, avg=171.98, stdev=40.68, samples=60 iops : min= 19, max= 64, avg=42.55, stdev=10.24, samples=60 lat (msec) : 50=0.08%, 100=1.21%, 250=88.59%, 500=10.12% cpu : usr=0.24%, sys=2.04%, ctx=4490, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180015: Wed Aug 19 02:45:26 2026 read: IOPS=40, BW=160KiB/s (164kB/s)(4984KiB/31085msec) slat (usec): min=2639, max=67416, avg=14214.87, stdev=8173.83 clat (msec): min=22, max=379, avg=182.62, stdev=49.74 lat (msec): min=33, max=393, avg=196.85, stdev=52.20 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 116], 10.00th=[ 127], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 205], 80.00th=[ 222], 90.00th=[ 253], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 334], 99.90th=[ 372], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 64, max= 252, per=24.74%, avg=159.54, stdev=41.70, samples=61 iops : min= 16, max= 63, avg=39.46, stdev=10.45, samples=61 write: IOPS=42, BW=169KiB/s (173kB/s)(5256KiB/31085msec); 0 zone resets slat (usec): min=58, max=13931, avg=319.97, stdev=757.12 clat (msec): min=61, max=364, avg=188.55, stdev=46.61 lat (msec): min=61, max=364, avg=188.87, stdev=46.61 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 126], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 207], 80.00th=[ 226], 90.00th=[ 251], 95.00th=[ 275], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 72, max= 264, per=24.96%, avg=167.72, stdev=46.42, samples=61 iops : min= 18, max= 66, avg=41.49, stdev=11.68, samples=61 lat (msec) : 50=0.04%, 100=0.98%, 250=88.20%, 500=10.78% cpu : usr=0.29%, sys=2.03%, ctx=4454, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180016: Wed Aug 19 02:45:26 2026 read: IOPS=40, BW=163KiB/s (167kB/s)(5060KiB/31063msec) slat (usec): min=2497, max=84444, avg=14243.05, stdev=8378.62 clat (msec): min=13, max=374, avg=182.18, stdev=49.53 lat (msec): min=18, max=395, avg=196.43, stdev=51.90 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 111], 10.00th=[ 126], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 222], 90.00th=[ 247], 95.00th=[ 275], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 71, max= 275, per=25.18%, avg=162.41, stdev=43.28, samples=61 iops : min= 17, max= 68, avg=40.16, stdev=10.87, samples=61 write: IOPS=41, BW=167KiB/s (171kB/s)(5180KiB/31063msec); 0 zone resets slat (usec): min=50, max=11094, avg=293.19, stdev=646.90 clat (msec): min=36, max=378, avg=189.07, stdev=49.20 lat (msec): min=36, max=378, avg=189.37, stdev=49.24 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 123], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 171], 50.00th=[ 184], 60.00th=[ 197], | 70.00th=[ 209], 80.00th=[ 228], 90.00th=[ 257], 95.00th=[ 284], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 372], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 71, max= 261, per=24.70%, avg=165.95, stdev=45.30, samples=61 iops : min= 17, max= 65, avg=41.07, stdev=11.40, samples=61 lat (msec) : 20=0.04%, 50=0.08%, 100=1.25%, 250=88.28%, 500=10.35% cpu : usr=0.24%, sys=2.03%, ctx=4703, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=646KiB/s (661kB/s), 160KiB/s-163KiB/s (164kB/s-167kB/s), io=19.6MiB (20.5MB), run=30744-31085msec WRITE: bw=672KiB/s (688kB/s), 167KiB/s-171KiB/s (171kB/s-175kB/s), io=20.4MiB (21.4MB), run=30744-31085msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180025: Wed Aug 19 02:45:36 2026 read: IOPS=1, BW=67.6MiB/s (70.8MB/s)(40.0MiB/592msec) slat (nsec): min=40657k, max=40657k, avg=40656664.00, stdev= 0.00 clat (nsec): min=527466k, max=527466k, avg=527465892.00, stdev= 0.00 lat (nsec): min=568127k, max=568127k, avg=568127328.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.35%, ctx=11, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.6MiB/s (70.8MB/s), 67.6MiB/s-67.6MiB/s (70.8MB/s-70.8MB/s), io=40.0MiB (41.9MB), run=592-592msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout