== sanity test 398c: run fio to test AIO ================= 12:32:42 (1783960362) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.560895 s, 74.8 MB/s osc.lustre-OST0000-osc-ffff8913e0ba7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179200: Mon Jul 13 12:33:27 2026 write: IOPS=73, BW=293KiB/s (300kB/s)(10.0MiB/34925msec); 0 zone resets slat (usec): min=35, max=24854, avg=312.87, stdev=1147.90 clat (msec): min=15, max=495, avg=216.58, stdev=87.16 lat (msec): min=16, max=495, avg=216.89, stdev=87.19 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 115], 10.00th=[ 128], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 194], 60.00th=[ 218], | 70.00th=[ 239], 80.00th=[ 284], 90.00th=[ 359], 95.00th=[ 401], | 99.00th=[ 451], 99.50th=[ 485], 99.90th=[ 493], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 123, max= 520, per=24.94%, avg=292.33, stdev=102.40, samples=69 iops : min= 30, max= 130, avg=72.62, stdev=25.62, samples=69 lat (msec) : 20=0.08%, 50=0.08%, 100=2.15%, 250=70.43%, 500=27.27% cpu : usr=0.11%, sys=1.00%, ctx=2193, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179201: Mon Jul 13 12:33:27 2026 write: IOPS=73, BW=294KiB/s (301kB/s)(10.0MiB/34781msec); 0 zone resets slat (usec): min=35, max=17225, avg=291.12, stdev=871.40 clat (msec): min=42, max=497, avg=215.95, stdev=87.13 lat (msec): min=42, max=497, avg=216.24, stdev=87.17 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 110], 10.00th=[ 125], 20.00th=[ 144], | 30.00th=[ 161], 40.00th=[ 180], 50.00th=[ 199], 60.00th=[ 218], | 70.00th=[ 243], 80.00th=[ 284], 90.00th=[ 355], 95.00th=[ 401], | 99.00th=[ 451], 99.50th=[ 477], 99.90th=[ 489], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 126, max= 608, per=25.01%, avg=293.10, stdev=109.43, samples=68 iops : min= 31, max= 152, avg=72.81, stdev=27.39, samples=68 lat (msec) : 50=0.08%, 100=3.71%, 250=69.30%, 500=26.91% cpu : usr=0.12%, sys=0.93%, ctx=2227, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179202: Mon Jul 13 12:33:27 2026 write: IOPS=74, BW=297KiB/s (304kB/s)(10.0MiB/34472msec); 0 zone resets slat (usec): min=39, max=40968, avg=421.18, stdev=1663.67 clat (msec): min=16, max=487, avg=214.85, stdev=86.76 lat (msec): min=16, max=488, avg=215.28, stdev=86.80 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 106], 10.00th=[ 123], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 194], 60.00th=[ 218], | 70.00th=[ 243], 80.00th=[ 279], 90.00th=[ 359], 95.00th=[ 397], | 99.00th=[ 460], 99.50th=[ 464], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 123, max= 528, per=25.22%, avg=295.63, stdev=105.25, samples=68 iops : min= 30, max= 132, avg=73.46, stdev=26.35, samples=68 lat (msec) : 20=0.08%, 50=0.08%, 100=3.71%, 250=68.40%, 500=27.73% cpu : usr=0.21%, sys=1.32%, ctx=2715, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179203: Mon Jul 13 12:33:27 2026 write: IOPS=73, BW=293KiB/s (300kB/s)(10.0MiB/34909msec); 0 zone resets slat (usec): min=49, max=37825, avg=355.45, stdev=1349.18 clat (msec): min=28, max=492, avg=216.43, stdev=86.15 lat (msec): min=28, max=492, avg=216.79, stdev=86.20 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 113], 10.00th=[ 127], 20.00th=[ 146], | 30.00th=[ 163], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 218], | 70.00th=[ 239], 80.00th=[ 279], 90.00th=[ 359], 95.00th=[ 393], | 99.00th=[ 451], 99.50th=[ 477], 99.90th=[ 489], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 123, max= 616, per=24.94%, avg=292.32, stdev=105.70, samples=69 iops : min= 30, max= 154, avg=72.62, stdev=26.45, samples=69 lat (msec) : 50=0.04%, 100=2.81%, 250=70.62%, 500=26.52% cpu : usr=0.17%, sys=1.09%, ctx=2713, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1173KiB/s (1201kB/s), 293KiB/s-297KiB/s (300kB/s-304kB/s), io=40.0MiB (41.9MB), run=34472-34925msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179218: Mon Jul 13 12:34:13 2026 read: IOPS=37, BW=150KiB/s (154kB/s)(5048KiB/33668msec) slat (usec): min=2806, max=65296, avg=14900.00, stdev=8449.03 clat (msec): min=63, max=406, avg=196.67, stdev=49.77 lat (msec): min=68, max=428, avg=211.58, stdev=52.44 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 122], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 264], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 347], 99.90th=[ 384], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 55, max= 238, per=25.17%, avg=149.27, stdev=39.64, samples=66 iops : min= 13, max= 59, avg=36.83, stdev= 9.92, samples=66 write: IOPS=38, BW=154KiB/s (158kB/s)(5192KiB/33668msec); 0 zone resets slat (usec): min=51, max=11066, avg=347.47, stdev=860.30 clat (msec): min=46, max=420, avg=204.57, stdev=50.54 lat (msec): min=46, max=420, avg=204.92, stdev=50.56 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 130], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 228], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 351], 99.50th=[ 376], 99.90th=[ 418], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 40, max= 294, per=24.96%, avg=153.98, stdev=40.41, samples=66 iops : min= 10, max= 73, avg=38.02, stdev=10.11, samples=66 lat (msec) : 50=0.04%, 100=1.48%, 250=82.70%, 500=15.78% cpu : usr=0.28%, sys=2.17%, ctx=4494, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179219: Mon Jul 13 12:34:13 2026 read: IOPS=37, BW=149KiB/s (152kB/s)(4976KiB/33492msec) slat (usec): min=94, max=74580, avg=15129.07, stdev=8592.36 clat (msec): min=22, max=487, avg=197.21, stdev=52.81 lat (msec): min=23, max=509, avg=212.35, stdev=55.61 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 122], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 207], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 334], 99.50th=[ 355], 99.90th=[ 426], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 71, max= 333, per=24.95%, avg=147.95, stdev=42.01, samples=66 iops : min= 17, max= 83, avg=36.50, stdev=10.52, samples=66 write: IOPS=39, BW=157KiB/s (161kB/s)(5264KiB/33492msec); 0 zone resets slat (usec): min=46, max=17968, avg=384.03, stdev=1062.44 clat (msec): min=61, max=445, avg=205.43, stdev=53.85 lat (msec): min=61, max=445, avg=205.82, stdev=53.80 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 130], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 351], 99.50th=[ 388], 99.90th=[ 414], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 77, max= 272, per=25.46%, avg=157.09, stdev=40.65, samples=66 iops : min= 19, max= 68, avg=38.80, stdev=10.23, samples=66 lat (msec) : 50=0.04%, 100=1.29%, 250=81.80%, 500=16.88% cpu : usr=0.24%, sys=2.31%, ctx=4484, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179220: Mon Jul 13 12:34:13 2026 read: IOPS=36, BW=148KiB/s (151kB/s)(4984KiB/33781msec) slat (usec): min=3083, max=57940, avg=15229.38, stdev=8621.92 clat (msec): min=66, max=396, avg=198.08, stdev=49.80 lat (msec): min=76, max=420, avg=213.32, stdev=52.55 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 125], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 264], 95.00th=[ 284], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 47, max= 230, per=24.81%, avg=147.15, stdev=38.94, samples=66 iops : min= 11, max= 57, avg=36.29, stdev= 9.79, samples=66 write: IOPS=38, BW=156KiB/s (159kB/s)(5256KiB/33781msec); 0 zone resets slat (usec): min=41, max=18443, avg=334.03, stdev=824.82 clat (msec): min=80, max=391, avg=204.85, stdev=50.05 lat (msec): min=80, max=391, avg=205.19, stdev=50.12 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 321], 99.50th=[ 347], 99.90th=[ 376], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 67, max= 284, per=24.99%, avg=154.21, stdev=42.43, samples=66 iops : min= 16, max= 71, avg=38.08, stdev=10.69, samples=66 lat (msec) : 100=1.41%, 250=81.21%, 500=17.38% cpu : usr=0.27%, sys=2.10%, ctx=4638, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179221: Mon Jul 13 12:34:13 2026 read: IOPS=37, BW=150KiB/s (153kB/s)(5060KiB/33807msec) slat (usec): min=3121, max=62931, avg=15221.88, stdev=8793.60 clat (msec): min=30, max=387, avg=196.56, stdev=51.04 lat (msec): min=38, max=411, avg=211.79, stdev=53.69 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 121], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 342], 99.90th=[ 376], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 33, max= 278, per=25.09%, avg=148.80, stdev=40.86, samples=66 iops : min= 8, max= 69, avg=36.73, stdev=10.29, samples=66 write: IOPS=38, BW=153KiB/s (157kB/s)(5180KiB/33807msec); 0 zone resets slat (usec): min=61, max=12936, avg=347.89, stdev=843.30 clat (msec): min=50, max=365, avg=205.96, stdev=54.10 lat (msec): min=50, max=365, avg=206.32, stdev=54.11 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 125], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 232], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 338], 99.50th=[ 347], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 47, max= 286, per=24.66%, avg=152.14, stdev=42.27, samples=66 iops : min= 11, max= 71, avg=37.55, stdev=10.59, samples=66 lat (msec) : 50=0.04%, 100=1.48%, 250=80.62%, 500=17.85% cpu : usr=0.27%, sys=2.28%, ctx=4777, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=594KiB/s (608kB/s), 148KiB/s-150KiB/s (151kB/s-154kB/s), io=19.6MiB (20.5MB), run=33492-33807msec WRITE: bw=618KiB/s (633kB/s), 153KiB/s-157KiB/s (157kB/s-161kB/s), io=20.4MiB (21.4MB), run=33492-33807msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179231: Mon Jul 13 12:34:25 2026 read: IOPS=1, BW=79.1MiB/s (82.9MB/s)(40.0MiB/506msec) slat (nsec): min=35105k, max=35105k, avg=35105422.00, stdev= 0.00 clat (nsec): min=434880k, max=434880k, avg=434879841.00, stdev= 0.00 lat (nsec): min=469991k, max=469991k, avg=469991197.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.57%, ctx=13, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=79.1MiB/s (82.9MB/s), 79.1MiB/s-79.1MiB/s (82.9MB/s-82.9MB/s), io=40.0MiB (41.9MB), run=506-506msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout