== sanity test 398c: run fio to test AIO ================= 03:57:05 (1790927825) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.381906 s, 110 MB/s osc.lustre-OST0000-osc-ffff8b16a0524800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180186: Fri Oct 2 03:57:41 2026 write: IOPS=96, BW=387KiB/s (396kB/s)(10.0MiB/26463msec); 0 zone resets slat (usec): min=27, max=21032, avg=188.24, stdev=709.90 clat (msec): min=32, max=422, avg=164.35, stdev=34.11 lat (msec): min=32, max=422, avg=164.54, stdev=34.12 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 110], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 207], 95.00th=[ 226], | 99.00th=[ 247], 99.50th=[ 279], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 422] bw ( KiB/s): min= 254, max= 542, per=25.06%, avg=387.37, stdev=75.19, samples=52 iops : min= 63, max= 135, avg=96.46, stdev=18.79, samples=52 lat (msec) : 50=0.04%, 100=1.76%, 250=97.30%, 500=0.90% cpu : usr=0.15%, sys=0.86%, ctx=2212, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180187: Fri Oct 2 03:57:41 2026 write: IOPS=97, BW=389KiB/s (398kB/s)(10.0MiB/26316msec); 0 zone resets slat (usec): min=28, max=17439, avg=238.06, stdev=750.68 clat (msec): min=20, max=365, avg=164.05, stdev=35.67 lat (msec): min=21, max=366, avg=164.29, stdev=35.66 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 108], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 211], 95.00th=[ 228], | 99.00th=[ 255], 99.50th=[ 271], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 263, max= 638, per=25.15%, avg=388.83, stdev=68.89, samples=52 iops : min= 65, max= 159, avg=96.85, stdev=17.20, samples=52 lat (msec) : 50=0.12%, 100=2.34%, 250=96.13%, 500=1.41% cpu : usr=0.22%, sys=1.16%, ctx=2813, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180188: Fri Oct 2 03:57:41 2026 write: IOPS=96, BW=387KiB/s (396kB/s)(10.0MiB/26473msec); 0 zone resets slat (usec): min=28, max=8974, avg=204.58, stdev=614.72 clat (msec): min=43, max=350, avg=164.34, stdev=34.75 lat (msec): min=44, max=350, avg=164.54, stdev=34.76 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 211], 95.00th=[ 228], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 252, max= 638, per=25.03%, avg=386.94, stdev=74.06, samples=52 iops : min= 63, max= 159, avg=96.37, stdev=18.49, samples=52 lat (msec) : 50=0.08%, 100=1.29%, 250=96.91%, 500=1.72% cpu : usr=0.17%, sys=0.92%, ctx=2793, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180189: Fri Oct 2 03:57:41 2026 write: IOPS=96, BW=387KiB/s (396kB/s)(10.0MiB/26478msec); 0 zone resets slat (usec): min=27, max=11573, avg=233.64, stdev=726.41 clat (msec): min=20, max=384, avg=164.37, stdev=34.41 lat (msec): min=20, max=385, avg=164.61, stdev=34.41 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 114], 10.00th=[ 125], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 169], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 209], 95.00th=[ 224], | 99.00th=[ 253], 99.50th=[ 271], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 262, max= 630, per=25.10%, avg=388.06, stdev=68.76, samples=52 iops : min= 65, max= 157, avg=96.63, stdev=17.15, samples=52 lat (msec) : 50=0.08%, 100=1.48%, 250=97.42%, 500=1.02% cpu : usr=0.17%, sys=1.05%, ctx=3067, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1547KiB/s (1584kB/s), 387KiB/s-389KiB/s (396kB/s-398kB/s), io=40.0MiB (41.9MB), run=26316-26478msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180203: Fri Oct 2 03:58:30 2026 read: IOPS=31, BW=125KiB/s (128kB/s)(5048KiB/40443msec) slat (msec): min=2, max=115, avg=18.03, stdev=11.87 clat (msec): min=93, max=541, avg=239.78, stdev=74.44 lat (msec): min=112, max=577, avg=257.82, stdev=77.91 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 178], | 30.00th=[ 194], 40.00th=[ 209], 50.00th=[ 226], 60.00th=[ 249], | 70.00th=[ 271], 80.00th=[ 300], 90.00th=[ 338], 95.00th=[ 380], | 99.00th=[ 477], 99.50th=[ 493], 99.90th=[ 523], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 36, max= 213, per=25.01%, avg=124.02, stdev=34.09, samples=80 iops : min= 9, max= 53, avg=30.53, stdev= 8.62, samples=80 write: IOPS=32, BW=128KiB/s (131kB/s)(5192KiB/40443msec); 0 zone resets slat (usec): min=33, max=36932, avg=257.13, stdev=1428.94 clat (msec): min=107, max=521, avg=243.83, stdev=73.69 lat (msec): min=107, max=521, avg=244.09, stdev=73.71 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 153], 10.00th=[ 165], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 211], 50.00th=[ 228], 60.00th=[ 247], | 70.00th=[ 275], 80.00th=[ 305], 90.00th=[ 347], 95.00th=[ 384], | 99.00th=[ 456], 99.50th=[ 481], 99.90th=[ 514], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 29, max= 247, per=24.79%, avg=127.93, stdev=44.04, samples=80 iops : min= 7, max= 61, avg=31.49, stdev=11.05, samples=80 lat (msec) : 100=0.04%, 250=61.21%, 500=38.59%, 750=0.16% cpu : usr=0.29%, sys=1.53%, ctx=4371, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180204: Fri Oct 2 03:58:30 2026 read: IOPS=31, BW=124KiB/s (127kB/s)(4976KiB/40004msec) slat (msec): min=2, max=123, avg=18.29, stdev=12.46 clat (msec): min=35, max=671, avg=239.89, stdev=80.24 lat (msec): min=47, max=678, avg=258.18, stdev=84.33 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 190], 40.00th=[ 209], 50.00th=[ 224], 60.00th=[ 247], | 70.00th=[ 271], 80.00th=[ 300], 90.00th=[ 342], 95.00th=[ 388], | 99.00th=[ 514], 99.50th=[ 542], 99.90th=[ 642], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 51, max= 199, per=24.96%, avg=123.82, stdev=35.13, samples=79 iops : min= 12, max= 49, avg=30.46, stdev= 8.88, samples=79 write: IOPS=32, BW=132KiB/s (135kB/s)(5264KiB/40004msec); 0 zone resets slat (usec): min=32, max=18966, avg=240.48, stdev=1063.25 clat (msec): min=96, max=665, avg=240.11, stdev=72.36 lat (msec): min=96, max=671, avg=240.36, stdev=72.44 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 157], 10.00th=[ 167], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 243], | 70.00th=[ 264], 80.00th=[ 292], 90.00th=[ 334], 95.00th=[ 380], | 99.00th=[ 464], 99.50th=[ 485], 99.90th=[ 625], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 55, max= 239, per=25.48%, avg=131.47, stdev=41.63, samples=79 iops : min= 13, max= 59, avg=32.38, stdev=10.46, samples=79 lat (msec) : 50=0.04%, 100=0.16%, 250=62.73%, 500=36.29%, 750=0.78% cpu : usr=0.25%, sys=1.42%, ctx=4040, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180205: Fri Oct 2 03:58:30 2026 read: IOPS=30, BW=123KiB/s (126kB/s)(4984KiB/40445msec) slat (msec): min=2, max=138, avg=18.52, stdev=12.66 clat (msec): min=40, max=608, avg=239.51, stdev=78.46 lat (msec): min=59, max=634, avg=258.04, stdev=82.26 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 138], 10.00th=[ 155], 20.00th=[ 178], | 30.00th=[ 194], 40.00th=[ 209], 50.00th=[ 224], 60.00th=[ 245], | 70.00th=[ 268], 80.00th=[ 300], 90.00th=[ 342], 95.00th=[ 388], | 99.00th=[ 472], 99.50th=[ 493], 99.90th=[ 575], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 39, max= 244, per=24.76%, avg=122.81, stdev=38.78, samples=80 iops : min= 9, max= 61, avg=30.21, stdev= 9.75, samples=80 write: IOPS=32, BW=130KiB/s (133kB/s)(5256KiB/40445msec); 0 zone resets slat (usec): min=31, max=20361, avg=252.23, stdev=1116.87 clat (msec): min=64, max=525, avg=245.63, stdev=71.99 lat (msec): min=64, max=525, avg=245.88, stdev=71.99 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 148], 10.00th=[ 169], 20.00th=[ 184], | 30.00th=[ 201], 40.00th=[ 218], 50.00th=[ 234], 60.00th=[ 251], | 70.00th=[ 275], 80.00th=[ 305], 90.00th=[ 342], 95.00th=[ 380], | 99.00th=[ 460], 99.50th=[ 489], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 39, max= 231, per=25.00%, avg=128.97, stdev=39.58, samples=80 iops : min= 9, max= 57, avg=31.75, stdev= 9.95, samples=80 lat (msec) : 50=0.04%, 100=0.43%, 250=60.51%, 500=38.63%, 750=0.39% cpu : usr=0.26%, sys=1.56%, ctx=4221, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180206: Fri Oct 2 03:58:30 2026 read: IOPS=31, BW=127KiB/s (130kB/s)(5060KiB/39948msec) slat (usec): min=49, max=101432, avg=17883.78, stdev=11750.83 clat (msec): min=12, max=621, avg=238.10, stdev=79.22 lat (msec): min=12, max=634, avg=255.99, stdev=82.91 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 207], 50.00th=[ 222], 60.00th=[ 241], | 70.00th=[ 264], 80.00th=[ 288], 90.00th=[ 338], 95.00th=[ 380], | 99.00th=[ 523], 99.50th=[ 542], 99.90th=[ 609], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 31, max= 223, per=25.40%, avg=126.00, stdev=40.45, samples=79 iops : min= 7, max= 55, avg=31.03, stdev=10.17, samples=79 write: IOPS=32, BW=130KiB/s (133kB/s)(5180KiB/39948msec); 0 zone resets slat (usec): min=31, max=22229, avg=223.16, stdev=891.54 clat (msec): min=91, max=776, avg=242.64, stdev=75.36 lat (msec): min=91, max=776, avg=242.86, stdev=75.35 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 186], | 30.00th=[ 199], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 247], | 70.00th=[ 268], 80.00th=[ 296], 90.00th=[ 342], 95.00th=[ 384], | 99.00th=[ 485], 99.50th=[ 498], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 47, max= 207, per=25.03%, avg=129.18, stdev=39.47, samples=79 iops : min= 11, max= 51, avg=31.78, stdev= 9.95, samples=79 lat (msec) : 20=0.04%, 50=0.12%, 100=0.27%, 250=62.93%, 500=35.66% lat (msec) : 750=0.90%, 1000=0.08% cpu : usr=0.28%, sys=1.45%, ctx=4222, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=496KiB/s (508kB/s), 123KiB/s-127KiB/s (126kB/s-130kB/s), io=19.6MiB (20.5MB), run=39948-40445msec WRITE: bw=517KiB/s (529kB/s), 128KiB/s-132KiB/s (131kB/s-135kB/s), io=20.4MiB (21.4MB), run=39948-40445msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180217: Fri Oct 2 03:58:46 2026 read: IOPS=1, BW=45.5MiB/s (47.7MB/s)(40.0MiB/880msec) slat (nsec): min=59331k, max=59331k, avg=59331136.00, stdev= 0.00 clat (nsec): min=794557k, max=794557k, avg=794557434.00, stdev= 0.00 lat (nsec): min=853921k, max=853921k, avg=853920682.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 793], 5.00th=[ 793], 10.00th=[ 793], 20.00th=[ 793], | 30.00th=[ 793], 40.00th=[ 793], 50.00th=[ 793], 60.00th=[ 793], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=1.48%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=40.0MiB (41.9MB), run=880-880msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout