== sanity test 398c: run fio to test AIO ================= 13:12:53 (1746119573) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.13158 s, 319 MB/s osc.lustre-OST0000-osc-ffff88009b4f9800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=13238: Thu May 1 13:13:48 2025 write: IOPS=48, BW=193KiB/s (198kB/s)(10.0MiB/53032msec) slat (usec): min=15, max=525, avg=103.09, stdev=56.53 clat (msec): min=75, max=521, avg=330.97, stdev=73.82 lat (msec): min=75, max=521, avg=331.08, stdev=73.83 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 192], 10.00th=[ 222], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 338], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 397], 90.00th=[ 418], 95.00th=[ 435], | 99.00th=[ 472], 99.50th=[ 481], 99.90th=[ 506], 99.95th=[ 514], | 99.99th=[ 523] bw ( KiB/s): min= 127, max= 384, per=24.87%, avg=191.97, stdev=46.78, samples=106 iops : min= 31, max= 96, avg=47.92, stdev=11.73, samples=106 lat (msec) : 100=0.62%, 250=13.48%, 500=85.78%, 750=0.12% cpu : usr=0.13%, sys=0.60%, ctx=1869, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=13239: Thu May 1 13:13:48 2025 write: IOPS=48, BW=193KiB/s (198kB/s)(10.0MiB/53031msec) slat (usec): min=16, max=1167, avg=108.45, stdev=66.38 clat (msec): min=87, max=514, avg=331.06, stdev=73.60 lat (msec): min=87, max=514, avg=331.17, stdev=73.60 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 194], 10.00th=[ 220], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 338], 60.00th=[ 359], | 70.00th=[ 376], 80.00th=[ 393], 90.00th=[ 418], 95.00th=[ 435], | 99.00th=[ 468], 99.50th=[ 481], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 514] bw ( KiB/s): min= 128, max= 384, per=24.86%, avg=191.92, stdev=42.75, samples=106 iops : min= 32, max= 96, avg=47.87, stdev=10.69, samples=106 lat (msec) : 100=0.62%, 250=13.67%, 500=85.66%, 750=0.04% cpu : usr=0.13%, sys=0.61%, ctx=2127, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=13240: Thu May 1 13:13:48 2025 write: IOPS=48, BW=193KiB/s (198kB/s)(10.0MiB/53010msec) slat (usec): min=16, max=983, avg=106.64, stdev=62.39 clat (msec): min=75, max=521, avg=330.82, stdev=73.80 lat (msec): min=75, max=521, avg=330.93, stdev=73.81 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 192], 10.00th=[ 228], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 338], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 393], 90.00th=[ 418], 95.00th=[ 439], | 99.00th=[ 477], 99.50th=[ 485], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 523] bw ( KiB/s): min= 128, max= 384, per=24.87%, avg=191.97, stdev=45.89, samples=106 iops : min= 32, max= 96, avg=47.92, stdev=11.52, samples=106 lat (msec) : 100=0.62%, 250=13.01%, 500=86.21%, 750=0.16% cpu : usr=0.11%, sys=0.63%, ctx=1972, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=13241: Thu May 1 13:13:48 2025 write: IOPS=48, BW=193KiB/s (198kB/s)(10.0MiB/52987msec) slat (usec): min=16, max=623, avg=104.85, stdev=57.37 clat (msec): min=46, max=511, avg=330.79, stdev=74.24 lat (msec): min=46, max=511, avg=330.90, stdev=74.24 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 190], 10.00th=[ 226], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 359], | 70.00th=[ 376], 80.00th=[ 393], 90.00th=[ 418], 95.00th=[ 435], | 99.00th=[ 472], 99.50th=[ 481], 99.90th=[ 498], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 127, max= 384, per=24.88%, avg=192.07, stdev=46.30, samples=105 iops : min= 31, max= 96, avg=47.95, stdev=11.61, samples=105 lat (msec) : 50=0.35%, 100=0.47%, 250=12.93%, 500=86.17%, 750=0.08% cpu : usr=0.12%, sys=0.62%, ctx=1981, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=772KiB/s (791kB/s), 193KiB/s-193KiB/s (198kB/s-198kB/s), io=40.0MiB (41.9MB), run=52987-53032msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13294: Thu May 1 13:14:22 2025 read: IOPS=38, BW=154KiB/s (157kB/s)(5048KiB/32876msec) slat (usec): min=960, max=4461, avg=2102.10, stdev=501.95 clat (msec): min=12, max=308, avg=132.01, stdev=28.08 lat (msec): min=14, max=310, avg=134.11, stdev=28.20 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 91], 10.00th=[ 102], 20.00th=[ 111], | 30.00th=[ 118], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 153], 90.00th=[ 165], 95.00th=[ 176], | 99.00th=[ 205], 99.50th=[ 211], 99.90th=[ 284], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 72, max= 224, per=25.17%, avg=153.29, stdev=41.32, samples=65 iops : min= 18, max= 56, avg=38.18, stdev=10.29, samples=65 write: IOPS=39, BW=158KiB/s (162kB/s)(5192KiB/32876msec) slat (usec): min=15, max=394, avg=104.63, stdev=51.30 clat (msec): min=114, max=476, avg=274.21, stdev=62.01 lat (msec): min=114, max=476, avg=274.32, stdev=62.01 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 174], 10.00th=[ 192], 20.00th=[ 222], | 30.00th=[ 243], 40.00th=[ 259], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 359], 95.00th=[ 380], | 99.00th=[ 422], 99.50th=[ 447], 99.90th=[ 468], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 72, max= 239, per=24.80%, avg=157.23, stdev=30.30, samples=65 iops : min= 18, max= 59, avg=39.17, stdev= 7.50, samples=65 lat (msec) : 20=0.23%, 50=0.23%, 100=4.26%, 250=62.58%, 500=32.70% cpu : usr=0.22%, sys=1.38%, ctx=3425, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13295: Thu May 1 13:14:22 2025 read: IOPS=37, BW=151KiB/s (155kB/s)(4976KiB/32912msec) slat (usec): min=823, max=4514, avg=2103.58, stdev=453.89 clat (msec): min=15, max=214, avg=132.79, stdev=27.05 lat (msec): min=16, max=216, avg=134.89, stdev=27.15 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 92], 10.00th=[ 101], 20.00th=[ 112], | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 167], 95.00th=[ 180], | 99.00th=[ 197], 99.50th=[ 205], 99.90th=[ 211], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 56, max= 240, per=24.63%, avg=149.97, stdev=42.16, samples=65 iops : min= 14, max= 60, avg=37.35, stdev=10.53, samples=65 write: IOPS=39, BW=160KiB/s (164kB/s)(5264KiB/32912msec) slat (usec): min=20, max=387, avg=96.36, stdev=44.55 clat (msec): min=89, max=466, avg=272.07, stdev=64.57 lat (msec): min=90, max=466, avg=272.17, stdev=64.58 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 167], 10.00th=[ 188], 20.00th=[ 218], | 30.00th=[ 239], 40.00th=[ 253], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 380], | 99.00th=[ 422], 99.50th=[ 439], 99.90th=[ 456], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 103, max= 239, per=24.99%, avg=158.46, stdev=32.88, samples=65 iops : min= 25, max= 59, avg=39.48, stdev= 8.18, samples=65 lat (msec) : 20=0.16%, 50=0.20%, 100=4.34%, 250=63.52%, 500=31.80% cpu : usr=0.17%, sys=1.40%, ctx=2945, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13296: Thu May 1 13:14:22 2025 read: IOPS=37, BW=151KiB/s (155kB/s)(4984KiB/32912msec) slat (usec): min=960, max=4587, avg=2116.22, stdev=492.85 clat (msec): min=16, max=246, avg=132.21, stdev=27.09 lat (msec): min=18, max=248, avg=134.33, stdev=27.22 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 92], 10.00th=[ 100], 20.00th=[ 112], | 30.00th=[ 120], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 165], 95.00th=[ 176], | 99.00th=[ 192], 99.50th=[ 197], 99.90th=[ 215], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 80, max= 296, per=24.91%, avg=151.69, stdev=46.32, samples=65 iops : min= 20, max= 74, avg=37.78, stdev=11.57, samples=65 write: IOPS=39, BW=160KiB/s (164kB/s)(5256KiB/32912msec) slat (usec): min=20, max=1667, avg=110.46, stdev=82.66 clat (msec): min=108, max=466, avg=272.63, stdev=61.84 lat (msec): min=108, max=467, avg=272.75, stdev=61.85 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 174], 10.00th=[ 190], 20.00th=[ 220], | 30.00th=[ 239], 40.00th=[ 255], 50.00th=[ 271], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 355], 95.00th=[ 380], | 99.00th=[ 422], 99.50th=[ 443], 99.90th=[ 460], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 96, max= 247, per=24.94%, avg=158.09, stdev=32.15, samples=65 iops : min= 24, max= 61, avg=39.38, stdev= 7.98, samples=65 lat (msec) : 20=0.20%, 50=0.12%, 100=4.84%, 250=62.93%, 500=31.91% cpu : usr=0.20%, sys=1.38%, ctx=3529, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13297: Thu May 1 13:14:22 2025 read: IOPS=38, BW=155KiB/s (159kB/s)(5060KiB/32643msec) slat (usec): min=79, max=4951, avg=2097.96, stdev=510.92 clat (usec): min=1819, max=351881, avg=132624.01, stdev=29034.85 lat (msec): min=3, max=353, avg=134.73, stdev=29.18 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 94], 10.00th=[ 102], 20.00th=[ 112], | 30.00th=[ 120], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 167], 95.00th=[ 180], | 99.00th=[ 194], 99.50th=[ 205], 99.90th=[ 292], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 48, max= 247, per=25.45%, avg=155.02, stdev=38.47, samples=65 iops : min= 12, max= 61, avg=38.62, stdev= 9.63, samples=65 write: IOPS=39, BW=159KiB/s (162kB/s)(5180KiB/32643msec) slat (usec): min=19, max=437, avg=101.31, stdev=46.90 clat (msec): min=34, max=509, avg=271.29, stdev=66.28 lat (msec): min=34, max=509, avg=271.40, stdev=66.28 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 165], 10.00th=[ 188], 20.00th=[ 215], | 30.00th=[ 239], 40.00th=[ 253], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 355], 95.00th=[ 376], | 99.00th=[ 439], 99.50th=[ 451], 99.90th=[ 493], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 104, max= 247, per=24.90%, avg=157.85, stdev=31.93, samples=65 iops : min= 26, max= 61, avg=39.32, stdev= 7.91, samples=65 lat (msec) : 2=0.04%, 4=0.08%, 10=0.16%, 50=0.43%, 100=4.34% lat (msec) : 250=63.67%, 500=31.25%, 750=0.04% cpu : usr=0.23%, sys=1.39%, ctx=3172, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=610KiB/s (624kB/s), 151KiB/s-155KiB/s (155kB/s-159kB/s), io=19.6MiB (20.5MB), run=32643-32912msec WRITE: bw=635KiB/s (650kB/s), 158KiB/s-160KiB/s (162kB/s-164kB/s), io=20.4MiB (21.4MB), run=32643-32912msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=13327: Thu May 1 13:14:24 2025 read: IOPS=5, BW=237MiB/s (248MB/s)(40.0MiB/169msec) slat (nsec): min=6367.5k, max=6367.5k, avg=6367454.00, stdev= 0.00 clat (nsec): min=156197k, max=156197k, avg=156196872.00, stdev= 0.00 lat (nsec): min=162576k, max=162576k, avg=162576334.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.17%, ctx=14, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=237MiB/s (248MB/s), 237MiB/s-237MiB/s (248MB/s-248MB/s), io=40.0MiB (41.9MB), run=169-169msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout