== sanity test 398c: run fio to test AIO ================= 21:05:38 (1745629538) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.146851 s, 286 MB/s osc.lustre-OST0000-osc-ffff8800863eb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=11403: Fri Apr 25 21:06:35 2025 write: IOPS=45, BW=184KiB/s (188kB/s)(10.0MiB/55734msec) slat (usec): min=13, max=351, avg=90.17, stdev=42.77 clat (msec): min=72, max=617, avg=347.87, stdev=69.44 lat (msec): min=72, max=617, avg=347.96, stdev=69.45 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 228], 10.00th=[ 257], 20.00th=[ 296], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 368], | 70.00th=[ 384], 80.00th=[ 405], 90.00th=[ 430], 95.00th=[ 451], | 99.00th=[ 493], 99.50th=[ 518], 99.90th=[ 550], 99.95th=[ 592], | 99.99th=[ 617] bw ( KiB/s): min= 127, max= 360, per=24.97%, avg=183.29, stdev=38.14, samples=111 iops : min= 31, max= 90, avg=45.72, stdev= 9.56, samples=111 lat (msec) : 100=0.70%, 250=8.32%, 500=90.23%, 750=0.74% cpu : usr=0.12%, sys=0.52%, ctx=1743, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=11404: Fri Apr 25 21:06:35 2025 write: IOPS=45, BW=184KiB/s (188kB/s)(10.0MiB/55734msec) slat (usec): min=14, max=531, avg=91.30, stdev=46.87 clat (msec): min=87, max=546, avg=347.91, stdev=68.43 lat (msec): min=87, max=546, avg=348.00, stdev=68.44 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 228], 10.00th=[ 255], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 368], | 70.00th=[ 384], 80.00th=[ 405], 90.00th=[ 430], 95.00th=[ 451], | 99.00th=[ 493], 99.50th=[ 510], 99.90th=[ 535], 99.95th=[ 542], | 99.99th=[ 550] bw ( KiB/s): min= 96, max= 384, per=24.90%, avg=182.74, stdev=39.44, samples=111 iops : min= 24, max= 96, avg=45.55, stdev= 9.88, samples=111 lat (msec) : 100=0.62%, 250=8.44%, 500=90.27%, 750=0.66% cpu : usr=0.10%, sys=0.53%, ctx=1784, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=11405: Fri Apr 25 21:06:35 2025 write: IOPS=46, BW=184KiB/s (188kB/s)(10.0MiB/55645msec) slat (usec): min=16, max=376, avg=93.13, stdev=43.18 clat (msec): min=37, max=640, avg=347.42, stdev=71.87 lat (msec): min=38, max=640, avg=347.51, stdev=71.87 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 228], 10.00th=[ 255], 20.00th=[ 296], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 368], | 70.00th=[ 384], 80.00th=[ 405], 90.00th=[ 430], 95.00th=[ 451], | 99.00th=[ 502], 99.50th=[ 523], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 119, max= 360, per=24.98%, avg=183.38, stdev=39.31, samples=111 iops : min= 29, max= 90, avg=45.81, stdev= 9.85, samples=111 lat (msec) : 50=0.04%, 100=0.78%, 250=7.97%, 500=90.20%, 750=1.02% cpu : usr=0.11%, sys=0.56%, ctx=1817, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=11406: Fri Apr 25 21:06:35 2025 write: IOPS=45, BW=184KiB/s (188kB/s)(10.0MiB/55713msec) slat (usec): min=14, max=710, avg=90.94, stdev=46.50 clat (msec): min=100, max=573, avg=347.82, stdev=68.42 lat (msec): min=100, max=573, avg=347.91, stdev=68.43 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 230], 10.00th=[ 259], 20.00th=[ 296], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 368], | 70.00th=[ 384], 80.00th=[ 405], 90.00th=[ 430], 95.00th=[ 451], | 99.00th=[ 502], 99.50th=[ 514], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 127, max= 384, per=24.97%, avg=183.30, stdev=37.97, samples=111 iops : min= 31, max= 96, avg=45.73, stdev= 9.50, samples=111 lat (msec) : 250=8.48%, 500=90.51%, 750=1.02% cpu : usr=0.08%, sys=0.54%, ctx=1784, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=735KiB/s (753kB/s), 184KiB/s-184KiB/s (188kB/s-188kB/s), io=40.0MiB (41.9MB), run=55645-55734msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11460: Fri Apr 25 21:07:07 2025 read: IOPS=41, BW=166KiB/s (170kB/s)(5048KiB/30399msec) slat (usec): min=902, max=5365, avg=1936.34, stdev=535.11 clat (usec): min=1542, max=202381, avg=122982.40, stdev=29243.46 lat (msec): min=2, max=204, avg=124.92, stdev=29.43 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 75], 10.00th=[ 86], 20.00th=[ 100], | 30.00th=[ 109], 40.00th=[ 116], 50.00th=[ 124], 60.00th=[ 131], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 161], 95.00th=[ 171], | 99.00th=[ 190], 99.50th=[ 194], 99.90th=[ 201], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 111, max= 288, per=24.97%, avg=164.80, stdev=39.59, samples=60 iops : min= 27, max= 72, avg=41.10, stdev= 9.93, samples=60 write: IOPS=42, BW=171KiB/s (175kB/s)(5192KiB/30399msec) slat (usec): min=13, max=1364, avg=78.10, stdev=52.03 clat (msec): min=94, max=431, avg=252.63, stdev=57.11 lat (msec): min=94, max=431, avg=252.71, stdev=57.12 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 161], 10.00th=[ 178], 20.00th=[ 201], | 30.00th=[ 220], 40.00th=[ 239], 50.00th=[ 253], 60.00th=[ 266], | 70.00th=[ 284], 80.00th=[ 300], 90.00th=[ 330], 95.00th=[ 355], | 99.00th=[ 388], 99.50th=[ 401], 99.90th=[ 422], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 104, max= 264, per=24.69%, avg=169.60, stdev=31.47, samples=60 iops : min= 26, max= 66, avg=42.30, stdev= 7.84, samples=60 lat (msec) : 2=0.04%, 10=0.04%, 20=0.08%, 50=0.27%, 100=9.80% lat (msec) : 250=63.63%, 500=26.13% cpu : usr=0.19%, sys=1.26%, ctx=2929, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11461: Fri Apr 25 21:07:07 2025 read: IOPS=40, BW=164KiB/s (168kB/s)(4976KiB/30400msec) slat (usec): min=827, max=4746, avg=1944.35, stdev=512.07 clat (usec): min=1824, max=200508, avg=123391.83, stdev=28427.57 lat (msec): min=3, max=203, avg=125.34, stdev=28.61 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 78], 10.00th=[ 89], 20.00th=[ 100], | 30.00th=[ 108], 40.00th=[ 115], 50.00th=[ 123], 60.00th=[ 131], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 161], 95.00th=[ 169], | 99.00th=[ 184], 99.50th=[ 190], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 80, max= 256, per=24.63%, avg=162.57, stdev=43.17, samples=60 iops : min= 20, max= 64, avg=40.57, stdev=10.80, samples=60 write: IOPS=43, BW=173KiB/s (177kB/s)(5264KiB/30400msec) slat (usec): min=15, max=269, avg=75.46, stdev=34.66 clat (msec): min=79, max=433, avg=250.52, stdev=58.75 lat (msec): min=79, max=433, avg=250.60, stdev=58.75 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 155], 10.00th=[ 176], 20.00th=[ 199], | 30.00th=[ 218], 40.00th=[ 236], 50.00th=[ 251], 60.00th=[ 266], | 70.00th=[ 279], 80.00th=[ 300], 90.00th=[ 330], 95.00th=[ 355], | 99.00th=[ 388], 99.50th=[ 397], 99.90th=[ 418], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 127, max= 272, per=25.04%, avg=172.03, stdev=34.67, samples=60 iops : min= 31, max= 68, avg=42.93, stdev= 8.68, samples=60 lat (msec) : 2=0.04%, 4=0.08%, 20=0.08%, 50=0.08%, 100=10.08% lat (msec) : 250=63.71%, 500=25.94% cpu : usr=0.16%, sys=1.29%, ctx=2924, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11462: Fri Apr 25 21:07:07 2025 read: IOPS=41, BW=164KiB/s (168kB/s)(4984KiB/30322msec) slat (usec): min=782, max=4383, avg=1933.43, stdev=515.12 clat (msec): min=2, max=207, avg=121.70, stdev=28.52 lat (msec): min=4, max=208, avg=123.64, stdev=28.72 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 74], 10.00th=[ 86], 20.00th=[ 97], | 30.00th=[ 109], 40.00th=[ 115], 50.00th=[ 123], 60.00th=[ 130], | 70.00th=[ 136], 80.00th=[ 146], 90.00th=[ 159], 95.00th=[ 167], | 99.00th=[ 190], 99.50th=[ 197], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 80, max= 304, per=24.85%, avg=164.02, stdev=46.00, samples=60 iops : min= 20, max= 76, avg=40.92, stdev=11.50, samples=60 write: IOPS=43, BW=173KiB/s (177kB/s)(5256KiB/30322msec) slat (usec): min=15, max=1526, avg=78.90, stdev=53.84 clat (msec): min=107, max=433, avg=251.48, stdev=55.91 lat (msec): min=107, max=433, avg=251.56, stdev=55.92 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 159], 10.00th=[ 178], 20.00th=[ 203], | 30.00th=[ 224], 40.00th=[ 239], 50.00th=[ 253], 60.00th=[ 266], | 70.00th=[ 279], 80.00th=[ 300], 90.00th=[ 326], 95.00th=[ 347], | 99.00th=[ 380], 99.50th=[ 401], 99.90th=[ 414], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 88, max= 264, per=25.00%, avg=171.75, stdev=31.96, samples=60 iops : min= 22, max= 66, avg=42.85, stdev= 8.00, samples=60 lat (msec) : 4=0.08%, 50=0.23%, 100=10.27%, 250=62.73%, 500=26.68% cpu : usr=0.17%, sys=1.28%, ctx=3010, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11463: Fri Apr 25 21:07:07 2025 read: IOPS=41, BW=167KiB/s (171kB/s)(5060KiB/30285msec) slat (usec): min=51, max=5956, avg=1942.05, stdev=564.19 clat (msec): min=2, max=208, avg=121.29, stdev=30.15 lat (msec): min=2, max=210, avg=123.23, stdev=30.36 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 74], 10.00th=[ 85], 20.00th=[ 97], | 30.00th=[ 108], 40.00th=[ 115], 50.00th=[ 123], 60.00th=[ 130], | 70.00th=[ 138], 80.00th=[ 148], 90.00th=[ 159], 95.00th=[ 167], | 99.00th=[ 188], 99.50th=[ 199], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 80, max= 288, per=25.37%, avg=167.42, stdev=46.51, samples=60 iops : min= 20, max= 72, avg=41.82, stdev=11.62, samples=60 write: IOPS=42, BW=171KiB/s (175kB/s)(5180KiB/30285msec) slat (usec): min=16, max=535, avg=77.35, stdev=39.15 clat (msec): min=29, max=443, avg=253.51, stdev=58.31 lat (msec): min=29, max=443, avg=253.59, stdev=58.32 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 159], 10.00th=[ 180], 20.00th=[ 205], | 30.00th=[ 224], 40.00th=[ 241], 50.00th=[ 255], 60.00th=[ 268], | 70.00th=[ 284], 80.00th=[ 300], 90.00th=[ 330], 95.00th=[ 351], | 99.00th=[ 384], 99.50th=[ 397], 99.90th=[ 430], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 104, max= 240, per=24.76%, avg=170.08, stdev=29.44, samples=60 iops : min= 26, max= 60, avg=42.48, stdev= 7.38, samples=60 lat (msec) : 4=0.12%, 10=0.12%, 20=0.12%, 50=0.43%, 100=10.70% lat (msec) : 250=61.84%, 500=26.68% cpu : usr=0.16%, sys=1.30%, ctx=2953, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=660KiB/s (676kB/s), 164KiB/s-167KiB/s (168kB/s-171kB/s), io=19.6MiB (20.5MB), run=30285-30400msec WRITE: bw=687KiB/s (704kB/s), 171KiB/s-173KiB/s (175kB/s-177kB/s), io=20.4MiB (21.4MB), run=30285-30400msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=11491: Fri Apr 25 21:07:08 2025 read: IOPS=6, BW=241MiB/s (253MB/s)(40.0MiB/166msec) slat (nsec): min=6037.4k, max=6037.4k, avg=6037413.00, stdev= 0.00 clat (nsec): min=153308k, max=153308k, avg=153307795.00, stdev= 0.00 lat (nsec): min=159354k, max=159354k, avg=159353991.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.64%, ctx=15, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=241MiB/s (253MB/s), 241MiB/s-241MiB/s (253MB/s-253MB/s), io=40.0MiB (41.9MB), run=166-166msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout