== sanity test 398c: run fio to test AIO ================= 22:14:15 (1746152055) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.143528 s, 292 MB/s osc.lustre-OST0000-osc-ffff88012a697800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=1758: Thu May 1 22:15:32 2025 write: IOPS=34, BW=137KiB/s (140kB/s)(10.0MiB/74863msec) slat (usec): min=11, max=3674, avg=104.57, stdev=178.67 clat (msec): min=130, max=871, avg=467.20, stdev=133.81 lat (msec): min=130, max=871, avg=467.30, stdev=133.81 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 241], 10.00th=[ 284], 20.00th=[ 347], | 30.00th=[ 393], 40.00th=[ 439], 50.00th=[ 477], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 584], 90.00th=[ 634], 95.00th=[ 667], | 99.00th=[ 776], 99.50th=[ 802], 99.90th=[ 852], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 56, max= 264, per=24.83%, avg=135.84, stdev=45.07, samples=149 iops : min= 14, max= 66, avg=33.72, stdev=11.30, samples=149 lat (msec) : 250=5.98%, 500=49.41%, 750=43.24%, 1000=1.37% cpu : usr=0.06%, sys=0.58%, ctx=1580, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1759: Thu May 1 22:15:32 2025 write: IOPS=34, BW=137KiB/s (140kB/s)(10.0MiB/74641msec) slat (usec): min=10, max=4423, avg=115.51, stdev=222.26 clat (msec): min=31, max=824, avg=466.04, stdev=136.71 lat (msec): min=31, max=824, avg=466.15, stdev=136.71 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 236], 10.00th=[ 284], 20.00th=[ 342], | 30.00th=[ 393], 40.00th=[ 435], 50.00th=[ 477], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 584], 90.00th=[ 634], 95.00th=[ 684], | 99.00th=[ 776], 99.50th=[ 785], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 48, max= 352, per=24.95%, avg=136.47, stdev=45.04, samples=149 iops : min= 12, max= 88, avg=33.99, stdev=11.31, samples=149 lat (msec) : 50=0.43%, 100=0.08%, 250=5.47%, 500=50.90%, 750=41.80% lat (msec) : 1000=1.33% cpu : usr=0.14%, sys=0.53%, ctx=1613, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1760: Thu May 1 22:15:32 2025 write: IOPS=34, BW=137KiB/s (140kB/s)(10.0MiB/74714msec) slat (usec): min=11, max=3952, avg=104.03, stdev=171.32 clat (msec): min=147, max=871, avg=466.39, stdev=134.36 lat (msec): min=147, max=871, avg=466.50, stdev=134.37 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 239], 10.00th=[ 284], 20.00th=[ 342], | 30.00th=[ 388], 40.00th=[ 439], 50.00th=[ 477], 60.00th=[ 510], | 70.00th=[ 550], 80.00th=[ 584], 90.00th=[ 634], 95.00th=[ 676], | 99.00th=[ 785], 99.50th=[ 818], 99.90th=[ 844], 99.95th=[ 860], | 99.99th=[ 869] bw ( KiB/s): min= 31, max= 272, per=24.95%, avg=136.48, stdev=45.92, samples=149 iops : min= 7, max= 68, avg=34.00, stdev=11.54, samples=149 lat (msec) : 250=5.86%, 500=51.64%, 750=40.94%, 1000=1.56% cpu : usr=0.26%, sys=0.39%, ctx=1524, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1761: Thu May 1 22:15:32 2025 write: IOPS=34, BW=137KiB/s (140kB/s)(10.0MiB/74832msec) slat (usec): min=11, max=2428, avg=111.76, stdev=185.65 clat (msec): min=45, max=870, avg=466.91, stdev=132.93 lat (msec): min=45, max=871, avg=467.03, stdev=132.94 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 245], 10.00th=[ 284], 20.00th=[ 342], | 30.00th=[ 397], 40.00th=[ 439], 50.00th=[ 477], 60.00th=[ 510], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 634], 95.00th=[ 676], | 99.00th=[ 768], 99.50th=[ 793], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 63, max= 367, per=24.81%, avg=135.68, stdev=43.83, samples=149 iops : min= 15, max= 91, avg=33.68, stdev=11.00, samples=149 lat (msec) : 50=0.04%, 250=5.31%, 500=51.48%, 750=41.72%, 1000=1.45% cpu : usr=0.16%, sys=0.49%, ctx=1525, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=547KiB/s (560kB/s), 137KiB/s-137KiB/s (140kB/s-140kB/s), io=40.0MiB (41.9MB), run=74641-74863msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1830: Thu May 1 22:16:22 2025 read: IOPS=25, BW=104KiB/s (106kB/s)(5048KiB/48694msec) slat (usec): min=1070, max=15684, avg=3946.60, stdev=1832.31 clat (msec): min=5, max=378, avg=198.16, stdev=53.86 lat (msec): min=7, max=386, avg=202.11, stdev=54.58 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 268], 95.00th=[ 296], | 99.00th=[ 355], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 16, max= 207, per=25.21%, avg=103.38, stdev=39.48, samples=97 iops : min= 4, max= 51, avg=25.69, stdev= 9.87, samples=97 write: IOPS=26, BW=107KiB/s (109kB/s)(5192KiB/48694msec) slat (usec): min=11, max=2632, avg=132.78, stdev=223.19 clat (msec): min=138, max=867, avg=402.98, stdev=132.05 lat (msec): min=138, max=867, avg=403.11, stdev=132.07 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 201], 10.00th=[ 247], 20.00th=[ 296], | 30.00th=[ 326], 40.00th=[ 359], 50.00th=[ 388], 60.00th=[ 418], | 70.00th=[ 460], 80.00th=[ 506], 90.00th=[ 575], 95.00th=[ 659], | 99.00th=[ 793], 99.50th=[ 802], 99.90th=[ 860], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 23, max= 184, per=24.85%, avg=106.10, stdev=32.82, samples=97 iops : min= 5, max= 46, avg=26.37, stdev= 8.25, samples=97 lat (msec) : 10=0.04%, 20=0.08%, 50=0.35%, 100=0.08%, 250=46.05% lat (msec) : 500=42.66%, 750=9.69%, 1000=1.05% cpu : usr=0.42%, sys=1.40%, ctx=2746, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1831: Thu May 1 22:16:22 2025 read: IOPS=25, BW=102KiB/s (105kB/s)(4976KiB/48725msec) slat (usec): min=1106, max=16953, avg=3872.59, stdev=1813.76 clat (msec): min=5, max=370, avg=198.97, stdev=54.05 lat (msec): min=11, max=377, avg=202.85, stdev=54.79 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 125], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 222], 80.00th=[ 241], 90.00th=[ 271], 95.00th=[ 300], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 368], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 31, max= 216, per=24.73%, avg=101.38, stdev=37.82, samples=97 iops : min= 7, max= 54, avg=25.18, stdev= 9.53, samples=97 write: IOPS=27, BW=108KiB/s (111kB/s)(5264KiB/48725msec) slat (usec): min=13, max=3594, avg=135.74, stdev=235.41 clat (msec): min=125, max=863, avg=400.08, stdev=129.49 lat (msec): min=125, max=863, avg=400.22, stdev=129.51 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 209], 10.00th=[ 241], 20.00th=[ 296], | 30.00th=[ 330], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 414], | 70.00th=[ 456], 80.00th=[ 514], 90.00th=[ 575], 95.00th=[ 634], | 99.00th=[ 768], 99.50th=[ 810], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 24, max= 184, per=25.15%, avg=107.40, stdev=32.37, samples=97 iops : min= 6, max= 46, avg=26.68, stdev= 8.07, samples=97 lat (msec) : 10=0.04%, 20=0.04%, 50=0.31%, 100=0.08%, 250=45.31% lat (msec) : 500=42.89%, 750=10.70%, 1000=0.62% cpu : usr=0.40%, sys=1.40%, ctx=2730, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1832: Thu May 1 22:16:22 2025 read: IOPS=25, BW=102KiB/s (105kB/s)(4984KiB/48833msec) slat (usec): min=1082, max=15039, avg=3886.95, stdev=1851.27 clat (msec): min=4, max=381, avg=198.48, stdev=53.97 lat (msec): min=7, max=387, avg=202.38, stdev=54.70 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 241], 90.00th=[ 271], 95.00th=[ 305], | 99.00th=[ 363], 99.50th=[ 368], 99.90th=[ 380], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 31, max= 240, per=24.66%, avg=101.10, stdev=41.44, samples=97 iops : min= 7, max= 60, avg=25.14, stdev=10.43, samples=97 write: IOPS=26, BW=108KiB/s (110kB/s)(5256KiB/48833msec) slat (usec): min=11, max=2727, avg=127.88, stdev=192.67 clat (msec): min=72, max=864, avg=402.06, stdev=133.82 lat (msec): min=72, max=864, avg=402.19, stdev=133.84 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 197], 10.00th=[ 245], 20.00th=[ 292], | 30.00th=[ 326], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 418], | 70.00th=[ 464], 80.00th=[ 514], 90.00th=[ 575], 95.00th=[ 651], | 99.00th=[ 768], 99.50th=[ 818], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 48, max= 184, per=24.84%, avg=106.05, stdev=30.14, samples=97 iops : min= 12, max= 46, avg=26.38, stdev= 7.54, samples=97 lat (msec) : 10=0.12%, 50=0.04%, 100=0.31%, 250=45.16%, 500=42.73% lat (msec) : 750=10.82%, 1000=0.82% cpu : usr=0.28%, sys=1.53%, ctx=2802, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1833: Thu May 1 22:16:22 2025 read: IOPS=26, BW=104KiB/s (107kB/s)(5060KiB/48468msec) slat (usec): min=23, max=16508, avg=3924.29, stdev=1850.85 clat (msec): min=4, max=373, avg=199.77, stdev=55.26 lat (msec): min=4, max=379, avg=203.70, stdev=56.04 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 209], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 271], 95.00th=[ 305], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 32, max= 215, per=25.48%, avg=104.45, stdev=36.63, samples=96 iops : min= 8, max= 53, avg=25.89, stdev= 9.22, samples=96 write: IOPS=26, BW=107KiB/s (109kB/s)(5180KiB/48468msec) slat (usec): min=11, max=2975, avg=124.85, stdev=210.67 clat (msec): min=28, max=855, avg=399.46, stdev=133.09 lat (msec): min=28, max=855, avg=399.59, stdev=133.10 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 197], 10.00th=[ 236], 20.00th=[ 288], | 30.00th=[ 326], 40.00th=[ 355], 50.00th=[ 388], 60.00th=[ 422], | 70.00th=[ 468], 80.00th=[ 510], 90.00th=[ 575], 95.00th=[ 642], | 99.00th=[ 743], 99.50th=[ 793], 99.90th=[ 818], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 32, max= 216, per=24.77%, avg=105.78, stdev=35.51, samples=96 iops : min= 8, max= 54, avg=26.22, stdev= 8.94, samples=96 lat (msec) : 10=0.20%, 20=0.04%, 50=0.39%, 100=0.62%, 250=45.55% lat (msec) : 500=42.19%, 750=10.59%, 1000=0.43% cpu : usr=0.56%, sys=1.28%, ctx=2783, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=411KiB/s (421kB/s), 102KiB/s-104KiB/s (105kB/s-107kB/s), io=19.6MiB (20.5MB), run=48468-48833msec WRITE: bw=428KiB/s (438kB/s), 107KiB/s-108KiB/s (109kB/s-111kB/s), io=20.4MiB (21.4MB), run=48468-48833msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=1876: Thu May 1 22:16:24 2025 read: IOPS=5, BW=222MiB/s (233MB/s)(40.0MiB/180msec) slat (nsec): min=3200.4k, max=3200.4k, avg=3200448.00, stdev= 0.00 clat (nsec): min=170607k, max=170607k, avg=170607391.00, stdev= 0.00 lat (nsec): min=173849k, max=173849k, avg=173848649.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=1.67%, ctx=19, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=40.0MiB (41.9MB), run=180-180msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout