== sanity test 398c: run fio to test AIO ================= 03:30:49 (1746084649) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0879759 s, 477 MB/s osc.lustre-OST0000-osc-ffff8800a9960800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=32146: Thu May 1 03:31:50 2025 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58910msec) slat (usec): min=11, max=2997, avg=113.05, stdev=182.22 clat (msec): min=70, max=729, avg=367.78, stdev=107.98 lat (msec): min=71, max=729, avg=367.89, stdev=107.99 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 188], 10.00th=[ 239], 20.00th=[ 288], | 30.00th=[ 326], 40.00th=[ 347], 50.00th=[ 363], 60.00th=[ 388], | 70.00th=[ 414], 80.00th=[ 443], 90.00th=[ 506], 95.00th=[ 550], | 99.00th=[ 667], 99.50th=[ 701], 99.90th=[ 718], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 72, max= 528, per=25.01%, avg=173.59, stdev=56.99, samples=117 iops : min= 18, max= 132, avg=43.28, stdev=14.27, samples=117 lat (msec) : 100=0.98%, 250=11.37%, 500=76.91%, 750=10.74% cpu : usr=0.34%, sys=0.47%, ctx=1741, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=32147: Thu May 1 03:31:50 2025 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58944msec) slat (usec): min=12, max=3165, avg=102.26, stdev=146.39 clat (msec): min=62, max=741, avg=367.93, stdev=108.81 lat (msec): min=62, max=741, avg=368.03, stdev=108.82 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 194], 10.00th=[ 234], 20.00th=[ 292], | 30.00th=[ 326], 40.00th=[ 347], 50.00th=[ 363], 60.00th=[ 388], | 70.00th=[ 409], 80.00th=[ 451], 90.00th=[ 506], 95.00th=[ 550], | 99.00th=[ 676], 99.50th=[ 693], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 743] bw ( KiB/s): min= 72, max= 512, per=24.97%, avg=173.26, stdev=56.36, samples=117 iops : min= 18, max= 128, avg=43.16, stdev=14.13, samples=117 lat (msec) : 100=1.09%, 250=11.17%, 500=76.45%, 750=11.29% cpu : usr=0.09%, sys=0.68%, ctx=1776, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=32148: Thu May 1 03:31:50 2025 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58944msec) slat (usec): min=11, max=3637, avg=101.30, stdev=150.32 clat (msec): min=63, max=741, avg=367.92, stdev=108.34 lat (msec): min=63, max=741, avg=368.02, stdev=108.35 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 192], 10.00th=[ 232], 20.00th=[ 292], | 30.00th=[ 326], 40.00th=[ 347], 50.00th=[ 363], 60.00th=[ 388], | 70.00th=[ 414], 80.00th=[ 447], 90.00th=[ 506], 95.00th=[ 550], | 99.00th=[ 659], 99.50th=[ 701], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 80, max= 512, per=24.95%, avg=173.14, stdev=56.97, samples=117 iops : min= 20, max= 128, avg=43.14, stdev=14.27, samples=117 lat (msec) : 100=1.02%, 250=11.33%, 500=76.91%, 750=10.74% cpu : usr=0.14%, sys=0.63%, ctx=1850, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=32149: Thu May 1 03:31:50 2025 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58889msec) slat (usec): min=13, max=2735, avg=106.43, stdev=146.57 clat (msec): min=18, max=842, avg=367.64, stdev=112.34 lat (msec): min=18, max=842, avg=367.75, stdev=112.35 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 184], 10.00th=[ 232], 20.00th=[ 288], | 30.00th=[ 321], 40.00th=[ 342], 50.00th=[ 363], 60.00th=[ 388], | 70.00th=[ 414], 80.00th=[ 447], 90.00th=[ 510], 95.00th=[ 550], | 99.00th=[ 676], 99.50th=[ 718], 99.90th=[ 827], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 80, max= 568, per=25.04%, avg=173.79, stdev=60.63, samples=117 iops : min= 20, max= 142, avg=43.33, stdev=15.19, samples=117 lat (msec) : 20=0.04%, 50=0.35%, 100=1.05%, 250=11.52%, 500=75.47% lat (msec) : 750=11.29%, 1000=0.27% cpu : usr=0.07%, sys=0.73%, ctx=1825, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=695KiB/s (712kB/s), 174KiB/s-174KiB/s (178kB/s-178kB/s), io=40.0MiB (41.9MB), run=58889-58944msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32206: Thu May 1 03:32:32 2025 read: IOPS=30, BW=123KiB/s (126kB/s)(5048KiB/40938msec) slat (usec): min=24, max=10382, avg=3233.48, stdev=1285.47 clat (msec): min=4, max=443, avg=168.54, stdev=38.70 lat (msec): min=4, max=449, avg=171.77, stdev=39.00 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 116], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 215], 95.00th=[ 228], | 99.00th=[ 253], 99.50th=[ 300], 99.90th=[ 376], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 56, max= 216, per=25.27%, avg=123.30, stdev=39.60, samples=81 iops : min= 14, max= 54, avg=30.63, stdev= 9.91, samples=81 write: IOPS=31, BW=127KiB/s (130kB/s)(5192KiB/40938msec) slat (usec): min=15, max=1783, avg=111.33, stdev=156.76 clat (msec): min=25, max=647, avg=337.14, stdev=92.78 lat (msec): min=25, max=647, avg=337.25, stdev=92.79 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 203], 10.00th=[ 224], 20.00th=[ 259], | 30.00th=[ 284], 40.00th=[ 305], 50.00th=[ 330], 60.00th=[ 351], | 70.00th=[ 380], 80.00th=[ 414], 90.00th=[ 468], 95.00th=[ 502], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 634], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 64, max= 215, per=24.89%, avg=126.46, stdev=30.44, samples=81 iops : min= 16, max= 53, avg=31.42, stdev= 7.59, samples=81 lat (msec) : 10=0.39%, 50=0.23%, 100=0.74%, 250=56.13%, 500=39.77% lat (msec) : 750=2.73% cpu : usr=0.16%, sys=1.75%, ctx=2703, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32207: Thu May 1 03:32:32 2025 read: IOPS=30, BW=121KiB/s (124kB/s)(4976KiB/41079msec) slat (usec): min=1056, max=9947, avg=3245.90, stdev=1238.24 clat (msec): min=4, max=308, avg=168.81, stdev=35.28 lat (msec): min=7, max=311, avg=172.06, stdev=35.48 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 117], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 228], | 99.00th=[ 253], 99.50th=[ 268], 99.90th=[ 305], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 48, max= 232, per=24.66%, avg=120.33, stdev=39.53, samples=82 iops : min= 12, max= 58, avg=29.89, stdev= 9.88, samples=82 write: IOPS=32, BW=128KiB/s (131kB/s)(5264KiB/41079msec) slat (usec): min=12, max=2797, avg=112.75, stdev=176.63 clat (msec): min=134, max=672, avg=336.18, stdev=90.73 lat (msec): min=134, max=672, avg=336.29, stdev=90.74 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 197], 10.00th=[ 228], 20.00th=[ 259], | 30.00th=[ 288], 40.00th=[ 305], 50.00th=[ 326], 60.00th=[ 351], | 70.00th=[ 380], 80.00th=[ 414], 90.00th=[ 464], 95.00th=[ 498], | 99.00th=[ 558], 99.50th=[ 592], 99.90th=[ 625], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 79, max= 200, per=25.09%, avg=127.45, stdev=28.23, samples=82 iops : min= 19, max= 50, avg=31.67, stdev= 7.07, samples=82 lat (msec) : 10=0.08%, 20=0.08%, 50=0.20%, 100=0.39%, 250=55.86% lat (msec) : 500=41.05%, 750=2.34% cpu : usr=0.11%, sys=1.81%, ctx=2775, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32208: Thu May 1 03:32:32 2025 read: IOPS=30, BW=121KiB/s (124kB/s)(4984KiB/41078msec) slat (usec): min=1126, max=10761, avg=3193.18, stdev=1262.06 clat (msec): min=20, max=310, avg=167.05, stdev=32.79 lat (msec): min=22, max=313, avg=170.25, stdev=32.97 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 115], 10.00th=[ 127], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 194], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 236], 99.50th=[ 245], 99.90th=[ 264], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 47, max= 232, per=24.84%, avg=121.20, stdev=42.74, samples=82 iops : min= 11, max= 58, avg=30.17, stdev=10.72, samples=82 write: IOPS=31, BW=128KiB/s (131kB/s)(5256KiB/41078msec) slat (usec): min=15, max=3076, avg=113.01, stdev=180.16 clat (msec): min=96, max=647, avg=338.01, stdev=90.47 lat (msec): min=96, max=647, avg=338.12, stdev=90.48 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 199], 10.00th=[ 224], 20.00th=[ 262], | 30.00th=[ 284], 40.00th=[ 309], 50.00th=[ 330], 60.00th=[ 355], | 70.00th=[ 384], 80.00th=[ 418], 90.00th=[ 464], 95.00th=[ 498], | 99.00th=[ 558], 99.50th=[ 567], 99.90th=[ 625], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 71, max= 192, per=24.95%, avg=126.76, stdev=28.20, samples=82 iops : min= 17, max= 48, avg=31.56, stdev= 7.05, samples=82 lat (msec) : 50=0.27%, 100=0.31%, 250=56.45%, 500=40.59%, 750=2.38% cpu : usr=0.49%, sys=1.47%, ctx=2783, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32209: Thu May 1 03:32:32 2025 read: IOPS=30, BW=123KiB/s (126kB/s)(5060KiB/41058msec) slat (usec): min=1122, max=9525, avg=3279.11, stdev=1287.20 clat (msec): min=21, max=290, avg=169.29, stdev=32.95 lat (msec): min=23, max=293, avg=172.57, stdev=33.17 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 117], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 211], 95.00th=[ 226], | 99.00th=[ 251], 99.50th=[ 257], 99.90th=[ 288], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 48, max= 248, per=25.16%, avg=122.77, stdev=39.28, samples=82 iops : min= 12, max= 62, avg=30.50, stdev= 9.89, samples=82 write: IOPS=31, BW=126KiB/s (129kB/s)(5180KiB/41058msec) slat (usec): min=12, max=1930, avg=116.12, stdev=170.18 clat (msec): min=133, max=645, avg=338.06, stdev=89.93 lat (msec): min=133, max=645, avg=338.18, stdev=89.93 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 197], 10.00th=[ 228], 20.00th=[ 262], | 30.00th=[ 284], 40.00th=[ 305], 50.00th=[ 326], 60.00th=[ 355], | 70.00th=[ 384], 80.00th=[ 418], 90.00th=[ 468], 95.00th=[ 498], | 99.00th=[ 558], 99.50th=[ 584], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 48, max= 192, per=24.61%, avg=125.01, stdev=29.72, samples=82 iops : min= 12, max= 48, avg=31.06, stdev= 7.39, samples=82 lat (msec) : 50=0.27%, 100=0.20%, 250=56.80%, 500=40.47%, 750=2.27% cpu : usr=0.16%, sys=1.81%, ctx=2815, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=489KiB/s (500kB/s), 121KiB/s-123KiB/s (124kB/s-126kB/s), io=19.6MiB (20.5MB), run=40938-41079msec WRITE: bw=509KiB/s (521kB/s), 126KiB/s-128KiB/s (129kB/s-131kB/s), io=20.4MiB (21.4MB), run=40938-41079msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=32246: Thu May 1 03:32:33 2025 read: IOPS=5, BW=238MiB/s (250MB/s)(40.0MiB/168msec) slat (nsec): min=10148k, max=10148k, avg=10148354.00, stdev= 0.00 clat (nsec): min=149532k, max=149532k, avg=149531833.00, stdev= 0.00 lat (nsec): min=159703k, max=159703k, avg=159703265.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] lat (msec) : 250=100.00% cpu : usr=0.60%, sys=4.19%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=238MiB/s (250MB/s), 238MiB/s-238MiB/s (250MB/s-250MB/s), io=40.0MiB (41.9MB), run=168-168msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout