== sanity test 398c: run fio to test AIO ================= 02:34:52 (1775198092) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.407584 s, 103 MB/s osc.lustre-OST0000-osc-ffff9f01c31fe800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149422: Fri Apr 3 02:37:44 2026 write: IOPS=15, BW=62.4KiB/s (63.9kB/s)(10.0MiB/164177msec); 0 zone resets slat (usec): min=42, max=15328, avg=295.80, stdev=847.16 clat (msec): min=101, max=2996, avg=1025.40, stdev=404.18 lat (msec): min=102, max=2996, avg=1025.70, stdev=404.15 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 542], 10.00th=[ 667], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1099], 80.00th=[ 1250], 90.00th=[ 1536], 95.00th=[ 1838], | 99.00th=[ 2400], 99.50th=[ 2702], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 240, per=25.45%, avg=63.11, stdev=33.77, samples=319 iops : min= 1, max= 60, avg=15.33, stdev= 8.49, samples=319 lat (msec) : 250=0.74%, 500=3.79%, 750=15.23%, 1000=37.70%, 2000=39.14% lat (msec) : >=2000=3.40% cpu : usr=0.05%, sys=0.26%, ctx=2023, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149423: Fri Apr 3 02:37:44 2026 write: IOPS=15, BW=62.2KiB/s (63.7kB/s)(10.0MiB/164703msec); 0 zone resets slat (usec): min=47, max=22829, avg=282.69, stdev=939.21 clat (msec): min=188, max=2994, avg=1028.09, stdev=396.48 lat (msec): min=188, max=2994, avg=1028.37, stdev=396.46 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1234], 90.00th=[ 1519], 95.00th=[ 1854], | 99.00th=[ 2400], 99.50th=[ 2668], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 256, per=25.38%, avg=62.94, stdev=29.47, samples=320 iops : min= 1, max= 64, avg=15.28, stdev= 7.43, samples=320 lat (msec) : 250=0.51%, 500=3.36%, 750=15.47%, 1000=37.27%, 2000=40.20% lat (msec) : >=2000=3.20% cpu : usr=0.04%, sys=0.24%, ctx=2197, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149424: Fri Apr 3 02:37:44 2026 write: IOPS=15, BW=62.2KiB/s (63.7kB/s)(10.0MiB/164714msec); 0 zone resets slat (usec): min=48, max=21347, avg=309.35, stdev=1014.63 clat (msec): min=200, max=2917, avg=1028.34, stdev=401.22 lat (msec): min=201, max=2917, avg=1028.65, stdev=401.22 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 567], 10.00th=[ 667], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1234], 90.00th=[ 1586], 95.00th=[ 1871], | 99.00th=[ 2433], 99.50th=[ 2601], 99.90th=[ 2769], 99.95th=[ 2802], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 256, per=24.98%, avg=61.95, stdev=33.90, samples=325 iops : min= 1, max= 64, avg=15.03, stdev= 8.51, samples=325 lat (msec) : 250=0.04%, 500=3.75%, 750=15.59%, 1000=37.58%, 2000=39.45% lat (msec) : >=2000=3.59% cpu : usr=0.04%, sys=0.27%, ctx=2248, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149425: Fri Apr 3 02:37:44 2026 write: IOPS=15, BW=62.2KiB/s (63.7kB/s)(10.0MiB/164703msec); 0 zone resets slat (usec): min=36, max=15514, avg=286.30, stdev=819.09 clat (msec): min=193, max=2901, avg=1028.24, stdev=398.97 lat (msec): min=197, max=2902, avg=1028.53, stdev=398.96 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1569], 95.00th=[ 1871], | 99.00th=[ 2333], 99.50th=[ 2601], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 256, per=25.38%, avg=62.93, stdev=30.80, samples=320 iops : min= 1, max= 64, avg=15.27, stdev= 7.73, samples=320 lat (msec) : 250=0.20%, 500=3.95%, 750=14.77%, 1000=37.93%, 2000=39.53% lat (msec) : >=2000=3.63% cpu : usr=0.04%, sys=0.24%, ctx=1979, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=249KiB/s (255kB/s), 62.2KiB/s-62.4KiB/s (63.7kB/s-63.9kB/s), io=40.0MiB (41.9MB), run=164177-164714msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149458: Fri Apr 3 02:39:38 2026 read: IOPS=12, BW=48.2KiB/s (49.3kB/s)(5048KiB/104787msec) slat (usec): min=59, max=54123, avg=13933.43, stdev=7587.96 clat (msec): min=11, max=818, avg=422.59, stdev=110.88 lat (msec): min=33, max=831, avg=436.53, stdev=112.29 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 259], 10.00th=[ 296], 20.00th=[ 338], | 30.00th=[ 359], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 472], 80.00th=[ 514], 90.00th=[ 567], 95.00th=[ 617], | 99.00th=[ 751], 99.50th=[ 768], 99.90th=[ 810], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 7, max= 120, per=25.19%, avg=47.62, stdev=22.36, samples=208 iops : min= 1, max= 30, avg=11.43, stdev= 5.65, samples=208 write: IOPS=12, BW=49.5KiB/s (50.7kB/s)(5192KiB/104787msec); 0 zone resets slat (usec): min=45, max=13620, avg=339.46, stdev=877.42 clat (msec): min=168, max=1535, avg=866.18, stdev=238.40 lat (msec): min=169, max=1535, avg=866.52, stdev=238.43 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 477], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1435], 99.50th=[ 1469], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 112, per=24.93%, avg=49.12, stdev=20.75, samples=207 iops : min= 1, max= 28, avg=11.81, stdev= 5.24, samples=207 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=2.19%, 500=38.67% lat (msec) : 750=25.39%, 1000=19.34%, 2000=14.14% cpu : usr=0.08%, sys=0.69%, ctx=5007, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149459: Fri Apr 3 02:39:38 2026 read: IOPS=11, BW=47.1KiB/s (48.2kB/s)(4976KiB/105700msec) slat (usec): min=2662, max=65671, avg=13804.61, stdev=7376.98 clat (msec): min=28, max=952, avg=418.00, stdev=113.74 lat (msec): min=34, max=969, avg=431.81, stdev=114.97 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 257], 10.00th=[ 292], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 460], 80.00th=[ 493], 90.00th=[ 558], 95.00th=[ 617], | 99.00th=[ 776], 99.50th=[ 827], 99.90th=[ 927], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 119, per=24.93%, avg=47.13, stdev=22.33, samples=207 iops : min= 1, max= 29, avg=11.31, stdev= 5.62, samples=207 write: IOPS=12, BW=49.8KiB/s (50.0kB/s)(5264KiB/105700msec); 0 zone resets slat (usec): min=56, max=15775, avg=336.33, stdev=942.24 clat (msec): min=249, max=1576, avg=873.87, stdev=236.46 lat (msec): min=249, max=1576, avg=874.21, stdev=236.46 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 485], 10.00th=[ 575], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 802], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1552], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 120, per=25.30%, avg=49.85, stdev=20.85, samples=207 iops : min= 1, max= 30, avg=11.99, stdev= 5.26, samples=207 lat (msec) : 50=0.12%, 100=0.12%, 250=1.91%, 500=40.20%, 750=22.07% lat (msec) : 1000=19.88%, 2000=15.70% cpu : usr=0.07%, sys=0.68%, ctx=4459, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149460: Fri Apr 3 02:39:38 2026 read: IOPS=11, BW=47.1KiB/s (48.3kB/s)(4984KiB/105738msec) slat (usec): min=2935, max=54192, avg=14064.39, stdev=7598.23 clat (msec): min=39, max=867, avg=421.27, stdev=111.77 lat (msec): min=46, max=876, avg=435.34, stdev=113.20 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 388], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 502], 90.00th=[ 558], 95.00th=[ 625], | 99.00th=[ 776], 99.50th=[ 818], 99.90th=[ 860], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 7, max= 136, per=24.68%, avg=46.64, stdev=21.55, samples=210 iops : min= 1, max= 34, avg=11.19, stdev= 5.46, samples=210 write: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5256KiB/105738msec); 0 zone resets slat (usec): min=54, max=10246, avg=329.37, stdev=834.39 clat (msec): min=186, max=1533, avg=871.80, stdev=233.90 lat (msec): min=186, max=1534, avg=872.13, stdev=233.93 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 502], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 860], 60.00th=[ 927], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1401], 99.50th=[ 1418], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 8, max= 104, per=24.85%, avg=48.96, stdev=20.04, samples=210 iops : min= 2, max= 26, avg=11.77, stdev= 5.04, samples=210 lat (msec) : 50=0.12%, 100=0.16%, 250=1.33%, 500=39.92%, 750=23.16% lat (msec) : 1000=19.88%, 2000=15.43% cpu : usr=0.09%, sys=0.68%, ctx=4824, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149461: Fri Apr 3 02:39:38 2026 read: IOPS=11, BW=47.0KiB/s (49.1kB/s)(5060KiB/105512msec) slat (usec): min=2904, max=52952, avg=14067.59, stdev=7375.44 clat (msec): min=16, max=858, avg=422.17, stdev=111.22 lat (msec): min=39, max=878, avg=436.25, stdev=112.25 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 384], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 506], 90.00th=[ 558], 95.00th=[ 617], | 99.00th=[ 768], 99.50th=[ 827], 99.90th=[ 852], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 7, max= 135, per=25.18%, avg=47.60, stdev=22.10, samples=209 iops : min= 1, max= 33, avg=11.43, stdev= 5.55, samples=209 write: IOPS=12, BW=49.1KiB/s (50.3kB/s)(5180KiB/105512msec); 0 zone resets slat (usec): min=51, max=22135, avg=337.96, stdev=984.73 clat (msec): min=151, max=1595, avg=873.88, stdev=238.71 lat (msec): min=151, max=1595, avg=874.22, stdev=238.67 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 506], 10.00th=[ 575], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 860], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1519], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 104, per=24.73%, avg=48.72, stdev=19.54, samples=208 iops : min= 1, max= 26, avg=11.70, stdev= 4.93, samples=208 lat (msec) : 20=0.04%, 50=0.12%, 100=0.04%, 250=1.45%, 500=39.69% lat (msec) : 750=24.38%, 1000=18.98%, 2000=15.31% cpu : usr=0.08%, sys=0.65%, ctx=4724, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=190KiB/s (194kB/s), 47.1KiB/s-48.2KiB/s (48.2kB/s-49.3kB/s), io=19.6MiB (20.5MB), run=104787-105738msec WRITE: bw=198KiB/s (202kB/s), 49.1KiB/s-49.8KiB/s (50.3kB/s-50.0kB/s), io=20.4MiB (21.4MB), run=104787-105738msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149480: Fri Apr 3 02:39:45 2026 read: IOPS=1, BW=74.1MiB/s (77.7MB/s)(40.0MiB/540msec) slat (nsec): min=18102k, max=18102k, avg=18101877.00, stdev= 0.00 clat (nsec): min=513035k, max=513035k, avg=513034526.00, stdev= 0.00 lat (nsec): min=531165k, max=531165k, avg=531165431.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.23%, ctx=26, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.1MiB/s (77.7MB/s), 74.1MiB/s-74.1MiB/s (77.7MB/s-77.7MB/s), io=40.0MiB (41.9MB), run=540-540msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout