-----============= acceptance-small: conf-sanity ============----- Thu Apr 17 05:46:02 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg357-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg357-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg357-server oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:47:34 (1744883254) Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2826 (time 1744883300.23 total 10.00 last 282.55) - open/close 6168 (time 1744883310.23 total 20.00 last 334.18) - open/close 9995 (time 1744883320.23 total 30.00 last 382.70) - open/close 14069 (time 1744883330.24 total 40.01 last 407.28) - open/close 18475 (time 1744883340.24 total 50.01 last 440.58) - open/close 20000 (time 1744883343.64 total 53.41 last 448.04) - open/close 23829 (time 1744883353.64 total 63.41 last 382.80) - open/close 27795 (time 1744883363.64 total 73.41 last 396.52) - open/close 30000 (time 1744883369.09 total 78.86 last 405.24) - open/close 34700 (time 1744883379.09 total 88.86 last 469.89) - open/close 39468 (time 1744883389.09 total 98.86 last 476.74) - open/close 43810 (time 1744883399.09 total 108.86 last 434.17) - open/close 47345 (time 1744883409.09 total 118.86 last 353.49) - open/close 50000 (time 1744883417.75 total 127.52 last 306.65) - open/close 53421 (time 1744883427.75 total 137.52 last 342.01) - open/close 55906 (time 1744883437.76 total 147.53 last 248.37) - open/close 58331 (time 1744883447.76 total 157.53 last 242.50) - open/close 60000 (time 1744883455.50 total 165.27 last 215.51) - open/close 62337 (time 1744883465.50 total 175.27 last 233.70) total: 64768 open/close in 182.07 seconds: 355.74 ops/second - /unlink 6273 (time 1744883483.30 total 10.00 last 627.23) - /unlink 10000 (time 1744883488.07 total 14.77 last 782.09) - /unlink 19222 (time 1744883498.07 total 24.77 last 922.19) - /unlink 28036 (time 1744883508.07 total 34.77 last 881.33) - /unlink 30000 (time 1744883510.23 total 36.93 last 908.06) - /unlink 39118 (time 1744883520.23 total 46.93 last 911.75) - /unlink 49036 (time 1744883530.23 total 56.93 last 991.76) - /unlink 57256 (time 1744883540.23 total 66.93 last 821.98) - /unlink 60000 (time 1744883544.57 total 71.27 last 632.76) total: 64768 /unlink in 76.47 seconds: 847.02 ops/second Waiting for MDT destroys to complete - open/close 3821 (time 1744883642.22 total 10.00 last 382.01) - open/close 7879 (time 1744883652.22 total 20.00 last 405.75) - open/close 10000 (time 1744883656.84 total 24.62 last 459.12) - open/close 15181 (time 1744883666.84 total 34.62 last 518.09) - open/close 20000 (time 1744883676.05 total 43.84 last 523.10) - open/close 24632 (time 1744883686.05 total 53.84 last 463.11) - open/close 29310 (time 1744883696.05 total 63.84 last 467.76) - open/close 34999 (time 1744883706.05 total 73.84 last 568.90) - open/close 40000 (time 1744883714.80 total 82.58 last 571.78) - open/close 45861 (time 1744883724.80 total 92.58 last 586.09) - open/close 50000 (time 1744883732.33 total 100.11 last 549.92) - open/close 54815 (time 1744883742.33 total 110.11 last 481.42) - open/close 60000 (time 1744883751.61 total 119.40 last 558.60) total: 64768 open/close in 128.24 seconds: 505.04 ops/second - /unlink 8019 (time 1744883771.36 total 10.00 last 801.90) - /unlink 10000 (time 1744883773.52 total 12.17 last 914.53) - /unlink 19610 (time 1744883783.52 total 22.17 last 960.97) - /unlink 29886 (time 1744883793.52 total 32.17 last 1027.58) - /unlink 39303 (time 1744883803.52 total 42.17 last 941.65) - /unlink 48358 (time 1744883813.52 total 52.17 last 905.49) - /unlink 57742 (time 1744883823.52 total 62.17 last 938.40) - /unlink 60000 (time 1744883826.02 total 64.66 last 904.99) total: 64768 /unlink in 69.55 seconds: 931.23 ops/second Waiting for MDT destroys to complete - open/close 5078 (time 1744883928.63 total 10.00 last 507.73) - open/close 10000 (time 1744883937.21 total 18.57 last 574.17) - open/close 15083 (time 1744883947.21 total 28.58 last 508.22) - open/close 20000 (time 1744883956.53 total 37.89 last 527.62) - open/close 26073 (time 1744883966.53 total 47.89 last 607.29) - open/close 29126 (time 1744883976.53 total 57.90 last 305.06) - open/close 30000 (time 1744883983.34 total 64.71 last 128.45) - open/close 30999 (time 1744883993.35 total 74.71 last 99.82) - open/close 33251 (time 1744884003.35 total 84.72 last 225.18) - open/close 35486 (time 1744884013.35 total 94.72 last 223.47) - open/close 37700 (time 1744884023.35 total 104.72 last 221.34) - open/close 39468 (time 1744884033.36 total 114.73 last 176.67) - open/close 40000 (time 1744884035.71 total 117.08 last 225.92) - open/close 43421 (time 1744884045.72 total 127.08 last 342.05) - open/close 46668 (time 1744884055.72 total 137.09 last 324.64) - open/close 50000 (time 1744884063.26 total 144.62 last 441.96) - open/close 52653 (time 1744884073.26 total 154.62 last 265.29) - open/close 55498 (time 1744884083.26 total 164.63 last 284.39) - open/close 57550 (time 1744884093.26 total 174.63 last 205.15) - open/close 59472 (time 1744884103.27 total 184.63 last 192.14) - open/close 60000 (time 1744884106.11 total 187.47 last 185.94) - open/close 61832 (time 1744884116.11 total 197.48 last 183.08) - open/close 63481 (time 1744884126.12 total 207.49 last 164.82) total: 64768 open/close in 215.33 seconds: 300.78 ops/second - /unlink 2973 (time 1744884145.28 total 10.00 last 297.22) - /unlink 6098 (time 1744884155.28 total 20.00 last 312.44) - /unlink 10000 (time 1744884163.19 total 27.91 last 493.86) - /unlink 15240 (time 1744884173.19 total 37.91 last 523.97) - /unlink 20000 (time 1744884182.44 total 47.16 last 514.41) - /unlink 23950 (time 1744884192.44 total 57.16 last 394.89) - /unlink 28189 (time 1744884202.44 total 67.16 last 423.89) - /unlink 30000 (time 1744884206.08 total 70.80 last 498.29) - /unlink 34456 (time 1744884216.08 total 80.80 last 445.58) - /unlink 40000 (time 1744884225.59 total 90.31 last 582.84) - /unlink 45164 (time 1744884235.59 total 100.31 last 516.30) - /unlink 50000 (time 1744884245.57 total 110.30 last 484.36) - /unlink 55868 (time 1744884255.58 total 120.30 last 586.77) - /unlink 60000 (time 1744884262.77 total 127.49 last 574.26) total: 64768 /unlink in 136.03 seconds: 476.12 ops/second Waiting for MDT destroys to complete - open/close 3315 (time 1744884288.03 total 10.00 last 331.47) - open/close 7672 (time 1744884298.03 total 20.00 last 435.69) - open/close 10000 (time 1744884305.16 total 27.13 last 326.78) - open/close 13067 (time 1744884315.16 total 37.13 last 306.64) - open/close 16435 (time 1744884325.16 total 47.13 last 336.77) - open/close 20000 (time 1744884335.06 total 57.03 last 360.14) - open/close 23870 (time 1744884345.06 total 67.03 last 386.94) - open/close 27751 (time 1744884355.06 total 77.03 last 388.02) - open/close 30000 (time 1744884360.30 total 82.27 last 428.87) - open/close 34294 (time 1744884370.31 total 92.28 last 429.35) - open/close 39396 (time 1744884380.31 total 102.28 last 510.18) - open/close 43985 (time 1744884390.31 total 112.28 last 458.89) - open/close 48357 (time 1744884400.31 total 122.28 last 437.16) - open/close 50000 (time 1744884403.82 total 125.79 last 468.19) - open/close 54423 (time 1744884413.82 total 135.79 last 442.18) - open/close 58282 (time 1744884423.82 total 145.79 last 385.82) - open/close 60000 (time 1744884428.04 total 150.01 last 407.03) - open/close 64328 (time 1744884438.04 total 160.01 last 432.75) total: 64768 open/close in 161.11 seconds: 402.01 ops/second - /unlink 7033 (time 1744884450.05 total 10.00 last 703.23) - /unlink 10000 (time 1744884454.41 total 14.36 last 680.04) - /unlink 17339 (time 1744884464.41 total 24.36 last 733.89) - /unlink 20000 (time 1744884468.07 total 28.02 last 727.65) - /unlink 26442 (time 1744884478.07 total 38.02 last 644.16) - /unlink 30000 (time 1744884482.61 total 42.56 last 783.44) - /unlink 38505 (time 1744884492.61 total 52.56 last 850.45) - /unlink 47589 (time 1744884502.61 total 62.56 last 908.39) - /unlink 50000 (time 1744884506.51 total 66.46 last 618.07) - /unlink 56795 (time 1744884516.52 total 76.47 last 679.34) - /unlink 60000 (time 1744884521.57 total 81.52 last 634.05) total: 64768 /unlink in 88.03 seconds: 735.79 ops/second Waiting for MDT destroys to complete - open/close 4030 (time 1744884594.11 total 10.00 last 402.95) - open/close 7923 (time 1744884604.11 total 20.00 last 389.26) - open/close 10000 (time 1744884609.09 total 24.98 last 417.36) - open/close 13805 (time 1744884619.09 total 34.98 last 380.48) - open/close 17707 (time 1744884629.09 total 44.98 last 390.18) - open/close 20000 (time 1744884635.43 total 51.32 last 361.94) - open/close 23822 (time 1744884645.43 total 61.32 last 382.16) - open/close 27996 (time 1744884655.43 total 71.32 last 417.39) - open/close 30000 (time 1744884661.05 total 76.93 last 356.73) - open/close 33629 (time 1744884671.05 total 86.93 last 362.89) - open/close 37354 (time 1744884681.05 total 96.94 last 372.46) - open/close 40000 (time 1744884688.26 total 104.15 last 366.63) - open/close 44084 (time 1744884698.26 total 114.15 last 408.36) - open/close 47997 (time 1744884708.27 total 124.15 last 391.27) - open/close 50000 (time 1744884713.10 total 128.98 last 414.69) - open/close 54285 (time 1744884723.10 total 138.99 last 428.41) - open/close 58976 (time 1744884733.10 total 148.99 last 469.05) - open/close 60000 (time 1744884735.65 total 151.54 last 400.98) - open/close 64545 (time 1744884745.65 total 161.54 last 454.48) total: 64768 open/close in 162.10 seconds: 399.56 ops/second - /unlink 9047 (time 1744884757.12 total 10.00 last 904.54) - /unlink 16803 (time 1744884767.12 total 20.00 last 775.56) - /unlink 20000 (time 1744884771.15 total 24.03 last 793.72) - /unlink 28572 (time 1744884781.15 total 34.03 last 857.18) - /unlink 38727 (time 1744884791.15 total 44.03 last 1015.45) - /unlink 48416 (time 1744884801.15 total 54.03 last 968.90) - /unlink 58458 (time 1744884811.15 total 64.03 last 1003.93) total: 64768 /unlink in 69.74 seconds: 928.68 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 PASS 106 (1676s) Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1787 sec ========== 06:15:49 (1744884949) === conf-sanity: start cleanup 06:15:49 (1744884949) === === conf-sanity: finish cleanup 06:15:50 (1744884950) ===