-----============= acceptance-small: conf-sanity ============----- Mon Apr 7 13:10:07 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg451-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg451-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg451-server oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg451-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg451-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:11:06 (1744045866) Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg451-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4943 (time 1744045900.23 total 10.00 last 494.26) - open/close 9733 (time 1744045910.23 total 20.00 last 478.93) - open/close 15430 (time 1744045920.23 total 30.00 last 569.66) - open/close 20000 (time 1744045928.68 total 38.46 last 540.59) - open/close 24303 (time 1744045938.68 total 48.46 last 430.26) - open/close 28401 (time 1744045948.68 total 58.46 last 409.78) - open/close 30000 (time 1744045952.51 total 62.28 last 418.09) - open/close 34389 (time 1744045962.51 total 72.28 last 438.88) - open/close 39219 (time 1744045972.51 total 82.29 last 482.92) - open/close 44615 (time 1744045982.51 total 92.29 last 539.52) - open/close 49713 (time 1744045992.51 total 102.29 last 509.78) - open/close 54270 (time 1744046002.51 total 112.29 last 455.64) - open/close 59410 (time 1744046012.52 total 122.29 last 513.93) - open/close 64061 (time 1744046022.52 total 132.29 last 465.02) total: 64768 open/close in 133.79 seconds: 484.09 ops/second - /unlink 9244 (time 1744046034.85 total 10.00 last 924.30) - /unlink 17238 (time 1744046044.85 total 20.00 last 799.36) - /unlink 20000 (time 1744046048.20 total 23.35 last 824.23) - /unlink 28905 (time 1744046058.20 total 33.35 last 890.44) - /unlink 38293 (time 1744046068.20 total 43.35 last 938.80) - /unlink 48246 (time 1744046078.20 total 53.35 last 995.25) - /unlink 50000 (time 1744046080.23 total 55.39 last 863.00) - /unlink 58419 (time 1744046090.23 total 65.39 last 841.80) - /unlink 60000 (time 1744046092.35 total 67.51 last 746.13) total: 64768 /unlink in 72.31 seconds: 895.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4929 (time 1744046133.54 total 10.00 last 492.87) - open/close 9597 (time 1744046143.54 total 20.00 last 466.80) - open/close 14769 (time 1744046153.54 total 30.00 last 517.14) - open/close 20000 (time 1744046162.83 total 39.29 last 562.95) - open/close 25368 (time 1744046172.83 total 49.30 last 536.73) - open/close 30000 (time 1744046181.11 total 57.57 last 559.80) - open/close 35587 (time 1744046191.11 total 67.57 last 558.70) - open/close 40000 (time 1744046200.66 total 77.13 last 461.77) - open/close 45708 (time 1744046210.67 total 87.13 last 570.64) - open/close 50000 (time 1744046218.37 total 94.83 last 557.38) - open/close 54839 (time 1744046228.37 total 104.83 last 483.85) - open/close 59546 (time 1744046238.37 total 114.83 last 470.64) - open/close 64695 (time 1744046248.37 total 124.83 last 514.89) total: 64768 open/close in 124.97 seconds: 518.27 ops/second - /unlink 9582 (time 1744046259.29 total 10.00 last 958.13) - /unlink 19775 (time 1744046269.29 total 20.00 last 1019.27) - /unlink 29995 (time 1744046279.29 total 30.00 last 1021.99) - /unlink 40000 (time 1744046289.26 total 39.97 last 1004.09) - /unlink 48981 (time 1744046299.26 total 49.97 last 898.03) - /unlink 57905 (time 1744046309.26 total 59.97 last 892.35) - /unlink 60000 (time 1744046311.55 total 62.26 last 912.87) total: 64768 /unlink in 67.66 seconds: 957.31 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5044 (time 1744046358.91 total 10.00 last 504.39) - open/close 10000 (time 1744046368.71 total 19.80 last 505.67) - open/close 15373 (time 1744046378.71 total 29.80 last 537.22) - open/close 20000 (time 1744046387.29 total 38.39 last 539.03) - open/close 25252 (time 1744046397.29 total 48.39 last 525.04) - open/close 30000 (time 1744046406.12 total 57.22 last 537.82) - open/close 35027 (time 1744046416.12 total 67.22 last 502.68) - open/close 40000 (time 1744046425.87 total 76.97 last 509.98) - open/close 45183 (time 1744046435.88 total 86.97 last 518.30) - open/close 50000 (time 1744046445.43 total 96.52 last 504.21) - open/close 55122 (time 1744046455.43 total 106.52 last 512.14) - open/close 60000 (time 1744046464.93 total 116.03 last 513.39) total: 64768 open/close in 125.31 seconds: 516.85 ops/second - /unlink 8823 (time 1744046485.03 total 10.00 last 882.28) - /unlink 17717 (time 1744046495.03 total 20.00 last 889.37) - /unlink 20000 (time 1744046497.57 total 22.54 last 899.70) - /unlink 28313 (time 1744046507.57 total 32.54 last 831.16) - /unlink 30000 (time 1744046509.62 total 34.58 last 825.85) - /unlink 37972 (time 1744046519.62 total 44.58 last 797.17) - /unlink 40000 (time 1744046521.96 total 46.93 last 863.51) - /unlink 48521 (time 1744046531.97 total 56.93 last 852.04) - /unlink 57599 (time 1744046541.97 total 66.93 last 907.80) - /unlink 60000 (time 1744046544.69 total 69.66 last 880.67) total: 64768 /unlink in 74.89 seconds: 864.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5206 (time 1744046584.47 total 10.00 last 520.52) - open/close 10000 (time 1744046593.86 total 19.40 last 510.09) - open/close 15133 (time 1744046603.87 total 29.40 last 513.25) - open/close 20000 (time 1744046613.53 total 39.06 last 503.79) - open/close 25059 (time 1744046623.53 total 49.06 last 505.89) - open/close 30000 (time 1744046633.45 total 58.99 last 497.78) - open/close 34982 (time 1744046643.45 total 68.99 last 498.11) - open/close 40000 (time 1744046653.31 total 78.84 last 509.28) - open/close 45170 (time 1744046663.31 total 88.84 last 516.98) - open/close 50000 (time 1744046672.77 total 98.31 last 510.46) - open/close 55061 (time 1744046682.77 total 108.31 last 506.07) - open/close 60000 (time 1744046692.60 total 118.13 last 502.51) total: 64768 open/close in 127.43 seconds: 508.27 ops/second - /unlink 9038 (time 1744046712.74 total 10.00 last 903.74) - /unlink 18164 (time 1744046722.74 total 20.00 last 912.58) - /unlink 20000 (time 1744046724.94 total 22.20 last 834.52) - /unlink 29058 (time 1744046734.94 total 32.20 last 905.71) - /unlink 38165 (time 1744046744.94 total 42.20 last 910.64) - /unlink 47282 (time 1744046754.94 total 52.20 last 911.61) - /unlink 50000 (time 1744046757.90 total 55.16 last 918.19) - /unlink 58882 (time 1744046767.90 total 65.16 last 888.16) total: 64768 /unlink in 71.75 seconds: 902.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4175 (time 1744046814.81 total 10.00 last 417.41) - open/close 9182 (time 1744046824.81 total 20.00 last 500.61) - open/close 13558 (time 1744046834.81 total 30.01 last 437.53) - open/close 18175 (time 1744046844.81 total 40.01 last 461.63) - open/close 20000 (time 1744046849.18 total 44.38 last 417.68) - open/close 24504 (time 1744046859.18 total 54.38 last 450.32) - open/close 29253 (time 1744046869.19 total 64.38 last 474.86) - open/close 34175 (time 1744046879.19 total 74.38 last 492.19) - open/close 38938 (time 1744046889.19 total 84.38 last 476.15) - open/close 40000 (time 1744046891.84 total 87.04 last 400.11) - open/close 44919 (time 1744046901.84 total 97.04 last 491.90) - open/close 49903 (time 1744046911.84 total 107.04 last 498.38) - open/close 54398 (time 1744046921.84 total 117.04 last 449.48) - open/close 58616 (time 1744046931.85 total 127.04 last 421.76) - open/close 60000 (time 1744046935.63 total 130.82 last 365.74) - open/close 64460 (time 1744046945.63 total 140.82 last 445.97) total: 64768 open/close in 141.39 seconds: 458.08 ops/second - /unlink 8694 (time 1744046957.04 total 10.00 last 869.14) - /unlink 14746 (time 1744046967.04 total 20.00 last 605.20) - /unlink 20000 (time 1744046976.84 total 29.81 last 536.00) - /unlink 25565 (time 1744046986.84 total 39.81 last 556.49) - /unlink 30000 (time 1744046994.05 total 47.01 last 615.17) - /unlink 35614 (time 1744047004.05 total 57.02 last 561.28) - /unlink 40000 (time 1744047010.38 total 63.34 last 693.46) - /unlink 48804 (time 1744047020.38 total 73.34 last 880.36) - /unlink 57030 (time 1744047030.38 total 83.34 last 822.58) - /unlink 60000 (time 1744047033.85 total 86.81 last 855.43) total: 64768 /unlink in 92.53 seconds: 699.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 PASS 106 (1213s) Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1290 sec ========== 13:31:37 (1744047097) === conf-sanity: start cleanup 13:31:37 (1744047097) === === conf-sanity: finish cleanup 13:31:38 (1744047098) ===