-----============= acceptance-small: conf-sanity ============----- Tue Nov 5 05:16:34 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg107-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg107-server oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:17:54 (1730801874) Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1664 (time 1730801924.09 total 10.01 last 166.30) - open/close 3141 (time 1730801934.09 total 20.01 last 147.65) - open/close 4862 (time 1730801944.09 total 30.01 last 172.04) - open/close 6419 (time 1730801954.09 total 40.01 last 155.69) - open/close 7844 (time 1730801964.09 total 50.01 last 142.49) - open/close 9174 (time 1730801974.10 total 60.02 last 132.93) - open/close 10000 (time 1730801980.65 total 66.57 last 126.05) - open/close 11499 (time 1730801990.66 total 76.58 last 149.76) - open/close 12925 (time 1730802000.66 total 86.58 last 142.56) - open/close 14415 (time 1730802010.67 total 96.59 last 148.95) - open/close 15880 (time 1730802020.67 total 106.59 last 146.43) - open/close 17185 (time 1730802030.68 total 116.60 last 130.38) - open/close 18617 (time 1730802040.68 total 126.60 last 143.19) - open/close 19912 (time 1730802050.69 total 136.61 last 129.44) - open/close 21248 (time 1730802060.69 total 146.61 last 133.56) - open/close 22579 (time 1730802070.69 total 156.61 last 133.10) - open/close 23744 (time 1730802080.70 total 166.62 last 116.34) - open/close 24955 (time 1730802090.71 total 176.63 last 121.04) - open/close 26157 (time 1730802100.71 total 186.63 last 120.16) - open/close 27423 (time 1730802110.72 total 196.64 last 126.47) - open/close 28924 (time 1730802120.73 total 206.65 last 149.96) - open/close 30000 (time 1730802127.74 total 213.66 last 153.51) - open/close 31909 (time 1730802137.74 total 223.66 last 190.87) - open/close 33695 (time 1730802147.74 total 233.66 last 178.55) - open/close 35588 (time 1730802157.75 total 243.67 last 189.20) - open/close 37455 (time 1730802167.75 total 253.67 last 186.63) - open/close 39448 (time 1730802177.76 total 263.68 last 199.21) - open/close 40000 (time 1730802180.46 total 266.39 last 203.93) - open/close 42009 (time 1730802190.47 total 276.39 last 200.86) - open/close 43709 (time 1730802200.47 total 286.39 last 169.98) - open/close 45463 (time 1730802210.47 total 296.39 last 175.31) - open/close 47026 (time 1730802220.47 total 306.40 last 156.26) - open/close 48670 (time 1730802230.48 total 316.40 last 164.37) - open/close 50000 (time 1730802237.28 total 323.20 last 195.56) - open/close 52313 (time 1730802247.28 total 333.20 last 231.27) - open/close 54391 (time 1730802257.28 total 343.20 last 207.79) - open/close 56544 (time 1730802267.28 total 353.20 last 215.27) - open/close 58736 (time 1730802277.28 total 363.20 last 219.15) - open/close 60000 (time 1730802282.82 total 368.74 last 228.26) - open/close 62257 (time 1730802292.82 total 378.74 last 225.64) - open/close 64510 (time 1730802302.83 total 388.75 last 225.22) total: 64768 open/close in 390.10 seconds: 166.03 ops/second - /unlink 3490 (time 1730802315.52 total 10.00 last 348.95) - /unlink 7245 (time 1730802325.52 total 20.00 last 375.45) - /unlink 10000 (time 1730802333.19 total 27.68 last 359.07) - /unlink 14034 (time 1730802343.19 total 37.68 last 403.29) - /unlink 18243 (time 1730802353.19 total 47.68 last 420.80) - /unlink 20000 (time 1730802357.75 total 52.24 last 385.55) - /unlink 24247 (time 1730802367.75 total 62.24 last 424.63) - /unlink 29239 (time 1730802377.76 total 72.24 last 499.12) - /unlink 34382 (time 1730802387.76 total 82.24 last 514.26) - /unlink 38715 (time 1730802397.76 total 92.24 last 433.24) - /unlink 40000 (time 1730802400.38 total 94.87 last 489.18) - /unlink 45582 (time 1730802410.39 total 104.87 last 558.11) - /unlink 50000 (time 1730802418.43 total 112.92 last 549.04) - /unlink 55709 (time 1730802428.43 total 122.92 last 570.85) - /unlink 60000 (time 1730802435.47 total 129.95 last 609.92) total: 64768 /unlink in 139.43 seconds: 464.53 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2878 (time 1730802471.57 total 10.01 last 287.64) - open/close 5736 (time 1730802481.57 total 20.01 last 285.78) - open/close 8235 (time 1730802491.57 total 30.01 last 249.78) - open/close 10000 (time 1730802499.12 total 37.56 last 233.93) - open/close 13365 (time 1730802509.12 total 47.56 last 336.46) - open/close 16748 (time 1730802519.13 total 57.57 last 337.77) - open/close 19612 (time 1730802529.14 total 67.58 last 286.31) - open/close 22332 (time 1730802539.14 total 77.58 last 271.97) - open/close 25306 (time 1730802549.14 total 87.58 last 297.34) - open/close 28210 (time 1730802559.14 total 97.58 last 290.30) - open/close 30000 (time 1730802565.91 total 104.35 last 264.44) - open/close 32938 (time 1730802575.91 total 114.35 last 293.78) - open/close 35758 (time 1730802585.92 total 124.35 last 281.93) - open/close 38417 (time 1730802595.92 total 134.36 last 265.87) - open/close 40000 (time 1730802601.14 total 139.58 last 303.06) - open/close 43209 (time 1730802611.14 total 149.58 last 320.87) - open/close 46200 (time 1730802621.14 total 159.58 last 299.10) - open/close 48899 (time 1730802631.15 total 169.58 last 269.81) - open/close 50000 (time 1730802635.75 total 174.19 last 239.10) - open/close 52862 (time 1730802645.75 total 184.19 last 286.19) - open/close 56438 (time 1730802655.75 total 194.19 last 357.57) - open/close 60000 (time 1730802665.05 total 203.49 last 382.87) - open/close 63573 (time 1730802675.06 total 213.49 last 357.26) total: 64768 open/close in 217.00 seconds: 298.47 ops/second - /unlink 6537 (time 1730802689.53 total 10.00 last 653.61) - /unlink 10000 (time 1730802694.19 total 14.66 last 742.68) - /unlink 17378 (time 1730802704.19 total 24.67 last 737.70) - /unlink 20000 (time 1730802707.73 total 28.20 last 741.35) - /unlink 27619 (time 1730802717.73 total 38.20 last 761.78) - /unlink 30000 (time 1730802720.58 total 41.05 last 835.71) - /unlink 38455 (time 1730802730.58 total 51.05 last 845.47) - /unlink 46707 (time 1730802740.58 total 61.05 last 825.10) - /unlink 50000 (time 1730802745.13 total 65.60 last 723.96) - /unlink 56082 (time 1730802755.13 total 75.60 last 608.17) - /unlink 60000 (time 1730802761.46 total 81.93 last 619.28) total: 64768 /unlink in 90.10 seconds: 718.88 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3424 (time 1730802795.94 total 10.00 last 342.37) - open/close 8557 (time 1730802805.94 total 20.00 last 513.21) - open/close 10000 (time 1730802808.86 total 22.91 last 495.50) - open/close 13671 (time 1730802818.86 total 32.91 last 367.09) - open/close 17681 (time 1730802828.86 total 42.92 last 400.95) - open/close 20000 (time 1730802835.20 total 49.26 last 365.46) - open/close 24413 (time 1730802845.20 total 59.26 last 441.21) - open/close 28692 (time 1730802855.20 total 69.26 last 427.88) - open/close 30000 (time 1730802858.27 total 72.33 last 426.72) - open/close 34408 (time 1730802868.27 total 82.33 last 440.71) - open/close 38881 (time 1730802878.35 total 92.41 last 443.78) - open/close 40000 (time 1730802881.74 total 95.80 last 330.19) - open/close 42975 (time 1730802891.74 total 105.80 last 297.38) - open/close 45733 (time 1730802901.75 total 115.81 last 275.76) - open/close 49301 (time 1730802911.75 total 125.81 last 356.69) - open/close 50000 (time 1730802913.93 total 127.99 last 320.52) - open/close 53257 (time 1730802923.93 total 137.99 last 325.65) - open/close 56999 (time 1730802933.93 total 147.99 last 374.20) - open/close 60000 (time 1730802941.39 total 155.45 last 402.19) - open/close 64604 (time 1730802951.39 total 165.45 last 460.32) total: 64768 open/close in 165.84 seconds: 390.53 ops/second - /unlink 8057 (time 1730802962.63 total 10.00 last 805.61) - /unlink 10000 (time 1730802965.05 total 12.42 last 803.99) - /unlink 16825 (time 1730802975.05 total 22.42 last 682.43) - /unlink 20000 (time 1730802979.08 total 26.45 last 788.17) - /unlink 28146 (time 1730802989.08 total 36.45 last 814.59) - /unlink 30000 (time 1730802991.42 total 38.79 last 792.54) - /unlink 38624 (time 1730803001.42 total 48.79 last 862.28) - /unlink 47161 (time 1730803011.42 total 58.79 last 853.63) - /unlink 50000 (time 1730803014.50 total 61.87 last 921.43) - /unlink 59030 (time 1730803024.50 total 71.87 last 902.92) total: 64768 /unlink in 77.95 seconds: 830.91 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5316 (time 1730803062.03 total 10.00 last 531.60) - open/close 10000 (time 1730803070.01 total 17.98 last 587.03) - open/close 15573 (time 1730803080.01 total 27.98 last 557.26) - open/close 20000 (time 1730803088.32 total 36.28 last 533.10) - open/close 25340 (time 1730803098.32 total 46.29 last 533.94) - open/close 30000 (time 1730803107.00 total 54.97 last 536.73) - open/close 35428 (time 1730803117.00 total 64.97 last 542.77) - open/close 40000 (time 1730803125.56 total 73.53 last 533.82) - open/close 45355 (time 1730803135.57 total 83.53 last 535.44) - open/close 50000 (time 1730803144.22 total 92.19 last 536.50) - open/close 55324 (time 1730803154.23 total 102.19 last 532.30) - open/close 60000 (time 1730803162.75 total 110.71 last 548.79) total: 64768 open/close in 119.33 seconds: 542.78 ops/second - /unlink 9228 (time 1730803182.20 total 10.00 last 922.74) - /unlink 18536 (time 1730803192.20 total 20.00 last 930.73) - /unlink 27663 (time 1730803202.20 total 30.00 last 912.65) - /unlink 30000 (time 1730803204.65 total 32.45 last 954.59) - /unlink 39293 (time 1730803214.65 total 42.45 last 929.23) - /unlink 48715 (time 1730803224.65 total 52.45 last 942.16) - /unlink 58283 (time 1730803234.65 total 62.45 last 956.80) total: 64768 /unlink in 70.59 seconds: 917.55 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5511 (time 1730803281.77 total 10.00 last 551.02) - open/close 10000 (time 1730803289.74 total 17.98 last 562.81) - open/close 15638 (time 1730803299.75 total 27.98 last 563.74) - open/close 20000 (time 1730803307.60 total 35.84 last 555.03) - open/close 25434 (time 1730803317.61 total 45.84 last 543.32) - open/close 30000 (time 1730803326.01 total 54.24 last 543.55) - open/close 35435 (time 1730803336.01 total 64.24 last 543.46) - open/close 40000 (time 1730803344.55 total 72.78 last 534.30) - open/close 45375 (time 1730803354.55 total 82.78 last 537.48) - open/close 50000 (time 1730803363.21 total 91.45 last 533.93) - open/close 55150 (time 1730803373.22 total 101.45 last 514.92) - open/close 60000 (time 1730803382.53 total 110.76 last 520.71) total: 64768 open/close in 119.75 seconds: 540.84 ops/second - /unlink 8681 (time 1730803402.38 total 10.00 last 868.07) - /unlink 17732 (time 1730803412.38 total 20.00 last 905.03) - /unlink 20000 (time 1730803414.84 total 22.46 last 921.36) - /unlink 29065 (time 1730803424.84 total 32.46 last 906.50) - /unlink 38241 (time 1730803434.84 total 42.46 last 917.59) - /unlink 40000 (time 1730803436.85 total 44.46 last 878.58) - /unlink 48957 (time 1730803446.85 total 54.47 last 895.63) - /unlink 58285 (time 1730803456.85 total 64.47 last 932.76) total: 64768 /unlink in 71.22 seconds: 909.41 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server unloading modules via unload_modules_local on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 PASS 106 (1637s) Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1733 sec ========== 05:45:28 (1730803528) === conf-sanity: start cleanup 05:45:28 (1730803528) === === conf-sanity: finish cleanup 05:45:29 (1730803529) ===