-----============= acceptance-small: conf-sanity ============----- Thu Jun 13 18:04:10 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg316-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg316-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg316-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg316-server oleg316-server: oleg316-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: detected 4 online CPUs by sysfs oleg316-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg316-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg316-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg316-server: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg316-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:05:08 (1718316308) Stopping clients: oleg316-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg316-client.virtnet /mnt/lustre2 (opts:-f) oleg316-server: oleg316-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: detected 4 online CPUs by sysfs oleg316-server: Force libcfs to create 2 CPU partitions oleg316-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg316-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg316-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock oleg316-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3969 (time 1718316342.32 total 10.00 last 396.89) - open/close 8253 (time 1718316352.33 total 20.00 last 428.36) - open/close 10000 (time 1718316356.43 total 24.11 last 425.29) - open/close 14140 (time 1718316366.43 total 34.11 last 413.96) - open/close 18616 (time 1718316376.44 total 44.11 last 447.56) - open/close 20000 (time 1718316379.69 total 47.36 last 425.54) - open/close 24142 (time 1718316389.69 total 57.36 last 414.13) - open/close 28530 (time 1718316399.69 total 67.36 last 438.79) - open/close 30000 (time 1718316402.37 total 70.04 last 549.28) - open/close 35200 (time 1718316412.37 total 80.04 last 519.92) - open/close 40000 (time 1718316421.31 total 88.99 last 536.77) - open/close 45557 (time 1718316431.31 total 98.99 last 555.60) - open/close 50000 (time 1718316439.78 total 107.46 last 524.52) - open/close 55344 (time 1718316449.78 total 117.46 last 534.34) - open/close 60000 (time 1718316459.45 total 127.12 last 481.70) - open/close 64606 (time 1718316469.45 total 137.13 last 460.53) total: 64768 open/close in 137.46 seconds: 471.16 ops/second - /unlink 7944 (time 1718316480.71 total 10.00 last 794.35) - /unlink 10000 (time 1718316483.32 total 12.60 last 789.47) - /unlink 17788 (time 1718316493.32 total 22.61 last 778.72) - /unlink 20000 (time 1718316496.45 total 25.73 last 707.04) - /unlink 26278 (time 1718316506.45 total 35.74 last 627.76) - /unlink 30000 (time 1718316511.65 total 40.93 last 715.83) - /unlink 37875 (time 1718316521.65 total 50.94 last 787.42) - /unlink 40000 (time 1718316524.16 total 53.45 last 844.45) - /unlink 48357 (time 1718316534.16 total 63.45 last 835.59) - /unlink 50000 (time 1718316536.38 total 65.66 last 742.94) - /unlink 57664 (time 1718316546.38 total 75.67 last 766.32) - /unlink 60000 (time 1718316549.48 total 78.77 last 753.25) total: 64768 /unlink in 84.56 seconds: 765.91 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5002 (time 1718316585.91 total 10.00 last 500.16) - open/close 10000 (time 1718316595.41 total 19.50 last 526.15) - open/close 15551 (time 1718316605.41 total 29.50 last 555.07) - open/close 20000 (time 1718316613.52 total 37.61 last 548.93) - open/close 24949 (time 1718316623.52 total 47.61 last 494.84) - open/close 29751 (time 1718316633.52 total 57.61 last 480.19) - open/close 34279 (time 1718316643.52 total 67.61 last 452.77) - open/close 38807 (time 1718316653.52 total 77.61 last 452.77) - open/close 40000 (time 1718316656.27 total 80.35 last 434.34) - open/close 44283 (time 1718316666.27 total 90.35 last 428.30) - open/close 48657 (time 1718316676.27 total 100.36 last 437.35) - open/close 50000 (time 1718316679.69 total 103.78 last 392.36) - open/close 54594 (time 1718316689.69 total 113.78 last 459.34) - open/close 59411 (time 1718316699.69 total 123.78 last 481.66) - open/close 64208 (time 1718316709.69 total 133.78 last 479.69) total: 64768 open/close in 134.96 seconds: 479.90 ops/second - /unlink 8422 (time 1718316721.75 total 10.00 last 842.12) - /unlink 16656 (time 1718316731.75 total 20.00 last 823.40) - /unlink 20000 (time 1718316735.76 total 24.02 last 832.84) - /unlink 28297 (time 1718316745.76 total 34.02 last 829.64) - /unlink 30000 (time 1718316747.78 total 36.04 last 842.24) - /unlink 38488 (time 1718316757.79 total 46.04 last 848.73) - /unlink 46866 (time 1718316767.79 total 56.04 last 837.75) - /unlink 50000 (time 1718316771.49 total 59.75 last 845.36) - /unlink 58957 (time 1718316781.49 total 69.75 last 895.65) total: 64768 /unlink in 76.13 seconds: 850.74 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5316 (time 1718316821.44 total 10.00 last 531.56) - open/close 10000 (time 1718316830.29 total 18.85 last 529.13) - open/close 15313 (time 1718316840.30 total 28.85 last 531.25) - open/close 20000 (time 1718316849.14 total 37.70 last 529.99) - open/close 25041 (time 1718316859.14 total 47.70 last 504.04) - open/close 29950 (time 1718316869.14 total 57.70 last 490.84) - open/close 34997 (time 1718316879.14 total 67.70 last 504.69) - open/close 40000 (time 1718316888.75 total 77.31 last 520.62) - open/close 45042 (time 1718316898.75 total 87.31 last 504.20) - open/close 49800 (time 1718316908.75 total 97.31 last 475.73) - open/close 54840 (time 1718316918.75 total 107.31 last 503.90) - open/close 59952 (time 1718316928.76 total 117.32 last 511.12) total: 64768 open/close in 127.27 seconds: 508.91 ops/second - /unlink 8493 (time 1718316949.60 total 10.00 last 849.24) - /unlink 17009 (time 1718316959.60 total 20.00 last 851.49) - /unlink 20000 (time 1718316964.25 total 24.65 last 643.01) - /unlink 27820 (time 1718316974.25 total 34.65 last 781.95) - /unlink 30000 (time 1718316976.91 total 37.32 last 819.21) - /unlink 37726 (time 1718316986.91 total 47.32 last 772.53) - /unlink 40000 (time 1718316989.63 total 50.04 last 834.93) - /unlink 47812 (time 1718316999.63 total 60.04 last 781.19) - /unlink 50000 (time 1718317002.63 total 63.03 last 731.11) - /unlink 57632 (time 1718317012.63 total 73.03 last 763.03) - /unlink 60000 (time 1718317015.75 total 76.15 last 759.35) total: 64768 /unlink in 82.37 seconds: 786.34 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3141 (time 1718317052.02 total 10.00 last 314.07) - open/close 7903 (time 1718317062.02 total 20.00 last 476.19) - open/close 10000 (time 1718317069.29 total 27.28 last 288.23) - open/close 11522 (time 1718317079.29 total 37.28 last 152.17) - open/close 13338 (time 1718317089.29 total 47.28 last 181.59) - open/close 14427 (time 1718317099.30 total 57.28 last 108.83) - open/close 15309 (time 1718317109.31 total 67.29 last 88.13) - open/close 16932 (time 1718317119.31 total 77.30 last 162.26) - open/close 18320 (time 1718317129.31 total 87.30 last 138.78) - open/close 20000 (time 1718317138.52 total 96.50 last 182.49) - open/close 21345 (time 1718317148.52 total 106.51 last 134.46) - open/close 22359 (time 1718317158.53 total 116.51 last 101.33) - open/close 23436 (time 1718317168.53 total 126.52 last 107.63) - open/close 24602 (time 1718317178.54 total 136.52 last 116.59) - open/close 25915 (time 1718317188.54 total 146.52 last 131.25) - open/close 27376 (time 1718317198.54 total 156.53 last 146.04) - open/close 28861 (time 1718317208.55 total 166.53 last 148.42) - open/close 30000 (time 1718317216.94 total 174.93 last 135.70) - open/close 31477 (time 1718317226.94 total 184.93 last 147.68) - open/close 33282 (time 1718317236.95 total 194.93 last 180.39) - open/close 34821 (time 1718317246.95 total 204.94 last 153.88) - open/close 36698 (time 1718317256.95 total 214.94 last 187.69) - open/close 38327 (time 1718317266.95 total 224.94 last 162.89) - open/close 39900 (time 1718317276.95 total 234.94 last 157.29) - open/close 41475 (time 1718317286.95 total 244.94 last 157.49) - open/close 43283 (time 1718317296.96 total 254.94 last 180.74) - open/close 45971 (time 1718317306.96 total 264.94 last 268.78) - open/close 49088 (time 1718317316.96 total 274.94 last 311.67) - open/close 50000 (time 1718317320.55 total 278.53 last 254.16) - open/close 53191 (time 1718317330.55 total 288.54 last 318.99) - open/close 55704 (time 1718317340.55 total 298.54 last 251.27) - open/close 59073 (time 1718317350.55 total 308.54 last 336.84) - open/close 60000 (time 1718317353.48 total 311.47 last 316.63) - open/close 62087 (time 1718317363.48 total 321.47 last 208.66) total: 64768 open/close in 329.33 seconds: 196.66 ops/second - /unlink 4481 (time 1718317382.29 total 10.00 last 448.07) - /unlink 8397 (time 1718317392.29 total 20.00 last 391.55) - /unlink 10000 (time 1718317396.12 total 23.84 last 418.13) - /unlink 13515 (time 1718317406.12 total 33.84 last 351.46) - /unlink 15899 (time 1718317416.13 total 43.84 last 238.38) - /unlink 18371 (time 1718317426.13 total 53.84 last 247.17) - /unlink 20000 (time 1718317430.59 total 58.30 last 365.20) - /unlink 23617 (time 1718317440.59 total 68.30 last 361.67) - /unlink 25758 (time 1718317450.59 total 78.30 last 214.04) - /unlink 27658 (time 1718317460.59 total 88.30 last 190.00) - /unlink 29491 (time 1718317470.59 total 98.31 last 183.25) - /unlink 30000 (time 1718317473.29 total 101.00 last 189.13) - /unlink 31938 (time 1718317483.29 total 111.00 last 193.72) - /unlink 34054 (time 1718317493.29 total 121.00 last 211.53) - /unlink 36317 (time 1718317503.30 total 131.01 last 226.23) - /unlink 38549 (time 1718317513.30 total 141.01 last 223.12) - /unlink 40000 (time 1718317519.94 total 147.65 last 218.61) - /unlink 42210 (time 1718317529.94 total 157.65 last 220.97) - /unlink 44333 (time 1718317539.94 total 167.65 last 212.21) - /unlink 46525 (time 1718317549.94 total 177.66 last 219.16) - /unlink 48885 (time 1718317559.95 total 187.66 last 235.94) - /unlink 50000 (time 1718317563.75 total 191.46 last 293.55) - /unlink 53310 (time 1718317573.75 total 201.46 last 330.95) - /unlink 56304 (time 1718317583.75 total 211.46 last 299.38) - /unlink 58836 (time 1718317593.75 total 221.46 last 253.16) - /unlink 60000 (time 1718317598.68 total 226.40 last 235.86) - /unlink 62564 (time 1718317608.69 total 236.40 last 256.35) total: 64768 /unlink in 245.29 seconds: 264.05 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1290 (time 1718317648.52 total 10.01 last 128.89) - open/close 2736 (time 1718317658.53 total 20.01 last 144.54) - open/close 4220 (time 1718317668.53 total 30.02 last 148.35) - open/close 6360 (time 1718317678.53 total 40.02 last 213.99) - open/close 8907 (time 1718317688.53 total 50.02 last 254.63) - open/close 10000 (time 1718317694.64 total 56.13 last 178.86) - open/close 12191 (time 1718317704.65 total 66.13 last 219.04) - open/close 14993 (time 1718317714.65 total 76.14 last 280.12) - open/close 19522 (time 1718317724.65 total 86.14 last 452.85) - open/close 23898 (time 1718317734.65 total 96.14 last 437.60) - open/close 27701 (time 1718317744.65 total 106.14 last 380.28) - open/close 30000 (time 1718317751.26 total 112.75 last 347.62) - open/close 35297 (time 1718317761.27 total 122.75 last 529.66) - open/close 40000 (time 1718317769.23 total 130.72 last 590.12) - open/close 45358 (time 1718317779.24 total 140.72 last 535.76) - open/close 50000 (time 1718317788.36 total 149.84 last 508.92) - open/close 55023 (time 1718317798.36 total 159.85 last 502.24) - open/close 59633 (time 1718317808.39 total 169.88 last 459.51) - open/close 63586 (time 1718317818.39 total 179.88 last 395.29) total: 64768 open/close in 182.29 seconds: 355.29 ops/second - /unlink 8063 (time 1718317831.78 total 10.00 last 806.21) - /unlink 10000 (time 1718317833.96 total 12.19 last 885.95) - /unlink 18059 (time 1718317843.97 total 22.19 last 805.79) - /unlink 20000 (time 1718317846.24 total 24.46 last 855.23) - /unlink 28206 (time 1718317856.24 total 34.46 last 820.55) - /unlink 30000 (time 1718317858.27 total 36.49 last 881.25) - /unlink 38721 (time 1718317868.27 total 46.50 last 872.02) - /unlink 47086 (time 1718317878.27 total 56.50 last 836.36) - /unlink 50000 (time 1718317882.46 total 60.68 last 696.44) - /unlink 56799 (time 1718317892.46 total 70.68 last 679.86) - /unlink 60000 (time 1718317896.08 total 74.31 last 883.45) total: 64768 /unlink in 79.41 seconds: 815.64 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server unloading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 PASS 106 (1634s) Stopping clients: oleg316-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg316-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 oleg316-server: oleg316-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: detected 4 online CPUs by sysfs oleg316-server: Force libcfs to create 2 CPU partitions oleg316-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg316-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg316-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1708 sec ========== 18:32:39 (1718317959) === conf-sanity: start cleanup 18:32:40 (1718317960) === === conf-sanity: finish cleanup 18:32:40 (1718317960) ===