-----============= acceptance-small: conf-sanity ============----- Fri May 10 05:19:39 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg307-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg307-server oleg307-server: oleg307-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg307-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg307-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg307-server: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg307-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:20:49 (1715332849) Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) oleg307-server: oleg307-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg307-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg307-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1625 (time 1715332894.80 total 10.00 last 162.48) - open/close 3531 (time 1715332904.80 total 20.00 last 190.57) - open/close 5257 (time 1715332914.80 total 30.00 last 172.57) - open/close 6974 (time 1715332924.80 total 40.01 last 171.67) - open/close 9225 (time 1715332934.80 total 50.01 last 225.04) - open/close 10000 (time 1715332938.44 total 53.65 last 212.94) - open/close 11869 (time 1715332948.45 total 63.65 last 186.79) - open/close 13501 (time 1715332958.45 total 73.66 last 163.19) - open/close 14975 (time 1715332968.45 total 83.66 last 147.39) - open/close 16499 (time 1715332978.45 total 93.66 last 152.34) - open/close 18287 (time 1715332988.46 total 103.66 last 178.77) - open/close 20000 (time 1715332996.45 total 111.65 last 214.35) - open/close 22336 (time 1715333006.45 total 121.65 last 233.59) - open/close 24861 (time 1715333016.45 total 131.66 last 252.46) - open/close 26724 (time 1715333026.45 total 141.66 last 186.29) - open/close 28781 (time 1715333036.45 total 151.66 last 205.67) - open/close 30000 (time 1715333043.96 total 159.16 last 162.37) - open/close 31862 (time 1715333053.97 total 169.17 last 186.09) - open/close 33622 (time 1715333063.97 total 179.17 last 175.97) - open/close 35199 (time 1715333073.97 total 189.18 last 157.65) - open/close 36930 (time 1715333083.97 total 199.18 last 173.07) - open/close 38641 (time 1715333093.98 total 209.18 last 171.02) - open/close 40000 (time 1715333100.67 total 215.87 last 203.18) - open/close 42386 (time 1715333110.67 total 225.87 last 238.58) - open/close 44962 (time 1715333120.67 total 235.87 last 257.55) - open/close 47047 (time 1715333130.67 total 245.87 last 208.49) - open/close 49126 (time 1715333140.67 total 255.88 last 207.83) - open/close 50000 (time 1715333144.54 total 259.75 last 225.91) - open/close 52035 (time 1715333154.54 total 269.75 last 203.49) - open/close 53864 (time 1715333164.54 total 279.75 last 182.86) - open/close 56665 (time 1715333174.54 total 289.75 last 280.08) - open/close 58891 (time 1715333184.55 total 299.75 last 222.55) - open/close 60000 (time 1715333189.89 total 305.09 last 207.68) - open/close 62461 (time 1715333199.89 total 315.09 last 246.09) - open/close 64681 (time 1715333209.89 total 325.09 last 221.98) total: 64768 open/close in 325.41 seconds: 199.04 ops/second - /unlink 4311 (time 1715333221.21 total 10.00 last 431.01) - /unlink 8628 (time 1715333231.21 total 20.00 last 431.69) - /unlink 10000 (time 1715333234.86 total 23.66 last 375.44) - /unlink 13209 (time 1715333244.86 total 33.66 last 320.86) - /unlink 16784 (time 1715333254.87 total 43.66 last 357.42) - /unlink 20000 (time 1715333263.45 total 52.24 last 374.66) - /unlink 23689 (time 1715333273.45 total 62.24 last 368.90) - /unlink 27131 (time 1715333283.45 total 72.25 last 344.15) - /unlink 30000 (time 1715333290.54 total 79.33 last 405.03) - /unlink 33719 (time 1715333300.54 total 89.33 last 371.82) - /unlink 37448 (time 1715333310.54 total 99.33 last 372.78) - /unlink 40000 (time 1715333317.34 total 106.13 last 375.59) - /unlink 43693 (time 1715333327.34 total 116.13 last 369.29) - /unlink 47695 (time 1715333337.34 total 126.13 last 400.12) - /unlink 50000 (time 1715333342.47 total 131.26 last 449.44) - /unlink 54802 (time 1715333352.47 total 141.26 last 480.17) - /unlink 58843 (time 1715333362.47 total 151.26 last 404.03) - /unlink 60000 (time 1715333365.39 total 154.18 last 396.50) - /unlink 63756 (time 1715333375.39 total 164.18 last 375.54) total: 64768 /unlink in 166.68 seconds: 388.57 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2168 (time 1715333407.02 total 10.00 last 216.76) - open/close 4799 (time 1715333417.03 total 20.01 last 262.95) - open/close 7300 (time 1715333427.03 total 30.01 last 250.09) - open/close 10000 (time 1715333435.61 total 38.59 last 314.49) - open/close 12490 (time 1715333445.61 total 48.60 last 248.94) - open/close 15143 (time 1715333455.61 total 58.60 last 265.30) - open/close 17757 (time 1715333465.62 total 68.60 last 261.34) - open/close 20000 (time 1715333473.97 total 76.95 last 268.55) - open/close 22955 (time 1715333483.97 total 86.95 last 295.49) - open/close 25854 (time 1715333493.97 total 96.95 last 289.87) - open/close 29008 (time 1715333503.97 total 106.95 last 315.36) - open/close 30000 (time 1715333507.76 total 110.74 last 262.10) - open/close 32590 (time 1715333517.76 total 120.74 last 258.97) - open/close 35016 (time 1715333527.76 total 130.74 last 242.52) - open/close 37282 (time 1715333537.76 total 140.75 last 226.53) - open/close 39632 (time 1715333547.77 total 150.75 last 234.92) - open/close 42460 (time 1715333557.77 total 160.75 last 282.70) - open/close 45385 (time 1715333567.77 total 170.75 last 292.47) - open/close 47813 (time 1715333577.78 total 180.76 last 242.62) - open/close 50000 (time 1715333587.26 total 190.24 last 230.73) - open/close 52449 (time 1715333597.26 total 200.24 last 244.87) - open/close 55223 (time 1715333607.26 total 210.24 last 277.33) - open/close 57464 (time 1715333617.26 total 220.25 last 224.03) - open/close 59783 (time 1715333627.33 total 230.31 last 230.47) - open/close 62039 (time 1715333637.34 total 240.32 last 225.26) - open/close 64499 (time 1715333647.34 total 250.33 last 245.93) total: 64768 open/close in 251.24 seconds: 257.79 ops/second - /unlink 4606 (time 1715333659.22 total 10.00 last 460.57) - /unlink 8532 (time 1715333669.22 total 20.00 last 392.59) - /unlink 10000 (time 1715333672.50 total 23.28 last 448.05) - /unlink 14304 (time 1715333682.50 total 33.28 last 430.36) - /unlink 18297 (time 1715333692.50 total 43.28 last 399.27) - /unlink 20000 (time 1715333697.02 total 47.80 last 376.73) - /unlink 23852 (time 1715333707.02 total 57.80 last 385.14) - /unlink 27609 (time 1715333717.03 total 67.80 last 375.63) - /unlink 30000 (time 1715333723.30 total 74.08 last 380.99) - /unlink 34025 (time 1715333733.30 total 84.08 last 402.50) - /unlink 37806 (time 1715333743.30 total 94.08 last 378.03) - /unlink 40000 (time 1715333748.67 total 99.45 last 408.56) - /unlink 44304 (time 1715333758.67 total 109.45 last 430.37) - /unlink 48673 (time 1715333768.68 total 119.45 last 436.89) - /unlink 50000 (time 1715333772.09 total 122.87 last 388.38) - /unlink 53842 (time 1715333782.09 total 132.87 last 384.15) - /unlink 57600 (time 1715333792.10 total 142.87 last 375.64) - /unlink 60000 (time 1715333797.54 total 148.32 last 440.63) total: 64768 /unlink in 157.71 seconds: 410.68 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3205 (time 1715333832.34 total 10.00 last 320.43) - open/close 6413 (time 1715333842.34 total 20.00 last 320.79) - open/close 9302 (time 1715333852.34 total 30.00 last 288.86) - open/close 10000 (time 1715333855.21 total 32.88 last 242.66) - open/close 13545 (time 1715333865.22 total 42.88 last 354.41) - open/close 17198 (time 1715333875.22 total 52.89 last 365.21) - open/close 20000 (time 1715333882.86 total 60.53 last 366.72) - open/close 22952 (time 1715333892.86 total 70.53 last 295.11) - open/close 25719 (time 1715333902.86 total 80.53 last 276.70) - open/close 28703 (time 1715333912.87 total 90.53 last 298.33) - open/close 30000 (time 1715333916.98 total 94.65 last 314.88) - open/close 32708 (time 1715333926.99 total 104.65 last 270.77) - open/close 35557 (time 1715333936.99 total 114.65 last 284.88) - open/close 39102 (time 1715333946.99 total 124.65 last 354.47) - open/close 40000 (time 1715333950.02 total 127.68 last 296.31) - open/close 42963 (time 1715333969.11 total 146.78 last 155.20) - open/close 45571 (time 1715333979.11 total 156.78 last 260.79) - open/close 48627 (time 1715333989.11 total 166.78 last 305.58) - open/close 50000 (time 1715333993.04 total 170.70 last 349.76) - open/close 53470 (time 1715334003.04 total 180.70 last 346.93) - open/close 56997 (time 1715334013.04 total 190.71 last 352.62) - open/close 60000 (time 1715334022.09 total 199.76 last 331.77) - open/close 63363 (time 1715334032.09 total 209.76 last 336.27) total: 64768 open/close in 214.20 seconds: 302.37 ops/second - /unlink 5909 (time 1715334047.54 total 10.00 last 590.84) - /unlink 10000 (time 1715334053.78 total 16.24 last 655.25) - /unlink 15005 (time 1715334063.78 total 26.24 last 500.49) - /unlink 20000 (time 1715334072.98 total 35.44 last 542.98) - /unlink 25826 (time 1715334082.98 total 45.44 last 582.55) - /unlink 30000 (time 1715334090.56 total 53.03 last 550.50) - /unlink 36769 (time 1715334100.56 total 63.03 last 676.84) - /unlink 40000 (time 1715334105.35 total 67.82 last 674.34) - /unlink 46757 (time 1715334115.35 total 77.82 last 675.69) - /unlink 50000 (time 1715334120.22 total 82.68 last 667.00) - /unlink 56555 (time 1715334130.22 total 92.68 last 655.46) - /unlink 60000 (time 1715334135.62 total 98.09 last 637.32) total: 64768 /unlink in 106.66 seconds: 607.26 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4030 (time 1715334172.58 total 10.00 last 402.89) - open/close 8023 (time 1715334182.59 total 20.00 last 399.24) - open/close 10000 (time 1715334187.66 total 25.08 last 389.27) - open/close 12892 (time 1715334197.67 total 35.08 last 289.16) - open/close 16011 (time 1715334207.67 total 45.09 last 311.86) - open/close 19638 (time 1715334217.67 total 55.09 last 362.64) - open/close 23504 (time 1715334227.67 total 65.09 last 386.57) - open/close 27942 (time 1715334237.67 total 75.09 last 443.72) - open/close 30000 (time 1715334241.84 total 79.26 last 493.70) - open/close 34872 (time 1715334251.84 total 89.26 last 487.19) - open/close 39305 (time 1715334261.84 total 99.26 last 443.28) - open/close 43738 (time 1715334271.84 total 109.26 last 443.29) - open/close 47876 (time 1715334281.84 total 119.26 last 413.72) - open/close 50000 (time 1715334286.82 total 124.24 last 426.66) - open/close 54570 (time 1715334296.82 total 134.24 last 456.95) - open/close 58754 (time 1715334306.82 total 144.24 last 418.34) - open/close 60000 (time 1715334310.36 total 147.78 last 352.23) - open/close 64665 (time 1715334320.36 total 157.78 last 466.46) total: 64768 open/close in 158.01 seconds: 409.89 ops/second - /unlink 8079 (time 1715334331.50 total 10.00 last 807.82) - /unlink 10000 (time 1715334334.01 total 12.50 last 768.08) - /unlink 17633 (time 1715334344.01 total 22.50 last 763.24) - /unlink 20000 (time 1715334347.34 total 25.84 last 709.22) - /unlink 26119 (time 1715334357.34 total 35.84 last 611.83) - /unlink 30000 (time 1715334362.32 total 40.82 last 779.63) - /unlink 37578 (time 1715334372.32 total 50.82 last 757.77) - /unlink 40000 (time 1715334375.48 total 53.98 last 766.15) - /unlink 48027 (time 1715334385.49 total 63.98 last 802.64) - /unlink 50000 (time 1715334388.02 total 66.51 last 778.92) - /unlink 57185 (time 1715334398.02 total 76.52 last 718.42) - /unlink 60000 (time 1715334401.62 total 80.12 last 781.42) total: 64768 /unlink in 86.43 seconds: 749.35 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4943 (time 1715334438.81 total 10.00 last 494.24) - open/close 10000 (time 1715334448.63 total 19.82 last 514.97) - open/close 15199 (time 1715334458.64 total 29.82 last 519.83) - open/close 20000 (time 1715334467.96 total 39.15 last 514.68) - open/close 24057 (time 1715334477.96 total 49.15 last 405.67) - open/close 28232 (time 1715334487.97 total 59.15 last 417.42) - open/close 30000 (time 1715334491.83 total 63.02 last 457.22) - open/close 34951 (time 1715334501.83 total 73.02 last 495.10) - open/close 40000 (time 1715334511.24 total 82.42 last 537.00) - open/close 45416 (time 1715334521.24 total 92.42 last 541.52) - open/close 50000 (time 1715334529.42 total 100.61 last 559.90) - open/close 55938 (time 1715334539.42 total 110.61 last 593.76) - open/close 60000 (time 1715334546.81 total 118.00 last 550.00) total: 64768 open/close in 127.07 seconds: 509.69 ops/second - /unlink 9526 (time 1715334566.66 total 10.00 last 952.59) - /unlink 19539 (time 1715334576.66 total 20.00 last 1001.22) - /unlink 29554 (time 1715334586.67 total 30.00 last 1001.43) - /unlink 39298 (time 1715334596.67 total 40.00 last 974.37) - /unlink 48827 (time 1715334606.67 total 50.00 last 952.82) - /unlink 58996 (time 1715334616.67 total 60.00 last 1016.85) total: 64768 /unlink in 66.19 seconds: 978.51 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server unloading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 PASS 106 (1828s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg307-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg307-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 oleg307-server: oleg307-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg307-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg307-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1915 sec ========== 05:51:35 (1715334695) === conf-sanity: start cleanup 05:51:35 (1715334695) === === conf-sanity: finish cleanup 05:51:35 (1715334695) ===