-----============= acceptance-small: conf-sanity ============----- Thu Apr 17 05:45:46 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg360-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg360-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg360-server oleg360-server: oleg360-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg360-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg360-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg360-server: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg360-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:46:43 (1744883203) Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) oleg360-server: oleg360-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg360-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg360-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2871 (time 1744883241.04 total 10.00 last 287.04) - open/close 6065 (time 1744883251.05 total 20.01 last 319.25) - open/close 8668 (time 1744883261.05 total 30.01 last 260.29) - open/close 10000 (time 1744883265.12 total 34.08 last 327.14) - open/close 12459 (time 1744883275.12 total 44.08 last 245.87) - open/close 15018 (time 1744883285.12 total 54.08 last 255.84) - open/close 17250 (time 1744883295.12 total 64.08 last 223.18) - open/close 19753 (time 1744883305.12 total 74.08 last 250.28) - open/close 22711 (time 1744883315.12 total 84.09 last 295.77) - open/close 25824 (time 1744883325.13 total 94.09 last 311.25) - open/close 29211 (time 1744883335.13 total 104.09 last 338.66) - open/close 30000 (time 1744883337.15 total 106.11 last 390.89) - open/close 33780 (time 1744883347.15 total 116.11 last 377.90) - open/close 37280 (time 1744883357.15 total 126.11 last 349.99) - open/close 40000 (time 1744883365.10 total 134.06 last 342.20) - open/close 43842 (time 1744883375.10 total 144.06 last 384.20) - open/close 47787 (time 1744883385.10 total 154.06 last 394.49) - open/close 50000 (time 1744883390.75 total 159.71 last 391.81) - open/close 53548 (time 1744883400.75 total 169.71 last 354.72) - open/close 56489 (time 1744883410.75 total 179.71 last 294.09) - open/close 59207 (time 1744883420.75 total 189.71 last 271.78) - open/close 60000 (time 1744883423.27 total 192.24 last 313.96) - open/close 62393 (time 1744883433.28 total 202.24 last 239.16) - open/close 64454 (time 1744883443.28 total 212.25 last 206.02) total: 64768 open/close in 213.90 seconds: 302.79 ops/second - /unlink 3229 (time 1744883456.05 total 10.00 last 322.84) - /unlink 6726 (time 1744883466.05 total 20.00 last 349.67) - /unlink 10000 (time 1744883471.78 total 25.73 last 571.49) - /unlink 15216 (time 1744883481.78 total 35.73 last 521.52) - /unlink 20000 (time 1744883489.12 total 43.07 last 651.64) - /unlink 28020 (time 1744883499.12 total 53.08 last 801.91) - /unlink 30000 (time 1744883501.84 total 55.79 last 728.29) - /unlink 37328 (time 1744883511.84 total 65.80 last 732.74) - /unlink 40000 (time 1744883515.65 total 69.60 last 701.74) - /unlink 47875 (time 1744883525.65 total 79.60 last 787.44) - /unlink 50000 (time 1744883528.22 total 82.18 last 826.34) - /unlink 57373 (time 1744883538.22 total 92.18 last 737.21) - /unlink 60000 (time 1744883541.90 total 95.85 last 715.08) total: 64768 /unlink in 103.19 seconds: 627.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3616 (time 1744883578.67 total 10.00 last 361.56) - open/close 7754 (time 1744883588.67 total 20.00 last 413.71) - open/close 10000 (time 1744883594.04 total 25.37 last 418.23) - open/close 14231 (time 1744883604.04 total 35.38 last 423.03) - open/close 18387 (time 1744883614.05 total 45.38 last 415.54) - open/close 20000 (time 1744883617.73 total 49.06 last 437.35) - open/close 23511 (time 1744883627.73 total 59.07 last 351.06) - open/close 26822 (time 1744883637.74 total 69.07 last 331.06) - open/close 30000 (time 1744883647.01 total 78.34 last 342.81) - open/close 33507 (time 1744883657.01 total 88.34 last 350.63) - open/close 37831 (time 1744883667.01 total 98.34 last 432.33) - open/close 40000 (time 1744883671.63 total 102.96 last 469.51) - open/close 44327 (time 1744883681.63 total 112.96 last 432.68) - open/close 48160 (time 1744883691.63 total 122.96 last 383.23) - open/close 50000 (time 1744883696.01 total 127.34 last 420.03) - open/close 54879 (time 1744883706.01 total 137.34 last 487.85) - open/close 59845 (time 1744883716.02 total 147.35 last 496.53) total: 64768 open/close in 156.84 seconds: 412.95 ops/second - /unlink 8178 (time 1744883736.31 total 10.00 last 817.75) - /unlink 10000 (time 1744883738.62 total 12.31 last 789.28) - /unlink 18203 (time 1744883748.62 total 22.31 last 820.28) - /unlink 27287 (time 1744883758.62 total 32.31 last 908.31) - /unlink 30000 (time 1744883761.76 total 35.45 last 863.45) - /unlink 37301 (time 1744883771.76 total 45.45 last 730.04) - /unlink 40000 (time 1744883775.14 total 48.83 last 799.78) - /unlink 48396 (time 1744883785.14 total 58.83 last 839.59) - /unlink 57725 (time 1744883795.14 total 68.83 last 932.83) - /unlink 60000 (time 1744883797.55 total 71.24 last 942.27) total: 64768 /unlink in 77.27 seconds: 838.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4731 (time 1744883833.61 total 10.00 last 473.03) - open/close 9829 (time 1744883843.61 total 20.00 last 509.72) - open/close 14371 (time 1744883853.62 total 30.01 last 454.09) - open/close 17696 (time 1744883863.62 total 40.01 last 332.43) - open/close 20000 (time 1744883870.05 total 46.44 last 358.43) - open/close 24555 (time 1744883880.05 total 56.44 last 455.44) - open/close 28416 (time 1744883890.05 total 66.44 last 385.96) - open/close 30000 (time 1744883894.16 total 70.55 last 385.75) - open/close 34216 (time 1744883904.16 total 80.55 last 421.55) - open/close 38447 (time 1744883914.16 total 90.55 last 423.04) - open/close 40000 (time 1744883917.44 total 93.83 last 474.05) - open/close 44523 (time 1744883927.44 total 103.83 last 452.15) - open/close 49303 (time 1744883937.44 total 113.83 last 477.93) - open/close 53798 (time 1744883947.44 total 123.83 last 449.50) - open/close 58630 (time 1744883957.44 total 133.83 last 483.17) - open/close 60000 (time 1744883960.00 total 136.39 last 534.93) total: 64768 open/close in 145.37 seconds: 445.53 ops/second - /unlink 3107 (time 1744883979.86 total 10.00 last 310.62) - /unlink 4642 (time 1744883989.86 total 20.01 last 153.44) - /unlink 7432 (time 1744883999.86 total 30.01 last 278.97) - /unlink 10000 (time 1744884006.96 total 37.10 last 361.85) - /unlink 13954 (time 1744884016.96 total 47.11 last 395.29) - /unlink 17085 (time 1744884026.97 total 57.11 last 312.89) - /unlink 20000 (time 1744884036.64 total 66.78 last 301.50) - /unlink 25569 (time 1744884046.64 total 76.78 last 556.75) - /unlink 30000 (time 1744884055.19 total 85.34 last 517.91) - /unlink 36710 (time 1744884065.20 total 95.34 last 670.90) - /unlink 40000 (time 1744884073.06 total 103.21 last 418.12) - /unlink 45015 (time 1744884083.07 total 113.21 last 501.40) - /unlink 48687 (time 1744884093.07 total 123.21 last 367.13) - /unlink 50000 (time 1744884097.31 total 127.46 last 309.41) - /unlink 53296 (time 1744884107.31 total 137.46 last 329.58) - /unlink 56374 (time 1744884117.31 total 147.46 last 307.75) - /unlink 59111 (time 1744884127.32 total 157.46 last 273.63) - /unlink 60000 (time 1744884130.72 total 160.86 last 261.37) - /unlink 62700 (time 1744884140.72 total 170.87 last 269.95) total: 64768 /unlink in 178.27 seconds: 363.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2544 (time 1744884174.15 total 10.00 last 254.30) - open/close 5070 (time 1744884184.15 total 20.01 last 252.55) - open/close 7183 (time 1744884194.16 total 30.01 last 211.23) - open/close 9481 (time 1744884204.17 total 40.02 last 229.50) - open/close 10000 (time 1744884206.38 total 42.23 last 234.96) - open/close 12336 (time 1744884216.38 total 52.23 last 233.58) - open/close 15209 (time 1744884226.38 total 62.24 last 287.21) - open/close 17867 (time 1744884236.39 total 72.24 last 265.66) - open/close 20000 (time 1744884245.29 total 81.15 last 239.50) - open/close 22850 (time 1744884255.30 total 91.15 last 284.89) - open/close 25762 (time 1744884265.30 total 101.15 last 291.15) - open/close 28159 (time 1744884275.30 total 111.15 last 239.70) - open/close 30000 (time 1744884281.07 total 116.93 last 318.79) - open/close 33046 (time 1744884291.08 total 126.93 last 304.55) - open/close 36638 (time 1744884301.08 total 136.93 last 359.14) - open/close 39371 (time 1744884311.08 total 146.93 last 273.27) - open/close 40000 (time 1744884313.80 total 149.66 last 230.78) - open/close 43130 (time 1744884323.81 total 159.66 last 312.93) - open/close 46035 (time 1744884333.81 total 169.66 last 290.41) - open/close 49577 (time 1744884343.81 total 179.67 last 354.10) - open/close 52853 (time 1744884353.81 total 189.67 last 327.53) - open/close 56489 (time 1744884363.82 total 199.67 last 363.41) - open/close 60000 (time 1744884372.77 total 208.62 last 392.31) - open/close 64643 (time 1744884382.77 total 218.62 last 464.22) total: 64768 open/close in 218.86 seconds: 295.94 ops/second - /unlink 7306 (time 1744884393.87 total 10.00 last 730.54) - /unlink 10000 (time 1744884397.82 total 13.95 last 681.77) - /unlink 17422 (time 1744884407.82 total 23.95 last 742.17) - /unlink 20000 (time 1744884411.39 total 27.52 last 723.17) - /unlink 26574 (time 1744884421.39 total 37.52 last 657.38) - /unlink 30000 (time 1744884426.58 total 42.71 last 660.23) - /unlink 37053 (time 1744884436.58 total 52.71 last 705.23) - /unlink 40000 (time 1744884440.98 total 57.11 last 670.01) - /unlink 46609 (time 1744884450.98 total 67.11 last 660.88) - /unlink 50000 (time 1744884456.23 total 72.36 last 645.03) - /unlink 56888 (time 1744884466.24 total 82.36 last 688.75) - /unlink 60000 (time 1744884470.98 total 87.11 last 655.76) total: 64768 /unlink in 94.95 seconds: 682.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3351 (time 1744884509.69 total 10.00 last 335.08) - open/close 6854 (time 1744884519.69 total 20.00 last 350.20) - open/close 10000 (time 1744884528.67 total 28.98 last 350.38) - open/close 14054 (time 1744884538.67 total 38.98 last 405.34) - open/close 17979 (time 1744884548.67 total 48.98 last 392.48) - open/close 20000 (time 1744884554.11 total 54.42 last 371.94) - open/close 23794 (time 1744884564.11 total 64.42 last 379.37) - open/close 27186 (time 1744884574.11 total 74.42 last 339.13) - open/close 30000 (time 1744884582.00 total 82.31 last 356.68) - open/close 33506 (time 1744884592.00 total 92.31 last 350.51) - open/close 36951 (time 1744884602.00 total 102.32 last 344.40) - open/close 40000 (time 1744884611.23 total 111.54 last 330.37) - open/close 43012 (time 1744884621.24 total 121.55 last 301.11) - open/close 46319 (time 1744884631.24 total 131.55 last 330.67) - open/close 49338 (time 1744884641.24 total 141.55 last 301.88) - open/close 52694 (time 1744884651.24 total 151.55 last 335.56) - open/close 55881 (time 1744884661.24 total 161.55 last 318.67) - open/close 58823 (time 1744884671.24 total 171.55 last 294.18) - open/close 60000 (time 1744884674.66 total 174.97 last 344.45) - open/close 63105 (time 1744884684.66 total 184.97 last 310.41) total: 64768 open/close in 189.92 seconds: 341.02 ops/second - /unlink 6326 (time 1744884700.53 total 10.00 last 632.57) - /unlink 10000 (time 1744884706.32 total 15.79 last 634.53) - /unlink 16886 (time 1744884716.32 total 25.79 last 688.54) - /unlink 20000 (time 1744884720.78 total 30.25 last 699.08) - /unlink 27420 (time 1744884730.78 total 40.25 last 741.96) - /unlink 30000 (time 1744884734.13 total 43.60 last 768.99) - /unlink 37182 (time 1744884744.13 total 53.60 last 718.11) - /unlink 40000 (time 1744884748.24 total 57.70 last 687.22) - /unlink 48201 (time 1744884758.24 total 67.70 last 820.08) - /unlink 50000 (time 1744884761.13 total 70.60 last 620.80) - /unlink 57388 (time 1744884771.13 total 80.60 last 738.75) - /unlink 60000 (time 1744884774.95 total 84.42 last 684.67) total: 64768 /unlink in 90.27 seconds: 717.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server unloading modules via unload_modules_local on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 PASS 106 (1617s) Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 oleg360-server: oleg360-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions oleg360-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg360-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1690 sec ========== 06:13:57 (1744884837) === conf-sanity: start cleanup 06:13:57 (1744884837) === === conf-sanity: finish cleanup 06:13:58 (1744884838) ===