-----============= acceptance-small: conf-sanity ============----- Tue Apr 1 07:51:38 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg436-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg436-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg436-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg436-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg436-server oleg436-server: oleg436-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing load_modules_local oleg436-server: Loading modules from /home/green/git/lustre-release/lustre oleg436-server: detected 4 online CPUs by sysfs oleg436-server: MODOPTS_LIBCFS= oleg436-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg436-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg436-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg436-server: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg436-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:52:32 (1743508352) Stopping clients: oleg436-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg436-client.virtnet /mnt/lustre2 (opts:-f) oleg436-server: oleg436-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing load_modules_local oleg436-server: Loading modules from /home/green/git/lustre-release/lustre oleg436-server: detected 4 online CPUs by sysfs oleg436-server: MODOPTS_LIBCFS= oleg436-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg436-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg436-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4924 (time 1743508386.58 total 10.00 last 492.40) - open/close 9786 (time 1743508396.59 total 20.00 last 486.17) - open/close 14532 (time 1743508406.59 total 30.00 last 474.53) - open/close 19114 (time 1743508416.59 total 40.00 last 458.18) - open/close 23792 (time 1743508426.59 total 50.00 last 467.76) - open/close 28462 (time 1743508436.59 total 60.00 last 466.96) - open/close 30000 (time 1743508439.86 total 63.28 last 469.68) - open/close 34115 (time 1743508449.87 total 73.28 last 411.44) - open/close 38975 (time 1743508459.87 total 83.28 last 485.93) - open/close 43905 (time 1743508469.87 total 93.28 last 492.92) - open/close 48756 (time 1743508479.87 total 103.29 last 485.01) - open/close 50000 (time 1743508482.58 total 106.00 last 458.35) - open/close 54654 (time 1743508492.59 total 116.00 last 465.34) - open/close 59633 (time 1743508502.59 total 126.00 last 497.88) total: 64768 open/close in 134.59 seconds: 481.21 ops/second - /unlink 9761 (time 1743508521.97 total 10.00 last 976.07) - /unlink 18955 (time 1743508531.97 total 20.00 last 919.35) - /unlink 27650 (time 1743508541.98 total 30.00 last 869.46) - /unlink 30000 (time 1743508544.57 total 32.59 last 906.11) - /unlink 38834 (time 1743508554.57 total 42.59 last 883.39) - /unlink 47206 (time 1743508564.57 total 52.60 last 837.16) - /unlink 50000 (time 1743508567.56 total 55.59 last 933.87) - /unlink 58412 (time 1743508577.56 total 65.59 last 841.16) - /unlink 60000 (time 1743508579.82 total 67.84 last 704.62) total: 64768 /unlink in 73.50 seconds: 881.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4600 (time 1743508619.73 total 10.00 last 459.99) - open/close 8980 (time 1743508629.73 total 20.00 last 437.96) - open/close 10000 (time 1743508631.97 total 22.24 last 455.09) - open/close 14953 (time 1743508641.97 total 32.24 last 495.29) - open/close 20000 (time 1743508651.89 total 42.16 last 508.93) - open/close 24975 (time 1743508661.89 total 52.16 last 497.43) - open/close 30000 (time 1743508670.66 total 60.94 last 572.69) - open/close 35475 (time 1743508680.66 total 70.94 last 547.45) - open/close 40000 (time 1743508687.82 total 78.09 last 632.39) - open/close 44724 (time 1743508697.82 total 88.09 last 472.38) - open/close 49581 (time 1743508707.82 total 98.09 last 485.68) - open/close 55453 (time 1743508717.82 total 108.09 last 587.16) - open/close 60000 (time 1743508726.52 total 116.79 last 522.94) total: 64768 open/close in 125.31 seconds: 516.86 ops/second - /unlink 10000 (time 1743508745.50 total 9.68 last 1032.62) - /unlink 19843 (time 1743508755.50 total 19.68 last 984.29) - /unlink 30000 (time 1743508765.33 total 29.51 last 1033.75) - /unlink 39696 (time 1743508775.33 total 39.51 last 969.47) - /unlink 49329 (time 1743508785.33 total 49.51 last 963.27) - /unlink 59306 (time 1743508795.33 total 59.51 last 997.63) total: 64768 /unlink in 65.26 seconds: 992.48 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5764 (time 1743508844.97 total 10.00 last 576.37) - open/close 10000 (time 1743508852.31 total 17.33 last 577.73) - open/close 15241 (time 1743508862.31 total 27.33 last 524.03) - open/close 20000 (time 1743508870.93 total 35.96 last 551.98) - open/close 26003 (time 1743508880.93 total 45.96 last 600.29) - open/close 30000 (time 1743508887.83 total 52.86 last 578.82) - open/close 35880 (time 1743508897.84 total 62.86 last 587.98) - open/close 40000 (time 1743508907.26 total 72.28 last 437.27) - open/close 44693 (time 1743508917.26 total 82.29 last 469.24) - open/close 48695 (time 1743508927.26 total 92.29 last 400.10) - open/close 50000 (time 1743508930.22 total 95.24 last 441.68) - open/close 54013 (time 1743508940.22 total 105.24 last 401.26) - open/close 57986 (time 1743508950.22 total 115.24 last 397.27) - open/close 60000 (time 1743508954.72 total 119.75 last 446.88) - open/close 64144 (time 1743508964.73 total 129.75 last 414.35) total: 64768 open/close in 131.16 seconds: 493.82 ops/second - /unlink 8173 (time 1743508976.96 total 10.00 last 817.15) - /unlink 10000 (time 1743508979.71 total 12.76 last 663.27) - /unlink 16058 (time 1743508989.71 total 22.76 last 605.72) - /unlink 20000 (time 1743508994.66 total 27.71 last 796.28) - /unlink 27126 (time 1743509004.67 total 37.71 last 712.44) - /unlink 30000 (time 1743509008.23 total 41.28 last 806.24) - /unlink 38497 (time 1743509018.23 total 51.28 last 849.65) - /unlink 40000 (time 1743509020.44 total 53.48 last 680.40) - /unlink 47117 (time 1743509030.44 total 63.49 last 711.65) - /unlink 50000 (time 1743509034.28 total 67.32 last 751.78) - /unlink 56407 (time 1743509044.28 total 77.32 last 640.66) - /unlink 60000 (time 1743509050.02 total 83.06 last 625.67) total: 64768 /unlink in 88.72 seconds: 730.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3556 (time 1743509086.12 total 10.00 last 355.58) - open/close 7682 (time 1743509096.12 total 20.00 last 412.50) - open/close 10000 (time 1743509101.43 total 25.31 last 436.57) - open/close 14787 (time 1743509111.43 total 35.32 last 478.57) - open/close 18863 (time 1743509121.43 total 45.32 last 407.59) - open/close 20000 (time 1743509124.48 total 48.37 last 372.75) - open/close 23595 (time 1743509134.49 total 58.37 last 359.45) - open/close 27607 (time 1743509144.49 total 68.37 last 401.19) - open/close 30000 (time 1743509150.47 total 74.35 last 400.01) - open/close 34073 (time 1743509160.47 total 84.35 last 407.25) - open/close 38045 (time 1743509170.47 total 94.35 last 397.16) - open/close 40000 (time 1743509175.61 total 99.49 last 380.38) - open/close 44468 (time 1743509185.61 total 109.49 last 446.75) - open/close 49059 (time 1743509195.61 total 119.49 last 459.06) - open/close 50000 (time 1743509197.81 total 121.70 last 427.34) - open/close 53879 (time 1743509207.82 total 131.70 last 387.85) - open/close 57842 (time 1743509217.82 total 141.70 last 396.21) - open/close 60000 (time 1743509224.53 total 148.41 last 321.61) - open/close 64099 (time 1743509234.53 total 158.41 last 409.84) total: 64768 open/close in 160.05 seconds: 404.66 ops/second - /unlink 6844 (time 1743509247.07 total 10.00 last 684.36) - /unlink 10000 (time 1743509251.03 total 13.96 last 797.05) - /unlink 17177 (time 1743509261.03 total 23.96 last 717.58) - /unlink 20000 (time 1743509265.01 total 27.94 last 708.95) - /unlink 27976 (time 1743509275.01 total 37.94 last 797.59) - /unlink 30000 (time 1743509277.77 total 40.70 last 733.31) - /unlink 37983 (time 1743509287.77 total 50.70 last 798.28) - /unlink 40000 (time 1743509290.26 total 53.19 last 810.90) - /unlink 48269 (time 1743509300.26 total 63.19 last 826.88) - /unlink 50000 (time 1743509302.26 total 65.19 last 865.01) - /unlink 57386 (time 1743509312.26 total 75.19 last 738.53) - /unlink 60000 (time 1743509315.67 total 78.60 last 768.53) total: 64768 /unlink in 84.21 seconds: 769.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4148 (time 1743509351.75 total 10.00 last 414.77) - open/close 8546 (time 1743509361.75 total 20.00 last 439.80) - open/close 10000 (time 1743509364.87 total 23.12 last 466.46) - open/close 14825 (time 1743509374.87 total 33.12 last 482.43) - open/close 19011 (time 1743509384.87 total 43.12 last 418.53) - open/close 20000 (time 1743509387.01 total 45.26 last 462.27) - open/close 24470 (time 1743509397.01 total 55.26 last 446.98) - open/close 29038 (time 1743509407.02 total 65.26 last 456.75) - open/close 30000 (time 1743509409.03 total 67.28 last 477.61) - open/close 35022 (time 1743509419.03 total 77.28 last 502.18) - open/close 39568 (time 1743509429.03 total 87.28 last 454.51) - open/close 44262 (time 1743509439.03 total 97.28 last 469.38) - open/close 48850 (time 1743509449.03 total 107.28 last 458.78) - open/close 50000 (time 1743509451.62 total 109.87 last 444.25) - open/close 54652 (time 1743509461.62 total 119.87 last 465.11) - open/close 59551 (time 1743509471.63 total 129.87 last 489.83) - open/close 63941 (time 1743509481.63 total 139.87 last 438.95) total: 64768 open/close in 141.68 seconds: 457.13 ops/second - /unlink 8303 (time 1743509494.35 total 10.00 last 830.27) - /unlink 10000 (time 1743509496.57 total 12.22 last 766.12) - /unlink 18061 (time 1743509506.57 total 22.22 last 806.06) - /unlink 20000 (time 1743509508.87 total 24.52 last 841.19) - /unlink 28048 (time 1743509518.87 total 34.52 last 804.74) - /unlink 30000 (time 1743509521.45 total 37.10 last 756.94) - /unlink 38573 (time 1743509531.45 total 47.10 last 857.17) - /unlink 46677 (time 1743509541.46 total 57.10 last 810.31) - /unlink 50000 (time 1743509545.64 total 61.28 last 794.72) - /unlink 57967 (time 1743509555.64 total 71.29 last 796.62) - /unlink 60000 (time 1743509558.04 total 73.69 last 846.54) total: 64768 /unlink in 79.78 seconds: 811.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg436-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server unloading modules via unload_modules_local on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 PASS 106 (1249s) Stopping clients: oleg436-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg436-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 oleg436-server: oleg436-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing load_modules_local oleg436-server: Loading modules from /home/green/git/lustre-release/lustre oleg436-server: detected 4 online CPUs by sysfs oleg436-server: MODOPTS_LIBCFS= oleg436-server: Force libcfs to create 2 CPU partitions oleg436-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg436-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1321 sec ========== 08:13:39 (1743509619) === conf-sanity: start cleanup 08:13:39 (1743509619) === === conf-sanity: finish cleanup 08:13:40 (1743509620) ===