-----============= acceptance-small: conf-sanity ============----- Thu May 1 17:23:34 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg246-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg246-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg246-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg246-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg246-server oleg246-server: oleg246-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg246-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg246-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg246-server: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg246-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server == conf-sanity test 45: long unlink handling in ptlrpcd == 17:24:58 (1746134698) start mds service on oleg246-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg246-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg246-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 PASS 45 (144s) == conf-sanity test 69: replace an OST with the same index ========================================================== 17:27:22 (1746134842) start mds service on oleg246-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg246-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 892 (time 1746134889.06 total 10.00 last 89.16) - open/close 1866 (time 1746134899.07 total 20.00 last 97.40) - open/close 2924 (time 1746134909.07 total 30.01 last 105.71) - open/close 3861 (time 1746134919.08 total 40.02 last 93.68) - open/close 4616 (time 1746134929.12 total 50.06 last 75.20) - open/close 5663 (time 1746134939.13 total 60.07 last 104.57) - open/close 6650 (time 1746134949.14 total 70.08 last 98.62) - open/close 7477 (time 1746134959.14 total 80.08 last 82.69) - open/close 8390 (time 1746134969.14 total 90.08 last 91.29) - open/close 9333 (time 1746134979.14 total 100.08 last 94.30) total: 10000 open/close in 106.40 seconds: 93.99 ops/second - unlinked 0 (time 1746134987 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1006 (time 1746135061.75 total 10.01 last 100.55) - open/close 1933 (time 1746135071.75 total 20.01 last 92.69) - open/close 2743 (time 1746135081.76 total 30.01 last 80.99) - open/close 3620 (time 1746135091.76 total 40.01 last 87.69) - open/close 4668 (time 1746135101.76 total 50.01 last 104.77) - open/close 5619 (time 1746135111.78 total 60.04 last 94.87) - open/close 6399 (time 1746135121.79 total 70.04 last 77.97) - open/close 7209 (time 1746135131.80 total 80.05 last 80.91) - open/close 8010 (time 1746135141.80 total 90.05 last 80.08) - open/close 9074 (time 1746135151.80 total 100.05 last 106.39) total: 10000 open/close in 109.90 seconds: 90.99 ops/second - unlinked 0 (time 1746135163 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1015 (time 1746135238.78 total 10.01 last 101.38) - open/close 1895 (time 1746135248.79 total 20.02 last 87.92) - open/close 2729 (time 1746135258.80 total 30.03 last 83.36) - open/close 3611 (time 1746135268.81 total 40.03 last 88.14) - open/close 4637 (time 1746135278.81 total 50.04 last 102.54) - open/close 5406 (time 1746135288.81 total 60.04 last 76.88) - open/close 6194 (time 1746135298.82 total 70.04 last 78.79) - open/close 6947 (time 1746135308.82 total 80.04 last 75.29) - open/close 7871 (time 1746135318.82 total 90.05 last 92.33) - open/close 8715 (time 1746135328.83 total 100.06 last 84.36) - open/close 9480 (time 1746135338.84 total 110.07 last 76.42) total: 10000 open/close in 118.02 seconds: 84.73 ops/second - unlinked 0 (time 1746135349 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1025 (time 1746135435.46 total 10.00 last 102.46) - open/close 1941 (time 1746135445.47 total 20.02 last 91.47) - open/close 2796 (time 1746135455.48 total 30.02 last 85.49) - open/close 3610 (time 1746135465.48 total 40.02 last 81.39) - open/close 4600 (time 1746135475.48 total 50.02 last 98.99) - open/close 5835 (time 1746135485.48 total 60.03 last 123.46) - open/close 6658 (time 1746135495.50 total 70.05 last 82.12) - open/close 7520 (time 1746135505.51 total 80.06 last 86.12) - open/close 8461 (time 1746135515.51 total 90.06 last 94.10) - open/close 9692 (time 1746135525.51 total 100.06 last 123.08) total: 10000 open/close in 103.31 seconds: 96.80 ops/second - unlinked 0 (time 1746135531 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1249 (time 1746135598.84 total 10.00 last 124.85) - open/close 2275 (time 1746135608.84 total 20.01 last 102.58) - open/close 3337 (time 1746135618.85 total 30.01 last 106.14) - open/close 4447 (time 1746135628.85 total 40.01 last 110.98) - open/close 5559 (time 1746135638.86 total 50.02 last 111.07) - open/close 6600 (time 1746135648.86 total 60.03 last 104.09) - open/close 7509 (time 1746135658.86 total 70.03 last 90.87) - open/close 8587 (time 1746135668.87 total 80.03 last 107.78) - open/close 9630 (time 1746135678.87 total 90.04 last 104.21) total: 10000 open/close in 94.19 seconds: 106.16 ops/second - unlinked 0 (time 1746135684 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 996 (time 1746135762.28 total 10.01 last 99.55) - open/close 2104 (time 1746135772.28 total 20.01 last 110.79) - open/close 2951 (time 1746135782.28 total 30.01 last 84.69) - open/close 3796 (time 1746135792.29 total 40.01 last 84.47) - open/close 4829 (time 1746135802.29 total 50.01 last 103.29) - open/close 5954 (time 1746135812.29 total 60.02 last 112.45) - open/close 6955 (time 1746135822.31 total 70.03 last 99.97) - open/close 7908 (time 1746135832.31 total 80.04 last 95.23) - open/close 9061 (time 1746135842.32 total 90.04 last 115.26) total: 10000 open/close in 98.61 seconds: 101.41 ops/second - unlinked 0 (time 1746135852 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1414 (time 1746135912.22 total 10.00 last 141.34) - open/close 2660 (time 1746135922.23 total 20.01 last 124.55) - open/close 3898 (time 1746135932.23 total 30.01 last 123.79) - open/close 5044 (time 1746135942.23 total 40.01 last 114.57) - open/close 6361 (time 1746135952.24 total 50.02 last 131.59) - open/close 7421 (time 1746135962.24 total 60.02 last 105.99) - open/close 8719 (time 1746135972.24 total 70.02 last 129.77) total: 10000 open/close in 78.83 seconds: 126.86 ops/second - unlinked 0 (time 1746135982 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1131 (time 1746136035.05 total 10.00 last 113.06) - open/close 2533 (time 1746136045.06 total 20.01 last 140.09) - open/close 3954 (time 1746136055.06 total 30.01 last 142.08) - open/close 5215 (time 1746136065.06 total 40.02 last 126.03) - open/close 6600 (time 1746136075.07 total 50.02 last 138.42) - open/close 7798 (time 1746136085.07 total 60.03 last 119.77) - open/close 9482 (time 1746136095.07 total 70.03 last 168.37) total: 10000 open/close in 73.11 seconds: 136.78 ops/second - unlinked 0 (time 1746136099 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1615 (time 1746136151.70 total 10.00 last 161.47) - open/close 3570 (time 1746136161.71 total 20.01 last 195.40) - open/close 5335 (time 1746136171.71 total 30.01 last 176.37) - open/close 6782 (time 1746136181.72 total 40.02 last 144.55) - open/close 8068 (time 1746136191.73 total 50.03 last 128.56) - open/close 9575 (time 1746136201.73 total 60.03 last 150.64) total: 10000 open/close in 62.47 seconds: 160.08 ops/second - unlinked 0 (time 1746136205 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1480 (time 1746136265.62 total 10.00 last 147.93) - open/close 2890 (time 1746136275.62 total 20.01 last 141.00) - open/close 4520 (time 1746136285.62 total 30.01 last 162.99) - open/close 6138 (time 1746136295.63 total 40.01 last 161.76) - open/close 7712 (time 1746136305.63 total 50.01 last 157.32) - open/close 9092 (time 1746136315.63 total 60.02 last 137.97) total: 10000 open/close in 66.92 seconds: 149.43 ops/second - unlinked 0 (time 1746136324 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.146@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.146@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg246-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg246-server: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg246-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre On OST0, 16141 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 PASS 69 (1569s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 17:53:31 (1746136411) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg246-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg246-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 oleg246-server: oleg246-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1832 sec ========== 17:54:07 (1746136447) === conf-sanity: start cleanup 17:54:08 (1746136448) === === conf-sanity: finish cleanup 17:54:10 (1746136450) ===