-----============= acceptance-small: conf-sanity ============----- Fri Feb 14 06:45:57 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg208-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 32b 32c 32e 32h 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg208-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg208-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg208-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg208-server oleg208-server: oleg208-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg208-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg208-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg208-server: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg208-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server == conf-sanity test 45: long unlink handling in ptlrpcd == 06:47:28 (1739533648) start mds service on oleg208-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg208-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg208-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 PASS 45 (137s) == conf-sanity test 69: replace an OST with the same index ========================================================== 06:49:45 (1739533785) start mds service on oleg208-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg208-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg208-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2592 (time 1739533824.74 total 10.00 last 259.16) - open/close 4880 (time 1739533834.74 total 20.00 last 228.78) - open/close 7069 (time 1739533844.75 total 30.01 last 218.78) - open/close 8853 (time 1739533854.75 total 40.01 last 178.34) total: 10000 open/close in 46.56 seconds: 214.80 ops/second - unlinked 0 (time 1739533862 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1846 (time 1739533904.55 total 10.00 last 184.58) - open/close 3438 (time 1739533914.55 total 20.00 last 159.17) - open/close 4957 (time 1739533924.57 total 30.02 last 151.69) - open/close 6615 (time 1739533934.57 total 40.02 last 165.75) - open/close 8200 (time 1739533944.57 total 50.02 last 158.45) - open/close 9691 (time 1739533954.58 total 60.02 last 149.08) total: 10000 open/close in 61.82 seconds: 161.76 ops/second - unlinked 0 (time 1739533957 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1849 (time 1739534001.57 total 10.00 last 184.84) - open/close 3488 (time 1739534011.58 total 20.00 last 163.89) - open/close 5636 (time 1739534021.58 total 30.01 last 214.77) - open/close 8106 (time 1739534031.58 total 40.01 last 246.88) total: 10000 open/close in 46.79 seconds: 213.73 ops/second - unlinked 0 (time 1739534040 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1948 (time 1739534083.03 total 10.00 last 194.74) - open/close 4155 (time 1739534093.03 total 20.00 last 220.67) - open/close 6366 (time 1739534103.03 total 30.00 last 221.09) - open/close 8132 (time 1739534113.04 total 40.01 last 176.46) total: 10000 open/close in 49.25 seconds: 203.03 ops/second - unlinked 0 (time 1739534123 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 2015 (time 1739534166.36 total 10.00 last 201.46) - open/close 4167 (time 1739534176.37 total 20.01 last 215.05) - open/close 6121 (time 1739534186.37 total 30.01 last 195.35) - open/close 8305 (time 1739534196.38 total 40.01 last 218.36) total: 10000 open/close in 48.51 seconds: 206.16 ops/second - unlinked 0 (time 1739534206 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 2298 (time 1739534246.73 total 10.00 last 229.73) - open/close 4712 (time 1739534256.74 total 20.01 last 241.27) - open/close 6718 (time 1739534266.74 total 30.01 last 200.54) - open/close 8612 (time 1739534276.74 total 40.01 last 189.34) total: 10000 open/close in 45.98 seconds: 217.50 ops/second - unlinked 0 (time 1739534283 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1987 (time 1739534321.57 total 10.00 last 198.66) - open/close 5207 (time 1739534331.57 total 20.00 last 321.96) - open/close 8678 (time 1739534341.57 total 30.00 last 347.07) total: 10000 open/close in 35.37 seconds: 282.75 ops/second - unlinked 0 (time 1739534348 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2034 (time 1739534383.37 total 10.00 last 203.33) - open/close 4125 (time 1739534393.37 total 20.00 last 209.08) - open/close 8064 (time 1739534403.37 total 30.00 last 393.89) total: 10000 open/close in 35.05 seconds: 285.30 ops/second - unlinked 0 (time 1739534409 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3827 (time 1739534434.30 total 10.00 last 382.69) - open/close 7147 (time 1739534444.30 total 20.00 last 331.94) total: 10000 open/close in 28.44 seconds: 351.56 ops/second - unlinked 0 (time 1739534453 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3358 (time 1739534481.85 total 10.00 last 335.79) - open/close 7160 (time 1739534491.85 total 20.00 last 380.11) total: 10000 open/close in 28.28 seconds: 353.63 ops/second - unlinked 0 (time 1739534501 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.108@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.108@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg208-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg208-server: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg208-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre On OST0, 17693 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 PASS 69 (776s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 07:02:41 (1739534561) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg208-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg208-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 oleg208-server: oleg208-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1025 sec ========== 07:03:02 (1739534582) === conf-sanity: start cleanup 07:03:03 (1739534583) === === conf-sanity: finish cleanup 07:03:04 (1739534584) ===