-----============= acceptance-small: conf-sanity ============----- Tue Mar 25 07:42:47 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg409-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg409-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg409-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg409-server oleg409-server: oleg409-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg409-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg409-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg409-server: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg409-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server == conf-sanity test 45: long unlink handling in ptlrpcd == 07:43:43 (1742903023) start mds service on oleg409-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg409-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg409-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server unloading modules via unload_modules_local on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 PASS 45 (125s) == conf-sanity test 69: replace an OST with the same index ========================================================== 07:45:48 (1742903148) start mds service on oleg409-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: Force libcfs to create 2 CPU partitions oleg409-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg409-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg409-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1512 (time 1742903189.47 total 10.00 last 151.19) - open/close 3001 (time 1742903199.48 total 20.00 last 148.87) - open/close 4632 (time 1742903209.48 total 30.00 last 163.08) - open/close 6114 (time 1742903219.49 total 40.01 last 148.07) - open/close 7723 (time 1742903229.49 total 50.02 last 160.82) - open/close 9403 (time 1742903239.49 total 60.02 last 167.98) total: 10000 open/close in 62.97 seconds: 158.81 ops/second - unlinked 0 (time 1742903243 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1990 (time 1742903288.63 total 10.00 last 199.00) - open/close 3986 (time 1742903298.63 total 20.00 last 199.59) - open/close 5549 (time 1742903308.64 total 30.01 last 156.22) - open/close 7107 (time 1742903318.64 total 40.01 last 155.72) - open/close 8993 (time 1742903328.65 total 50.01 last 188.54) total: 10000 open/close in 54.37 seconds: 183.92 ops/second - unlinked 0 (time 1742903334 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 2110 (time 1742903377.09 total 10.01 last 210.89) - open/close 4275 (time 1742903387.09 total 20.01 last 216.43) - open/close 6716 (time 1742903397.09 total 30.01 last 244.04) - open/close 9263 (time 1742903407.10 total 40.02 last 254.58) total: 10000 open/close in 43.59 seconds: 229.39 ops/second - unlinked 0 (time 1742903411 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 3647 (time 1742903444.87 total 10.00 last 364.68) - open/close 7809 (time 1742903454.88 total 20.00 last 416.16) total: 10000 open/close in 24.32 seconds: 411.16 ops/second - unlinked 0 (time 1742903460 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3811 (time 1742903485.68 total 10.00 last 381.08) - open/close 7444 (time 1742903495.69 total 20.00 last 363.27) total: 10000 open/close in 28.02 seconds: 356.92 ops/second - unlinked 0 (time 1742903504 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 4096 (time 1742903533.20 total 10.00 last 409.50) - open/close 7680 (time 1742903543.20 total 20.01 last 358.26) total: 10000 open/close in 26.72 seconds: 374.19 ops/second - unlinked 0 (time 1742903550 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 2922 (time 1742903575.94 total 10.00 last 292.15) - open/close 6828 (time 1742903585.94 total 20.00 last 390.54) total: 10000 open/close in 27.82 seconds: 359.47 ops/second - unlinked 0 (time 1742903594 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3658 (time 1742903621.27 total 10.00 last 365.76) - open/close 8104 (time 1742903631.27 total 20.00 last 444.57) total: 10000 open/close in 25.34 seconds: 394.69 ops/second - unlinked 0 (time 1742903637 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3807 (time 1742903666.64 total 10.00 last 380.64) - open/close 8422 (time 1742903676.64 total 20.00 last 461.48) total: 10000 open/close in 23.65 seconds: 422.79 ops/second - unlinked 0 (time 1742903681 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 4543 (time 1742903703.60 total 10.00 last 454.27) - open/close 9698 (time 1742903713.60 total 20.00 last 515.47) total: 10000 open/close in 20.63 seconds: 484.65 ops/second - unlinked 0 (time 1742903715 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.109@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.109@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg409-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg409-server: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg409-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 16 sec mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre On OST0, 6789 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server unloading modules via unload_modules_local on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 PASS 69 (630s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 07:56:19 (1742903779) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (0s) Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg409-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 oleg409-server: oleg409-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: Force libcfs to create 2 CPU partitions oleg409-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg409-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg409-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 830 sec =========== 07:56:38 (1742903798) === conf-sanity: start cleanup 07:56:39 (1742903799) === === conf-sanity: finish cleanup 07:56:39 (1742903799) ===