-----============= acceptance-small: conf-sanity ============----- Thu Apr 17 05:46:08 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg135-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg135-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg135-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg135-server oleg135-server: oleg135-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg135-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg135-server: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg135-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server == conf-sanity test 45: long unlink handling in ptlrpcd == 05:47:12 (1744883232) start mds service on oleg135-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg135-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 PASS 45 (133s) == conf-sanity test 69: replace an OST with the same index ========================================================== 05:49:25 (1744883365) start mds service on oleg135-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2601 (time 1744883406.82 total 10.00 last 260.06) - open/close 5380 (time 1744883416.82 total 20.00 last 277.87) - open/close 8122 (time 1744883426.83 total 30.01 last 274.06) total: 10000 open/close in 38.21 seconds: 261.73 ops/second - unlinked 0 (time 1744883436 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2174 (time 1744883474.19 total 10.00 last 217.37) - open/close 4469 (time 1744883484.19 total 20.00 last 229.44) - open/close 7525 (time 1744883494.19 total 30.01 last 305.54) total: 10000 open/close in 37.67 seconds: 265.47 ops/second - unlinked 0 (time 1744883502 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3486 (time 1744883529.12 total 10.00 last 348.55) - open/close 6583 (time 1744883539.12 total 20.00 last 309.66) - open/close 9963 (time 1744883549.12 total 30.01 last 337.90) total: 10000 open/close in 30.11 seconds: 332.13 ops/second - unlinked 0 (time 1744883550 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 3413 (time 1744883583.37 total 10.00 last 341.20) - open/close 6011 (time 1744883593.37 total 20.00 last 259.75) - open/close 8608 (time 1744883603.37 total 30.01 last 259.68) total: 10000 open/close in 35.22 seconds: 283.93 ops/second - unlinked 0 (time 1744883609 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2992 (time 1744883640.65 total 10.00 last 299.20) - open/close 6126 (time 1744883650.65 total 20.00 last 313.38) - open/close 9134 (time 1744883660.65 total 30.00 last 300.73) total: 10000 open/close in 32.52 seconds: 307.47 ops/second - unlinked 0 (time 1744883664 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3499 (time 1744883694.00 total 10.00 last 349.84) - open/close 7444 (time 1744883704.00 total 20.00 last 394.47) total: 10000 open/close in 26.39 seconds: 378.96 ops/second - unlinked 0 (time 1744883711 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3345 (time 1744883742.16 total 10.00 last 334.44) - open/close 6350 (time 1744883752.17 total 20.01 last 300.39) - open/close 9317 (time 1744883762.17 total 30.01 last 296.53) total: 10000 open/close in 33.11 seconds: 302.02 ops/second - unlinked 0 (time 1744883766 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 3400 (time 1744883796.54 total 10.00 last 339.94) - open/close 6195 (time 1744883806.55 total 20.00 last 279.43) - open/close 9008 (time 1744883816.55 total 30.01 last 281.28) total: 10000 open/close in 32.99 seconds: 303.08 ops/second - unlinked 0 (time 1744883820 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3436 (time 1744883849.64 total 10.00 last 343.57) - open/close 7350 (time 1744883859.64 total 20.00 last 391.26) total: 10000 open/close in 27.71 seconds: 360.85 ops/second - unlinked 0 (time 1744883868 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3059 (time 1744883895.77 total 10.00 last 305.89) - open/close 6411 (time 1744883905.77 total 20.00 last 335.19) - open/close 9729 (time 1744883915.78 total 30.00 last 331.72) total: 10000 open/close in 30.94 seconds: 323.20 ops/second - unlinked 0 (time 1744883917 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.135@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.135@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg135-server: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg135-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 PASS 69 (607s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 05:59:33 (1744883973) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg135-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 oleg135-server: oleg135-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 843 sec =========== 06:00:11 (1744884011) === conf-sanity: start cleanup 06:00:12 (1744884012) === === conf-sanity: finish cleanup 06:00:13 (1744884013) ===