-----============= acceptance-small: conf-sanity ============----- Fri Apr 5 18:46:20 EDT 2024 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg207-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg207-server oleg207-server: oleg207-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg207-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 45: long unlink handling in ptlrpcd == 18:47:15 (1712357235) start mds service on oleg207-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock oleg207-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg207-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock oleg207-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 PASS 45 (128s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg207-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg207-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg207-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 oleg207-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg207-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 69: replace an OST with the same index ========================================================== 18:49:26 (1712357366) start mds service on oleg207-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock oleg207-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2232 (time 1712357406.97 total 10.00 last 223.18) - open/close 4470 (time 1712357416.97 total 20.00 last 223.76) - open/close 6272 (time 1712357426.97 total 30.01 last 180.11) - open/close 7896 (time 1712357436.98 total 40.01 last 162.33) - open/close 9632 (time 1712357446.98 total 50.02 last 173.51) total: 10000 open/close in 52.40 seconds: 190.83 ops/second - unlinked 0 (time 1712357450 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 2481 (time 1712357499.36 total 10.00 last 248.08) - open/close 4992 (time 1712357509.36 total 20.01 last 250.98) - open/close 6587 (time 1712357519.37 total 30.01 last 159.47) - open/close 8134 (time 1712357529.37 total 40.01 last 154.60) - open/close 9559 (time 1712357539.38 total 50.02 last 142.43) total: 10000 open/close in 53.22 seconds: 187.91 ops/second - unlinked 0 (time 1712357543 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 3037 (time 1712357583.79 total 10.00 last 303.66) - open/close 5702 (time 1712357593.79 total 20.00 last 266.47) - open/close 8013 (time 1712357603.80 total 30.00 last 231.05) total: 10000 open/close in 39.82 seconds: 251.14 ops/second - unlinked 0 (time 1712357614 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 4218 (time 1712357651.15 total 10.00 last 421.75) - open/close 7072 (time 1712357661.16 total 20.00 last 285.35) - open/close 9487 (time 1712357671.16 total 30.00 last 241.46) total: 10000 open/close in 32.50 seconds: 307.70 ops/second - unlinked 0 (time 1712357674 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 3847 (time 1712357714.27 total 10.00 last 384.67) - open/close 6478 (time 1712357724.27 total 20.00 last 263.10) - open/close 8535 (time 1712357734.27 total 30.00 last 205.66) total: 10000 open/close in 38.18 seconds: 261.92 ops/second - unlinked 0 (time 1712357743 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 3923 (time 1712357780.50 total 10.00 last 392.18) - open/close 6927 (time 1712357790.50 total 20.00 last 300.37) - open/close 9141 (time 1712357800.51 total 30.01 last 221.31) total: 10000 open/close in 33.80 seconds: 295.84 ops/second - unlinked 0 (time 1712357805 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 3437 (time 1712357843.00 total 10.00 last 343.70) - open/close 6799 (time 1712357853.00 total 20.00 last 336.19) - open/close 9361 (time 1712357863.00 total 30.00 last 256.12) total: 10000 open/close in 32.73 seconds: 305.57 ops/second - unlinked 0 (time 1712357866 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 2925 (time 1712357916.60 total 10.00 last 292.43) - open/close 5135 (time 1712357926.60 total 20.01 last 220.88) - open/close 6468 (time 1712357936.60 total 30.01 last 133.30) - open/close 7896 (time 1712357946.61 total 40.01 last 142.70) - open/close 9278 (time 1712357956.62 total 50.02 last 138.13) total: 10000 open/close in 54.79 seconds: 182.52 ops/second - unlinked 0 (time 1712357962 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 3002 (time 1712358007.36 total 10.00 last 300.20) - open/close 4599 (time 1712358017.37 total 20.00 last 159.63) - open/close 6213 (time 1712358027.37 total 30.01 last 161.33) - open/close 7797 (time 1712358037.37 total 40.01 last 158.35) - open/close 9288 (time 1712358047.38 total 50.02 last 149.02) total: 10000 open/close in 54.95 seconds: 182.00 ops/second - unlinked 0 (time 1712358053 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 4299 (time 1712358090.79 total 10.00 last 429.83) - open/close 7113 (time 1712358100.80 total 20.00 last 281.33) - open/close 9166 (time 1712358110.80 total 30.01 last 205.27) total: 10000 open/close in 34.91 seconds: 286.45 ops/second - unlinked 0 (time 1712358116 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.107@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg207-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 16 sec mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock oleg207-server@tcp:/lustre /mnt/lustre On OST0, 8197 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 PASS 69 (828s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg207-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg207-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg207-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 oleg207-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg207-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 19:03:16 (1712358196) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg207-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg207-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 oleg207-server: oleg207-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1036 sec ========== 19:03:37 (1712358217) === conf-sanity: start cleanup 19:03:37 (1712358217) === === conf-sanity: finish cleanup 19:03:37 (1712358217) ===