-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:47:03 EDT 2024 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg442-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg442-server oleg442-server: oleg442-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg442-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg442-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg442-server: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg442-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 45: long unlink handling in ptlrpcd == 15:47:58 (1710100078) start mds service on oleg442-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg442-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock oleg442-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg442-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock oleg442-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server unloading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 PASS 45 (122s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg442-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg442-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg442-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 oleg442-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg442-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 69: replace an OST with the same index ========================================================== 15:50:02 (1710100202) start mds service on oleg442-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg442-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg442-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg442-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock oleg442-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 4234 (time 1710100237.65 total 10.00 last 423.31) - open/close 7863 (time 1710100247.65 total 20.00 last 362.83) total: 10000 open/close in 27.34 seconds: 365.78 ops/second - unlinked 0 (time 1710100255 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 4268 (time 1710100286.83 total 10.00 last 426.73) - open/close 7499 (time 1710100296.83 total 20.00 last 323.04) - open/close 9879 (time 1710100306.83 total 30.00 last 237.99) total: 10000 open/close in 30.50 seconds: 327.92 ops/second - unlinked 0 (time 1710100308 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2822 (time 1710100342.07 total 10.00 last 282.18) - open/close 4757 (time 1710100352.07 total 20.00 last 193.43) - open/close 6125 (time 1710100362.08 total 30.01 last 136.77) - open/close 7193 (time 1710100372.09 total 40.02 last 106.69) - open/close 8101 (time 1710100382.09 total 50.02 last 90.77) - open/close 9615 (time 1710100392.09 total 60.02 last 151.36) total: 10000 open/close in 61.96 seconds: 161.38 ops/second - unlinked 0 (time 1710100394 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 3384 (time 1710100443.29 total 10.00 last 338.23) - open/close 4449 (time 1710100453.30 total 20.01 last 106.43) - open/close 5611 (time 1710100463.30 total 30.02 last 116.16) - open/close 7146 (time 1710100473.31 total 40.02 last 153.46) - open/close 8431 (time 1710100483.31 total 50.02 last 128.43) - open/close 9645 (time 1710100493.32 total 60.03 last 121.37) total: 10000 open/close in 63.21 seconds: 158.20 ops/second - unlinked 0 (time 1710100497 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 3034 (time 1710100541.55 total 10.01 last 303.23) - open/close 4987 (time 1710100551.55 total 20.01 last 195.23) - open/close 6440 (time 1710100561.55 total 30.01 last 145.28) - open/close 8212 (time 1710100571.56 total 40.02 last 177.10) - open/close 9982 (time 1710100581.56 total 50.02 last 176.99) total: 10000 open/close in 50.09 seconds: 199.64 ops/second - unlinked 0 (time 1710100582 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 3391 (time 1710100630.32 total 10.00 last 339.07) - open/close 5891 (time 1710100640.32 total 20.00 last 249.96) - open/close 8375 (time 1710100650.33 total 30.00 last 248.35) total: 10000 open/close in 37.36 seconds: 267.69 ops/second - unlinked 0 (time 1710100658 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 2297 (time 1710100706.76 total 10.00 last 229.67) - open/close 4445 (time 1710100716.76 total 20.00 last 214.75) - open/close 6196 (time 1710100726.77 total 30.01 last 174.96) - open/close 7266 (time 1710100736.77 total 40.01 last 106.99) - open/close 8196 (time 1710100746.77 total 50.01 last 92.98) - open/close 9379 (time 1710100756.78 total 60.02 last 118.24) total: 10000 open/close in 65.93 seconds: 151.69 ops/second - unlinked 0 (time 1710100763 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1535 (time 1710100837.25 total 10.01 last 153.36) - open/close 3112 (time 1710100847.25 total 20.01 last 157.68) - open/close 4638 (time 1710100857.25 total 30.01 last 152.57) - open/close 5905 (time 1710100867.25 total 40.01 last 126.67) - open/close 7008 (time 1710100877.25 total 50.01 last 110.29) - open/close 7990 (time 1710100887.26 total 60.02 last 98.17) - open/close 8817 (time 1710100897.26 total 70.02 last 82.65) - open/close 9009 (time 1710100910.71 total 83.48 last 14.27) - open/close 9865 (time 1710100920.73 total 93.49 last 85.50) total: 10000 open/close in 95.11 seconds: 105.14 ops/second - unlinked 0 (time 1710100923 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1459 (time 1710100997.53 total 10.00 last 145.90) - open/close 2799 (time 1710101007.54 total 20.00 last 133.95) - open/close 3952 (time 1710101017.55 total 30.01 last 115.18) - open/close 4822 (time 1710101027.55 total 40.02 last 86.97) - open/close 5635 (time 1710101037.56 total 50.03 last 81.24) - open/close 6363 (time 1710101047.56 total 60.03 last 72.78) - open/close 7058 (time 1710101057.57 total 70.03 last 69.46) - open/close 7795 (time 1710101077.97 total 90.44 last 36.12) - open/close 8497 (time 1710101087.98 total 100.44 last 70.14) - open/close 9132 (time 1710101097.98 total 110.45 last 63.47) - open/close 9742 (time 1710101107.99 total 120.46 last 60.96) total: 10000 open/close in 124.64 seconds: 80.23 ops/second - unlinked 0 (time 1710101114 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 648 (time 1710101213.39 total 15.95 last 40.64) - open/close 1689 (time 1710101223.39 total 25.95 last 104.04) - open/close 2609 (time 1710101233.40 total 35.95 last 91.98) - open/close 3514 (time 1710101243.40 total 45.96 last 90.46) - open/close 4324 (time 1710101253.42 total 55.97 last 80.87) - open/close 5056 (time 1710101263.44 total 66.00 last 73.03) - open/close 5782 (time 1710101273.44 total 76.00 last 72.57) - open/close 6485 (time 1710101283.45 total 86.01 last 70.23) - open/close 7199 (time 1710101293.46 total 96.02 last 71.33) - open/close 7895 (time 1710101303.46 total 106.02 last 69.58) - open/close 8615 (time 1710101313.48 total 116.04 last 71.86) - open/close 9231 (time 1710101323.49 total 126.05 last 61.56) - open/close 9920 (time 1710101333.49 total 136.05 last 68.89) total: 10000 open/close in 137.20 seconds: 72.89 ops/second - unlinked 0 (time 1710101336 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.142@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.142@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg442-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg442-server: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg442-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock oleg442-server@tcp:/lustre /mnt/lustre On OST0, 18749 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server unloading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 PASS 69 (1275s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg442-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg442-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg442-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 oleg442-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg442-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:11:20 (1710101480) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg442-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg442-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 oleg442-server: oleg442-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg442-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1496 sec ========== 16:12:00 (1710101520) === conf-sanity: start cleanup 16:12:01 (1710101521) === === conf-sanity: finish cleanup 16:12:01 (1710101521) ===