-----============= acceptance-small: conf-sanity ============----- Tue Jan 7 10:22:23 EST 2025 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg438-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg438-server oleg438-server: oleg438-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg438-server: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg438-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server == conf-sanity test 45: long unlink handling in ptlrpcd == 10:23:50 (1736263430) start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server LNET ready to unload unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local oleg438-server: LNET ready to unload modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 PASS 45 (149s) == conf-sanity test 69: replace an OST with the same index ========================================================== 10:26:20 (1736263580) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1460 (time 1736263625.90 total 10.00 last 145.97) - open/close 3088 (time 1736263635.90 total 20.01 last 162.71) - open/close 4742 (time 1736263645.90 total 30.01 last 165.34) - open/close 6231 (time 1736263655.90 total 40.01 last 148.89) - open/close 8208 (time 1736263665.91 total 50.01 last 197.65) total: 10000 open/close in 58.13 seconds: 172.02 ops/second - unlinked 0 (time 1736263675 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2340 (time 1736263709.94 total 10.00 last 233.97) - open/close 4647 (time 1736263719.94 total 20.01 last 230.60) - open/close 6555 (time 1736263729.94 total 30.01 last 190.80) - open/close 8421 (time 1736263739.94 total 40.01 last 186.58) total: 10000 open/close in 48.15 seconds: 207.68 ops/second - unlinked 0 (time 1736263749 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1819 (time 1736263790.25 total 10.00 last 181.82) - open/close 3878 (time 1736263800.25 total 20.01 last 205.87) - open/close 5702 (time 1736263810.26 total 30.01 last 182.37) - open/close 7474 (time 1736263820.26 total 40.01 last 177.14) - open/close 9690 (time 1736263830.26 total 50.01 last 221.54) total: 10000 open/close in 51.64 seconds: 193.65 ops/second - unlinked 0 (time 1736263833 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2423 (time 1736263867.79 total 10.00 last 242.29) - open/close 4566 (time 1736263877.80 total 20.00 last 214.22) - open/close 6850 (time 1736263887.80 total 30.01 last 228.33) - open/close 9124 (time 1736263897.80 total 40.01 last 227.35) total: 10000 open/close in 42.66 seconds: 234.43 ops/second - unlinked 0 (time 1736263901 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2958 (time 1736263931.73 total 10.00 last 295.74) - open/close 5673 (time 1736263941.73 total 20.00 last 271.48) - open/close 7690 (time 1736263951.73 total 30.00 last 201.68) - open/close 9901 (time 1736263961.73 total 40.01 last 221.04) total: 10000 open/close in 40.42 seconds: 247.43 ops/second - unlinked 0 (time 1736263963 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2532 (time 1736263998.97 total 10.00 last 253.15) - open/close 5641 (time 1736264008.97 total 20.00 last 310.84) - open/close 7886 (time 1736264018.98 total 30.01 last 224.44) total: 10000 open/close in 39.11 seconds: 255.68 ops/second - unlinked 0 (time 1736264029 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2273 (time 1736264063.01 total 10.00 last 227.26) - open/close 4386 (time 1736264073.02 total 20.01 last 211.20) - open/close 6751 (time 1736264083.03 total 30.01 last 236.37) - open/close 8878 (time 1736264093.03 total 40.01 last 212.69) total: 10000 open/close in 45.09 seconds: 221.79 ops/second - unlinked 0 (time 1736264099 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2209 (time 1736264136.38 total 10.01 last 220.77) - open/close 4249 (time 1736264146.39 total 20.01 last 203.93) - open/close 6654 (time 1736264156.39 total 30.01 last 240.49) - open/close 8844 (time 1736264166.39 total 40.01 last 218.99) total: 10000 open/close in 45.33 seconds: 220.60 ops/second - unlinked 0 (time 1736264173 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 3233 (time 1736264206.71 total 10.00 last 323.21) - open/close 6157 (time 1736264216.71 total 20.00 last 292.33) - open/close 9325 (time 1736264226.72 total 30.01 last 316.75) total: 10000 open/close in 31.97 seconds: 312.83 ops/second - unlinked 0 (time 1736264229 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3693 (time 1736264256.59 total 10.00 last 369.23) - open/close 7364 (time 1736264266.59 total 20.00 last 367.03) total: 10000 open/close in 27.88 seconds: 358.72 ops/second - unlinked 0 (time 1736264275 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.138@tcp sys.timeout=20 mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.138@tcp lustre:sys.timeout=20 lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg438-server: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg438-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre On OST0, 291 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server LNET ready to unload unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local oleg438-server: LNET ready to unload modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 PASS 69 (755s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 10:38:55 (1736264335) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: oleg438-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1009 sec ========== 10:39:13 (1736264353)