-----============= acceptance-small: conf-sanity ============----- Wed Jan 15 18:48:02 EST 2025 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg126-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg126-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg126-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg126-server oleg126-server: oleg126-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: detected 4 online CPUs by sysfs oleg126-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg126-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server == conf-sanity test 45: long unlink handling in ptlrpcd == 18:49:37 (1736984977) start mds service on oleg126-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg126-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server LNET ready to unload unloading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing unload_modules_local oleg126-server: LNET ready to unload modules unloaded. pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 PASS 45 (148s) == conf-sanity test 69: replace an OST with the same index ========================================================== 18:52:05 (1736985125) start mds service on oleg126-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: detected 4 online CPUs by sysfs oleg126-server: Force libcfs to create 2 CPU partitions oleg126-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg126-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg126-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1805 (time 1736985170.18 total 10.00 last 180.49) - open/close 3659 (time 1736985180.18 total 20.00 last 185.40) - open/close 5440 (time 1736985190.18 total 30.00 last 178.06) - open/close 7413 (time 1736985200.19 total 40.01 last 197.24) - open/close 9417 (time 1736985210.19 total 50.01 last 200.30) total: 10000 open/close in 52.85 seconds: 189.22 ops/second - unlinked 0 (time 1736985214 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2637 (time 1736985250.55 total 10.00 last 263.70) - open/close 5008 (time 1736985260.55 total 20.00 last 237.05) - open/close 7146 (time 1736985270.55 total 30.00 last 213.76) - open/close 9062 (time 1736985280.55 total 40.01 last 191.52) total: 10000 open/close in 44.52 seconds: 224.64 ops/second - unlinked 0 (time 1736985286 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2812 (time 1736985323.00 total 10.00 last 281.12) - open/close 5167 (time 1736985333.00 total 20.01 last 235.44) - open/close 6954 (time 1736985343.00 total 30.01 last 178.61) - open/close 8644 (time 1736985353.01 total 40.01 last 168.98) total: 10000 open/close in 47.27 seconds: 211.54 ops/second - unlinked 0 (time 1736985361 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1972 (time 1736985402.56 total 10.00 last 197.12) - open/close 4032 (time 1736985412.57 total 20.01 last 205.89) - open/close 6189 (time 1736985422.57 total 30.01 last 215.65) - open/close 8298 (time 1736985432.57 total 40.01 last 210.90) total: 10000 open/close in 49.47 seconds: 202.15 ops/second - unlinked 0 (time 1736985443 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1976 (time 1736985484.07 total 10.00 last 197.55) - open/close 3954 (time 1736985494.07 total 20.00 last 197.78) - open/close 5957 (time 1736985504.08 total 30.01 last 200.27) - open/close 7882 (time 1736985514.08 total 40.01 last 192.47) - open/close 9634 (time 1736985524.08 total 50.01 last 175.12) total: 10000 open/close in 52.26 seconds: 191.35 ops/second - unlinked 0 (time 1736985527 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2296 (time 1736985561.80 total 10.00 last 229.58) - open/close 4086 (time 1736985571.81 total 20.01 last 178.88) - open/close 6086 (time 1736985581.82 total 30.02 last 199.84) - open/close 8088 (time 1736985591.82 total 40.02 last 200.15) total: 10000 open/close in 49.21 seconds: 203.21 ops/second - unlinked 0 (time 1736985602 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2717 (time 1736985636.75 total 10.00 last 271.65) - open/close 6562 (time 1736985646.75 total 20.00 last 384.46) total: 10000 open/close in 29.51 seconds: 338.82 ops/second - unlinked 0 (time 1736985657 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3101 (time 1736985688.36 total 10.00 last 310.08) - open/close 6395 (time 1736985698.37 total 20.00 last 329.32) - open/close 9282 (time 1736985708.37 total 30.00 last 288.66) total: 10000 open/close in 32.27 seconds: 309.89 ops/second - unlinked 0 (time 1736985711 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 4283 (time 1736985737.91 total 10.00 last 428.26) - open/close 8082 (time 1736985747.91 total 20.00 last 379.82) total: 10000 open/close in 25.80 seconds: 387.64 ops/second - unlinked 0 (time 1736985754 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 4443 (time 1736985781.04 total 10.00 last 444.26) - open/close 8528 (time 1736985791.05 total 20.00 last 408.47) total: 10000 open/close in 23.39 seconds: 427.46 ops/second - unlinked 0 (time 1736985795 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.126@tcp lustre:sys.timeout=20 lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg126-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre On OST0, 291 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server LNET ready to unload unloading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing unload_modules_local oleg126-server: LNET ready to unload modules unloaded. pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 PASS 69 (721s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 19:04:06 (1736985846) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg126-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg126-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 oleg126-server: oleg126-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: detected 4 online CPUs by sysfs oleg126-server: Force libcfs to create 2 CPU partitions oleg126-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg126-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 981 sec =========== 19:04:24 (1736985864)