-----============= acceptance-small: conf-sanity ============----- Fri Jun 14 16:51:41 EDT 2024 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg138-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg138-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg138-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg138-server oleg138-server: oleg138-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: detected 4 online CPUs by sysfs oleg138-server: Force libcfs to create 2 CPU partitions oleg138-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg138-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg138-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg138-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server == conf-sanity test 45: long unlink handling in ptlrpcd == 16:52:57 (1718398377) start mds service on oleg138-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg138-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock oleg138-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg138-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg138-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock oleg138-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server LNET ready to unload unloading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing unload_modules_local oleg138-server: LNET ready to unload modules unloaded. pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 PASS 45 (130s) == conf-sanity test 69: replace an OST with the same index ========================================================== 16:55:07 (1718398507) start mds service on oleg138-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: detected 4 online CPUs by sysfs oleg138-server: Force libcfs to create 2 CPU partitions oleg138-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg138-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg138-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock oleg138-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2851 (time 1718398539.07 total 10.00 last 284.97) - open/close 5063 (time 1718398549.07 total 20.01 last 221.13) - open/close 7766 (time 1718398559.08 total 30.01 last 270.19) total: 10000 open/close in 37.16 seconds: 269.07 ops/second - unlinked 0 (time 1718398567 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4283 (time 1718398591.94 total 10.00 last 428.30) - open/close 7827 (time 1718398601.94 total 20.00 last 354.33) total: 10000 open/close in 26.22 seconds: 381.32 ops/second - unlinked 0 (time 1718398609 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 3649 (time 1718398632.43 total 10.00 last 364.86) - open/close 6988 (time 1718398642.43 total 20.00 last 333.81) total: 10000 open/close in 29.48 seconds: 339.27 ops/second - unlinked 0 (time 1718398652 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 2872 (time 1718398680.31 total 10.00 last 287.06) - open/close 6085 (time 1718398690.32 total 20.01 last 321.24) - open/close 9562 (time 1718398700.32 total 30.01 last 347.65) total: 10000 open/close in 31.20 seconds: 320.54 ops/second - unlinked 0 (time 1718398702 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3650 (time 1718398728.53 total 10.00 last 364.93) - open/close 7369 (time 1718398738.54 total 20.00 last 371.83) total: 10000 open/close in 26.83 seconds: 372.78 ops/second - unlinked 0 (time 1718398746 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3892 (time 1718398772.90 total 10.00 last 389.13) - open/close 7533 (time 1718398782.90 total 20.00 last 364.09) total: 10000 open/close in 27.35 seconds: 365.69 ops/second - unlinked 0 (time 1718398791 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3473 (time 1718398818.60 total 10.00 last 347.30) - open/close 6342 (time 1718398828.60 total 20.00 last 286.84) - open/close 9730 (time 1718398838.60 total 30.01 last 338.68) total: 10000 open/close in 30.76 seconds: 325.07 ops/second - unlinked 0 (time 1718398840 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3386 (time 1718398866.47 total 10.00 last 338.58) - open/close 6989 (time 1718398876.48 total 20.00 last 360.26) total: 10000 open/close in 28.09 seconds: 355.98 ops/second - unlinked 0 (time 1718398885 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 4470 (time 1718398911.28 total 10.00 last 446.98) - open/close 8322 (time 1718398921.28 total 20.00 last 385.11) total: 10000 open/close in 23.78 seconds: 420.56 ops/second - unlinked 0 (time 1718398925 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4612 (time 1718398950.27 total 10.00 last 461.19) - open/close 8565 (time 1718398960.27 total 20.00 last 395.27) total: 10000 open/close in 23.41 seconds: 427.15 ops/second - unlinked 0 (time 1718398964 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.138@tcp lustre:sys.timeout=20 lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg138-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg138-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock oleg138-server@tcp:/lustre /mnt/lustre On OST0, 1837 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server LNET ready to unload unloading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing unload_modules_local oleg138-server: LNET ready to unload modules unloaded. pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2 PASS 69 (530s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 17:03:57 (1718399037) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg138-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg138-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 oleg138-server: oleg138-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: detected 4 online CPUs by sysfs oleg138-server: Force libcfs to create 2 CPU partitions oleg138-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg138-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg138-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 766 sec =========== 17:04:28 (1718399068)