== conf-sanity test 69: replace an OST with the same index ========================================================== 08:05:04 (1721217904) start mds service on oleg411-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: detected 4 online CPUs by sysfs oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg411-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg411-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock oleg411-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 4697 (time 1721217947.07 total 10.00 last 469.66) - open/close 8214 (time 1721217957.07 total 20.00 last 351.64) total: 10000 open/close in 25.85 seconds: 386.87 ops/second - unlinked 0 (time 1721217964 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3185 (time 1721217992.27 total 10.00 last 318.39) - open/close 6096 (time 1721218002.27 total 20.00 last 291.05) - open/close 8759 (time 1721218012.28 total 30.01 last 266.24) total: 10000 open/close in 34.63 seconds: 288.80 ops/second - unlinked 0 (time 1721218018 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3846 (time 1721218047.66 total 10.00 last 384.45) - open/close 6509 (time 1721218057.66 total 20.01 last 266.26) - open/close 9144 (time 1721218067.66 total 30.01 last 263.42) total: 10000 open/close in 33.22 seconds: 301.00 ops/second - unlinked 0 (time 1721218072 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2670 (time 1721218102.15 total 10.00 last 266.97) - open/close 5310 (time 1721218112.15 total 20.00 last 263.94) - open/close 7960 (time 1721218122.15 total 30.00 last 264.98) total: 10000 open/close in 36.43 seconds: 274.47 ops/second - unlinked 0 (time 1721218129 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 2819 (time 1721218156.68 total 10.00 last 281.86) - open/close 5762 (time 1721218166.68 total 20.00 last 294.24) - open/close 8894 (time 1721218176.68 total 30.00 last 313.15) total: 10000 open/close in 33.83 seconds: 295.55 ops/second - unlinked 0 (time 1721218181 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2916 (time 1721218210.45 total 10.00 last 291.54) - open/close 5554 (time 1721218220.45 total 20.00 last 263.77) - open/close 8204 (time 1721218230.45 total 30.00 last 264.99) total: 10000 open/close in 36.66 seconds: 272.75 ops/second - unlinked 0 (time 1721218238 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2862 (time 1721218268.66 total 10.00 last 286.11) - open/close 5538 (time 1721218278.66 total 20.00 last 267.57) - open/close 8210 (time 1721218288.66 total 30.01 last 267.18) total: 10000 open/close in 36.22 seconds: 276.07 ops/second - unlinked 0 (time 1721218296 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 3278 (time 1721218326.39 total 10.00 last 327.70) - open/close 6105 (time 1721218336.39 total 20.01 last 282.61) - open/close 8862 (time 1721218346.39 total 30.01 last 275.63) total: 10000 open/close in 34.24 seconds: 292.06 ops/second - unlinked 0 (time 1721218351 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 2827 (time 1721218375.63 total 10.00 last 282.65) - open/close 5883 (time 1721218385.63 total 20.01 last 305.50) - open/close 8822 (time 1721218395.63 total 30.01 last 293.88) total: 10000 open/close in 34.54 seconds: 289.48 ops/second - unlinked 0 (time 1721218401 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2706 (time 1721218430.92 total 10.00 last 270.52) - open/close 5515 (time 1721218440.93 total 20.01 last 280.83) - open/close 8642 (time 1721218450.93 total 30.01 last 312.63) total: 10000 open/close in 34.58 seconds: 289.20 ops/second - unlinked 0 (time 1721218456 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg411-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg411-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock oleg411-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server LNET ready to unload unloading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: LNET ready to unload modules unloaded. pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2