== conf-sanity test 69: replace an OST with the same index ========================================================== 08:49:31 (1747399771) start mds service on oleg139-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: detected 4 online CPUs by sysfs oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg139-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 1987 (time 1747399831.68 total 10.00 last 198.67) - open/close 4160 (time 1747399841.68 total 20.00 last 217.29) - open/close 6171 (time 1747399851.68 total 30.00 last 201.10) - open/close 8239 (time 1747399861.68 total 40.00 last 206.78) total: 10000 open/close in 48.09 seconds: 207.95 ops/second - unlinked 0 (time 1747399871 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 2323 (time 1747399910.13 total 10.00 last 232.21) - open/close 4771 (time 1747399920.13 total 20.01 last 244.77) - open/close 7079 (time 1747399930.13 total 30.01 last 230.79) - open/close 9097 (time 1747399940.13 total 40.01 last 201.72) total: 10000 open/close in 43.92 seconds: 227.68 ops/second - unlinked 0 (time 1747399945 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2046 (time 1747399980.86 total 10.00 last 204.57) - open/close 3930 (time 1747399990.86 total 20.00 last 188.36) - open/close 6006 (time 1747400000.87 total 30.00 last 207.58) - open/close 8026 (time 1747400010.87 total 40.01 last 201.91) total: 10000 open/close in 48.05 seconds: 208.13 ops/second - unlinked 0 (time 1747400020 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2284 (time 1747400052.78 total 10.00 last 228.38) - open/close 4570 (time 1747400062.78 total 20.00 last 228.58) - open/close 7282 (time 1747400072.78 total 30.00 last 271.18) - open/close 9772 (time 1747400082.79 total 40.00 last 248.96) total: 10000 open/close in 41.05 seconds: 243.60 ops/second - unlinked 0 (time 1747400085 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2496 (time 1747400117.51 total 10.00 last 249.56) - open/close 5382 (time 1747400127.51 total 20.00 last 288.59) - open/close 7995 (time 1747400137.51 total 30.00 last 261.28) total: 10000 open/close in 37.01 seconds: 270.19 ops/second - unlinked 0 (time 1747400145 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3241 (time 1747400173.66 total 10.00 last 324.09) - open/close 6436 (time 1747400183.66 total 20.00 last 319.42) - open/close 9627 (time 1747400193.66 total 30.00 last 319.05) total: 10000 open/close in 31.14 seconds: 321.16 ops/second - unlinked 0 (time 1747400195 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2662 (time 1747400225.41 total 10.00 last 266.19) - open/close 5562 (time 1747400235.41 total 20.00 last 290.00) - open/close 8566 (time 1747400245.41 total 30.00 last 300.37) total: 10000 open/close in 34.91 seconds: 286.48 ops/second - unlinked 0 (time 1747400251 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 2968 (time 1747400279.53 total 10.00 last 296.76) - open/close 5637 (time 1747400289.53 total 20.00 last 266.87) - open/close 8916 (time 1747400299.53 total 30.00 last 327.85) total: 10000 open/close in 32.56 seconds: 307.11 ops/second - unlinked 0 (time 1747400303 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3439 (time 1747400328.62 total 10.00 last 343.89) - open/close 6808 (time 1747400338.62 total 20.00 last 336.89) - open/close 9990 (time 1747400348.62 total 30.00 last 318.14) total: 10000 open/close in 30.03 seconds: 333.01 ops/second - unlinked 0 (time 1747400349 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3823 (time 1747400374.63 total 10.00 last 382.16) - open/close 7772 (time 1747400384.63 total 20.01 last 394.85) total: 10000 open/close in 25.52 seconds: 391.92 ops/second - unlinked 0 (time 1747400391 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg139-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg139-server LNET ready to unload unloading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local oleg139-server: LNET ready to unload modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2