== conf-sanity test 69: replace an OST with the same index ========================================================== 08:50:18 (1747399818) start mds service on oleg246-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg246-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg246-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock oleg246-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 1674 (time 1747399865.54 total 10.00 last 167.38) - open/close 3353 (time 1747399875.55 total 20.01 last 167.79) - open/close 4931 (time 1747399885.55 total 30.01 last 157.76) - open/close 6523 (time 1747399895.56 total 40.01 last 159.13) - open/close 8036 (time 1747399905.56 total 50.01 last 151.30) - open/close 9674 (time 1747399915.56 total 60.02 last 163.78) total: 10000 open/close in 61.94 seconds: 161.44 ops/second - unlinked 0 (time 1747399919 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1658 (time 1747399964.47 total 10.01 last 165.63) - open/close 3092 (time 1747399974.47 total 20.01 last 143.35) - open/close 4820 (time 1747399984.47 total 30.02 last 172.78) - open/close 6541 (time 1747399994.48 total 40.02 last 172.02) - open/close 8054 (time 1747400004.48 total 50.02 last 151.27) - open/close 9702 (time 1747400014.48 total 60.02 last 164.78) total: 10000 open/close in 61.81 seconds: 161.79 ops/second - unlinked 0 (time 1747400017 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1970 (time 1747400060.14 total 10.00 last 196.94) - open/close 3761 (time 1747400070.15 total 20.01 last 179.00) - open/close 5403 (time 1747400080.15 total 30.01 last 164.18) - open/close 7184 (time 1747400090.15 total 40.01 last 178.05) - open/close 9158 (time 1747400100.15 total 50.01 last 197.36) total: 10000 open/close in 53.96 seconds: 185.31 ops/second - unlinked 0 (time 1747400105 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 1926 (time 1747400141.06 total 10.00 last 192.59) - open/close 3799 (time 1747400151.06 total 20.00 last 187.22) - open/close 5650 (time 1747400161.07 total 30.01 last 184.95) - open/close 7677 (time 1747400171.07 total 40.02 last 202.64) - open/close 9782 (time 1747400181.07 total 50.02 last 210.44) total: 10000 open/close in 51.17 seconds: 195.44 ops/second - unlinked 0 (time 1747400183 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2168 (time 1747400216.42 total 10.00 last 216.73) - open/close 4127 (time 1747400226.42 total 20.01 last 195.77) - open/close 5903 (time 1747400236.42 total 30.01 last 177.58) - open/close 7814 (time 1747400246.43 total 40.02 last 190.99) - open/close 9837 (time 1747400256.43 total 50.02 last 202.28) total: 10000 open/close in 50.83 seconds: 196.74 ops/second - unlinked 0 (time 1747400258 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2113 (time 1747400291.78 total 10.00 last 211.23) - open/close 4458 (time 1747400301.79 total 20.00 last 234.48) - open/close 6662 (time 1747400311.79 total 30.00 last 220.39) - open/close 9082 (time 1747400321.79 total 40.01 last 241.88) total: 10000 open/close in 44.90 seconds: 222.72 ops/second - unlinked 0 (time 1747400328 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2973 (time 1747400358.98 total 10.00 last 297.21) - open/close 5850 (time 1747400368.98 total 20.00 last 287.65) - open/close 8322 (time 1747400378.98 total 30.01 last 247.19) total: 10000 open/close in 36.01 seconds: 277.70 ops/second - unlinked 0 (time 1747400386 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2243 (time 1747400421.17 total 10.00 last 224.25) - open/close 4396 (time 1747400431.17 total 20.01 last 215.23) - open/close 6544 (time 1747400441.17 total 30.01 last 214.74) - open/close 8963 (time 1747400451.18 total 40.01 last 241.83) total: 10000 open/close in 44.18 seconds: 226.35 ops/second - unlinked 0 (time 1747400456 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2417 (time 1747400485.13 total 10.00 last 241.63) - open/close 5667 (time 1747400495.13 total 20.00 last 324.99) - open/close 9221 (time 1747400505.13 total 30.00 last 355.40) total: 10000 open/close in 31.81 seconds: 314.35 ops/second - unlinked 0 (time 1747400507 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3106 (time 1747400534.62 total 10.00 last 310.52) - open/close 5956 (time 1747400544.62 total 20.01 last 284.91) - open/close 9410 (time 1747400554.63 total 30.01 last 345.38) total: 10000 open/close in 31.80 seconds: 314.43 ops/second - unlinked 0 (time 1747400557 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.146@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg246-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg246-server: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg246-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec oleg246-server: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg246-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock oleg246-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server LNET ready to unload unloading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local oleg246-server: LNET ready to unload modules unloaded. pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2