== conf-sanity test 69: replace an OST with the same index ========================================================== 16:53:26 (1718398406) start mds service on oleg343-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: detected 4 online CPUs by sysfs oleg343-server: Force libcfs to create 2 CPU partitions oleg343-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg343-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg343-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre On OST0, 49700 inodes available. Want 99872. rc=0 - open/close 1716 (time 1718398454.09 total 10.00 last 171.58) - open/close 3504 (time 1718398464.10 total 20.01 last 178.70) - open/close 5381 (time 1718398474.10 total 30.01 last 187.67) - open/close 7329 (time 1718398484.10 total 40.01 last 194.77) - open/close 9561 (time 1718398494.10 total 50.01 last 223.18) total: 10000 open/close in 51.60 seconds: 193.81 ops/second - unlinked 0 (time 1718398496 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2849 (time 1718398532.77 total 10.00 last 284.89) - open/close 5325 (time 1718398542.78 total 20.01 last 247.42) - open/close 7492 (time 1718398552.78 total 30.01 last 216.64) total: 10000 open/close in 39.70 seconds: 251.88 ops/second - unlinked 0 (time 1718398563 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2484 (time 1718398598.31 total 10.00 last 248.37) - open/close 5207 (time 1718398608.31 total 20.00 last 272.25) - open/close 7844 (time 1718398618.31 total 30.00 last 263.67) total: 10000 open/close in 38.18 seconds: 261.94 ops/second - unlinked 0 (time 1718398627 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2004 (time 1718398658.17 total 10.00 last 200.33) - open/close 4107 (time 1718398668.18 total 20.01 last 210.23) - open/close 6239 (time 1718398678.18 total 30.01 last 213.17) - open/close 9297 (time 1718398688.18 total 40.01 last 305.78) total: 10000 open/close in 42.25 seconds: 236.67 ops/second - unlinked 0 (time 1718398691 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3025 (time 1718398720.29 total 10.00 last 302.44) - open/close 6405 (time 1718398730.29 total 20.00 last 338.00) - open/close 9517 (time 1718398740.29 total 30.00 last 311.18) total: 10000 open/close in 31.48 seconds: 317.64 ops/second - unlinked 0 (time 1718398742 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2797 (time 1718398772.28 total 10.00 last 279.58) - open/close 6059 (time 1718398782.28 total 20.01 last 326.16) - open/close 9206 (time 1718398792.28 total 30.01 last 314.68) total: 10000 open/close in 32.67 seconds: 306.08 ops/second - unlinked 0 (time 1718398796 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 2617 (time 1718398822.71 total 10.00 last 261.70) - open/close 5509 (time 1718398832.71 total 20.00 last 289.13) - open/close 8267 (time 1718398842.72 total 30.00 last 275.77) total: 10000 open/close in 36.83 seconds: 271.49 ops/second - unlinked 0 (time 1718398850 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3398 (time 1718398878.48 total 10.00 last 339.71) - open/close 7501 (time 1718398888.48 total 20.00 last 410.30) total: 10000 open/close in 26.08 seconds: 383.42 ops/second - unlinked 0 (time 1718398895 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 3691 (time 1718398918.58 total 10.00 last 369.09) - open/close 7904 (time 1718398928.58 total 20.00 last 421.28) total: 10000 open/close in 24.62 seconds: 406.17 ops/second - unlinked 0 (time 1718398934 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 3720 (time 1718398957.66 total 10.00 last 371.98) - open/close 7773 (time 1718398967.66 total 20.00 last 405.27) total: 10000 open/close in 24.97 seconds: 400.56 ops/second - unlinked 0 (time 1718398973 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg343-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg343-server: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg343-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre On OST0, 10235 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server LNET ready to unload unloading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local oleg343-server: LNET ready to unload modules unloaded. pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2