== conf-sanity test 69: replace an OST with the same index ========================================================== 23:03:01 (1789182181) start mds service on oleg444-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 11 11:59 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: kmemleak disabled oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg444-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg444-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg444-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 530 (time 1789182300.74 total 10.03 last 52.82) - open/close 1156 (time 1789182310.76 total 20.05 last 62.49) - open/close 1785 (time 1789182320.77 total 30.06 last 62.84) - open/close 2437 (time 1789182330.77 total 40.07 last 65.15) - open/close 3017 (time 1789182340.78 total 50.08 last 57.96) - open/close 3686 (time 1789182350.79 total 60.09 last 66.82) - open/close 4444 (time 1789182360.80 total 70.10 last 75.71) - open/close 5139 (time 1789182370.81 total 80.10 last 69.50) - open/close 5835 (time 1789182380.81 total 90.10 last 69.57) - open/close 6510 (time 1789182390.81 total 100.11 last 67.49) - open/close 7135 (time 1789182400.83 total 110.12 last 62.40) - open/close 7814 (time 1789182410.84 total 120.13 last 67.83) - open/close 8562 (time 1789182420.84 total 130.13 last 74.78) - open/close 9292 (time 1789182430.84 total 140.14 last 72.99) total: 10000 open/close in 149.76 seconds: 66.77 ops/second - unlinked 0 (time 1789182445 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 637 (time 1789182543.05 total 10.00 last 63.68) - open/close 1327 (time 1789182553.07 total 20.02 last 68.86) - open/close 1997 (time 1789182563.07 total 30.03 last 66.98) - open/close 2832 (time 1789182573.08 total 40.03 last 83.43) - open/close 3546 (time 1789182583.10 total 50.05 last 71.30) - open/close 4151 (time 1789182593.10 total 60.05 last 60.48) - open/close 4855 (time 1789182603.10 total 70.05 last 70.39) - open/close 5621 (time 1789182613.11 total 80.07 last 76.51) - open/close 6286 (time 1789182623.12 total 90.07 last 66.44) - open/close 6984 (time 1789182633.13 total 100.08 last 69.78) - open/close 7664 (time 1789182643.14 total 110.09 last 67.89) - open/close 8247 (time 1789182653.16 total 120.11 last 58.22) - open/close 8905 (time 1789182663.16 total 130.11 last 65.77) - open/close 9534 (time 1789182673.18 total 140.14 last 62.76) total: 10000 open/close in 146.75 seconds: 68.14 ops/second - unlinked 0 (time 1789182684 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 629 (time 1789182788.04 total 10.01 last 62.87) - open/close 1231 (time 1789182798.06 total 20.03 last 60.08) - open/close 1919 (time 1789182808.07 total 30.04 last 68.70) - open/close 2635 (time 1789182818.09 total 40.06 last 71.49) - open/close 3326 (time 1789182828.10 total 50.06 last 69.03) - open/close 3963 (time 1789182838.10 total 60.07 last 63.70) - open/close 4560 (time 1789182848.10 total 70.07 last 59.68) - open/close 5222 (time 1789182858.10 total 80.07 last 66.18) - open/close 5937 (time 1789182868.12 total 90.09 last 71.39) - open/close 6603 (time 1789182878.13 total 100.10 last 66.52) - open/close 7261 (time 1789182888.14 total 110.11 last 65.74) - open/close 7994 (time 1789182898.15 total 120.12 last 73.23) - open/close 8670 (time 1789182908.15 total 130.12 last 67.57) - open/close 9355 (time 1789182918.17 total 140.14 last 68.40) total: 10000 open/close in 150.00 seconds: 66.67 ops/second - unlinked 0 (time 1789182932 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 761 (time 1789183024.96 total 10.01 last 76.00) - open/close 1549 (time 1789183034.96 total 20.01 last 78.79) - open/close 2321 (time 1789183044.99 total 30.04 last 77.02) - open/close 3053 (time 1789183054.99 total 40.04 last 73.16) - open/close 3789 (time 1789183065.00 total 50.05 last 73.58) - open/close 4443 (time 1789183075.01 total 60.05 last 65.34) - open/close 5204 (time 1789183085.02 total 70.07 last 76.00) - open/close 5863 (time 1789183095.03 total 80.08 last 65.82) - open/close 6446 (time 1789183105.03 total 90.08 last 58.27) - open/close 7238 (time 1789183115.04 total 100.09 last 79.12) - open/close 7915 (time 1789183125.06 total 110.11 last 67.61) - open/close 8646 (time 1789183135.08 total 120.13 last 72.94) - open/close 9363 (time 1789183145.09 total 130.14 last 71.64) - open/close 9969 (time 1789183155.10 total 140.15 last 60.52) total: 10000 open/close in 140.76 seconds: 71.04 ops/second - unlinked 0 (time 1789183161 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 758 (time 1789183256.01 total 10.02 last 75.66) - open/close 1419 (time 1789183266.01 total 20.02 last 66.07) - open/close 2108 (time 1789183276.03 total 30.04 last 68.79) - open/close 2807 (time 1789183286.04 total 40.05 last 69.82) - open/close 3528 (time 1789183296.06 total 50.07 last 71.97) - open/close 4254 (time 1789183306.07 total 60.08 last 72.48) - open/close 4967 (time 1789183316.08 total 70.09 last 71.27) - open/close 5734 (time 1789183326.08 total 80.09 last 76.70) - open/close 6483 (time 1789183336.09 total 90.10 last 74.84) - open/close 7135 (time 1789183346.09 total 100.10 last 65.17) - open/close 7771 (time 1789183356.09 total 110.10 last 63.59) - open/close 8389 (time 1789183366.12 total 120.13 last 61.63) - open/close 9118 (time 1789183376.14 total 130.15 last 72.76) - open/close 9739 (time 1789183386.14 total 140.16 last 62.06) total: 10000 open/close in 143.22 seconds: 69.82 ops/second - unlinked 0 (time 1789183394 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 752 (time 1789183487.00 total 10.01 last 75.15) - open/close 1427 (time 1789183497.02 total 20.03 last 67.35) - open/close 2175 (time 1789183507.03 total 30.03 last 74.76) - open/close 2902 (time 1789183517.03 total 40.04 last 72.68) - open/close 3658 (time 1789183527.04 total 50.05 last 75.51) - open/close 4366 (time 1789183537.05 total 60.05 last 70.79) - open/close 5122 (time 1789183547.06 total 70.06 last 75.52) - open/close 5871 (time 1789183557.06 total 80.07 last 74.86) - open/close 6552 (time 1789183567.07 total 90.07 last 68.05) - open/close 7287 (time 1789183577.08 total 100.08 last 73.45) - open/close 7986 (time 1789183587.08 total 110.08 last 69.89) - open/close 8718 (time 1789183597.10 total 120.10 last 73.07) - open/close 9278 (time 1789183607.10 total 130.11 last 55.97) - open/close 9886 (time 1789183617.10 total 140.11 last 60.79) total: 10000 open/close in 142.05 seconds: 70.40 ops/second - unlinked 0 (time 1789183623 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 756 (time 1789183714.32 total 10.00 last 75.60) - open/close 1505 (time 1789183724.33 total 20.01 last 74.84) - open/close 2236 (time 1789183734.33 total 30.01 last 73.06) - open/close 2962 (time 1789183744.35 total 40.03 last 72.48) - open/close 3708 (time 1789183754.35 total 50.03 last 74.58) - open/close 4475 (time 1789183764.36 total 60.04 last 76.68) - open/close 5248 (time 1789183774.37 total 70.05 last 77.19) - open/close 5981 (time 1789183784.38 total 80.06 last 73.25) - open/close 6709 (time 1789183794.38 total 90.06 last 72.76) - open/close 7376 (time 1789183804.43 total 100.11 last 66.36) - open/close 7973 (time 1789183814.44 total 110.12 last 59.66) - open/close 8647 (time 1789183824.44 total 120.12 last 67.38) - open/close 9374 (time 1789183834.45 total 130.13 last 72.65) total: 10000 open/close in 138.32 seconds: 72.30 ops/second - unlinked 0 (time 1789183848 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 716 (time 1789183939.75 total 10.01 last 71.53) - open/close 1471 (time 1789183949.76 total 20.02 last 75.43) - open/close 2138 (time 1789183959.77 total 30.03 last 66.66) - open/close 2814 (time 1789183969.79 total 40.05 last 67.47) - open/close 3437 (time 1789183979.80 total 50.05 last 62.25) - open/close 4128 (time 1789183989.80 total 60.06 last 69.08) - open/close 4759 (time 1789183999.82 total 70.08 last 62.96) - open/close 5357 (time 1789184009.85 total 80.10 last 59.67) - open/close 5948 (time 1789184019.85 total 90.11 last 59.06) - open/close 6646 (time 1789184029.88 total 100.13 last 69.63) - open/close 7248 (time 1789184039.88 total 110.14 last 60.17) - open/close 7952 (time 1789184049.89 total 120.14 last 70.36) - open/close 8692 (time 1789184059.89 total 130.15 last 73.96) - open/close 9428 (time 1789184069.90 total 140.16 last 73.55) total: 10000 open/close in 147.95 seconds: 67.59 ops/second - unlinked 0 (time 1789184083 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 656 (time 1789184187.23 total 10.01 last 65.57) - open/close 1269 (time 1789184197.23 total 20.01 last 61.29) - open/close 2003 (time 1789184207.24 total 30.01 last 73.35) - open/close 2712 (time 1789184217.24 total 40.01 last 70.90) - open/close 3365 (time 1789184227.25 total 50.02 last 65.24) - open/close 4025 (time 1789184237.26 total 60.04 last 65.91) - open/close 4645 (time 1789184247.27 total 70.05 last 61.92) - open/close 5366 (time 1789184257.28 total 80.05 last 72.09) - open/close 6123 (time 1789184267.28 total 90.06 last 75.64) - open/close 6812 (time 1789184277.29 total 100.06 last 68.87) - open/close 7505 (time 1789184287.29 total 110.07 last 69.27) - open/close 8340 (time 1789184297.30 total 120.07 last 83.46) - open/close 9104 (time 1789184307.30 total 130.08 last 76.35) - open/close 9871 (time 1789184317.30 total 140.08 last 76.69) total: 10000 open/close in 141.94 seconds: 70.45 ops/second - unlinked 0 (time 1789184324 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 737 (time 1789184419.26 total 10.01 last 73.64) - open/close 1430 (time 1789184429.26 total 20.01 last 69.29) - open/close 2064 (time 1789184439.27 total 30.02 last 63.34) - open/close 2833 (time 1789184449.28 total 40.03 last 76.85) - open/close 3456 (time 1789184459.28 total 50.03 last 62.29) - open/close 4079 (time 1789184469.29 total 60.03 last 62.26) - open/close 4632 (time 1789184479.29 total 70.04 last 55.28) - open/close 5217 (time 1789184489.30 total 80.05 last 58.44) - open/close 5789 (time 1789184499.31 total 90.06 last 57.13) - open/close 6454 (time 1789184509.31 total 100.06 last 66.50) - open/close 7146 (time 1789184519.33 total 110.08 last 69.06) - open/close 7826 (time 1789184529.34 total 120.09 last 67.92) - open/close 8486 (time 1789184539.36 total 130.11 last 65.86) - open/close 9146 (time 1789184549.37 total 140.12 last 65.98) - open/close 9768 (time 1789184559.37 total 150.12 last 62.17) total: 10000 open/close in 153.82 seconds: 65.01 ops/second - unlinked 0 (time 1789184568 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg444-server unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: kmemleak disabled modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2