== conf-sanity test 69: replace an OST with the same index ========================================================== 12:05:56 (1788797156) start mds service on oleg445-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 7 10:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: kmemleak disabled oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg445-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg445-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg445-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 700 (time 1788797288.73 total 10.02 last 69.86) - open/close 1326 (time 1788797298.74 total 20.02 last 62.59) - open/close 1988 (time 1788797308.74 total 30.03 last 66.16) - open/close 2729 (time 1788797318.74 total 40.03 last 74.09) - open/close 3444 (time 1788797328.75 total 50.03 last 71.48) - open/close 4129 (time 1788797338.77 total 60.05 last 68.35) - open/close 4650 (time 1788797348.78 total 70.07 last 52.03) - open/close 5260 (time 1788797358.80 total 80.08 last 60.89) - open/close 5903 (time 1788797368.81 total 90.09 last 64.26) - open/close 6627 (time 1788797378.81 total 100.09 last 72.38) - open/close 7297 (time 1788797388.81 total 110.10 last 66.98) - open/close 8117 (time 1788797398.82 total 120.10 last 81.94) - open/close 8797 (time 1788797408.84 total 130.12 last 67.86) - open/close 9433 (time 1788797418.86 total 140.14 last 63.48) total: 10000 open/close in 149.03 seconds: 67.10 ops/second - unlinked 0 (time 1788797432 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 627 (time 1788797534.78 total 10.01 last 62.63) - open/close 1197 (time 1788797544.79 total 20.02 last 56.95) - open/close 1911 (time 1788797554.80 total 30.02 last 71.38) - open/close 2641 (time 1788797564.80 total 40.03 last 72.97) - open/close 3374 (time 1788797574.81 total 50.04 last 73.24) - open/close 4011 (time 1788797584.82 total 60.05 last 63.61) - open/close 4645 (time 1788797594.83 total 70.05 last 63.37) - open/close 5253 (time 1788797604.84 total 80.07 last 60.72) - open/close 5904 (time 1788797614.85 total 90.08 last 65.05) - open/close 6562 (time 1788797624.86 total 100.09 last 65.73) - open/close 7138 (time 1788797634.87 total 110.10 last 57.53) - open/close 7809 (time 1788797644.88 total 120.11 last 67.02) - open/close 8584 (time 1788797654.89 total 130.12 last 77.46) - open/close 9300 (time 1788797664.89 total 140.12 last 71.59) - open/close 9979 (time 1788797674.90 total 150.13 last 67.84) total: 10000 open/close in 150.40 seconds: 66.49 ops/second - unlinked 0 (time 1788797679 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 700 (time 1788797774.19 total 10.01 last 69.92) - open/close 1472 (time 1788797784.19 total 20.01 last 77.20) - open/close 2173 (time 1788797794.20 total 30.02 last 70.07) - open/close 2920 (time 1788797804.20 total 40.02 last 74.66) - open/close 3651 (time 1788797814.21 total 50.03 last 73.04) - open/close 4393 (time 1788797824.21 total 60.03 last 74.20) - open/close 5069 (time 1788797834.22 total 70.04 last 67.54) - open/close 5868 (time 1788797844.23 total 80.05 last 79.84) - open/close 6598 (time 1788797854.23 total 90.05 last 72.99) - open/close 7313 (time 1788797864.25 total 100.07 last 71.38) - open/close 7951 (time 1788797874.28 total 110.10 last 63.60) - open/close 8689 (time 1788797884.29 total 120.11 last 73.74) - open/close 9402 (time 1788797894.29 total 130.11 last 71.23) total: 10000 open/close in 138.21 seconds: 72.36 ops/second - unlinked 0 (time 1788797906 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 716 (time 1788797994.37 total 10.00 last 71.56) - open/close 1495 (time 1788798004.37 total 20.01 last 77.88) - open/close 2235 (time 1788798014.39 total 30.03 last 73.84) - open/close 2961 (time 1788798024.39 total 40.03 last 72.58) - open/close 3685 (time 1788798034.40 total 50.04 last 72.38) - open/close 4439 (time 1788798044.41 total 60.04 last 75.33) - open/close 5201 (time 1788798054.41 total 70.05 last 76.17) - open/close 5909 (time 1788798064.41 total 80.05 last 70.80) - open/close 6710 (time 1788798074.42 total 90.06 last 80.00) - open/close 7432 (time 1788798084.43 total 100.06 last 72.18) - open/close 8208 (time 1788798094.43 total 110.07 last 77.56) - open/close 8952 (time 1788798104.43 total 120.07 last 74.37) - open/close 9627 (time 1788798114.44 total 130.08 last 67.48) total: 10000 open/close in 135.79 seconds: 73.64 ops/second - unlinked 0 (time 1788798124 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 765 (time 1788798219.25 total 10.00 last 76.48) - open/close 1499 (time 1788798229.26 total 20.01 last 73.33) - open/close 2261 (time 1788798239.26 total 30.01 last 76.19) - open/close 3069 (time 1788798249.28 total 40.03 last 80.70) - open/close 3877 (time 1788798259.28 total 50.03 last 80.73) - open/close 4641 (time 1788798269.29 total 60.04 last 76.36) - open/close 5393 (time 1788798279.30 total 70.05 last 75.14) - open/close 6140 (time 1788798289.30 total 80.05 last 74.67) - open/close 6703 (time 1788798299.31 total 90.06 last 56.27) - open/close 7270 (time 1788798309.31 total 100.06 last 56.67) - open/close 7945 (time 1788798319.33 total 110.07 last 67.41) - open/close 8623 (time 1788798329.33 total 120.08 last 67.78) - open/close 9302 (time 1788798339.34 total 130.09 last 67.81) - open/close 9943 (time 1788798349.35 total 140.10 last 64.07) total: 10000 open/close in 140.77 seconds: 71.04 ops/second - unlinked 0 (time 1788798354 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 685 (time 1788798452.64 total 10.00 last 68.49) - open/close 1445 (time 1788798462.65 total 20.02 last 75.88) - open/close 2141 (time 1788798472.68 total 30.04 last 69.44) - open/close 2869 (time 1788798482.69 total 40.05 last 72.72) - open/close 3595 (time 1788798492.69 total 50.06 last 72.57) - open/close 4362 (time 1788798502.70 total 60.07 last 76.62) - open/close 5060 (time 1788798512.71 total 70.07 last 69.76) - open/close 5813 (time 1788798522.71 total 80.08 last 75.26) - open/close 6531 (time 1788798532.72 total 90.09 last 71.75) - open/close 7283 (time 1788798542.72 total 100.09 last 75.18) - open/close 8077 (time 1788798552.73 total 110.10 last 79.34) - open/close 8848 (time 1788798562.74 total 120.10 last 77.07) - open/close 9663 (time 1788798572.74 total 130.11 last 81.44) total: 10000 open/close in 134.97 seconds: 74.09 ops/second - unlinked 0 (time 1788798581 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 639 (time 1788798683.95 total 10.02 last 63.79) - open/close 1357 (time 1788798693.95 total 20.02 last 71.78) - open/close 2095 (time 1788798703.96 total 30.03 last 73.72) - open/close 2873 (time 1788798713.97 total 40.04 last 77.74) - open/close 3589 (time 1788798723.97 total 50.04 last 71.58) - open/close 4317 (time 1788798733.98 total 60.05 last 72.77) - open/close 4976 (time 1788798743.98 total 70.05 last 65.85) - open/close 5642 (time 1788798754.00 total 80.07 last 66.49) - open/close 6426 (time 1788798764.01 total 90.08 last 78.33) - open/close 7115 (time 1788798774.04 total 100.11 last 68.69) - open/close 7810 (time 1788798784.04 total 110.11 last 69.47) - open/close 8529 (time 1788798794.04 total 120.12 last 71.89) - open/close 9247 (time 1788798804.05 total 130.12 last 71.75) - open/close 9994 (time 1788798814.06 total 140.13 last 74.64) total: 10000 open/close in 140.21 seconds: 71.32 ops/second - unlinked 0 (time 1788798817 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 599 (time 1788798912.07 total 10.01 last 59.82) - open/close 1252 (time 1788798922.07 total 20.01 last 65.29) - open/close 2008 (time 1788798932.08 total 30.03 last 75.51) - open/close 2720 (time 1788798942.08 total 40.03 last 71.19) - open/close 3492 (time 1788798952.09 total 50.03 last 77.20) - open/close 4249 (time 1788798962.09 total 60.03 last 75.69) - open/close 5021 (time 1788798972.09 total 70.03 last 77.20) - open/close 5742 (time 1788798982.10 total 80.05 last 71.98) - open/close 6500 (time 1788798992.11 total 90.06 last 75.73) - open/close 7266 (time 1788799002.14 total 100.08 last 76.43) - open/close 7992 (time 1788799012.15 total 110.09 last 72.52) - open/close 8718 (time 1788799022.15 total 120.09 last 72.56) - open/close 9516 (time 1788799032.16 total 130.10 last 79.77) total: 10000 open/close in 137.09 seconds: 72.94 ops/second - unlinked 0 (time 1788799043 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 754 (time 1788799135.56 total 10.01 last 75.34) - open/close 1569 (time 1788799145.57 total 20.02 last 81.43) - open/close 2281 (time 1788799155.58 total 30.02 last 71.15) - open/close 3037 (time 1788799165.60 total 40.04 last 75.46) - open/close 3719 (time 1788799175.60 total 50.05 last 68.15) - open/close 4522 (time 1788799185.60 total 60.05 last 80.29) - open/close 5238 (time 1788799195.61 total 70.06 last 71.54) - open/close 5874 (time 1788799205.62 total 80.07 last 63.53) - open/close 6560 (time 1788799215.63 total 90.08 last 68.56) - open/close 7172 (time 1788799225.63 total 100.08 last 61.18) - open/close 7725 (time 1788799235.64 total 110.08 last 55.28) - open/close 8426 (time 1788799245.64 total 120.08 last 70.10) - open/close 9087 (time 1788799255.64 total 130.08 last 66.10) - open/close 9731 (time 1788799265.64 total 140.08 last 64.39) total: 10000 open/close in 144.20 seconds: 69.35 ops/second - unlinked 0 (time 1788799274 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 745 (time 1788799367.68 total 10.00 last 74.48) - open/close 1489 (time 1788799377.68 total 20.01 last 74.35) - open/close 2168 (time 1788799387.70 total 30.02 last 67.83) - open/close 2826 (time 1788799397.70 total 40.03 last 65.75) - open/close 3593 (time 1788799407.71 total 50.04 last 76.64) - open/close 4424 (time 1788799417.72 total 60.04 last 83.06) - open/close 5154 (time 1788799427.73 total 70.05 last 72.92) - open/close 5936 (time 1788799437.73 total 80.05 last 78.17) - open/close 6706 (time 1788799447.73 total 90.06 last 76.99) - open/close 7502 (time 1788799457.74 total 100.06 last 79.55) - open/close 8125 (time 1788799467.75 total 110.07 last 62.24) - open/close 8845 (time 1788799477.75 total 120.07 last 72.00) - open/close 9554 (time 1788799487.75 total 130.08 last 70.85) total: 10000 open/close in 135.92 seconds: 73.57 ops/second - unlinked 0 (time 1788799497 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.145@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.145@tcp sys.timeout=20 start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg445-server unloading modules via unload_modules_local on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: kmemleak disabled modules unloaded. pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2