== conf-sanity test 69: replace an OST with the same index ========================================================== 00:16:47 (1788841007) start mds service on oleg138-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:41 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: kmemleak disabled oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions oleg138-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg138-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg138-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg138-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg138-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg138-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 928 (time 1788841125.25 total 10.00 last 92.75) - open/close 1714 (time 1788841135.27 total 20.02 last 78.48) - open/close 2604 (time 1788841145.29 total 30.04 last 88.80) - open/close 3476 (time 1788841155.30 total 40.05 last 87.13) - open/close 4194 (time 1788841165.31 total 50.06 last 71.73) - open/close 5004 (time 1788841175.31 total 60.06 last 80.97) - open/close 5758 (time 1788841185.31 total 70.06 last 75.40) - open/close 6440 (time 1788841195.31 total 80.07 last 68.18) - open/close 7261 (time 1788841205.33 total 90.08 last 82.00) - open/close 8081 (time 1788841215.34 total 100.09 last 81.90) - open/close 8753 (time 1788841225.35 total 110.10 last 67.15) - open/close 9385 (time 1788841235.35 total 120.10 last 63.17) total: 10000 open/close in 126.50 seconds: 79.05 ops/second - unlinked 0 (time 1788841246 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 977 (time 1788841332.13 total 10.01 last 97.61) - open/close 1813 (time 1788841342.13 total 20.01 last 83.57) - open/close 2638 (time 1788841352.14 total 30.02 last 82.41) - open/close 3543 (time 1788841362.15 total 40.03 last 90.46) - open/close 4408 (time 1788841372.15 total 50.03 last 86.45) - open/close 5132 (time 1788841382.15 total 60.04 last 72.40) - open/close 5811 (time 1788841392.18 total 70.06 last 67.70) - open/close 6329 (time 1788841402.19 total 80.08 last 51.74) - open/close 7047 (time 1788841412.19 total 90.08 last 71.79) - open/close 7977 (time 1788841422.21 total 100.09 last 92.85) - open/close 8952 (time 1788841432.21 total 110.10 last 97.47) - open/close 9898 (time 1788841442.23 total 120.12 last 94.42) total: 10000 open/close in 121.66 seconds: 82.20 ops/second - unlinked 0 (time 1788841448 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 831 (time 1788841543.48 total 10.01 last 83.04) - open/close 1657 (time 1788841553.48 total 20.01 last 82.58) - open/close 2586 (time 1788841563.49 total 30.02 last 92.80) - open/close 3324 (time 1788841573.51 total 40.04 last 73.67) - open/close 4016 (time 1788841583.52 total 50.05 last 69.10) - open/close 4738 (time 1788841593.54 total 60.07 last 72.08) - open/close 5525 (time 1788841603.55 total 70.08 last 78.61) - open/close 6318 (time 1788841613.56 total 80.09 last 79.21) - open/close 7156 (time 1788841623.59 total 90.12 last 83.61) - open/close 7837 (time 1788841633.59 total 100.12 last 68.09) - open/close 8602 (time 1788841643.59 total 110.12 last 76.46) - open/close 9536 (time 1788841653.60 total 120.13 last 93.34) total: 10000 open/close in 125.83 seconds: 79.47 ops/second - unlinked 0 (time 1788841664 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 850 (time 1788841755.44 total 10.00 last 84.98) - open/close 1659 (time 1788841765.44 total 20.00 last 80.88) - open/close 2532 (time 1788841775.46 total 30.03 last 87.11) - open/close 3461 (time 1788841785.46 total 40.03 last 92.87) - open/close 4359 (time 1788841795.47 total 50.04 last 89.72) - open/close 5241 (time 1788841805.49 total 60.05 last 88.07) - open/close 6279 (time 1788841815.49 total 70.06 last 103.74) - open/close 7101 (time 1788841825.49 total 80.06 last 82.20) - open/close 8117 (time 1788841835.50 total 90.06 last 101.57) - open/close 9091 (time 1788841845.50 total 100.07 last 97.38) - open/close 9962 (time 1788841855.51 total 110.08 last 87.01) total: 10000 open/close in 110.65 seconds: 90.37 ops/second - unlinked 0 (time 1788841860 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 793 (time 1788841941.81 total 10.03 last 79.09) - open/close 1515 (time 1788841951.81 total 20.03 last 72.15) - open/close 2325 (time 1788841961.82 total 30.04 last 80.97) - open/close 3147 (time 1788841971.82 total 40.04 last 82.17) - open/close 4013 (time 1788841981.83 total 50.05 last 86.49) - open/close 4768 (time 1788841991.84 total 60.06 last 75.42) - open/close 5478 (time 1788842001.85 total 70.07 last 70.95) - open/close 6196 (time 1788842011.86 total 80.08 last 71.74) - open/close 6780 (time 1788842021.88 total 90.10 last 58.29) - open/close 7345 (time 1788842031.90 total 100.12 last 56.35) - open/close 8143 (time 1788842041.92 total 110.14 last 79.71) - open/close 8640 (time 1788842051.93 total 120.15 last 49.62) - open/close 9116 (time 1788842061.94 total 130.16 last 47.55) - open/close 9924 (time 1788842071.95 total 140.17 last 80.75) total: 10000 open/close in 141.00 seconds: 70.92 ops/second - unlinked 0 (time 1788842078 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 758 (time 1788842160.68 total 10.00 last 75.79) - open/close 1329 (time 1788842170.69 total 20.00 last 57.08) - open/close 2143 (time 1788842180.69 total 30.01 last 81.34) - open/close 3035 (time 1788842190.70 total 40.02 last 89.16) - open/close 3819 (time 1788842200.70 total 50.02 last 78.39) - open/close 4598 (time 1788842210.71 total 60.03 last 77.83) - open/close 5396 (time 1788842220.71 total 70.03 last 79.77) - open/close 6279 (time 1788842230.72 total 80.04 last 88.24) - open/close 7257 (time 1788842240.72 total 90.04 last 97.79) - open/close 8231 (time 1788842250.72 total 100.04 last 97.38) - open/close 9176 (time 1788842260.72 total 110.04 last 94.50) total: 10000 open/close in 119.39 seconds: 83.76 ops/second - unlinked 0 (time 1788842273 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 780 (time 1788842356.29 total 10.00 last 77.98) - open/close 1693 (time 1788842366.30 total 20.01 last 91.20) - open/close 2494 (time 1788842376.31 total 30.01 last 80.09) - open/close 3208 (time 1788842386.31 total 40.02 last 71.36) - open/close 3926 (time 1788842396.31 total 50.02 last 71.78) - open/close 4829 (time 1788842406.32 total 60.03 last 90.27) - open/close 5632 (time 1788842416.32 total 70.03 last 80.27) - open/close 6327 (time 1788842426.34 total 80.05 last 69.38) - open/close 7154 (time 1788842436.36 total 90.07 last 82.55) - open/close 7936 (time 1788842446.37 total 100.08 last 78.13) - open/close 8756 (time 1788842456.37 total 110.08 last 81.98) - open/close 9527 (time 1788842466.37 total 120.08 last 77.08) total: 10000 open/close in 125.02 seconds: 79.98 ops/second - unlinked 0 (time 1788842476 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 818 (time 1788842557.58 total 10.02 last 81.66) - open/close 1681 (time 1788842567.59 total 20.03 last 86.23) - open/close 2435 (time 1788842577.60 total 30.03 last 75.34) - open/close 3386 (time 1788842587.60 total 40.04 last 95.08) - open/close 4081 (time 1788842597.61 total 50.05 last 69.42) - open/close 4875 (time 1788842607.62 total 60.05 last 79.35) - open/close 5798 (time 1788842617.63 total 70.06 last 92.25) - open/close 6567 (time 1788842627.63 total 80.07 last 76.84) - open/close 7465 (time 1788842637.64 total 90.07 last 89.75) - open/close 8380 (time 1788842647.64 total 100.07 last 91.48) - open/close 9106 (time 1788842657.64 total 110.08 last 72.58) - open/close 9781 (time 1788842667.65 total 120.08 last 67.45) total: 10000 open/close in 122.83 seconds: 81.41 ops/second - unlinked 0 (time 1788842675 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1010 (time 1788842758.45 total 10.01 last 100.91) - open/close 1835 (time 1788842768.48 total 20.04 last 82.26) - open/close 2385 (time 1788842778.50 total 30.06 last 54.90) - open/close 3097 (time 1788842788.50 total 40.06 last 71.14) - open/close 3783 (time 1788842798.51 total 50.07 last 68.56) - open/close 4615 (time 1788842808.52 total 60.08 last 83.14) - open/close 5360 (time 1788842818.53 total 70.09 last 74.43) - open/close 6230 (time 1788842828.53 total 80.09 last 86.97) - open/close 7112 (time 1788842838.54 total 90.10 last 88.13) - open/close 8003 (time 1788842848.54 total 100.10 last 89.05) - open/close 8768 (time 1788842858.55 total 110.11 last 76.44) - open/close 9560 (time 1788842868.55 total 120.11 last 79.20) total: 10000 open/close in 126.25 seconds: 79.21 ops/second - unlinked 0 (time 1788842879 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 757 (time 1788842964.76 total 10.00 last 75.68) - open/close 1484 (time 1788842974.76 total 20.01 last 72.67) - open/close 2399 (time 1788842984.77 total 30.01 last 91.45) - open/close 3264 (time 1788842994.77 total 40.02 last 86.46) - open/close 3971 (time 1788843004.78 total 50.03 last 70.64) - open/close 4566 (time 1788843014.79 total 60.03 last 59.47) - open/close 5204 (time 1788843024.79 total 70.04 last 63.76) - open/close 5861 (time 1788843034.79 total 80.04 last 65.69) - open/close 6630 (time 1788843044.80 total 90.05 last 76.85) - open/close 7303 (time 1788843054.80 total 100.05 last 67.28) - open/close 7862 (time 1788843064.81 total 110.05 last 55.88) - open/close 8495 (time 1788843074.81 total 120.06 last 63.27) - open/close 9081 (time 1788843084.82 total 130.06 last 58.58) - open/close 9780 (time 1788843094.82 total 140.06 last 69.88) total: 10000 open/close in 142.75 seconds: 70.05 ops/second - unlinked 0 (time 1788843102 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg138-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg138-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg138-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 start ost1 service on oleg138-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg138-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg138-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg138-server unloading modules via unload_modules_local on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing unload_modules_local oleg138-server: kmemleak disabled modules unloaded. pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2