== conf-sanity test 69: replace an OST with the same index ========================================================== 22:39:44 (1787884784) start mds service on oleg645-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754920 Aug 26 15:28 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: kmemleak disabled oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg645-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg645-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg645-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg645-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 826 (time 1787884899.63 total 10.01 last 82.54) - open/close 1585 (time 1787884909.64 total 20.02 last 75.81) - open/close 2424 (time 1787884919.65 total 30.02 last 83.86) - open/close 3213 (time 1787884929.67 total 40.05 last 78.72) - open/close 4048 (time 1787884939.68 total 50.06 last 83.44) - open/close 4872 (time 1787884949.68 total 60.06 last 82.37) - open/close 5742 (time 1787884959.70 total 70.07 last 86.90) - open/close 6592 (time 1787884969.70 total 80.08 last 84.95) - open/close 7311 (time 1787884979.71 total 90.08 last 71.85) - open/close 7964 (time 1787884989.72 total 100.09 last 65.22) - open/close 8675 (time 1787884999.75 total 110.12 last 70.92) - open/close 9437 (time 1787885009.75 total 120.12 last 76.19) total: 10000 open/close in 128.23 seconds: 77.99 ops/second - unlinked 0 (time 1787885023 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 805 (time 1787885113.42 total 10.03 last 80.23) - open/close 1604 (time 1787885123.43 total 20.04 last 79.83) - open/close 2489 (time 1787885133.44 total 30.05 last 88.40) - open/close 3319 (time 1787885143.45 total 40.06 last 82.95) - open/close 4222 (time 1787885153.45 total 50.06 last 90.27) - open/close 5045 (time 1787885163.45 total 60.07 last 82.27) - open/close 5755 (time 1787885173.45 total 70.07 last 70.98) - open/close 6605 (time 1787885183.47 total 80.08 last 84.91) - open/close 7386 (time 1787885193.47 total 90.08 last 78.09) - open/close 8085 (time 1787885203.47 total 100.09 last 69.85) - open/close 8812 (time 1787885213.48 total 110.09 last 72.68) - open/close 9651 (time 1787885223.48 total 120.10 last 83.84) total: 10000 open/close in 124.16 seconds: 80.54 ops/second - unlinked 0 (time 1787885231 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 754 (time 1787885311.31 total 10.00 last 75.38) - open/close 1581 (time 1787885321.32 total 20.01 last 82.62) - open/close 2413 (time 1787885331.35 total 30.04 last 82.99) - open/close 3217 (time 1787885341.35 total 40.04 last 80.38) - open/close 4165 (time 1787885351.36 total 50.05 last 94.72) - open/close 5043 (time 1787885361.36 total 60.05 last 87.78) - open/close 5849 (time 1787885371.36 total 70.05 last 80.58) - open/close 6530 (time 1787885381.37 total 80.05 last 68.08) - open/close 7315 (time 1787885391.37 total 90.06 last 78.49) - open/close 8161 (time 1787885401.37 total 100.06 last 84.55) - open/close 8986 (time 1787885411.38 total 110.07 last 82.46) - open/close 9806 (time 1787885421.39 total 120.07 last 81.93) total: 10000 open/close in 122.30 seconds: 81.77 ops/second - unlinked 0 (time 1787885427 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 817 (time 1787885508.84 total 10.01 last 81.65) - open/close 1688 (time 1787885518.85 total 20.01 last 87.03) - open/close 2467 (time 1787885528.86 total 30.02 last 77.83) - open/close 3200 (time 1787885538.86 total 40.02 last 73.29) - open/close 3946 (time 1787885548.87 total 50.04 last 74.51) - open/close 4777 (time 1787885558.88 total 60.04 last 83.04) - open/close 5574 (time 1787885568.88 total 70.04 last 79.69) - open/close 6375 (time 1787885578.89 total 80.06 last 79.99) - open/close 7279 (time 1787885588.90 total 90.07 last 90.29) - open/close 8134 (time 1787885598.91 total 100.08 last 85.44) - open/close 8965 (time 1787885608.92 total 110.08 last 83.05) - open/close 9714 (time 1787885618.94 total 120.11 last 74.74) total: 10000 open/close in 123.90 seconds: 80.71 ops/second - unlinked 0 (time 1787885627 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 760 (time 1787885710.66 total 10.00 last 75.99) - open/close 1536 (time 1787885720.66 total 20.01 last 77.57) - open/close 2320 (time 1787885730.67 total 30.01 last 78.32) - open/close 3106 (time 1787885740.68 total 40.02 last 78.56) - open/close 4004 (time 1787885750.68 total 50.02 last 89.77) - open/close 4811 (time 1787885760.69 total 60.03 last 80.66) - open/close 5568 (time 1787885770.70 total 70.04 last 75.63) - open/close 6334 (time 1787885780.70 total 80.04 last 76.59) - open/close 7102 (time 1787885790.72 total 90.06 last 76.61) - open/close 7802 (time 1787885800.74 total 100.08 last 69.86) - open/close 8389 (time 1787885810.75 total 110.09 last 58.67) - open/close 9090 (time 1787885820.77 total 120.11 last 69.96) - open/close 9802 (time 1787885830.78 total 130.12 last 71.08) total: 10000 open/close in 133.52 seconds: 74.89 ops/second - unlinked 0 (time 1787885839 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 815 (time 1787885924.64 total 10.01 last 81.41) - open/close 1513 (time 1787885934.65 total 20.01 last 69.78) - open/close 2392 (time 1787885944.65 total 30.02 last 87.89) - open/close 3256 (time 1787885954.66 total 40.03 last 86.31) - open/close 4050 (time 1787885964.66 total 50.03 last 79.39) - open/close 4890 (time 1787885974.66 total 60.03 last 83.96) - open/close 5618 (time 1787885984.67 total 70.03 last 72.78) - open/close 6529 (time 1787885994.67 total 80.04 last 91.06) - open/close 7337 (time 1787886004.68 total 90.05 last 80.74) - open/close 8102 (time 1787886014.69 total 100.06 last 76.41) - open/close 8939 (time 1787886024.70 total 110.07 last 83.63) - open/close 9842 (time 1787886034.71 total 120.08 last 90.20) total: 10000 open/close in 122.24 seconds: 81.80 ops/second - unlinked 0 (time 1787886041 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 632 (time 1787886126.51 total 10.01 last 63.12) - open/close 1364 (time 1787886136.52 total 20.02 last 73.12) - open/close 2082 (time 1787886146.53 total 30.03 last 71.75) - open/close 2753 (time 1787886156.54 total 40.04 last 67.03) - open/close 3383 (time 1787886166.54 total 50.04 last 62.99) - open/close 4166 (time 1787886176.54 total 60.04 last 78.29) - open/close 4898 (time 1787886186.55 total 70.05 last 73.18) - open/close 5559 (time 1787886196.57 total 80.07 last 65.97) - open/close 6351 (time 1787886206.57 total 90.07 last 79.14) - open/close 7080 (time 1787886216.58 total 100.08 last 72.84) - open/close 7830 (time 1787886226.60 total 110.10 last 74.89) - open/close 8552 (time 1787886236.60 total 120.10 last 72.17) - open/close 9241 (time 1787886246.61 total 130.11 last 68.84) total: 10000 open/close in 139.82 seconds: 71.52 ops/second - unlinked 0 (time 1787886260 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 670 (time 1787886357.03 total 10.00 last 67.00) - open/close 1349 (time 1787886367.03 total 20.00 last 67.89) - open/close 2114 (time 1787886377.04 total 30.01 last 76.41) - open/close 2826 (time 1787886387.04 total 40.01 last 71.19) - open/close 3661 (time 1787886397.05 total 50.03 last 83.41) - open/close 4417 (time 1787886407.06 total 60.03 last 75.55) - open/close 5214 (time 1787886417.07 total 70.04 last 79.63) - open/close 5916 (time 1787886427.08 total 80.05 last 70.16) - open/close 6500 (time 1787886437.08 total 90.05 last 58.37) - open/close 7041 (time 1787886447.10 total 100.07 last 54.00) - open/close 7690 (time 1787886457.11 total 110.08 last 64.85) - open/close 8526 (time 1787886467.11 total 120.08 last 83.55) - open/close 9279 (time 1787886477.12 total 130.09 last 75.24) total: 10000 open/close in 139.28 seconds: 71.80 ops/second - unlinked 0 (time 1787886490 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 671 (time 1787886575.94 total 10.03 last 66.90) - open/close 1382 (time 1787886585.95 total 20.04 last 71.05) - open/close 2233 (time 1787886595.95 total 30.04 last 85.07) - open/close 3021 (time 1787886605.96 total 40.06 last 78.67) - open/close 3798 (time 1787886615.97 total 50.07 last 77.63) - open/close 4477 (time 1787886625.98 total 60.07 last 67.86) - open/close 5269 (time 1787886635.99 total 70.08 last 79.14) - open/close 6009 (time 1787886646.00 total 80.09 last 73.90) - open/close 6670 (time 1787886656.01 total 90.10 last 66.03) - open/close 7390 (time 1787886666.02 total 100.11 last 71.94) - open/close 8176 (time 1787886676.04 total 110.14 last 78.40) - open/close 8847 (time 1787886686.08 total 120.17 last 66.88) - open/close 9533 (time 1787886696.09 total 130.18 last 68.52) total: 10000 open/close in 138.93 seconds: 71.98 ops/second - unlinked 0 (time 1787886709 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 770 (time 1787886800.30 total 10.02 last 76.83) - open/close 1600 (time 1787886810.30 total 20.03 last 82.97) - open/close 2440 (time 1787886820.31 total 30.03 last 83.92) - open/close 3262 (time 1787886830.32 total 40.04 last 82.16) - open/close 4086 (time 1787886840.33 total 50.05 last 82.30) - open/close 4909 (time 1787886850.33 total 60.05 last 82.28) - open/close 5703 (time 1787886860.33 total 70.06 last 79.37) - open/close 6461 (time 1787886870.34 total 80.06 last 75.77) - open/close 7278 (time 1787886880.36 total 90.08 last 81.51) - open/close 8037 (time 1787886890.36 total 100.09 last 75.87) - open/close 8777 (time 1787886900.37 total 110.10 last 73.94) - open/close 9612 (time 1787886910.38 total 120.10 last 83.42) total: 10000 open/close in 124.45 seconds: 80.35 ops/second - unlinked 0 (time 1787886919 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg645-server: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg645-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg645-server: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg645-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.145@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.145@tcp sys.timeout=20 start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg645-server: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg645-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg645-server: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg645-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg645-server unloading modules via unload_modules_local on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing unload_modules_local oleg645-server: kmemleak disabled modules unloaded. pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2