== conf-sanity test 69: replace an OST with the same index ========================================================== 00:09:19 (1789445359) start mds service on oleg605-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1933448 Sep 14 13:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg605-server' oleg605-server: oleg605-server.virtnet: executing load_modules_local oleg605-server: kmemleak disabled oleg605-server: Loading modules from /home/green/git/lustre-release/lustre oleg605-server: detected 4 online CPUs by sysfs oleg605-server: MODOPTS_LIBCFS= oleg605-server: Force libcfs to create 2 CPU partitions oleg605-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg605-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg605-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg605-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg605-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg605-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 939 (time 1789445460.89 total 10.00 last 93.86) - open/close 1952 (time 1789445470.90 total 20.01 last 101.26) - open/close 2995 (time 1789445480.90 total 30.01 last 104.30) - open/close 4085 (time 1789445490.90 total 40.02 last 108.91) - open/close 5069 (time 1789445500.91 total 50.02 last 98.34) - open/close 6096 (time 1789445510.92 total 60.03 last 102.63) - open/close 7104 (time 1789445520.92 total 70.04 last 100.73) - open/close 8163 (time 1789445530.93 total 80.04 last 105.87) - open/close 9119 (time 1789445540.94 total 90.06 last 95.43) total: 10000 open/close in 98.61 seconds: 101.41 ops/second - unlinked 0 (time 1789445552 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 932 (time 1789445624.06 total 10.01 last 93.13) - open/close 1755 (time 1789445634.06 total 20.01 last 82.27) - open/close 2675 (time 1789445644.09 total 30.04 last 91.77) - open/close 3695 (time 1789445654.09 total 40.04 last 101.95) - open/close 4664 (time 1789445664.09 total 50.04 last 96.88) - open/close 5568 (time 1789445674.11 total 60.06 last 90.24) - open/close 6569 (time 1789445684.12 total 70.07 last 100.03) - open/close 7580 (time 1789445694.13 total 80.08 last 100.97) - open/close 8598 (time 1789445704.15 total 90.10 last 101.66) - open/close 9586 (time 1789445714.15 total 100.10 last 98.73) total: 10000 open/close in 104.22 seconds: 95.96 ops/second - unlinked 0 (time 1789445721 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 939 (time 1789445791.96 total 10.00 last 93.87) - open/close 1926 (time 1789445801.97 total 20.01 last 98.59) - open/close 2926 (time 1789445811.97 total 30.02 last 99.97) - open/close 3925 (time 1789445821.98 total 40.02 last 99.86) - open/close 4827 (time 1789445831.98 total 50.02 last 90.20) - open/close 5744 (time 1789445841.99 total 60.03 last 91.63) - open/close 6705 (time 1789445851.99 total 70.04 last 96.03) - open/close 7653 (time 1789445862.01 total 80.06 last 94.63) - open/close 8588 (time 1789445872.02 total 90.06 last 93.46) - open/close 9390 (time 1789445882.02 total 100.06 last 80.16) total: 10000 open/close in 106.74 seconds: 93.68 ops/second - unlinked 0 (time 1789445892 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1011 (time 1789445966.05 total 10.00 last 101.07) - open/close 1932 (time 1789445976.06 total 20.01 last 92.07) - open/close 2808 (time 1789445986.06 total 30.01 last 87.54) - open/close 3762 (time 1789445996.07 total 40.02 last 95.30) - open/close 4783 (time 1789446006.07 total 50.02 last 102.10) - open/close 5793 (time 1789446016.08 total 60.03 last 100.95) - open/close 6828 (time 1789446026.08 total 70.03 last 103.44) - open/close 7712 (time 1789446036.09 total 80.04 last 88.37) - open/close 8615 (time 1789446046.10 total 90.05 last 90.23) - open/close 9581 (time 1789446056.10 total 100.05 last 96.57) total: 10000 open/close in 103.75 seconds: 96.39 ops/second - unlinked 0 (time 1789446063 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 801 (time 1789446135.65 total 10.01 last 80.03) - open/close 1771 (time 1789446145.65 total 20.01 last 96.96) - open/close 2875 (time 1789446155.66 total 30.02 last 110.27) - open/close 3925 (time 1789446165.67 total 40.03 last 104.96) - open/close 4932 (time 1789446175.67 total 50.03 last 100.68) - open/close 5865 (time 1789446185.68 total 60.04 last 93.25) - open/close 6660 (time 1789446195.68 total 70.04 last 79.48) - open/close 7468 (time 1789446205.69 total 80.05 last 80.74) - open/close 8438 (time 1789446215.69 total 90.05 last 96.98) - open/close 9387 (time 1789446225.69 total 100.05 last 94.85) total: 10000 open/close in 106.39 seconds: 93.99 ops/second - unlinked 0 (time 1789446235 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 901 (time 1789446310.66 total 10.00 last 90.08) - open/close 1931 (time 1789446320.67 total 20.01 last 102.95) - open/close 2936 (time 1789446330.67 total 30.01 last 100.43) - open/close 3927 (time 1789446340.68 total 40.02 last 99.06) - open/close 4871 (time 1789446350.68 total 50.02 last 94.36) - open/close 5912 (time 1789446360.69 total 60.03 last 104.06) - open/close 6800 (time 1789446370.69 total 70.03 last 88.79) - open/close 7753 (time 1789446380.71 total 80.05 last 95.11) - open/close 8701 (time 1789446390.71 total 90.05 last 94.80) - open/close 9648 (time 1789446400.71 total 100.06 last 94.63) total: 10000 open/close in 103.41 seconds: 96.70 ops/second - unlinked 0 (time 1789446407 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 760 (time 1789446480.62 total 10.01 last 75.95) - open/close 1750 (time 1789446490.63 total 20.01 last 98.93) - open/close 2764 (time 1789446500.63 total 30.02 last 101.38) - open/close 3814 (time 1789446510.63 total 40.02 last 105.00) - open/close 4803 (time 1789446520.63 total 50.02 last 98.89) - open/close 5846 (time 1789446530.64 total 60.02 last 104.28) - open/close 6785 (time 1789446540.64 total 70.02 last 93.88) - open/close 7823 (time 1789446550.64 total 80.02 last 103.75) - open/close 8810 (time 1789446560.65 total 90.04 last 98.59) - open/close 9798 (time 1789446570.66 total 100.04 last 98.76) total: 10000 open/close in 102.06 seconds: 97.98 ops/second - unlinked 0 (time 1789446576 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 960 (time 1789446648.97 total 10.01 last 95.89) - open/close 1947 (time 1789446658.98 total 20.02 last 98.59) - open/close 2935 (time 1789446668.99 total 30.03 last 98.70) - open/close 3880 (time 1789446679.00 total 40.04 last 94.44) - open/close 4806 (time 1789446689.01 total 50.04 last 92.54) - open/close 5705 (time 1789446699.01 total 60.05 last 89.86) - open/close 6698 (time 1789446709.01 total 70.05 last 99.28) - open/close 7677 (time 1789446719.02 total 80.06 last 97.78) - open/close 8631 (time 1789446729.03 total 90.07 last 95.39) - open/close 9597 (time 1789446739.03 total 100.07 last 96.56) total: 10000 open/close in 104.42 seconds: 95.77 ops/second - unlinked 0 (time 1789446747 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1020 (time 1789446819.68 total 10.01 last 101.94) - open/close 1977 (time 1789446829.69 total 20.01 last 95.64) - open/close 3015 (time 1789446839.70 total 30.02 last 103.70) - open/close 4103 (time 1789446849.70 total 40.02 last 108.80) - open/close 5128 (time 1789446859.71 total 50.03 last 102.43) - open/close 6233 (time 1789446869.71 total 60.03 last 110.47) - open/close 7257 (time 1789446879.71 total 70.04 last 102.36) - open/close 8236 (time 1789446889.72 total 80.04 last 97.85) - open/close 9241 (time 1789446899.73 total 90.05 last 100.44) total: 10000 open/close in 97.27 seconds: 102.80 ops/second - unlinked 0 (time 1789446910 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 938 (time 1789446980.74 total 10.00 last 93.80) - open/close 1909 (time 1789446990.74 total 20.00 last 97.09) - open/close 2845 (time 1789447000.75 total 30.01 last 93.55) - open/close 3894 (time 1789447010.75 total 40.01 last 104.87) - open/close 4904 (time 1789447020.76 total 50.02 last 100.94) - open/close 5873 (time 1789447030.76 total 60.02 last 96.88) - open/close 6782 (time 1789447040.76 total 70.02 last 90.88) - open/close 7765 (time 1789447050.77 total 80.03 last 98.21) - open/close 8734 (time 1789447060.77 total 90.03 last 96.88) - open/close 9764 (time 1789447070.78 total 100.04 last 102.92) total: 10000 open/close in 102.22 seconds: 97.83 ops/second - unlinked 0 (time 1789447077 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg605-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg605-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg605-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg605-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg605-client: oleg605-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg605-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg605-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg605-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 start ost1 service on oleg605-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg605-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg605-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg605-server stop mds service on oleg605-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg605-server stop mds service on oleg605-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg605-server unloading modules via unload_modules_local on: 'oleg605-server' oleg605-server: oleg605-server.virtnet: executing unload_modules_local oleg605-server: kmemleak disabled modules unloaded. pdsh@oleg605-client: oleg605-client: ssh exited with exit code 2 pdsh@oleg605-client: oleg605-server: ssh exited with exit code 2 pdsh@oleg605-client: oleg605-client: ssh exited with exit code 2 pdsh@oleg605-client: oleg605-server: ssh exited with exit code 2