== conf-sanity test 69: replace an OST with the same index ========================================================== 16:57:30 (1788901050) start mds service on oleg141-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1952240 Sep 8 13:38 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: kmemleak disabled oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: MODOPTS_LIBCFS= oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg141-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg141-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg141-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg141-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 4265 (time 1788901107.23 total 10.00 last 426.44) - open/close 8109 (time 1788901117.23 total 20.00 last 384.38) total: 10000 open/close in 24.99 seconds: 400.08 ops/second - unlinked 0 (time 1788901123 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 829 (time 1788901174.13 total 10.00 last 82.87) - open/close 1568 (time 1788901184.16 total 20.03 last 73.69) - open/close 2201 (time 1788901194.16 total 30.03 last 63.29) - open/close 3398 (time 1788901204.17 total 40.04 last 119.57) - open/close 4650 (time 1788901214.17 total 50.05 last 125.19) - open/close 5888 (time 1788901224.18 total 60.05 last 123.74) - open/close 7217 (time 1788901234.18 total 70.05 last 132.88) - open/close 8554 (time 1788901244.18 total 80.06 last 133.64) - open/close 9743 (time 1788901254.18 total 90.06 last 118.88) total: 10000 open/close in 92.08 seconds: 108.61 ops/second - unlinked 0 (time 1788901258 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1327 (time 1788901313.33 total 10.00 last 132.67) - open/close 2485 (time 1788901323.33 total 20.01 last 115.73) - open/close 3686 (time 1788901333.34 total 30.01 last 120.06) - open/close 4951 (time 1788901343.34 total 40.01 last 126.49) - open/close 6049 (time 1788901353.35 total 50.03 last 109.66) - open/close 7176 (time 1788901363.36 total 60.04 last 112.58) - open/close 8383 (time 1788901373.36 total 70.04 last 120.67) - open/close 9620 (time 1788901383.38 total 80.05 last 123.54) total: 10000 open/close in 83.60 seconds: 119.62 ops/second - unlinked 0 (time 1788901390 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1008 (time 1788901447.13 total 10.00 last 100.77) - open/close 2206 (time 1788901457.14 total 20.01 last 119.66) - open/close 3306 (time 1788901467.15 total 30.02 last 109.92) - open/close 4575 (time 1788901477.15 total 40.03 last 126.84) - open/close 5808 (time 1788901487.16 total 50.03 last 123.25) - open/close 7172 (time 1788901497.16 total 60.03 last 136.35) - open/close 8458 (time 1788901507.17 total 70.04 last 128.52) - open/close 9588 (time 1788901517.17 total 80.05 last 112.94) total: 10000 open/close in 83.34 seconds: 119.98 ops/second - unlinked 0 (time 1788901523 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1231 (time 1788901577.94 total 10.01 last 123.02) - open/close 2460 (time 1788901587.94 total 20.01 last 122.88) - open/close 3635 (time 1788901597.95 total 30.01 last 117.44) - open/close 4934 (time 1788901607.95 total 40.01 last 129.88) - open/close 6154 (time 1788901617.96 total 50.02 last 121.91) - open/close 7384 (time 1788901627.96 total 60.03 last 122.92) - open/close 8725 (time 1788901637.97 total 70.03 last 134.02) total: 10000 open/close in 78.33 seconds: 127.67 ops/second - unlinked 0 (time 1788901648 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1320 (time 1788901695.10 total 10.00 last 131.98) - open/close 2853 (time 1788901705.10 total 20.01 last 153.24) - open/close 4338 (time 1788901715.11 total 30.01 last 148.43) - open/close 5816 (time 1788901725.11 total 40.01 last 147.75) - open/close 7298 (time 1788901735.11 total 50.01 last 148.18) - open/close 8755 (time 1788901745.11 total 60.01 last 145.70) total: 10000 open/close in 68.70 seconds: 145.56 ops/second - unlinked 0 (time 1788901755 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1888 (time 1788901801.67 total 10.01 last 188.69) - open/close 3558 (time 1788901811.68 total 20.01 last 166.92) - open/close 5061 (time 1788901821.69 total 30.02 last 150.21) - open/close 6552 (time 1788901831.69 total 40.02 last 149.06) - open/close 7976 (time 1788901841.69 total 50.02 last 142.37) - open/close 9408 (time 1788901851.69 total 60.02 last 143.18) total: 10000 open/close in 63.73 seconds: 156.92 ops/second - unlinked 0 (time 1788901857 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1892 (time 1788901901.72 total 10.00 last 189.19) - open/close 3567 (time 1788901911.73 total 20.00 last 167.46) - open/close 5188 (time 1788901921.73 total 30.01 last 162.04) - open/close 6807 (time 1788901931.73 total 40.01 last 161.85) - open/close 8224 (time 1788901941.73 total 50.01 last 141.69) - open/close 9798 (time 1788901951.74 total 60.01 last 157.37) total: 10000 open/close in 61.35 seconds: 163.00 ops/second - unlinked 0 (time 1788901955 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1762 (time 1788901996.86 total 10.00 last 176.17) - open/close 3519 (time 1788902006.86 total 20.00 last 175.68) - open/close 5214 (time 1788902016.86 total 30.00 last 169.49) - open/close 6804 (time 1788902026.86 total 40.01 last 158.92) - open/close 8227 (time 1788902036.87 total 50.01 last 142.24) - open/close 9748 (time 1788902046.87 total 60.02 last 152.04) total: 10000 open/close in 61.29 seconds: 163.16 ops/second - unlinked 0 (time 1788902049 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1528 (time 1788902096.35 total 10.00 last 152.76) - open/close 3060 (time 1788902106.35 total 20.00 last 153.20) - open/close 4641 (time 1788902116.35 total 30.00 last 158.09) - open/close 7587 (time 1788902126.35 total 40.00 last 294.58) total: 10000 open/close in 44.84 seconds: 223.04 ops/second - unlinked 0 (time 1788902132 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg141-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg141-server: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg141-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg141-server: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg141-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre On OST0, 16450 used inodes rc=0 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.141@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.141@tcp sys.timeout=20 start ost1 service on oleg141-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg141-server: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg141-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg141-server: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg141-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg141-server unloading modules via unload_modules_local on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local oleg141-server: kmemleak disabled modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2