== conf-sanity test 69: replace an OST with the same index ========================================================== 19:51:18 (1789602678) start mds service on oleg146-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 16 12:13 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: kmemleak disabled oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: detected 4 online CPUs by sysfs oleg146-server: MODOPTS_LIBCFS= oleg146-server: Force libcfs to create 2 CPU partitions oleg146-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg146-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg146-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg146-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg146-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg146-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 727 (time 1789602792.05 total 10.01 last 72.66) - open/close 1475 (time 1789602802.07 total 20.02 last 74.72) - open/close 2267 (time 1789602812.08 total 30.03 last 79.11) - open/close 3028 (time 1789602822.08 total 40.04 last 76.04) - open/close 3769 (time 1789602832.09 total 50.04 last 74.07) - open/close 4551 (time 1789602842.09 total 60.04 last 78.19) - open/close 5252 (time 1789602852.10 total 70.05 last 70.01) - open/close 6135 (time 1789602862.11 total 80.06 last 88.25) - open/close 6952 (time 1789602872.12 total 90.07 last 81.65) - open/close 7735 (time 1789602882.13 total 100.08 last 78.20) - open/close 8536 (time 1789602892.13 total 110.09 last 80.05) - open/close 9296 (time 1789602902.14 total 120.09 last 75.99) - open/close 9856 (time 1789602912.15 total 130.10 last 55.93) total: 10000 open/close in 132.42 seconds: 75.52 ops/second - unlinked 0 (time 1789602919 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 745 (time 1789603006.52 total 10.01 last 74.44) - open/close 1511 (time 1789603016.54 total 20.03 last 76.45) - open/close 2293 (time 1789603026.56 total 30.04 last 78.12) - open/close 2999 (time 1789603036.56 total 40.04 last 70.58) - open/close 3786 (time 1789603046.56 total 50.04 last 78.67) - open/close 4555 (time 1789603056.57 total 60.05 last 76.86) - open/close 5377 (time 1789603066.57 total 70.05 last 82.17) - open/close 6165 (time 1789603076.59 total 80.07 last 78.68) - open/close 7007 (time 1789603086.59 total 90.07 last 84.16) - open/close 8013 (time 1789603096.59 total 100.08 last 100.58) - open/close 8843 (time 1789603106.60 total 110.08 last 82.93) - open/close 9663 (time 1789603116.61 total 120.09 last 81.96) total: 10000 open/close in 124.62 seconds: 80.25 ops/second - unlinked 0 (time 1789603125 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 583 (time 1789603212.15 total 10.00 last 58.28) - open/close 1431 (time 1789603222.16 total 20.01 last 84.72) - open/close 2194 (time 1789603232.16 total 30.02 last 76.25) - open/close 3011 (time 1789603242.17 total 40.03 last 81.62) - open/close 3693 (time 1789603252.18 total 50.04 last 68.15) - open/close 4489 (time 1789603262.20 total 60.06 last 79.45) - open/close 5265 (time 1789603272.21 total 70.07 last 77.52) - open/close 5991 (time 1789603282.22 total 80.08 last 72.53) - open/close 6879 (time 1789603292.23 total 90.08 last 88.75) - open/close 7702 (time 1789603302.23 total 100.09 last 82.23) - open/close 8410 (time 1789603312.24 total 110.09 last 70.78) - open/close 9228 (time 1789603322.25 total 120.10 last 81.70) - open/close 9882 (time 1789603332.26 total 130.12 last 65.32) total: 10000 open/close in 131.68 seconds: 75.94 ops/second - unlinked 0 (time 1789603339 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 848 (time 1789603434.20 total 10.01 last 84.73) - open/close 1559 (time 1789603444.21 total 20.01 last 71.09) - open/close 2476 (time 1789603454.22 total 30.02 last 91.58) - open/close 3306 (time 1789603464.22 total 40.02 last 82.99) - open/close 4075 (time 1789603474.24 total 50.04 last 76.74) - open/close 4881 (time 1789603484.24 total 60.04 last 80.59) - open/close 5692 (time 1789603494.25 total 70.05 last 81.05) - open/close 6530 (time 1789603504.26 total 80.07 last 83.68) - open/close 7280 (time 1789603514.27 total 90.07 last 74.97) - open/close 8099 (time 1789603524.28 total 100.08 last 81.82) - open/close 8878 (time 1789603534.28 total 110.09 last 77.85) - open/close 9737 (time 1789603544.29 total 120.10 last 85.82) total: 10000 open/close in 123.10 seconds: 81.24 ops/second - unlinked 0 (time 1789603551 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 741 (time 1789603643.75 total 10.01 last 74.06) - open/close 1579 (time 1789603653.76 total 20.01 last 83.75) - open/close 2487 (time 1789603663.78 total 30.03 last 90.61) - open/close 3153 (time 1789603673.78 total 40.03 last 66.59) - open/close 3803 (time 1789603683.79 total 50.05 last 64.92) - open/close 4327 (time 1789603693.80 total 60.05 last 52.37) - open/close 4936 (time 1789603703.80 total 70.05 last 60.88) - open/close 5695 (time 1789603713.81 total 80.06 last 75.83) - open/close 6493 (time 1789603723.82 total 90.07 last 79.74) - open/close 7250 (time 1789603733.84 total 100.10 last 75.50) - open/close 7719 (time 1789603743.85 total 110.10 last 46.89) - open/close 8621 (time 1789603753.85 total 120.11 last 90.12) - open/close 9278 (time 1789603763.85 total 130.11 last 65.70) total: 10000 open/close in 139.78 seconds: 71.54 ops/second - unlinked 0 (time 1789603778 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 928 (time 1789603872.11 total 10.00 last 92.80) - open/close 1833 (time 1789603882.12 total 20.01 last 90.44) - open/close 2614 (time 1789603892.13 total 30.02 last 78.03) - open/close 3339 (time 1789603902.14 total 40.02 last 72.45) - open/close 4002 (time 1789603912.15 total 50.04 last 66.18) - open/close 4502 (time 1789603922.17 total 60.05 last 49.93) - open/close 5080 (time 1789603932.17 total 70.05 last 57.80) - open/close 5734 (time 1789603942.18 total 80.06 last 65.35) - open/close 6419 (time 1789603952.18 total 90.06 last 68.49) - open/close 7180 (time 1789603962.18 total 100.07 last 76.05) - open/close 7829 (time 1789603972.19 total 110.08 last 64.84) - open/close 8722 (time 1789603982.20 total 120.09 last 89.23) - open/close 9571 (time 1789603992.21 total 130.09 last 84.87) total: 10000 open/close in 135.45 seconds: 73.83 ops/second - unlinked 0 (time 1789604002 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 593 (time 1789604089.38 total 10.05 last 59.01) - open/close 1069 (time 1789604099.40 total 20.06 last 47.55) - open/close 1557 (time 1789604109.41 total 30.07 last 48.75) - open/close 2071 (time 1789604119.41 total 40.07 last 51.39) - open/close 2635 (time 1789604129.42 total 50.08 last 56.34) - open/close 3395 (time 1789604139.46 total 60.12 last 75.70) - open/close 4094 (time 1789604149.47 total 70.13 last 69.83) - open/close 4807 (time 1789604159.48 total 80.14 last 71.25) - open/close 5599 (time 1789604169.48 total 90.14 last 79.18) - open/close 6277 (time 1789604179.49 total 100.15 last 67.75) - open/close 7046 (time 1789604189.49 total 110.15 last 76.89) - open/close 7796 (time 1789604199.49 total 120.16 last 74.96) - open/close 8540 (time 1789604209.50 total 130.16 last 74.35) - open/close 9428 (time 1789604219.51 total 140.17 last 88.74) total: 10000 open/close in 148.35 seconds: 67.41 ops/second - unlinked 0 (time 1789604233 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 782 (time 1789604321.70 total 10.01 last 78.14) - open/close 1487 (time 1789604331.72 total 20.02 last 70.39) - open/close 2166 (time 1789604341.73 total 30.03 last 67.86) - open/close 2957 (time 1789604351.73 total 40.03 last 79.08) - open/close 3714 (time 1789604361.73 total 50.03 last 75.68) - open/close 4419 (time 1789604371.74 total 60.04 last 70.46) - open/close 5261 (time 1789604381.74 total 70.04 last 84.18) - open/close 6056 (time 1789604391.75 total 80.05 last 79.47) - open/close 6846 (time 1789604401.75 total 90.05 last 78.96) - open/close 7655 (time 1789604411.76 total 100.06 last 80.84) - open/close 8503 (time 1789604421.76 total 110.06 last 84.78) - open/close 9178 (time 1789604431.78 total 120.08 last 67.37) - open/close 9936 (time 1789604441.79 total 130.09 last 75.75) total: 10000 open/close in 130.88 seconds: 76.41 ops/second - unlinked 0 (time 1789604448 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 790 (time 1789604542.60 total 10.00 last 78.98) - open/close 1643 (time 1789604552.61 total 20.01 last 85.24) - open/close 2486 (time 1789604562.62 total 30.02 last 84.20) - open/close 3315 (time 1789604572.63 total 40.03 last 82.81) - open/close 4060 (time 1789604582.64 total 50.04 last 74.46) - open/close 4758 (time 1789604592.65 total 60.05 last 69.69) - open/close 5594 (time 1789604602.66 total 70.06 last 83.55) - open/close 6303 (time 1789604612.67 total 80.07 last 70.85) - open/close 6934 (time 1789604622.69 total 90.09 last 62.97) - open/close 7512 (time 1789604632.69 total 100.09 last 57.79) - open/close 8237 (time 1789604642.69 total 110.09 last 72.49) - open/close 8852 (time 1789604652.69 total 120.09 last 61.48) - open/close 9545 (time 1789604662.71 total 130.11 last 69.16) total: 10000 open/close in 135.98 seconds: 73.54 ops/second - unlinked 0 (time 1789604673 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 830 (time 1789604760.48 total 10.01 last 82.88) - open/close 1613 (time 1789604770.48 total 20.01 last 78.30) - open/close 2405 (time 1789604780.50 total 30.03 last 79.09) - open/close 3098 (time 1789604790.51 total 40.04 last 69.23) - open/close 3887 (time 1789604800.51 total 50.04 last 78.88) - open/close 4457 (time 1789604810.52 total 60.05 last 56.97) - open/close 5256 (time 1789604820.52 total 70.05 last 79.88) - open/close 6056 (time 1789604830.54 total 80.07 last 79.80) - open/close 6841 (time 1789604840.55 total 90.08 last 78.46) - open/close 7717 (time 1789604850.56 total 100.09 last 87.48) - open/close 8548 (time 1789604860.57 total 110.10 last 83.05) - open/close 9364 (time 1789604870.58 total 120.11 last 81.51) total: 10000 open/close in 128.83 seconds: 77.62 ops/second - unlinked 0 (time 1789604884 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.146@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg146-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg146-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg146-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.146@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.146@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.146@tcp sys.timeout=20 start ost1 service on oleg146-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg146-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg146-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server unloading modules via unload_modules_local on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing unload_modules_local oleg146-server: kmemleak disabled modules unloaded. pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2