== conf-sanity test 69: replace an OST with the same index ========================================================== 16:55:38 (1788814538) start mds service on oleg143-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931936 Sep 7 16:07 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: kmemleak disabled oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: MODOPTS_LIBCFS= oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg143-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg143-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg143-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 941 (time 1788814630.93 total 10.01 last 94.04) - open/close 1916 (time 1788814640.93 total 20.01 last 97.45) - open/close 2816 (time 1788814650.94 total 30.02 last 89.93) - open/close 3753 (time 1788814660.95 total 40.03 last 93.62) - open/close 4694 (time 1788814670.95 total 50.03 last 94.10) - open/close 5637 (time 1788814680.95 total 60.03 last 94.30) - open/close 6558 (time 1788814690.95 total 70.03 last 92.06) - open/close 7548 (time 1788814700.96 total 80.04 last 98.90) - open/close 8436 (time 1788814710.99 total 90.07 last 88.59) - open/close 9365 (time 1788814721.00 total 100.08 last 92.77) total: 10000 open/close in 107.19 seconds: 93.30 ops/second - unlinked 0 (time 1788814731 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 920 (time 1788814804.17 total 10.01 last 91.95) - open/close 2054 (time 1788814814.19 total 20.02 last 113.25) - open/close 3062 (time 1788814824.19 total 30.02 last 100.77) - open/close 4094 (time 1788814834.19 total 40.02 last 103.17) - open/close 5142 (time 1788814844.20 total 50.03 last 104.69) - open/close 6197 (time 1788814854.20 total 60.03 last 105.50) - open/close 7134 (time 1788814864.21 total 70.05 last 93.60) - open/close 8093 (time 1788814874.23 total 80.06 last 95.75) - open/close 9002 (time 1788814884.23 total 90.06 last 90.86) - open/close 9956 (time 1788814894.24 total 100.08 last 95.29) total: 10000 open/close in 100.56 seconds: 99.44 ops/second - unlinked 0 (time 1788814897 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 966 (time 1788814964.97 total 10.01 last 96.52) - open/close 1954 (time 1788814974.97 total 20.01 last 98.79) - open/close 2834 (time 1788814984.97 total 30.01 last 87.98) - open/close 3787 (time 1788814994.98 total 40.02 last 95.22) - open/close 4664 (time 1788815004.98 total 50.02 last 87.69) - open/close 5691 (time 1788815014.99 total 60.02 last 102.68) - open/close 6610 (time 1788815024.99 total 70.03 last 91.88) - open/close 7642 (time 1788815034.99 total 80.03 last 103.18) - open/close 8681 (time 1788815045.00 total 90.04 last 103.79) - open/close 9694 (time 1788815055.01 total 100.04 last 101.25) total: 10000 open/close in 103.25 seconds: 96.85 ops/second - unlinked 0 (time 1788815061 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 914 (time 1788815128.51 total 10.00 last 91.38) - open/close 1864 (time 1788815138.51 total 20.01 last 94.97) - open/close 2794 (time 1788815148.51 total 30.01 last 92.99) - open/close 3805 (time 1788815158.52 total 40.02 last 101.00) - open/close 4815 (time 1788815168.53 total 50.02 last 100.91) - open/close 5796 (time 1788815178.53 total 60.03 last 98.09) - open/close 6907 (time 1788815188.53 total 70.03 last 111.09) - open/close 7892 (time 1788815198.54 total 80.03 last 98.44) - open/close 8894 (time 1788815208.55 total 90.05 last 100.06) - open/close 9967 (time 1788815218.56 total 100.05 last 107.25) total: 10000 open/close in 100.40 seconds: 99.60 ops/second - unlinked 0 (time 1788815221 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1092 (time 1788815286.78 total 10.00 last 109.17) - open/close 2153 (time 1788815296.79 total 20.01 last 105.99) - open/close 3182 (time 1788815306.80 total 30.02 last 102.83) - open/close 4310 (time 1788815316.80 total 40.02 last 112.79) - open/close 5395 (time 1788815326.81 total 50.03 last 108.44) - open/close 6518 (time 1788815336.82 total 60.04 last 112.13) - open/close 7505 (time 1788815346.83 total 70.04 last 98.69) - open/close 8416 (time 1788815356.83 total 80.05 last 91.04) - open/close 9595 (time 1788815366.84 total 90.06 last 117.79) total: 10000 open/close in 93.50 seconds: 106.96 ops/second - unlinked 0 (time 1788815372 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1199 (time 1788815430.16 total 10.00 last 119.87) - open/close 2479 (time 1788815440.16 total 20.01 last 127.91) - open/close 3661 (time 1788815450.16 total 30.01 last 118.17) - open/close 4835 (time 1788815460.17 total 40.01 last 117.39) - open/close 6046 (time 1788815470.17 total 50.02 last 121.03) - open/close 7248 (time 1788815480.18 total 60.02 last 120.15) - open/close 8509 (time 1788815490.18 total 70.02 last 126.10) - open/close 9659 (time 1788815500.18 total 80.02 last 114.99) total: 10000 open/close in 82.98 seconds: 120.51 ops/second - unlinked 0 (time 1788815505 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1392 (time 1788815561.35 total 10.00 last 139.20) - open/close 2754 (time 1788815571.36 total 20.01 last 136.10) - open/close 4197 (time 1788815581.36 total 30.01 last 144.25) - open/close 5511 (time 1788815591.37 total 40.02 last 131.35) - open/close 7085 (time 1788815601.37 total 50.02 last 157.35) - open/close 8447 (time 1788815611.38 total 60.03 last 136.05) - open/close 9943 (time 1788815621.38 total 70.03 last 149.58) total: 10000 open/close in 70.35 seconds: 142.14 ops/second - unlinked 0 (time 1788815623 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1337 (time 1788815677.61 total 10.00 last 133.65) - open/close 2551 (time 1788815687.62 total 20.01 last 121.35) - open/close 3807 (time 1788815697.62 total 30.01 last 125.52) - open/close 5051 (time 1788815707.63 total 40.02 last 124.31) - open/close 6282 (time 1788815717.64 total 50.03 last 123.04) - open/close 7479 (time 1788815727.64 total 60.03 last 119.69) - open/close 8675 (time 1788815737.64 total 70.03 last 119.56) - open/close 9966 (time 1788815747.65 total 80.04 last 129.03) total: 10000 open/close in 80.25 seconds: 124.61 ops/second - unlinked 0 (time 1788815750 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1235 (time 1788815808.48 total 10.01 last 123.37) - open/close 2454 (time 1788815818.48 total 20.01 last 121.89) - open/close 3795 (time 1788815828.48 total 30.01 last 134.10) - open/close 5080 (time 1788815838.48 total 40.01 last 128.49) - open/close 6395 (time 1788815848.48 total 50.01 last 131.46) - open/close 7670 (time 1788815858.50 total 60.03 last 127.37) - open/close 8897 (time 1788815868.50 total 70.03 last 122.62) total: 10000 open/close in 78.89 seconds: 126.76 ops/second - unlinked 0 (time 1788815879 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1164 (time 1788815933.08 total 10.00 last 116.35) - open/close 2332 (time 1788815943.08 total 20.01 last 116.78) - open/close 3556 (time 1788815953.08 total 30.01 last 122.39) - open/close 4307 (time 1788815963.08 total 40.01 last 75.10) - open/close 4857 (time 1788815973.10 total 50.02 last 54.93) - open/close 5339 (time 1788815983.12 total 60.05 last 48.07) - open/close 5788 (time 1788815993.13 total 70.05 last 44.88) - open/close 6231 (time 1788816003.13 total 80.06 last 44.27) - open/close 6712 (time 1788816013.14 total 90.06 last 48.08) - open/close 7461 (time 1788816023.15 total 100.07 last 74.83) - open/close 8032 (time 1788816033.16 total 110.09 last 57.03) - open/close 8773 (time 1788816043.17 total 120.10 last 74.01) - open/close 9594 (time 1788816053.19 total 130.11 last 81.99) total: 10000 open/close in 134.71 seconds: 74.23 ops/second - unlinked 0 (time 1788816062 ; total 1 ; last 1) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg143-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.143@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.143@tcp sys.timeout=20 start ost1 service on oleg143-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server unloading modules via unload_modules_local on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local oleg143-server: kmemleak disabled modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2