== conf-sanity test 69: replace an OST with the same index ========================================================== 04:13:47 (1788423227) start mds service on oleg618-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921464 Sep 3 03:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg618-server' oleg618-server: oleg618-server.virtnet: executing load_modules_local oleg618-server: kmemleak disabled oleg618-server: Loading modules from /home/green/git/lustre-release/lustre oleg618-server: detected 4 online CPUs by sysfs oleg618-server: MODOPTS_LIBCFS= oleg618-server: Force libcfs to create 2 CPU partitions oleg618-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg618-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg618-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg618-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg618-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg618-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-OST0000 oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg618-client.virtnet: -o user_xattr,flock 192.168.206.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 944 (time 1788423338.68 total 10.01 last 94.32) - open/close 1875 (time 1788423348.68 total 20.01 last 93.09) - open/close 2881 (time 1788423358.69 total 30.02 last 100.47) - open/close 3838 (time 1788423368.71 total 40.04 last 95.55) - open/close 4804 (time 1788423378.72 total 50.05 last 96.51) - open/close 5762 (time 1788423388.72 total 60.06 last 95.73) - open/close 6729 (time 1788423398.73 total 70.06 last 96.64) - open/close 7640 (time 1788423408.74 total 80.07 last 91.04) - open/close 8526 (time 1788423418.74 total 90.07 last 88.58) - open/close 9382 (time 1788423428.75 total 100.08 last 85.52) total: 10000 open/close in 108.32 seconds: 92.32 ops/second - unlinked 0 (time 1788423442 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 860 (time 1788423522.88 total 10.00 last 85.98) - open/close 1724 (time 1788423532.89 total 20.01 last 86.36) - open/close 2624 (time 1788423542.89 total 30.01 last 89.99) - open/close 3587 (time 1788423552.89 total 40.01 last 96.25) - open/close 4604 (time 1788423562.90 total 50.02 last 101.68) - open/close 5532 (time 1788423572.91 total 60.03 last 92.66) - open/close 6429 (time 1788423582.91 total 70.03 last 89.68) - open/close 7302 (time 1788423592.91 total 80.03 last 87.30) - open/close 8262 (time 1788423602.92 total 90.04 last 95.95) - open/close 9199 (time 1788423612.93 total 100.05 last 93.59) total: 10000 open/close in 109.08 seconds: 91.67 ops/second - unlinked 0 (time 1788423626 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 676 (time 1788423710.13 total 10.01 last 67.55) - open/close 1347 (time 1788423720.13 total 20.01 last 67.10) - open/close 2240 (time 1788423730.13 total 30.01 last 89.27) - open/close 3091 (time 1788423740.13 total 40.01 last 85.08) - open/close 3961 (time 1788423750.13 total 50.02 last 86.99) - open/close 4942 (time 1788423760.14 total 60.03 last 97.99) - open/close 5868 (time 1788423770.15 total 70.03 last 92.55) - open/close 6759 (time 1788423780.15 total 80.03 last 89.08) - open/close 7562 (time 1788423790.18 total 90.06 last 80.07) - open/close 8235 (time 1788423800.19 total 100.07 last 67.27) - open/close 9022 (time 1788423810.19 total 110.07 last 78.65) - open/close 9978 (time 1788423820.20 total 120.08 last 95.53) total: 10000 open/close in 120.24 seconds: 83.16 ops/second - unlinked 0 (time 1788423824 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 643 (time 1788423909.32 total 10.00 last 64.29) - open/close 1257 (time 1788423919.34 total 20.02 last 61.26) - open/close 1875 (time 1788423929.34 total 30.03 last 61.79) - open/close 2657 (time 1788423939.35 total 40.03 last 78.16) - open/close 3396 (time 1788423949.36 total 50.04 last 73.84) - open/close 4183 (time 1788423959.36 total 60.04 last 78.69) - open/close 4974 (time 1788423969.36 total 70.04 last 79.07) - open/close 5788 (time 1788423979.37 total 80.05 last 81.34) - open/close 6565 (time 1788423989.37 total 90.06 last 77.66) - open/close 7425 (time 1788423999.39 total 100.07 last 85.86) - open/close 8448 (time 1788424009.40 total 110.08 last 102.24) - open/close 9330 (time 1788424019.40 total 120.08 last 88.17) total: 10000 open/close in 127.26 seconds: 78.58 ops/second - unlinked 0 (time 1788424031 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 969 (time 1788424114.48 total 10.01 last 96.82) - open/close 1888 (time 1788424124.49 total 20.02 last 91.83) - open/close 2931 (time 1788424134.49 total 30.02 last 104.21) - open/close 3762 (time 1788424144.50 total 40.03 last 83.05) - open/close 4601 (time 1788424154.51 total 50.04 last 83.82) - open/close 5316 (time 1788424164.52 total 60.05 last 71.46) - open/close 6148 (time 1788424174.52 total 70.05 last 83.14) - open/close 7006 (time 1788424184.52 total 80.05 last 85.79) - open/close 7871 (time 1788424194.53 total 90.06 last 86.45) - open/close 8742 (time 1788424204.54 total 100.07 last 87.04) - open/close 9547 (time 1788424214.55 total 110.08 last 80.36) total: 10000 open/close in 117.23 seconds: 85.30 ops/second - unlinked 0 (time 1788424227 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 785 (time 1788424309.68 total 10.01 last 78.39) - open/close 1449 (time 1788424319.70 total 20.03 last 66.28) - open/close 2179 (time 1788424329.70 total 30.04 last 72.97) - open/close 2928 (time 1788424339.73 total 40.07 last 74.64) - open/close 3604 (time 1788424349.74 total 50.08 last 67.58) - open/close 4528 (time 1788424359.74 total 60.08 last 92.38) - open/close 5537 (time 1788424369.76 total 70.09 last 100.73) - open/close 6485 (time 1788424379.76 total 80.09 last 94.79) - open/close 7304 (time 1788424389.78 total 90.11 last 81.76) - open/close 7928 (time 1788424399.78 total 100.11 last 62.39) - open/close 8584 (time 1788424409.78 total 110.12 last 65.58) - open/close 9279 (time 1788424419.78 total 120.12 last 69.47) total: 10000 open/close in 128.45 seconds: 77.85 ops/second - unlinked 0 (time 1788424432 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 848 (time 1788424507.68 total 10.00 last 84.79) - open/close 1762 (time 1788424517.69 total 20.01 last 91.34) - open/close 2661 (time 1788424527.69 total 30.01 last 89.85) - open/close 3451 (time 1788424537.69 total 40.02 last 78.98) - open/close 4451 (time 1788424547.70 total 50.02 last 99.93) - open/close 5394 (time 1788424557.71 total 60.03 last 94.25) - open/close 6382 (time 1788424567.71 total 70.03 last 98.80) - open/close 7329 (time 1788424577.71 total 80.03 last 94.66) - open/close 8305 (time 1788424587.71 total 90.04 last 97.58) - open/close 9201 (time 1788424597.72 total 100.04 last 89.57) total: 10000 open/close in 108.54 seconds: 92.13 ops/second - unlinked 0 (time 1788424610 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 790 (time 1788424684.09 total 10.02 last 78.86) - open/close 1623 (time 1788424694.10 total 20.03 last 83.22) - open/close 2467 (time 1788424704.11 total 30.03 last 84.38) - open/close 3245 (time 1788424714.13 total 40.05 last 77.66) - open/close 4178 (time 1788424724.14 total 50.06 last 93.17) - open/close 5113 (time 1788424734.15 total 60.07 last 93.45) - open/close 6039 (time 1788424744.15 total 70.07 last 92.54) - open/close 6976 (time 1788424754.15 total 80.08 last 93.67) - open/close 7975 (time 1788424764.16 total 90.08 last 99.88) - open/close 9042 (time 1788424774.17 total 100.09 last 106.59) total: 10000 open/close in 109.60 seconds: 91.24 ops/second - unlinked 0 (time 1788424787 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 984 (time 1788424860.62 total 10.00 last 98.38) - open/close 2008 (time 1788424870.63 total 20.01 last 102.29) - open/close 2994 (time 1788424880.63 total 30.01 last 98.59) - open/close 3981 (time 1788424890.64 total 40.02 last 98.65) - open/close 4864 (time 1788424900.65 total 50.04 last 88.16) - open/close 5871 (time 1788424910.66 total 60.04 last 100.68) - open/close 6751 (time 1788424920.66 total 70.04 last 88.00) - open/close 7744 (time 1788424930.67 total 80.05 last 99.16) - open/close 8705 (time 1788424940.67 total 90.05 last 96.10) - open/close 9563 (time 1788424950.68 total 100.06 last 85.73) total: 10000 open/close in 104.62 seconds: 95.58 ops/second - unlinked 0 (time 1788424959 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1070 (time 1788425038.45 total 10.01 last 106.93) - open/close 2016 (time 1788425048.46 total 20.01 last 94.56) - open/close 2994 (time 1788425058.47 total 30.02 last 97.68) - open/close 4024 (time 1788425068.47 total 40.02 last 102.99) - open/close 4869 (time 1788425078.49 total 50.05 last 84.31) - open/close 5853 (time 1788425088.50 total 60.05 last 98.34) - open/close 6886 (time 1788425098.51 total 70.06 last 103.23) - open/close 7797 (time 1788425108.51 total 80.06 last 91.06) - open/close 8664 (time 1788425118.51 total 90.07 last 86.67) - open/close 9465 (time 1788425128.52 total 100.07 last 80.08) total: 10000 open/close in 107.88 seconds: 92.70 ops/second - unlinked 0 (time 1788425141 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg618-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg618-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg618-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg618-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-OST0000 oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg618-server: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg618-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg618-server: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg618-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg618-client.virtnet: -o user_xattr,flock 192.168.206.118@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg618-client: oleg618-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg618-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg618-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg618-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.118@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.118@tcp sys.timeout=20 start ost1 service on oleg618-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-OST0000 oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg618-server: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg618-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg618-server: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg618-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg618-client.virtnet: -o user_xattr,flock 192.168.206.118@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg618-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg618-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg618-server stop mds service on oleg618-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg618-server stop mds service on oleg618-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg618-server unloading modules via unload_modules_local on: 'oleg618-server' oleg618-server: oleg618-server.virtnet: executing unload_modules_local oleg618-server: kmemleak disabled modules unloaded. pdsh@oleg618-client: oleg618-server: ssh exited with exit code 2 pdsh@oleg618-client: oleg618-client: ssh exited with exit code 2 pdsh@oleg618-client: oleg618-client: ssh exited with exit code 2 pdsh@oleg618-client: oleg618-server: ssh exited with exit code 2