== conf-sanity test 69: replace an OST with the same index ========================================================== 00:36:14 (1788842174) start mds service on oleg209-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:40 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: kmemleak disabled oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg209-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg209-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 564 (time 1788842264.83 total 10.01 last 56.36) - open/close 1073 (time 1788842274.83 total 20.01 last 50.86) - open/close 1736 (time 1788842284.85 total 30.03 last 66.22) - open/close 2382 (time 1788842294.85 total 40.03 last 64.60) - open/close 3138 (time 1788842304.85 total 50.04 last 75.54) - open/close 3825 (time 1788842314.86 total 60.04 last 68.67) - open/close 4414 (time 1788842324.86 total 70.04 last 58.87) - open/close 4998 (time 1788842334.87 total 80.05 last 58.35) - open/close 5632 (time 1788842344.89 total 90.07 last 63.30) - open/close 6252 (time 1788842354.89 total 100.07 last 61.98) - open/close 6897 (time 1788842364.92 total 110.10 last 64.34) - open/close 7444 (time 1788842374.93 total 120.11 last 54.61) - open/close 7994 (time 1788842384.95 total 130.13 last 54.88) - open/close 8578 (time 1788842394.96 total 140.14 last 58.39) - open/close 9279 (time 1788842404.96 total 150.14 last 70.09) - open/close 9961 (time 1788842414.97 total 160.15 last 68.12) total: 10000 open/close in 160.64 seconds: 62.25 ops/second - unlinked 0 (time 1788842420 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 714 (time 1788842519.91 total 10.01 last 71.35) - open/close 1438 (time 1788842529.91 total 20.01 last 72.40) - open/close 2124 (time 1788842539.92 total 30.01 last 68.57) - open/close 2794 (time 1788842549.94 total 40.04 last 66.84) - open/close 3555 (time 1788842559.95 total 50.05 last 76.02) - open/close 4280 (time 1788842569.95 total 60.05 last 72.50) - open/close 4962 (time 1788842579.97 total 70.06 last 68.11) - open/close 5716 (time 1788842589.98 total 80.07 last 75.32) - open/close 6554 (time 1788842600.00 total 90.10 last 83.58) - open/close 7288 (time 1788842610.01 total 100.11 last 73.32) - open/close 7995 (time 1788842620.02 total 110.11 last 70.68) - open/close 8743 (time 1788842630.03 total 120.12 last 74.71) - open/close 9417 (time 1788842640.04 total 130.13 last 67.35) total: 10000 open/close in 137.72 seconds: 72.61 ops/second - unlinked 0 (time 1788842652 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 620 (time 1788842754.15 total 10.00 last 62.00) - open/close 1225 (time 1788842764.16 total 20.01 last 60.44) - open/close 1899 (time 1788842774.18 total 30.03 last 67.25) - open/close 2527 (time 1788842784.19 total 40.04 last 62.74) - open/close 3213 (time 1788842794.20 total 50.04 last 68.59) - open/close 3866 (time 1788842804.21 total 60.06 last 65.20) - open/close 4383 (time 1788842814.23 total 70.08 last 51.58) - open/close 5102 (time 1788842824.26 total 80.10 last 71.74) - open/close 5724 (time 1788842834.26 total 90.11 last 62.17) - open/close 6356 (time 1788842844.27 total 100.12 last 63.14) - open/close 7069 (time 1788842854.28 total 110.13 last 71.24) - open/close 7756 (time 1788842864.30 total 120.15 last 68.54) - open/close 8330 (time 1788842874.31 total 130.16 last 57.34) - open/close 9081 (time 1788842884.32 total 140.17 last 75.05) - open/close 9836 (time 1788842894.34 total 150.19 last 75.33) total: 10000 open/close in 152.76 seconds: 65.46 ops/second - unlinked 0 (time 1788842901 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 647 (time 1788843002.75 total 10.01 last 64.66) - open/close 1239 (time 1788843012.75 total 20.01 last 59.15) - open/close 1974 (time 1788843022.77 total 30.03 last 73.35) - open/close 2620 (time 1788843032.78 total 40.04 last 64.55) - open/close 3317 (time 1788843042.80 total 50.06 last 69.56) - open/close 4022 (time 1788843052.81 total 60.07 last 70.41) - open/close 4747 (time 1788843062.82 total 70.08 last 72.48) - open/close 5471 (time 1788843072.82 total 80.09 last 72.34) - open/close 5992 (time 1788843082.84 total 90.10 last 52.04) - open/close 6658 (time 1788843092.84 total 100.10 last 66.58) - open/close 7353 (time 1788843102.84 total 110.10 last 69.49) - open/close 8004 (time 1788843112.84 total 120.10 last 65.09) - open/close 8660 (time 1788843122.85 total 130.11 last 65.53) - open/close 9338 (time 1788843132.86 total 140.12 last 67.77) - open/close 9972 (time 1788843142.87 total 150.13 last 63.34) total: 10000 open/close in 150.69 seconds: 66.36 ops/second - unlinked 0 (time 1788843149 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 714 (time 1788843277.34 total 10.01 last 71.35) - open/close 1392 (time 1788843287.36 total 20.03 last 67.66) - open/close 2107 (time 1788843297.37 total 30.03 last 71.49) - open/close 2724 (time 1788843307.37 total 40.04 last 61.66) - open/close 3389 (time 1788843317.37 total 50.04 last 66.50) - open/close 4003 (time 1788843327.38 total 60.04 last 61.39) - open/close 4625 (time 1788843337.40 total 70.06 last 62.04) - open/close 5175 (time 1788843347.42 total 80.08 last 54.90) - open/close 5771 (time 1788843357.42 total 90.09 last 59.57) - open/close 6556 (time 1788843367.42 total 100.09 last 78.49) - open/close 7133 (time 1788843377.44 total 110.10 last 57.61) - open/close 7800 (time 1788843387.45 total 120.11 last 66.64) - open/close 8537 (time 1788843397.45 total 130.12 last 73.67) - open/close 9203 (time 1788843407.46 total 140.13 last 66.54) - open/close 9693 (time 1788843417.47 total 150.14 last 48.95) total: 10000 open/close in 155.88 seconds: 64.15 ops/second - unlinked 0 (time 1788843427 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 703 (time 1788843542.49 total 10.03 last 70.07) - open/close 1246 (time 1788843552.50 total 20.04 last 54.24) - open/close 1908 (time 1788843562.51 total 30.05 last 66.15) - open/close 2604 (time 1788843572.51 total 40.05 last 69.59) - open/close 3313 (time 1788843582.52 total 50.06 last 70.84) - open/close 4047 (time 1788843592.52 total 60.06 last 73.40) - open/close 4776 (time 1788843602.55 total 70.09 last 72.69) - open/close 5469 (time 1788843612.55 total 80.09 last 69.30) - open/close 6088 (time 1788843622.56 total 90.10 last 61.83) - open/close 6769 (time 1788843632.57 total 100.11 last 68.03) - open/close 7404 (time 1788843642.58 total 110.12 last 63.44) - open/close 8071 (time 1788843652.59 total 120.13 last 66.64) - open/close 8729 (time 1788843662.60 total 130.14 last 65.77) - open/close 9347 (time 1788843672.66 total 140.20 last 61.42) - open/close 9844 (time 1788843682.66 total 150.21 last 49.66) total: 10000 open/close in 153.68 seconds: 65.07 ops/second - unlinked 0 (time 1788843692 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 701 (time 1788843796.33 total 10.01 last 70.02) - open/close 1426 (time 1788843806.34 total 20.02 last 72.45) - open/close 2187 (time 1788843816.34 total 30.02 last 76.09) - open/close 3022 (time 1788843826.37 total 40.05 last 83.22) - open/close 3769 (time 1788843836.38 total 50.06 last 74.68) - open/close 4423 (time 1788843846.39 total 60.07 last 65.33) - open/close 5031 (time 1788843856.39 total 70.07 last 60.79) - open/close 5750 (time 1788843866.40 total 80.07 last 71.87) - open/close 6389 (time 1788843876.40 total 90.08 last 63.84) - open/close 6991 (time 1788843886.41 total 100.09 last 60.17) - open/close 7568 (time 1788843896.44 total 110.11 last 57.55) - open/close 8192 (time 1788843906.44 total 120.12 last 62.38) - open/close 8942 (time 1788843916.44 total 130.12 last 74.99) - open/close 9614 (time 1788843926.44 total 140.12 last 67.19) total: 10000 open/close in 145.90 seconds: 68.54 ops/second - unlinked 0 (time 1788843937 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 691 (time 1788844052.29 total 10.01 last 69.01) - open/close 1456 (time 1788844062.29 total 20.02 last 76.48) - open/close 2211 (time 1788844072.30 total 30.02 last 75.49) - open/close 2934 (time 1788844082.31 total 40.03 last 72.19) - open/close 3653 (time 1788844092.33 total 50.05 last 71.78) - open/close 4359 (time 1788844102.34 total 60.06 last 70.54) - open/close 4920 (time 1788844112.34 total 70.06 last 56.09) - open/close 5669 (time 1788844122.34 total 80.06 last 74.88) - open/close 6458 (time 1788844132.34 total 90.06 last 78.88) - open/close 7159 (time 1788844142.35 total 100.07 last 70.05) - open/close 7829 (time 1788844152.36 total 110.08 last 66.95) - open/close 8562 (time 1788844162.36 total 120.08 last 73.25) - open/close 9264 (time 1788844172.37 total 130.09 last 70.14) total: 10000 open/close in 140.07 seconds: 71.39 ops/second - unlinked 0 (time 1788844186 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 717 (time 1788844283.77 total 10.01 last 71.65) - open/close 1382 (time 1788844293.78 total 20.02 last 66.40) - open/close 2115 (time 1788844303.78 total 30.02 last 73.29) - open/close 2909 (time 1788844313.79 total 40.03 last 79.37) - open/close 3501 (time 1788844323.79 total 50.03 last 59.20) - open/close 4175 (time 1788844333.79 total 60.04 last 67.35) - open/close 4862 (time 1788844343.80 total 70.04 last 68.66) - open/close 5570 (time 1788844353.83 total 80.07 last 70.59) - open/close 6379 (time 1788844363.85 total 90.09 last 80.75) - open/close 7179 (time 1788844373.86 total 100.10 last 79.93) - open/close 7855 (time 1788844383.86 total 110.10 last 67.58) - open/close 8588 (time 1788844393.87 total 120.11 last 73.23) - open/close 9282 (time 1788844403.88 total 130.12 last 69.34) total: 10000 open/close in 138.88 seconds: 72.01 ops/second - unlinked 0 (time 1788844417 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 651 (time 1788844517.78 total 10.00 last 65.07) - open/close 1419 (time 1788844527.78 total 20.01 last 76.77) - open/close 2217 (time 1788844537.78 total 30.01 last 79.78) - open/close 2843 (time 1788844547.79 total 40.01 last 62.59) - open/close 3561 (time 1788844557.79 total 50.02 last 71.78) - open/close 4166 (time 1788844567.79 total 60.02 last 60.49) - open/close 4814 (time 1788844577.79 total 70.02 last 64.79) - open/close 5485 (time 1788844587.80 total 80.02 last 67.08) - open/close 6290 (time 1788844597.80 total 90.02 last 80.48) - open/close 7136 (time 1788844607.81 total 100.04 last 84.48) - open/close 7685 (time 1788844617.82 total 110.05 last 54.86) - open/close 8145 (time 1788844627.83 total 120.05 last 45.97) - open/close 8778 (time 1788844637.83 total 130.05 last 63.29) - open/close 9435 (time 1788844647.84 total 140.06 last 65.65) total: 10000 open/close in 150.05 seconds: 66.64 ops/second - unlinked 0 (time 1788844662 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.109@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.109@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg209-server: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg209-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.109@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.109@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.109@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.109@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.202.109@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg209-server: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg209-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: kmemleak disabled modules unloaded. pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2