== conf-sanity test 69: replace an OST with the same index ========================================================== 09:58:22 (1788530302) start mds service on oleg639-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921072 Sep 3 15:55 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg639-server' oleg639-server: oleg639-server.virtnet: executing load_modules_local oleg639-server: kmemleak disabled oleg639-server: Loading modules from /home/green/git/lustre-release/lustre oleg639-server: detected 4 online CPUs by sysfs oleg639-server: MODOPTS_LIBCFS= oleg639-server: Force libcfs to create 2 CPU partitions oleg639-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg639-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg639-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg639-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg639-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1 Started lustre-OST0000 oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg639-client.virtnet: -o user_xattr,flock 192.168.206.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 624 (time 1788530386.14 total 10.03 last 62.22) - open/close 1267 (time 1788530396.15 total 20.04 last 64.23) - open/close 1916 (time 1788530406.17 total 30.05 last 64.81) - open/close 2625 (time 1788530416.18 total 40.06 last 70.84) - open/close 3303 (time 1788530426.18 total 50.06 last 67.79) - open/close 4008 (time 1788530436.19 total 60.07 last 70.43) - open/close 4680 (time 1788530446.20 total 70.08 last 67.13) - open/close 5353 (time 1788530456.20 total 80.09 last 67.26) - open/close 6013 (time 1788530466.21 total 90.10 last 65.95) - open/close 6664 (time 1788530476.22 total 100.11 last 65.04) - open/close 7392 (time 1788530486.24 total 110.13 last 72.67) - open/close 8085 (time 1788530496.24 total 120.13 last 69.28) - open/close 8752 (time 1788530506.25 total 130.13 last 66.67) - open/close 9462 (time 1788530516.25 total 140.14 last 70.95) total: 10000 open/close in 147.41 seconds: 67.84 ops/second - unlinked 0 (time 1788530527 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 748 (time 1788530628.42 total 10.00 last 74.79) - open/close 1439 (time 1788530638.43 total 20.01 last 69.05) - open/close 2152 (time 1788530648.43 total 30.02 last 71.25) - open/close 2860 (time 1788530658.45 total 40.03 last 70.68) - open/close 3493 (time 1788530668.47 total 50.05 last 63.18) - open/close 4230 (time 1788530678.47 total 60.06 last 73.66) - open/close 4969 (time 1788530688.48 total 70.06 last 73.87) - open/close 5692 (time 1788530698.49 total 80.07 last 72.23) - open/close 6386 (time 1788530708.50 total 90.08 last 69.32) - open/close 7088 (time 1788530718.50 total 100.08 last 70.19) - open/close 7834 (time 1788530728.51 total 110.09 last 74.54) - open/close 8586 (time 1788530738.52 total 120.10 last 75.13) - open/close 9306 (time 1788530748.52 total 130.10 last 71.99) total: 10000 open/close in 139.99 seconds: 71.43 ops/second - unlinked 0 (time 1788530762 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 671 (time 1788530863.95 total 10.00 last 67.08) - open/close 1379 (time 1788530873.96 total 20.01 last 70.77) - open/close 1939 (time 1788530883.97 total 30.02 last 55.94) - open/close 2677 (time 1788530893.97 total 40.02 last 73.79) - open/close 3381 (time 1788530903.98 total 50.03 last 70.34) - open/close 4144 (time 1788530913.99 total 60.04 last 76.21) - open/close 4834 (time 1788530924.00 total 70.05 last 68.91) - open/close 5527 (time 1788530934.01 total 80.06 last 69.26) - open/close 6192 (time 1788530944.01 total 90.06 last 66.49) - open/close 6902 (time 1788530954.01 total 100.07 last 70.95) - open/close 7618 (time 1788530964.02 total 110.07 last 71.56) - open/close 8333 (time 1788530974.02 total 120.08 last 71.47) - open/close 9053 (time 1788530984.03 total 130.08 last 71.97) - open/close 9786 (time 1788530994.03 total 140.08 last 73.30) total: 10000 open/close in 143.01 seconds: 69.93 ops/second - unlinked 0 (time 1788531000 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 721 (time 1788531101.28 total 10.00 last 72.07) - open/close 1457 (time 1788531111.31 total 20.03 last 73.39) - open/close 2156 (time 1788531121.31 total 30.04 last 69.86) - open/close 2869 (time 1788531131.32 total 40.05 last 71.25) - open/close 3520 (time 1788531141.34 total 50.06 last 65.00) - open/close 4250 (time 1788531151.34 total 60.07 last 72.97) - open/close 4957 (time 1788531161.34 total 70.07 last 70.69) - open/close 5740 (time 1788531171.35 total 80.07 last 78.27) - open/close 6418 (time 1788531181.36 total 90.08 last 67.70) - open/close 7087 (time 1788531191.36 total 100.08 last 66.90) - open/close 7773 (time 1788531201.37 total 110.09 last 68.56) - open/close 8463 (time 1788531211.38 total 120.10 last 68.92) - open/close 9191 (time 1788531221.38 total 130.10 last 72.79) - open/close 9948 (time 1788531231.40 total 140.12 last 75.57) total: 10000 open/close in 140.99 seconds: 70.93 ops/second - unlinked 0 (time 1788531236 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 744 (time 1788531337.12 total 10.02 last 74.24) - open/close 1464 (time 1788531347.13 total 20.03 last 71.94) - open/close 2213 (time 1788531357.13 total 30.03 last 74.89) - open/close 2930 (time 1788531367.13 total 40.04 last 71.66) - open/close 3684 (time 1788531377.15 total 50.05 last 75.30) - open/close 4448 (time 1788531387.16 total 60.06 last 76.32) - open/close 5197 (time 1788531397.16 total 70.06 last 74.88) - open/close 5875 (time 1788531407.16 total 80.07 last 67.75) - open/close 6631 (time 1788531417.17 total 90.08 last 75.56) - open/close 7397 (time 1788531427.18 total 100.09 last 76.49) - open/close 8140 (time 1788531437.20 total 110.11 last 74.17) - open/close 8898 (time 1788531447.21 total 120.12 last 75.70) - open/close 9646 (time 1788531457.23 total 130.13 last 74.70) total: 10000 open/close in 135.93 seconds: 73.57 ops/second - unlinked 0 (time 1788531467 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 707 (time 1788531570.91 total 10.01 last 70.64) - open/close 1460 (time 1788531580.92 total 20.02 last 75.23) - open/close 2186 (time 1788531590.94 total 30.04 last 72.44) - open/close 2918 (time 1788531600.95 total 40.04 last 73.19) - open/close 3613 (time 1788531610.95 total 50.05 last 69.45) - open/close 4277 (time 1788531620.96 total 60.06 last 66.36) - open/close 4944 (time 1788531630.96 total 70.06 last 66.67) - open/close 5643 (time 1788531641.02 total 80.12 last 69.49) - open/close 6351 (time 1788531651.05 total 90.14 last 70.63) - open/close 7047 (time 1788531661.06 total 100.16 last 69.50) - open/close 7715 (time 1788531671.07 total 110.17 last 66.74) - open/close 8393 (time 1788531681.08 total 120.18 last 67.74) - open/close 9071 (time 1788531691.09 total 130.18 last 67.76) - open/close 9746 (time 1788531701.11 total 140.21 last 67.32) total: 10000 open/close in 143.74 seconds: 69.57 ops/second - unlinked 0 (time 1788531708 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 755 (time 1788531808.46 total 10.01 last 75.45) - open/close 1507 (time 1788531818.47 total 20.01 last 75.17) - open/close 2229 (time 1788531828.47 total 30.01 last 72.18) - open/close 2944 (time 1788531838.49 total 40.03 last 71.40) - open/close 3662 (time 1788531848.49 total 50.03 last 71.77) - open/close 4416 (time 1788531858.49 total 60.04 last 75.38) - open/close 5151 (time 1788531868.50 total 70.04 last 73.47) - open/close 5912 (time 1788531878.50 total 80.04 last 76.10) - open/close 6644 (time 1788531888.51 total 90.05 last 73.14) - open/close 7377 (time 1788531898.51 total 100.05 last 73.26) - open/close 8110 (time 1788531908.52 total 110.06 last 73.23) - open/close 8890 (time 1788531918.53 total 120.07 last 77.96) - open/close 9614 (time 1788531928.53 total 130.08 last 72.36) total: 10000 open/close in 135.09 seconds: 74.02 ops/second - unlinked 0 (time 1788531937 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 714 (time 1788532039.52 total 10.00 last 71.38) - open/close 1480 (time 1788532049.53 total 20.01 last 76.55) - open/close 2264 (time 1788532059.54 total 30.02 last 78.32) - open/close 2995 (time 1788532069.55 total 40.03 last 73.01) - open/close 3731 (time 1788532079.55 total 50.03 last 73.59) - open/close 4369 (time 1788532089.57 total 60.05 last 63.70) - open/close 4997 (time 1788532099.58 total 70.05 last 62.76) - open/close 5744 (time 1788532109.58 total 80.06 last 74.64) - open/close 6460 (time 1788532119.59 total 90.07 last 71.56) - open/close 7223 (time 1788532129.59 total 100.07 last 76.30) - open/close 7887 (time 1788532139.59 total 110.07 last 66.39) - open/close 8570 (time 1788532149.60 total 120.08 last 68.27) - open/close 9292 (time 1788532159.60 total 130.08 last 72.14) - open/close 9951 (time 1788532169.61 total 140.09 last 65.86) total: 10000 open/close in 140.66 seconds: 71.09 ops/second - unlinked 0 (time 1788532174 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 722 (time 1788532277.10 total 10.00 last 72.20) - open/close 1391 (time 1788532287.11 total 20.01 last 66.85) - open/close 2068 (time 1788532297.11 total 30.01 last 67.69) - open/close 2809 (time 1788532307.11 total 40.01 last 74.06) - open/close 3496 (time 1788532317.12 total 50.03 last 68.62) - open/close 4151 (time 1788532327.13 total 60.03 last 65.48) - open/close 4923 (time 1788532337.13 total 70.03 last 77.16) - open/close 5618 (time 1788532347.14 total 80.04 last 69.43) - open/close 6366 (time 1788532357.15 total 90.05 last 74.74) - open/close 7089 (time 1788532367.16 total 100.06 last 72.23) - open/close 7778 (time 1788532377.16 total 110.06 last 68.88) - open/close 8417 (time 1788532387.16 total 120.07 last 63.89) - open/close 9085 (time 1788532397.17 total 130.07 last 66.79) - open/close 9786 (time 1788532407.17 total 140.07 last 70.08) total: 10000 open/close in 142.90 seconds: 69.98 ops/second - unlinked 0 (time 1788532413 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 717 (time 1788532519.49 total 10.00 last 71.67) - open/close 1396 (time 1788532529.54 total 20.06 last 67.53) - open/close 2089 (time 1788532539.55 total 30.07 last 69.23) - open/close 2843 (time 1788532549.56 total 40.08 last 75.36) - open/close 3578 (time 1788532559.56 total 50.08 last 73.48) - open/close 4283 (time 1788532569.58 total 60.10 last 70.36) - open/close 4782 (time 1788532579.59 total 70.11 last 49.84) - open/close 5491 (time 1788532589.59 total 80.11 last 70.89) - open/close 6192 (time 1788532599.60 total 90.12 last 70.03) - open/close 6917 (time 1788532609.61 total 100.12 last 72.49) - open/close 7648 (time 1788532619.61 total 110.13 last 73.08) - open/close 8352 (time 1788532629.62 total 120.14 last 70.33) - open/close 9110 (time 1788532639.62 total 130.14 last 75.76) - open/close 9790 (time 1788532649.64 total 140.16 last 67.87) total: 10000 open/close in 142.68 seconds: 70.09 ops/second - unlinked 0 (time 1788532655 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg639-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg639-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg639-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg639-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1 Started lustre-OST0000 oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg639-server: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg639-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg639-client.virtnet: -o user_xattr,flock 192.168.206.139@tcp:/lustre /mnt/lustre On OST0, 13502 used inodes rc=0 pdsh@oleg639-client: oleg639-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg639-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg639-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg639-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.139@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.139@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.139@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg639-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1 Started lustre-OST0000 oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg639-server: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg639-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg639-client.virtnet: -o user_xattr,flock 192.168.206.139@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg639-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg639-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg639-server stop mds service on oleg639-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg639-server unloading modules via unload_modules_local on: 'oleg639-server' oleg639-server: oleg639-server.virtnet: executing unload_modules_local oleg639-server: kmemleak disabled modules unloaded. pdsh@oleg639-client: oleg639-client: ssh exited with exit code 2 pdsh@oleg639-client: oleg639-server: ssh exited with exit code 2 pdsh@oleg639-client: oleg639-server: ssh exited with exit code 2 pdsh@oleg639-client: oleg639-client: ssh exited with exit code 2