== conf-sanity test 69: replace an OST with the same index ========================================================== 01:39:48 (1788845988) start mds service on oleg237-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: kmemleak disabled oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg237-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 675 (time 1788846080.20 total 10.00 last 67.47) - open/close 1348 (time 1788846090.21 total 20.02 last 67.23) - open/close 2038 (time 1788846100.24 total 30.05 last 68.76) - open/close 2765 (time 1788846110.25 total 40.06 last 72.63) - open/close 3443 (time 1788846120.26 total 50.07 last 67.73) - open/close 4253 (time 1788846130.27 total 60.08 last 80.96) - open/close 4852 (time 1788846140.27 total 70.08 last 59.88) - open/close 5587 (time 1788846150.27 total 80.08 last 73.49) - open/close 6264 (time 1788846160.28 total 90.09 last 67.64) - open/close 7014 (time 1788846170.29 total 100.10 last 74.91) - open/close 7752 (time 1788846180.30 total 110.11 last 73.76) - open/close 8427 (time 1788846190.31 total 120.12 last 67.42) - open/close 9072 (time 1788846200.32 total 130.13 last 64.41) - open/close 9714 (time 1788846210.33 total 140.13 last 64.18) total: 10000 open/close in 144.47 seconds: 69.22 ops/second - unlinked 0 (time 1788846218 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 581 (time 1788846317.26 total 10.02 last 58.00) - open/close 1146 (time 1788846327.26 total 20.02 last 56.48) - open/close 1919 (time 1788846337.27 total 30.03 last 77.25) - open/close 2726 (time 1788846347.27 total 40.03 last 80.67) - open/close 3460 (time 1788846357.27 total 50.03 last 73.38) - open/close 4197 (time 1788846367.28 total 60.04 last 73.63) - open/close 4919 (time 1788846377.29 total 70.05 last 72.15) - open/close 5590 (time 1788846387.30 total 80.06 last 67.06) - open/close 6299 (time 1788846397.31 total 90.07 last 70.82) - open/close 6985 (time 1788846407.31 total 100.07 last 68.58) - open/close 7598 (time 1788846417.32 total 110.08 last 61.23) - open/close 8169 (time 1788846427.33 total 120.09 last 57.05) - open/close 8616 (time 1788846437.34 total 130.10 last 44.68) - open/close 9254 (time 1788846447.34 total 140.10 last 63.79) - open/close 9979 (time 1788846457.34 total 150.10 last 72.44) total: 10000 open/close in 150.46 seconds: 66.46 ops/second - unlinked 0 (time 1788846463 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 647 (time 1788846579.81 total 10.01 last 64.65) - open/close 1410 (time 1788846589.82 total 20.01 last 76.27) - open/close 1988 (time 1788846599.84 total 30.04 last 57.65) - open/close 2660 (time 1788846609.85 total 40.04 last 67.15) - open/close 3282 (time 1788846619.86 total 50.05 last 62.14) - open/close 3983 (time 1788846629.86 total 60.06 last 70.09) - open/close 4712 (time 1788846639.86 total 70.06 last 72.90) - open/close 5415 (time 1788846649.88 total 80.07 last 70.19) - open/close 6159 (time 1788846659.88 total 90.07 last 74.39) - open/close 6879 (time 1788846669.88 total 100.08 last 71.97) - open/close 7571 (time 1788846679.89 total 110.08 last 69.16) - open/close 8211 (time 1788846689.90 total 120.09 last 63.93) - open/close 8817 (time 1788846699.92 total 130.11 last 60.50) - open/close 9476 (time 1788846709.93 total 140.12 last 65.84) total: 10000 open/close in 147.14 seconds: 67.96 ops/second - unlinked 0 (time 1788846721 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 725 (time 1788846825.49 total 10.02 last 72.39) - open/close 1227 (time 1788846835.50 total 20.02 last 50.17) - open/close 1800 (time 1788846845.50 total 30.02 last 57.28) - open/close 2434 (time 1788846855.52 total 40.05 last 63.27) - open/close 3096 (time 1788846865.53 total 50.06 last 66.12) - open/close 3758 (time 1788846875.54 total 60.06 last 66.18) - open/close 4403 (time 1788846885.54 total 70.06 last 64.49) - open/close 4994 (time 1788846895.54 total 80.06 last 59.10) - open/close 5652 (time 1788846905.54 total 90.06 last 65.80) - open/close 6344 (time 1788846915.54 total 100.06 last 69.19) - open/close 6972 (time 1788846925.55 total 110.07 last 62.77) - open/close 7655 (time 1788846935.55 total 120.07 last 68.28) - open/close 8452 (time 1788846945.55 total 130.07 last 79.70) - open/close 9215 (time 1788846955.56 total 140.08 last 76.22) - open/close 9901 (time 1788846965.57 total 150.10 last 68.51) total: 10000 open/close in 151.98 seconds: 65.80 ops/second - unlinked 0 (time 1788846973 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 659 (time 1788847085.31 total 10.01 last 65.84) - open/close 1295 (time 1788847095.34 total 20.04 last 63.42) - open/close 1776 (time 1788847105.36 total 30.05 last 48.02) - open/close 2368 (time 1788847115.37 total 40.07 last 59.11) - open/close 3006 (time 1788847125.38 total 50.08 last 63.75) - open/close 3675 (time 1788847135.39 total 60.08 last 66.85) - open/close 4337 (time 1788847145.39 total 70.08 last 66.19) - open/close 5067 (time 1788847155.40 total 80.10 last 72.90) - open/close 5760 (time 1788847165.41 total 90.11 last 69.25) - open/close 6582 (time 1788847175.42 total 100.12 last 82.09) - open/close 7344 (time 1788847185.42 total 110.12 last 76.19) - open/close 8004 (time 1788847195.43 total 120.12 last 65.98) - open/close 8625 (time 1788847205.44 total 130.13 last 62.04) - open/close 9256 (time 1788847215.44 total 140.14 last 63.04) - open/close 9786 (time 1788847225.46 total 150.15 last 52.94) total: 10000 open/close in 153.87 seconds: 64.99 ops/second - unlinked 0 (time 1788847234 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 674 (time 1788847337.72 total 10.00 last 67.40) - open/close 1334 (time 1788847347.73 total 20.01 last 65.92) - open/close 1930 (time 1788847357.73 total 30.02 last 59.58) - open/close 2638 (time 1788847367.75 total 40.03 last 70.68) - open/close 3287 (time 1788847377.76 total 50.05 last 64.81) - open/close 3992 (time 1788847387.77 total 60.06 last 70.44) - open/close 4800 (time 1788847397.78 total 70.06 last 80.77) - open/close 5294 (time 1788847407.80 total 80.08 last 49.29) - open/close 5849 (time 1788847417.81 total 90.09 last 55.45) - open/close 6482 (time 1788847427.82 total 100.10 last 63.22) - open/close 7158 (time 1788847437.82 total 110.11 last 67.56) - open/close 7752 (time 1788847447.83 total 120.11 last 59.40) - open/close 8363 (time 1788847457.83 total 130.11 last 61.08) - open/close 8988 (time 1788847467.84 total 140.12 last 62.45) - open/close 9571 (time 1788847477.85 total 150.13 last 58.23) total: 10000 open/close in 157.12 seconds: 63.65 ops/second - unlinked 0 (time 1788847489 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 689 (time 1788847603.72 total 10.00 last 68.89) - open/close 1411 (time 1788847613.73 total 20.01 last 72.11) - open/close 2120 (time 1788847623.73 total 30.02 last 70.86) - open/close 2782 (time 1788847633.74 total 40.02 last 66.18) - open/close 3245 (time 1788847643.74 total 50.03 last 46.27) - open/close 3826 (time 1788847653.78 total 60.07 last 57.87) - open/close 4515 (time 1788847663.79 total 70.08 last 68.86) - open/close 5198 (time 1788847673.80 total 80.08 last 68.24) - open/close 5799 (time 1788847683.81 total 90.09 last 60.04) - open/close 6433 (time 1788847693.81 total 100.10 last 63.37) - open/close 7094 (time 1788847703.82 total 110.11 last 66.03) - open/close 7776 (time 1788847713.83 total 120.11 last 68.17) - open/close 8483 (time 1788847723.84 total 130.13 last 70.62) - open/close 9182 (time 1788847733.85 total 140.13 last 69.86) - open/close 9841 (time 1788847743.88 total 150.16 last 65.70) total: 10000 open/close in 152.12 seconds: 65.74 ops/second - unlinked 0 (time 1788847750 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 713 (time 1788847863.38 total 10.01 last 71.23) - open/close 1439 (time 1788847873.38 total 20.01 last 72.58) - open/close 2152 (time 1788847883.38 total 30.01 last 71.29) - open/close 2880 (time 1788847893.39 total 40.02 last 72.77) - open/close 3496 (time 1788847903.39 total 50.02 last 61.57) - open/close 4183 (time 1788847913.39 total 60.02 last 68.70) - open/close 4985 (time 1788847923.39 total 70.02 last 80.19) - open/close 5674 (time 1788847933.41 total 80.04 last 68.81) - open/close 6361 (time 1788847943.43 total 90.06 last 68.52) - open/close 7157 (time 1788847953.44 total 100.07 last 79.55) - open/close 7911 (time 1788847963.45 total 110.08 last 75.34) - open/close 8519 (time 1788847973.45 total 120.08 last 60.78) - open/close 9239 (time 1788847983.47 total 130.10 last 71.86) - open/close 9934 (time 1788847993.49 total 140.12 last 69.35) total: 10000 open/close in 141.38 seconds: 70.73 ops/second - unlinked 0 (time 1788847998 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 561 (time 1788848101.96 total 10.02 last 55.97) - open/close 1066 (time 1788848111.97 total 20.03 last 50.48) - open/close 1696 (time 1788848121.97 total 30.03 last 62.98) - open/close 2326 (time 1788848131.98 total 40.04 last 62.92) - open/close 2924 (time 1788848141.99 total 50.05 last 59.74) - open/close 3616 (time 1788848152.00 total 60.06 last 69.17) - open/close 4352 (time 1788848162.00 total 70.06 last 73.59) - open/close 5000 (time 1788848172.00 total 80.06 last 64.77) - open/close 5564 (time 1788848182.01 total 90.07 last 56.36) - open/close 6088 (time 1788848192.01 total 100.08 last 52.38) - open/close 6733 (time 1788848202.01 total 110.08 last 64.50) - open/close 7399 (time 1788848212.03 total 120.09 last 66.53) - open/close 8220 (time 1788848222.03 total 130.10 last 82.03) - open/close 8952 (time 1788848232.04 total 140.10 last 73.15) - open/close 9673 (time 1788848242.04 total 150.11 last 72.07) total: 10000 open/close in 154.32 seconds: 64.80 ops/second - unlinked 0 (time 1788848250 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 684 (time 1788848351.18 total 10.02 last 68.25) - open/close 1359 (time 1788848361.19 total 20.03 last 67.43) - open/close 2092 (time 1788848371.19 total 30.03 last 73.30) - open/close 2843 (time 1788848381.20 total 40.04 last 75.07) - open/close 3510 (time 1788848391.20 total 50.04 last 66.67) - open/close 4248 (time 1788848401.21 total 60.04 last 73.78) - open/close 5002 (time 1788848411.21 total 70.05 last 75.35) - open/close 5754 (time 1788848421.21 total 80.05 last 75.19) - open/close 6479 (time 1788848431.22 total 90.05 last 72.49) - open/close 7128 (time 1788848441.23 total 100.07 last 64.82) - open/close 7844 (time 1788848451.23 total 110.07 last 71.59) - open/close 8511 (time 1788848461.23 total 120.07 last 66.67) - open/close 9165 (time 1788848471.25 total 130.08 last 65.32) - open/close 9891 (time 1788848481.25 total 140.09 last 72.57) total: 10000 open/close in 141.57 seconds: 70.64 ops/second - unlinked 0 (time 1788848488 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.137@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.137@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.137@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: kmemleak disabled modules unloaded. pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2