== conf-sanity test 69: replace an OST with the same index ========================================================== 08:29:23 (1790771363) start mds service on oleg160-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1952976 Sep 29 14:03 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: kmemleak disabled oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: detected 4 online CPUs by sysfs oleg160-server: MODOPTS_LIBCFS= oleg160-server: Force libcfs to create 2 CPU partitions oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg160-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg160-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 694 (time 1790771464.02 total 10.00 last 69.39) - open/close 1673 (time 1790771474.02 total 20.00 last 97.89) - open/close 2366 (time 1790771484.02 total 30.00 last 69.30) - open/close 3201 (time 1790771494.02 total 40.00 last 83.49) - open/close 4031 (time 1790771504.02 total 50.00 last 82.98) - open/close 4785 (time 1790771514.03 total 60.01 last 75.39) - open/close 5542 (time 1790771524.05 total 70.03 last 75.48) - open/close 6255 (time 1790771534.06 total 80.04 last 71.25) - open/close 6957 (time 1790771544.06 total 90.04 last 70.18) - open/close 7701 (time 1790771554.09 total 100.07 last 74.22) - open/close 8445 (time 1790771564.09 total 110.07 last 74.36) - open/close 9169 (time 1790771574.10 total 120.08 last 72.36) - open/close 9935 (time 1790771584.10 total 130.08 last 76.56) total: 10000 open/close in 130.81 seconds: 76.45 ops/second - unlinked 0 (time 1790771589 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 737 (time 1790771686.92 total 10.00 last 73.69) - open/close 1459 (time 1790771696.93 total 20.01 last 72.11) - open/close 2232 (time 1790771706.93 total 30.02 last 77.27) - open/close 2991 (time 1790771716.94 total 40.02 last 75.86) - open/close 3792 (time 1790771726.94 total 50.03 last 80.04) - open/close 4568 (time 1790771736.95 total 60.04 last 77.55) - open/close 5306 (time 1790771746.96 total 70.05 last 73.71) - open/close 6011 (time 1790771756.97 total 80.05 last 70.47) - open/close 6807 (time 1790771766.98 total 90.06 last 79.52) - open/close 7630 (time 1790771776.98 total 100.06 last 82.29) - open/close 8371 (time 1790771786.99 total 110.08 last 74.01) - open/close 9035 (time 1790771796.99 total 120.08 last 66.38) - open/close 9752 (time 1790771807.00 total 130.08 last 71.67) total: 10000 open/close in 132.93 seconds: 75.23 ops/second - unlinked 0 (time 1790771814 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 668 (time 1790771911.14 total 10.01 last 66.75) - open/close 1542 (time 1790771921.15 total 20.01 last 87.34) - open/close 2321 (time 1790771931.15 total 30.02 last 77.85) - open/close 3062 (time 1790771941.16 total 40.03 last 74.04) - open/close 3849 (time 1790771951.16 total 50.03 last 78.68) - open/close 4591 (time 1790771961.18 total 60.04 last 74.12) - open/close 5228 (time 1790771971.18 total 70.05 last 63.65) - open/close 6003 (time 1790771981.18 total 80.05 last 77.48) - open/close 6845 (time 1790771991.19 total 90.06 last 84.17) - open/close 7464 (time 1790772001.19 total 100.06 last 61.89) - open/close 8191 (time 1790772011.20 total 110.07 last 72.60) - open/close 9022 (time 1790772021.20 total 120.07 last 83.09) - open/close 9928 (time 1790772031.24 total 130.11 last 90.27) total: 10000 open/close in 131.00 seconds: 76.33 ops/second - unlinked 0 (time 1790772036 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 836 (time 1790772129.48 total 10.00 last 83.56) - open/close 1793 (time 1790772139.48 total 20.01 last 95.65) - open/close 2720 (time 1790772149.49 total 30.01 last 92.67) - open/close 3632 (time 1790772159.50 total 40.03 last 91.05) - open/close 4543 (time 1790772169.52 total 50.05 last 90.90) - open/close 5530 (time 1790772179.53 total 60.06 last 98.63) - open/close 6415 (time 1790772189.54 total 70.06 last 88.45) - open/close 7129 (time 1790772199.54 total 80.07 last 71.37) - open/close 7961 (time 1790772209.56 total 90.09 last 83.01) - open/close 8719 (time 1790772219.57 total 100.10 last 75.73) - open/close 9459 (time 1790772229.58 total 110.10 last 73.98) total: 10000 open/close in 117.01 seconds: 85.46 ops/second - unlinked 0 (time 1790772241 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 807 (time 1790772336.64 total 10.00 last 80.67) - open/close 1522 (time 1790772346.64 total 20.01 last 71.48) - open/close 2331 (time 1790772356.65 total 30.01 last 80.85) - open/close 3078 (time 1790772366.65 total 40.01 last 74.69) - open/close 3806 (time 1790772376.66 total 50.02 last 72.75) - open/close 4563 (time 1790772386.67 total 60.03 last 75.63) - open/close 5368 (time 1790772396.68 total 70.04 last 80.42) - open/close 6291 (time 1790772406.68 total 80.04 last 92.27) - open/close 7195 (time 1790772416.68 total 90.05 last 90.38) - open/close 7969 (time 1790772426.69 total 100.05 last 77.35) - open/close 8761 (time 1790772436.70 total 110.06 last 79.11) - open/close 9606 (time 1790772446.70 total 120.07 last 84.44) total: 10000 open/close in 125.62 seconds: 79.61 ops/second - unlinked 0 (time 1790772456 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 684 (time 1790772551.21 total 10.01 last 68.33) - open/close 1484 (time 1790772561.22 total 20.01 last 79.96) - open/close 2265 (time 1790772571.22 total 30.02 last 78.09) - open/close 3133 (time 1790772581.24 total 40.04 last 86.62) - open/close 3960 (time 1790772591.24 total 50.04 last 82.68) - open/close 4705 (time 1790772601.26 total 60.06 last 74.39) - open/close 5448 (time 1790772611.27 total 70.06 last 74.24) - open/close 6289 (time 1790772621.27 total 80.07 last 84.09) - open/close 7018 (time 1790772631.27 total 90.07 last 72.87) - open/close 7800 (time 1790772641.28 total 100.07 last 78.16) - open/close 8480 (time 1790772651.29 total 110.08 last 67.95) - open/close 9127 (time 1790772661.29 total 120.09 last 64.67) - open/close 9884 (time 1790772671.29 total 130.09 last 75.68) total: 10000 open/close in 131.45 seconds: 76.07 ops/second - unlinked 0 (time 1790772677 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 756 (time 1790772776.06 total 10.01 last 75.51) - open/close 1479 (time 1790772786.07 total 20.02 last 72.26) - open/close 2262 (time 1790772796.07 total 30.02 last 78.27) - open/close 3043 (time 1790772806.08 total 40.03 last 78.03) - open/close 3760 (time 1790772816.09 total 50.04 last 71.62) - open/close 4608 (time 1790772826.10 total 60.05 last 84.72) - open/close 5437 (time 1790772836.10 total 70.06 last 82.86) - open/close 6219 (time 1790772846.11 total 80.06 last 78.18) - open/close 6912 (time 1790772856.12 total 90.07 last 69.22) - open/close 7606 (time 1790772866.13 total 100.08 last 69.31) - open/close 8282 (time 1790772876.13 total 110.09 last 67.58) - open/close 9047 (time 1790772886.19 total 120.14 last 76.06) - open/close 9590 (time 1790772896.21 total 130.16 last 54.22) total: 10000 open/close in 137.44 seconds: 72.76 ops/second - unlinked 0 (time 1790772911 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 675 (time 1790773007.90 total 10.00 last 67.49) - open/close 1341 (time 1790773017.92 total 20.02 last 66.50) - open/close 2097 (time 1790773027.94 total 30.04 last 75.43) - open/close 2539 (time 1790773037.94 total 40.04 last 44.19) - open/close 3116 (time 1790773047.96 total 50.06 last 57.60) - open/close 3747 (time 1790773057.97 total 60.07 last 63.03) - open/close 4502 (time 1790773067.98 total 70.08 last 75.45) - open/close 5195 (time 1790773077.99 total 80.09 last 69.23) - open/close 5841 (time 1790773088.00 total 90.10 last 64.54) - open/close 6547 (time 1790773098.00 total 100.10 last 70.58) - open/close 7151 (time 1790773108.00 total 110.10 last 60.40) - open/close 7880 (time 1790773118.01 total 120.11 last 72.83) - open/close 8629 (time 1790773128.02 total 130.12 last 74.83) - open/close 9325 (time 1790773138.02 total 140.12 last 69.58) total: 10000 open/close in 148.52 seconds: 67.33 ops/second - unlinked 0 (time 1790773151 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 693 (time 1790773245.89 total 10.01 last 69.24) - open/close 1399 (time 1790773255.90 total 20.02 last 70.53) - open/close 2094 (time 1790773265.91 total 30.03 last 69.43) - open/close 2786 (time 1790773275.91 total 40.03 last 69.19) - open/close 3642 (time 1790773285.91 total 50.04 last 85.54) - open/close 4523 (time 1790773295.92 total 60.04 last 88.09) - open/close 5255 (time 1790773305.94 total 70.06 last 73.03) - open/close 5959 (time 1790773315.95 total 80.07 last 70.36) - open/close 6723 (time 1790773325.95 total 90.07 last 76.37) - open/close 7427 (time 1790773335.95 total 100.07 last 70.39) - open/close 8264 (time 1790773345.95 total 110.08 last 83.68) - open/close 9013 (time 1790773355.96 total 120.08 last 74.84) - open/close 9822 (time 1790773365.96 total 130.08 last 80.88) total: 10000 open/close in 132.72 seconds: 75.35 ops/second - unlinked 0 (time 1790773374 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 460 (time 1790773475.17 total 10.02 last 45.93) - open/close 1322 (time 1790773485.18 total 20.03 last 86.10) - open/close 2064 (time 1790773495.19 total 30.03 last 74.15) - open/close 2797 (time 1790773505.20 total 40.05 last 73.21) - open/close 3516 (time 1790773515.21 total 50.06 last 71.79) - open/close 4239 (time 1790773525.23 total 60.08 last 72.18) - open/close 4891 (time 1790773535.24 total 70.09 last 65.13) - open/close 5449 (time 1790773545.25 total 80.10 last 55.75) - open/close 6111 (time 1790773555.25 total 90.10 last 66.18) - open/close 6896 (time 1790773565.25 total 100.10 last 78.49) - open/close 7603 (time 1790773575.27 total 110.11 last 70.61) - open/close 8246 (time 1790773585.27 total 120.12 last 64.26) - open/close 8992 (time 1790773595.28 total 130.13 last 74.55) - open/close 9781 (time 1790773605.31 total 140.16 last 78.63) total: 10000 open/close in 143.17 seconds: 69.85 ops/second - unlinked 0 (time 1790773612 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.160@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.160@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.160@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.160@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.201.160@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server unloading modules via unload_modules_local on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: kmemleak disabled modules unloaded. pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2