== conf-sanity test 69: replace an OST with the same index ========================================================== 20:09:49 (1790726989) start mds service on oleg214-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1946856 Sep 29 10:18 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: kmemleak disabled oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: MODOPTS_LIBCFS= oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg214-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg214-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg214-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg214-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 671 (time 1790727104.84 total 10.01 last 67.02) - open/close 1437 (time 1790727114.84 total 20.01 last 76.60) - open/close 2168 (time 1790727124.86 total 30.03 last 72.94) - open/close 2963 (time 1790727134.87 total 40.04 last 79.44) - open/close 3633 (time 1790727144.87 total 50.04 last 67.00) - open/close 4387 (time 1790727154.89 total 60.06 last 75.29) - open/close 5288 (time 1790727164.89 total 70.06 last 90.10) - open/close 6068 (time 1790727174.89 total 80.07 last 77.94) - open/close 6722 (time 1790727184.90 total 90.07 last 65.35) - open/close 7419 (time 1790727194.91 total 100.09 last 69.61) - open/close 8057 (time 1790727204.94 total 110.11 last 63.66) - open/close 8810 (time 1790727214.94 total 120.11 last 75.29) - open/close 9645 (time 1790727224.94 total 130.11 last 83.48) total: 10000 open/close in 134.56 seconds: 74.32 ops/second - unlinked 0 (time 1790727234 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 880 (time 1790727327.83 total 10.00 last 87.96) - open/close 1751 (time 1790727337.84 total 20.01 last 87.03) - open/close 2550 (time 1790727347.85 total 30.02 last 79.85) - open/close 3270 (time 1790727357.86 total 40.03 last 71.91) - open/close 4090 (time 1790727367.87 total 50.04 last 81.89) - open/close 4975 (time 1790727377.88 total 60.05 last 88.42) - open/close 5801 (time 1790727387.89 total 70.06 last 82.55) - open/close 6619 (time 1790727397.89 total 80.06 last 81.79) - open/close 7346 (time 1790727407.89 total 90.07 last 72.67) - open/close 8148 (time 1790727417.90 total 100.07 last 80.17) - open/close 8929 (time 1790727427.90 total 110.07 last 78.09) - open/close 9741 (time 1790727437.90 total 120.08 last 81.15) total: 10000 open/close in 123.15 seconds: 81.20 ops/second - unlinked 0 (time 1790727445 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 751 (time 1790727535.78 total 10.00 last 75.09) - open/close 1499 (time 1790727545.79 total 20.01 last 74.75) - open/close 2366 (time 1790727555.80 total 30.02 last 86.55) - open/close 3152 (time 1790727565.82 total 40.04 last 78.49) - open/close 3991 (time 1790727575.83 total 50.05 last 83.77) - open/close 4758 (time 1790727585.84 total 60.07 last 76.61) - open/close 5472 (time 1790727595.85 total 70.07 last 71.37) - open/close 6228 (time 1790727605.85 total 80.07 last 75.58) - open/close 6987 (time 1790727615.85 total 90.07 last 75.89) - open/close 7683 (time 1790727625.88 total 100.10 last 69.43) - open/close 8489 (time 1790727635.88 total 110.10 last 80.59) - open/close 9160 (time 1790727645.88 total 120.11 last 67.06) - open/close 9924 (time 1790727655.89 total 130.11 last 76.39) total: 10000 open/close in 131.50 seconds: 76.05 ops/second - unlinked 0 (time 1790727662 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 758 (time 1790727760.42 total 10.01 last 75.69) - open/close 1448 (time 1790727770.43 total 20.02 last 68.97) - open/close 2171 (time 1790727780.44 total 30.03 last 72.24) - open/close 2845 (time 1790727790.45 total 40.04 last 67.35) - open/close 3539 (time 1790727800.45 total 50.04 last 69.37) - open/close 4226 (time 1790727810.46 total 60.05 last 68.64) - open/close 5144 (time 1790727820.47 total 70.06 last 91.67) - open/close 5893 (time 1790727830.48 total 80.07 last 74.83) - open/close 6707 (time 1790727840.49 total 90.08 last 81.34) - open/close 7495 (time 1790727850.49 total 100.08 last 78.78) - open/close 8316 (time 1790727860.50 total 110.09 last 82.00) - open/close 9178 (time 1790727870.51 total 120.10 last 86.17) - open/close 9845 (time 1790727880.52 total 130.11 last 66.61) total: 10000 open/close in 132.47 seconds: 75.49 ops/second - unlinked 0 (time 1790727887 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 688 (time 1790727981.18 total 10.00 last 68.77) - open/close 1404 (time 1790727991.18 total 20.00 last 71.59) - open/close 2183 (time 1790728001.18 total 30.00 last 77.90) - open/close 2951 (time 1790728011.19 total 40.02 last 76.70) - open/close 3728 (time 1790728021.21 total 50.04 last 77.55) - open/close 4693 (time 1790728031.22 total 60.04 last 96.45) - open/close 5471 (time 1790728041.22 total 70.04 last 77.79) - open/close 6051 (time 1790728051.22 total 80.05 last 57.98) - open/close 6795 (time 1790728061.23 total 90.05 last 74.36) - open/close 7496 (time 1790728071.23 total 100.05 last 70.09) - open/close 8119 (time 1790728081.23 total 110.06 last 62.29) - open/close 8786 (time 1790728091.24 total 120.06 last 66.65) - open/close 9476 (time 1790728101.24 total 130.07 last 68.98) total: 10000 open/close in 136.86 seconds: 73.07 ops/second - unlinked 0 (time 1790728113 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 865 (time 1790728209.52 total 10.00 last 86.47) - open/close 1670 (time 1790728219.52 total 20.01 last 80.49) - open/close 2413 (time 1790728229.53 total 30.01 last 74.27) - open/close 3192 (time 1790728239.53 total 40.01 last 77.90) - open/close 3994 (time 1790728249.55 total 50.03 last 80.03) - open/close 4689 (time 1790728259.56 total 60.04 last 69.44) - open/close 5391 (time 1790728269.58 total 70.06 last 70.06) - open/close 6163 (time 1790728279.58 total 80.06 last 77.19) - open/close 7055 (time 1790728289.59 total 90.07 last 89.12) - open/close 7730 (time 1790728299.59 total 100.08 last 67.47) - open/close 8245 (time 1790728309.61 total 110.10 last 51.39) - open/close 8864 (time 1790728319.62 total 120.11 last 61.83) - open/close 9584 (time 1790728329.63 total 130.11 last 71.96) total: 10000 open/close in 135.31 seconds: 73.90 ops/second - unlinked 0 (time 1790728339 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 630 (time 1790728429.00 total 10.01 last 62.91) - open/close 1390 (time 1790728439.01 total 20.02 last 75.94) - open/close 2264 (time 1790728449.02 total 30.03 last 87.35) - open/close 3128 (time 1790728459.02 total 40.03 last 86.39) - open/close 3948 (time 1790728469.03 total 50.04 last 81.88) - open/close 4816 (time 1790728479.03 total 60.05 last 86.79) - open/close 5673 (time 1790728489.04 total 70.05 last 85.66) - open/close 6525 (time 1790728499.04 total 80.06 last 85.15) - open/close 7469 (time 1790728509.05 total 90.06 last 94.34) - open/close 8278 (time 1790728519.06 total 100.07 last 80.83) - open/close 9139 (time 1790728529.07 total 110.08 last 86.03) total: 10000 open/close in 119.07 seconds: 83.98 ops/second - unlinked 0 (time 1790728541 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 703 (time 1790728619.53 total 10.01 last 70.26) - open/close 1728 (time 1790728629.53 total 20.01 last 102.50) - open/close 2636 (time 1790728639.54 total 30.01 last 90.75) - open/close 3598 (time 1790728649.54 total 40.01 last 96.18) - open/close 4498 (time 1790728659.55 total 50.02 last 89.91) - open/close 5513 (time 1790728669.56 total 60.03 last 101.42) - open/close 6406 (time 1790728679.56 total 70.03 last 89.29) - open/close 7355 (time 1790728689.57 total 80.04 last 94.83) - open/close 8247 (time 1790728699.57 total 90.04 last 89.16) - open/close 9170 (time 1790728709.57 total 100.05 last 92.28) total: 10000 open/close in 108.50 seconds: 92.16 ops/second - unlinked 0 (time 1790728721 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 991 (time 1790728797.52 total 10.00 last 99.08) - open/close 1995 (time 1790728807.52 total 20.00 last 100.37) - open/close 3007 (time 1790728817.52 total 30.01 last 101.19) - open/close 4046 (time 1790728827.53 total 40.01 last 103.84) - open/close 5047 (time 1790728837.53 total 50.01 last 100.07) - open/close 5921 (time 1790728847.53 total 60.02 last 87.38) - open/close 6736 (time 1790728857.54 total 70.03 last 81.42) - open/close 7605 (time 1790728867.55 total 80.03 last 86.87) - open/close 8543 (time 1790728877.55 total 90.03 last 93.78) - open/close 9394 (time 1790728887.56 total 100.04 last 85.04) total: 10000 open/close in 107.43 seconds: 93.08 ops/second - unlinked 0 (time 1790728898 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 859 (time 1790728984.29 total 10.00 last 85.88) - open/close 1626 (time 1790728994.31 total 20.01 last 76.62) - open/close 2531 (time 1790729004.31 total 30.01 last 90.48) - open/close 3468 (time 1790729014.31 total 40.02 last 93.68) - open/close 4374 (time 1790729024.31 total 50.02 last 90.58) - open/close 5312 (time 1790729034.32 total 60.03 last 93.73) - open/close 6037 (time 1790729044.32 total 70.03 last 72.48) - open/close 6677 (time 1790729054.33 total 80.04 last 63.95) - open/close 7316 (time 1790729064.34 total 90.05 last 63.83) - open/close 8017 (time 1790729074.36 total 100.07 last 69.96) - open/close 8893 (time 1790729084.37 total 110.07 last 87.55) - open/close 9925 (time 1790729094.37 total 120.08 last 103.15) total: 10000 open/close in 120.75 seconds: 82.81 ops/second - unlinked 0 (time 1790729100 ; total 1 ; last 1) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre On OST0, 20520 used inodes rc=0 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.114@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.114@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.202.114@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server unloading modules via unload_modules_local on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local oleg214-server: kmemleak disabled modules unloaded. pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2