== conf-sanity test 69: replace an OST with the same index ========================================================== 21:24:09 (1789608249) start mds service on oleg340-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931640 Sep 16 13:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: kmemleak disabled oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: MODOPTS_LIBCFS= oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg340-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg340-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg340-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg340-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 713 (time 1789608367.37 total 10.01 last 71.26) - open/close 1408 (time 1789608377.37 total 20.01 last 69.50) - open/close 2144 (time 1789608387.37 total 30.01 last 73.58) - open/close 2834 (time 1789608397.37 total 40.01 last 69.00) - open/close 3599 (time 1789608407.38 total 50.01 last 76.45) - open/close 4349 (time 1789608417.40 total 60.03 last 74.85) - open/close 5112 (time 1789608427.41 total 70.04 last 76.27) - open/close 5796 (time 1789608437.41 total 80.04 last 68.39) - open/close 6540 (time 1789608447.42 total 90.05 last 74.34) - open/close 7177 (time 1789608457.43 total 100.06 last 63.63) - open/close 7904 (time 1789608467.44 total 110.07 last 72.62) - open/close 8644 (time 1789608477.46 total 120.09 last 73.86) - open/close 9449 (time 1789608487.46 total 130.09 last 80.46) total: 10000 open/close in 138.07 seconds: 72.42 ops/second - unlinked 0 (time 1789608500 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 812 (time 1789608606.09 total 10.00 last 81.17) - open/close 1526 (time 1789608616.10 total 20.01 last 71.36) - open/close 2275 (time 1789608626.11 total 30.02 last 74.78) - open/close 3054 (time 1789608636.12 total 40.03 last 77.84) - open/close 3791 (time 1789608646.12 total 50.04 last 73.67) - open/close 4415 (time 1789608656.14 total 60.05 last 62.30) - open/close 5081 (time 1789608666.14 total 70.05 last 66.58) - open/close 5746 (time 1789608676.15 total 80.06 last 66.44) - open/close 6320 (time 1789608686.18 total 90.09 last 57.26) - open/close 6978 (time 1789608696.20 total 100.11 last 65.65) - open/close 7640 (time 1789608706.20 total 110.11 last 66.20) - open/close 8318 (time 1789608716.20 total 120.11 last 67.79) - open/close 9028 (time 1789608726.21 total 130.12 last 70.97) - open/close 9756 (time 1789608736.21 total 140.12 last 72.78) total: 10000 open/close in 143.58 seconds: 69.65 ops/second - unlinked 0 (time 1789608744 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 707 (time 1789608844.99 total 10.01 last 70.66) - open/close 1358 (time 1789608855.00 total 20.01 last 65.05) - open/close 1916 (time 1789608865.03 total 30.04 last 55.65) - open/close 2617 (time 1789608875.03 total 40.04 last 70.10) - open/close 3280 (time 1789608885.03 total 50.05 last 66.25) - open/close 4060 (time 1789608895.04 total 60.06 last 77.96) - open/close 4867 (time 1789608905.04 total 70.06 last 80.70) - open/close 5552 (time 1789608915.06 total 80.07 last 68.38) - open/close 6318 (time 1789608925.07 total 90.08 last 76.54) - open/close 7014 (time 1789608935.07 total 100.09 last 69.57) - open/close 7555 (time 1789608945.07 total 110.09 last 54.09) - open/close 8212 (time 1789608955.08 total 120.09 last 65.66) - open/close 8852 (time 1789608965.09 total 130.10 last 63.95) - open/close 9406 (time 1789608975.09 total 140.11 last 55.36) - open/close 9869 (time 1789608985.10 total 150.12 last 46.27) total: 10000 open/close in 151.92 seconds: 65.82 ops/second - unlinked 0 (time 1789608991 ; total 1 ; last 1) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 749 (time 1789609089.34 total 10.01 last 74.83) - open/close 1396 (time 1789609099.37 total 20.04 last 64.53) - open/close 2126 (time 1789609109.38 total 30.05 last 72.93) - open/close 2840 (time 1789609119.39 total 40.06 last 71.27) - open/close 3590 (time 1789609129.40 total 50.07 last 74.95) - open/close 4339 (time 1789609139.41 total 60.08 last 74.81) - open/close 4993 (time 1789609149.42 total 70.09 last 65.36) - open/close 5680 (time 1789609159.42 total 80.09 last 68.67) - open/close 6363 (time 1789609169.46 total 90.13 last 68.01) - open/close 7039 (time 1789609179.46 total 100.14 last 67.60) - open/close 7688 (time 1789609189.47 total 110.14 last 64.85) - open/close 8331 (time 1789609199.48 total 120.15 last 64.22) - open/close 9021 (time 1789609209.48 total 130.16 last 69.00) - open/close 9728 (time 1789609219.49 total 140.17 last 70.63) total: 10000 open/close in 143.88 seconds: 69.50 ops/second - unlinked 0 (time 1789609227 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 748 (time 1789609331.09 total 10.00 last 74.79) - open/close 1301 (time 1789609341.11 total 20.02 last 55.19) - open/close 1970 (time 1789609351.11 total 30.02 last 66.90) - open/close 2597 (time 1789609361.11 total 40.03 last 62.67) - open/close 3228 (time 1789609371.13 total 50.05 last 62.97) - open/close 3835 (time 1789609381.15 total 60.06 last 60.63) - open/close 4368 (time 1789609391.16 total 70.07 last 53.24) - open/close 5041 (time 1789609401.16 total 80.07 last 67.29) - open/close 5709 (time 1789609411.16 total 90.07 last 66.79) - open/close 6407 (time 1789609421.16 total 100.08 last 69.78) - open/close 7112 (time 1789609431.19 total 110.10 last 70.30) - open/close 7725 (time 1789609441.24 total 120.15 last 61.00) - open/close 8311 (time 1789609451.25 total 130.16 last 58.57) - open/close 8969 (time 1789609461.25 total 140.16 last 65.76) - open/close 9643 (time 1789609471.27 total 150.19 last 67.25) total: 10000 open/close in 156.00 seconds: 64.10 ops/second - unlinked 0 (time 1789609481 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 624 (time 1789609586.69 total 10.01 last 62.33) - open/close 1131 (time 1789609596.69 total 20.01 last 50.70) - open/close 1794 (time 1789609606.70 total 30.03 last 66.20) - open/close 2486 (time 1789609616.71 total 40.03 last 69.18) - open/close 3131 (time 1789609626.72 total 50.04 last 64.43) - open/close 3831 (time 1789609636.73 total 60.05 last 69.95) - open/close 4469 (time 1789609646.75 total 70.08 last 63.62) - open/close 5143 (time 1789609656.76 total 80.08 last 67.38) - open/close 5826 (time 1789609666.79 total 90.11 last 68.10) - open/close 6445 (time 1789609676.79 total 100.11 last 61.88) - open/close 7036 (time 1789609686.79 total 110.12 last 59.07) - open/close 7760 (time 1789609696.81 total 120.14 last 72.28) - open/close 8452 (time 1789609706.81 total 130.14 last 69.20) - open/close 9090 (time 1789609716.81 total 140.14 last 63.80) - open/close 9779 (time 1789609726.82 total 150.15 last 68.83) total: 10000 open/close in 153.16 seconds: 65.29 ops/second - unlinked 0 (time 1789609733 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 583 (time 1789609838.04 total 10.01 last 58.26) - open/close 1158 (time 1789609848.05 total 20.01 last 57.48) - open/close 1741 (time 1789609858.05 total 30.02 last 58.27) - open/close 2372 (time 1789609868.06 total 40.02 last 63.06) - open/close 3034 (time 1789609878.07 total 50.03 last 66.11) - open/close 3648 (time 1789609888.09 total 60.05 last 61.31) - open/close 4312 (time 1789609898.09 total 70.05 last 66.38) - open/close 4895 (time 1789609908.10 total 80.06 last 58.26) - open/close 5483 (time 1789609918.10 total 90.06 last 58.80) - open/close 6049 (time 1789609928.10 total 100.07 last 56.55) - open/close 6748 (time 1789609938.12 total 110.08 last 69.79) - open/close 7352 (time 1789609948.12 total 120.08 last 60.40) - open/close 8087 (time 1789609958.12 total 130.08 last 73.49) - open/close 8719 (time 1789609968.13 total 140.09 last 63.18) - open/close 9323 (time 1789609978.13 total 150.09 last 60.37) - open/close 9966 (time 1789609988.13 total 160.09 last 64.29) total: 10000 open/close in 160.71 seconds: 62.22 ops/second - unlinked 0 (time 1789609993 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 773 (time 1789610104.52 total 10.01 last 77.21) - open/close 1451 (time 1789610114.53 total 20.02 last 67.73) - open/close 2175 (time 1789610124.54 total 30.03 last 72.34) - open/close 2810 (time 1789610134.55 total 40.04 last 63.45) - open/close 3404 (time 1789610144.57 total 50.06 last 59.28) - open/close 4003 (time 1789610154.57 total 60.06 last 59.89) - open/close 4524 (time 1789610164.58 total 70.07 last 52.06) - open/close 5172 (time 1789610174.61 total 80.10 last 64.61) - open/close 5944 (time 1789610184.62 total 90.11 last 77.12) - open/close 6620 (time 1789610194.63 total 100.12 last 67.53) - open/close 7336 (time 1789610204.64 total 110.13 last 71.55) - open/close 8036 (time 1789610214.64 total 120.13 last 69.95) - open/close 8631 (time 1789610224.67 total 130.16 last 59.34) - open/close 9322 (time 1789610234.67 total 140.16 last 69.08) total: 10000 open/close in 150.06 seconds: 66.64 ops/second - unlinked 0 (time 1789610249 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 678 (time 1789610352.21 total 10.00 last 67.80) - open/close 1367 (time 1789610362.23 total 20.01 last 68.81) - open/close 2089 (time 1789610372.23 total 30.01 last 72.19) - open/close 2826 (time 1789610382.24 total 40.02 last 73.64) - open/close 3583 (time 1789610392.24 total 50.02 last 75.69) - open/close 4315 (time 1789610402.24 total 60.03 last 73.17) - open/close 5027 (time 1789610412.24 total 70.03 last 71.19) - open/close 5712 (time 1789610422.25 total 80.04 last 68.45) - open/close 6385 (time 1789610432.25 total 90.04 last 67.28) - open/close 7123 (time 1789610442.27 total 100.05 last 73.71) - open/close 7874 (time 1789610452.28 total 110.06 last 75.02) - open/close 8553 (time 1789610462.30 total 120.08 last 67.78) - open/close 9177 (time 1789610472.31 total 130.10 last 62.29) - open/close 9836 (time 1789610482.32 total 140.10 last 65.86) total: 10000 open/close in 142.95 seconds: 69.96 ops/second - unlinked 0 (time 1789610489 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 722 (time 1789610596.14 total 10.02 last 72.04) - open/close 1430 (time 1789610606.16 total 20.04 last 70.67) - open/close 2198 (time 1789610616.18 total 30.06 last 76.67) - open/close 2935 (time 1789610626.18 total 40.07 last 73.64) - open/close 3706 (time 1789610636.19 total 50.07 last 77.03) - open/close 4415 (time 1789610646.20 total 60.08 last 70.88) - open/close 5177 (time 1789610656.20 total 70.08 last 76.16) - open/close 5905 (time 1789610666.22 total 80.10 last 72.69) - open/close 6607 (time 1789610676.22 total 90.10 last 70.17) - open/close 7312 (time 1789610686.22 total 100.10 last 70.48) - open/close 8063 (time 1789610696.23 total 110.11 last 75.06) - open/close 8793 (time 1789610706.24 total 120.12 last 72.92) - open/close 9487 (time 1789610716.25 total 130.13 last 69.32) total: 10000 open/close in 137.96 seconds: 72.48 ops/second - unlinked 0 (time 1789610728 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.140@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.140@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg340-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre On OST0, 20520 used inodes rc=0 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.140@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.140@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.140@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.140@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.140@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg340-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server unloading modules via unload_modules_local on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local oleg340-server: kmemleak disabled modules unloaded. pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2