== conf-sanity test 69: replace an OST with the same index ========================================================== 01:50:46 (1788587446) start mds service on oleg155-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 11:05 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing load_modules_local oleg155-server: kmemleak disabled oleg155-server: Loading modules from /home/green/git/lustre-release/lustre oleg155-server: detected 4 online CPUs by sysfs oleg155-server: MODOPTS_LIBCFS= oleg155-server: Force libcfs to create 2 CPU partitions oleg155-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg155-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg155-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg155-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg155-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 659 (time 1788587547.91 total 10.00 last 65.87) - open/close 1258 (time 1788587557.91 total 20.00 last 59.90) - open/close 1961 (time 1788587567.92 total 30.02 last 70.19) - open/close 2595 (time 1788587577.93 total 40.03 last 63.34) - open/close 3357 (time 1788587587.93 total 50.03 last 76.19) - open/close 4016 (time 1788587597.95 total 60.05 last 65.80) - open/close 4698 (time 1788587607.95 total 70.05 last 68.16) - open/close 5393 (time 1788587617.96 total 80.05 last 69.48) - open/close 6173 (time 1788587627.96 total 90.06 last 77.96) - open/close 6863 (time 1788587637.96 total 100.06 last 68.99) - open/close 7551 (time 1788587647.96 total 110.06 last 68.80) - open/close 8149 (time 1788587657.97 total 120.07 last 59.75) - open/close 8907 (time 1788587668.05 total 130.15 last 75.19) - open/close 9541 (time 1788587678.06 total 140.16 last 63.35) total: 10000 open/close in 149.23 seconds: 67.01 ops/second - unlinked 0 (time 1788587693 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 598 (time 1788587800.64 total 10.00 last 59.78) - open/close 1350 (time 1788587810.64 total 20.00 last 75.19) - open/close 1949 (time 1788587820.65 total 30.01 last 59.85) - open/close 2524 (time 1788587830.65 total 40.01 last 57.49) - open/close 3220 (time 1788587840.65 total 50.01 last 69.60) - open/close 3760 (time 1788587850.66 total 60.02 last 53.96) - open/close 4342 (time 1788587860.67 total 70.03 last 58.14) - open/close 5109 (time 1788587870.67 total 80.04 last 76.65) - open/close 5754 (time 1788587880.69 total 90.06 last 64.38) - open/close 6341 (time 1788587890.71 total 100.08 last 58.57) - open/close 6976 (time 1788587900.75 total 110.11 last 63.30) - open/close 7551 (time 1788587910.75 total 120.11 last 57.49) - open/close 8271 (time 1788587920.75 total 130.11 last 71.99) - open/close 8923 (time 1788587930.75 total 140.11 last 65.19) - open/close 9602 (time 1788587940.76 total 150.12 last 67.84) total: 10000 open/close in 155.56 seconds: 64.28 ops/second - unlinked 0 (time 1788587950 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 610 (time 1788588068.79 total 10.01 last 60.97) - open/close 1265 (time 1788588078.79 total 20.01 last 65.46) - open/close 1889 (time 1788588088.80 total 30.02 last 62.34) - open/close 2508 (time 1788588098.81 total 40.03 last 61.85) - open/close 3119 (time 1788588108.82 total 50.04 last 61.06) - open/close 3578 (time 1788588118.82 total 60.04 last 45.89) - open/close 4024 (time 1788588128.85 total 70.07 last 44.46) - open/close 4607 (time 1788588138.85 total 80.07 last 58.29) - open/close 5243 (time 1788588148.85 total 90.07 last 63.59) - open/close 5877 (time 1788588158.86 total 100.08 last 63.38) - open/close 6512 (time 1788588168.87 total 110.09 last 63.43) - open/close 6966 (time 1788588178.87 total 120.09 last 45.38) - open/close 7440 (time 1788588188.88 total 130.10 last 47.38) - open/close 8026 (time 1788588198.90 total 140.12 last 58.48) - open/close 8612 (time 1788588208.90 total 150.12 last 58.58) - open/close 9234 (time 1788588218.91 total 160.13 last 62.13) - open/close 9978 (time 1788588228.92 total 170.14 last 74.36) total: 10000 open/close in 170.57 seconds: 58.63 ops/second - unlinked 0 (time 1788588234 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 624 (time 1788588349.89 total 10.00 last 62.40) - open/close 1079 (time 1788588359.90 total 20.01 last 45.46) - open/close 1817 (time 1788588369.92 total 30.03 last 73.68) - open/close 2551 (time 1788588379.93 total 40.03 last 73.34) - open/close 3190 (time 1788588389.94 total 50.05 last 63.79) - open/close 4094 (time 1788588399.95 total 60.06 last 90.33) - open/close 4828 (time 1788588409.96 total 70.07 last 73.35) - open/close 5568 (time 1788588419.97 total 80.08 last 73.93) - open/close 6307 (time 1788588429.98 total 90.09 last 73.82) - open/close 7080 (time 1788588439.98 total 100.09 last 77.29) - open/close 7864 (time 1788588449.98 total 110.09 last 78.37) - open/close 8594 (time 1788588460.00 total 120.10 last 72.91) - open/close 9252 (time 1788588470.00 total 130.10 last 65.80) - open/close 9903 (time 1788588480.01 total 140.11 last 65.03) total: 10000 open/close in 141.98 seconds: 70.43 ops/second - unlinked 0 (time 1788588486 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 892 (time 1788588589.97 total 10.00 last 89.20) - open/close 1746 (time 1788588599.97 total 20.00 last 85.40) - open/close 2592 (time 1788588609.98 total 30.00 last 84.57) - open/close 3426 (time 1788588619.98 total 40.01 last 83.38) - open/close 4144 (time 1788588630.00 total 50.03 last 71.66) - open/close 4844 (time 1788588640.01 total 60.04 last 69.94) - open/close 5594 (time 1788588650.01 total 70.04 last 74.98) - open/close 6342 (time 1788588660.02 total 80.04 last 74.75) - open/close 7149 (time 1788588670.03 total 90.05 last 80.62) - open/close 7967 (time 1788588680.03 total 100.06 last 81.74) - open/close 8705 (time 1788588690.04 total 110.07 last 73.75) - open/close 9546 (time 1788588700.06 total 120.09 last 83.96) total: 10000 open/close in 126.26 seconds: 79.20 ops/second - unlinked 0 (time 1788588710 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 557 (time 1788588807.51 total 10.01 last 55.63) - open/close 1235 (time 1788588817.52 total 20.02 last 67.77) - open/close 1959 (time 1788588827.53 total 30.02 last 72.35) - open/close 2725 (time 1788588837.53 total 40.03 last 76.55) - open/close 3524 (time 1788588847.54 total 50.04 last 79.84) - open/close 4271 (time 1788588857.55 total 60.05 last 74.61) - open/close 5027 (time 1788588867.56 total 70.05 last 75.58) - open/close 5889 (time 1788588877.56 total 80.06 last 86.17) - open/close 6523 (time 1788588887.59 total 90.08 last 63.23) - open/close 7306 (time 1788588897.59 total 100.09 last 78.23) - open/close 8072 (time 1788588907.60 total 110.10 last 76.58) - open/close 8758 (time 1788588917.61 total 120.11 last 68.52) - open/close 9479 (time 1788588927.64 total 130.14 last 71.88) total: 10000 open/close in 136.89 seconds: 73.05 ops/second - unlinked 0 (time 1788588939 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 771 (time 1788589033.94 total 10.01 last 77.04) - open/close 1576 (time 1788589043.96 total 20.03 last 80.34) - open/close 2385 (time 1788589053.97 total 30.03 last 80.86) - open/close 3180 (time 1788589063.98 total 40.04 last 79.42) - open/close 3980 (time 1788589074.00 total 50.06 last 79.83) - open/close 4694 (time 1788589084.00 total 60.07 last 71.39) - open/close 5339 (time 1788589094.00 total 70.07 last 64.47) - open/close 5971 (time 1788589104.01 total 80.07 last 63.19) - open/close 6613 (time 1788589114.01 total 90.08 last 64.17) - open/close 7458 (time 1788589124.02 total 100.09 last 84.43) - open/close 8354 (time 1788589134.03 total 110.10 last 89.49) - open/close 9064 (time 1788589144.03 total 120.10 last 70.99) - open/close 9962 (time 1788589154.04 total 130.10 last 89.76) total: 10000 open/close in 130.48 seconds: 76.64 ops/second - unlinked 0 (time 1788589158 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 743 (time 1788589246.65 total 10.00 last 74.28) - open/close 1431 (time 1788589256.65 total 20.00 last 68.79) - open/close 2341 (time 1788589266.66 total 30.01 last 90.95) - open/close 3134 (time 1788589276.66 total 40.01 last 79.27) - open/close 3974 (time 1788589286.66 total 50.02 last 83.98) - open/close 4840 (time 1788589296.68 total 60.03 last 86.48) - open/close 5713 (time 1788589306.70 total 70.05 last 87.09) - open/close 6591 (time 1788589316.70 total 80.05 last 87.80) - open/close 7446 (time 1788589326.71 total 90.06 last 85.43) - open/close 8283 (time 1788589336.71 total 100.07 last 83.66) - open/close 9108 (time 1788589346.72 total 110.07 last 82.49) - open/close 9991 (time 1788589356.72 total 120.07 last 88.30) total: 10000 open/close in 120.16 seconds: 83.22 ops/second - unlinked 0 (time 1788589360 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 761 (time 1788589452.82 total 10.01 last 76.05) - open/close 1519 (time 1788589462.82 total 20.01 last 75.77) - open/close 2418 (time 1788589472.83 total 30.02 last 89.78) - open/close 3196 (time 1788589482.84 total 40.03 last 77.75) - open/close 4021 (time 1788589492.85 total 50.04 last 82.45) - open/close 4762 (time 1788589502.85 total 60.04 last 74.07) - open/close 5552 (time 1788589512.87 total 70.06 last 78.85) - open/close 6445 (time 1788589522.87 total 80.06 last 89.30) - open/close 7269 (time 1788589532.88 total 90.07 last 82.31) - open/close 8050 (time 1788589542.88 total 100.07 last 78.07) - open/close 8822 (time 1788589552.91 total 110.10 last 77.00) - open/close 9622 (time 1788589562.92 total 120.11 last 79.95) total: 10000 open/close in 125.03 seconds: 79.98 ops/second - unlinked 0 (time 1788589573 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 734 (time 1788589673.27 total 10.01 last 73.36) - open/close 1462 (time 1788589683.27 total 20.01 last 72.79) - open/close 2283 (time 1788589693.28 total 30.02 last 82.02) - open/close 3191 (time 1788589703.28 total 40.02 last 90.78) - open/close 4048 (time 1788589713.29 total 50.02 last 85.65) - open/close 4878 (time 1788589723.29 total 60.03 last 82.96) - open/close 5778 (time 1788589733.30 total 70.04 last 89.93) - open/close 6594 (time 1788589743.30 total 80.04 last 81.60) - open/close 7535 (time 1788589753.31 total 90.04 last 94.05) - open/close 8416 (time 1788589763.32 total 100.06 last 87.98) - open/close 9271 (time 1788589773.32 total 110.06 last 85.48) total: 10000 open/close in 118.59 seconds: 84.32 ops/second - unlinked 0 (time 1788589785 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.155@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.155@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg155-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg155-server: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg155-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.155@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.155@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.155@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.155@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.155@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg155-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg155-server: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg155-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server unloading modules via unload_modules_local on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing unload_modules_local oleg155-server: kmemleak disabled modules unloaded. pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2