== conf-sanity test 69: replace an OST with the same index ========================================================== 08:07:15 (1788264435) start mds service on oleg652-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1756096 Sep 1 06:05 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg652-server' oleg652-server: oleg652-server.virtnet: executing load_modules_local oleg652-server: kmemleak disabled oleg652-server: Loading modules from /home/green/git/lustre-release/lustre oleg652-server: detected 4 online CPUs by sysfs oleg652-server: MODOPTS_LIBCFS= oleg652-server: Force libcfs to create 2 CPU partitions oleg652-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg652-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg652-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg652-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg652-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-OST0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg652-client.virtnet: -o user_xattr,flock 192.168.206.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 608 (time 1788264522.08 total 10.03 last 60.64) - open/close 1191 (time 1788264532.08 total 20.03 last 58.28) - open/close 1654 (time 1788264542.09 total 30.03 last 46.28) - open/close 2240 (time 1788264552.10 total 40.04 last 58.54) - open/close 2868 (time 1788264562.12 total 50.07 last 62.67) - open/close 3573 (time 1788264572.13 total 60.08 last 70.40) - open/close 4250 (time 1788264582.15 total 70.10 last 67.55) - open/close 4895 (time 1788264592.15 total 80.10 last 64.49) - open/close 5546 (time 1788264602.16 total 90.11 last 65.05) - open/close 6181 (time 1788264612.17 total 100.12 last 63.44) - open/close 6862 (time 1788264622.20 total 110.15 last 67.92) - open/close 7557 (time 1788264632.20 total 120.15 last 69.47) - open/close 8295 (time 1788264642.21 total 130.16 last 73.76) - open/close 8946 (time 1788264652.22 total 140.17 last 65.03) - open/close 9613 (time 1788264662.22 total 150.17 last 66.69) total: 10000 open/close in 156.24 seconds: 64.00 ops/second - unlinked 0 (time 1788264672 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 702 (time 1788264781.57 total 10.00 last 70.18) - open/close 1395 (time 1788264791.58 total 20.01 last 69.25) - open/close 2150 (time 1788264801.59 total 30.02 last 75.45) - open/close 2834 (time 1788264811.61 total 40.04 last 68.25) - open/close 3527 (time 1788264821.62 total 50.05 last 69.19) - open/close 4228 (time 1788264831.64 total 60.07 last 70.02) - open/close 4958 (time 1788264841.64 total 70.07 last 73.00) - open/close 5678 (time 1788264851.64 total 80.07 last 71.96) - open/close 6367 (time 1788264861.65 total 90.08 last 68.85) - open/close 7017 (time 1788264871.66 total 100.09 last 64.94) - open/close 7683 (time 1788264881.67 total 110.10 last 66.56) - open/close 8409 (time 1788264891.67 total 120.10 last 72.58) - open/close 9108 (time 1788264901.67 total 130.10 last 69.88) - open/close 9843 (time 1788264911.68 total 140.11 last 73.42) total: 10000 open/close in 142.25 seconds: 70.30 ops/second - unlinked 0 (time 1788264917 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 749 (time 1788265014.15 total 10.00 last 74.89) - open/close 1389 (time 1788265024.16 total 20.02 last 63.91) - open/close 2097 (time 1788265034.17 total 30.03 last 70.71) - open/close 2773 (time 1788265044.17 total 40.03 last 67.59) - open/close 3474 (time 1788265054.18 total 50.04 last 70.04) - open/close 4098 (time 1788265064.19 total 60.05 last 62.34) - open/close 4741 (time 1788265074.21 total 70.06 last 64.21) - open/close 5282 (time 1788265084.21 total 80.06 last 54.09) - open/close 5870 (time 1788265094.22 total 90.07 last 58.75) - open/close 6521 (time 1788265104.22 total 100.07 last 65.09) - open/close 7170 (time 1788265114.23 total 110.08 last 64.83) - open/close 7895 (time 1788265124.24 total 120.09 last 72.44) - open/close 8600 (time 1788265134.24 total 130.10 last 70.47) - open/close 9317 (time 1788265144.25 total 140.10 last 71.67) total: 10000 open/close in 149.65 seconds: 66.82 ops/second - unlinked 0 (time 1788265157 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 735 (time 1788265252.38 total 10.00 last 73.47) - open/close 1451 (time 1788265262.39 total 20.02 last 71.50) - open/close 2217 (time 1788265272.40 total 30.03 last 76.54) - open/close 2924 (time 1788265282.41 total 40.04 last 70.63) - open/close 3747 (time 1788265292.41 total 50.04 last 82.30) - open/close 4484 (time 1788265302.42 total 60.05 last 73.64) - open/close 5251 (time 1788265312.42 total 70.05 last 76.68) - open/close 5973 (time 1788265322.43 total 80.06 last 72.13) - open/close 6710 (time 1788265332.44 total 90.06 last 73.66) - open/close 7420 (time 1788265342.45 total 100.07 last 70.94) - open/close 8171 (time 1788265352.45 total 110.08 last 75.05) - open/close 8888 (time 1788265362.46 total 120.09 last 71.64) - open/close 9589 (time 1788265372.46 total 130.09 last 70.08) total: 10000 open/close in 135.71 seconds: 73.69 ops/second - unlinked 0 (time 1788265382 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 762 (time 1788265480.01 total 10.03 last 76.00) - open/close 1555 (time 1788265490.01 total 20.03 last 79.30) - open/close 2352 (time 1788265500.01 total 30.03 last 79.68) - open/close 3127 (time 1788265510.02 total 40.04 last 77.43) - open/close 3922 (time 1788265520.02 total 50.05 last 79.44) - open/close 4677 (time 1788265530.03 total 60.05 last 75.44) - open/close 5458 (time 1788265540.05 total 70.07 last 77.95) - open/close 6222 (time 1788265550.06 total 80.08 last 76.32) - open/close 6983 (time 1788265560.07 total 90.09 last 76.02) - open/close 7773 (time 1788265570.10 total 100.12 last 78.77) - open/close 8441 (time 1788265580.11 total 110.14 last 66.72) - open/close 9083 (time 1788265590.12 total 120.14 last 64.18) - open/close 9728 (time 1788265600.13 total 130.15 last 64.42) total: 10000 open/close in 133.96 seconds: 74.65 ops/second - unlinked 0 (time 1788265608 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 739 (time 1788265709.03 total 10.01 last 73.79) - open/close 1470 (time 1788265719.03 total 20.02 last 73.06) - open/close 2160 (time 1788265729.04 total 30.03 last 68.96) - open/close 2887 (time 1788265739.05 total 40.04 last 72.62) - open/close 3630 (time 1788265749.05 total 50.04 last 74.29) - open/close 4322 (time 1788265759.05 total 60.04 last 69.17) - open/close 4975 (time 1788265769.07 total 70.06 last 65.20) - open/close 5621 (time 1788265779.08 total 80.06 last 64.56) - open/close 6295 (time 1788265789.09 total 90.07 last 67.33) - open/close 6995 (time 1788265799.09 total 100.08 last 69.98) - open/close 7702 (time 1788265809.10 total 110.09 last 70.64) - open/close 8357 (time 1788265819.11 total 120.10 last 65.41) - open/close 9007 (time 1788265829.12 total 130.10 last 64.97) - open/close 9693 (time 1788265839.13 total 140.11 last 68.54) total: 10000 open/close in 144.60 seconds: 69.15 ops/second - unlinked 0 (time 1788265847 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 757 (time 1788265944.92 total 10.01 last 75.62) - open/close 1507 (time 1788265954.93 total 20.02 last 74.93) - open/close 2247 (time 1788265964.93 total 30.02 last 73.99) - open/close 2960 (time 1788265974.94 total 40.04 last 71.19) - open/close 3653 (time 1788265984.96 total 50.05 last 69.19) - open/close 4347 (time 1788265994.97 total 60.06 last 69.34) - open/close 4972 (time 1788266004.97 total 70.07 last 62.46) - open/close 5553 (time 1788266014.97 total 80.07 last 58.10) - open/close 6193 (time 1788266024.99 total 90.08 last 63.92) - open/close 6699 (time 1788266035.00 total 100.09 last 50.53) - open/close 7382 (time 1788266045.00 total 110.10 last 68.28) - open/close 8112 (time 1788266055.01 total 120.10 last 72.98) - open/close 8832 (time 1788266065.01 total 130.11 last 71.93) - open/close 9567 (time 1788266075.04 total 140.14 last 73.29) total: 10000 open/close in 145.80 seconds: 68.58 ops/second - unlinked 0 (time 1788266084 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 748 (time 1788266193.43 total 10.01 last 74.74) - open/close 1478 (time 1788266203.44 total 20.01 last 72.99) - open/close 2205 (time 1788266213.44 total 30.01 last 72.69) - open/close 2931 (time 1788266223.44 total 40.02 last 72.56) - open/close 3663 (time 1788266233.45 total 50.02 last 73.15) - open/close 4352 (time 1788266243.45 total 60.03 last 68.86) - open/close 5127 (time 1788266253.46 total 70.03 last 77.46) - open/close 5875 (time 1788266263.47 total 80.04 last 74.75) - open/close 6623 (time 1788266273.48 total 90.05 last 74.70) - open/close 7306 (time 1788266283.49 total 100.06 last 68.24) - open/close 7961 (time 1788266293.49 total 110.07 last 65.46) - open/close 8608 (time 1788266303.51 total 120.08 last 64.62) - open/close 9284 (time 1788266313.51 total 130.08 last 67.57) - open/close 9943 (time 1788266323.52 total 140.10 last 65.83) total: 10000 open/close in 140.71 seconds: 71.07 ops/second - unlinked 0 (time 1788266328 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 633 (time 1788266427.58 total 10.00 last 63.30) - open/close 1316 (time 1788266437.59 total 20.00 last 68.27) - open/close 1945 (time 1788266447.59 total 30.01 last 62.89) - open/close 2787 (time 1788266457.60 total 40.02 last 84.12) - open/close 3502 (time 1788266467.60 total 50.02 last 71.49) - open/close 4241 (time 1788266477.62 total 60.03 last 73.78) - open/close 4966 (time 1788266487.63 total 70.04 last 72.43) - open/close 5678 (time 1788266497.63 total 80.05 last 71.17) - open/close 6411 (time 1788266507.63 total 90.05 last 73.27) - open/close 7149 (time 1788266517.64 total 100.06 last 73.73) - open/close 7890 (time 1788266527.66 total 110.07 last 74.01) - open/close 8621 (time 1788266537.68 total 120.10 last 72.92) - open/close 9402 (time 1788266547.69 total 130.10 last 78.06) total: 10000 open/close in 138.64 seconds: 72.13 ops/second - unlinked 0 (time 1788266559 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 767 (time 1788266648.33 total 10.00 last 76.68) - open/close 1509 (time 1788266658.34 total 20.01 last 74.11) - open/close 2345 (time 1788266668.35 total 30.02 last 83.56) - open/close 3171 (time 1788266678.35 total 40.02 last 82.57) - open/close 3963 (time 1788266688.36 total 50.03 last 79.16) - open/close 4791 (time 1788266698.36 total 60.03 last 82.79) - open/close 5604 (time 1788266708.37 total 70.04 last 81.24) - open/close 6560 (time 1788266718.37 total 80.04 last 95.57) - open/close 7435 (time 1788266728.37 total 90.04 last 87.48) - open/close 8193 (time 1788266738.39 total 100.06 last 75.68) - open/close 8952 (time 1788266748.40 total 110.07 last 75.82) - open/close 9745 (time 1788266758.40 total 120.07 last 79.27) total: 10000 open/close in 123.37 seconds: 81.06 ops/second - unlinked 0 (time 1788266765 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg652-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg652-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg652-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.152@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.152@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg652-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-OST0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg652-server: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg652-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg652-client.virtnet: -o user_xattr,flock 192.168.206.152@tcp:/lustre /mnt/lustre On OST0, 814 used inodes rc=0 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg652-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg652-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg652-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.152@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.152@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.206.152@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.206.152@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.206.152@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg652-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-OST0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg652-server: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg652-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg652-client.virtnet: -o user_xattr,flock 192.168.206.152@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg652-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg652-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg652-server stop mds service on oleg652-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg652-server unloading modules via unload_modules_local on: 'oleg652-server' oleg652-server: oleg652-server.virtnet: executing unload_modules_local oleg652-server: kmemleak disabled modules unloaded. pdsh@oleg652-client: oleg652-server: ssh exited with exit code 2 pdsh@oleg652-client: oleg652-client: ssh exited with exit code 2 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 2 pdsh@oleg652-client: oleg652-client: ssh exited with exit code 2