== conf-sanity test 69: replace an OST with the same index ========================================================== 21:15:41 (1787793341) start mds service on oleg224-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1755968 Aug 26 20:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: kmemleak disabled oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg224-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 707 (time 1787793438.57 total 10.00 last 70.68) - open/close 1255 (time 1787793448.57 total 20.01 last 54.79) - open/close 2089 (time 1787793458.58 total 30.01 last 83.34) - open/close 2810 (time 1787793468.59 total 40.02 last 72.02) - open/close 3434 (time 1787793478.60 total 50.04 last 62.32) - open/close 4054 (time 1787793488.61 total 60.05 last 61.92) - open/close 4627 (time 1787793498.62 total 70.05 last 57.28) - open/close 5194 (time 1787793508.63 total 80.06 last 56.64) - open/close 5712 (time 1787793518.65 total 90.09 last 51.66) - open/close 6277 (time 1787793528.66 total 100.09 last 56.47) - open/close 6867 (time 1787793538.69 total 110.12 last 58.85) - open/close 7540 (time 1787793548.73 total 120.16 last 67.00) - open/close 8097 (time 1787793558.73 total 130.17 last 55.70) - open/close 8673 (time 1787793568.74 total 140.17 last 57.56) - open/close 9256 (time 1787793578.74 total 150.18 last 58.26) - open/close 9870 (time 1787793588.75 total 160.18 last 61.38) total: 10000 open/close in 162.12 seconds: 61.68 ops/second - unlinked 0 (time 1787793595 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 544 (time 1787793712.90 total 10.02 last 54.28) - open/close 982 (time 1787793722.90 total 20.02 last 43.80) - open/close 1559 (time 1787793732.91 total 30.03 last 57.66) - open/close 2189 (time 1787793742.92 total 40.04 last 62.94) - open/close 2736 (time 1787793752.95 total 50.07 last 54.51) - open/close 3471 (time 1787793762.96 total 60.08 last 73.45) - open/close 4037 (time 1787793772.97 total 70.09 last 56.56) - open/close 4611 (time 1787793782.97 total 80.09 last 57.39) - open/close 5169 (time 1787793792.99 total 90.11 last 55.66) - open/close 5660 (time 1787793803.00 total 100.12 last 49.06) - open/close 6224 (time 1787793813.00 total 110.12 last 56.40) - open/close 6878 (time 1787793823.01 total 120.13 last 65.34) - open/close 7637 (time 1787793833.02 total 130.14 last 75.86) - open/close 8286 (time 1787793843.02 total 140.14 last 64.89) - open/close 8887 (time 1787793853.02 total 150.14 last 60.10) - open/close 9631 (time 1787793863.02 total 160.14 last 74.38) total: 10000 open/close in 165.35 seconds: 60.48 ops/second - unlinked 0 (time 1787793873 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 641 (time 1787793977.67 total 10.01 last 64.05) - open/close 1279 (time 1787793987.68 total 20.02 last 63.70) - open/close 1978 (time 1787793997.68 total 30.03 last 69.89) - open/close 2642 (time 1787794007.69 total 40.03 last 66.34) - open/close 3394 (time 1787794017.69 total 50.04 last 75.18) - open/close 4033 (time 1787794027.70 total 60.04 last 63.88) - open/close 4735 (time 1787794037.70 total 70.04 last 70.19) - open/close 5435 (time 1787794047.70 total 80.04 last 69.99) - open/close 6221 (time 1787794057.72 total 90.06 last 78.49) - open/close 6953 (time 1787794067.73 total 100.07 last 73.12) - open/close 7759 (time 1787794077.73 total 110.07 last 80.55) - open/close 8351 (time 1787794087.74 total 120.08 last 59.15) - open/close 8978 (time 1787794097.74 total 130.08 last 62.69) - open/close 9577 (time 1787794107.75 total 140.09 last 59.85) total: 10000 open/close in 146.81 seconds: 68.12 ops/second - unlinked 0 (time 1787794119 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 709 (time 1787794230.94 total 10.02 last 70.79) - open/close 1375 (time 1787794240.94 total 20.02 last 66.59) - open/close 2131 (time 1787794250.96 total 30.04 last 75.45) - open/close 2773 (time 1787794260.97 total 40.05 last 64.14) - open/close 3381 (time 1787794270.98 total 50.06 last 60.74) - open/close 3979 (time 1787794280.99 total 60.07 last 59.72) - open/close 4664 (time 1787794290.99 total 70.07 last 68.50) - open/close 5314 (time 1787794301.00 total 80.08 last 64.94) - open/close 6034 (time 1787794311.01 total 90.09 last 71.90) - open/close 6718 (time 1787794321.03 total 100.11 last 68.30) - open/close 7357 (time 1787794331.03 total 110.11 last 63.88) - open/close 8051 (time 1787794341.03 total 120.11 last 69.39) - open/close 8707 (time 1787794351.04 total 130.12 last 65.52) - open/close 9286 (time 1787794361.06 total 140.14 last 57.83) - open/close 9907 (time 1787794371.06 total 150.14 last 62.09) total: 10000 open/close in 151.44 seconds: 66.03 ops/second - unlinked 0 (time 1787794376 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 677 (time 1787794491.55 total 10.02 last 67.55) - open/close 1355 (time 1787794501.56 total 20.03 last 67.76) - open/close 2099 (time 1787794511.57 total 30.04 last 74.34) - open/close 2860 (time 1787794521.58 total 40.04 last 76.03) - open/close 3573 (time 1787794531.59 total 50.06 last 71.22) - open/close 4212 (time 1787794541.61 total 60.08 last 63.76) - open/close 4813 (time 1787794551.62 total 70.08 last 60.06) - open/close 5483 (time 1787794561.63 total 80.10 last 66.93) - open/close 6093 (time 1787794571.63 total 90.10 last 60.98) - open/close 6914 (time 1787794581.63 total 100.10 last 82.08) - open/close 7732 (time 1787794591.64 total 110.11 last 81.76) - open/close 8545 (time 1787794601.65 total 120.12 last 81.16) - open/close 9316 (time 1787794611.66 total 130.13 last 77.06) - open/close 9821 (time 1787794621.67 total 140.14 last 50.46) total: 10000 open/close in 143.44 seconds: 69.72 ops/second - unlinked 0 (time 1787794629 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 746 (time 1787794733.77 total 10.01 last 74.52) - open/close 1391 (time 1787794743.77 total 20.01 last 64.48) - open/close 2054 (time 1787794753.78 total 30.02 last 66.24) - open/close 2732 (time 1787794763.81 total 40.05 last 67.62) - open/close 3388 (time 1787794773.82 total 50.06 last 65.51) - open/close 4142 (time 1787794783.83 total 60.07 last 75.31) - open/close 4687 (time 1787794793.84 total 70.08 last 54.45) - open/close 5179 (time 1787794803.85 total 80.09 last 49.15) - open/close 5860 (time 1787794813.86 total 90.10 last 68.03) - open/close 6469 (time 1787794823.88 total 100.12 last 60.81) - open/close 7203 (time 1787794833.88 total 110.12 last 73.38) - open/close 7874 (time 1787794843.89 total 120.13 last 67.03) - open/close 8462 (time 1787794853.89 total 130.14 last 58.77) - open/close 9216 (time 1787794863.90 total 140.15 last 75.33) - open/close 9819 (time 1787794873.91 total 150.15 last 60.27) total: 10000 open/close in 152.45 seconds: 65.59 ops/second - unlinked 0 (time 1787794880 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 701 (time 1787794987.52 total 10.00 last 70.09) - open/close 1388 (time 1787794997.53 total 20.02 last 68.59) - open/close 2182 (time 1787795007.54 total 30.02 last 79.36) - open/close 2806 (time 1787795017.55 total 40.04 last 62.29) - open/close 3592 (time 1787795027.57 total 50.05 last 78.51) - open/close 4367 (time 1787795037.58 total 60.06 last 77.42) - open/close 5084 (time 1787795047.58 total 70.07 last 71.66) - open/close 5741 (time 1787795057.59 total 80.07 last 65.67) - open/close 6359 (time 1787795067.59 total 90.07 last 61.79) - open/close 7013 (time 1787795077.59 total 100.08 last 65.37) - open/close 7703 (time 1787795087.60 total 110.08 last 68.97) - open/close 8440 (time 1787795097.60 total 120.09 last 73.67) - open/close 9062 (time 1787795107.61 total 130.10 last 62.15) - open/close 9808 (time 1787795117.62 total 140.10 last 74.55) total: 10000 open/close in 142.49 seconds: 70.18 ops/second - unlinked 0 (time 1787795124 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 654 (time 1787795235.15 total 10.00 last 65.38) - open/close 1386 (time 1787795245.15 total 20.01 last 73.16) - open/close 2097 (time 1787795255.16 total 30.02 last 71.04) - open/close 2786 (time 1787795265.16 total 40.02 last 68.90) - open/close 3560 (time 1787795275.17 total 50.03 last 77.33) - open/close 4224 (time 1787795285.18 total 60.03 last 66.35) - open/close 4842 (time 1787795295.18 total 70.04 last 61.77) - open/close 5534 (time 1787795305.20 total 80.06 last 69.04) - open/close 6118 (time 1787795315.21 total 90.07 last 58.35) - open/close 6719 (time 1787795325.23 total 100.09 last 59.98) - open/close 7283 (time 1787795335.24 total 110.10 last 56.36) - open/close 7933 (time 1787795345.24 total 120.10 last 65.00) - open/close 8743 (time 1787795355.25 total 130.10 last 80.94) - open/close 9466 (time 1787795365.26 total 140.12 last 72.19) total: 10000 open/close in 149.11 seconds: 67.07 ops/second - unlinked 0 (time 1787795379 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 711 (time 1787795485.38 total 10.01 last 71.06) - open/close 1397 (time 1787795495.39 total 20.01 last 68.55) - open/close 2109 (time 1787795505.39 total 30.02 last 71.17) - open/close 2765 (time 1787795515.40 total 40.02 last 65.59) - open/close 3406 (time 1787795525.40 total 50.02 last 64.06) - open/close 4062 (time 1787795535.45 total 60.07 last 65.29) - open/close 4665 (time 1787795545.45 total 70.07 last 60.30) - open/close 5373 (time 1787795555.45 total 80.08 last 70.76) - open/close 6116 (time 1787795565.48 total 90.10 last 74.12) - open/close 6830 (time 1787795575.48 total 100.10 last 71.39) - open/close 7611 (time 1787795585.49 total 110.11 last 78.05) - open/close 8311 (time 1787795595.49 total 120.11 last 69.99) - open/close 9053 (time 1787795605.51 total 130.13 last 74.08) - open/close 9693 (time 1787795615.51 total 140.13 last 63.99) total: 10000 open/close in 144.24 seconds: 69.33 ops/second - unlinked 0 (time 1787795624 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 618 (time 1787795739.21 total 10.01 last 61.73) - open/close 1289 (time 1787795749.22 total 20.02 last 67.05) - open/close 2007 (time 1787795759.23 total 30.02 last 71.76) - open/close 2578 (time 1787795769.24 total 40.04 last 57.02) - open/close 3196 (time 1787795779.25 total 50.05 last 61.75) - open/close 3877 (time 1787795789.26 total 60.06 last 68.03) - open/close 4586 (time 1787795799.27 total 70.07 last 70.81) - open/close 5341 (time 1787795809.28 total 80.08 last 75.43) - open/close 5952 (time 1787795819.29 total 90.08 last 61.09) - open/close 6568 (time 1787795829.29 total 100.09 last 61.57) - open/close 7279 (time 1787795839.30 total 110.10 last 71.01) - open/close 7969 (time 1787795849.31 total 120.11 last 68.92) - open/close 8643 (time 1787795859.32 total 130.12 last 67.35) - open/close 9391 (time 1787795869.33 total 140.12 last 74.76) total: 10000 open/close in 147.68 seconds: 67.72 ops/second - unlinked 0 (time 1787795880 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.124@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg224-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre On OST0, 16454 used inodes rc=0 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.124@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.124@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg224-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: kmemleak disabled modules unloaded. pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2