== conf-sanity test 69: replace an OST with the same index ========================================================== 10:18:24 (1789049904) start mds service on oleg109-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931568 Sep 10 01:50 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: kmemleak disabled oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: detected 4 online CPUs by sysfs oleg109-server: MODOPTS_LIBCFS= oleg109-server: Force libcfs to create 2 CPU partitions oleg109-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg109-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg109-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg109-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg109-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 574 (time 1789050006.02 total 10.01 last 57.36) - open/close 1336 (time 1789050016.03 total 20.01 last 76.14) - open/close 2172 (time 1789050026.03 total 30.02 last 83.56) - open/close 2801 (time 1789050036.05 total 40.03 last 62.80) - open/close 3501 (time 1789050046.05 total 50.04 last 69.97) - open/close 4240 (time 1789050056.06 total 60.04 last 73.87) - open/close 5000 (time 1789050066.07 total 70.06 last 75.87) - open/close 5757 (time 1789050076.08 total 80.07 last 75.63) - open/close 6380 (time 1789050086.09 total 90.07 last 62.27) - open/close 7095 (time 1789050096.09 total 100.07 last 71.50) - open/close 7810 (time 1789050106.09 total 110.07 last 71.49) - open/close 8464 (time 1789050116.09 total 120.08 last 65.38) - open/close 9178 (time 1789050126.09 total 130.08 last 71.39) - open/close 9908 (time 1789050136.10 total 140.09 last 72.93) total: 10000 open/close in 141.53 seconds: 70.66 ops/second - unlinked 0 (time 1789050141 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 605 (time 1789050246.31 total 10.00 last 60.49) - open/close 1295 (time 1789050256.32 total 20.01 last 68.93) - open/close 2020 (time 1789050266.32 total 30.02 last 72.47) - open/close 2700 (time 1789050276.33 total 40.02 last 67.95) - open/close 3390 (time 1789050286.34 total 50.04 last 68.91) - open/close 4130 (time 1789050296.35 total 60.05 last 73.91) - open/close 4811 (time 1789050306.36 total 70.06 last 68.05) - open/close 5574 (time 1789050316.37 total 80.06 last 76.28) - open/close 6359 (time 1789050326.37 total 90.06 last 78.46) - open/close 7048 (time 1789050336.37 total 100.07 last 68.90) - open/close 7774 (time 1789050346.41 total 110.10 last 72.35) - open/close 8347 (time 1789050356.41 total 120.10 last 57.28) - open/close 9021 (time 1789050366.42 total 130.12 last 67.31) - open/close 9799 (time 1789050376.43 total 140.12 last 77.76) total: 10000 open/close in 142.77 seconds: 70.04 ops/second - unlinked 0 (time 1789050384 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 684 (time 1789050486.37 total 10.03 last 68.18) - open/close 1270 (time 1789050496.38 total 20.04 last 58.56) - open/close 1878 (time 1789050506.39 total 30.06 last 60.69) - open/close 2543 (time 1789050516.40 total 40.06 last 66.48) - open/close 3230 (time 1789050526.41 total 50.07 last 68.64) - open/close 3791 (time 1789050536.42 total 60.08 last 56.04) - open/close 4409 (time 1789050546.43 total 70.09 last 61.74) - open/close 5049 (time 1789050556.44 total 80.10 last 63.92) - open/close 5702 (time 1789050566.46 total 90.12 last 65.17) - open/close 6486 (time 1789050576.47 total 100.13 last 78.35) - open/close 7225 (time 1789050586.48 total 110.14 last 73.79) - open/close 7857 (time 1789050596.48 total 120.14 last 63.20) - open/close 8515 (time 1789050606.50 total 130.16 last 65.68) - open/close 9042 (time 1789050616.51 total 140.17 last 52.65) - open/close 9699 (time 1789050626.52 total 150.19 last 65.60) total: 10000 open/close in 154.49 seconds: 64.73 ops/second - unlinked 0 (time 1789050635 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 714 (time 1789050740.52 total 10.01 last 71.35) - open/close 1426 (time 1789050750.53 total 20.01 last 71.16) - open/close 2136 (time 1789050760.55 total 30.03 last 70.85) - open/close 2691 (time 1789050770.56 total 40.05 last 55.41) - open/close 3341 (time 1789050780.58 total 50.07 last 64.88) - open/close 4160 (time 1789050790.60 total 60.08 last 81.78) - open/close 4862 (time 1789050800.60 total 70.09 last 70.16) - open/close 5612 (time 1789050810.62 total 80.10 last 74.88) - open/close 6150 (time 1789050820.62 total 90.11 last 53.78) - open/close 6721 (time 1789050830.64 total 100.13 last 56.99) - open/close 7366 (time 1789050840.67 total 110.16 last 64.32) - open/close 8197 (time 1789050850.68 total 120.16 last 83.04) - open/close 8891 (time 1789050860.68 total 130.16 last 69.40) - open/close 9518 (time 1789050870.70 total 140.18 last 62.60) total: 10000 open/close in 147.76 seconds: 67.68 ops/second - unlinked 0 (time 1789050883 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 701 (time 1789050986.52 total 10.00 last 70.08) - open/close 1441 (time 1789050996.52 total 20.01 last 73.96) - open/close 2275 (time 1789051006.53 total 30.02 last 83.30) - open/close 3069 (time 1789051016.54 total 40.03 last 79.34) - open/close 3895 (time 1789051026.55 total 50.04 last 82.49) - open/close 4637 (time 1789051036.56 total 60.05 last 74.16) - open/close 5361 (time 1789051046.58 total 70.07 last 72.23) - open/close 5933 (time 1789051056.59 total 80.08 last 57.13) - open/close 6758 (time 1789051066.60 total 90.09 last 82.43) - open/close 7559 (time 1789051076.61 total 100.09 last 80.07) - open/close 8302 (time 1789051086.62 total 110.10 last 74.22) - open/close 9052 (time 1789051096.62 total 120.11 last 74.96) - open/close 9773 (time 1789051106.63 total 130.11 last 72.08) total: 10000 open/close in 132.97 seconds: 75.21 ops/second - unlinked 0 (time 1789051113 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 707 (time 1789051212.80 total 10.00 last 70.69) - open/close 1451 (time 1789051222.80 total 20.00 last 74.40) - open/close 2152 (time 1789051232.80 total 30.01 last 70.07) - open/close 2900 (time 1789051242.81 total 40.01 last 74.78) - open/close 3664 (time 1789051252.83 total 50.03 last 76.23) - open/close 4329 (time 1789051262.83 total 60.03 last 66.50) - open/close 4950 (time 1789051272.83 total 70.03 last 62.10) - open/close 5635 (time 1789051282.84 total 80.04 last 68.45) - open/close 6406 (time 1789051292.85 total 90.05 last 77.01) - open/close 7204 (time 1789051302.85 total 100.06 last 79.78) - open/close 7937 (time 1789051312.86 total 110.06 last 73.25) - open/close 8660 (time 1789051322.87 total 120.07 last 72.25) - open/close 9395 (time 1789051332.88 total 130.08 last 73.43) total: 10000 open/close in 139.56 seconds: 71.65 ops/second - unlinked 0 (time 1789051347 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 706 (time 1789051446.76 total 10.01 last 70.52) - open/close 1429 (time 1789051456.77 total 20.02 last 72.23) - open/close 2286 (time 1789051466.78 total 30.03 last 85.65) - open/close 3086 (time 1789051476.79 total 40.04 last 79.89) - open/close 3756 (time 1789051486.80 total 50.05 last 66.94) - open/close 4404 (time 1789051496.81 total 60.05 last 64.77) - open/close 5027 (time 1789051506.82 total 70.07 last 62.20) - open/close 5698 (time 1789051516.83 total 80.08 last 67.03) - open/close 6540 (time 1789051526.85 total 90.09 last 84.09) - open/close 7237 (time 1789051536.85 total 100.10 last 69.66) - open/close 7943 (time 1789051546.86 total 110.11 last 70.55) - open/close 8599 (time 1789051556.87 total 120.12 last 65.54) - open/close 9344 (time 1789051566.88 total 130.12 last 74.45) total: 10000 open/close in 138.27 seconds: 72.32 ops/second - unlinked 0 (time 1789051579 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 643 (time 1789051690.06 total 10.00 last 64.27) - open/close 1372 (time 1789051700.09 total 20.03 last 72.68) - open/close 2048 (time 1789051710.09 total 30.04 last 67.60) - open/close 2802 (time 1789051720.09 total 40.04 last 75.36) - open/close 3426 (time 1789051730.09 total 50.04 last 62.40) - open/close 4113 (time 1789051740.11 total 60.06 last 68.60) - open/close 4923 (time 1789051750.12 total 70.06 last 80.95) - open/close 5655 (time 1789051760.13 total 80.08 last 73.06) - open/close 6420 (time 1789051770.14 total 90.09 last 76.43) - open/close 7202 (time 1789051780.16 total 100.10 last 78.11) - open/close 7970 (time 1789051790.17 total 110.11 last 76.73) - open/close 8812 (time 1789051800.17 total 120.12 last 84.16) - open/close 9546 (time 1789051810.17 total 130.12 last 73.39) total: 10000 open/close in 136.00 seconds: 73.53 ops/second - unlinked 0 (time 1789051820 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 715 (time 1789051920.09 total 10.02 last 71.37) - open/close 1336 (time 1789051930.09 total 20.03 last 62.05) - open/close 2042 (time 1789051940.10 total 30.03 last 70.58) - open/close 2778 (time 1789051950.11 total 40.04 last 73.52) - open/close 3538 (time 1789051960.11 total 50.04 last 75.98) - open/close 4350 (time 1789051970.12 total 60.06 last 81.09) - open/close 5097 (time 1789051980.13 total 70.07 last 74.63) - open/close 5726 (time 1789051990.14 total 80.07 last 62.84) - open/close 6505 (time 1789052000.15 total 90.08 last 77.87) - open/close 7253 (time 1789052010.15 total 100.08 last 74.80) - open/close 8038 (time 1789052020.16 total 110.09 last 78.42) - open/close 8789 (time 1789052030.17 total 120.10 last 75.03) - open/close 9489 (time 1789052040.18 total 130.11 last 69.90) total: 10000 open/close in 137.78 seconds: 72.58 ops/second - unlinked 0 (time 1789052052 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 625 (time 1789052161.20 total 10.00 last 62.49) - open/close 1338 (time 1789052171.20 total 20.00 last 71.29) - open/close 2080 (time 1789052181.20 total 30.01 last 74.15) - open/close 2836 (time 1789052191.21 total 40.01 last 75.58) - open/close 3683 (time 1789052201.21 total 50.01 last 84.68) - open/close 4583 (time 1789052211.22 total 60.02 last 89.94) - open/close 5304 (time 1789052221.22 total 70.02 last 72.07) - open/close 6073 (time 1789052231.22 total 80.03 last 76.87) - open/close 6815 (time 1789052241.23 total 90.04 last 74.14) - open/close 7464 (time 1789052251.23 total 100.04 last 64.90) - open/close 8137 (time 1789052261.24 total 110.04 last 67.28) - open/close 8718 (time 1789052271.25 total 120.06 last 58.01) - open/close 9420 (time 1789052281.27 total 130.07 last 70.10) - open/close 9802 (time 1789052291.27 total 140.08 last 38.18) total: 10000 open/close in 144.76 seconds: 69.08 ops/second - unlinked 0 (time 1789052300 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.109@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg109-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg109-server: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg109-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.109@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.109@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.109@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.109@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg109-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg109-server: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg109-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server unloading modules via unload_modules_local on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing unload_modules_local oleg109-server: kmemleak disabled modules unloaded. pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2