== conf-sanity test 69: replace an OST with the same index ========================================================== 19:26:45 (1789428405) start mds service on oleg203-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 11:26 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: kmemleak disabled oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg203-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg203-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg203-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 850 (time 1789428482.92 total 10.01 last 84.94) - open/close 1589 (time 1789428492.93 total 20.01 last 73.85) - open/close 2414 (time 1789428502.93 total 30.02 last 82.44) - open/close 3166 (time 1789428512.95 total 40.03 last 75.10) - open/close 3928 (time 1789428522.95 total 50.04 last 76.18) - open/close 4888 (time 1789428532.95 total 60.04 last 95.96) - open/close 5655 (time 1789428542.96 total 70.05 last 76.63) - open/close 6432 (time 1789428552.96 total 80.05 last 77.67) - open/close 7240 (time 1789428562.97 total 90.06 last 80.76) - open/close 8062 (time 1789428572.97 total 100.06 last 82.20) - open/close 8837 (time 1789428582.97 total 110.06 last 77.49) - open/close 9613 (time 1789428592.98 total 120.07 last 77.55) total: 10000 open/close in 124.92 seconds: 80.05 ops/second - unlinked 0 (time 1789428602 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 711 (time 1789428697.98 total 10.01 last 71.03) - open/close 1475 (time 1789428707.98 total 20.01 last 76.39) - open/close 2210 (time 1789428717.99 total 30.01 last 73.48) - open/close 2913 (time 1789428727.99 total 40.02 last 70.26) - open/close 3675 (time 1789428738.05 total 50.08 last 75.73) - open/close 4396 (time 1789428748.06 total 60.09 last 72.03) - open/close 5284 (time 1789428758.10 total 70.13 last 88.47) - open/close 6067 (time 1789428768.10 total 80.13 last 78.29) - open/close 6874 (time 1789428778.12 total 90.15 last 80.57) - open/close 7634 (time 1789428788.12 total 100.15 last 75.97) - open/close 8362 (time 1789428798.14 total 110.17 last 72.68) - open/close 9105 (time 1789428808.15 total 120.18 last 74.24) - open/close 9754 (time 1789428818.16 total 130.18 last 64.84) total: 10000 open/close in 132.90 seconds: 75.25 ops/second - unlinked 0 (time 1789428824 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 668 (time 1789428922.08 total 10.01 last 66.76) - open/close 1410 (time 1789428932.09 total 20.01 last 74.15) - open/close 2161 (time 1789428942.10 total 30.02 last 75.06) - open/close 2863 (time 1789428952.10 total 40.02 last 70.18) - open/close 3683 (time 1789428962.11 total 50.03 last 81.93) - open/close 4467 (time 1789428972.12 total 60.04 last 78.32) - open/close 5168 (time 1789428982.13 total 70.05 last 70.04) - open/close 5962 (time 1789428992.13 total 80.05 last 79.35) - open/close 6694 (time 1789429002.14 total 90.06 last 73.15) - open/close 7400 (time 1789429012.14 total 100.07 last 70.57) - open/close 8173 (time 1789429022.15 total 110.07 last 77.29) - open/close 8846 (time 1789429032.16 total 120.08 last 67.20) - open/close 9572 (time 1789429042.17 total 130.09 last 72.54) total: 10000 open/close in 136.00 seconds: 73.53 ops/second - unlinked 0 (time 1789429051 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 760 (time 1789429151.46 total 10.00 last 75.97) - open/close 1537 (time 1789429161.47 total 20.02 last 77.56) - open/close 2320 (time 1789429171.48 total 30.03 last 78.27) - open/close 3164 (time 1789429181.48 total 40.03 last 84.36) - open/close 4002 (time 1789429191.48 total 50.03 last 83.79) - open/close 4755 (time 1789429201.48 total 60.03 last 75.29) - open/close 5535 (time 1789429211.50 total 70.05 last 77.90) - open/close 6269 (time 1789429221.50 total 80.05 last 73.35) - open/close 6988 (time 1789429231.51 total 90.06 last 71.83) - open/close 7637 (time 1789429241.51 total 100.06 last 64.90) - open/close 8348 (time 1789429251.53 total 110.07 last 71.02) - open/close 9102 (time 1789429261.53 total 120.08 last 75.35) - open/close 9901 (time 1789429271.54 total 130.08 last 79.88) total: 10000 open/close in 131.31 seconds: 76.15 ops/second - unlinked 0 (time 1789429276 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 837 (time 1789429361.89 total 10.01 last 83.63) - open/close 1555 (time 1789429371.90 total 20.02 last 71.74) - open/close 2374 (time 1789429381.91 total 30.03 last 81.83) - open/close 3254 (time 1789429391.91 total 40.03 last 87.98) - open/close 4151 (time 1789429401.91 total 50.03 last 89.69) - open/close 4989 (time 1789429411.92 total 60.04 last 83.75) - open/close 5835 (time 1789429421.92 total 70.04 last 84.57) - open/close 6606 (time 1789429431.95 total 80.07 last 76.87) - open/close 7410 (time 1789429441.96 total 90.08 last 80.31) - open/close 8249 (time 1789429451.97 total 100.09 last 83.83) - open/close 9036 (time 1789429461.98 total 110.09 last 78.65) - open/close 9836 (time 1789429471.99 total 120.10 last 79.93) total: 10000 open/close in 122.34 seconds: 81.74 ops/second - unlinked 0 (time 1789429478 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 790 (time 1789429573.02 total 10.00 last 79.00) - open/close 1521 (time 1789429583.03 total 20.01 last 73.04) - open/close 2294 (time 1789429593.04 total 30.02 last 77.22) - open/close 3088 (time 1789429603.04 total 40.02 last 79.35) - open/close 3968 (time 1789429613.05 total 50.03 last 87.95) - open/close 4702 (time 1789429623.05 total 60.03 last 73.38) - open/close 5361 (time 1789429633.06 total 70.04 last 65.84) - open/close 6049 (time 1789429643.07 total 80.05 last 68.74) - open/close 6797 (time 1789429653.09 total 90.07 last 74.62) - open/close 7536 (time 1789429663.10 total 100.08 last 73.83) - open/close 8247 (time 1789429673.12 total 110.10 last 70.97) - open/close 8984 (time 1789429683.13 total 120.11 last 73.60) - open/close 9707 (time 1789429693.14 total 130.12 last 72.26) total: 10000 open/close in 133.78 seconds: 74.75 ops/second - unlinked 0 (time 1789429700 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 839 (time 1789429796.60 total 10.02 last 83.77) - open/close 1556 (time 1789429806.61 total 20.02 last 71.67) - open/close 2323 (time 1789429816.62 total 30.03 last 76.65) - open/close 3089 (time 1789429826.63 total 40.04 last 76.47) - open/close 3770 (time 1789429836.63 total 50.05 last 68.08) - open/close 4392 (time 1789429846.64 total 60.05 last 62.15) - open/close 5188 (time 1789429856.64 total 70.06 last 79.59) - open/close 5936 (time 1789429866.65 total 80.06 last 74.78) - open/close 6754 (time 1789429876.67 total 90.08 last 81.65) - open/close 7482 (time 1789429886.68 total 100.09 last 72.69) - open/close 8208 (time 1789429896.68 total 110.09 last 72.58) - open/close 8920 (time 1789429906.69 total 120.10 last 71.17) - open/close 9721 (time 1789429916.69 total 130.10 last 80.06) total: 10000 open/close in 133.32 seconds: 75.01 ops/second - unlinked 0 (time 1789429923 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 672 (time 1789430019.09 total 10.00 last 67.17) - open/close 1291 (time 1789430029.09 total 20.01 last 61.87) - open/close 2044 (time 1789430039.10 total 30.02 last 75.22) - open/close 2964 (time 1789430049.11 total 40.03 last 91.91) - open/close 3642 (time 1789430059.14 total 50.05 last 67.63) - open/close 4319 (time 1789430069.17 total 60.09 last 67.45) - open/close 5024 (time 1789430079.17 total 70.09 last 70.49) - open/close 5762 (time 1789430089.18 total 80.10 last 73.77) - open/close 6471 (time 1789430099.18 total 90.10 last 70.86) - open/close 7232 (time 1789430109.19 total 100.10 last 76.08) - open/close 8031 (time 1789430119.20 total 110.12 last 79.79) - open/close 8746 (time 1789430129.21 total 120.13 last 71.44) - open/close 9415 (time 1789430139.21 total 130.13 last 66.88) total: 10000 open/close in 138.72 seconds: 72.09 ops/second - unlinked 0 (time 1789430152 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 689 (time 1789430255.74 total 10.03 last 68.69) - open/close 1486 (time 1789430265.75 total 20.04 last 79.62) - open/close 2169 (time 1789430275.75 total 30.04 last 68.30) - open/close 2844 (time 1789430285.77 total 40.06 last 67.37) - open/close 3524 (time 1789430295.78 total 50.07 last 67.92) - open/close 4212 (time 1789430305.78 total 60.07 last 68.78) - open/close 4824 (time 1789430315.81 total 70.10 last 61.02) - open/close 5497 (time 1789430325.81 total 80.11 last 67.27) - open/close 6225 (time 1789430335.83 total 90.12 last 72.69) - open/close 6986 (time 1789430345.84 total 100.14 last 76.00) - open/close 7712 (time 1789430355.85 total 110.14 last 72.55) - open/close 8381 (time 1789430365.86 total 120.15 last 66.86) - open/close 9013 (time 1789430375.87 total 130.16 last 63.14) - open/close 9650 (time 1789430385.87 total 140.17 last 63.66) total: 10000 open/close in 145.72 seconds: 68.63 ops/second - unlinked 0 (time 1789430397 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 789 (time 1789430495.40 total 10.00 last 78.87) - open/close 1503 (time 1789430505.41 total 20.01 last 71.34) - open/close 2466 (time 1789430515.43 total 30.03 last 96.12) - open/close 3172 (time 1789430525.43 total 40.03 last 70.58) - open/close 3919 (time 1789430535.43 total 50.03 last 74.70) - open/close 4762 (time 1789430545.44 total 60.04 last 84.21) - open/close 5500 (time 1789430555.46 total 70.06 last 73.71) - open/close 6263 (time 1789430565.47 total 80.07 last 76.17) - open/close 7011 (time 1789430575.48 total 90.08 last 74.77) - open/close 7765 (time 1789430585.50 total 100.10 last 75.23) - open/close 8449 (time 1789430595.51 total 110.11 last 68.30) - open/close 9131 (time 1789430605.52 total 120.12 last 68.16) - open/close 9844 (time 1789430615.53 total 130.13 last 71.24) total: 10000 open/close in 132.61 seconds: 75.41 ops/second - unlinked 0 (time 1789430622 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.103@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg203-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.103@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.103@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg203-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local oleg203-server: kmemleak disabled modules unloaded. pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2