== conf-sanity test 69: replace an OST with the same index ========================================================== 22:05:09 (1789351509) start mds service on oleg129-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg129-server: detected 4 online CPUs by sysfs oleg129-server: MODOPTS_LIBCFS= oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg129-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg129-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 844 (time 1789351602.42 total 10.00 last 84.39) - open/close 1767 (time 1789351612.43 total 20.02 last 92.16) - open/close 2646 (time 1789351622.43 total 30.02 last 87.90) - open/close 3521 (time 1789351632.43 total 40.02 last 87.49) - open/close 4513 (time 1789351642.44 total 50.03 last 99.11) - open/close 5498 (time 1789351652.45 total 60.03 last 98.43) - open/close 6474 (time 1789351662.45 total 70.04 last 97.57) - open/close 7407 (time 1789351672.47 total 80.06 last 93.13) - open/close 8366 (time 1789351682.47 total 90.06 last 95.89) - open/close 9273 (time 1789351692.48 total 100.06 last 90.68) total: 10000 open/close in 107.50 seconds: 93.02 ops/second - unlinked 0 (time 1789351703 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1034 (time 1789351780.63 total 10.00 last 103.35) - open/close 1936 (time 1789351790.64 total 20.01 last 90.16) - open/close 2918 (time 1789351800.66 total 30.03 last 98.00) - open/close 3914 (time 1789351810.66 total 40.04 last 99.54) - open/close 4934 (time 1789351820.67 total 50.04 last 101.93) - open/close 6009 (time 1789351830.68 total 60.05 last 107.38) - open/close 6972 (time 1789351840.69 total 70.06 last 96.25) - open/close 7895 (time 1789351850.69 total 80.07 last 92.23) - open/close 8850 (time 1789351860.70 total 90.07 last 95.42) - open/close 9793 (time 1789351870.71 total 100.09 last 94.20) total: 10000 open/close in 102.26 seconds: 97.79 ops/second - unlinked 0 (time 1789351876 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 938 (time 1789351958.52 total 10.00 last 93.79) - open/close 1848 (time 1789351968.52 total 20.00 last 90.98) - open/close 2768 (time 1789351978.54 total 30.02 last 91.84) - open/close 3658 (time 1789351988.54 total 40.03 last 88.96) - open/close 4528 (time 1789351998.55 total 50.04 last 86.91) - open/close 5351 (time 1789352008.56 total 60.04 last 82.24) - open/close 6212 (time 1789352018.56 total 70.04 last 86.09) - open/close 7261 (time 1789352028.57 total 80.05 last 104.81) - open/close 8215 (time 1789352038.58 total 90.06 last 95.29) - open/close 9161 (time 1789352048.59 total 100.07 last 94.54) total: 10000 open/close in 109.37 seconds: 91.43 ops/second - unlinked 0 (time 1789352061 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1078 (time 1789352139.19 total 10.00 last 107.78) - open/close 2079 (time 1789352149.21 total 20.02 last 99.94) - open/close 2929 (time 1789352159.22 total 30.03 last 84.89) - open/close 4041 (time 1789352169.23 total 40.04 last 111.14) - open/close 5066 (time 1789352179.23 total 50.04 last 102.49) - open/close 6181 (time 1789352189.23 total 60.04 last 111.48) - open/close 7455 (time 1789352199.24 total 70.05 last 127.32) - open/close 8684 (time 1789352209.24 total 80.05 last 122.83) - open/close 9689 (time 1789352219.25 total 90.06 last 100.46) total: 10000 open/close in 93.33 seconds: 107.14 ops/second - unlinked 0 (time 1789352225 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1092 (time 1789352297.91 total 10.00 last 109.17) - open/close 2213 (time 1789352307.92 total 20.01 last 112.02) - open/close 3264 (time 1789352317.92 total 30.02 last 105.03) - open/close 4313 (time 1789352327.92 total 40.02 last 104.89) - open/close 5313 (time 1789352337.93 total 50.03 last 99.89) - open/close 6482 (time 1789352347.94 total 60.03 last 116.85) - open/close 7662 (time 1789352357.95 total 70.04 last 117.91) - open/close 8730 (time 1789352367.95 total 80.05 last 106.75) - open/close 9765 (time 1789352377.99 total 90.09 last 103.06) total: 10000 open/close in 92.32 seconds: 108.32 ops/second - unlinked 0 (time 1789352383 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1069 (time 1789352450.52 total 10.00 last 106.89) - open/close 2229 (time 1789352460.54 total 20.01 last 115.84) - open/close 3423 (time 1789352470.54 total 30.02 last 119.36) - open/close 4583 (time 1789352480.55 total 40.02 last 115.94) - open/close 5823 (time 1789352490.55 total 50.03 last 123.93) - open/close 7117 (time 1789352500.56 total 60.03 last 129.34) - open/close 8333 (time 1789352510.57 total 70.05 last 121.47) - open/close 9541 (time 1789352520.57 total 80.05 last 120.77) total: 10000 open/close in 83.68 seconds: 119.50 ops/second - unlinked 0 (time 1789352526 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1372 (time 1789352586.41 total 10.01 last 137.13) - open/close 2863 (time 1789352596.42 total 20.01 last 149.07) - open/close 4481 (time 1789352606.42 total 30.01 last 161.74) - open/close 6103 (time 1789352616.42 total 40.01 last 162.18) - open/close 7761 (time 1789352626.43 total 50.02 last 165.70) - open/close 9674 (time 1789352636.43 total 60.02 last 191.18) total: 10000 open/close in 62.04 seconds: 161.20 ops/second - unlinked 0 (time 1789352640 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1624 (time 1789352683.71 total 10.00 last 162.33) - open/close 3412 (time 1789352693.72 total 20.01 last 178.63) - open/close 4940 (time 1789352703.73 total 30.02 last 152.65) - open/close 6698 (time 1789352713.74 total 40.03 last 175.73) - open/close 8974 (time 1789352723.74 total 50.03 last 227.58) total: 10000 open/close in 55.58 seconds: 179.91 ops/second - unlinked 0 (time 1789352730 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1592 (time 1789352773.91 total 10.00 last 159.15) - open/close 3283 (time 1789352783.91 total 20.01 last 169.06) - open/close 5238 (time 1789352793.91 total 30.01 last 195.45) - open/close 7209 (time 1789352803.92 total 40.02 last 196.95) - open/close 9340 (time 1789352813.92 total 50.02 last 213.04) total: 10000 open/close in 54.75 seconds: 182.64 ops/second - unlinked 0 (time 1789352821 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 979 (time 1789352912.28 total 10.00 last 97.86) - open/close 1873 (time 1789352922.28 total 20.01 last 89.37) - open/close 2937 (time 1789352932.29 total 30.01 last 106.32) - open/close 3783 (time 1789352942.30 total 40.02 last 84.54) - open/close 4828 (time 1789352952.30 total 50.02 last 104.49) - open/close 5851 (time 1789352962.30 total 60.03 last 102.26) - open/close 6912 (time 1789352972.31 total 70.04 last 105.98) - open/close 7880 (time 1789352982.32 total 80.04 last 96.77) - open/close 8920 (time 1789352992.38 total 90.11 last 103.29) - open/close 9924 (time 1789353002.39 total 100.11 last 100.39) total: 10000 open/close in 100.98 seconds: 99.03 ops/second - unlinked 0 (time 1789353006 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg129-server: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg129-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre On OST0, 16854 used inodes rc=0 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.129@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.129@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg129-server: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg129-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server unloading modules via unload_modules_local on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local oleg129-server: modules unloaded. pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2