== conf-sanity test 69: replace an OST with the same index ========================================================== 00:11:02 (1788927062) start mds service on oleg647-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing load_modules_local oleg647-server: Loading modules from /home/green/git/lustre-release/lustre oleg647-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg647-server: detected 4 online CPUs by sysfs oleg647-server: MODOPTS_LIBCFS= oleg647-server: Force libcfs to create 2 CPU partitions oleg647-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg647-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg647-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg647-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg647-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 807 (time 1788927148.73 total 10.02 last 80.53) - open/close 1613 (time 1788927158.73 total 20.03 last 80.52) - open/close 2499 (time 1788927168.75 total 30.04 last 88.49) - open/close 3329 (time 1788927178.75 total 40.05 last 82.96) - open/close 4087 (time 1788927188.76 total 50.06 last 75.72) - open/close 4917 (time 1788927198.77 total 60.07 last 82.94) - open/close 5771 (time 1788927208.77 total 70.07 last 85.38) - open/close 6641 (time 1788927218.78 total 80.08 last 86.92) - open/close 7540 (time 1788927228.79 total 90.09 last 89.82) - open/close 8412 (time 1788927238.80 total 100.09 last 87.14) - open/close 9253 (time 1788927248.81 total 110.10 last 84.00) total: 10000 open/close in 118.94 seconds: 84.08 ops/second - unlinked 0 (time 1788927262 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 680 (time 1788927357.11 total 10.01 last 67.94) - open/close 1289 (time 1788927367.11 total 20.01 last 60.90) - open/close 2011 (time 1788927377.12 total 30.01 last 72.16) - open/close 2709 (time 1788927387.13 total 40.03 last 69.71) - open/close 3505 (time 1788927397.14 total 50.04 last 79.49) - open/close 4200 (time 1788927407.16 total 60.06 last 69.35) - open/close 4868 (time 1788927417.18 total 70.08 last 66.70) - open/close 5549 (time 1788927427.18 total 80.08 last 68.06) - open/close 6416 (time 1788927437.20 total 90.10 last 86.54) - open/close 7326 (time 1788927447.21 total 100.11 last 90.95) - open/close 8228 (time 1788927457.22 total 110.11 last 90.14) - open/close 9036 (time 1788927467.22 total 120.12 last 80.74) - open/close 9861 (time 1788927477.23 total 130.13 last 82.44) total: 10000 open/close in 131.86 seconds: 75.84 ops/second - unlinked 0 (time 1788927482 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 875 (time 1788927579.06 total 10.00 last 87.49) - open/close 1682 (time 1788927589.07 total 20.01 last 80.67) - open/close 2602 (time 1788927599.08 total 30.02 last 91.89) - open/close 3398 (time 1788927609.08 total 40.02 last 79.57) - open/close 4203 (time 1788927619.09 total 50.03 last 80.40) - open/close 4854 (time 1788927629.10 total 60.04 last 65.06) - open/close 5689 (time 1788927639.11 total 70.05 last 83.41) - open/close 6547 (time 1788927649.12 total 80.06 last 85.69) - open/close 7380 (time 1788927659.13 total 90.07 last 83.27) - open/close 8196 (time 1788927669.13 total 100.07 last 81.57) - open/close 8973 (time 1788927679.14 total 110.07 last 77.67) - open/close 9629 (time 1788927689.14 total 120.08 last 65.56) total: 10000 open/close in 125.89 seconds: 79.43 ops/second - unlinked 0 (time 1788927698 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 801 (time 1788927791.69 total 10.01 last 80.04) - open/close 1513 (time 1788927801.70 total 20.02 last 71.14) - open/close 2203 (time 1788927811.71 total 30.02 last 68.95) - open/close 3022 (time 1788927821.72 total 40.04 last 81.78) - open/close 3780 (time 1788927831.74 total 50.05 last 75.68) - open/close 4619 (time 1788927841.75 total 60.06 last 83.82) - open/close 5495 (time 1788927851.76 total 70.07 last 87.53) - open/close 6344 (time 1788927861.76 total 80.08 last 84.86) - open/close 7237 (time 1788927871.77 total 90.09 last 89.20) - open/close 8058 (time 1788927881.79 total 100.10 last 82.00) - open/close 8904 (time 1788927891.79 total 110.10 last 84.55) - open/close 9695 (time 1788927901.80 total 120.12 last 79.00) total: 10000 open/close in 123.82 seconds: 80.76 ops/second - unlinked 0 (time 1788927909 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 680 (time 1788928008.06 total 10.00 last 67.97) - open/close 1255 (time 1788928018.06 total 20.01 last 57.48) - open/close 1810 (time 1788928028.11 total 30.05 last 55.26) - open/close 2275 (time 1788928038.11 total 40.05 last 46.49) - open/close 2828 (time 1788928048.11 total 50.05 last 55.29) - open/close 3462 (time 1788928058.11 total 60.06 last 63.36) - open/close 4198 (time 1788928068.12 total 70.06 last 73.58) - open/close 5031 (time 1788928078.12 total 80.07 last 83.26) - open/close 5907 (time 1788928088.12 total 90.07 last 87.59) - open/close 6650 (time 1788928098.13 total 100.08 last 74.24) - open/close 7408 (time 1788928108.14 total 110.08 last 75.77) - open/close 8167 (time 1788928118.15 total 120.09 last 75.79) - open/close 8978 (time 1788928128.16 total 130.10 last 81.05) - open/close 9857 (time 1788928138.16 total 140.10 last 87.90) total: 10000 open/close in 141.78 seconds: 70.53 ops/second - unlinked 0 (time 1788928144 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 677 (time 1788928238.20 total 10.01 last 67.60) - open/close 1475 (time 1788928248.20 total 20.02 last 79.77) - open/close 2295 (time 1788928258.21 total 30.03 last 81.94) - open/close 3162 (time 1788928268.21 total 40.03 last 86.69) - open/close 3946 (time 1788928278.21 total 50.03 last 78.37) - open/close 4681 (time 1788928288.23 total 60.04 last 73.40) - open/close 5469 (time 1788928298.24 total 70.05 last 78.73) - open/close 6172 (time 1788928308.25 total 80.07 last 70.21) - open/close 6952 (time 1788928318.25 total 90.07 last 78.00) - open/close 7731 (time 1788928328.25 total 100.07 last 77.90) - open/close 8440 (time 1788928338.26 total 110.08 last 70.82) - open/close 9182 (time 1788928348.26 total 120.08 last 74.17) - open/close 9840 (time 1788928358.27 total 130.08 last 65.77) total: 10000 open/close in 132.21 seconds: 75.64 ops/second - unlinked 0 (time 1788928364 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 860 (time 1788928463.69 total 10.01 last 85.95) - open/close 1703 (time 1788928473.69 total 20.01 last 84.28) - open/close 2550 (time 1788928483.70 total 30.02 last 84.61) - open/close 3326 (time 1788928493.70 total 40.02 last 77.56) - open/close 3971 (time 1788928503.73 total 50.05 last 64.36) - open/close 4467 (time 1788928513.73 total 60.05 last 49.59) - open/close 5088 (time 1788928523.74 total 70.06 last 62.02) - open/close 5662 (time 1788928533.74 total 80.06 last 57.39) - open/close 6418 (time 1788928543.75 total 90.07 last 75.55) - open/close 7144 (time 1788928553.75 total 100.07 last 72.56) - open/close 7918 (time 1788928563.76 total 110.08 last 77.39) - open/close 8708 (time 1788928573.76 total 120.08 last 78.97) - open/close 9523 (time 1788928583.76 total 130.08 last 81.47) total: 10000 open/close in 135.67 seconds: 73.71 ops/second - unlinked 0 (time 1788928593 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 833 (time 1788928684.19 total 10.00 last 83.29) - open/close 1633 (time 1788928694.21 total 20.02 last 79.87) - open/close 2281 (time 1788928704.23 total 30.03 last 64.69) - open/close 3003 (time 1788928714.23 total 40.03 last 72.19) - open/close 3661 (time 1788928724.23 total 50.04 last 65.78) - open/close 4217 (time 1788928734.26 total 60.07 last 55.45) - open/close 4723 (time 1788928744.27 total 70.08 last 50.52) - open/close 5221 (time 1788928754.30 total 80.10 last 49.68) - open/close 5801 (time 1788928764.30 total 90.11 last 57.96) - open/close 6428 (time 1788928774.31 total 100.12 last 62.65) - open/close 7153 (time 1788928784.32 total 110.13 last 72.45) - open/close 7882 (time 1788928794.34 total 120.14 last 72.78) - open/close 8522 (time 1788928804.37 total 130.18 last 63.78) - open/close 9044 (time 1788928814.38 total 140.19 last 52.14) - open/close 9623 (time 1788928824.38 total 150.19 last 57.89) total: 10000 open/close in 157.13 seconds: 63.64 ops/second - unlinked 0 (time 1788928836 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 824 (time 1788928930.13 total 10.01 last 82.30) - open/close 1543 (time 1788928940.14 total 20.03 last 71.79) - open/close 2211 (time 1788928950.16 total 30.05 last 66.66) - open/close 2892 (time 1788928960.17 total 40.05 last 68.05) - open/close 3834 (time 1788928970.17 total 50.06 last 94.18) - open/close 4608 (time 1788928980.17 total 60.06 last 77.39) - open/close 5383 (time 1788928990.17 total 70.06 last 77.48) - open/close 6165 (time 1788929000.17 total 80.06 last 78.20) - open/close 6716 (time 1788929010.18 total 90.07 last 55.04) - open/close 7320 (time 1788929020.19 total 100.08 last 60.36) - open/close 8139 (time 1788929030.20 total 110.09 last 81.83) - open/close 8973 (time 1788929040.23 total 120.12 last 83.15) - open/close 9684 (time 1788929050.23 total 130.12 last 71.09) total: 10000 open/close in 133.96 seconds: 74.65 ops/second - unlinked 0 (time 1788929057 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 847 (time 1788929148.91 total 10.00 last 84.69) - open/close 1649 (time 1788929158.92 total 20.01 last 80.11) - open/close 2353 (time 1788929168.95 total 30.03 last 70.25) - open/close 3095 (time 1788929178.96 total 40.04 last 74.13) - open/close 3883 (time 1788929188.96 total 50.05 last 78.76) - open/close 4713 (time 1788929198.97 total 60.06 last 82.94) - open/close 5522 (time 1788929208.98 total 70.06 last 80.84) - open/close 6202 (time 1788929218.99 total 80.07 last 67.94) - open/close 7062 (time 1788929229.00 total 90.09 last 85.87) - open/close 7820 (time 1788929239.01 total 100.10 last 75.74) - open/close 8569 (time 1788929249.02 total 110.10 last 74.84) - open/close 9426 (time 1788929259.02 total 120.11 last 85.67) total: 10000 open/close in 127.41 seconds: 78.49 ops/second - unlinked 0 (time 1788929270 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg647-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg647-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg647-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.147@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.147@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg647-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg647-server: oleg647-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg647-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg647-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg647-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg647-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.147@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.147@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.206.147@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.206.147@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.206.147@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg647-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg647-server: oleg647-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg647-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg647-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg647-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg647-server stop mds service on oleg647-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg647-server unloading modules via unload_modules_local on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing unload_modules_local oleg647-server: modules unloaded. pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2