== conf-sanity test 69: replace an OST with the same index ========================================================== 11:15:15 (1776266115) start mds service on oleg152-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg152-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock oleg152-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 642 (time 1776266204.98 total 10.01 last 64.14) - open/close 1276 (time 1776266214.98 total 20.01 last 63.38) - open/close 1923 (time 1776266224.99 total 30.02 last 64.65) - open/close 2483 (time 1776266234.99 total 40.03 last 55.97) - open/close 3066 (time 1776266245.00 total 50.03 last 58.29) - open/close 3750 (time 1776266255.00 total 60.03 last 68.39) - open/close 4424 (time 1776266265.00 total 70.04 last 67.36) - open/close 5090 (time 1776266275.01 total 80.04 last 66.56) - open/close 5917 (time 1776266285.02 total 90.05 last 82.64) - open/close 6613 (time 1776266295.02 total 100.05 last 69.60) - open/close 7270 (time 1776266305.03 total 110.06 last 65.64) - open/close 7914 (time 1776266315.04 total 120.07 last 64.34) - open/close 8584 (time 1776266325.04 total 130.07 last 66.98) - open/close 9245 (time 1776266335.04 total 140.07 last 66.08) - open/close 9857 (time 1776266345.05 total 150.09 last 61.13) total: 10000 open/close in 151.91 seconds: 65.83 ops/second - unlinked 0 (time 1776266350 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 674 (time 1776266455.05 total 10.00 last 67.38) - open/close 1305 (time 1776266465.06 total 20.01 last 63.04) - open/close 1858 (time 1776266475.06 total 30.01 last 55.29) - open/close 2488 (time 1776266485.07 total 40.02 last 62.96) - open/close 3178 (time 1776266495.08 total 50.04 last 68.90) - open/close 3909 (time 1776266505.08 total 60.04 last 73.08) - open/close 4481 (time 1776266515.08 total 70.04 last 57.19) - open/close 5059 (time 1776266525.09 total 80.05 last 57.77) - open/close 5742 (time 1776266535.10 total 90.06 last 68.24) - open/close 6397 (time 1776266545.11 total 100.06 last 65.44) - open/close 6971 (time 1776266555.14 total 110.10 last 57.22) - open/close 7592 (time 1776266565.15 total 120.11 last 62.02) - open/close 8260 (time 1776266575.15 total 130.11 last 66.79) - open/close 8866 (time 1776266585.16 total 140.12 last 60.54) - open/close 9515 (time 1776266595.16 total 150.12 last 64.90) total: 10000 open/close in 157.96 seconds: 63.31 ops/second - unlinked 0 (time 1776266607 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 710 (time 1776266709.95 total 10.00 last 70.99) - open/close 1490 (time 1776266719.95 total 20.00 last 78.00) - open/close 2066 (time 1776266729.97 total 30.02 last 57.50) - open/close 2656 (time 1776266739.97 total 40.03 last 58.96) - open/close 3395 (time 1776266749.99 total 50.04 last 73.81) - open/close 4082 (time 1776266759.99 total 60.04 last 68.66) - open/close 4831 (time 1776266770.00 total 70.05 last 74.86) - open/close 5581 (time 1776266780.01 total 80.06 last 74.91) - open/close 6212 (time 1776266790.02 total 90.07 last 63.03) - open/close 6775 (time 1776266800.03 total 100.08 last 56.23) - open/close 7320 (time 1776266810.03 total 110.09 last 54.49) - open/close 7925 (time 1776266820.06 total 120.11 last 60.35) - open/close 8601 (time 1776266830.07 total 130.13 last 67.49) - open/close 9271 (time 1776266840.08 total 140.14 last 66.94) total: 10000 open/close in 149.32 seconds: 66.97 ops/second - unlinked 0 (time 1776266854 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 674 (time 1776266955.74 total 10.02 last 67.26) - open/close 1273 (time 1776266965.75 total 20.03 last 59.86) - open/close 2013 (time 1776266975.76 total 30.04 last 73.93) - open/close 2667 (time 1776266985.77 total 40.05 last 65.33) - open/close 3282 (time 1776266995.77 total 50.05 last 61.49) - open/close 3946 (time 1776267005.77 total 60.05 last 66.40) - open/close 4601 (time 1776267015.77 total 70.06 last 65.47) - open/close 5237 (time 1776267025.79 total 80.07 last 63.53) - open/close 5862 (time 1776267035.79 total 90.07 last 62.47) - open/close 6522 (time 1776267045.80 total 100.08 last 65.94) - open/close 7138 (time 1776267055.83 total 110.11 last 61.43) - open/close 7681 (time 1776267065.83 total 120.12 last 54.26) - open/close 8476 (time 1776267075.87 total 130.15 last 79.23) - open/close 9103 (time 1776267085.92 total 140.20 last 62.38) - open/close 9793 (time 1776267095.93 total 150.21 last 68.93) total: 10000 open/close in 152.93 seconds: 65.39 ops/second - unlinked 0 (time 1776267103 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 604 (time 1776267214.14 total 10.02 last 60.29) - open/close 1280 (time 1776267224.16 total 20.04 last 67.47) - open/close 1932 (time 1776267234.16 total 30.04 last 65.18) - open/close 2553 (time 1776267244.18 total 40.06 last 61.99) - open/close 3231 (time 1776267254.19 total 50.07 last 67.69) - open/close 3886 (time 1776267264.21 total 60.09 last 65.42) - open/close 4569 (time 1776267274.21 total 70.09 last 68.29) - open/close 5112 (time 1776267284.21 total 80.09 last 54.30) - open/close 5653 (time 1776267294.22 total 90.10 last 54.02) - open/close 6281 (time 1776267304.24 total 100.12 last 62.71) - open/close 6815 (time 1776267314.26 total 110.14 last 53.26) - open/close 7420 (time 1776267324.27 total 120.15 last 60.47) - open/close 8037 (time 1776267334.27 total 130.15 last 61.68) - open/close 8718 (time 1776267344.28 total 140.16 last 68.02) - open/close 9290 (time 1776267354.29 total 150.17 last 57.14) - open/close 9840 (time 1776267364.32 total 160.20 last 54.85) total: 10000 open/close in 163.22 seconds: 61.27 ops/second - unlinked 0 (time 1776267374 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 578 (time 1776267479.34 total 10.00 last 57.79) - open/close 1258 (time 1776267489.35 total 20.01 last 67.93) - open/close 1970 (time 1776267499.37 total 30.03 last 71.07) - open/close 2691 (time 1776267509.37 total 40.03 last 72.07) - open/close 3384 (time 1776267519.38 total 50.04 last 69.25) - open/close 4040 (time 1776267529.39 total 60.05 last 65.54) - open/close 4635 (time 1776267539.39 total 70.05 last 59.49) - open/close 5225 (time 1776267549.39 total 80.05 last 59.00) - open/close 6085 (time 1776267559.40 total 90.06 last 85.94) - open/close 6884 (time 1776267569.40 total 100.07 last 79.85) - open/close 7612 (time 1776267579.41 total 110.07 last 72.78) - open/close 8334 (time 1776267589.42 total 120.08 last 72.13) - open/close 9046 (time 1776267599.42 total 130.08 last 71.19) - open/close 9655 (time 1776267609.43 total 140.09 last 60.85) total: 10000 open/close in 144.80 seconds: 69.06 ops/second - unlinked 0 (time 1776267619 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 743 (time 1776267719.75 total 10.01 last 74.19) - open/close 1371 (time 1776267729.75 total 20.02 last 62.78) - open/close 2186 (time 1776267739.76 total 30.03 last 81.43) - open/close 2968 (time 1776267749.77 total 40.03 last 78.15) - open/close 3747 (time 1776267759.77 total 50.04 last 77.84) - open/close 4481 (time 1776267769.78 total 60.05 last 73.37) - open/close 5268 (time 1776267779.78 total 70.05 last 78.66) - open/close 6093 (time 1776267789.79 total 80.06 last 82.46) - open/close 6787 (time 1776267799.82 total 90.08 last 69.22) - open/close 7447 (time 1776267809.82 total 100.09 last 65.95) - open/close 8198 (time 1776267819.83 total 110.10 last 75.04) - open/close 8960 (time 1776267829.84 total 120.10 last 76.16) - open/close 9727 (time 1776267839.84 total 130.11 last 76.65) total: 10000 open/close in 133.44 seconds: 74.94 ops/second - unlinked 0 (time 1776267847 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 809 (time 1776267945.93 total 10.01 last 80.84) - open/close 1550 (time 1776267955.94 total 20.02 last 74.04) - open/close 2393 (time 1776267965.94 total 30.02 last 84.27) - open/close 3164 (time 1776267975.95 total 40.02 last 77.08) - open/close 3910 (time 1776267985.96 total 50.03 last 74.50) - open/close 4617 (time 1776267995.96 total 60.04 last 70.67) - open/close 5421 (time 1776268005.97 total 70.04 last 80.36) - open/close 6241 (time 1776268015.97 total 80.05 last 81.96) - open/close 7026 (time 1776268025.98 total 90.05 last 78.49) - open/close 7871 (time 1776268035.99 total 100.06 last 84.41) - open/close 8702 (time 1776268045.99 total 110.06 last 83.08) - open/close 9425 (time 1776268056.00 total 120.07 last 72.23) total: 10000 open/close in 127.73 seconds: 78.29 ops/second - unlinked 0 (time 1776268068 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 765 (time 1776268169.07 total 10.00 last 76.50) - open/close 1536 (time 1776268179.08 total 20.01 last 77.06) - open/close 2305 (time 1776268189.08 total 30.01 last 76.86) - open/close 3082 (time 1776268199.09 total 40.02 last 77.62) - open/close 3802 (time 1776268209.10 total 50.03 last 71.93) - open/close 4379 (time 1776268219.10 total 60.03 last 57.70) - open/close 4995 (time 1776268229.11 total 70.04 last 61.56) - open/close 5709 (time 1776268239.12 total 80.05 last 71.33) - open/close 6459 (time 1776268249.13 total 90.06 last 74.93) - open/close 7233 (time 1776268259.15 total 100.07 last 77.27) - open/close 7846 (time 1776268269.19 total 110.12 last 61.02) - open/close 8348 (time 1776268279.21 total 120.14 last 50.11) - open/close 8959 (time 1776268289.22 total 130.14 last 61.07) - open/close 9632 (time 1776268299.22 total 140.15 last 67.27) total: 10000 open/close in 145.19 seconds: 68.87 ops/second - unlinked 0 (time 1776268309 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 773 (time 1776268406.45 total 10.01 last 77.25) - open/close 1424 (time 1776268416.45 total 20.01 last 65.09) - open/close 2163 (time 1776268426.46 total 30.02 last 73.84) - open/close 2968 (time 1776268436.47 total 40.02 last 80.45) - open/close 3751 (time 1776268446.47 total 50.02 last 78.29) - open/close 4517 (time 1776268456.48 total 60.03 last 76.51) - open/close 5321 (time 1776268466.48 total 70.04 last 80.39) - open/close 6145 (time 1776268476.48 total 80.04 last 82.37) - open/close 6842 (time 1776268486.50 total 90.05 last 69.61) - open/close 7485 (time 1776268496.53 total 100.09 last 64.06) - open/close 8111 (time 1776268506.54 total 110.09 last 62.58) - open/close 8825 (time 1776268516.55 total 120.11 last 71.30) - open/close 9544 (time 1776268526.56 total 130.12 last 71.84) total: 10000 open/close in 136.68 seconds: 73.16 ops/second - unlinked 0 (time 1776268538 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.152@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg152-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock oleg152-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg152-server: LNET ready to unload oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2