== conf-sanity test 69: replace an OST with the same index ========================================================== 13:33:58 (1776188038) start mds service on oleg248-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg248-server: detected 4 online CPUs by sysfs oleg248-server: Force libcfs to create 2 CPU partitions oleg248-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg248-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg248-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg248-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock oleg248-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 602 (time 1776188147.46 total 10.00 last 60.19) - open/close 1358 (time 1776188157.46 total 20.00 last 75.59) - open/close 2140 (time 1776188167.48 total 30.03 last 78.03) - open/close 2929 (time 1776188177.48 total 40.03 last 78.88) - open/close 3646 (time 1776188187.49 total 50.04 last 71.62) - open/close 4502 (time 1776188197.50 total 60.05 last 85.52) - open/close 5357 (time 1776188207.52 total 70.06 last 85.38) - open/close 6146 (time 1776188217.53 total 80.08 last 78.76) - open/close 6963 (time 1776188227.53 total 90.08 last 81.70) - open/close 7905 (time 1776188237.54 total 100.09 last 94.14) - open/close 8750 (time 1776188247.56 total 110.10 last 84.37) - open/close 9554 (time 1776188257.57 total 120.11 last 80.32) total: 10000 open/close in 125.69 seconds: 79.56 ops/second - unlinked 0 (time 1776188266 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 875 (time 1776188354.19 total 10.01 last 87.45) - open/close 1712 (time 1776188364.19 total 20.01 last 83.68) - open/close 2546 (time 1776188374.20 total 30.01 last 83.37) - open/close 3318 (time 1776188384.20 total 40.01 last 77.19) - open/close 4080 (time 1776188394.20 total 50.02 last 76.16) - open/close 4687 (time 1776188404.21 total 60.03 last 60.65) - open/close 5486 (time 1776188414.22 total 70.03 last 79.84) - open/close 6264 (time 1776188424.22 total 80.04 last 77.77) - open/close 6930 (time 1776188434.23 total 90.05 last 66.53) - open/close 7627 (time 1776188444.24 total 100.05 last 69.68) - open/close 8387 (time 1776188454.24 total 110.05 last 75.98) - open/close 9122 (time 1776188464.24 total 120.05 last 73.50) - open/close 9814 (time 1776188474.24 total 130.06 last 69.16) total: 10000 open/close in 132.56 seconds: 75.44 ops/second - unlinked 0 (time 1776188480 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 836 (time 1776188572.23 total 10.01 last 83.55) - open/close 1651 (time 1776188582.23 total 20.01 last 81.50) - open/close 2268 (time 1776188592.24 total 30.02 last 61.64) - open/close 2927 (time 1776188602.26 total 40.03 last 65.81) - open/close 3662 (time 1776188612.26 total 50.03 last 73.50) - open/close 4420 (time 1776188622.27 total 60.04 last 75.73) - open/close 5147 (time 1776188632.28 total 70.05 last 72.63) - open/close 5783 (time 1776188642.30 total 80.07 last 63.45) - open/close 6417 (time 1776188652.30 total 90.07 last 63.37) - open/close 7135 (time 1776188662.31 total 100.08 last 71.73) - open/close 7884 (time 1776188672.31 total 110.08 last 74.90) - open/close 8631 (time 1776188682.32 total 120.09 last 74.65) - open/close 9373 (time 1776188692.33 total 130.10 last 74.14) total: 10000 open/close in 138.62 seconds: 72.14 ops/second - unlinked 0 (time 1776188705 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 862 (time 1776188801.42 total 10.01 last 86.09) - open/close 1599 (time 1776188811.42 total 20.02 last 73.66) - open/close 2285 (time 1776188821.42 total 30.02 last 68.60) - open/close 2946 (time 1776188831.43 total 40.03 last 66.03) - open/close 3653 (time 1776188841.45 total 50.04 last 70.62) - open/close 4564 (time 1776188851.46 total 60.05 last 90.98) - open/close 5325 (time 1776188861.46 total 70.05 last 76.08) - open/close 6122 (time 1776188871.47 total 80.06 last 79.63) - open/close 6971 (time 1776188881.49 total 90.09 last 84.71) - open/close 7719 (time 1776188891.50 total 100.09 last 74.76) - open/close 8542 (time 1776188901.50 total 110.09 last 82.28) - open/close 9270 (time 1776188911.50 total 120.10 last 72.78) total: 10000 open/close in 129.19 seconds: 77.41 ops/second - unlinked 0 (time 1776188926 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 737 (time 1776189023.64 total 10.01 last 73.64) - open/close 1485 (time 1776189033.64 total 20.01 last 74.75) - open/close 2268 (time 1776189043.65 total 30.02 last 78.24) - open/close 3181 (time 1776189053.65 total 40.03 last 91.27) - open/close 4085 (time 1776189063.66 total 50.03 last 90.37) - open/close 4860 (time 1776189073.66 total 60.03 last 77.48) - open/close 5691 (time 1776189083.67 total 70.04 last 83.03) - open/close 6496 (time 1776189093.68 total 80.05 last 80.41) - open/close 7287 (time 1776189103.68 total 90.05 last 79.10) - open/close 8154 (time 1776189113.69 total 100.06 last 86.65) - open/close 9009 (time 1776189123.70 total 110.07 last 85.41) - open/close 9882 (time 1776189133.70 total 120.08 last 87.25) total: 10000 open/close in 121.61 seconds: 82.23 ops/second - unlinked 0 (time 1776189138 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 745 (time 1776189226.19 total 10.02 last 74.34) - open/close 1540 (time 1776189236.19 total 20.02 last 79.49) - open/close 2215 (time 1776189246.19 total 30.03 last 67.48) - open/close 2956 (time 1776189256.21 total 40.04 last 73.98) - open/close 3714 (time 1776189266.22 total 50.06 last 75.69) - open/close 4504 (time 1776189276.23 total 60.07 last 78.92) - open/close 5295 (time 1776189286.24 total 70.07 last 79.05) - open/close 6055 (time 1776189296.25 total 80.09 last 75.91) - open/close 6766 (time 1776189306.26 total 90.10 last 71.01) - open/close 7583 (time 1776189316.26 total 100.10 last 81.70) - open/close 8411 (time 1776189326.27 total 110.10 last 82.76) - open/close 9210 (time 1776189336.27 total 120.10 last 79.89) - open/close 9918 (time 1776189346.28 total 130.11 last 70.72) total: 10000 open/close in 131.27 seconds: 76.18 ops/second - unlinked 0 (time 1776189352 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 777 (time 1776189446.50 total 10.01 last 77.63) - open/close 1451 (time 1776189456.52 total 20.03 last 67.27) - open/close 2186 (time 1776189466.53 total 30.03 last 73.49) - open/close 2881 (time 1776189476.53 total 40.03 last 69.49) - open/close 3576 (time 1776189486.54 total 50.04 last 69.44) - open/close 4295 (time 1776189496.54 total 60.05 last 71.85) - open/close 5014 (time 1776189506.55 total 70.06 last 71.81) - open/close 5757 (time 1776189516.56 total 80.06 last 74.27) - open/close 6322 (time 1776189526.57 total 90.07 last 56.45) - open/close 6886 (time 1776189536.57 total 100.07 last 56.38) - open/close 7593 (time 1776189546.57 total 110.08 last 70.68) - open/close 8340 (time 1776189556.58 total 120.08 last 74.67) - open/close 9079 (time 1776189566.58 total 130.09 last 73.88) - open/close 9838 (time 1776189576.58 total 140.09 last 75.89) total: 10000 open/close in 142.66 seconds: 70.10 ops/second - unlinked 0 (time 1776189584 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 634 (time 1776189686.07 total 10.01 last 63.37) - open/close 1316 (time 1776189696.08 total 20.01 last 68.17) - open/close 2025 (time 1776189706.08 total 30.01 last 70.89) - open/close 2627 (time 1776189716.08 total 40.01 last 60.19) - open/close 3376 (time 1776189726.09 total 50.02 last 74.84) - open/close 4115 (time 1776189736.09 total 60.02 last 73.87) - open/close 4838 (time 1776189746.10 total 70.03 last 72.26) - open/close 5572 (time 1776189756.11 total 80.05 last 73.28) - open/close 6313 (time 1776189766.12 total 90.05 last 74.07) - open/close 7093 (time 1776189776.12 total 100.05 last 77.98) - open/close 7782 (time 1776189786.13 total 110.06 last 68.85) - open/close 8267 (time 1776189796.14 total 120.08 last 48.42) - open/close 8775 (time 1776189806.15 total 130.08 last 50.78) - open/close 9543 (time 1776189816.16 total 140.10 last 76.66) total: 10000 open/close in 146.43 seconds: 68.29 ops/second - unlinked 0 (time 1776189827 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 771 (time 1776189933.93 total 10.01 last 77.04) - open/close 1587 (time 1776189943.96 total 20.03 last 81.38) - open/close 2361 (time 1776189953.96 total 30.03 last 77.40) - open/close 3113 (time 1776189963.98 total 40.05 last 75.08) - open/close 3913 (time 1776189973.99 total 50.07 last 79.87) - open/close 4651 (time 1776189984.00 total 60.07 last 73.75) - open/close 5407 (time 1776189994.00 total 70.08 last 75.58) - open/close 6175 (time 1776190004.00 total 80.08 last 76.78) - open/close 7012 (time 1776190014.01 total 90.08 last 83.67) - open/close 7866 (time 1776190024.02 total 100.09 last 85.31) - open/close 8603 (time 1776190034.02 total 110.09 last 73.70) - open/close 9375 (time 1776190044.03 total 120.10 last 77.14) total: 10000 open/close in 128.63 seconds: 77.74 ops/second - unlinked 0 (time 1776190056 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 794 (time 1776190155.36 total 10.00 last 79.38) - open/close 1446 (time 1776190165.37 total 20.01 last 65.15) - open/close 2171 (time 1776190175.38 total 30.02 last 72.45) - open/close 2878 (time 1776190185.39 total 40.03 last 70.62) - open/close 3483 (time 1776190195.40 total 50.04 last 60.45) - open/close 4074 (time 1776190205.41 total 60.05 last 59.03) - open/close 4755 (time 1776190215.41 total 70.05 last 68.07) - open/close 5483 (time 1776190225.42 total 80.06 last 72.75) - open/close 6242 (time 1776190235.44 total 90.08 last 75.77) - open/close 7040 (time 1776190245.44 total 100.08 last 79.79) - open/close 7853 (time 1776190255.45 total 110.08 last 81.24) - open/close 8579 (time 1776190265.45 total 120.09 last 72.56) - open/close 9260 (time 1776190275.46 total 130.10 last 68.01) total: 10000 open/close in 139.90 seconds: 71.48 ops/second - unlinked 0 (time 1776190289 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.148@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg248-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg248-server: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg248-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg248-server: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg248-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock oleg248-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg248-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing unload_modules_local oleg248-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg248-server: LNET ready to unload oleg248-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2