== conf-sanity test 69: replace an OST with the same index ========================================================== 14:33:58 (1776191638) start mds service on oleg440-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 532 (time 1776191723.57 total 10.01 last 53.12) - open/close 1086 (time 1776191733.58 total 20.02 last 55.35) - open/close 1639 (time 1776191743.60 total 30.04 last 55.21) - open/close 2241 (time 1776191753.60 total 40.05 last 60.15) - open/close 2843 (time 1776191763.62 total 50.07 last 60.10) - open/close 3466 (time 1776191773.63 total 60.07 last 62.26) - open/close 4006 (time 1776191783.64 total 70.08 last 53.95) - open/close 4507 (time 1776191793.64 total 80.09 last 50.08) - open/close 5090 (time 1776191803.67 total 90.11 last 58.16) - open/close 5687 (time 1776191813.67 total 100.11 last 59.68) - open/close 6294 (time 1776191823.68 total 110.12 last 60.66) - open/close 6911 (time 1776191833.69 total 120.13 last 61.62) - open/close 7460 (time 1776191843.69 total 130.14 last 54.88) - open/close 8010 (time 1776191853.76 total 140.20 last 54.65) - open/close 8525 (time 1776191863.77 total 150.22 last 51.41) - open/close 9123 (time 1776191873.79 total 160.23 last 59.73) - open/close 9700 (time 1776191883.80 total 170.25 last 57.61) total: 10000 open/close in 175.24 seconds: 57.07 ops/second - unlinked 0 (time 1776191892 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 623 (time 1776192005.11 total 10.00 last 62.29) - open/close 1237 (time 1776192015.11 total 20.01 last 61.35) - open/close 1807 (time 1776192025.11 total 30.01 last 57.00) - open/close 2426 (time 1776192035.12 total 40.02 last 61.85) - open/close 3043 (time 1776192045.15 total 50.05 last 61.51) - open/close 3706 (time 1776192055.16 total 60.05 last 66.27) - open/close 4322 (time 1776192065.16 total 70.06 last 61.60) - open/close 4952 (time 1776192075.16 total 80.06 last 62.97) - open/close 5637 (time 1776192085.18 total 90.07 last 68.42) - open/close 6277 (time 1776192095.21 total 100.11 last 63.79) - open/close 6888 (time 1776192105.22 total 110.11 last 61.05) - open/close 7516 (time 1776192115.22 total 120.12 last 62.78) - open/close 8082 (time 1776192125.23 total 130.13 last 56.55) - open/close 8642 (time 1776192135.23 total 140.13 last 56.00) - open/close 9200 (time 1776192145.24 total 150.14 last 55.73) - open/close 9691 (time 1776192155.25 total 160.15 last 49.06) total: 10000 open/close in 166.56 seconds: 60.04 ops/second - unlinked 0 (time 1776192166 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 574 (time 1776192285.23 total 10.01 last 57.37) - open/close 1182 (time 1776192295.24 total 20.01 last 60.75) - open/close 1825 (time 1776192305.27 total 30.04 last 64.10) - open/close 2413 (time 1776192315.27 total 40.05 last 58.78) - open/close 2952 (time 1776192325.28 total 50.05 last 53.86) - open/close 3502 (time 1776192335.28 total 60.06 last 54.99) - open/close 4078 (time 1776192345.29 total 70.06 last 57.57) - open/close 4669 (time 1776192355.30 total 80.08 last 59.00) - open/close 5241 (time 1776192365.31 total 90.09 last 57.16) - open/close 5760 (time 1776192375.33 total 100.10 last 51.82) - open/close 6331 (time 1776192385.34 total 110.12 last 56.99) - open/close 6919 (time 1776192395.36 total 120.14 last 58.69) - open/close 7536 (time 1776192405.38 total 130.15 last 61.63) - open/close 8087 (time 1776192415.39 total 140.16 last 55.03) - open/close 8704 (time 1776192425.39 total 150.17 last 61.66) - open/close 9279 (time 1776192435.42 total 160.20 last 57.35) - open/close 9883 (time 1776192445.43 total 170.21 last 60.31) total: 10000 open/close in 172.21 seconds: 58.07 ops/second - unlinked 0 (time 1776192451 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 488 (time 1776192565.59 total 10.01 last 48.77) - open/close 1047 (time 1776192575.59 total 20.01 last 55.90) - open/close 1637 (time 1776192585.60 total 30.01 last 58.95) - open/close 2268 (time 1776192595.60 total 40.02 last 63.06) - open/close 2797 (time 1776192605.61 total 50.03 last 52.84) - open/close 3388 (time 1776192615.63 total 60.05 last 58.99) - open/close 3983 (time 1776192625.65 total 70.07 last 59.37) - open/close 4517 (time 1776192635.68 total 80.09 last 53.29) - open/close 4983 (time 1776192645.70 total 90.12 last 46.47) - open/close 5559 (time 1776192655.71 total 100.12 last 57.59) - open/close 6168 (time 1776192665.72 total 110.14 last 60.81) - open/close 6796 (time 1776192675.72 total 120.14 last 62.79) - open/close 7428 (time 1776192685.73 total 130.15 last 63.13) - open/close 8080 (time 1776192695.73 total 140.15 last 65.19) - open/close 8718 (time 1776192705.75 total 150.16 last 63.72) - open/close 9371 (time 1776192715.75 total 160.17 last 65.29) total: 10000 open/close in 170.11 seconds: 58.78 ops/second - unlinked 0 (time 1776192731 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 420 (time 1776192847.13 total 10.02 last 41.92) - open/close 899 (time 1776192857.14 total 20.03 last 47.83) - open/close 1383 (time 1776192867.15 total 30.04 last 48.37) - open/close 1889 (time 1776192877.17 total 40.06 last 50.50) - open/close 2452 (time 1776192887.18 total 50.07 last 56.25) - open/close 3085 (time 1776192897.18 total 60.07 last 63.28) - open/close 3700 (time 1776192907.19 total 70.07 last 61.48) - open/close 4301 (time 1776192917.20 total 80.09 last 60.04) - open/close 4806 (time 1776192927.22 total 90.11 last 50.38) - open/close 5293 (time 1776192937.23 total 100.12 last 48.68) - open/close 5825 (time 1776192947.25 total 110.14 last 53.07) - open/close 6360 (time 1776192957.26 total 120.14 last 53.48) - open/close 6928 (time 1776192967.27 total 130.16 last 56.73) - open/close 7537 (time 1776192977.28 total 140.17 last 60.84) - open/close 8139 (time 1776192987.28 total 150.17 last 60.19) - open/close 8708 (time 1776192997.28 total 160.17 last 56.87) - open/close 9259 (time 1776193007.29 total 170.17 last 55.09) - open/close 9828 (time 1776193017.29 total 180.18 last 56.85) total: 10000 open/close in 183.41 seconds: 54.52 ops/second - unlinked 0 (time 1776193025 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 597 (time 1776193141.87 total 10.01 last 59.61) - open/close 1231 (time 1776193151.87 total 20.02 last 63.39) - open/close 1817 (time 1776193161.89 total 30.03 last 58.52) - open/close 2357 (time 1776193171.90 total 40.04 last 53.94) - open/close 2917 (time 1776193181.92 total 50.06 last 55.89) - open/close 3553 (time 1776193191.92 total 60.06 last 63.58) - open/close 4145 (time 1776193201.94 total 70.08 last 59.08) - open/close 4733 (time 1776193211.95 total 80.09 last 58.75) - open/close 5360 (time 1776193221.95 total 90.10 last 62.66) - open/close 5989 (time 1776193231.96 total 100.10 last 62.90) - open/close 6610 (time 1776193241.96 total 110.10 last 62.09) - open/close 7174 (time 1776193251.96 total 120.11 last 56.38) - open/close 7729 (time 1776193261.97 total 130.11 last 55.47) - open/close 8237 (time 1776193271.98 total 140.12 last 50.75) - open/close 8693 (time 1776193281.99 total 150.13 last 45.56) - open/close 9085 (time 1776193292.01 total 160.15 last 39.12) - open/close 9560 (time 1776193302.01 total 170.16 last 47.47) total: 10000 open/close in 177.21 seconds: 56.43 ops/second - unlinked 0 (time 1776193313 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 581 (time 1776193432.93 total 10.00 last 58.08) - open/close 1190 (time 1776193442.93 total 20.00 last 60.90) - open/close 1772 (time 1776193452.94 total 30.02 last 58.12) - open/close 2317 (time 1776193462.95 total 40.03 last 54.45) - open/close 2889 (time 1776193472.96 total 50.04 last 57.12) - open/close 3406 (time 1776193482.98 total 60.05 last 51.63) - open/close 4001 (time 1776193492.99 total 70.06 last 59.44) - open/close 4596 (time 1776193502.99 total 80.07 last 59.49) - open/close 5114 (time 1776193513.00 total 90.07 last 51.75) - open/close 5724 (time 1776193523.01 total 100.08 last 60.94) - open/close 6296 (time 1776193533.02 total 110.10 last 57.10) - open/close 6859 (time 1776193543.03 total 120.11 last 56.25) - open/close 7285 (time 1776193553.05 total 130.13 last 42.53) - open/close 7744 (time 1776193563.06 total 140.14 last 45.85) - open/close 8170 (time 1776193573.07 total 150.15 last 42.54) - open/close 8603 (time 1776193583.07 total 160.15 last 43.30) - open/close 9123 (time 1776193593.09 total 170.17 last 51.93) - open/close 9640 (time 1776193603.10 total 180.18 last 51.62) total: 10000 open/close in 186.62 seconds: 53.59 ops/second - unlinked 0 (time 1776193614 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 541 (time 1776193733.30 total 10.01 last 54.06) - open/close 1149 (time 1776193743.30 total 20.01 last 60.80) - open/close 1686 (time 1776193753.31 total 30.02 last 53.62) - open/close 2272 (time 1776193763.34 total 40.05 last 58.44) - open/close 2866 (time 1776193773.34 total 50.05 last 59.40) - open/close 3423 (time 1776193783.34 total 60.05 last 55.67) - open/close 3985 (time 1776193793.34 total 70.05 last 56.20) - open/close 4553 (time 1776193803.36 total 80.07 last 56.70) - open/close 5099 (time 1776193813.36 total 90.07 last 54.60) - open/close 5545 (time 1776193823.37 total 100.08 last 44.55) - open/close 6146 (time 1776193833.38 total 110.10 last 60.03) - open/close 6684 (time 1776193843.40 total 120.11 last 53.72) - open/close 7290 (time 1776193853.41 total 130.12 last 60.57) - open/close 7853 (time 1776193863.42 total 140.13 last 56.24) - open/close 8478 (time 1776193873.43 total 150.14 last 62.44) - open/close 9167 (time 1776193883.44 total 160.15 last 68.83) - open/close 9741 (time 1776193893.45 total 170.16 last 57.33) total: 10000 open/close in 175.05 seconds: 57.13 ops/second - unlinked 0 (time 1776193903 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 512 (time 1776194022.67 total 10.00 last 51.19) - open/close 1095 (time 1776194032.68 total 20.01 last 58.25) - open/close 1701 (time 1776194042.68 total 30.01 last 60.58) - open/close 2278 (time 1776194052.69 total 40.02 last 57.68) - open/close 2901 (time 1776194062.70 total 50.03 last 62.21) - open/close 3518 (time 1776194072.71 total 60.04 last 61.65) - open/close 4171 (time 1776194082.71 total 70.05 last 65.26) - open/close 4797 (time 1776194092.72 total 80.05 last 62.57) - open/close 5335 (time 1776194102.73 total 90.06 last 53.76) - open/close 5937 (time 1776194112.73 total 100.06 last 60.18) - open/close 6516 (time 1776194122.74 total 110.07 last 57.85) - open/close 7125 (time 1776194132.74 total 120.07 last 60.89) - open/close 7651 (time 1776194142.74 total 130.08 last 52.58) - open/close 8076 (time 1776194152.75 total 140.09 last 42.46) - open/close 8596 (time 1776194162.76 total 150.09 last 51.96) - open/close 9235 (time 1776194172.78 total 160.11 last 63.77) - open/close 9861 (time 1776194182.79 total 170.12 last 62.54) total: 10000 open/close in 172.70 seconds: 57.90 ops/second - unlinked 0 (time 1776194189 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 578 (time 1776194305.87 total 10.01 last 57.75) - open/close 1101 (time 1776194315.88 total 20.02 last 52.25) - open/close 1689 (time 1776194325.89 total 30.02 last 58.79) - open/close 2285 (time 1776194335.91 total 40.05 last 59.45) - open/close 2889 (time 1776194345.91 total 50.05 last 60.39) - open/close 3489 (time 1776194355.91 total 60.05 last 59.99) - open/close 4108 (time 1776194365.92 total 70.05 last 61.86) - open/close 4747 (time 1776194375.92 total 80.06 last 63.90) - open/close 5346 (time 1776194385.93 total 90.06 last 59.87) - open/close 5971 (time 1776194395.93 total 100.07 last 62.46) - open/close 6520 (time 1776194405.93 total 110.07 last 54.89) - open/close 7106 (time 1776194415.95 total 120.08 last 58.50) - open/close 7692 (time 1776194425.97 total 130.11 last 58.48) - open/close 8287 (time 1776194435.98 total 140.11 last 59.47) - open/close 8914 (time 1776194445.98 total 150.12 last 62.65) - open/close 9537 (time 1776194455.99 total 160.12 last 62.27) total: 10000 open/close in 167.17 seconds: 59.82 ops/second - unlinked 0 (time 1776194467 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.140@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre On OST0, 291 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg440-server: LNET ready to unload oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2