== conf-sanity test 69: replace an OST with the same index ========================================================== 16:57:00 (1776200220) start mds service on oleg242-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: detected 4 online CPUs by sysfs oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg242-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock oleg242-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 874 (time 1776200291.28 total 10.00 last 87.37) - open/close 1697 (time 1776200301.29 total 20.01 last 82.26) - open/close 2577 (time 1776200311.30 total 30.02 last 87.93) - open/close 3408 (time 1776200321.31 total 40.03 last 82.99) - open/close 4278 (time 1776200331.32 total 50.04 last 86.94) - open/close 5141 (time 1776200341.32 total 60.04 last 86.27) - open/close 6097 (time 1776200351.33 total 70.05 last 95.51) - open/close 6991 (time 1776200361.33 total 80.05 last 89.38) - open/close 7856 (time 1776200371.34 total 90.06 last 86.46) - open/close 8736 (time 1776200381.36 total 100.08 last 87.76) - open/close 9633 (time 1776200391.38 total 110.10 last 89.59) total: 10000 open/close in 113.77 seconds: 87.90 ops/second - unlinked 0 (time 1776200399 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 879 (time 1776200487.33 total 10.00 last 87.87) - open/close 1788 (time 1776200497.34 total 20.01 last 90.86) - open/close 2708 (time 1776200507.35 total 30.02 last 91.85) - open/close 3713 (time 1776200517.36 total 40.03 last 100.45) - open/close 4585 (time 1776200527.36 total 50.03 last 87.17) - open/close 5485 (time 1776200537.38 total 60.06 last 89.78) - open/close 6418 (time 1776200547.38 total 70.06 last 93.30) - open/close 7349 (time 1776200557.40 total 80.07 last 92.98) - open/close 8180 (time 1776200567.40 total 90.07 last 83.09) - open/close 9091 (time 1776200577.40 total 100.07 last 91.08) - open/close 9968 (time 1776200587.40 total 110.08 last 87.66) total: 10000 open/close in 110.44 seconds: 90.55 ops/second - unlinked 0 (time 1776200592 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 939 (time 1776200681.50 total 10.01 last 93.83) - open/close 1903 (time 1776200691.51 total 20.01 last 96.35) - open/close 2861 (time 1776200701.51 total 30.02 last 95.72) - open/close 3842 (time 1776200711.52 total 40.02 last 98.05) - open/close 4578 (time 1776200721.54 total 50.05 last 73.45) - open/close 5513 (time 1776200731.55 total 60.06 last 93.40) - open/close 6326 (time 1776200741.55 total 70.06 last 81.28) - open/close 7324 (time 1776200751.57 total 80.07 last 99.65) - open/close 8214 (time 1776200761.57 total 90.08 last 88.92) - open/close 8997 (time 1776200771.58 total 100.08 last 78.28) - open/close 9892 (time 1776200781.59 total 110.09 last 89.40) total: 10000 open/close in 111.18 seconds: 89.94 ops/second - unlinked 0 (time 1776200787 ; total 1 ; last 1) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 769 (time 1776200869.25 total 10.01 last 76.79) - open/close 1557 (time 1776200879.25 total 20.02 last 78.77) - open/close 2513 (time 1776200889.25 total 30.02 last 95.58) - open/close 3259 (time 1776200899.26 total 40.02 last 74.57) - open/close 4136 (time 1776200909.27 total 50.03 last 87.63) - open/close 4826 (time 1776200919.27 total 60.04 last 68.97) - open/close 5393 (time 1776200929.28 total 70.04 last 56.67) - open/close 6054 (time 1776200939.28 total 80.05 last 66.05) - open/close 7072 (time 1776200949.28 total 90.05 last 101.79) - open/close 8068 (time 1776200959.29 total 100.05 last 99.57) - open/close 8970 (time 1776200969.30 total 110.06 last 90.10) - open/close 9827 (time 1776200979.31 total 120.08 last 85.60) total: 10000 open/close in 122.13 seconds: 81.88 ops/second - unlinked 0 (time 1776200984 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 869 (time 1776201069.74 total 10.00 last 86.87) - open/close 1767 (time 1776201079.76 total 20.02 last 89.65) - open/close 2470 (time 1776201089.76 total 30.03 last 70.23) - open/close 3139 (time 1776201099.77 total 40.03 last 66.87) - open/close 4006 (time 1776201109.78 total 50.04 last 86.64) - open/close 4940 (time 1776201119.78 total 60.05 last 93.37) - open/close 5919 (time 1776201129.78 total 70.05 last 97.88) - open/close 6791 (time 1776201139.79 total 80.06 last 87.10) - open/close 7572 (time 1776201149.80 total 90.07 last 78.05) - open/close 8502 (time 1776201159.80 total 100.07 last 92.98) - open/close 9474 (time 1776201169.81 total 110.07 last 97.15) total: 10000 open/close in 115.61 seconds: 86.50 ops/second - unlinked 0 (time 1776201179 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1020 (time 1776201260.04 total 10.00 last 102.00) - open/close 2085 (time 1776201270.04 total 20.00 last 106.49) - open/close 3050 (time 1776201280.04 total 30.00 last 96.48) - open/close 3755 (time 1776201290.04 total 40.00 last 70.49) - open/close 4485 (time 1776201300.05 total 50.01 last 72.97) - open/close 5295 (time 1776201310.05 total 60.02 last 80.93) - open/close 6173 (time 1776201320.07 total 70.03 last 87.67) - open/close 6978 (time 1776201330.08 total 80.04 last 80.45) - open/close 7704 (time 1776201340.08 total 90.04 last 72.58) - open/close 8513 (time 1776201350.08 total 100.04 last 80.89) - open/close 9434 (time 1776201360.09 total 110.05 last 92.03) total: 10000 open/close in 117.07 seconds: 85.42 ops/second - unlinked 0 (time 1776201370 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 802 (time 1776201454.38 total 10.02 last 80.02) - open/close 1575 (time 1776201464.39 total 20.03 last 77.26) - open/close 2318 (time 1776201474.39 total 30.03 last 74.28) - open/close 3137 (time 1776201484.39 total 40.03 last 81.90) - open/close 4051 (time 1776201494.41 total 50.04 last 91.28) - open/close 4945 (time 1776201504.41 total 60.05 last 89.34) - open/close 5861 (time 1776201514.42 total 70.06 last 91.54) - open/close 6612 (time 1776201524.43 total 80.07 last 74.98) - open/close 7194 (time 1776201534.46 total 90.10 last 58.04) - open/close 7796 (time 1776201544.46 total 100.10 last 60.19) - open/close 8682 (time 1776201554.47 total 110.11 last 88.55) - open/close 9407 (time 1776201564.47 total 120.11 last 72.46) total: 10000 open/close in 127.51 seconds: 78.42 ops/second - unlinked 0 (time 1776201576 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 857 (time 1776201661.11 total 10.01 last 85.63) - open/close 1673 (time 1776201671.11 total 20.01 last 81.59) - open/close 2524 (time 1776201681.13 total 30.02 last 84.99) - open/close 3372 (time 1776201691.13 total 40.02 last 84.78) - open/close 4254 (time 1776201701.13 total 50.02 last 88.20) - open/close 4954 (time 1776201711.14 total 60.03 last 69.94) - open/close 5859 (time 1776201721.14 total 70.04 last 90.43) - open/close 6661 (time 1776201731.16 total 80.06 last 80.04) - open/close 7498 (time 1776201741.17 total 90.06 last 83.67) - open/close 8284 (time 1776201751.17 total 100.07 last 78.58) - open/close 9177 (time 1776201761.17 total 110.07 last 89.29) total: 10000 open/close in 118.77 seconds: 84.20 ops/second - unlinked 0 (time 1776201773 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 911 (time 1776201863.63 total 10.00 last 91.06) - open/close 1747 (time 1776201873.63 total 20.01 last 83.55) - open/close 2600 (time 1776201883.64 total 30.01 last 85.28) - open/close 3405 (time 1776201893.64 total 40.02 last 80.47) - open/close 4260 (time 1776201903.65 total 50.03 last 85.42) - open/close 5108 (time 1776201913.65 total 60.03 last 84.76) - open/close 5886 (time 1776201923.67 total 70.04 last 77.69) - open/close 6799 (time 1776201933.67 total 80.04 last 91.30) - open/close 7699 (time 1776201943.67 total 90.05 last 89.94) - open/close 8592 (time 1776201953.68 total 100.06 last 89.22) - open/close 9258 (time 1776201963.70 total 110.07 last 66.52) - open/close 9986 (time 1776201973.70 total 120.08 last 72.73) total: 10000 open/close in 120.25 seconds: 83.16 ops/second - unlinked 0 (time 1776201978 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 755 (time 1776202064.73 total 10.01 last 75.45) - open/close 1597 (time 1776202074.73 total 20.01 last 84.20) - open/close 2416 (time 1776202084.74 total 30.02 last 81.83) - open/close 3372 (time 1776202094.74 total 40.02 last 95.56) - open/close 4254 (time 1776202104.74 total 50.02 last 88.17) - open/close 5164 (time 1776202114.75 total 60.02 last 90.99) - open/close 6131 (time 1776202124.77 total 70.05 last 96.48) - open/close 7096 (time 1776202134.79 total 80.06 last 96.33) - open/close 7785 (time 1776202144.79 total 90.07 last 68.85) - open/close 8502 (time 1776202154.80 total 100.07 last 71.68) - open/close 9315 (time 1776202164.80 total 110.08 last 81.24) total: 10000 open/close in 117.06 seconds: 85.43 ops/second - unlinked 0 (time 1776202175 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg242-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock oleg242-server@tcp:/lustre /mnt/lustre On OST0, 3509 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg242-server: LNET ready to unload oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2