== conf-sanity test 69: replace an OST with the same index ========================================================== 16:02:12 (1776196932) start mds service on oleg318-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg318-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 656 (time 1776197011.52 total 10.01 last 65.55) - open/close 1290 (time 1776197021.53 total 20.02 last 63.34) - open/close 1942 (time 1776197031.54 total 30.03 last 65.14) - open/close 2521 (time 1776197041.55 total 40.03 last 57.85) - open/close 3227 (time 1776197051.57 total 50.05 last 70.47) - open/close 3855 (time 1776197061.58 total 60.06 last 62.75) - open/close 4521 (time 1776197071.60 total 70.08 last 66.46) - open/close 5170 (time 1776197081.61 total 80.09 last 64.83) - open/close 5889 (time 1776197091.62 total 90.10 last 71.83) - open/close 6587 (time 1776197101.62 total 100.11 last 69.77) - open/close 7222 (time 1776197111.63 total 110.11 last 63.47) - open/close 7878 (time 1776197121.63 total 120.11 last 65.59) - open/close 8580 (time 1776197131.65 total 130.14 last 70.05) - open/close 9133 (time 1776197141.66 total 140.15 last 55.23) - open/close 9612 (time 1776197151.68 total 150.17 last 47.82) total: 10000 open/close in 156.45 seconds: 63.92 ops/second - unlinked 0 (time 1776197162 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 596 (time 1776197267.61 total 10.02 last 59.49) - open/close 1189 (time 1776197277.62 total 20.02 last 59.27) - open/close 1864 (time 1776197287.63 total 30.03 last 67.45) - open/close 2489 (time 1776197297.63 total 40.03 last 62.49) - open/close 3104 (time 1776197307.63 total 50.04 last 61.46) - open/close 3753 (time 1776197317.64 total 60.05 last 64.85) - open/close 4403 (time 1776197327.65 total 70.06 last 64.94) - open/close 5087 (time 1776197337.67 total 80.07 last 68.30) - open/close 5781 (time 1776197347.68 total 90.09 last 69.31) - open/close 6433 (time 1776197357.71 total 100.11 last 65.03) - open/close 7068 (time 1776197367.71 total 110.12 last 63.46) - open/close 7713 (time 1776197377.72 total 120.12 last 64.46) - open/close 8355 (time 1776197387.72 total 130.13 last 64.18) - open/close 8926 (time 1776197397.72 total 140.13 last 57.10) - open/close 9535 (time 1776197407.73 total 150.14 last 60.84) total: 10000 open/close in 158.35 seconds: 63.15 ops/second - unlinked 0 (time 1776197421 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 648 (time 1776197526.57 total 10.00 last 64.79) - open/close 1298 (time 1776197536.57 total 20.01 last 64.96) - open/close 1967 (time 1776197546.59 total 30.02 last 66.80) - open/close 2633 (time 1776197556.59 total 40.02 last 66.59) - open/close 3288 (time 1776197566.59 total 50.03 last 65.49) - open/close 3915 (time 1776197576.59 total 60.03 last 62.67) - open/close 4585 (time 1776197586.61 total 70.04 last 66.93) - open/close 5237 (time 1776197596.63 total 80.07 last 65.05) - open/close 5885 (time 1776197606.64 total 90.07 last 64.75) - open/close 6529 (time 1776197616.64 total 100.07 last 64.39) - open/close 7166 (time 1776197626.64 total 110.08 last 63.66) - open/close 7793 (time 1776197636.66 total 120.10 last 62.57) - open/close 8477 (time 1776197646.67 total 130.11 last 68.34) - open/close 9116 (time 1776197656.68 total 140.12 last 63.83) - open/close 9683 (time 1776197666.71 total 150.15 last 56.54) total: 10000 open/close in 155.20 seconds: 64.43 ops/second - unlinked 0 (time 1776197676 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 716 (time 1776197784.76 total 10.00 last 71.60) - open/close 1368 (time 1776197794.77 total 20.00 last 65.20) - open/close 2030 (time 1776197804.78 total 30.01 last 66.13) - open/close 2706 (time 1776197814.78 total 40.01 last 67.58) - open/close 3356 (time 1776197824.78 total 50.02 last 64.99) - open/close 4040 (time 1776197834.81 total 60.04 last 68.23) - open/close 4697 (time 1776197844.82 total 70.06 last 65.59) - open/close 5348 (time 1776197854.86 total 80.09 last 64.90) - open/close 6044 (time 1776197864.86 total 90.10 last 69.54) - open/close 6697 (time 1776197874.87 total 100.11 last 65.24) - open/close 7358 (time 1776197884.88 total 110.11 last 66.06) - open/close 7995 (time 1776197894.88 total 120.11 last 63.70) - open/close 8637 (time 1776197904.90 total 130.14 last 64.05) - open/close 9365 (time 1776197914.91 total 140.15 last 72.71) total: 10000 open/close in 148.80 seconds: 67.21 ops/second - unlinked 0 (time 1776197926 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 647 (time 1776198025.87 total 10.01 last 64.63) - open/close 1277 (time 1776198035.88 total 20.02 last 62.91) - open/close 1922 (time 1776198045.89 total 30.03 last 64.47) - open/close 2592 (time 1776198055.91 total 40.05 last 66.86) - open/close 3261 (time 1776198065.92 total 50.06 last 66.81) - open/close 3857 (time 1776198075.95 total 60.09 last 59.44) - open/close 4499 (time 1776198085.96 total 70.10 last 64.13) - open/close 5251 (time 1776198095.96 total 80.11 last 75.16) - open/close 5921 (time 1776198105.98 total 90.12 last 66.92) - open/close 6512 (time 1776198115.98 total 100.12 last 59.09) - open/close 7125 (time 1776198125.99 total 110.13 last 61.26) - open/close 7814 (time 1776198136.00 total 120.14 last 68.82) - open/close 8515 (time 1776198146.00 total 130.14 last 70.06) - open/close 9226 (time 1776198156.01 total 140.15 last 71.06) - open/close 9983 (time 1776198166.02 total 150.16 last 75.62) total: 10000 open/close in 150.45 seconds: 66.47 ops/second - unlinked 0 (time 1776198170 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 697 (time 1776198277.93 total 10.02 last 69.59) - open/close 1398 (time 1776198287.95 total 20.03 last 70.00) - open/close 2184 (time 1776198297.95 total 30.03 last 78.58) - open/close 2847 (time 1776198307.96 total 40.04 last 66.22) - open/close 3539 (time 1776198317.97 total 50.06 last 69.11) - open/close 4035 (time 1776198327.98 total 60.06 last 49.59) - open/close 4593 (time 1776198337.99 total 70.07 last 55.74) - open/close 5229 (time 1776198347.99 total 80.07 last 63.58) - open/close 5859 (time 1776198358.00 total 90.08 last 62.94) - open/close 6578 (time 1776198368.01 total 100.10 last 71.82) - open/close 7257 (time 1776198378.02 total 110.10 last 67.84) - open/close 7955 (time 1776198388.03 total 120.11 last 69.76) - open/close 8673 (time 1776198398.04 total 130.12 last 71.73) - open/close 9372 (time 1776198408.04 total 140.12 last 69.89) - open/close 9975 (time 1776198418.04 total 150.13 last 60.26) total: 10000 open/close in 150.44 seconds: 66.47 ops/second - unlinked 0 (time 1776198422 ; total 1 ; last 1) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 588 (time 1776198520.76 total 10.00 last 58.79) - open/close 1063 (time 1776198530.76 total 20.00 last 47.50) - open/close 1614 (time 1776198540.76 total 30.00 last 55.10) - open/close 2260 (time 1776198550.77 total 40.01 last 64.53) - open/close 2978 (time 1776198560.77 total 50.02 last 71.78) - open/close 3726 (time 1776198570.78 total 60.02 last 74.74) - open/close 4412 (time 1776198580.78 total 70.02 last 68.60) - open/close 5097 (time 1776198590.78 total 80.03 last 68.48) - open/close 5769 (time 1776198600.79 total 90.04 last 67.14) - open/close 6482 (time 1776198610.80 total 100.05 last 71.21) - open/close 7198 (time 1776198620.81 total 110.05 last 71.57) - open/close 7932 (time 1776198630.81 total 120.06 last 73.38) - open/close 8586 (time 1776198640.82 total 130.07 last 65.30) - open/close 9269 (time 1776198650.82 total 140.07 last 68.30) - open/close 9933 (time 1776198660.84 total 150.08 last 66.33) total: 10000 open/close in 151.39 seconds: 66.06 ops/second - unlinked 0 (time 1776198666 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 720 (time 1776198767.01 total 10.01 last 71.96) - open/close 1386 (time 1776198777.02 total 20.01 last 66.56) - open/close 2079 (time 1776198787.02 total 30.01 last 69.29) - open/close 2713 (time 1776198797.03 total 40.03 last 63.32) - open/close 3369 (time 1776198807.05 total 50.04 last 65.52) - open/close 4051 (time 1776198817.07 total 60.06 last 68.05) - open/close 4766 (time 1776198827.07 total 70.07 last 71.46) - open/close 5495 (time 1776198837.08 total 80.07 last 72.85) - open/close 6201 (time 1776198847.10 total 90.09 last 70.48) - open/close 6996 (time 1776198857.11 total 100.10 last 79.44) - open/close 7680 (time 1776198867.12 total 110.11 last 68.32) - open/close 8428 (time 1776198877.12 total 120.11 last 74.77) - open/close 9183 (time 1776198887.12 total 130.12 last 75.49) total: 10000 open/close in 140.03 seconds: 71.42 ops/second - unlinked 0 (time 1776198899 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 712 (time 1776199006.82 total 10.01 last 71.15) - open/close 1373 (time 1776199016.83 total 20.01 last 66.07) - open/close 2006 (time 1776199026.84 total 30.02 last 63.23) - open/close 2683 (time 1776199036.84 total 40.02 last 67.69) - open/close 3370 (time 1776199046.84 total 50.03 last 68.65) - open/close 4021 (time 1776199056.85 total 60.04 last 65.07) - open/close 4689 (time 1776199066.85 total 70.04 last 66.79) - open/close 5301 (time 1776199076.85 total 80.04 last 61.18) - open/close 5871 (time 1776199086.85 total 90.04 last 57.00) - open/close 6450 (time 1776199096.87 total 100.05 last 57.83) - open/close 7096 (time 1776199106.87 total 110.05 last 64.60) - open/close 7762 (time 1776199116.87 total 120.06 last 66.57) - open/close 8430 (time 1776199126.88 total 130.07 last 66.72) - open/close 9141 (time 1776199136.89 total 140.07 last 71.06) - open/close 9866 (time 1776199146.89 total 150.08 last 72.48) total: 10000 open/close in 151.97 seconds: 65.80 ops/second - unlinked 0 (time 1776199152 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 689 (time 1776199249.47 total 10.00 last 68.87) - open/close 1367 (time 1776199259.49 total 20.03 last 67.63) - open/close 2094 (time 1776199269.49 total 30.03 last 72.69) - open/close 2837 (time 1776199279.51 total 40.04 last 74.21) - open/close 3653 (time 1776199289.51 total 50.05 last 81.58) - open/close 4369 (time 1776199299.51 total 60.05 last 71.59) - open/close 5065 (time 1776199309.51 total 70.05 last 69.58) - open/close 5764 (time 1776199319.52 total 80.05 last 69.90) - open/close 6460 (time 1776199329.53 total 90.06 last 69.52) - open/close 7150 (time 1776199339.54 total 100.07 last 68.93) - open/close 7812 (time 1776199349.54 total 110.08 last 66.19) - open/close 8504 (time 1776199359.55 total 120.09 last 69.10) - open/close 9057 (time 1776199369.57 total 130.10 last 55.22) - open/close 9647 (time 1776199379.59 total 140.13 last 58.87) total: 10000 open/close in 146.28 seconds: 68.36 ops/second - unlinked 0 (time 1776199391 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.118@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.118@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg318-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg318-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg318-server: LNET ready to unload oleg318-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2