== conf-sanity test 69: replace an OST with the same index ========================================================== 13:32:02 (1776187922) start mds service on oleg357-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg357-server: detected 4 online CPUs by sysfs oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 750 (time 1776187996.80 total 10.02 last 74.85) - open/close 1540 (time 1776188006.80 total 20.02 last 79.00) - open/close 2290 (time 1776188016.81 total 30.03 last 74.91) - open/close 2999 (time 1776188026.82 total 40.04 last 70.86) - open/close 3666 (time 1776188036.82 total 50.04 last 66.67) - open/close 4335 (time 1776188046.83 total 60.05 last 66.83) - open/close 5046 (time 1776188056.84 total 70.06 last 71.03) - open/close 5625 (time 1776188066.85 total 80.07 last 57.84) - open/close 6274 (time 1776188076.86 total 90.08 last 64.83) - open/close 6978 (time 1776188086.86 total 100.09 last 70.38) - open/close 7533 (time 1776188096.88 total 110.11 last 55.39) - open/close 8195 (time 1776188106.90 total 120.12 last 66.09) - open/close 8860 (time 1776188116.91 total 130.13 last 66.44) - open/close 9536 (time 1776188126.91 total 140.13 last 67.60) total: 10000 open/close in 149.08 seconds: 67.08 ops/second - unlinked 0 (time 1776188141 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 719 (time 1776188232.81 total 10.00 last 71.87) - open/close 1413 (time 1776188242.82 total 20.01 last 69.33) - open/close 2114 (time 1776188252.83 total 30.03 last 70.01) - open/close 2870 (time 1776188262.84 total 40.04 last 75.54) - open/close 3536 (time 1776188272.86 total 50.05 last 66.52) - open/close 4236 (time 1776188282.86 total 60.06 last 69.94) - open/close 4969 (time 1776188292.87 total 70.06 last 73.26) - open/close 5682 (time 1776188302.87 total 80.06 last 71.29) - open/close 6434 (time 1776188312.87 total 90.07 last 75.18) - open/close 7202 (time 1776188322.88 total 100.07 last 76.77) - open/close 7870 (time 1776188332.89 total 110.08 last 66.72) - open/close 8537 (time 1776188342.90 total 120.09 last 66.65) - open/close 9169 (time 1776188352.90 total 130.09 last 63.19) - open/close 9744 (time 1776188362.90 total 140.10 last 57.47) total: 10000 open/close in 143.99 seconds: 69.45 ops/second - unlinked 0 (time 1776188370 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 671 (time 1776188468.34 total 10.01 last 67.01) - open/close 1319 (time 1776188478.34 total 20.02 last 64.77) - open/close 2004 (time 1776188488.35 total 30.03 last 68.43) - open/close 2647 (time 1776188498.37 total 40.05 last 64.17) - open/close 3291 (time 1776188508.38 total 50.05 last 64.36) - open/close 3998 (time 1776188518.40 total 60.07 last 70.58) - open/close 4629 (time 1776188528.40 total 70.08 last 63.06) - open/close 5331 (time 1776188538.41 total 80.08 last 70.17) - open/close 6011 (time 1776188548.41 total 90.09 last 67.97) - open/close 6667 (time 1776188558.42 total 100.09 last 65.57) - open/close 7308 (time 1776188568.43 total 110.11 last 64.00) - open/close 8055 (time 1776188578.44 total 120.11 last 74.66) - open/close 8734 (time 1776188588.44 total 130.11 last 67.89) - open/close 9424 (time 1776188598.44 total 140.12 last 68.99) total: 10000 open/close in 148.01 seconds: 67.56 ops/second - unlinked 0 (time 1776188610 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 770 (time 1776188702.64 total 10.01 last 76.96) - open/close 1480 (time 1776188712.65 total 20.02 last 70.92) - open/close 2286 (time 1776188722.65 total 30.02 last 80.54) - open/close 3018 (time 1776188732.67 total 40.04 last 73.06) - open/close 3709 (time 1776188742.68 total 50.05 last 69.07) - open/close 4441 (time 1776188752.68 total 60.05 last 73.15) - open/close 5181 (time 1776188762.69 total 70.06 last 73.96) - open/close 5865 (time 1776188772.70 total 80.07 last 68.34) - open/close 6559 (time 1776188782.71 total 90.08 last 69.34) - open/close 7297 (time 1776188792.71 total 100.08 last 73.79) - open/close 8019 (time 1776188802.72 total 110.09 last 72.10) - open/close 8667 (time 1776188812.73 total 120.10 last 64.75) - open/close 9444 (time 1776188822.75 total 130.12 last 77.58) total: 10000 open/close in 137.38 seconds: 72.79 ops/second - unlinked 0 (time 1776188833 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 744 (time 1776188926.20 total 10.00 last 74.39) - open/close 1456 (time 1776188936.21 total 20.01 last 71.10) - open/close 2174 (time 1776188946.22 total 30.02 last 71.76) - open/close 2935 (time 1776188956.23 total 40.03 last 76.04) - open/close 3682 (time 1776188966.24 total 50.04 last 74.63) - open/close 4450 (time 1776188976.24 total 60.04 last 76.77) - open/close 5171 (time 1776188986.24 total 70.04 last 72.08) - open/close 5809 (time 1776188996.25 total 80.05 last 63.77) - open/close 6564 (time 1776189006.25 total 90.05 last 75.47) - open/close 7206 (time 1776189016.26 total 100.06 last 64.17) - open/close 7884 (time 1776189026.26 total 110.06 last 67.77) - open/close 8530 (time 1776189036.26 total 120.06 last 64.59) - open/close 9204 (time 1776189046.28 total 130.08 last 67.29) - open/close 9823 (time 1776189056.29 total 140.09 last 61.83) total: 10000 open/close in 143.04 seconds: 69.91 ops/second - unlinked 0 (time 1776189063 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 679 (time 1776189169.56 total 10.00 last 67.88) - open/close 1314 (time 1776189179.56 total 20.00 last 63.49) - open/close 1995 (time 1776189189.56 total 30.01 last 68.09) - open/close 2642 (time 1776189199.57 total 40.01 last 64.66) - open/close 3331 (time 1776189209.58 total 50.02 last 68.82) - open/close 4052 (time 1776189219.58 total 60.03 last 72.07) - open/close 4768 (time 1776189229.60 total 70.04 last 71.50) - open/close 5451 (time 1776189239.60 total 80.05 last 68.27) - open/close 6168 (time 1776189249.61 total 90.05 last 71.64) - open/close 6821 (time 1776189259.61 total 100.06 last 65.27) - open/close 7507 (time 1776189269.64 total 110.08 last 68.44) - open/close 8247 (time 1776189279.64 total 120.08 last 74.00) - open/close 8982 (time 1776189289.65 total 130.10 last 73.41) - open/close 9782 (time 1776189299.66 total 140.10 last 79.96) total: 10000 open/close in 143.09 seconds: 69.88 ops/second - unlinked 0 (time 1776189306 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 637 (time 1776189403.44 total 10.02 last 63.56) - open/close 1311 (time 1776189413.46 total 20.03 last 67.31) - open/close 2048 (time 1776189423.47 total 30.04 last 73.63) - open/close 2765 (time 1776189433.47 total 40.05 last 71.65) - open/close 3443 (time 1776189443.48 total 50.06 last 67.72) - open/close 4114 (time 1776189453.50 total 60.08 last 66.98) - open/close 4816 (time 1776189463.51 total 70.09 last 70.13) - open/close 5469 (time 1776189473.54 total 80.12 last 65.09) - open/close 6140 (time 1776189483.55 total 90.13 last 67.07) - open/close 6800 (time 1776189493.55 total 100.13 last 66.00) - open/close 7501 (time 1776189503.56 total 110.14 last 70.03) - open/close 8199 (time 1776189513.57 total 120.14 last 69.74) - open/close 8937 (time 1776189523.57 total 130.15 last 73.75) - open/close 9622 (time 1776189533.57 total 140.15 last 68.49) total: 10000 open/close in 145.56 seconds: 68.70 ops/second - unlinked 0 (time 1776189542 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 630 (time 1776189647.27 total 10.00 last 62.99) - open/close 1263 (time 1776189657.27 total 20.00 last 63.28) - open/close 1867 (time 1776189667.28 total 30.01 last 60.35) - open/close 2542 (time 1776189677.29 total 40.02 last 67.45) - open/close 3233 (time 1776189687.29 total 50.02 last 69.09) - open/close 3903 (time 1776189697.30 total 60.03 last 66.96) - open/close 4549 (time 1776189707.30 total 70.03 last 64.58) - open/close 5166 (time 1776189717.30 total 80.03 last 61.67) - open/close 5723 (time 1776189727.31 total 90.04 last 55.65) - open/close 6233 (time 1776189737.32 total 100.05 last 50.95) - open/close 6809 (time 1776189747.34 total 110.07 last 57.51) - open/close 7414 (time 1776189757.35 total 120.08 last 60.45) - open/close 8060 (time 1776189767.37 total 130.10 last 64.46) - open/close 8642 (time 1776189777.38 total 140.11 last 58.15) - open/close 9256 (time 1776189787.39 total 150.12 last 61.35) - open/close 9889 (time 1776189797.40 total 160.13 last 63.23) total: 10000 open/close in 161.91 seconds: 61.76 ops/second - unlinked 0 (time 1776189804 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 617 (time 1776189912.41 total 10.01 last 61.67) - open/close 1302 (time 1776189922.41 total 20.01 last 68.49) - open/close 1895 (time 1776189932.42 total 30.02 last 59.25) - open/close 2470 (time 1776189942.43 total 40.03 last 57.43) - open/close 3196 (time 1776189952.44 total 50.03 last 72.59) - open/close 3903 (time 1776189962.44 total 60.04 last 70.64) - open/close 4559 (time 1776189972.46 total 70.05 last 65.53) - open/close 5230 (time 1776189982.46 total 80.05 last 67.06) - open/close 5900 (time 1776189992.47 total 90.06 last 66.93) - open/close 6575 (time 1776190002.50 total 100.09 last 67.33) - open/close 7257 (time 1776190012.51 total 110.10 last 68.13) - open/close 7857 (time 1776190022.51 total 120.11 last 59.95) - open/close 8511 (time 1776190032.52 total 130.11 last 65.38) - open/close 9169 (time 1776190042.52 total 140.11 last 65.78) - open/close 9850 (time 1776190052.53 total 150.12 last 68.05) total: 10000 open/close in 152.34 seconds: 65.64 ops/second - unlinked 0 (time 1776190058 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 729 (time 1776190152.47 total 10.00 last 72.87) - open/close 1491 (time 1776190162.48 total 20.02 last 76.11) - open/close 2252 (time 1776190172.48 total 30.02 last 76.07) - open/close 2923 (time 1776190182.49 total 40.03 last 67.01) - open/close 3674 (time 1776190192.51 total 50.05 last 75.00) - open/close 4399 (time 1776190202.52 total 60.06 last 72.42) - open/close 5186 (time 1776190212.53 total 70.06 last 78.64) - open/close 5987 (time 1776190222.53 total 80.07 last 80.06) - open/close 6836 (time 1776190232.54 total 90.07 last 84.87) - open/close 7698 (time 1776190242.54 total 100.08 last 86.15) - open/close 8512 (time 1776190252.54 total 110.08 last 81.37) - open/close 9396 (time 1776190262.55 total 120.08 last 88.39) total: 10000 open/close in 128.23 seconds: 77.99 ops/second - unlinked 0 (time 1776190275 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.157@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg357-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local oleg357-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg357-server: LNET ready to unload oleg357-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2