== conf-sanity test 69: replace an OST with the same index ========================================================== 12:23:38 (1781281418) start mds service on oleg306-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg306-server: detected 4 online CPUs by sysfs oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg306-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock oleg306-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 612 (time 1781281524.48 total 10.00 last 61.20) - open/close 1273 (time 1781281534.48 total 20.01 last 66.05) - open/close 1893 (time 1781281544.49 total 30.02 last 61.93) - open/close 2576 (time 1781281554.51 total 40.03 last 68.23) - open/close 3239 (time 1781281564.52 total 50.04 last 66.21) - open/close 3933 (time 1781281574.52 total 60.04 last 69.40) - open/close 4672 (time 1781281584.53 total 70.05 last 73.84) - open/close 5370 (time 1781281594.54 total 80.06 last 69.73) - open/close 6117 (time 1781281604.54 total 90.06 last 74.69) - open/close 6828 (time 1781281614.54 total 100.06 last 71.10) - open/close 7503 (time 1781281624.54 total 110.07 last 67.47) - open/close 8098 (time 1781281634.57 total 120.10 last 59.31) - open/close 8783 (time 1781281644.58 total 130.10 last 68.47) - open/close 9424 (time 1781281654.58 total 140.10 last 64.09) total: 10000 open/close in 148.04 seconds: 67.55 ops/second - unlinked 0 (time 1781281667 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 738 (time 1781281768.95 total 10.00 last 73.78) - open/close 1449 (time 1781281778.96 total 20.01 last 71.05) - open/close 2172 (time 1781281788.97 total 30.02 last 72.24) - open/close 2950 (time 1781281798.97 total 40.02 last 77.79) - open/close 3685 (time 1781281808.98 total 50.03 last 73.42) - open/close 4343 (time 1781281818.99 total 60.04 last 65.74) - open/close 5092 (time 1781281829.00 total 70.05 last 74.83) - open/close 5805 (time 1781281839.01 total 80.06 last 71.23) - open/close 6545 (time 1781281849.04 total 90.08 last 73.81) - open/close 7194 (time 1781281859.04 total 100.09 last 64.87) - open/close 7931 (time 1781281869.05 total 110.10 last 73.59) - open/close 8604 (time 1781281879.07 total 120.12 last 67.22) - open/close 9254 (time 1781281889.07 total 130.12 last 64.97) - open/close 9904 (time 1781281899.07 total 140.12 last 64.98) total: 10000 open/close in 141.58 seconds: 70.63 ops/second - unlinked 0 (time 1781281904 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 824 (time 1781282000.33 total 10.01 last 82.31) - open/close 1600 (time 1781282010.34 total 20.01 last 77.59) - open/close 2403 (time 1781282020.34 total 30.02 last 80.23) - open/close 3149 (time 1781282030.35 total 40.03 last 74.56) - open/close 3842 (time 1781282040.35 total 50.03 last 69.29) - open/close 4567 (time 1781282050.38 total 60.05 last 72.31) - open/close 5289 (time 1781282060.38 total 70.05 last 72.20) - open/close 6049 (time 1781282070.38 total 80.05 last 75.99) - open/close 6742 (time 1781282080.38 total 90.06 last 69.27) - open/close 7478 (time 1781282090.39 total 100.07 last 73.54) - open/close 8185 (time 1781282100.40 total 110.07 last 70.64) - open/close 8916 (time 1781282110.41 total 120.09 last 73.01) - open/close 9595 (time 1781282120.42 total 130.09 last 67.87) total: 10000 open/close in 135.80 seconds: 73.64 ops/second - unlinked 0 (time 1781282129 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 758 (time 1781282225.72 total 10.01 last 75.76) - open/close 1549 (time 1781282235.73 total 20.02 last 79.01) - open/close 2239 (time 1781282245.74 total 30.02 last 68.99) - open/close 3008 (time 1781282255.74 total 40.02 last 76.90) - open/close 3713 (time 1781282265.74 total 50.02 last 70.47) - open/close 4386 (time 1781282275.75 total 60.04 last 67.21) - open/close 5130 (time 1781282285.76 total 70.04 last 74.36) - open/close 5845 (time 1781282295.77 total 80.06 last 71.40) - open/close 6458 (time 1781282305.78 total 90.06 last 61.26) - open/close 7123 (time 1781282315.79 total 100.07 last 66.44) - open/close 7811 (time 1781282325.79 total 110.07 last 68.79) - open/close 8522 (time 1781282335.80 total 120.09 last 71.00) - open/close 9222 (time 1781282345.81 total 130.09 last 69.96) - open/close 9912 (time 1781282355.82 total 140.10 last 68.95) total: 10000 open/close in 141.35 seconds: 70.75 ops/second - unlinked 0 (time 1781282361 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 710 (time 1781282453.62 total 10.01 last 70.93) - open/close 1382 (time 1781282463.64 total 20.03 last 67.07) - open/close 2141 (time 1781282473.64 total 30.03 last 75.88) - open/close 2889 (time 1781282483.65 total 40.04 last 74.77) - open/close 3638 (time 1781282493.66 total 50.05 last 74.83) - open/close 4366 (time 1781282503.66 total 60.05 last 72.76) - open/close 4960 (time 1781282513.69 total 70.07 last 59.26) - open/close 5616 (time 1781282523.70 total 80.08 last 65.54) - open/close 6361 (time 1781282533.71 total 90.09 last 74.41) - open/close 7072 (time 1781282543.71 total 100.10 last 71.05) - open/close 7812 (time 1781282553.72 total 110.11 last 73.96) - open/close 8458 (time 1781282563.73 total 120.12 last 64.52) - open/close 9262 (time 1781282573.74 total 130.13 last 80.35) total: 10000 open/close in 139.98 seconds: 71.44 ops/second - unlinked 0 (time 1781282587 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 775 (time 1781282684.34 total 10.01 last 77.44) - open/close 1484 (time 1781282694.35 total 20.02 last 70.82) - open/close 2220 (time 1781282704.35 total 30.02 last 73.59) - open/close 2958 (time 1781282714.36 total 40.03 last 73.70) - open/close 3619 (time 1781282724.36 total 50.04 last 66.09) - open/close 4324 (time 1781282734.37 total 60.04 last 70.46) - open/close 4986 (time 1781282744.37 total 70.04 last 66.19) - open/close 5631 (time 1781282754.38 total 80.05 last 64.48) - open/close 6352 (time 1781282764.38 total 90.05 last 72.08) - open/close 7011 (time 1781282774.38 total 100.05 last 65.87) - open/close 7714 (time 1781282784.39 total 110.06 last 70.25) - open/close 8418 (time 1781282794.39 total 120.06 last 70.39) - open/close 9175 (time 1781282804.41 total 130.08 last 75.55) - open/close 9879 (time 1781282814.42 total 140.09 last 70.36) total: 10000 open/close in 141.82 seconds: 70.51 ops/second - unlinked 0 (time 1781282820 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 757 (time 1781282917.01 total 10.01 last 75.61) - open/close 1459 (time 1781282927.03 total 20.03 last 70.08) - open/close 2172 (time 1781282937.04 total 30.04 last 71.22) - open/close 2908 (time 1781282947.04 total 40.04 last 73.58) - open/close 3631 (time 1781282957.07 total 50.07 last 72.13) - open/close 4361 (time 1781282967.08 total 60.07 last 72.95) - open/close 5130 (time 1781282977.08 total 70.08 last 76.86) - open/close 5845 (time 1781282987.08 total 80.08 last 71.49) - open/close 6515 (time 1781282997.09 total 90.08 last 66.98) - open/close 7290 (time 1781283007.10 total 100.10 last 77.36) - open/close 7980 (time 1781283017.11 total 110.11 last 68.97) - open/close 8657 (time 1781283027.12 total 120.12 last 67.64) - open/close 9380 (time 1781283037.14 total 130.14 last 72.14) total: 10000 open/close in 139.65 seconds: 71.61 ops/second - unlinked 0 (time 1781283050 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 726 (time 1781283156.80 total 10.00 last 72.57) - open/close 1376 (time 1781283166.82 total 20.03 last 64.83) - open/close 2062 (time 1781283176.83 total 30.04 last 68.52) - open/close 2763 (time 1781283186.83 total 40.04 last 70.09) - open/close 3517 (time 1781283196.84 total 50.04 last 75.39) - open/close 4155 (time 1781283206.84 total 60.04 last 63.80) - open/close 4877 (time 1781283216.85 total 70.06 last 72.07) - open/close 5538 (time 1781283226.87 total 80.08 last 65.99) - open/close 6168 (time 1781283236.87 total 90.08 last 62.99) - open/close 6848 (time 1781283246.88 total 100.09 last 67.93) - open/close 7559 (time 1781283256.90 total 110.11 last 70.98) - open/close 8253 (time 1781283266.92 total 120.12 last 69.28) - open/close 8685 (time 1781283276.92 total 130.13 last 43.18) - open/close 9338 (time 1781283286.92 total 140.13 last 65.30) total: 10000 open/close in 149.52 seconds: 66.88 ops/second - unlinked 0 (time 1781283301 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 585 (time 1781283405.99 total 10.00 last 58.47) - open/close 1326 (time 1781283415.99 total 20.01 last 74.09) - open/close 1986 (time 1781283426.00 total 30.01 last 65.97) - open/close 2611 (time 1781283436.01 total 40.02 last 62.42) - open/close 3334 (time 1781283446.02 total 50.03 last 72.23) - open/close 4013 (time 1781283456.03 total 60.04 last 67.87) - open/close 4649 (time 1781283466.03 total 70.04 last 63.59) - open/close 5297 (time 1781283476.03 total 80.04 last 64.78) - open/close 6009 (time 1781283486.04 total 90.05 last 71.12) - open/close 6683 (time 1781283496.04 total 100.06 last 67.39) - open/close 7402 (time 1781283506.05 total 110.06 last 71.89) - open/close 8110 (time 1781283516.05 total 120.06 last 70.79) - open/close 8770 (time 1781283526.05 total 130.06 last 65.99) - open/close 9470 (time 1781283536.05 total 140.06 last 69.98) total: 10000 open/close in 147.74 seconds: 67.69 ops/second - unlinked 0 (time 1781283548 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 607 (time 1781283668.05 total 10.01 last 60.67) - open/close 1273 (time 1781283678.06 total 20.02 last 66.53) - open/close 1996 (time 1781283688.07 total 30.03 last 72.20) - open/close 2635 (time 1781283698.08 total 40.04 last 63.87) - open/close 3335 (time 1781283708.09 total 50.05 last 69.90) - open/close 4003 (time 1781283718.11 total 60.06 last 66.72) - open/close 4666 (time 1781283728.13 total 70.08 last 66.16) - open/close 5247 (time 1781283738.13 total 80.09 last 58.07) - open/close 5863 (time 1781283748.15 total 90.11 last 61.50) - open/close 6480 (time 1781283758.16 total 100.12 last 61.62) - open/close 7088 (time 1781283768.17 total 110.12 last 60.78) - open/close 7688 (time 1781283778.17 total 120.13 last 59.96) - open/close 8183 (time 1781283788.19 total 130.15 last 49.41) - open/close 8587 (time 1781283798.21 total 140.17 last 40.32) - open/close 9060 (time 1781283808.22 total 150.18 last 47.25) - open/close 9714 (time 1781283818.23 total 160.18 last 65.36) total: 10000 open/close in 164.30 seconds: 60.86 ops/second - unlinked 0 (time 1781283826 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.106@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.106@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg306-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg306-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock oleg306-server@tcp:/lustre /mnt/lustre On OST0, 10519 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg306-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local oleg306-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg306-server: LNET ready to unload oleg306-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2