== conf-sanity test 69: replace an OST with the same index ========================================================== 17:21:54 (1776201714) start mds service on oleg203-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg203-server: detected 4 online CPUs by sysfs oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 710 (time 1776201793.33 total 10.01 last 70.95) - open/close 1416 (time 1776201803.33 total 20.01 last 70.60) - open/close 2146 (time 1776201813.34 total 30.02 last 72.91) - open/close 2870 (time 1776201823.36 total 40.04 last 72.29) - open/close 3473 (time 1776201833.36 total 50.04 last 60.27) - open/close 4046 (time 1776201843.36 total 60.04 last 57.29) - open/close 4706 (time 1776201853.37 total 70.05 last 65.98) - open/close 5406 (time 1776201863.37 total 80.05 last 69.98) - open/close 6110 (time 1776201873.38 total 90.06 last 70.34) - open/close 6768 (time 1776201883.38 total 100.06 last 65.79) - open/close 7326 (time 1776201893.39 total 110.06 last 55.76) - open/close 7976 (time 1776201903.41 total 120.09 last 64.85) - open/close 8671 (time 1776201913.41 total 130.09 last 69.48) - open/close 9300 (time 1776201923.42 total 140.10 last 62.88) - open/close 9996 (time 1776201933.44 total 150.12 last 69.41) total: 10000 open/close in 150.21 seconds: 66.57 ops/second - unlinked 0 (time 1776201938 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 695 (time 1776202039.57 total 10.00 last 69.49) - open/close 1435 (time 1776202049.57 total 20.01 last 73.95) - open/close 2070 (time 1776202059.58 total 30.01 last 63.46) - open/close 2759 (time 1776202069.58 total 40.02 last 68.89) - open/close 3489 (time 1776202079.59 total 50.03 last 72.90) - open/close 4194 (time 1776202089.62 total 60.06 last 70.30) - open/close 4950 (time 1776202099.63 total 70.07 last 75.53) - open/close 5714 (time 1776202109.63 total 80.07 last 76.38) - open/close 6368 (time 1776202119.63 total 90.07 last 65.39) - open/close 7036 (time 1776202129.65 total 100.08 last 66.73) - open/close 7636 (time 1776202139.65 total 110.08 last 60.00) - open/close 8110 (time 1776202149.65 total 120.09 last 47.38) - open/close 8860 (time 1776202159.66 total 130.10 last 74.92) - open/close 9574 (time 1776202169.68 total 140.11 last 71.30) total: 10000 open/close in 145.63 seconds: 68.67 ops/second - unlinked 0 (time 1776202179 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 740 (time 1776202275.91 total 10.01 last 73.95) - open/close 1484 (time 1776202285.93 total 20.02 last 74.28) - open/close 2195 (time 1776202295.93 total 30.03 last 71.08) - open/close 2973 (time 1776202305.93 total 40.03 last 77.78) - open/close 3725 (time 1776202315.94 total 50.04 last 75.15) - open/close 4502 (time 1776202325.95 total 60.04 last 77.66) - open/close 5254 (time 1776202335.96 total 70.06 last 75.08) - open/close 6022 (time 1776202345.97 total 80.06 last 76.74) - open/close 6780 (time 1776202355.98 total 90.08 last 75.70) - open/close 7616 (time 1776202366.00 total 100.09 last 83.48) - open/close 8436 (time 1776202376.00 total 110.09 last 82.00) - open/close 9081 (time 1776202386.00 total 120.10 last 64.46) - open/close 9774 (time 1776202396.03 total 130.12 last 69.11) total: 10000 open/close in 133.74 seconds: 74.77 ops/second - unlinked 0 (time 1776202404 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 785 (time 1776202509.16 total 10.00 last 78.48) - open/close 1494 (time 1776202519.17 total 20.01 last 70.86) - open/close 2234 (time 1776202529.17 total 30.01 last 73.96) - open/close 2967 (time 1776202539.19 total 40.03 last 73.18) - open/close 3768 (time 1776202549.19 total 50.03 last 80.06) - open/close 4470 (time 1776202559.20 total 60.04 last 70.14) - open/close 5180 (time 1776202569.20 total 70.04 last 70.99) - open/close 5902 (time 1776202579.20 total 80.05 last 72.20) - open/close 6546 (time 1776202589.22 total 90.06 last 64.31) - open/close 7191 (time 1776202599.22 total 100.06 last 64.46) - open/close 7922 (time 1776202609.23 total 110.07 last 73.07) - open/close 8530 (time 1776202619.23 total 120.07 last 60.77) - open/close 9136 (time 1776202629.24 total 130.08 last 60.57) - open/close 9836 (time 1776202639.25 total 140.09 last 69.93) total: 10000 open/close in 142.12 seconds: 70.36 ops/second - unlinked 0 (time 1776202645 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 605 (time 1776202742.06 total 10.00 last 60.49) - open/close 1288 (time 1776202752.07 total 20.01 last 68.23) - open/close 2143 (time 1776202762.07 total 30.02 last 85.47) - open/close 2851 (time 1776202772.07 total 40.02 last 70.80) - open/close 3563 (time 1776202782.09 total 50.03 last 71.09) - open/close 4219 (time 1776202792.10 total 60.04 last 65.53) - open/close 4764 (time 1776202802.11 total 70.05 last 54.44) - open/close 5528 (time 1776202812.11 total 80.05 last 76.39) - open/close 6117 (time 1776202822.11 total 90.06 last 58.88) - open/close 6805 (time 1776202832.12 total 100.07 last 68.72) - open/close 7493 (time 1776202842.13 total 110.07 last 68.77) - open/close 8187 (time 1776202852.14 total 120.08 last 69.32) - open/close 8794 (time 1776202862.16 total 130.10 last 60.57) - open/close 9407 (time 1776202872.19 total 140.13 last 61.15) total: 10000 open/close in 148.45 seconds: 67.36 ops/second - unlinked 0 (time 1776202884 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 712 (time 1776202987.46 total 10.01 last 71.12) - open/close 1477 (time 1776202997.47 total 20.02 last 76.45) - open/close 2244 (time 1776203007.47 total 30.02 last 76.65) - open/close 3022 (time 1776203017.48 total 40.03 last 77.79) - open/close 3726 (time 1776203027.48 total 50.03 last 70.34) - open/close 4366 (time 1776203037.49 total 60.04 last 63.99) - open/close 4977 (time 1776203047.50 total 70.05 last 61.01) - open/close 5398 (time 1776203057.51 total 80.06 last 42.04) - open/close 6038 (time 1776203067.52 total 90.07 last 63.99) - open/close 6758 (time 1776203077.53 total 100.08 last 71.91) - open/close 7328 (time 1776203087.54 total 110.09 last 56.93) - open/close 7968 (time 1776203097.56 total 120.11 last 63.88) - open/close 8527 (time 1776203107.57 total 130.12 last 55.85) - open/close 9191 (time 1776203117.57 total 140.12 last 66.38) - open/close 9897 (time 1776203127.58 total 150.13 last 70.53) total: 10000 open/close in 151.42 seconds: 66.04 ops/second - unlinked 0 (time 1776203132 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 884 (time 1776203227.28 total 10.00 last 88.39) - open/close 1697 (time 1776203237.28 total 20.00 last 81.28) - open/close 2385 (time 1776203247.29 total 30.01 last 68.73) - open/close 3133 (time 1776203257.31 total 40.03 last 74.67) - open/close 3799 (time 1776203267.31 total 50.03 last 66.59) - open/close 4469 (time 1776203277.33 total 60.05 last 66.88) - open/close 5132 (time 1776203287.34 total 70.06 last 66.25) - open/close 5673 (time 1776203297.36 total 80.08 last 54.00) - open/close 6289 (time 1776203307.36 total 90.08 last 61.57) - open/close 7053 (time 1776203317.37 total 100.09 last 76.36) - open/close 7877 (time 1776203327.38 total 110.10 last 82.33) - open/close 8692 (time 1776203337.38 total 120.11 last 81.43) - open/close 9448 (time 1776203347.39 total 130.11 last 75.57) total: 10000 open/close in 138.32 seconds: 72.30 ops/second - unlinked 0 (time 1776203360 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 637 (time 1776203458.39 total 10.00 last 63.70) - open/close 1227 (time 1776203468.40 total 20.01 last 58.96) - open/close 1851 (time 1776203478.42 total 30.03 last 62.27) - open/close 2516 (time 1776203488.44 total 40.04 last 66.40) - open/close 3181 (time 1776203498.44 total 50.05 last 66.46) - open/close 3905 (time 1776203508.44 total 60.05 last 72.39) - open/close 4679 (time 1776203518.45 total 70.06 last 77.35) - open/close 5475 (time 1776203528.46 total 80.07 last 79.52) - open/close 6331 (time 1776203538.46 total 90.07 last 85.59) - open/close 7095 (time 1776203548.47 total 100.08 last 76.31) - open/close 7894 (time 1776203558.48 total 110.09 last 79.83) - open/close 8610 (time 1776203568.48 total 120.09 last 71.59) - open/close 9351 (time 1776203578.49 total 130.10 last 74.06) total: 10000 open/close in 138.25 seconds: 72.33 ops/second - unlinked 0 (time 1776203590 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 595 (time 1776203691.30 total 10.01 last 59.45) - open/close 1321 (time 1776203701.30 total 20.01 last 72.60) - open/close 2027 (time 1776203711.30 total 30.01 last 70.59) - open/close 2745 (time 1776203721.31 total 40.02 last 71.76) - open/close 3439 (time 1776203731.32 total 50.02 last 69.33) - open/close 4199 (time 1776203741.32 total 60.03 last 75.96) - open/close 4986 (time 1776203751.33 total 70.04 last 78.65) - open/close 5756 (time 1776203761.33 total 80.04 last 76.99) - open/close 6462 (time 1776203771.33 total 90.04 last 70.58) - open/close 7200 (time 1776203781.34 total 100.05 last 73.76) - open/close 7952 (time 1776203791.36 total 110.07 last 75.02) - open/close 8634 (time 1776203801.38 total 120.09 last 68.07) - open/close 9322 (time 1776203811.38 total 130.09 last 68.78) total: 10000 open/close in 139.66 seconds: 71.60 ops/second - unlinked 0 (time 1776203825 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 740 (time 1776203922.07 total 10.01 last 73.95) - open/close 1478 (time 1776203932.09 total 20.02 last 73.69) - open/close 2232 (time 1776203942.09 total 30.02 last 75.40) - open/close 2925 (time 1776203952.09 total 40.03 last 69.24) - open/close 3651 (time 1776203962.10 total 50.03 last 72.59) - open/close 4393 (time 1776203972.10 total 60.04 last 74.16) - open/close 5026 (time 1776203982.11 total 70.04 last 63.26) - open/close 5595 (time 1776203992.12 total 80.06 last 56.82) - open/close 6269 (time 1776204002.12 total 90.06 last 67.39) - open/close 6910 (time 1776204012.13 total 100.07 last 64.02) - open/close 7537 (time 1776204022.14 total 110.08 last 62.64) - open/close 8234 (time 1776204032.16 total 120.09 last 69.61) - open/close 9000 (time 1776204042.16 total 130.09 last 76.60) - open/close 9663 (time 1776204052.16 total 140.09 last 66.30) total: 10000 open/close in 144.76 seconds: 69.08 ops/second - unlinked 0 (time 1776204060 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.103@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local oleg203-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg203-server: LNET ready to unload oleg203-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2