== conf-sanity test 69: replace an OST with the same index ========================================================== 09:23:38 (1788441818) start mds service on oleg348-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg348-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg348-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 709 (time 1788441908.57 total 10.01 last 70.81) - open/close 1496 (time 1788441918.58 total 20.02 last 78.61) - open/close 2262 (time 1788441928.60 total 30.04 last 76.47) - open/close 3078 (time 1788441938.60 total 40.05 last 81.56) - open/close 3840 (time 1788441948.62 total 50.06 last 76.09) - open/close 4579 (time 1788441958.63 total 60.08 last 73.78) - open/close 5413 (time 1788441968.64 total 70.08 last 83.36) - open/close 6181 (time 1788441978.66 total 80.10 last 76.68) - open/close 6934 (time 1788441988.66 total 90.11 last 75.23) - open/close 7721 (time 1788441998.67 total 100.11 last 78.67) - open/close 8435 (time 1788442008.69 total 110.13 last 71.23) - open/close 9213 (time 1788442018.70 total 120.14 last 77.75) - open/close 9945 (time 1788442028.72 total 130.16 last 73.03) total: 10000 open/close in 130.82 seconds: 76.44 ops/second - unlinked 0 (time 1788442033 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 788 (time 1788442136.96 total 10.00 last 78.77) - open/close 1542 (time 1788442146.97 total 20.01 last 75.32) - open/close 2317 (time 1788442156.97 total 30.02 last 77.49) - open/close 3104 (time 1788442166.98 total 40.02 last 78.67) - open/close 3824 (time 1788442176.98 total 50.02 last 71.98) - open/close 4519 (time 1788442187.00 total 60.04 last 69.39) - open/close 5275 (time 1788442197.00 total 70.04 last 75.56) - open/close 6026 (time 1788442207.01 total 80.06 last 75.01) - open/close 6762 (time 1788442217.02 total 90.06 last 73.55) - open/close 7498 (time 1788442227.03 total 100.07 last 73.53) - open/close 8287 (time 1788442237.04 total 110.08 last 78.82) - open/close 9012 (time 1788442247.05 total 120.10 last 72.39) - open/close 9725 (time 1788442257.08 total 130.12 last 71.13) total: 10000 open/close in 133.84 seconds: 74.71 ops/second - unlinked 0 (time 1788442265 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 773 (time 1788442366.52 total 10.02 last 77.18) - open/close 1484 (time 1788442376.53 total 20.02 last 71.05) - open/close 2226 (time 1788442386.53 total 30.02 last 74.20) - open/close 2995 (time 1788442396.53 total 40.03 last 76.86) - open/close 3724 (time 1788442406.54 total 50.04 last 72.82) - open/close 4491 (time 1788442416.55 total 60.05 last 76.65) - open/close 5204 (time 1788442426.55 total 70.05 last 71.28) - open/close 5973 (time 1788442436.57 total 80.06 last 76.78) - open/close 6734 (time 1788442446.61 total 90.10 last 75.80) - open/close 7472 (time 1788442456.63 total 100.12 last 73.64) - open/close 8175 (time 1788442466.64 total 110.14 last 70.22) - open/close 8881 (time 1788442476.64 total 120.14 last 70.57) - open/close 9629 (time 1788442486.65 total 130.15 last 74.72) total: 10000 open/close in 135.54 seconds: 73.78 ops/second - unlinked 0 (time 1788442495 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 726 (time 1788442597.92 total 10.01 last 72.55) - open/close 1387 (time 1788442607.94 total 20.03 last 65.97) - open/close 2035 (time 1788442617.95 total 30.03 last 64.77) - open/close 2740 (time 1788442627.95 total 40.04 last 70.46) - open/close 3478 (time 1788442637.96 total 50.04 last 73.78) - open/close 4266 (time 1788442647.97 total 60.05 last 78.74) - open/close 5067 (time 1788442657.97 total 70.06 last 80.04) - open/close 5818 (time 1788442667.98 total 80.06 last 75.04) - open/close 6489 (time 1788442677.98 total 90.06 last 67.09) - open/close 7104 (time 1788442688.00 total 100.09 last 61.36) - open/close 7831 (time 1788442698.01 total 110.10 last 72.63) - open/close 8558 (time 1788442708.02 total 120.10 last 72.68) - open/close 9294 (time 1788442718.03 total 130.11 last 73.50) total: 10000 open/close in 138.72 seconds: 72.09 ops/second - unlinked 0 (time 1788442730 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 800 (time 1788442824.18 total 10.00 last 79.98) - open/close 1528 (time 1788442834.19 total 20.01 last 72.76) - open/close 2257 (time 1788442844.19 total 30.01 last 72.89) - open/close 3018 (time 1788442854.20 total 40.02 last 76.05) - open/close 3788 (time 1788442864.20 total 50.02 last 76.93) - open/close 4420 (time 1788442874.20 total 60.02 last 63.20) - open/close 5096 (time 1788442884.22 total 70.04 last 67.48) - open/close 5874 (time 1788442894.22 total 80.04 last 77.79) - open/close 6649 (time 1788442904.24 total 90.06 last 77.41) - open/close 7261 (time 1788442914.24 total 100.06 last 61.15) - open/close 7880 (time 1788442924.25 total 110.07 last 61.85) - open/close 8598 (time 1788442934.27 total 120.09 last 71.66) - open/close 9330 (time 1788442944.28 total 130.10 last 73.14) total: 10000 open/close in 138.64 seconds: 72.13 ops/second - unlinked 0 (time 1788442957 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 716 (time 1788443059.69 total 10.01 last 71.56) - open/close 1493 (time 1788443069.70 total 20.02 last 77.60) - open/close 2243 (time 1788443079.72 total 30.03 last 74.89) - open/close 3010 (time 1788443089.73 total 40.04 last 76.63) - open/close 3725 (time 1788443099.73 total 50.04 last 71.49) - open/close 4420 (time 1788443109.74 total 60.05 last 69.44) - open/close 5140 (time 1788443119.74 total 70.05 last 71.99) - open/close 5900 (time 1788443129.75 total 80.06 last 75.94) - open/close 6588 (time 1788443139.75 total 90.07 last 68.75) - open/close 7368 (time 1788443149.76 total 100.08 last 77.93) - open/close 8157 (time 1788443159.77 total 110.08 last 78.87) - open/close 8792 (time 1788443169.78 total 120.10 last 63.42) - open/close 9485 (time 1788443179.78 total 130.10 last 69.29) total: 10000 open/close in 137.10 seconds: 72.94 ops/second - unlinked 0 (time 1788443191 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 746 (time 1788443287.72 total 10.00 last 74.58) - open/close 1538 (time 1788443297.73 total 20.01 last 79.11) - open/close 2268 (time 1788443307.73 total 30.02 last 72.99) - open/close 2986 (time 1788443317.74 total 40.02 last 71.77) - open/close 3788 (time 1788443327.75 total 50.03 last 80.12) - open/close 4557 (time 1788443337.76 total 60.04 last 76.79) - open/close 5339 (time 1788443347.76 total 70.05 last 78.18) - open/close 6095 (time 1788443357.76 total 80.05 last 75.60) - open/close 6894 (time 1788443367.77 total 90.05 last 79.88) - open/close 7740 (time 1788443377.77 total 100.05 last 84.56) - open/close 8554 (time 1788443387.78 total 110.06 last 81.32) - open/close 9277 (time 1788443397.79 total 120.08 last 72.21) total: 10000 open/close in 129.38 seconds: 77.29 ops/second - unlinked 0 (time 1788443411 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 813 (time 1788443504.43 total 10.00 last 81.26) - open/close 1564 (time 1788443514.45 total 20.02 last 75.02) - open/close 2366 (time 1788443524.45 total 30.02 last 80.14) - open/close 3142 (time 1788443534.48 total 40.05 last 77.42) - open/close 3703 (time 1788443544.48 total 50.05 last 56.06) - open/close 4324 (time 1788443554.49 total 60.06 last 62.06) - open/close 5052 (time 1788443564.50 total 70.07 last 72.70) - open/close 5839 (time 1788443574.51 total 80.08 last 78.68) - open/close 6460 (time 1788443584.51 total 90.08 last 62.05) - open/close 7192 (time 1788443594.52 total 100.09 last 73.17) - open/close 8064 (time 1788443604.52 total 110.09 last 87.17) - open/close 8836 (time 1788443614.53 total 120.10 last 77.16) - open/close 9498 (time 1788443624.54 total 130.11 last 66.10) total: 10000 open/close in 136.10 seconds: 73.47 ops/second - unlinked 0 (time 1788443635 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 721 (time 1788443746.20 total 10.01 last 72.01) - open/close 1453 (time 1788443756.20 total 20.01 last 73.19) - open/close 2087 (time 1788443766.21 total 30.02 last 63.34) - open/close 2691 (time 1788443776.21 total 40.03 last 60.37) - open/close 3429 (time 1788443786.22 total 50.03 last 73.77) - open/close 4261 (time 1788443796.23 total 60.05 last 83.07) - open/close 5064 (time 1788443806.24 total 70.05 last 80.26) - open/close 5831 (time 1788443816.25 total 80.06 last 76.61) - open/close 6580 (time 1788443826.26 total 90.07 last 74.85) - open/close 7281 (time 1788443836.27 total 100.09 last 69.97) - open/close 8003 (time 1788443846.28 total 110.09 last 72.16) - open/close 8771 (time 1788443856.28 total 120.10 last 76.79) - open/close 9560 (time 1788443866.29 total 130.11 last 78.81) total: 10000 open/close in 136.38 seconds: 73.32 ops/second - unlinked 0 (time 1788443876 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 685 (time 1788443978.14 total 10.01 last 68.41) - open/close 1314 (time 1788443988.14 total 20.02 last 62.88) - open/close 2069 (time 1788443998.15 total 30.02 last 75.45) - open/close 2774 (time 1788444008.16 total 40.04 last 70.42) - open/close 3346 (time 1788444018.19 total 50.06 last 57.07) - open/close 3759 (time 1788444028.19 total 60.06 last 41.28) - open/close 4211 (time 1788444038.20 total 70.08 last 45.14) - open/close 4802 (time 1788444048.21 total 80.08 last 59.05) - open/close 5541 (time 1788444058.22 total 90.09 last 73.84) - open/close 6283 (time 1788444068.22 total 100.10 last 74.17) - open/close 6976 (time 1788444078.24 total 110.11 last 69.21) - open/close 7657 (time 1788444088.27 total 120.14 last 67.91) - open/close 8393 (time 1788444098.27 total 130.14 last 73.58) - open/close 8917 (time 1788444108.28 total 140.16 last 52.32) - open/close 9518 (time 1788444118.29 total 150.16 last 60.06) total: 10000 open/close in 158.29 seconds: 63.17 ops/second - unlinked 0 (time 1788444132 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.148@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.148@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg348-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre On OST0, 10478 used inodes rc=0 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.148@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.148@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.148@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.148@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.148@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg348-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server unloading modules via unload_modules_local on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local oleg348-server: modules unloaded. pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2