== conf-sanity test 69: replace an OST with the same index ========================================================== 09:11:45 (1788441105) start mds service on oleg302-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg302-server: detected 4 online CPUs by sysfs oleg302-server: MODOPTS_LIBCFS= oleg302-server: Force libcfs to create 2 CPU partitions oleg302-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg302-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg302-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 670 (time 1788441195.27 total 10.00 last 66.97) - open/close 1277 (time 1788441205.28 total 20.01 last 60.67) - open/close 2091 (time 1788441215.28 total 30.01 last 81.40) - open/close 2841 (time 1788441225.28 total 40.01 last 74.97) - open/close 3586 (time 1788441235.29 total 50.02 last 74.44) - open/close 4348 (time 1788441245.30 total 60.03 last 76.12) - open/close 4997 (time 1788441255.32 total 70.05 last 64.77) - open/close 5653 (time 1788441265.33 total 80.06 last 65.52) - open/close 6417 (time 1788441275.33 total 90.07 last 76.37) - open/close 7196 (time 1788441285.36 total 100.10 last 77.68) - open/close 8005 (time 1788441295.37 total 110.10 last 80.84) - open/close 8785 (time 1788441305.38 total 120.12 last 77.91) - open/close 9561 (time 1788441315.38 total 130.12 last 77.59) total: 10000 open/close in 136.20 seconds: 73.42 ops/second - unlinked 0 (time 1788441325 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 639 (time 1788441421.87 total 10.00 last 63.88) - open/close 1360 (time 1788441431.87 total 20.01 last 72.07) - open/close 2186 (time 1788441441.90 total 30.03 last 82.41) - open/close 2926 (time 1788441451.91 total 40.04 last 73.93) - open/close 3631 (time 1788441461.94 total 50.08 last 70.25) - open/close 4280 (time 1788441471.95 total 60.09 last 64.83) - open/close 5081 (time 1788441481.96 total 70.09 last 80.07) - open/close 5795 (time 1788441491.97 total 80.11 last 71.28) - open/close 6431 (time 1788441501.99 total 90.13 last 63.47) - open/close 7192 (time 1788441512.00 total 100.13 last 76.06) - open/close 7988 (time 1788441522.01 total 110.15 last 79.47) - open/close 8803 (time 1788441532.02 total 120.16 last 81.43) - open/close 9512 (time 1788441542.03 total 130.16 last 70.87) total: 10000 open/close in 136.53 seconds: 73.25 ops/second - unlinked 0 (time 1788441551 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 490 (time 1788441645.90 total 10.01 last 48.95) - open/close 1026 (time 1788441655.91 total 20.02 last 53.56) - open/close 1602 (time 1788441665.92 total 30.03 last 57.51) - open/close 2276 (time 1788441675.92 total 40.04 last 67.39) - open/close 3045 (time 1788441685.92 total 50.04 last 76.89) - open/close 3771 (time 1788441695.93 total 60.04 last 72.57) - open/close 4526 (time 1788441705.94 total 70.05 last 75.43) - open/close 5274 (time 1788441715.94 total 80.06 last 74.74) - open/close 5951 (time 1788441725.96 total 90.07 last 67.58) - open/close 6656 (time 1788441735.97 total 100.08 last 70.46) - open/close 7369 (time 1788441745.97 total 110.08 last 71.29) - open/close 8104 (time 1788441755.98 total 120.09 last 73.43) - open/close 8907 (time 1788441765.99 total 130.10 last 80.23) - open/close 9634 (time 1788441775.99 total 140.11 last 72.66) total: 10000 open/close in 144.78 seconds: 69.07 ops/second - unlinked 0 (time 1788441784 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 777 (time 1788441889.42 total 10.01 last 77.65) - open/close 1597 (time 1788441899.44 total 20.02 last 81.87) - open/close 2433 (time 1788441909.44 total 30.03 last 83.57) - open/close 3196 (time 1788441919.46 total 40.04 last 76.18) - open/close 4022 (time 1788441929.46 total 50.04 last 82.58) - open/close 4797 (time 1788441939.47 total 60.05 last 77.46) - open/close 5544 (time 1788441949.49 total 70.07 last 74.56) - open/close 6285 (time 1788441959.50 total 80.08 last 74.00) - open/close 6955 (time 1788441969.50 total 90.08 last 66.99) - open/close 7727 (time 1788441979.50 total 100.09 last 77.18) - open/close 8499 (time 1788441989.51 total 110.10 last 77.13) - open/close 9262 (time 1788441999.52 total 120.10 last 76.24) total: 10000 open/close in 129.62 seconds: 77.15 ops/second - unlinked 0 (time 1788442013 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 798 (time 1788442113.48 total 10.00 last 79.80) - open/close 1568 (time 1788442123.48 total 20.00 last 76.98) - open/close 2347 (time 1788442133.48 total 30.01 last 77.87) - open/close 3173 (time 1788442143.49 total 40.01 last 82.56) - open/close 3819 (time 1788442153.52 total 50.05 last 64.38) - open/close 4546 (time 1788442163.53 total 60.06 last 72.62) - open/close 5344 (time 1788442173.54 total 70.06 last 79.75) - open/close 6100 (time 1788442183.54 total 80.07 last 75.56) - open/close 6884 (time 1788442193.55 total 90.07 last 78.39) - open/close 7664 (time 1788442203.55 total 100.08 last 77.94) - open/close 8456 (time 1788442213.56 total 110.09 last 79.13) - open/close 9209 (time 1788442223.57 total 120.09 last 75.24) - open/close 9953 (time 1788442233.58 total 130.10 last 74.33) total: 10000 open/close in 130.86 seconds: 76.42 ops/second - unlinked 0 (time 1788442238 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 561 (time 1788442336.43 total 10.00 last 56.08) - open/close 1259 (time 1788442346.44 total 20.01 last 69.77) - open/close 2011 (time 1788442356.45 total 30.02 last 75.10) - open/close 2791 (time 1788442366.46 total 40.02 last 77.96) - open/close 3571 (time 1788442376.49 total 50.06 last 77.72) - open/close 4310 (time 1788442386.50 total 60.07 last 73.86) - open/close 5128 (time 1788442396.51 total 70.08 last 81.71) - open/close 5938 (time 1788442406.52 total 80.08 last 80.95) - open/close 6708 (time 1788442416.52 total 90.09 last 76.98) - open/close 7283 (time 1788442426.54 total 100.11 last 57.39) - open/close 7864 (time 1788442436.54 total 110.11 last 58.10) - open/close 8591 (time 1788442446.54 total 120.11 last 72.67) - open/close 9395 (time 1788442456.55 total 130.12 last 80.36) total: 10000 open/close in 138.09 seconds: 72.42 ops/second - unlinked 0 (time 1788442469 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 786 (time 1788442575.80 total 10.00 last 78.59) - open/close 1608 (time 1788442585.81 total 20.01 last 82.12) - open/close 2398 (time 1788442595.82 total 30.02 last 78.95) - open/close 3087 (time 1788442605.83 total 40.03 last 68.83) - open/close 3680 (time 1788442615.84 total 50.03 last 59.26) - open/close 4328 (time 1788442625.85 total 60.05 last 64.70) - open/close 5080 (time 1788442635.85 total 70.05 last 75.18) - open/close 5880 (time 1788442645.86 total 80.06 last 79.93) - open/close 6585 (time 1788442655.88 total 90.07 last 70.41) - open/close 7389 (time 1788442665.89 total 100.08 last 80.31) - open/close 8128 (time 1788442675.89 total 110.09 last 73.88) - open/close 8962 (time 1788442685.90 total 120.09 last 83.33) - open/close 9628 (time 1788442695.90 total 130.10 last 66.55) total: 10000 open/close in 135.02 seconds: 74.06 ops/second - unlinked 0 (time 1788442704 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 765 (time 1788442801.88 total 10.01 last 76.44) - open/close 1592 (time 1788442811.89 total 20.02 last 82.62) - open/close 2368 (time 1788442821.89 total 30.02 last 77.59) - open/close 3163 (time 1788442831.89 total 40.02 last 79.50) - open/close 3960 (time 1788442841.90 total 50.03 last 79.64) - open/close 4773 (time 1788442851.90 total 60.03 last 81.27) - open/close 5443 (time 1788442861.90 total 70.03 last 66.98) - open/close 6031 (time 1788442871.93 total 80.06 last 58.63) - open/close 6740 (time 1788442881.94 total 90.07 last 70.88) - open/close 7445 (time 1788442891.95 total 100.08 last 70.43) - open/close 8225 (time 1788442901.95 total 110.08 last 77.95) - open/close 8908 (time 1788442911.97 total 120.10 last 68.18) - open/close 9579 (time 1788442921.97 total 130.10 last 67.09) total: 10000 open/close in 137.12 seconds: 72.93 ops/second - unlinked 0 (time 1788442932 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 745 (time 1788443029.21 total 10.00 last 74.49) - open/close 1389 (time 1788443039.22 total 20.01 last 64.32) - open/close 1992 (time 1788443049.24 total 30.04 last 60.17) - open/close 2733 (time 1788443059.25 total 40.04 last 74.07) - open/close 3473 (time 1788443069.29 total 50.08 last 73.68) - open/close 4270 (time 1788443079.29 total 60.09 last 79.67) - open/close 5037 (time 1788443089.30 total 70.09 last 76.67) - open/close 5816 (time 1788443099.31 total 80.11 last 77.77) - open/close 6613 (time 1788443109.32 total 90.12 last 79.64) - open/close 7432 (time 1788443119.32 total 100.12 last 81.88) - open/close 8198 (time 1788443129.33 total 110.12 last 76.58) - open/close 9001 (time 1788443139.34 total 120.13 last 80.23) - open/close 9713 (time 1788443149.34 total 130.14 last 71.15) total: 10000 open/close in 134.06 seconds: 74.59 ops/second - unlinked 0 (time 1788443158 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 762 (time 1788443257.78 total 10.00 last 76.20) - open/close 1533 (time 1788443267.78 total 20.01 last 77.05) - open/close 2221 (time 1788443277.79 total 30.01 last 68.76) - open/close 2936 (time 1788443287.80 total 40.02 last 71.44) - open/close 3721 (time 1788443297.80 total 50.03 last 78.46) - open/close 4454 (time 1788443307.81 total 60.03 last 73.26) - open/close 5221 (time 1788443317.81 total 70.03 last 76.70) - open/close 6002 (time 1788443327.81 total 80.04 last 78.08) - open/close 6816 (time 1788443337.82 total 90.04 last 81.35) - open/close 7564 (time 1788443347.83 total 100.06 last 74.68) - open/close 8350 (time 1788443357.84 total 110.06 last 78.57) - open/close 9088 (time 1788443367.84 total 120.06 last 73.77) - open/close 9814 (time 1788443377.85 total 130.07 last 72.55) total: 10000 open/close in 132.38 seconds: 75.54 ops/second - unlinked 0 (time 1788443383 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.102@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.102@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg302-server: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg302-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.102@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.102@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.102@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.102@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.102@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg302-server: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg302-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server unloading modules via unload_modules_local on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: modules unloaded. pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2