== conf-sanity test 69: replace an OST with the same index ========================================================== 04:47:22 (1787734042) start mds service on oleg239-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions oleg239-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg239-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg239-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg239-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 793 (time 1787734145.94 total 10.01 last 79.25) - open/close 1627 (time 1787734155.95 total 20.01 last 83.34) - open/close 2443 (time 1787734165.95 total 30.02 last 81.57) - open/close 3207 (time 1787734175.96 total 40.02 last 76.37) - open/close 4073 (time 1787734185.96 total 50.02 last 86.58) - open/close 4930 (time 1787734195.97 total 60.03 last 85.62) - open/close 5601 (time 1787734205.97 total 70.03 last 67.09) - open/close 6363 (time 1787734215.98 total 80.04 last 76.14) - open/close 7169 (time 1787734225.98 total 90.04 last 80.57) - open/close 8116 (time 1787734235.99 total 100.06 last 94.59) - open/close 8779 (time 1787734246.01 total 110.07 last 66.21) - open/close 9472 (time 1787734256.01 total 120.08 last 69.25) total: 10000 open/close in 127.84 seconds: 78.23 ops/second - unlinked 0 (time 1787734269 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 785 (time 1787734363.09 total 10.01 last 78.40) - open/close 1566 (time 1787734373.11 total 20.03 last 77.98) - open/close 2378 (time 1787734383.12 total 30.04 last 81.08) - open/close 3244 (time 1787734393.14 total 40.06 last 86.48) - open/close 3900 (time 1787734403.14 total 50.07 last 65.55) - open/close 4677 (time 1787734413.15 total 60.07 last 77.69) - open/close 5417 (time 1787734423.15 total 70.07 last 73.97) - open/close 6209 (time 1787734433.16 total 80.08 last 79.11) - open/close 7037 (time 1787734443.17 total 90.09 last 82.73) - open/close 8009 (time 1787734453.17 total 100.09 last 97.19) - open/close 8754 (time 1787734463.17 total 110.09 last 74.47) - open/close 9566 (time 1787734473.17 total 120.09 last 81.20) total: 10000 open/close in 125.41 seconds: 79.74 ops/second - unlinked 0 (time 1787734482 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 816 (time 1787734582.90 total 10.01 last 81.56) - open/close 1558 (time 1787734592.90 total 20.01 last 74.20) - open/close 2300 (time 1787734602.91 total 30.01 last 74.13) - open/close 2992 (time 1787734612.91 total 40.02 last 69.17) - open/close 3786 (time 1787734622.92 total 50.03 last 79.36) - open/close 4495 (time 1787734632.93 total 60.04 last 70.82) - open/close 5260 (time 1787734642.94 total 70.05 last 76.43) - open/close 6032 (time 1787734652.94 total 80.05 last 77.16) - open/close 6787 (time 1787734662.95 total 90.06 last 75.44) - open/close 7610 (time 1787734672.96 total 100.06 last 82.28) - open/close 8408 (time 1787734682.96 total 110.06 last 79.79) - open/close 9274 (time 1787734692.96 total 120.07 last 86.54) - open/close 9960 (time 1787734702.98 total 130.09 last 68.49) total: 10000 open/close in 130.66 seconds: 76.53 ops/second - unlinked 0 (time 1787734707 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 764 (time 1787734802.85 total 10.01 last 76.35) - open/close 1604 (time 1787734812.86 total 20.01 last 83.96) - open/close 2362 (time 1787734822.87 total 30.02 last 75.73) - open/close 3086 (time 1787734832.88 total 40.04 last 72.29) - open/close 3879 (time 1787734842.90 total 50.05 last 79.17) - open/close 4678 (time 1787734852.91 total 60.06 last 79.81) - open/close 5458 (time 1787734862.92 total 70.07 last 77.96) - open/close 6292 (time 1787734872.92 total 80.07 last 83.39) - open/close 7101 (time 1787734882.92 total 90.07 last 80.89) - open/close 7858 (time 1787734892.95 total 100.10 last 75.45) - open/close 8710 (time 1787734902.96 total 110.11 last 85.13) - open/close 9504 (time 1787734912.96 total 120.11 last 79.39) total: 10000 open/close in 125.86 seconds: 79.46 ops/second - unlinked 0 (time 1787734922 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 705 (time 1787735017.60 total 10.00 last 70.48) - open/close 1573 (time 1787735027.60 total 20.00 last 86.80) - open/close 2235 (time 1787735037.61 total 30.01 last 66.19) - open/close 3025 (time 1787735047.61 total 40.01 last 78.99) - open/close 3802 (time 1787735057.61 total 50.01 last 77.70) - open/close 4579 (time 1787735067.62 total 60.02 last 77.61) - open/close 5368 (time 1787735077.62 total 70.02 last 78.88) - open/close 6157 (time 1787735087.63 total 80.03 last 78.85) - open/close 6997 (time 1787735097.63 total 90.03 last 83.95) - open/close 7774 (time 1787735107.63 total 100.03 last 77.69) - open/close 8524 (time 1787735117.64 total 110.04 last 74.98) - open/close 9267 (time 1787735127.66 total 120.06 last 74.13) - open/close 9996 (time 1787735137.67 total 130.06 last 72.86) total: 10000 open/close in 130.09 seconds: 76.87 ops/second - unlinked 0 (time 1787735141 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 748 (time 1787735234.57 total 10.00 last 74.78) - open/close 1594 (time 1787735244.58 total 20.01 last 84.53) - open/close 2462 (time 1787735254.59 total 30.02 last 86.76) - open/close 3257 (time 1787735264.60 total 40.03 last 79.40) - open/close 3975 (time 1787735274.60 total 50.03 last 71.79) - open/close 4730 (time 1787735284.61 total 60.04 last 75.45) - open/close 5479 (time 1787735294.61 total 70.04 last 74.88) - open/close 6229 (time 1787735304.62 total 80.05 last 74.93) - open/close 7054 (time 1787735314.62 total 90.05 last 82.46) - open/close 7806 (time 1787735324.63 total 100.06 last 75.16) - open/close 8597 (time 1787735334.63 total 110.06 last 79.05) - open/close 9396 (time 1787735344.64 total 120.07 last 79.84) total: 10000 open/close in 127.39 seconds: 78.50 ops/second - unlinked 0 (time 1787735355 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 795 (time 1787735447.44 total 10.01 last 79.44) - open/close 1491 (time 1787735457.44 total 20.01 last 69.56) - open/close 2213 (time 1787735467.45 total 30.03 last 72.11) - open/close 2901 (time 1787735477.46 total 40.03 last 68.78) - open/close 3773 (time 1787735487.46 total 50.03 last 87.15) - open/close 4583 (time 1787735497.46 total 60.03 last 81.00) - open/close 5279 (time 1787735507.47 total 70.04 last 69.55) - open/close 6120 (time 1787735517.47 total 80.04 last 84.09) - open/close 6943 (time 1787735527.48 total 90.05 last 82.26) - open/close 7820 (time 1787735537.48 total 100.05 last 87.69) - open/close 8681 (time 1787735547.49 total 110.06 last 86.01) - open/close 9502 (time 1787735557.49 total 120.06 last 82.04) total: 10000 open/close in 126.64 seconds: 78.96 ops/second - unlinked 0 (time 1787735567 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 819 (time 1787735651.44 total 10.00 last 81.88) - open/close 1744 (time 1787735661.44 total 20.01 last 92.46) - open/close 2620 (time 1787735671.45 total 30.01 last 87.56) - open/close 3493 (time 1787735681.45 total 40.01 last 87.30) - open/close 4368 (time 1787735691.47 total 50.04 last 87.26) - open/close 5258 (time 1787735701.48 total 60.04 last 88.98) - open/close 6131 (time 1787735711.49 total 70.06 last 87.18) - open/close 6973 (time 1787735721.49 total 80.06 last 84.15) - open/close 7744 (time 1787735731.50 total 90.06 last 77.07) - open/close 8476 (time 1787735741.51 total 100.08 last 73.11) - open/close 9265 (time 1787735751.51 total 110.08 last 78.90) total: 10000 open/close in 119.48 seconds: 83.70 ops/second - unlinked 0 (time 1787735764 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 911 (time 1787735853.81 total 10.02 last 90.94) - open/close 1712 (time 1787735863.82 total 20.03 last 79.99) - open/close 2505 (time 1787735873.83 total 30.04 last 79.22) - open/close 3356 (time 1787735883.84 total 40.05 last 85.02) - open/close 4155 (time 1787735893.84 total 50.05 last 79.90) - open/close 4966 (time 1787735903.85 total 60.06 last 81.02) - open/close 5718 (time 1787735913.86 total 70.06 last 75.18) - open/close 6516 (time 1787735923.86 total 80.06 last 79.79) - open/close 7333 (time 1787735933.86 total 90.06 last 81.70) - open/close 8102 (time 1787735943.87 total 100.08 last 76.79) - open/close 8923 (time 1787735953.88 total 110.09 last 82.03) - open/close 9724 (time 1787735963.89 total 120.10 last 80.03) total: 10000 open/close in 123.82 seconds: 80.76 ops/second - unlinked 0 (time 1787735971 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 876 (time 1787736058.55 total 10.01 last 87.52) - open/close 1737 (time 1787736068.55 total 20.02 last 86.04) - open/close 2584 (time 1787736078.56 total 30.03 last 84.60) - open/close 3338 (time 1787736088.57 total 40.03 last 75.36) - open/close 4139 (time 1787736098.58 total 50.04 last 80.06) - open/close 4894 (time 1787736108.58 total 60.05 last 75.44) - open/close 5765 (time 1787736118.59 total 70.05 last 87.04) - open/close 6578 (time 1787736128.60 total 80.06 last 81.25) - open/close 7343 (time 1787736138.62 total 90.08 last 76.34) - open/close 8165 (time 1787736148.62 total 100.09 last 82.16) - open/close 8951 (time 1787736158.62 total 110.09 last 78.60) - open/close 9809 (time 1787736168.62 total 120.09 last 85.79) total: 10000 open/close in 122.18 seconds: 81.84 ops/second - unlinked 0 (time 1787736175 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg239-server: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg239-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg239-server: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg239-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre On OST0, 20520 used inodes rc=0 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.139@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.139@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.139@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg239-server: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg239-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg239-server: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg239-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg239-server unloading modules via unload_modules_local on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local oleg239-server: modules unloaded. pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2