== conf-sanity test 69: replace an OST with the same index ========================================================== 16:30:17 (1787862617) start mds service on oleg620-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg620-server' oleg620-server: oleg620-server.virtnet: executing load_modules_local oleg620-server: Loading modules from /home/green/git/lustre-release/lustre oleg620-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg620-server: detected 4 online CPUs by sysfs oleg620-server: MODOPTS_LIBCFS= oleg620-server: Force libcfs to create 2 CPU partitions oleg620-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg620-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg620-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg620-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0000 oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 639 (time 1787862707.73 total 10.00 last 63.89) - open/close 1323 (time 1787862717.73 total 20.00 last 68.39) - open/close 2062 (time 1787862727.73 total 30.00 last 73.90) - open/close 2823 (time 1787862737.74 total 40.01 last 76.08) - open/close 3568 (time 1787862747.75 total 50.01 last 74.44) - open/close 4167 (time 1787862757.76 total 60.03 last 59.82) - open/close 4885 (time 1787862767.76 total 70.03 last 71.76) - open/close 5536 (time 1787862777.77 total 80.03 last 65.09) - open/close 6267 (time 1787862787.78 total 90.05 last 73.02) - open/close 6879 (time 1787862797.80 total 100.07 last 61.04) - open/close 7601 (time 1787862807.81 total 110.08 last 72.17) - open/close 8207 (time 1787862817.81 total 120.08 last 60.58) - open/close 8863 (time 1787862827.81 total 130.08 last 65.59) - open/close 9488 (time 1787862837.83 total 140.10 last 62.40) total: 10000 open/close in 147.64 seconds: 67.73 ops/second - unlinked 0 (time 1787862849 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 706 (time 1787862955.25 total 10.00 last 70.60) - open/close 1473 (time 1787862965.25 total 20.01 last 76.66) - open/close 2174 (time 1787862975.25 total 30.01 last 70.09) - open/close 2820 (time 1787862985.26 total 40.01 last 64.55) - open/close 3443 (time 1787862995.27 total 50.03 last 62.21) - open/close 4094 (time 1787863005.29 total 60.05 last 65.00) - open/close 4844 (time 1787863015.30 total 70.05 last 74.94) - open/close 5539 (time 1787863025.30 total 80.05 last 69.50) - open/close 6203 (time 1787863035.31 total 90.06 last 66.35) - open/close 6929 (time 1787863045.31 total 100.07 last 72.56) - open/close 7655 (time 1787863055.31 total 110.07 last 72.58) - open/close 8400 (time 1787863065.32 total 120.07 last 74.49) - open/close 9116 (time 1787863075.33 total 130.08 last 71.52) - open/close 9830 (time 1787863085.34 total 140.09 last 71.34) total: 10000 open/close in 142.47 seconds: 70.19 ops/second - unlinked 0 (time 1787863092 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 675 (time 1787863203.70 total 10.02 last 67.33) - open/close 1341 (time 1787863213.71 total 20.03 last 66.55) - open/close 2097 (time 1787863223.72 total 30.04 last 75.52) - open/close 2793 (time 1787863233.72 total 40.05 last 69.56) - open/close 3528 (time 1787863243.73 total 50.06 last 73.43) - open/close 4169 (time 1787863253.75 total 60.08 last 63.97) - open/close 4779 (time 1787863263.76 total 70.09 last 60.94) - open/close 5456 (time 1787863273.77 total 80.09 last 67.66) - open/close 6048 (time 1787863283.80 total 90.12 last 59.03) - open/close 6669 (time 1787863293.80 total 100.13 last 62.06) - open/close 7370 (time 1787863303.80 total 110.13 last 70.10) - open/close 8078 (time 1787863313.81 total 120.13 last 70.75) - open/close 8762 (time 1787863323.83 total 130.15 last 68.30) - open/close 9432 (time 1787863333.83 total 140.15 last 66.98) total: 10000 open/close in 148.17 seconds: 67.49 ops/second - unlinked 0 (time 1787863346 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 657 (time 1787863452.83 total 10.01 last 65.62) - open/close 1232 (time 1787863462.84 total 20.02 last 57.45) - open/close 1931 (time 1787863472.86 total 30.04 last 69.75) - open/close 2630 (time 1787863482.87 total 40.05 last 69.85) - open/close 3282 (time 1787863492.88 total 50.06 last 65.12) - open/close 3948 (time 1787863502.89 total 60.07 last 66.57) - open/close 4679 (time 1787863512.90 total 70.07 last 73.04) - open/close 5366 (time 1787863522.90 total 80.08 last 68.67) - open/close 5990 (time 1787863532.90 total 90.08 last 62.40) - open/close 6706 (time 1787863542.92 total 100.09 last 71.48) - open/close 7412 (time 1787863552.93 total 110.11 last 70.49) - open/close 8116 (time 1787863562.96 total 120.13 last 70.23) - open/close 8784 (time 1787863572.97 total 130.14 last 66.75) - open/close 9440 (time 1787863582.99 total 140.16 last 65.47) total: 10000 open/close in 148.43 seconds: 67.37 ops/second - unlinked 0 (time 1787863595 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 635 (time 1787863703.29 total 10.00 last 63.47) - open/close 1352 (time 1787863713.30 total 20.01 last 71.67) - open/close 2054 (time 1787863723.30 total 30.02 last 70.15) - open/close 2763 (time 1787863733.32 total 40.03 last 70.80) - open/close 3449 (time 1787863743.32 total 50.03 last 68.58) - open/close 4056 (time 1787863753.33 total 60.04 last 60.68) - open/close 4813 (time 1787863763.34 total 70.05 last 75.62) - open/close 5539 (time 1787863773.35 total 80.06 last 72.52) - open/close 6267 (time 1787863783.35 total 90.07 last 72.75) - open/close 6981 (time 1787863793.37 total 100.08 last 71.31) - open/close 7700 (time 1787863803.37 total 110.08 last 71.90) - open/close 8415 (time 1787863813.38 total 120.09 last 71.41) - open/close 9173 (time 1787863823.38 total 130.09 last 75.77) - open/close 9846 (time 1787863833.39 total 140.10 last 67.27) total: 10000 open/close in 142.34 seconds: 70.25 ops/second - unlinked 0 (time 1787863840 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 634 (time 1787863958.27 total 10.02 last 63.30) - open/close 1297 (time 1787863968.28 total 20.03 last 66.22) - open/close 1990 (time 1787863978.29 total 30.04 last 69.24) - open/close 2703 (time 1787863988.30 total 40.05 last 71.24) - open/close 3380 (time 1787863998.32 total 50.06 last 67.58) - open/close 4102 (time 1787864008.33 total 60.08 last 72.10) - open/close 4823 (time 1787864018.34 total 70.09 last 72.04) - open/close 5467 (time 1787864028.35 total 80.09 last 64.35) - open/close 6150 (time 1787864038.35 total 90.10 last 68.29) - open/close 6801 (time 1787864048.36 total 100.11 last 65.04) - open/close 7492 (time 1787864058.36 total 110.11 last 69.07) - open/close 8191 (time 1787864068.37 total 120.11 last 69.88) - open/close 8876 (time 1787864078.38 total 130.12 last 68.42) - open/close 9552 (time 1787864088.38 total 140.13 last 67.57) total: 10000 open/close in 147.23 seconds: 67.92 ops/second - unlinked 0 (time 1787864100 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 656 (time 1787864207.95 total 10.00 last 65.60) - open/close 1273 (time 1787864217.95 total 20.00 last 61.68) - open/close 2032 (time 1787864227.97 total 30.03 last 75.72) - open/close 2750 (time 1787864238.01 total 40.06 last 71.54) - open/close 3413 (time 1787864248.02 total 50.07 last 66.24) - open/close 4091 (time 1787864258.04 total 60.09 last 67.67) - open/close 4762 (time 1787864268.04 total 70.09 last 67.08) - open/close 5417 (time 1787864278.04 total 80.10 last 65.48) - open/close 6129 (time 1787864288.05 total 90.11 last 71.14) - open/close 6791 (time 1787864298.06 total 100.11 last 66.16) - open/close 7469 (time 1787864308.06 total 110.11 last 67.79) - open/close 8130 (time 1787864318.06 total 120.12 last 66.08) - open/close 8825 (time 1787864328.08 total 130.14 last 69.35) - open/close 9456 (time 1787864338.08 total 140.14 last 63.09) total: 10000 open/close in 147.95 seconds: 67.59 ops/second - unlinked 0 (time 1787864349 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 678 (time 1787864456.42 total 10.00 last 67.78) - open/close 1374 (time 1787864466.42 total 20.01 last 69.58) - open/close 2103 (time 1787864476.43 total 30.01 last 72.85) - open/close 2801 (time 1787864486.45 total 40.03 last 69.65) - open/close 3444 (time 1787864496.45 total 50.03 last 64.30) - open/close 4144 (time 1787864506.46 total 60.05 last 69.92) - open/close 4836 (time 1787864516.47 total 70.06 last 69.14) - open/close 5565 (time 1787864526.50 total 80.08 last 72.73) - open/close 6281 (time 1787864536.50 total 90.09 last 71.54) - open/close 6990 (time 1787864546.52 total 100.10 last 70.81) - open/close 7703 (time 1787864556.53 total 110.11 last 71.24) - open/close 8294 (time 1787864566.54 total 120.12 last 59.04) - open/close 8958 (time 1787864576.54 total 130.12 last 66.36) - open/close 9673 (time 1787864586.56 total 140.14 last 71.37) total: 10000 open/close in 144.83 seconds: 69.05 ops/second - unlinked 0 (time 1787864595 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 675 (time 1787864697.96 total 10.01 last 67.43) - open/close 1327 (time 1787864707.96 total 20.01 last 65.19) - open/close 2066 (time 1787864717.97 total 30.02 last 73.86) - open/close 2851 (time 1787864727.97 total 40.02 last 78.47) - open/close 3528 (time 1787864737.97 total 50.02 last 67.68) - open/close 4255 (time 1787864747.98 total 60.04 last 72.62) - open/close 4983 (time 1787864758.01 total 70.06 last 72.64) - open/close 5707 (time 1787864768.01 total 80.06 last 72.36) - open/close 6447 (time 1787864778.03 total 90.08 last 73.86) - open/close 7164 (time 1787864788.05 total 100.11 last 71.53) - open/close 7938 (time 1787864798.06 total 110.11 last 77.38) - open/close 8635 (time 1787864808.06 total 120.11 last 69.68) - open/close 9340 (time 1787864818.07 total 130.13 last 70.40) total: 10000 open/close in 139.03 seconds: 71.93 ops/second - unlinked 0 (time 1787864831 ; total 1 ; last 1) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 736 (time 1787864930.36 total 10.00 last 73.59) - open/close 1483 (time 1787864940.37 total 20.00 last 74.69) - open/close 2223 (time 1787864950.37 total 30.01 last 73.98) - open/close 2941 (time 1787864960.37 total 40.01 last 71.77) - open/close 3597 (time 1787864970.38 total 50.01 last 65.57) - open/close 4245 (time 1787864980.39 total 60.02 last 64.74) - open/close 4872 (time 1787864990.39 total 70.03 last 62.69) - open/close 5549 (time 1787865000.40 total 80.03 last 67.65) - open/close 6208 (time 1787865010.41 total 90.04 last 65.83) - open/close 6969 (time 1787865020.42 total 100.06 last 76.00) - open/close 7699 (time 1787865030.43 total 110.06 last 72.94) - open/close 8406 (time 1787865040.44 total 120.08 last 70.59) - open/close 9147 (time 1787865050.46 total 130.09 last 74.00) - open/close 9860 (time 1787865060.48 total 140.11 last 71.15) total: 10000 open/close in 141.97 seconds: 70.44 ops/second - unlinked 0 (time 1787865066 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg620-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg620-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg620-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.120@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.120@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg620-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0000 oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg620-server: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg620-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre On OST0, 20327 used inodes rc=0 pdsh@oleg620-client: oleg620-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg620-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg620-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg620-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.120@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.120@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.206.120@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.206.120@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.206.120@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg620-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0000 oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg620-server: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg620-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg620-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg620-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg620-server stop mds service on oleg620-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg620-server unloading modules via unload_modules_local on: 'oleg620-server' oleg620-server: oleg620-server.virtnet: executing unload_modules_local oleg620-server: modules unloaded. pdsh@oleg620-client: oleg620-server: ssh exited with exit code 2 pdsh@oleg620-client: oleg620-client: ssh exited with exit code 2 pdsh@oleg620-client: oleg620-client: ssh exited with exit code 2 pdsh@oleg620-client: oleg620-server: ssh exited with exit code 2