== conf-sanity test 69: replace an OST with the same index ========================================================== 07:38:21 (1787830701) start mds service on oleg204-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg204-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 559 (time 1787830792.00 total 10.03 last 55.76) - open/close 1129 (time 1787830802.01 total 20.03 last 56.95) - open/close 1799 (time 1787830812.03 total 30.05 last 66.89) - open/close 2387 (time 1787830822.03 total 40.05 last 58.79) - open/close 2958 (time 1787830832.03 total 50.05 last 57.09) - open/close 3647 (time 1787830842.04 total 60.06 last 68.86) - open/close 4545 (time 1787830852.04 total 70.06 last 89.77) - open/close 5424 (time 1787830862.04 total 80.07 last 87.88) - open/close 6134 (time 1787830872.05 total 90.08 last 70.92) - open/close 6806 (time 1787830882.06 total 100.08 last 67.17) - open/close 7528 (time 1787830892.07 total 110.09 last 72.13) - open/close 8226 (time 1787830902.08 total 120.11 last 69.70) - open/close 8815 (time 1787830912.09 total 130.12 last 58.83) - open/close 9466 (time 1787830922.12 total 140.15 last 64.93) total: 10000 open/close in 148.02 seconds: 67.56 ops/second - unlinked 0 (time 1787830934 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 707 (time 1787831037.39 total 10.01 last 70.63) - open/close 1410 (time 1787831047.40 total 20.02 last 70.25) - open/close 2126 (time 1787831057.41 total 30.03 last 71.50) - open/close 2714 (time 1787831067.42 total 40.03 last 58.78) - open/close 3313 (time 1787831077.42 total 50.04 last 59.90) - open/close 3859 (time 1787831087.43 total 60.05 last 54.53) - open/close 4519 (time 1787831097.43 total 70.05 last 65.98) - open/close 5267 (time 1787831107.44 total 80.06 last 74.74) - open/close 6007 (time 1787831117.44 total 90.06 last 73.99) - open/close 6750 (time 1787831127.44 total 100.06 last 74.27) - open/close 7403 (time 1787831137.45 total 110.06 last 65.29) - open/close 8123 (time 1787831147.45 total 120.07 last 71.95) - open/close 8865 (time 1787831157.45 total 130.07 last 74.19) - open/close 9567 (time 1787831167.46 total 140.08 last 70.17) total: 10000 open/close in 146.89 seconds: 68.08 ops/second - unlinked 0 (time 1787831178 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 740 (time 1787831281.96 total 10.01 last 73.92) - open/close 1503 (time 1787831291.97 total 20.02 last 76.21) - open/close 2225 (time 1787831301.97 total 30.03 last 72.18) - open/close 2751 (time 1787831311.99 total 40.04 last 52.50) - open/close 3378 (time 1787831321.99 total 50.05 last 62.68) - open/close 4136 (time 1787831332.02 total 60.07 last 75.62) - open/close 4724 (time 1787831342.02 total 70.07 last 58.78) - open/close 5444 (time 1787831352.04 total 80.09 last 71.86) - open/close 6173 (time 1787831362.04 total 90.10 last 72.88) - open/close 6968 (time 1787831372.04 total 100.10 last 79.47) - open/close 7701 (time 1787831382.07 total 110.12 last 73.12) - open/close 8359 (time 1787831392.08 total 120.14 last 65.71) - open/close 9031 (time 1787831402.09 total 130.14 last 67.18) - open/close 9749 (time 1787831412.10 total 140.15 last 71.73) total: 10000 open/close in 143.65 seconds: 69.62 ops/second - unlinked 0 (time 1787831420 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 787 (time 1787831523.61 total 10.00 last 78.67) - open/close 1549 (time 1787831533.62 total 20.01 last 76.12) - open/close 2186 (time 1787831543.63 total 30.02 last 63.64) - open/close 2729 (time 1787831553.63 total 40.03 last 54.26) - open/close 3553 (time 1787831563.64 total 50.03 last 82.37) - open/close 4465 (time 1787831573.64 total 60.04 last 91.14) - open/close 5229 (time 1787831583.64 total 70.04 last 76.38) - open/close 5916 (time 1787831593.65 total 80.04 last 68.68) - open/close 6689 (time 1787831603.66 total 90.05 last 77.24) - open/close 7355 (time 1787831613.67 total 100.06 last 66.52) - open/close 8045 (time 1787831623.68 total 110.08 last 68.89) - open/close 8655 (time 1787831633.71 total 120.11 last 60.81) - open/close 9288 (time 1787831643.72 total 130.11 last 63.28) total: 10000 open/close in 139.59 seconds: 71.64 ops/second - unlinked 0 (time 1787831658 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 758 (time 1787831765.53 total 10.01 last 75.70) - open/close 1510 (time 1787831775.54 total 20.02 last 75.14) - open/close 2300 (time 1787831785.55 total 30.03 last 78.94) - open/close 2962 (time 1787831795.56 total 40.04 last 66.14) - open/close 3528 (time 1787831805.58 total 50.06 last 56.45) - open/close 4156 (time 1787831815.59 total 60.07 last 62.78) - open/close 4879 (time 1787831825.59 total 70.07 last 72.27) - open/close 5609 (time 1787831835.60 total 80.08 last 72.93) - open/close 6317 (time 1787831845.61 total 90.09 last 70.73) - open/close 6938 (time 1787831855.63 total 100.11 last 61.97) - open/close 7679 (time 1787831865.63 total 110.12 last 74.07) - open/close 8382 (time 1787831875.64 total 120.12 last 70.25) - open/close 9007 (time 1787831885.66 total 130.14 last 62.40) - open/close 9697 (time 1787831895.66 total 140.14 last 68.97) total: 10000 open/close in 144.33 seconds: 69.28 ops/second - unlinked 0 (time 1787831904 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 549 (time 1787832002.30 total 10.00 last 54.90) - open/close 1287 (time 1787832012.30 total 20.00 last 73.80) - open/close 1988 (time 1787832022.30 total 30.00 last 70.10) - open/close 2725 (time 1787832032.30 total 40.01 last 73.64) - open/close 3517 (time 1787832042.32 total 50.02 last 79.08) - open/close 4214 (time 1787832052.34 total 60.04 last 69.59) - open/close 4917 (time 1787832062.34 total 70.04 last 70.27) - open/close 5647 (time 1787832072.35 total 80.05 last 72.96) - open/close 6323 (time 1787832082.35 total 90.05 last 67.59) - open/close 7125 (time 1787832092.36 total 100.06 last 80.13) - open/close 7893 (time 1787832102.36 total 110.06 last 76.78) - open/close 8443 (time 1787832112.42 total 120.12 last 54.69) - open/close 8964 (time 1787832122.42 total 130.12 last 52.10) - open/close 9680 (time 1787832132.43 total 140.14 last 71.49) total: 10000 open/close in 144.99 seconds: 68.97 ops/second - unlinked 0 (time 1787832142 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 647 (time 1787832244.07 total 10.02 last 64.59) - open/close 1321 (time 1787832254.08 total 20.03 last 67.31) - open/close 1934 (time 1787832264.09 total 30.04 last 61.24) - open/close 2642 (time 1787832274.10 total 40.05 last 70.73) - open/close 3307 (time 1787832284.11 total 50.06 last 66.43) - open/close 3963 (time 1787832294.12 total 60.07 last 65.53) - open/close 4702 (time 1787832304.13 total 70.08 last 73.81) - open/close 5416 (time 1787832314.14 total 80.09 last 71.34) - open/close 6107 (time 1787832324.15 total 90.09 last 69.07) - open/close 6793 (time 1787832334.16 total 100.11 last 68.51) - open/close 7423 (time 1787832344.16 total 110.11 last 63.00) - open/close 8140 (time 1787832354.16 total 120.11 last 71.67) - open/close 8698 (time 1787832364.16 total 130.11 last 55.80) - open/close 9394 (time 1787832374.18 total 140.13 last 69.50) total: 10000 open/close in 149.97 seconds: 66.68 ops/second - unlinked 0 (time 1787832391 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 654 (time 1787832500.46 total 10.00 last 65.37) - open/close 1238 (time 1787832510.47 total 20.01 last 58.37) - open/close 1827 (time 1787832520.48 total 30.02 last 58.84) - open/close 2434 (time 1787832530.49 total 40.04 last 60.61) - open/close 3075 (time 1787832540.50 total 50.04 last 64.08) - open/close 3696 (time 1787832550.50 total 60.04 last 62.08) - open/close 4353 (time 1787832560.52 total 70.07 last 65.55) - open/close 4946 (time 1787832570.52 total 80.07 last 59.29) - open/close 5428 (time 1787832580.54 total 90.08 last 48.12) - open/close 5971 (time 1787832590.55 total 100.09 last 54.27) - open/close 6307 (time 1787832600.55 total 110.09 last 33.58) - open/close 6787 (time 1787832610.58 total 120.12 last 47.86) - open/close 7403 (time 1787832620.59 total 130.13 last 61.56) - open/close 8055 (time 1787832630.60 total 140.14 last 65.13) - open/close 8617 (time 1787832640.60 total 150.14 last 56.18) - open/close 9252 (time 1787832650.60 total 160.15 last 63.48) - open/close 9964 (time 1787832660.61 total 170.15 last 71.15) total: 10000 open/close in 170.61 seconds: 58.61 ops/second - unlinked 0 (time 1787832665 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 547 (time 1787832785.20 total 10.00 last 54.70) - open/close 1132 (time 1787832795.20 total 20.00 last 58.49) - open/close 1699 (time 1787832805.22 total 30.02 last 56.58) - open/close 2291 (time 1787832815.25 total 40.05 last 59.03) - open/close 2904 (time 1787832825.26 total 50.06 last 61.23) - open/close 3575 (time 1787832835.27 total 60.06 last 67.08) - open/close 4118 (time 1787832845.29 total 70.08 last 54.20) - open/close 4746 (time 1787832855.29 total 80.09 last 62.74) - open/close 5482 (time 1787832865.30 total 90.10 last 73.57) - open/close 6090 (time 1787832875.30 total 100.10 last 60.79) - open/close 6579 (time 1787832885.30 total 110.10 last 48.89) - open/close 7007 (time 1787832895.31 total 120.10 last 42.78) - open/close 7514 (time 1787832905.33 total 130.12 last 50.60) - open/close 8122 (time 1787832915.34 total 140.14 last 60.71) - open/close 8750 (time 1787832925.35 total 150.14 last 62.77) - open/close 9296 (time 1787832935.37 total 160.17 last 54.48) - open/close 9854 (time 1787832945.37 total 170.17 last 55.77) total: 10000 open/close in 173.09 seconds: 57.77 ops/second - unlinked 0 (time 1787832952 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 604 (time 1787833065.46 total 10.01 last 60.34) - open/close 1182 (time 1787833075.47 total 20.02 last 57.74) - open/close 1754 (time 1787833085.50 total 30.05 last 57.05) - open/close 2304 (time 1787833095.51 total 40.06 last 54.93) - open/close 2844 (time 1787833105.53 total 50.08 last 53.90) - open/close 3364 (time 1787833115.54 total 60.09 last 51.94) - open/close 4088 (time 1787833125.54 total 70.09 last 72.39) - open/close 4819 (time 1787833135.54 total 80.09 last 73.09) - open/close 5410 (time 1787833145.56 total 90.10 last 59.02) - open/close 6154 (time 1787833155.57 total 100.12 last 74.31) - open/close 6748 (time 1787833165.57 total 110.12 last 59.39) - open/close 7406 (time 1787833175.58 total 120.13 last 65.76) - open/close 8010 (time 1787833185.59 total 130.14 last 60.30) - open/close 8665 (time 1787833195.60 total 140.15 last 65.43) - open/close 9309 (time 1787833205.62 total 150.16 last 64.32) total: 10000 open/close in 158.76 seconds: 62.99 ops/second - unlinked 0 (time 1787833219 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.104@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre On OST0, 18710 used inodes rc=0 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.104@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.104@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2