== conf-sanity test 69: replace an OST with the same index ========================================================== 14:09:49 (1781028589) start mds service on oleg437-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg437-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre kernel.panic_on_warn = 0 seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 647 (time 1781028663.01 total 10.01 last 64.67) - open/close 1283 (time 1781028673.03 total 20.02 last 63.52) - open/close 1977 (time 1781028683.04 total 30.03 last 69.33) - open/close 2672 (time 1781028693.05 total 40.04 last 69.39) - open/close 3326 (time 1781028703.06 total 50.05 last 65.37) - open/close 4033 (time 1781028713.07 total 60.06 last 70.60) - open/close 4717 (time 1781028723.07 total 70.06 last 68.38) - open/close 5381 (time 1781028733.08 total 80.07 last 66.37) - open/close 6040 (time 1781028743.09 total 90.08 last 65.82) - open/close 6743 (time 1781028753.11 total 100.10 last 70.18) - open/close 7428 (time 1781028763.12 total 110.12 last 68.38) - open/close 8096 (time 1781028773.13 total 120.12 last 66.77) - open/close 8694 (time 1781028783.14 total 130.13 last 59.73) - open/close 9361 (time 1781028793.16 total 140.15 last 66.59) total: 10000 open/close in 149.01 seconds: 67.11 ops/second - unlinked 0 (time 1781028805 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 751 (time 1781028910.55 total 10.01 last 75.03) - open/close 1530 (time 1781028920.56 total 20.01 last 77.88) - open/close 2302 (time 1781028930.57 total 30.02 last 77.11) - open/close 3057 (time 1781028940.58 total 40.04 last 75.42) - open/close 3817 (time 1781028950.58 total 50.04 last 75.97) - open/close 4594 (time 1781028960.60 total 60.06 last 77.58) - open/close 5292 (time 1781028970.62 total 70.08 last 69.63) - open/close 6028 (time 1781028980.64 total 80.09 last 73.51) - open/close 6814 (time 1781028990.64 total 90.10 last 78.53) - open/close 7569 (time 1781029000.66 total 100.12 last 75.39) - open/close 8288 (time 1781029010.67 total 110.13 last 71.80) - open/close 9030 (time 1781029020.68 total 120.14 last 74.12) - open/close 9777 (time 1781029030.69 total 130.14 last 74.67) total: 10000 open/close in 133.22 seconds: 75.06 ops/second - unlinked 0 (time 1781029037 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 795 (time 1781029126.86 total 10.01 last 79.44) - open/close 1565 (time 1781029136.87 total 20.01 last 76.95) - open/close 2283 (time 1781029146.87 total 30.02 last 71.78) - open/close 3018 (time 1781029156.88 total 40.02 last 73.47) - open/close 3806 (time 1781029166.88 total 50.03 last 78.76) - open/close 4560 (time 1781029176.89 total 60.03 last 75.35) - open/close 5317 (time 1781029186.89 total 70.03 last 75.69) - open/close 6058 (time 1781029196.90 total 80.05 last 74.01) - open/close 6812 (time 1781029206.92 total 90.06 last 75.29) - open/close 7565 (time 1781029216.92 total 100.06 last 75.30) - open/close 8344 (time 1781029226.92 total 110.06 last 77.88) - open/close 9124 (time 1781029236.92 total 120.07 last 77.99) - open/close 9892 (time 1781029246.93 total 130.07 last 76.76) total: 10000 open/close in 131.49 seconds: 76.05 ops/second - unlinked 0 (time 1781029251 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 813 (time 1781029342.60 total 10.00 last 81.26) - open/close 1660 (time 1781029352.61 total 20.01 last 84.63) - open/close 2414 (time 1781029362.61 total 30.02 last 75.37) - open/close 3171 (time 1781029372.62 total 40.02 last 75.64) - open/close 3953 (time 1781029382.64 total 50.04 last 78.05) - open/close 4736 (time 1781029392.65 total 60.05 last 78.24) - open/close 5534 (time 1781029402.66 total 70.06 last 79.73) - open/close 6389 (time 1781029412.66 total 80.06 last 85.48) - open/close 7329 (time 1781029422.66 total 90.07 last 93.97) - open/close 8191 (time 1781029432.67 total 100.07 last 86.14) - open/close 8951 (time 1781029442.67 total 110.08 last 75.96) - open/close 9726 (time 1781029452.69 total 120.09 last 77.41) total: 10000 open/close in 123.56 seconds: 80.93 ops/second - unlinked 0 (time 1781029458 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 899 (time 1781029540.72 total 10.00 last 89.88) - open/close 1801 (time 1781029550.72 total 20.01 last 90.16) - open/close 2665 (time 1781029560.73 total 30.01 last 86.35) - open/close 3517 (time 1781029570.73 total 40.02 last 85.16) - open/close 4513 (time 1781029580.74 total 50.03 last 99.51) - open/close 5423 (time 1781029590.75 total 60.03 last 90.96) - open/close 6302 (time 1781029600.76 total 70.04 last 87.79) - open/close 7265 (time 1781029610.77 total 80.05 last 96.21) - open/close 8197 (time 1781029620.78 total 90.06 last 93.11) - open/close 9265 (time 1781029630.78 total 100.07 last 106.74) total: 10000 open/close in 108.07 seconds: 92.53 ops/second - unlinked 0 (time 1781029641 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1064 (time 1781029713.02 total 10.01 last 106.32) - open/close 2113 (time 1781029723.02 total 20.01 last 104.88) - open/close 3080 (time 1781029733.03 total 30.02 last 96.62) - open/close 4005 (time 1781029743.03 total 40.02 last 92.48) - open/close 4906 (time 1781029753.04 total 50.03 last 90.01) - open/close 5811 (time 1781029763.05 total 60.03 last 90.47) - open/close 6808 (time 1781029773.05 total 70.03 last 99.70) - open/close 7897 (time 1781029783.06 total 80.04 last 108.80) - open/close 8941 (time 1781029793.06 total 90.04 last 104.38) total: 10000 open/close in 99.77 seconds: 100.23 ops/second - unlinked 0 (time 1781029805 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1100 (time 1781029868.79 total 10.01 last 109.93) - open/close 2175 (time 1781029878.80 total 20.01 last 107.46) - open/close 3198 (time 1781029888.81 total 30.02 last 102.20) - open/close 4314 (time 1781029898.81 total 40.02 last 111.59) - open/close 5451 (time 1781029908.81 total 50.03 last 113.64) - open/close 6679 (time 1781029918.82 total 60.03 last 122.73) - open/close 8000 (time 1781029928.82 total 70.04 last 132.05) - open/close 9313 (time 1781029938.83 total 80.04 last 131.22) total: 10000 open/close in 85.85 seconds: 116.48 ops/second - unlinked 0 (time 1781029946 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1532 (time 1781030001.86 total 10.00 last 153.16) - open/close 2783 (time 1781030011.86 total 20.01 last 125.06) - open/close 4145 (time 1781030021.86 total 30.01 last 136.17) - open/close 5586 (time 1781030031.86 total 40.01 last 144.08) - open/close 6939 (time 1781030041.87 total 50.02 last 135.21) - open/close 8419 (time 1781030051.89 total 60.04 last 147.66) - open/close 9851 (time 1781030061.90 total 70.05 last 143.06) total: 10000 open/close in 71.36 seconds: 140.14 ops/second - unlinked 0 (time 1781030065 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1481 (time 1781030119.75 total 10.01 last 148.02) - open/close 2984 (time 1781030129.76 total 20.01 last 150.23) - open/close 4881 (time 1781030139.76 total 30.01 last 189.69) - open/close 6420 (time 1781030149.76 total 40.02 last 153.82) - open/close 8197 (time 1781030159.77 total 50.02 last 177.58) - open/close 9945 (time 1781030169.77 total 60.03 last 174.73) total: 10000 open/close in 60.29 seconds: 165.85 ops/second - unlinked 0 (time 1781030171 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1844 (time 1781030216.47 total 10.00 last 184.39) - open/close 3963 (time 1781030226.47 total 20.00 last 211.89) - open/close 6274 (time 1781030236.47 total 30.00 last 231.07) - open/close 8434 (time 1781030246.47 total 40.01 last 215.92) total: 10000 open/close in 46.61 seconds: 214.54 ops/second - unlinked 0 (time 1781030254 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre kernel.panic_on_warn = 0 On OST0, 19430 used inodes rc=0 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.137@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.137@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.137@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre kernel.panic_on_warn = 0 umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local oleg437-server: modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2