== conf-sanity test 69: replace an OST with the same index ========================================================== 04:51:31 (1779699091) start mds service on oleg238-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: MODOPTS_LIBCFS= oleg238-server: Force libcfs to create 2 CPU partitions oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg238-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 627 (time 1779699181.86 total 10.02 last 62.59) - open/close 1193 (time 1779699191.88 total 20.03 last 56.51) - open/close 1850 (time 1779699201.88 total 30.03 last 65.69) - open/close 2510 (time 1779699211.88 total 40.04 last 65.97) - open/close 3360 (time 1779699221.93 total 50.08 last 84.61) - open/close 4032 (time 1779699231.96 total 60.12 last 66.98) - open/close 4760 (time 1779699241.97 total 70.13 last 72.72) - open/close 5314 (time 1779699251.97 total 80.13 last 55.40) - open/close 5968 (time 1779699261.98 total 90.13 last 65.37) - open/close 6717 (time 1779699271.98 total 100.14 last 74.89) - open/close 7433 (time 1779699281.98 total 110.14 last 71.57) - open/close 8188 (time 1779699291.99 total 120.15 last 75.41) - open/close 8910 (time 1779699301.99 total 130.15 last 72.18) - open/close 9777 (time 1779699312.00 total 140.16 last 86.65) total: 10000 open/close in 142.85 seconds: 70.00 ops/second - unlinked 0 (time 1779699319 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 856 (time 1779699415.77 total 10.01 last 85.54) - open/close 1612 (time 1779699425.78 total 20.02 last 75.52) - open/close 2476 (time 1779699435.78 total 30.02 last 86.34) - open/close 3150 (time 1779699445.79 total 40.03 last 67.33) - open/close 3828 (time 1779699455.81 total 50.05 last 67.71) - open/close 4602 (time 1779699465.81 total 60.05 last 77.35) - open/close 5379 (time 1779699475.84 total 70.08 last 77.51) - open/close 6054 (time 1779699485.85 total 80.09 last 67.45) - open/close 6658 (time 1779699495.86 total 90.10 last 60.33) - open/close 7212 (time 1779699505.86 total 100.10 last 55.40) - open/close 7753 (time 1779699515.86 total 110.10 last 54.07) - open/close 8351 (time 1779699525.86 total 120.10 last 59.80) - open/close 9065 (time 1779699535.87 total 130.11 last 71.33) - open/close 9856 (time 1779699545.88 total 140.12 last 79.07) total: 10000 open/close in 142.30 seconds: 70.28 ops/second - unlinked 0 (time 1779699552 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 811 (time 1779699655.29 total 10.01 last 81.01) - open/close 1551 (time 1779699665.29 total 20.01 last 73.98) - open/close 2240 (time 1779699675.30 total 30.02 last 68.85) - open/close 3028 (time 1779699685.31 total 40.04 last 78.68) - open/close 3813 (time 1779699695.32 total 50.04 last 78.45) - open/close 4560 (time 1779699705.34 total 60.06 last 74.55) - open/close 5387 (time 1779699715.34 total 70.07 last 82.66) - open/close 6179 (time 1779699725.35 total 80.08 last 79.14) - open/close 6820 (time 1779699735.35 total 90.08 last 64.08) - open/close 7269 (time 1779699745.39 total 100.11 last 44.75) - open/close 7783 (time 1779699755.39 total 110.11 last 51.38) - open/close 8479 (time 1779699765.40 total 120.13 last 69.51) - open/close 9158 (time 1779699775.41 total 130.14 last 67.81) - open/close 9798 (time 1779699785.43 total 140.16 last 63.87) total: 10000 open/close in 143.23 seconds: 69.82 ops/second - unlinked 0 (time 1779699792 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 709 (time 1779699900.95 total 10.00 last 70.88) - open/close 1431 (time 1779699910.97 total 20.02 last 72.07) - open/close 2108 (time 1779699920.98 total 30.03 last 67.65) - open/close 2984 (time 1779699930.98 total 40.03 last 87.56) - open/close 3666 (time 1779699940.98 total 50.03 last 68.20) - open/close 4422 (time 1779699950.99 total 60.04 last 75.56) - open/close 5087 (time 1779699960.99 total 70.05 last 66.46) - open/close 5741 (time 1779699971.01 total 80.06 last 65.29) - open/close 6414 (time 1779699981.01 total 90.06 last 67.30) - open/close 7047 (time 1779699991.02 total 100.07 last 63.24) - open/close 7744 (time 1779700001.02 total 110.08 last 69.67) - open/close 8426 (time 1779700011.03 total 120.08 last 68.18) - open/close 9203 (time 1779700021.03 total 130.09 last 77.64) - open/close 9934 (time 1779700031.04 total 140.09 last 73.05) total: 10000 open/close in 140.87 seconds: 70.99 ops/second - unlinked 0 (time 1779700036 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 853 (time 1779700133.43 total 10.01 last 85.23) - open/close 1653 (time 1779700143.44 total 20.02 last 79.90) - open/close 2406 (time 1779700153.45 total 30.03 last 75.25) - open/close 3091 (time 1779700163.45 total 40.03 last 68.49) - open/close 3850 (time 1779700173.45 total 50.03 last 75.88) - open/close 4527 (time 1779700183.47 total 60.05 last 67.59) - open/close 5238 (time 1779700193.47 total 70.06 last 71.04) - open/close 5824 (time 1779700203.51 total 80.09 last 58.40) - open/close 6497 (time 1779700213.52 total 90.10 last 67.24) - open/close 7227 (time 1779700223.52 total 100.10 last 72.98) - open/close 7988 (time 1779700233.54 total 110.13 last 75.92) - open/close 8685 (time 1779700243.55 total 120.13 last 69.69) - open/close 9445 (time 1779700253.55 total 130.14 last 75.95) total: 10000 open/close in 136.94 seconds: 73.02 ops/second - unlinked 0 (time 1779700264 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 738 (time 1779700363.68 total 10.01 last 73.72) - open/close 1508 (time 1779700373.69 total 20.02 last 76.92) - open/close 2206 (time 1779700383.69 total 30.02 last 69.78) - open/close 2933 (time 1779700393.69 total 40.03 last 72.68) - open/close 3619 (time 1779700403.71 total 50.05 last 68.46) - open/close 4290 (time 1779700413.73 total 60.06 last 67.02) - open/close 5103 (time 1779700423.73 total 70.06 last 81.30) - open/close 5857 (time 1779700433.73 total 80.06 last 75.37) - open/close 6599 (time 1779700443.73 total 90.06 last 74.19) - open/close 7351 (time 1779700453.74 total 100.07 last 75.16) - open/close 8189 (time 1779700463.74 total 110.07 last 83.79) - open/close 9014 (time 1779700473.74 total 120.07 last 82.47) - open/close 9790 (time 1779700483.74 total 130.08 last 77.57) total: 10000 open/close in 132.70 seconds: 75.36 ops/second - unlinked 0 (time 1779700490 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 682 (time 1779700590.01 total 10.00 last 68.17) - open/close 1316 (time 1779700600.01 total 20.00 last 63.40) - open/close 1982 (time 1779700610.02 total 30.02 last 66.53) - open/close 2468 (time 1779700620.04 total 40.03 last 48.51) - open/close 2985 (time 1779700630.05 total 50.04 last 51.66) - open/close 3645 (time 1779700640.06 total 60.05 last 65.95) - open/close 4163 (time 1779700650.06 total 70.05 last 51.79) - open/close 4758 (time 1779700660.06 total 80.05 last 59.47) - open/close 5424 (time 1779700670.07 total 90.07 last 66.51) - open/close 6083 (time 1779700680.10 total 100.10 last 65.72) - open/close 6724 (time 1779700690.11 total 110.10 last 64.06) - open/close 7372 (time 1779700700.14 total 120.13 last 64.59) - open/close 8023 (time 1779700710.15 total 130.14 last 65.06) - open/close 8644 (time 1779700720.15 total 140.14 last 62.10) - open/close 9378 (time 1779700730.16 total 150.16 last 73.28) total: 10000 open/close in 159.38 seconds: 62.74 ops/second - unlinked 0 (time 1779700744 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 749 (time 1779700849.69 total 10.01 last 74.80) - open/close 1428 (time 1779700859.71 total 20.03 last 67.78) - open/close 2099 (time 1779700869.72 total 30.04 last 67.06) - open/close 2753 (time 1779700879.73 total 40.05 last 65.32) - open/close 3413 (time 1779700889.73 total 50.05 last 65.99) - open/close 4207 (time 1779700899.74 total 60.06 last 79.34) - open/close 4987 (time 1779700909.74 total 70.06 last 77.99) - open/close 5714 (time 1779700919.75 total 80.07 last 72.64) - open/close 6497 (time 1779700929.78 total 90.10 last 78.07) - open/close 7117 (time 1779700939.78 total 100.10 last 61.97) - open/close 7909 (time 1779700949.79 total 110.11 last 79.18) - open/close 8555 (time 1779700959.79 total 120.11 last 64.55) - open/close 9327 (time 1779700969.82 total 130.14 last 77.01) total: 10000 open/close in 139.13 seconds: 71.87 ops/second - unlinked 0 (time 1779700983 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 723 (time 1779701079.33 total 10.02 last 72.18) - open/close 1513 (time 1779701089.35 total 20.03 last 78.90) - open/close 2304 (time 1779701099.36 total 30.04 last 79.03) - open/close 3058 (time 1779701109.37 total 40.05 last 75.30) - open/close 3830 (time 1779701119.38 total 50.06 last 77.15) - open/close 4537 (time 1779701129.38 total 60.06 last 70.70) - open/close 5264 (time 1779701139.39 total 70.07 last 72.62) - open/close 5994 (time 1779701149.39 total 80.07 last 72.98) - open/close 6787 (time 1779701159.39 total 90.07 last 79.29) - open/close 7467 (time 1779701169.40 total 100.08 last 67.94) - open/close 8198 (time 1779701179.40 total 110.08 last 73.08) - open/close 8881 (time 1779701189.41 total 120.09 last 68.24) - open/close 9583 (time 1779701199.41 total 130.10 last 70.19) total: 10000 open/close in 135.89 seconds: 73.59 ops/second - unlinked 0 (time 1779701209 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 764 (time 1779701307.10 total 10.01 last 76.30) - open/close 1525 (time 1779701317.12 total 20.03 last 75.96) - open/close 2284 (time 1779701327.14 total 30.05 last 75.76) - open/close 2956 (time 1779701337.14 total 40.05 last 67.17) - open/close 3669 (time 1779701347.15 total 50.07 last 71.20) - open/close 4479 (time 1779701357.15 total 60.07 last 80.99) - open/close 5124 (time 1779701367.19 total 70.10 last 64.27) - open/close 5894 (time 1779701377.20 total 80.11 last 76.92) - open/close 6519 (time 1779701387.21 total 90.12 last 62.46) - open/close 7214 (time 1779701397.22 total 100.13 last 69.41) - open/close 7927 (time 1779701407.22 total 110.14 last 71.29) - open/close 8630 (time 1779701417.24 total 120.15 last 70.17) - open/close 9335 (time 1779701427.25 total 130.16 last 70.43) total: 10000 open/close in 139.96 seconds: 71.45 ops/second - unlinked 0 (time 1779701440 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.138@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.138@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.138@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.138@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.138@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: modules unloaded. pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2