== conf-sanity test 69: replace an OST with the same index ========================================================== 16:11:19 (1773605479)
start mds service on oleg401-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg401-server'
oleg401-server: oleg401-server.virtnet: executing load_modules_local
oleg401-server: Loading modules from /home/green/git/lustre-release/lustre
oleg401-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg401-server: detected 4 online CPUs by sysfs
oleg401-server: MODOPTS_LIBCFS=
oleg401-server: Force libcfs to create 2 CPU partitions
oleg401-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg401-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg401-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all
pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg401-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all
pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1
Started lustre-OST0000
oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg401-client.virtnet:  -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11872 inodes available. Want 99872. rc=0
 - open/close 638 (time 1773605571.89 total 10.01 last 63.77)
 - open/close 1320 (time 1773605581.90 total 20.01 last 68.13)
 - open/close 1971 (time 1773605591.91 total 30.02 last 65.05)
 - open/close 2570 (time 1773605601.91 total 40.03 last 59.88)
 - open/close 3072 (time 1773605611.93 total 50.04 last 50.12)
 - open/close 3596 (time 1773605621.94 total 60.05 last 52.35)
 - open/close 4094 (time 1773605631.95 total 70.06 last 49.75)
 - open/close 4660 (time 1773605641.95 total 80.06 last 56.57)
 - open/close 5254 (time 1773605651.97 total 90.08 last 59.31)
 - open/close 5863 (time 1773605661.98 total 100.09 last 60.80)
 - open/close 6429 (time 1773605671.98 total 110.10 last 56.60)
 - open/close 7183 (time 1773605682.00 total 120.11 last 75.30)
 - open/close 7881 (time 1773605692.01 total 130.12 last 69.73)
 - open/close 8497 (time 1773605702.03 total 140.14 last 61.49)
 - open/close 9097 (time 1773605712.05 total 150.16 last 59.87)
 - open/close 9622 (time 1773605722.05 total 160.16 last 52.48)
total: 10000 open/close in 166.90 seconds: 59.91 ops/second
 - unlinked 0 (time 1773605733 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 659 (time 1773605842.62 total 10.00 last 65.88)
 - open/close 1251 (time 1773605852.63 total 20.01 last 59.15)
 - open/close 1753 (time 1773605862.64 total 30.03 last 50.13)
 - open/close 2324 (time 1773605872.66 total 40.05 last 56.98)
 - open/close 2955 (time 1773605882.67 total 50.05 last 63.05)
 - open/close 3657 (time 1773605892.67 total 60.06 last 70.18)
 - open/close 4368 (time 1773605902.68 total 70.06 last 71.06)
 - open/close 5002 (time 1773605912.69 total 80.07 last 63.34)
 - open/close 5559 (time 1773605922.69 total 90.08 last 55.67)
 - open/close 6129 (time 1773605932.70 total 100.08 last 56.96)
 - open/close 6576 (time 1773605942.71 total 110.10 last 44.63)
 - open/close 7097 (time 1773605952.72 total 120.10 last 52.08)
 - open/close 7682 (time 1773605962.73 total 130.12 last 58.40)
 - open/close 8407 (time 1773605972.74 total 140.13 last 72.46)
 - open/close 9021 (time 1773605982.75 total 150.13 last 61.37)
 - open/close 9737 (time 1773605992.75 total 160.14 last 71.54)
total: 10000 open/close in 163.38 seconds: 61.21 ops/second
 - unlinked 0 (time 1773606000 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 734 (time 1773606105.77 total 10.00 last 73.37)
 - open/close 1387 (time 1773606115.78 total 20.01 last 65.27)
 - open/close 2165 (time 1773606125.80 total 30.03 last 77.65)
 - open/close 2837 (time 1773606135.81 total 40.04 last 67.14)
 - open/close 3487 (time 1773606145.84 total 50.06 last 64.81)
 - open/close 4115 (time 1773606155.84 total 60.07 last 62.78)
 - open/close 4593 (time 1773606165.85 total 70.08 last 47.75)
 - open/close 5128 (time 1773606175.85 total 80.08 last 53.47)
 - open/close 5765 (time 1773606185.86 total 90.08 last 63.69)
 - open/close 6307 (time 1773606195.88 total 100.11 last 54.07)
 - open/close 6865 (time 1773606205.90 total 110.12 last 55.72)
 - open/close 7532 (time 1773606215.94 total 120.17 last 66.38)
 - open/close 8198 (time 1773606225.95 total 130.18 last 66.55)
 - open/close 8936 (time 1773606235.95 total 140.18 last 73.80)
 - open/close 9631 (time 1773606245.97 total 150.19 last 69.41)
total: 10000 open/close in 154.89 seconds: 64.56 ops/second
 - unlinked 0 (time 1773606256 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 708 (time 1773606357.74 total 10.01 last 70.74)
 - open/close 1234 (time 1773606367.76 total 20.03 last 52.48)
 - open/close 1675 (time 1773606377.76 total 30.04 last 44.08)
 - open/close 2229 (time 1773606387.78 total 40.05 last 55.32)
 - open/close 2955 (time 1773606397.81 total 50.08 last 72.39)
 - open/close 3511 (time 1773606407.82 total 60.09 last 55.53)
 - open/close 4074 (time 1773606417.82 total 70.10 last 56.27)
 - open/close 4621 (time 1773606427.83 total 80.10 last 54.66)
 - open/close 5300 (time 1773606437.84 total 90.11 last 67.83)
 - open/close 5966 (time 1773606447.85 total 100.12 last 66.56)
 - open/close 6739 (time 1773606457.85 total 110.12 last 77.27)
 - open/close 7606 (time 1773606467.86 total 120.14 last 86.59)
 - open/close 8348 (time 1773606477.94 total 130.21 last 73.65)
 - open/close 9066 (time 1773606487.95 total 140.22 last 71.75)
 - open/close 9870 (time 1773606497.95 total 150.22 last 80.36)
total: 10000 open/close in 152.95 seconds: 65.38 ops/second
 - unlinked 0 (time 1773606505 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 598 (time 1773606611.14 total 10.06 last 59.47)
 - open/close 1173 (time 1773606621.17 total 20.08 last 57.33)
 - open/close 1845 (time 1773606631.18 total 30.09 last 67.17)
 - open/close 2655 (time 1773606641.18 total 40.10 last 80.95)
 - open/close 3419 (time 1773606651.18 total 50.10 last 76.40)
 - open/close 4015 (time 1773606661.19 total 60.10 last 59.59)
 - open/close 4737 (time 1773606671.20 total 70.11 last 72.12)
 - open/close 5300 (time 1773606681.21 total 80.12 last 56.25)
 - open/close 5923 (time 1773606691.21 total 90.12 last 62.28)
 - open/close 6483 (time 1773606701.22 total 100.13 last 55.96)
 - open/close 7183 (time 1773606711.26 total 110.17 last 69.70)
 - open/close 7708 (time 1773606721.27 total 120.18 last 52.46)
 - open/close 8303 (time 1773606731.29 total 130.20 last 59.38)
 - open/close 8873 (time 1773606741.30 total 140.21 last 56.93)
 - open/close 9459 (time 1773606751.31 total 150.22 last 58.57)
total: 10000 open/close in 159.60 seconds: 62.66 ops/second
 - unlinked 0 (time 1773606766 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 651 (time 1773606881.07 total 10.00 last 65.08)
 - open/close 1230 (time 1773606891.07 total 20.00 last 57.89)
 - open/close 1737 (time 1773606901.08 total 30.01 last 50.66)
 - open/close 2394 (time 1773606911.08 total 40.01 last 65.69)
 - open/close 2915 (time 1773606921.09 total 50.02 last 52.06)
 - open/close 3479 (time 1773606931.10 total 60.03 last 56.34)
 - open/close 4144 (time 1773606941.10 total 70.03 last 66.48)
 - open/close 4840 (time 1773606951.11 total 80.04 last 69.53)
 - open/close 5519 (time 1773606961.11 total 90.05 last 67.86)
 - open/close 6139 (time 1773606971.12 total 100.06 last 61.94)
 - open/close 6812 (time 1773606981.13 total 110.06 last 67.29)
 - open/close 7453 (time 1773606991.13 total 120.07 last 64.06)
 - open/close 8220 (time 1773607001.14 total 130.08 last 76.63)
 - open/close 8927 (time 1773607011.15 total 140.09 last 70.64)
 - open/close 9700 (time 1773607021.16 total 150.10 last 77.22)
total: 10000 open/close in 153.88 seconds: 64.99 ops/second
 - unlinked 0 (time 1773607030 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 663 (time 1773607130.64 total 10.00 last 66.27)
 - open/close 1374 (time 1773607140.64 total 20.00 last 71.10)
 - open/close 1934 (time 1773607150.65 total 30.01 last 55.98)
 - open/close 2586 (time 1773607160.65 total 40.01 last 65.18)
 - open/close 3377 (time 1773607170.65 total 50.01 last 79.08)
 - open/close 4120 (time 1773607180.66 total 60.02 last 74.23)
 - open/close 4651 (time 1773607190.67 total 70.03 last 53.07)
 - open/close 5299 (time 1773607200.68 total 80.04 last 64.75)
 - open/close 5763 (time 1773607210.68 total 90.04 last 46.38)
 - open/close 6441 (time 1773607220.69 total 100.05 last 67.72)
 - open/close 6972 (time 1773607230.69 total 110.05 last 53.10)
 - open/close 7736 (time 1773607240.72 total 120.08 last 76.20)
 - open/close 8416 (time 1773607250.72 total 130.08 last 67.97)
 - open/close 9169 (time 1773607260.74 total 140.10 last 75.19)
 - open/close 9878 (time 1773607270.75 total 150.11 last 70.85)
total: 10000 open/close in 151.68 seconds: 65.93 ops/second
 - unlinked 0 (time 1773607277 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 529 (time 1773607381.84 total 10.00 last 52.88)
 - open/close 1131 (time 1773607391.85 total 20.02 last 60.13)
 - open/close 1736 (time 1773607401.85 total 30.02 last 60.47)
 - open/close 2339 (time 1773607411.87 total 40.04 last 60.18)
 - open/close 2786 (time 1773607421.88 total 50.05 last 44.66)
 - open/close 3230 (time 1773607431.89 total 60.06 last 44.34)
 - open/close 3794 (time 1773607441.91 total 70.08 last 56.29)
 - open/close 4350 (time 1773607451.94 total 80.10 last 55.47)
 - open/close 4983 (time 1773607461.95 total 90.11 last 63.23)
 - open/close 5588 (time 1773607471.95 total 100.12 last 60.45)
 - open/close 6197 (time 1773607481.98 total 110.15 last 60.76)
 - open/close 6820 (time 1773607491.98 total 120.15 last 62.28)
 - open/close 7396 (time 1773607501.99 total 130.16 last 57.56)
 - open/close 8072 (time 1773607512.00 total 140.17 last 67.51)
 - open/close 8748 (time 1773607522.01 total 150.18 last 67.53)
 - open/close 9514 (time 1773607532.01 total 160.18 last 76.59)
total: 10000 open/close in 168.01 seconds: 59.52 ops/second
 - unlinked 0 (time 1773607545 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 703 (time 1773607654.54 total 10.00 last 70.28)
 - open/close 1341 (time 1773607664.55 total 20.01 last 63.74)
 - open/close 2066 (time 1773607674.55 total 30.01 last 72.50)
 - open/close 2694 (time 1773607684.55 total 40.01 last 62.79)
 - open/close 3360 (time 1773607694.56 total 50.02 last 66.58)
 - open/close 4026 (time 1773607704.56 total 60.02 last 66.55)
 - open/close 4732 (time 1773607714.57 total 70.03 last 70.57)
 - open/close 5389 (time 1773607724.57 total 80.03 last 65.70)
 - open/close 6008 (time 1773607734.59 total 90.05 last 61.76)
 - open/close 6697 (time 1773607744.60 total 100.06 last 68.82)
 - open/close 7423 (time 1773607754.61 total 110.07 last 72.57)
 - open/close 8104 (time 1773607764.61 total 120.07 last 68.08)
 - open/close 8756 (time 1773607774.62 total 130.08 last 65.15)
 - open/close 9397 (time 1773607784.62 total 140.08 last 64.10)
total: 10000 open/close in 148.72 seconds: 67.24 ops/second
 - unlinked 0 (time 1773607798 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 754 (time 1773607903.80 total 10.01 last 75.35)
 - open/close 1557 (time 1773607913.81 total 20.01 last 80.28)
 - open/close 2296 (time 1773607923.81 total 30.01 last 73.90)
 - open/close 3095 (time 1773607933.82 total 40.02 last 79.80)
 - open/close 3775 (time 1773607943.82 total 50.02 last 68.00)
 - open/close 4491 (time 1773607953.83 total 60.03 last 71.52)
 - open/close 5178 (time 1773607963.83 total 70.04 last 68.69)
 - open/close 5940 (time 1773607973.84 total 80.04 last 76.15)
 - open/close 6688 (time 1773607983.84 total 90.04 last 74.80)
 - open/close 7534 (time 1773607993.84 total 100.05 last 84.57)
 - open/close 8270 (time 1773608003.85 total 110.05 last 73.54)
 - open/close 8938 (time 1773608013.85 total 120.06 last 66.78)
 - open/close 9668 (time 1773608023.86 total 130.06 last 72.95)
total: 10000 open/close in 136.08 seconds: 73.49 ops/second
 - unlinked 0 (time 1773608034 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg401-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg401-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.101@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.101@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg401-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all
pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1
Started lustre-OST0000
oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg401-server: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg401-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg401-client.virtnet:  -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre
On OST0, 19621 used inodes rc=0
pdsh@oleg401-client: oleg401-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg401-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg401-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.101@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.101@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.101@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.101@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.204.101@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg401-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all
pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1
Started lustre-OST0000
oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg401-server: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg401-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg401-client.virtnet:  -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg401-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg401-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server
stop mds service on oleg401-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server

unloading modules via unload_modules_local on: 'oleg401-server'
oleg401-server: oleg401-server.virtnet: executing unload_modules_local
oleg401-server: 
modules unloaded.
pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2
pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2
pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2
pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2