== conf-sanity test 69: replace an OST with the same index ========================================================== 01:56:21 (1787032581)
start mds service on oleg138-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg138-server'
oleg138-server: oleg138-server.virtnet: executing load_modules_local
oleg138-server: Loading modules from /home/green/git/lustre-release/lustre
oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg138-server: detected 4 online CPUs by sysfs
oleg138-server: MODOPTS_LIBCFS=
oleg138-server: Force libcfs to create 2 CPU partitions
oleg138-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg138-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg138-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all
pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg138-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all
pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1
Started lustre-OST0000
oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg138-client.virtnet:  -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 822 (time 1787032665.00 total 10.00 last 82.19)
 - open/close 1615 (time 1787032675.00 total 20.00 last 79.28)
 - open/close 2526 (time 1787032685.01 total 30.01 last 91.01)
 - open/close 3466 (time 1787032695.03 total 40.03 last 93.80)
 - open/close 4232 (time 1787032705.04 total 50.04 last 76.54)
 - open/close 5169 (time 1787032715.05 total 60.05 last 93.63)
 - open/close 5919 (time 1787032725.05 total 70.05 last 74.99)
 - open/close 6761 (time 1787032735.05 total 80.05 last 84.16)
 - open/close 7595 (time 1787032745.06 total 90.06 last 83.35)
 - open/close 8349 (time 1787032755.07 total 100.07 last 75.36)
 - open/close 9069 (time 1787032765.07 total 110.07 last 71.97)
 - open/close 9896 (time 1787032775.07 total 120.07 last 82.67)
total: 10000 open/close in 121.39 seconds: 82.38 ops/second
 - unlinked 0 (time 1787032780 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 749 (time 1787032874.21 total 10.01 last 74.85)
 - open/close 1525 (time 1787032884.21 total 20.01 last 77.57)
 - open/close 2380 (time 1787032894.21 total 30.01 last 85.47)
 - open/close 3180 (time 1787032904.21 total 40.02 last 79.99)
 - open/close 4083 (time 1787032914.22 total 50.02 last 90.27)
 - open/close 4946 (time 1787032924.22 total 60.02 last 86.25)
 - open/close 5840 (time 1787032934.23 total 70.03 last 89.35)
 - open/close 6668 (time 1787032944.23 total 80.03 last 82.77)
 - open/close 7434 (time 1787032954.23 total 90.04 last 76.58)
 - open/close 8245 (time 1787032964.24 total 100.04 last 81.08)
 - open/close 9025 (time 1787032974.24 total 110.04 last 78.00)
 - open/close 9741 (time 1787032984.26 total 120.06 last 71.45)
total: 10000 open/close in 123.10 seconds: 81.23 ops/second
 - unlinked 0 (time 1787032991 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 821 (time 1787033087.24 total 10.01 last 82.04)
 - open/close 1528 (time 1787033097.24 total 20.01 last 70.67)
 - open/close 2402 (time 1787033107.24 total 30.01 last 87.39)
 - open/close 3292 (time 1787033117.25 total 40.02 last 88.97)
 - open/close 4239 (time 1787033127.25 total 50.02 last 94.67)
 - open/close 5109 (time 1787033137.26 total 60.02 last 86.95)
 - open/close 6047 (time 1787033147.28 total 70.05 last 93.56)
 - open/close 6991 (time 1787033157.28 total 80.05 last 94.39)
 - open/close 7795 (time 1787033167.29 total 90.06 last 80.37)
 - open/close 8587 (time 1787033177.29 total 100.06 last 79.18)
 - open/close 9457 (time 1787033187.30 total 110.06 last 86.95)
total: 10000 open/close in 116.70 seconds: 85.69 ops/second
 - unlinked 0 (time 1787033198 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 807 (time 1787033295.44 total 10.01 last 80.60)
 - open/close 1687 (time 1787033305.46 total 20.03 last 87.82)
 - open/close 2602 (time 1787033315.47 total 30.04 last 91.44)
 - open/close 3357 (time 1787033325.48 total 40.05 last 75.43)
 - open/close 4096 (time 1787033335.48 total 50.05 last 73.89)
 - open/close 4959 (time 1787033345.49 total 60.06 last 86.18)
 - open/close 5776 (time 1787033355.49 total 70.06 last 81.70)
 - open/close 6591 (time 1787033365.50 total 80.07 last 81.49)
 - open/close 7504 (time 1787033375.50 total 90.07 last 91.23)
 - open/close 8325 (time 1787033385.51 total 100.08 last 82.01)
 - open/close 9028 (time 1787033395.53 total 110.10 last 70.20)
 - open/close 9802 (time 1787033405.53 total 120.10 last 77.37)
total: 10000 open/close in 122.40 seconds: 81.70 ops/second
 - unlinked 0 (time 1787033412 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 804 (time 1787033502.40 total 10.00 last 80.38)
 - open/close 1537 (time 1787033512.41 total 20.01 last 73.22)
 - open/close 2324 (time 1787033522.41 total 30.01 last 78.70)
 - open/close 3198 (time 1787033532.42 total 40.02 last 87.33)
 - open/close 4011 (time 1787033542.43 total 50.02 last 81.28)
 - open/close 4927 (time 1787033552.44 total 60.04 last 91.44)
 - open/close 5736 (time 1787033562.45 total 70.05 last 80.87)
 - open/close 6658 (time 1787033572.45 total 80.05 last 92.16)
 - open/close 7492 (time 1787033582.48 total 90.08 last 83.14)
 - open/close 8450 (time 1787033592.48 total 100.08 last 95.78)
 - open/close 9280 (time 1787033602.51 total 110.11 last 82.78)
total: 10000 open/close in 118.85 seconds: 84.14 ops/second
 - unlinked 0 (time 1787033614 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 904 (time 1787033702.90 total 10.01 last 90.32)
 - open/close 1819 (time 1787033712.91 total 20.02 last 91.41)
 - open/close 2786 (time 1787033722.91 total 30.02 last 96.67)
 - open/close 3709 (time 1787033732.91 total 40.02 last 92.29)
 - open/close 4577 (time 1787033742.91 total 50.03 last 86.77)
 - open/close 5417 (time 1787033752.92 total 60.03 last 83.95)
 - open/close 6194 (time 1787033762.92 total 70.04 last 77.67)
 - open/close 7068 (time 1787033772.94 total 80.05 last 87.30)
 - open/close 7974 (time 1787033782.94 total 90.05 last 90.57)
 - open/close 8852 (time 1787033792.95 total 100.06 last 87.69)
 - open/close 9758 (time 1787033802.96 total 110.07 last 90.57)
total: 10000 open/close in 112.61 seconds: 88.80 ops/second
 - unlinked 0 (time 1787033809 ; total 0 ; last 0)
total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second
 - open/close 941 (time 1787033890.50 total 10.00 last 94.06)
 - open/close 1981 (time 1787033900.50 total 20.01 last 103.97)
 - open/close 2829 (time 1787033910.51 total 30.01 last 84.73)
 - open/close 3734 (time 1787033920.51 total 40.02 last 90.49)
 - open/close 4685 (time 1787033930.51 total 50.02 last 95.10)
 - open/close 5766 (time 1787033940.52 total 60.02 last 108.03)
 - open/close 6669 (time 1787033950.53 total 70.03 last 90.24)
 - open/close 7741 (time 1787033960.53 total 80.03 last 107.17)
 - open/close 8625 (time 1787033970.53 total 90.04 last 88.36)
 - open/close 9594 (time 1787033980.54 total 100.04 last 96.83)
total: 10000 open/close in 104.59 seconds: 95.61 ops/second
 - unlinked 0 (time 1787033988 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 710 (time 1787034087.04 total 10.02 last 70.88)
 - open/close 1486 (time 1787034097.04 total 20.02 last 77.60)
 - open/close 2338 (time 1787034107.06 total 30.04 last 85.01)
 - open/close 3135 (time 1787034117.06 total 40.04 last 79.69)
 - open/close 3982 (time 1787034127.07 total 50.05 last 84.64)
 - open/close 4799 (time 1787034137.10 total 60.08 last 81.48)
 - open/close 5619 (time 1787034147.10 total 70.08 last 81.99)
 - open/close 6451 (time 1787034157.11 total 80.09 last 83.08)
 - open/close 7202 (time 1787034167.13 total 90.10 last 75.00)
 - open/close 8039 (time 1787034177.13 total 100.11 last 83.68)
 - open/close 8934 (time 1787034187.13 total 110.11 last 89.46)
 - open/close 9681 (time 1787034197.14 total 120.12 last 74.67)
total: 10000 open/close in 123.99 seconds: 80.65 ops/second
 - unlinked 0 (time 1787034205 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 882 (time 1787034293.75 total 10.00 last 88.16)
 - open/close 1685 (time 1787034303.75 total 20.01 last 80.24)
 - open/close 2495 (time 1787034313.77 total 30.03 last 80.88)
 - open/close 3183 (time 1787034323.77 total 40.03 last 68.77)
 - open/close 3942 (time 1787034333.79 total 50.05 last 75.77)
 - open/close 4792 (time 1787034343.81 total 60.06 last 84.88)
 - open/close 5546 (time 1787034353.81 total 70.07 last 75.37)
 - open/close 6374 (time 1787034363.82 total 80.08 last 82.73)
 - open/close 7226 (time 1787034373.84 total 90.10 last 84.99)
 - open/close 8000 (time 1787034383.84 total 100.10 last 77.40)
 - open/close 8604 (time 1787034393.85 total 110.11 last 60.35)
 - open/close 9392 (time 1787034403.86 total 120.12 last 78.73)
total: 10000 open/close in 127.88 seconds: 78.20 ops/second
 - unlinked 0 (time 1787034415 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 877 (time 1787034517.08 total 10.01 last 87.60)
 - open/close 1654 (time 1787034527.08 total 20.01 last 77.69)
 - open/close 2396 (time 1787034537.09 total 30.02 last 74.14)
 - open/close 3133 (time 1787034547.09 total 40.02 last 73.69)
 - open/close 3930 (time 1787034557.10 total 50.03 last 79.65)
 - open/close 4663 (time 1787034567.12 total 60.05 last 73.14)
 - open/close 5392 (time 1787034577.13 total 70.06 last 72.85)
 - open/close 6185 (time 1787034587.14 total 80.06 last 79.25)
 - open/close 6921 (time 1787034597.14 total 90.06 last 73.59)
 - open/close 7714 (time 1787034607.15 total 100.08 last 79.17)
 - open/close 8619 (time 1787034617.16 total 110.08 last 90.46)
 - open/close 9401 (time 1787034627.16 total 120.08 last 78.20)
total: 10000 open/close in 127.90 seconds: 78.18 ops/second
 - unlinked 0 (time 1787034638 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg138-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg138-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.138@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg138-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all
pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1
Started lustre-OST0000
oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg138-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg138-client.virtnet:  -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg138-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg138-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.138@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.138@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.138@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.138@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg138-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all
pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1
Started lustre-OST0000
oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg138-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg138-client.virtnet:  -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg138-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg138-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server
stop mds service on oleg138-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server

unloading modules via unload_modules_local on: 'oleg138-server'
oleg138-server: oleg138-server.virtnet: executing unload_modules_local
oleg138-server: 
modules unloaded.
pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2
pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2
pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2
pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2