== conf-sanity test 69: replace an OST with the same index ========================================================== 19:07:51 (1763510871)
start mds service on oleg201-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg201-server'
oleg201-server: oleg201-server.virtnet: executing load_modules_local
oleg201-server: Loading modules from /home/green/git/lustre-release/lustre
oleg201-server: detected 4 online CPUs by sysfs
oleg201-server: MODOPTS_LIBCFS=
oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
oleg201-server: Force libcfs to create 2 CPU partitions
oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all
pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg201-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all
pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1
Started lustre-OST0000
oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg201-client.virtnet:  -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11872 inodes available. Want 99872. rc=0
 - open/close 616 (time 1763510944.97 total 10.02 last 61.51)
 - open/close 1278 (time 1763510954.97 total 20.02 last 66.19)
 - open/close 2012 (time 1763510964.98 total 30.03 last 73.32)
 - open/close 2754 (time 1763510974.99 total 40.04 last 74.11)
 - open/close 3469 (time 1763510985.00 total 50.05 last 71.44)
 - open/close 4166 (time 1763510995.01 total 60.06 last 69.61)
 - open/close 4901 (time 1763511005.02 total 70.06 last 73.47)
 - open/close 5608 (time 1763511015.03 total 80.08 last 70.62)
 - open/close 6310 (time 1763511025.03 total 90.08 last 70.18)
 - open/close 6993 (time 1763511035.03 total 100.08 last 68.28)
 - open/close 7677 (time 1763511045.04 total 110.08 last 68.39)
 - open/close 8317 (time 1763511055.04 total 120.09 last 63.97)
 - open/close 9026 (time 1763511065.05 total 130.10 last 70.82)
 - open/close 9734 (time 1763511075.05 total 140.10 last 70.80)
total: 10000 open/close in 143.80 seconds: 69.54 ops/second
 - unlinked 0 (time 1763511081 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 751 (time 1763511177.15 total 10.01 last 75.05)
 - open/close 1481 (time 1763511187.16 total 20.02 last 72.92)
 - open/close 2217 (time 1763511197.16 total 30.02 last 73.59)
 - open/close 2903 (time 1763511207.18 total 40.03 last 68.49)
 - open/close 3571 (time 1763511217.18 total 50.03 last 66.80)
 - open/close 4297 (time 1763511227.19 total 60.04 last 72.56)
 - open/close 4952 (time 1763511237.19 total 70.04 last 65.50)
 - open/close 5605 (time 1763511247.19 total 80.04 last 65.28)
 - open/close 6302 (time 1763511257.22 total 90.08 last 69.48)
 - open/close 6978 (time 1763511267.23 total 100.09 last 67.52)
 - open/close 7654 (time 1763511277.25 total 110.10 last 67.51)
 - open/close 8338 (time 1763511287.26 total 120.11 last 68.33)
 - open/close 9088 (time 1763511297.26 total 130.11 last 75.00)
 - open/close 9791 (time 1763511307.27 total 140.12 last 70.20)
total: 10000 open/close in 143.45 seconds: 69.71 ops/second
 - unlinked 0 (time 1763511314 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 785 (time 1763511405.97 total 10.01 last 78.43)
 - open/close 1475 (time 1763511415.99 total 20.03 last 68.88)
 - open/close 2212 (time 1763511426.00 total 30.03 last 73.66)
 - open/close 2891 (time 1763511436.00 total 40.04 last 67.85)
 - open/close 3626 (time 1763511446.01 total 50.05 last 73.44)
 - open/close 4327 (time 1763511456.01 total 60.05 last 70.09)
 - open/close 5049 (time 1763511466.01 total 70.05 last 72.19)
 - open/close 5769 (time 1763511476.02 total 80.05 last 71.96)
 - open/close 6465 (time 1763511486.03 total 90.06 last 69.56)
 - open/close 7164 (time 1763511496.03 total 100.06 last 69.88)
 - open/close 7815 (time 1763511506.04 total 110.07 last 65.03)
 - open/close 8471 (time 1763511516.06 total 120.09 last 65.48)
 - open/close 9200 (time 1763511526.06 total 130.10 last 72.86)
 - open/close 9920 (time 1763511536.07 total 140.10 last 71.98)
total: 10000 open/close in 140.96 seconds: 70.94 ops/second
 - unlinked 0 (time 1763511540 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 676 (time 1763511642.09 total 10.00 last 67.58)
 - open/close 1314 (time 1763511652.09 total 20.01 last 63.77)
 - open/close 2064 (time 1763511662.11 total 30.02 last 74.88)
 - open/close 2758 (time 1763511672.11 total 40.03 last 69.37)
 - open/close 3490 (time 1763511682.12 total 50.04 last 73.14)
 - open/close 4170 (time 1763511692.13 total 60.04 last 67.97)
 - open/close 4906 (time 1763511702.14 total 70.06 last 73.49)
 - open/close 5590 (time 1763511712.15 total 80.07 last 68.35)
 - open/close 6333 (time 1763511722.15 total 90.07 last 74.29)
 - open/close 7081 (time 1763511732.15 total 100.07 last 74.79)
 - open/close 7756 (time 1763511742.15 total 110.07 last 67.49)
 - open/close 8440 (time 1763511752.16 total 120.07 last 68.38)
 - open/close 9149 (time 1763511762.16 total 130.07 last 70.90)
 - open/close 9867 (time 1763511772.16 total 140.08 last 71.75)
total: 10000 open/close in 141.92 seconds: 70.46 ops/second
 - unlinked 0 (time 1763511778 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 690 (time 1763511879.43 total 10.01 last 68.94)
 - open/close 1345 (time 1763511889.44 total 20.02 last 65.44)
 - open/close 2040 (time 1763511899.44 total 30.02 last 69.46)
 - open/close 2733 (time 1763511909.45 total 40.03 last 69.24)
 - open/close 3426 (time 1763511919.46 total 50.04 last 69.23)
 - open/close 4157 (time 1763511929.47 total 60.05 last 73.07)
 - open/close 4835 (time 1763511939.49 total 70.07 last 67.62)
 - open/close 5518 (time 1763511949.50 total 80.08 last 68.26)
 - open/close 6160 (time 1763511959.52 total 90.10 last 64.04)
 - open/close 6783 (time 1763511969.53 total 100.11 last 62.29)
 - open/close 7449 (time 1763511979.54 total 110.12 last 66.53)
 - open/close 8114 (time 1763511989.55 total 120.13 last 66.41)
 - open/close 8804 (time 1763511999.55 total 130.13 last 69.00)
 - open/close 9440 (time 1763512009.57 total 140.15 last 63.50)
total: 10000 open/close in 149.28 seconds: 66.99 ops/second
 - unlinked 0 (time 1763512022 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 534 (time 1763512135.09 total 10.02 last 53.31)
 - open/close 1072 (time 1763512145.10 total 20.02 last 53.78)
 - open/close 1652 (time 1763512155.10 total 30.02 last 57.99)
 - open/close 2278 (time 1763512165.11 total 40.03 last 62.54)
 - open/close 2850 (time 1763512175.12 total 50.05 last 57.10)
 - open/close 3409 (time 1763512185.14 total 60.06 last 55.82)
 - open/close 4000 (time 1763512195.17 total 70.09 last 58.92)
 - open/close 4584 (time 1763512205.17 total 80.10 last 58.37)
 - open/close 5128 (time 1763512215.20 total 90.12 last 54.26)
 - open/close 5696 (time 1763512225.23 total 100.15 last 56.65)
 - open/close 6317 (time 1763512235.24 total 110.17 last 62.00)
 - open/close 6965 (time 1763512245.25 total 120.18 last 64.74)
 - open/close 7657 (time 1763512255.26 total 130.18 last 69.16)
 - open/close 8257 (time 1763512265.27 total 140.19 last 59.93)
 - open/close 8838 (time 1763512275.28 total 150.21 last 58.01)
 - open/close 9408 (time 1763512285.32 total 160.24 last 56.81)
total: 10000 open/close in 170.14 seconds: 58.78 ops/second
 - unlinked 0 (time 1763512299 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 588 (time 1763512406.91 total 10.00 last 58.79)
 - open/close 1185 (time 1763512416.92 total 20.01 last 59.66)
 - open/close 1800 (time 1763512426.93 total 30.02 last 61.44)
 - open/close 2437 (time 1763512436.93 total 40.02 last 63.68)
 - open/close 3070 (time 1763512446.94 total 50.03 last 63.23)
 - open/close 3726 (time 1763512456.95 total 60.03 last 65.58)
 - open/close 4379 (time 1763512466.96 total 70.05 last 65.19)
 - open/close 5034 (time 1763512476.97 total 80.06 last 65.48)
 - open/close 5717 (time 1763512486.99 total 90.07 last 68.17)
 - open/close 6405 (time 1763512496.99 total 100.08 last 68.77)
 - open/close 7097 (time 1763512507.02 total 110.11 last 68.99)
 - open/close 7783 (time 1763512517.02 total 120.11 last 68.58)
 - open/close 8452 (time 1763512527.04 total 130.13 last 66.77)
 - open/close 9104 (time 1763512537.06 total 140.15 last 65.08)
 - open/close 9766 (time 1763512547.07 total 150.16 last 66.16)
total: 10000 open/close in 153.60 seconds: 65.10 ops/second
 - unlinked 0 (time 1763512554 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 627 (time 1763512660.72 total 10.00 last 62.69)
 - open/close 1231 (time 1763512670.73 total 20.02 last 60.29)
 - open/close 1873 (time 1763512680.74 total 30.03 last 64.13)
 - open/close 2496 (time 1763512690.75 total 40.03 last 62.28)
 - open/close 3190 (time 1763512700.75 total 50.04 last 69.39)
 - open/close 3940 (time 1763512710.76 total 60.05 last 74.91)
 - open/close 4695 (time 1763512720.77 total 70.05 last 75.44)
 - open/close 5414 (time 1763512730.77 total 80.06 last 71.87)
 - open/close 6112 (time 1763512740.78 total 90.07 last 69.72)
 - open/close 6840 (time 1763512750.80 total 100.08 last 72.72)
 - open/close 7567 (time 1763512760.80 total 110.08 last 72.68)
 - open/close 8256 (time 1763512770.81 total 120.09 last 68.84)
 - open/close 8976 (time 1763512780.81 total 130.10 last 71.95)
 - open/close 9673 (time 1763512790.83 total 140.11 last 69.61)
total: 10000 open/close in 144.65 seconds: 69.13 ops/second
 - unlinked 0 (time 1763512799 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 732 (time 1763512895.25 total 10.01 last 73.16)
 - open/close 1456 (time 1763512905.25 total 20.01 last 72.39)
 - open/close 2217 (time 1763512915.26 total 30.02 last 76.02)
 - open/close 2947 (time 1763512925.28 total 40.03 last 72.88)
 - open/close 3653 (time 1763512935.29 total 50.05 last 70.51)
 - open/close 4368 (time 1763512945.30 total 60.06 last 71.43)
 - open/close 4989 (time 1763512955.31 total 70.07 last 62.05)
 - open/close 5703 (time 1763512965.32 total 80.07 last 71.34)
 - open/close 6446 (time 1763512975.33 total 90.08 last 74.25)
 - open/close 7207 (time 1763512985.33 total 100.08 last 76.07)
 - open/close 7960 (time 1763512995.34 total 110.09 last 75.26)
 - open/close 8688 (time 1763513005.34 total 120.10 last 72.76)
 - open/close 9406 (time 1763513015.35 total 130.10 last 71.77)
total: 10000 open/close in 138.96 seconds: 71.96 ops/second
 - unlinked 0 (time 1763513028 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 665 (time 1763513122.73 total 10.00 last 66.49)
 - open/close 1349 (time 1763513132.74 total 20.01 last 68.34)
 - open/close 1994 (time 1763513142.75 total 30.02 last 64.44)
 - open/close 2709 (time 1763513152.75 total 40.02 last 71.47)
 - open/close 3420 (time 1763513162.76 total 50.04 last 71.00)
 - open/close 4130 (time 1763513172.77 total 60.04 last 70.96)
 - open/close 4814 (time 1763513182.78 total 70.05 last 68.34)
 - open/close 5527 (time 1763513192.78 total 80.05 last 71.29)
 - open/close 6239 (time 1763513202.79 total 90.07 last 71.10)
 - open/close 6876 (time 1763513212.80 total 100.07 last 63.67)
 - open/close 7593 (time 1763513222.80 total 110.08 last 71.68)
 - open/close 8251 (time 1763513232.81 total 120.09 last 65.71)
 - open/close 8927 (time 1763513242.82 total 130.10 last 67.55)
 - open/close 9642 (time 1763513252.82 total 140.10 last 71.49)
total: 10000 open/close in 145.07 seconds: 68.93 ops/second
 - unlinked 0 (time 1763513261 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg201-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg201-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.101@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.101@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg201-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all
pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1
Started lustre-OST0000
oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg201-client.virtnet:  -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre
On OST0, 20325 used inodes rc=0
pdsh@oleg201-client: oleg201-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg201-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg201-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.101@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.101@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.101@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.101@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.101@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg201-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all
pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1
Started lustre-OST0000
oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg201-client.virtnet:  -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg201-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg201-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server
stop mds service on oleg201-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg201-server'
oleg201-server: oleg201-server.virtnet: executing unload_modules_local
oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2
pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2
pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2
pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2