== conf-sanity test 69: replace an OST with the same index ========================================================== 02:03:24 (1768547004)
start mds service on oleg233-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg233-server'
oleg233-server: oleg233-server.virtnet: executing load_modules_local
oleg233-server: Loading modules from /home/green/git/lustre-release/lustre
oleg233-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg233-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg233-server: detected 4 online CPUs by sysfs
oleg233-server: MODOPTS_LIBCFS=
oleg233-server: Force libcfs to create 2 CPU partitions
oleg233-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg233-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg233-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1
Started lustre-OST0000
oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg233-client.virtnet:  -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11872 inodes available. Want 99872. rc=0
 - open/close 1139 (time 1768547072.72 total 10.01 last 113.84)
 - open/close 2292 (time 1768547082.72 total 20.01 last 115.29)
 - open/close 3547 (time 1768547092.73 total 30.01 last 125.48)
 - open/close 4696 (time 1768547102.74 total 40.02 last 114.74)
 - open/close 5957 (time 1768547112.75 total 50.03 last 126.02)
 - open/close 7239 (time 1768547122.75 total 60.03 last 128.15)
 - open/close 8288 (time 1768547132.76 total 70.04 last 104.85)
 - open/close 9483 (time 1768547142.76 total 80.04 last 119.45)
total: 10000 open/close in 84.55 seconds: 118.28 ops/second
 - unlinked 0 (time 1768547149 ; total 0 ; last 0)
total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second
 - open/close 1471 (time 1768547210.76 total 10.00 last 147.06)
 - open/close 2924 (time 1768547220.76 total 20.00 last 145.27)
 - open/close 4253 (time 1768547230.76 total 30.01 last 132.87)
 - open/close 5732 (time 1768547240.77 total 40.01 last 147.84)
 - open/close 7029 (time 1768547250.77 total 50.02 last 129.64)
 - open/close 8306 (time 1768547260.77 total 60.02 last 127.68)
 - open/close 9669 (time 1768547270.78 total 70.02 last 136.20)
total: 10000 open/close in 72.42 seconds: 138.08 ops/second
 - unlinked 0 (time 1768547275 ; total 0 ; last 0)
total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second
 - open/close 1385 (time 1768547332.90 total 10.01 last 138.40)
 - open/close 2796 (time 1768547342.91 total 20.01 last 141.07)
 - open/close 3933 (time 1768547352.91 total 30.01 last 113.68)
 - open/close 4898 (time 1768547362.91 total 40.01 last 96.48)
 - open/close 5817 (time 1768547372.93 total 50.04 last 91.66)
 - open/close 6335 (time 1768547382.95 total 60.06 last 51.70)
 - open/close 6945 (time 1768547393.11 total 70.21 last 60.09)
 - open/close 7451 (time 1768547403.11 total 80.21 last 50.58)
 - open/close 7857 (time 1768547413.15 total 90.25 last 40.45)
 - open/close 8312 (time 1768547423.18 total 100.28 last 45.35)
 - open/close 8573 (time 1768547433.22 total 110.33 last 25.99)
 - open/close 8946 (time 1768547443.23 total 120.33 last 37.28)
 - open/close 9373 (time 1768547453.26 total 130.37 last 42.55)
 - open/close 9941 (time 1768547463.30 total 140.41 last 56.57)
total: 10000 open/close in 142.02 seconds: 70.41 ops/second
 - unlinked 0 (time 1768547472 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 749 (time 1768547566.11 total 10.00 last 74.87)
 - open/close 1547 (time 1768547576.12 total 20.01 last 79.74)
 - open/close 2249 (time 1768547586.13 total 30.03 last 70.09)
 - open/close 2941 (time 1768547596.15 total 40.04 last 69.11)
 - open/close 3578 (time 1768547606.15 total 50.04 last 63.69)
 - open/close 4286 (time 1768547616.15 total 60.05 last 70.75)
 - open/close 5074 (time 1768547626.16 total 70.06 last 78.74)
 - open/close 5859 (time 1768547636.18 total 80.07 last 78.39)
 - open/close 6670 (time 1768547646.18 total 90.08 last 81.06)
 - open/close 7531 (time 1768547656.19 total 100.08 last 86.02)
 - open/close 8312 (time 1768547666.20 total 110.10 last 78.01)
 - open/close 9045 (time 1768547676.21 total 120.11 last 73.23)
 - open/close 9794 (time 1768547686.22 total 130.11 last 74.85)
total: 10000 open/close in 133.46 seconds: 74.93 ops/second
 - unlinked 0 (time 1768547694 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 645 (time 1768547793.06 total 10.01 last 64.44)
 - open/close 1342 (time 1768547803.06 total 20.01 last 69.70)
 - open/close 2068 (time 1768547813.07 total 30.02 last 72.52)
 - open/close 2777 (time 1768547823.08 total 40.03 last 70.82)
 - open/close 3439 (time 1768547833.09 total 50.04 last 66.14)
 - open/close 4063 (time 1768547843.11 total 60.06 last 62.28)
 - open/close 4827 (time 1768547853.12 total 70.07 last 76.37)
 - open/close 5592 (time 1768547863.13 total 80.08 last 76.39)
 - open/close 6292 (time 1768547873.15 total 90.10 last 69.86)
 - open/close 6937 (time 1768547883.18 total 100.13 last 64.33)
 - open/close 7698 (time 1768547893.18 total 110.13 last 76.07)
 - open/close 8361 (time 1768547903.18 total 120.13 last 66.30)
 - open/close 9021 (time 1768547913.21 total 130.16 last 65.82)
 - open/close 9752 (time 1768547923.22 total 140.16 last 73.07)
total: 10000 open/close in 143.40 seconds: 69.74 ops/second
 - unlinked 0 (time 1768547930 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 791 (time 1768548032.18 total 10.01 last 79.02)
 - open/close 1579 (time 1768548042.19 total 20.02 last 78.75)
 - open/close 2285 (time 1768548052.19 total 30.02 last 70.56)
 - open/close 3028 (time 1768548062.20 total 40.03 last 74.24)
 - open/close 3830 (time 1768548072.21 total 50.04 last 80.10)
 - open/close 4550 (time 1768548082.22 total 60.05 last 71.96)
 - open/close 5238 (time 1768548092.22 total 70.05 last 68.76)
 - open/close 6002 (time 1768548102.33 total 80.16 last 75.61)
 - open/close 6724 (time 1768548112.34 total 90.17 last 72.12)
 - open/close 7342 (time 1768548122.34 total 100.17 last 61.77)
 - open/close 8034 (time 1768548132.35 total 110.18 last 69.16)
 - open/close 8730 (time 1768548142.36 total 120.19 last 69.53)
 - open/close 9415 (time 1768548152.37 total 130.20 last 68.40)
total: 10000 open/close in 137.52 seconds: 72.72 ops/second
 - unlinked 0 (time 1768548164 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 515 (time 1768548262.45 total 10.01 last 51.45)
 - open/close 1256 (time 1768548272.46 total 20.01 last 74.08)
 - open/close 1876 (time 1768548282.46 total 30.02 last 61.97)
 - open/close 2423 (time 1768548292.47 total 40.03 last 54.64)
 - open/close 2952 (time 1768548302.47 total 50.03 last 52.89)
 - open/close 3817 (time 1768548312.49 total 60.04 last 86.39)
 - open/close 4633 (time 1768548322.49 total 70.04 last 81.59)
 - open/close 5534 (time 1768548332.50 total 80.05 last 90.03)
 - open/close 6310 (time 1768548342.50 total 90.05 last 77.58)
 - open/close 7111 (time 1768548352.51 total 100.06 last 80.03)
 - open/close 7930 (time 1768548362.52 total 110.08 last 81.77)
 - open/close 8685 (time 1768548372.54 total 120.10 last 75.35)
 - open/close 9443 (time 1768548382.55 total 130.11 last 75.74)
total: 10000 open/close in 137.89 seconds: 72.52 ops/second
 - unlinked 0 (time 1768548397 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 909 (time 1768548493.16 total 10.01 last 90.85)
 - open/close 1757 (time 1768548503.18 total 20.03 last 84.62)
 - open/close 2628 (time 1768548513.19 total 30.03 last 87.04)
 - open/close 3483 (time 1768548523.20 total 40.04 last 85.44)
 - open/close 4220 (time 1768548533.21 total 50.05 last 73.62)
 - open/close 4898 (time 1768548543.24 total 60.08 last 67.60)
 - open/close 5421 (time 1768548553.24 total 70.09 last 52.27)
 - open/close 5977 (time 1768548563.27 total 80.12 last 55.43)
 - open/close 6560 (time 1768548573.28 total 90.13 last 58.24)
 - open/close 7319 (time 1768548583.31 total 100.15 last 75.74)
 - open/close 8036 (time 1768548593.32 total 110.16 last 71.59)
 - open/close 8745 (time 1768548603.33 total 120.17 last 70.82)
 - open/close 9572 (time 1768548613.34 total 130.18 last 82.62)
total: 10000 open/close in 135.15 seconds: 73.99 ops/second
 - unlinked 0 (time 1768548622 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 775 (time 1768548725.90 total 10.01 last 77.40)
 - open/close 1528 (time 1768548735.91 total 20.02 last 75.27)
 - open/close 2252 (time 1768548745.91 total 30.02 last 72.37)
 - open/close 3071 (time 1768548755.91 total 40.02 last 81.89)
 - open/close 3916 (time 1768548765.91 total 50.03 last 84.47)
 - open/close 4774 (time 1768548775.92 total 60.03 last 85.77)
 - open/close 5649 (time 1768548785.92 total 70.03 last 87.48)
 - open/close 6583 (time 1768548795.92 total 80.03 last 93.38)
 - open/close 7488 (time 1768548805.93 total 90.04 last 90.43)
 - open/close 8266 (time 1768548815.94 total 100.05 last 77.70)
 - open/close 9064 (time 1768548825.94 total 110.05 last 79.80)
 - open/close 9872 (time 1768548835.95 total 120.07 last 80.70)
total: 10000 open/close in 121.43 seconds: 82.35 ops/second
 - unlinked 0 (time 1768548841 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 776 (time 1768548928.17 total 10.03 last 77.36)
 - open/close 1554 (time 1768548938.17 total 20.04 last 77.74)
 - open/close 2302 (time 1768548948.18 total 30.04 last 74.79)
 - open/close 3121 (time 1768548958.18 total 40.05 last 81.85)
 - open/close 3844 (time 1768548968.19 total 50.06 last 72.23)
 - open/close 4554 (time 1768548978.20 total 60.07 last 70.94)
 - open/close 5195 (time 1768548988.21 total 70.07 last 64.04)
 - open/close 5982 (time 1768548998.22 total 80.08 last 78.65)
 - open/close 6788 (time 1768549008.23 total 90.10 last 80.47)
 - open/close 7542 (time 1768549018.25 total 100.11 last 75.28)
 - open/close 8372 (time 1768549028.26 total 110.12 last 82.93)
 - open/close 9198 (time 1768549038.26 total 120.12 last 82.59)
total: 10000 open/close in 129.82 seconds: 77.03 ops/second
 - unlinked 0 (time 1768549052 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg233-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg233-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.133@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.133@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg233-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1
Started lustre-OST0000
oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg233-server: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg233-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg233-client.virtnet:  -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre
On OST0, 20325 used inodes rc=0
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg233-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg233-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.133@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.133@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.133@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.133@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.133@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg233-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1
Started lustre-OST0000
oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg233-server: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg233-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg233-client.virtnet:  -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg233-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg233-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server
stop mds service on oleg233-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg233-server'
oleg233-server: oleg233-server.virtnet: executing unload_modules_local
oleg233-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg233-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2
pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2