== conf-sanity test 69: replace an OST with the same index ========================================================== 18:46:13 (1763336773) start mds service on oleg447-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg447-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg447-server: detected 4 online CPUs by sysfs oleg447-server: MODOPTS_LIBCFS= oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 711 (time 1763336847.43 total 10.01 last 71.02) - open/close 1453 (time 1763336857.43 total 20.01 last 74.19) - open/close 2141 (time 1763336867.44 total 30.03 last 68.71) - open/close 2864 (time 1763336877.45 total 40.04 last 72.22) - open/close 3601 (time 1763336887.46 total 50.05 last 73.63) - open/close 4192 (time 1763336897.47 total 60.05 last 59.09) - open/close 4882 (time 1763336907.47 total 70.06 last 68.95) - open/close 5578 (time 1763336917.49 total 80.07 last 69.50) - open/close 6245 (time 1763336927.50 total 90.08 last 66.64) - open/close 6895 (time 1763336937.51 total 100.09 last 64.92) - open/close 7581 (time 1763336947.51 total 110.10 last 68.56) - open/close 8272 (time 1763336957.53 total 120.12 last 68.97) - open/close 8849 (time 1763336967.55 total 130.13 last 57.63) - open/close 9479 (time 1763336977.55 total 140.14 last 62.95) total: 10000 open/close in 147.42 seconds: 67.84 ops/second - unlinked 0 (time 1763336989 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 678 (time 1763337088.02 total 10.00 last 67.80) - open/close 1397 (time 1763337098.02 total 20.00 last 71.89) - open/close 2096 (time 1763337108.03 total 30.00 last 69.88) - open/close 2803 (time 1763337118.03 total 40.00 last 70.70) - open/close 3503 (time 1763337128.03 total 50.01 last 69.99) - open/close 4182 (time 1763337138.03 total 60.01 last 67.86) - open/close 4872 (time 1763337148.06 total 70.03 last 68.84) - open/close 5517 (time 1763337158.06 total 80.04 last 64.49) - open/close 6188 (time 1763337168.07 total 90.04 last 67.06) - open/close 6859 (time 1763337178.07 total 100.05 last 67.08) - open/close 7547 (time 1763337188.08 total 110.06 last 68.72) - open/close 8238 (time 1763337198.08 total 120.06 last 69.08) - open/close 8965 (time 1763337208.09 total 130.07 last 72.66) - open/close 9687 (time 1763337218.11 total 140.09 last 72.06) total: 10000 open/close in 144.68 seconds: 69.12 ops/second - unlinked 0 (time 1763337226 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 704 (time 1763337321.97 total 10.01 last 70.31) - open/close 1345 (time 1763337331.97 total 20.02 last 64.06) - open/close 2038 (time 1763337341.98 total 30.03 last 69.24) - open/close 2670 (time 1763337351.99 total 40.04 last 63.13) - open/close 3384 (time 1763337361.99 total 50.04 last 71.39) - open/close 4112 (time 1763337372.01 total 60.05 last 72.70) - open/close 4843 (time 1763337382.01 total 70.06 last 73.05) - open/close 5580 (time 1763337392.04 total 80.08 last 73.52) - open/close 6283 (time 1763337402.05 total 90.09 last 70.24) - open/close 6979 (time 1763337412.05 total 100.10 last 69.57) - open/close 7683 (time 1763337422.06 total 110.10 last 70.37) - open/close 8442 (time 1763337432.06 total 120.11 last 75.86) - open/close 9042 (time 1763337442.07 total 130.12 last 59.95) - open/close 9687 (time 1763337452.07 total 140.12 last 64.49) total: 10000 open/close in 144.76 seconds: 69.08 ops/second - unlinked 0 (time 1763337460 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 698 (time 1763337558.22 total 10.01 last 69.73) - open/close 1429 (time 1763337568.23 total 20.02 last 73.02) - open/close 2160 (time 1763337578.24 total 30.03 last 73.00) - open/close 2867 (time 1763337588.25 total 40.04 last 70.67) - open/close 3589 (time 1763337598.26 total 50.05 last 72.11) - open/close 4252 (time 1763337608.26 total 60.05 last 66.28) - open/close 4909 (time 1763337618.26 total 70.05 last 65.69) - open/close 5613 (time 1763337628.26 total 80.05 last 70.40) - open/close 6295 (time 1763337638.26 total 90.06 last 68.20) - open/close 7013 (time 1763337648.27 total 100.06 last 71.75) - open/close 7727 (time 1763337658.28 total 110.07 last 71.34) - open/close 8420 (time 1763337668.29 total 120.08 last 69.24) - open/close 9092 (time 1763337678.30 total 130.10 last 67.08) - open/close 9795 (time 1763337688.31 total 140.10 last 70.28) total: 10000 open/close in 142.97 seconds: 69.94 ops/second - unlinked 0 (time 1763337695 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 705 (time 1763337791.47 total 10.01 last 70.41) - open/close 1414 (time 1763337801.47 total 20.02 last 70.87) - open/close 2077 (time 1763337811.48 total 30.02 last 66.29) - open/close 2783 (time 1763337821.49 total 40.03 last 70.53) - open/close 3425 (time 1763337831.49 total 50.04 last 64.14) - open/close 4096 (time 1763337841.50 total 60.04 last 67.09) - open/close 4808 (time 1763337851.50 total 70.05 last 71.15) - open/close 5498 (time 1763337861.51 total 80.05 last 68.95) - open/close 6220 (time 1763337871.52 total 90.06 last 72.12) - open/close 6899 (time 1763337881.53 total 100.07 last 67.85) - open/close 7592 (time 1763337891.53 total 110.08 last 69.26) - open/close 8270 (time 1763337901.55 total 120.09 last 67.70) - open/close 9030 (time 1763337911.56 total 130.10 last 75.93) - open/close 9737 (time 1763337921.56 total 140.10 last 70.69) total: 10000 open/close in 143.98 seconds: 69.45 ops/second - unlinked 0 (time 1763337929 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 659 (time 1763338029.20 total 10.01 last 65.84) - open/close 1295 (time 1763338039.20 total 20.01 last 63.57) - open/close 1978 (time 1763338049.20 total 30.02 last 68.30) - open/close 2644 (time 1763338059.21 total 40.02 last 66.59) - open/close 3347 (time 1763338069.22 total 50.03 last 70.18) - open/close 3976 (time 1763338079.23 total 60.04 last 62.84) - open/close 4685 (time 1763338089.24 total 70.05 last 70.85) - open/close 5382 (time 1763338099.24 total 80.05 last 69.69) - open/close 6039 (time 1763338109.25 total 90.06 last 65.66) - open/close 6817 (time 1763338119.28 total 100.09 last 77.55) - open/close 7519 (time 1763338129.29 total 110.10 last 70.14) - open/close 8251 (time 1763338139.30 total 120.11 last 73.13) - open/close 8894 (time 1763338149.30 total 130.11 last 64.29) - open/close 9532 (time 1763338159.30 total 140.11 last 63.80) total: 10000 open/close in 146.45 seconds: 68.28 ops/second - unlinked 0 (time 1763338169 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 723 (time 1763338264.12 total 10.01 last 72.22) - open/close 1401 (time 1763338274.13 total 20.02 last 67.76) - open/close 2161 (time 1763338284.14 total 30.03 last 75.93) - open/close 2880 (time 1763338294.14 total 40.03 last 71.90) - open/close 3608 (time 1763338304.15 total 50.04 last 72.69) - open/close 4326 (time 1763338314.15 total 60.04 last 71.80) - open/close 5041 (time 1763338324.16 total 70.05 last 71.45) - open/close 5752 (time 1763338334.17 total 80.06 last 71.04) - open/close 6427 (time 1763338344.18 total 90.07 last 67.42) - open/close 7026 (time 1763338354.18 total 100.07 last 59.90) - open/close 7657 (time 1763338364.20 total 110.09 last 62.97) - open/close 8318 (time 1763338374.21 total 120.10 last 66.06) - open/close 8952 (time 1763338384.21 total 130.10 last 63.35) - open/close 9561 (time 1763338394.22 total 140.11 last 60.84) total: 10000 open/close in 146.64 seconds: 68.20 ops/second - unlinked 0 (time 1763338405 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 632 (time 1763338510.10 total 10.01 last 63.16) - open/close 1267 (time 1763338520.11 total 20.02 last 63.43) - open/close 1976 (time 1763338530.12 total 30.03 last 70.80) - open/close 2673 (time 1763338540.13 total 40.04 last 69.66) - open/close 3340 (time 1763338550.14 total 50.05 last 66.64) - open/close 4055 (time 1763338560.14 total 60.05 last 71.45) - open/close 4750 (time 1763338570.15 total 70.06 last 69.46) - open/close 5437 (time 1763338580.16 total 80.07 last 68.63) - open/close 6184 (time 1763338590.16 total 90.07 last 74.70) - open/close 6813 (time 1763338600.16 total 100.07 last 62.90) - open/close 7444 (time 1763338610.17 total 110.08 last 63.06) - open/close 8148 (time 1763338620.17 total 120.09 last 70.34) - open/close 8816 (time 1763338630.18 total 130.09 last 66.74) - open/close 9516 (time 1763338640.19 total 140.10 last 69.94) total: 10000 open/close in 146.38 seconds: 68.32 ops/second - unlinked 0 (time 1763338650 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 589 (time 1763338752.90 total 10.00 last 58.89) - open/close 1191 (time 1763338762.92 total 20.01 last 60.12) - open/close 1785 (time 1763338772.92 total 30.02 last 59.36) - open/close 2423 (time 1763338782.93 total 40.02 last 63.78) - open/close 3074 (time 1763338792.96 total 50.05 last 64.90) - open/close 3656 (time 1763338802.96 total 60.06 last 58.18) - open/close 4286 (time 1763338812.97 total 70.07 last 62.91) - open/close 4900 (time 1763338822.98 total 80.07 last 61.39) - open/close 5497 (time 1763338832.98 total 90.08 last 59.68) - open/close 6120 (time 1763338842.98 total 100.08 last 62.30) - open/close 6763 (time 1763338852.98 total 110.08 last 64.28) - open/close 7433 (time 1763338862.98 total 120.08 last 66.99) - open/close 8078 (time 1763338872.99 total 130.09 last 64.45) - open/close 8713 (time 1763338883.00 total 140.10 last 63.44) - open/close 9369 (time 1763338893.02 total 150.11 last 65.51) total: 10000 open/close in 159.90 seconds: 62.54 ops/second - unlinked 0 (time 1763338907 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 727 (time 1763339010.91 total 10.00 last 72.69) - open/close 1465 (time 1763339020.91 total 20.01 last 73.76) - open/close 2191 (time 1763339030.92 total 30.02 last 72.51) - open/close 2900 (time 1763339040.93 total 40.03 last 70.84) - open/close 3639 (time 1763339050.94 total 50.04 last 73.84) - open/close 4366 (time 1763339060.96 total 60.05 last 72.58) - open/close 5095 (time 1763339070.96 total 70.06 last 72.88) - open/close 5856 (time 1763339080.97 total 80.07 last 76.02) - open/close 6667 (time 1763339090.97 total 90.07 last 81.08) - open/close 7411 (time 1763339100.98 total 100.08 last 74.34) - open/close 8175 (time 1763339110.98 total 110.08 last 76.39) - open/close 8863 (time 1763339120.99 total 120.09 last 68.72) - open/close 9610 (time 1763339131.01 total 130.10 last 74.59) total: 10000 open/close in 135.24 seconds: 73.94 ops/second - unlinked 0 (time 1763339139 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.147@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg447-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre On OST0, 14885 used inodes rc=0 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.147@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.147@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.147@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.204.147@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg447-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local oleg447-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg447-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2