== conf-sanity test 69: replace an OST with the same index ========================================================== 01:10:02 (1769753402) start mds service on oleg402-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg402-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 535 (time 1769753493.99 total 10.01 last 53.46) - open/close 999 (time 1769753504.00 total 20.02 last 46.35) - open/close 1475 (time 1769753514.00 total 30.02 last 47.59) - open/close 2128 (time 1769753524.01 total 40.03 last 65.22) - open/close 2723 (time 1769753534.01 total 50.03 last 59.50) - open/close 3413 (time 1769753544.01 total 60.03 last 68.99) - open/close 4082 (time 1769753554.04 total 70.06 last 66.75) - open/close 4649 (time 1769753564.05 total 80.07 last 56.65) - open/close 5117 (time 1769753574.05 total 90.07 last 46.76) - open/close 5594 (time 1769753584.06 total 100.08 last 47.69) - open/close 6195 (time 1769753594.07 total 110.09 last 60.03) - open/close 6765 (time 1769753604.07 total 120.09 last 56.98) - open/close 7294 (time 1769753614.08 total 130.10 last 52.85) - open/close 7803 (time 1769753624.09 total 140.11 last 50.83) - open/close 8503 (time 1769753634.10 total 150.12 last 69.99) - open/close 9112 (time 1769753644.11 total 160.13 last 60.82) - open/close 9796 (time 1769753654.13 total 170.15 last 68.26) total: 10000 open/close in 173.68 seconds: 57.58 ops/second - unlinked 0 (time 1769753663 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 755 (time 1769753767.84 total 10.00 last 75.48) - open/close 1404 (time 1769753777.85 total 20.01 last 64.89) - open/close 2070 (time 1769753787.86 total 30.02 last 66.53) - open/close 2880 (time 1769753797.87 total 40.03 last 80.92) - open/close 3602 (time 1769753807.87 total 50.03 last 72.16) - open/close 4326 (time 1769753817.87 total 60.03 last 72.37) - open/close 5095 (time 1769753827.90 total 70.06 last 76.69) - open/close 5707 (time 1769753837.92 total 80.08 last 61.12) - open/close 6351 (time 1769753847.93 total 90.09 last 64.34) - open/close 7034 (time 1769753857.93 total 100.09 last 68.30) - open/close 7686 (time 1769753867.94 total 110.10 last 65.13) - open/close 8380 (time 1769753877.95 total 120.11 last 69.33) - open/close 9064 (time 1769753887.96 total 130.12 last 68.31) - open/close 9864 (time 1769753897.96 total 140.12 last 79.96) total: 10000 open/close in 141.81 seconds: 70.51 ops/second - unlinked 0 (time 1769753904 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 615 (time 1769754008.55 total 10.00 last 61.48) - open/close 1237 (time 1769754018.55 total 20.00 last 62.20) - open/close 1929 (time 1769754028.55 total 30.00 last 69.19) - open/close 2545 (time 1769754038.55 total 40.01 last 61.57) - open/close 3217 (time 1769754048.56 total 50.01 last 67.17) - open/close 3767 (time 1769754058.57 total 60.03 last 54.93) - open/close 4439 (time 1769754068.57 total 70.03 last 67.16) - open/close 4990 (time 1769754078.58 total 80.04 last 55.05) - open/close 5598 (time 1769754088.59 total 90.04 last 60.79) - open/close 6056 (time 1769754098.60 total 100.05 last 45.75) - open/close 6583 (time 1769754108.62 total 110.08 last 52.56) - open/close 7245 (time 1769754118.63 total 120.09 last 66.15) - open/close 7976 (time 1769754128.65 total 130.11 last 72.97) - open/close 8585 (time 1769754138.66 total 140.11 last 60.86) - open/close 9264 (time 1769754148.67 total 150.13 last 67.79) - open/close 9934 (time 1769754158.67 total 160.13 last 67.00) total: 10000 open/close in 161.30 seconds: 62.00 ops/second - unlinked 0 (time 1769754163 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 732 (time 1769754264.31 total 10.01 last 73.10) - open/close 1336 (time 1769754274.32 total 20.03 last 60.33) - open/close 1987 (time 1769754284.32 total 30.03 last 65.07) - open/close 2720 (time 1769754294.33 total 40.03 last 73.27) - open/close 3346 (time 1769754304.33 total 50.04 last 62.58) - open/close 3975 (time 1769754314.33 total 60.04 last 62.87) - open/close 4652 (time 1769754324.34 total 70.05 last 67.68) - open/close 5315 (time 1769754334.36 total 80.07 last 66.17) - open/close 5946 (time 1769754344.36 total 90.07 last 63.09) - open/close 6606 (time 1769754354.37 total 100.07 last 65.95) - open/close 7255 (time 1769754364.37 total 110.07 last 64.90) - open/close 7931 (time 1769754374.37 total 120.08 last 67.60) - open/close 8696 (time 1769754384.38 total 130.09 last 76.37) - open/close 9352 (time 1769754394.40 total 140.11 last 65.52) total: 10000 open/close in 150.10 seconds: 66.62 ops/second - unlinked 0 (time 1769754409 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 585 (time 1769754518.69 total 10.01 last 58.46) - open/close 1246 (time 1769754528.72 total 20.04 last 65.90) - open/close 1846 (time 1769754538.72 total 30.04 last 59.97) - open/close 2532 (time 1769754548.72 total 40.04 last 68.59) - open/close 3098 (time 1769754558.73 total 50.05 last 56.55) - open/close 3631 (time 1769754568.75 total 60.07 last 53.23) - open/close 4222 (time 1769754578.77 total 70.09 last 58.98) - open/close 4764 (time 1769754588.77 total 80.09 last 54.20) - open/close 5235 (time 1769754598.78 total 90.10 last 47.03) - open/close 5974 (time 1769754608.80 total 100.12 last 73.79) - open/close 6596 (time 1769754618.80 total 110.12 last 62.19) - open/close 7204 (time 1769754628.81 total 120.13 last 60.70) - open/close 7829 (time 1769754638.83 total 130.14 last 62.43) - open/close 8534 (time 1769754648.84 total 140.16 last 70.42) - open/close 9175 (time 1769754658.84 total 150.16 last 64.05) - open/close 9879 (time 1769754668.85 total 160.17 last 70.36) total: 10000 open/close in 161.71 seconds: 61.84 ops/second - unlinked 0 (time 1769754674 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 693 (time 1769754781.43 total 10.01 last 69.20) - open/close 1273 (time 1769754791.44 total 20.02 last 57.96) - open/close 1928 (time 1769754801.44 total 30.02 last 65.49) - open/close 2574 (time 1769754811.45 total 40.03 last 64.54) - open/close 3251 (time 1769754821.46 total 50.04 last 67.64) - open/close 3897 (time 1769754831.46 total 60.04 last 64.59) - open/close 4574 (time 1769754841.47 total 70.05 last 67.64) - open/close 5261 (time 1769754851.48 total 80.06 last 68.63) - open/close 5909 (time 1769754861.49 total 90.07 last 64.77) - open/close 6538 (time 1769754871.49 total 100.07 last 62.85) - open/close 7192 (time 1769754881.50 total 110.09 last 65.33) - open/close 7909 (time 1769754891.51 total 120.09 last 71.65) - open/close 8667 (time 1769754901.52 total 130.10 last 75.76) - open/close 9288 (time 1769754911.53 total 140.11 last 62.04) - open/close 9938 (time 1769754921.55 total 150.13 last 64.82) total: 10000 open/close in 151.23 seconds: 66.12 ops/second - unlinked 0 (time 1769754927 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 661 (time 1769755031.25 total 10.01 last 66.04) - open/close 1330 (time 1769755041.25 total 20.01 last 66.87) - open/close 2009 (time 1769755051.26 total 30.02 last 67.86) - open/close 2689 (time 1769755061.26 total 40.02 last 67.99) - open/close 3383 (time 1769755071.27 total 50.03 last 69.31) - open/close 4064 (time 1769755081.28 total 60.04 last 68.08) - open/close 4680 (time 1769755091.28 total 70.04 last 61.55) - open/close 5345 (time 1769755101.29 total 80.05 last 66.45) - open/close 6043 (time 1769755111.30 total 90.06 last 69.74) - open/close 6683 (time 1769755121.31 total 100.07 last 63.94) - open/close 7335 (time 1769755131.32 total 110.08 last 65.16) - open/close 7930 (time 1769755141.33 total 120.10 last 59.39) - open/close 8451 (time 1769755151.35 total 130.11 last 52.04) - open/close 9052 (time 1769755161.35 total 140.11 last 60.08) - open/close 9699 (time 1769755171.36 total 150.12 last 64.63) total: 10000 open/close in 154.90 seconds: 64.56 ops/second - unlinked 0 (time 1769755180 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 580 (time 1769755289.66 total 10.01 last 57.93) - open/close 1195 (time 1769755299.68 total 20.03 last 61.39) - open/close 1919 (time 1769755309.69 total 30.04 last 72.35) - open/close 2677 (time 1769755319.70 total 40.05 last 75.72) - open/close 3386 (time 1769755329.70 total 50.05 last 70.87) - open/close 4085 (time 1769755339.72 total 60.07 last 69.79) - open/close 4755 (time 1769755349.73 total 70.08 last 66.93) - open/close 5471 (time 1769755359.73 total 80.08 last 71.60) - open/close 6181 (time 1769755369.75 total 90.10 last 70.87) - open/close 6815 (time 1769755379.75 total 100.10 last 63.39) - open/close 7498 (time 1769755389.75 total 110.10 last 68.26) - open/close 8141 (time 1769755399.76 total 120.11 last 64.25) - open/close 8840 (time 1769755409.77 total 130.12 last 69.83) - open/close 9540 (time 1769755419.77 total 140.12 last 69.98) total: 10000 open/close in 146.02 seconds: 68.48 ops/second - unlinked 0 (time 1769755430 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 679 (time 1769755541.71 total 10.00 last 67.89) - open/close 1354 (time 1769755551.71 total 20.01 last 67.46) - open/close 2029 (time 1769755561.72 total 30.01 last 67.47) - open/close 2768 (time 1769755571.72 total 40.01 last 73.88) - open/close 3409 (time 1769755581.73 total 50.02 last 64.05) - open/close 4059 (time 1769755591.73 total 60.03 last 64.95) - open/close 4782 (time 1769755601.74 total 70.03 last 72.27) - open/close 5472 (time 1769755611.75 total 80.04 last 68.94) - open/close 6152 (time 1769755621.76 total 90.06 last 67.91) - open/close 6838 (time 1769755631.78 total 100.07 last 68.47) - open/close 7489 (time 1769755641.78 total 110.08 last 65.09) - open/close 8108 (time 1769755651.78 total 120.08 last 61.88) - open/close 8713 (time 1769755661.79 total 130.09 last 60.45) - open/close 9352 (time 1769755671.80 total 140.09 last 63.87) - open/close 9958 (time 1769755681.81 total 150.10 last 60.54) total: 10000 open/close in 150.98 seconds: 66.23 ops/second - unlinked 0 (time 1769755687 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 641 (time 1769755795.37 total 10.01 last 64.02) - open/close 1280 (time 1769755805.38 total 20.02 last 63.87) - open/close 1952 (time 1769755815.38 total 30.02 last 67.18) - open/close 2620 (time 1769755825.38 total 40.03 last 66.77) - open/close 3245 (time 1769755835.39 total 50.03 last 62.48) - open/close 3767 (time 1769755845.40 total 60.04 last 52.15) - open/close 4441 (time 1769755855.41 total 70.05 last 67.32) - open/close 5046 (time 1769755865.41 total 80.05 last 60.49) - open/close 5665 (time 1769755875.43 total 90.07 last 61.79) - open/close 6413 (time 1769755885.44 total 100.08 last 74.74) - open/close 7159 (time 1769755895.44 total 110.08 last 74.57) - open/close 7846 (time 1769755905.44 total 120.08 last 68.68) - open/close 8511 (time 1769755915.45 total 130.09 last 66.46) - open/close 9200 (time 1769755925.47 total 140.12 last 68.73) - open/close 9855 (time 1769755935.48 total 150.13 last 65.43) total: 10000 open/close in 152.76 seconds: 65.46 ops/second - unlinked 0 (time 1769755944 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.102@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.102@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.102@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.102@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.204.102@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local oleg402-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg402-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2