== conf-sanity test 69: replace an OST with the same index ========================================================== 15:54:50 (1786305290) start mds service on oleg122-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg122-server: detected 4 online CPUs by sysfs oleg122-server: MODOPTS_LIBCFS= oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg122-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 816 (time 1786305384.86 total 10.00 last 81.57) - open/close 1562 (time 1786305394.87 total 20.02 last 74.50) - open/close 2319 (time 1786305404.88 total 30.02 last 75.68) - open/close 3016 (time 1786305414.88 total 40.03 last 69.65) - open/close 3769 (time 1786305424.89 total 50.03 last 75.26) - open/close 4443 (time 1786305434.90 total 60.04 last 67.33) - open/close 5143 (time 1786305444.90 total 70.05 last 69.97) - open/close 5909 (time 1786305454.91 total 80.05 last 76.54) - open/close 6692 (time 1786305464.92 total 90.06 last 78.25) - open/close 7381 (time 1786305474.92 total 100.07 last 68.88) - open/close 8040 (time 1786305484.95 total 110.09 last 65.73) - open/close 8594 (time 1786305494.96 total 120.11 last 55.31) - open/close 9450 (time 1786305504.97 total 130.11 last 85.56) total: 10000 open/close in 137.01 seconds: 72.99 ops/second - unlinked 0 (time 1786305516 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 767 (time 1786305622.29 total 10.01 last 76.63) - open/close 1416 (time 1786305632.29 total 20.01 last 64.88) - open/close 2101 (time 1786305642.30 total 30.03 last 68.41) - open/close 2750 (time 1786305652.31 total 40.03 last 64.87) - open/close 3346 (time 1786305662.33 total 50.05 last 59.46) - open/close 4087 (time 1786305672.36 total 60.08 last 73.89) - open/close 4799 (time 1786305682.37 total 70.09 last 71.12) - open/close 5549 (time 1786305692.38 total 80.10 last 74.96) - open/close 6296 (time 1786305702.38 total 90.11 last 74.63) - open/close 7011 (time 1786305712.40 total 100.13 last 71.36) - open/close 7779 (time 1786305722.41 total 110.13 last 76.78) - open/close 8583 (time 1786305732.41 total 120.14 last 80.34) - open/close 9395 (time 1786305742.42 total 130.14 last 81.16) total: 10000 open/close in 138.89 seconds: 72.00 ops/second - unlinked 0 (time 1786305755 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 848 (time 1786305860.53 total 10.01 last 84.75) - open/close 1605 (time 1786305870.53 total 20.01 last 75.70) - open/close 2322 (time 1786305880.54 total 30.01 last 71.68) - open/close 2925 (time 1786305890.54 total 40.02 last 60.25) - open/close 3575 (time 1786305900.56 total 50.03 last 64.90) - open/close 4266 (time 1786305910.57 total 60.04 last 69.05) - open/close 5000 (time 1786305920.57 total 70.05 last 73.34) - open/close 5665 (time 1786305930.58 total 80.05 last 66.45) - open/close 6375 (time 1786305940.59 total 90.07 last 70.92) - open/close 7129 (time 1786305950.60 total 100.07 last 75.38) - open/close 7835 (time 1786305960.60 total 110.07 last 70.60) - open/close 8621 (time 1786305970.60 total 120.07 last 78.58) - open/close 9435 (time 1786305980.61 total 130.08 last 81.35) total: 10000 open/close in 138.86 seconds: 72.01 ops/second - unlinked 0 (time 1786305994 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 781 (time 1786306099.84 total 10.00 last 78.09) - open/close 1461 (time 1786306109.84 total 20.00 last 67.98) - open/close 2155 (time 1786306119.85 total 30.01 last 69.34) - open/close 2884 (time 1786306129.86 total 40.02 last 72.87) - open/close 3686 (time 1786306139.87 total 50.03 last 80.14) - open/close 4324 (time 1786306149.88 total 60.04 last 63.68) - open/close 4825 (time 1786306159.92 total 70.08 last 49.90) - open/close 5399 (time 1786306169.92 total 80.09 last 57.40) - open/close 6069 (time 1786306179.93 total 90.09 last 66.97) - open/close 6759 (time 1786306189.93 total 100.09 last 68.99) - open/close 7531 (time 1786306199.94 total 110.10 last 77.17) - open/close 8221 (time 1786306210.00 total 120.16 last 68.58) - open/close 8924 (time 1786306220.01 total 130.17 last 70.24) - open/close 9536 (time 1786306230.03 total 140.19 last 61.04) total: 10000 open/close in 147.29 seconds: 67.89 ops/second - unlinked 0 (time 1786306242 ; total 1 ; last 1) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 882 (time 1786306345.43 total 10.01 last 88.14) - open/close 1555 (time 1786306355.44 total 20.02 last 67.22) - open/close 2300 (time 1786306365.45 total 30.03 last 74.42) - open/close 2970 (time 1786306375.45 total 40.03 last 66.99) - open/close 3580 (time 1786306385.46 total 50.04 last 60.98) - open/close 4258 (time 1786306395.47 total 60.05 last 67.70) - open/close 4883 (time 1786306405.48 total 70.06 last 62.44) - open/close 5627 (time 1786306415.49 total 80.07 last 74.33) - open/close 6244 (time 1786306425.50 total 90.08 last 61.64) - open/close 6850 (time 1786306435.51 total 100.09 last 60.57) - open/close 7548 (time 1786306445.55 total 110.13 last 69.47) - open/close 8228 (time 1786306455.56 total 120.14 last 67.95) - open/close 8908 (time 1786306465.56 total 130.14 last 68.00) - open/close 9477 (time 1786306475.56 total 140.14 last 56.90) total: 10000 open/close in 148.45 seconds: 67.36 ops/second - unlinked 0 (time 1786306488 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 758 (time 1786306593.16 total 10.01 last 75.72) - open/close 1517 (time 1786306603.16 total 20.02 last 75.83) - open/close 2238 (time 1786306613.17 total 30.02 last 72.08) - open/close 2963 (time 1786306623.18 total 40.03 last 72.42) - open/close 3691 (time 1786306633.18 total 50.03 last 72.79) - open/close 4407 (time 1786306643.20 total 60.05 last 71.45) - open/close 5121 (time 1786306653.21 total 70.06 last 71.35) - open/close 5780 (time 1786306663.21 total 80.07 last 65.87) - open/close 6506 (time 1786306673.22 total 90.08 last 72.52) - open/close 7184 (time 1786306683.23 total 100.08 last 67.79) - open/close 7902 (time 1786306693.23 total 110.09 last 71.75) - open/close 8627 (time 1786306703.24 total 120.10 last 72.42) - open/close 9363 (time 1786306713.24 total 130.10 last 73.58) total: 10000 open/close in 138.83 seconds: 72.03 ops/second - unlinked 0 (time 1786306726 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 661 (time 1786306838.63 total 10.02 last 65.94) - open/close 1287 (time 1786306848.63 total 20.03 last 62.56) - open/close 2000 (time 1786306858.64 total 30.04 last 71.27) - open/close 2665 (time 1786306868.65 total 40.05 last 66.41) - open/close 3371 (time 1786306878.67 total 50.07 last 70.45) - open/close 4078 (time 1786306888.68 total 60.07 last 70.67) - open/close 4709 (time 1786306898.69 total 70.09 last 62.99) - open/close 5414 (time 1786306908.71 total 80.11 last 70.40) - open/close 6204 (time 1786306918.71 total 90.11 last 78.98) - open/close 6891 (time 1786306928.71 total 100.11 last 68.68) - open/close 7392 (time 1786306938.74 total 110.14 last 49.96) - open/close 8033 (time 1786306948.75 total 120.15 last 64.05) - open/close 8771 (time 1786306958.75 total 130.15 last 73.77) - open/close 9507 (time 1786306968.76 total 140.16 last 73.55) total: 10000 open/close in 147.00 seconds: 68.03 ops/second - unlinked 0 (time 1786306980 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 667 (time 1786307094.12 total 10.01 last 66.61) - open/close 1352 (time 1786307104.12 total 20.02 last 68.45) - open/close 2109 (time 1786307114.12 total 30.02 last 75.68) - open/close 2773 (time 1786307124.14 total 40.04 last 66.29) - open/close 3459 (time 1786307134.15 total 50.05 last 68.54) - open/close 4134 (time 1786307144.16 total 60.06 last 67.40) - open/close 4748 (time 1786307154.18 total 70.08 last 61.31) - open/close 5415 (time 1786307164.20 total 80.10 last 66.55) - open/close 6096 (time 1786307174.22 total 90.12 last 67.97) - open/close 6591 (time 1786307184.23 total 100.13 last 49.46) - open/close 7025 (time 1786307194.23 total 110.13 last 43.39) - open/close 7773 (time 1786307204.23 total 120.13 last 74.80) - open/close 8400 (time 1786307214.23 total 130.13 last 62.68) - open/close 8940 (time 1786307224.24 total 140.13 last 53.99) - open/close 9649 (time 1786307234.24 total 150.14 last 70.86) total: 10000 open/close in 153.93 seconds: 64.96 ops/second - unlinked 0 (time 1786307242 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 584 (time 1786307355.02 total 10.02 last 58.31) - open/close 1158 (time 1786307365.04 total 20.03 last 57.33) - open/close 1900 (time 1786307375.04 total 30.03 last 74.19) - open/close 2590 (time 1786307385.04 total 40.03 last 68.97) - open/close 3392 (time 1786307395.05 total 50.04 last 80.17) - open/close 4159 (time 1786307405.06 total 60.05 last 76.63) - open/close 4958 (time 1786307415.06 total 70.05 last 79.89) - open/close 5611 (time 1786307425.07 total 80.06 last 65.21) - open/close 6199 (time 1786307435.13 total 90.12 last 58.44) - open/close 6765 (time 1786307445.14 total 100.13 last 56.58) - open/close 7373 (time 1786307455.14 total 110.13 last 60.80) - open/close 7971 (time 1786307465.14 total 120.13 last 59.78) - open/close 8638 (time 1786307475.15 total 130.14 last 66.62) - open/close 9156 (time 1786307485.15 total 140.14 last 51.80) - open/close 9863 (time 1786307495.16 total 150.15 last 70.67) total: 10000 open/close in 152.24 seconds: 65.69 ops/second - unlinked 0 (time 1786307502 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 665 (time 1786307616.43 total 10.01 last 66.45) - open/close 1289 (time 1786307626.44 total 20.01 last 62.36) - open/close 1963 (time 1786307636.45 total 30.02 last 67.35) - open/close 2674 (time 1786307646.46 total 40.03 last 71.02) - open/close 3320 (time 1786307656.47 total 50.04 last 64.53) - open/close 4111 (time 1786307666.47 total 60.05 last 79.08) - open/close 5026 (time 1786307676.48 total 70.05 last 91.47) - open/close 5820 (time 1786307686.49 total 80.06 last 79.33) - open/close 6516 (time 1786307696.49 total 90.06 last 69.60) - open/close 7182 (time 1786307706.51 total 100.08 last 66.46) - open/close 7819 (time 1786307716.52 total 110.09 last 63.63) - open/close 8481 (time 1786307726.53 total 120.10 last 66.14) - open/close 9125 (time 1786307736.54 total 130.11 last 64.32) - open/close 9833 (time 1786307746.54 total 140.12 last 70.78) total: 10000 open/close in 143.60 seconds: 69.64 ops/second - unlinked 0 (time 1786307755 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.122@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.122@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.122@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.122@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.201.122@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server unloading modules via unload_modules_local on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: modules unloaded. pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2