== conf-sanity test 69: replace an OST with the same index ========================================================== 16:38:19 (1776199099) start mds service on oleg422-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg422-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg422-server: detected 4 online CPUs by sysfs oleg422-server: Force libcfs to create 2 CPU partitions oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg422-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg422-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg422-client.virtnet: -o user_xattr,flock oleg422-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 574 (time 1776199221.65 total 10.01 last 57.34) - open/close 1173 (time 1776199231.66 total 20.02 last 59.83) - open/close 1785 (time 1776199241.67 total 30.03 last 61.19) - open/close 2402 (time 1776199251.67 total 40.03 last 61.69) - open/close 3051 (time 1776199261.67 total 50.03 last 64.87) - open/close 3648 (time 1776199271.67 total 60.03 last 59.69) - open/close 4269 (time 1776199281.68 total 70.04 last 62.04) - open/close 4864 (time 1776199291.69 total 80.05 last 59.48) - open/close 5510 (time 1776199301.70 total 90.05 last 64.55) - open/close 6159 (time 1776199311.71 total 100.06 last 64.84) - open/close 6833 (time 1776199321.71 total 110.07 last 67.35) - open/close 7489 (time 1776199331.72 total 120.08 last 65.56) - open/close 8167 (time 1776199341.73 total 130.09 last 67.73) - open/close 8755 (time 1776199351.73 total 140.09 last 58.79) - open/close 9378 (time 1776199361.74 total 150.10 last 62.26) total: 10000 open/close in 159.71 seconds: 62.61 ops/second - unlinked 0 (time 1776199376 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 645 (time 1776199491.02 total 10.01 last 64.44) - open/close 1313 (time 1776199501.05 total 20.04 last 66.60) - open/close 1882 (time 1776199511.06 total 30.05 last 56.83) - open/close 2491 (time 1776199521.06 total 40.06 last 60.88) - open/close 3107 (time 1776199531.07 total 50.06 last 61.56) - open/close 3759 (time 1776199541.07 total 60.06 last 65.19) - open/close 4299 (time 1776199551.07 total 70.07 last 53.97) - open/close 4838 (time 1776199561.09 total 80.09 last 53.81) - open/close 5370 (time 1776199571.11 total 90.11 last 53.10) - open/close 5873 (time 1776199581.11 total 100.11 last 50.29) - open/close 6422 (time 1776199591.14 total 110.13 last 54.75) - open/close 6933 (time 1776199601.15 total 120.14 last 51.07) - open/close 7509 (time 1776199611.15 total 130.15 last 57.55) - open/close 8109 (time 1776199621.15 total 140.15 last 60.00) - open/close 8762 (time 1776199631.16 total 150.15 last 65.29) - open/close 9408 (time 1776199641.16 total 160.16 last 64.56) total: 10000 open/close in 169.50 seconds: 59.00 ops/second - unlinked 0 (time 1776199655 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 644 (time 1776199761.03 total 10.00 last 64.37) - open/close 1271 (time 1776199771.03 total 20.01 last 62.66) - open/close 1852 (time 1776199781.04 total 30.02 last 58.07) - open/close 2483 (time 1776199791.06 total 40.03 last 62.99) - open/close 3123 (time 1776199801.07 total 50.05 last 63.91) - open/close 3709 (time 1776199811.07 total 60.05 last 58.60) - open/close 4342 (time 1776199821.08 total 70.06 last 63.24) - open/close 4958 (time 1776199831.09 total 80.07 last 61.54) - open/close 5594 (time 1776199841.11 total 90.08 last 63.49) - open/close 6223 (time 1776199851.12 total 100.09 last 62.85) - open/close 6859 (time 1776199861.13 total 110.10 last 63.53) - open/close 7440 (time 1776199871.14 total 120.12 last 58.02) - open/close 8043 (time 1776199881.14 total 130.12 last 60.29) - open/close 8658 (time 1776199891.15 total 140.13 last 61.44) - open/close 9288 (time 1776199901.16 total 150.13 last 62.96) - open/close 9860 (time 1776199911.16 total 160.14 last 57.17) total: 10000 open/close in 162.20 seconds: 61.65 ops/second - unlinked 0 (time 1776199918 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 596 (time 1776200028.01 total 10.00 last 59.59) - open/close 1217 (time 1776200038.02 total 20.01 last 62.03) - open/close 1867 (time 1776200048.03 total 30.02 last 64.97) - open/close 2510 (time 1776200058.03 total 40.02 last 64.29) - open/close 3143 (time 1776200068.04 total 50.03 last 63.24) - open/close 3804 (time 1776200078.04 total 60.03 last 66.08) - open/close 4420 (time 1776200088.05 total 70.05 last 61.53) - open/close 5016 (time 1776200098.07 total 80.06 last 59.49) - open/close 5676 (time 1776200108.08 total 90.07 last 65.93) - open/close 6274 (time 1776200118.08 total 100.08 last 59.79) - open/close 6946 (time 1776200128.10 total 110.10 last 67.07) - open/close 7532 (time 1776200138.13 total 120.12 last 58.46) - open/close 8143 (time 1776200148.14 total 130.13 last 61.04) - open/close 8779 (time 1776200158.16 total 140.15 last 63.48) - open/close 9366 (time 1776200168.16 total 150.15 last 58.67) - open/close 9938 (time 1776200178.19 total 160.18 last 57.06) total: 10000 open/close in 161.73 seconds: 61.83 ops/second - unlinked 0 (time 1776200186 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 678 (time 1776200294.07 total 10.01 last 67.76) - open/close 1327 (time 1776200304.08 total 20.01 last 64.87) - open/close 1926 (time 1776200314.09 total 30.02 last 59.86) - open/close 2556 (time 1776200324.09 total 40.03 last 62.96) - open/close 3222 (time 1776200334.10 total 50.03 last 66.56) - open/close 3919 (time 1776200344.10 total 60.03 last 69.69) - open/close 4611 (time 1776200354.10 total 70.04 last 69.19) - open/close 5307 (time 1776200364.11 total 80.04 last 69.57) - open/close 5978 (time 1776200374.11 total 90.05 last 67.05) - open/close 6593 (time 1776200384.13 total 100.06 last 61.43) - open/close 7123 (time 1776200394.13 total 110.06 last 52.97) - open/close 7759 (time 1776200404.14 total 120.07 last 63.55) - open/close 8313 (time 1776200414.15 total 130.08 last 55.35) - open/close 8939 (time 1776200424.16 total 140.10 last 62.50) - open/close 9559 (time 1776200434.18 total 150.11 last 61.89) total: 10000 open/close in 157.17 seconds: 63.62 ops/second - unlinked 0 (time 1776200445 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 645 (time 1776200560.48 total 10.00 last 64.49) - open/close 1271 (time 1776200570.49 total 20.01 last 62.55) - open/close 1918 (time 1776200580.49 total 30.01 last 64.69) - open/close 2499 (time 1776200590.50 total 40.02 last 58.03) - open/close 3115 (time 1776200600.52 total 50.03 last 61.52) - open/close 3742 (time 1776200610.52 total 60.04 last 62.65) - open/close 4304 (time 1776200620.53 total 70.05 last 56.16) - open/close 4965 (time 1776200630.55 total 80.07 last 65.97) - open/close 5614 (time 1776200640.58 total 90.10 last 64.72) - open/close 6267 (time 1776200650.58 total 100.10 last 65.28) - open/close 6933 (time 1776200660.58 total 110.10 last 66.60) - open/close 7573 (time 1776200670.58 total 120.10 last 64.00) - open/close 8142 (time 1776200680.60 total 130.11 last 56.82) - open/close 8874 (time 1776200690.60 total 140.12 last 73.18) - open/close 9528 (time 1776200700.62 total 150.13 last 65.29) total: 10000 open/close in 157.65 seconds: 63.43 ops/second - unlinked 0 (time 1776200712 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 670 (time 1776200829.53 total 10.02 last 66.87) - open/close 1274 (time 1776200839.54 total 20.04 last 60.30) - open/close 1856 (time 1776200849.54 total 30.04 last 58.20) - open/close 2519 (time 1776200859.55 total 40.04 last 66.25) - open/close 3116 (time 1776200869.56 total 50.05 last 59.66) - open/close 3691 (time 1776200879.57 total 60.07 last 57.40) - open/close 4349 (time 1776200889.58 total 70.07 last 65.77) - open/close 5013 (time 1776200899.60 total 80.09 last 66.29) - open/close 5669 (time 1776200909.60 total 90.09 last 65.57) - open/close 6316 (time 1776200919.60 total 100.10 last 64.67) - open/close 7012 (time 1776200929.61 total 110.11 last 69.56) - open/close 7691 (time 1776200939.63 total 120.13 last 67.75) - open/close 8333 (time 1776200949.64 total 130.13 last 64.16) - open/close 8982 (time 1776200959.65 total 140.15 last 64.80) - open/close 9618 (time 1776200969.66 total 150.15 last 63.56) total: 10000 open/close in 156.20 seconds: 64.02 ops/second - unlinked 0 (time 1776200980 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 689 (time 1776201080.16 total 10.01 last 68.84) - open/close 1378 (time 1776201090.16 total 20.01 last 68.90) - open/close 2024 (time 1776201100.18 total 30.02 last 64.53) - open/close 2706 (time 1776201110.18 total 40.02 last 68.19) - open/close 3303 (time 1776201120.18 total 50.03 last 59.66) - open/close 3971 (time 1776201130.19 total 60.04 last 66.73) - open/close 4555 (time 1776201140.21 total 70.05 last 58.33) - open/close 5131 (time 1776201150.22 total 80.06 last 57.55) - open/close 5776 (time 1776201160.22 total 90.07 last 64.46) - open/close 6369 (time 1776201170.24 total 100.08 last 59.22) - open/close 6939 (time 1776201180.25 total 110.10 last 56.91) - open/close 7615 (time 1776201190.26 total 120.11 last 67.53) - open/close 8276 (time 1776201200.27 total 130.12 last 66.04) - open/close 8949 (time 1776201210.28 total 140.12 last 67.25) - open/close 9590 (time 1776201220.30 total 150.14 last 64.00) total: 10000 open/close in 157.04 seconds: 63.68 ops/second - unlinked 0 (time 1776201231 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 694 (time 1776201336.47 total 10.00 last 69.39) - open/close 1324 (time 1776201346.48 total 20.01 last 62.97) - open/close 1949 (time 1776201356.50 total 30.03 last 62.36) - open/close 2592 (time 1776201366.51 total 40.04 last 64.21) - open/close 3235 (time 1776201376.53 total 50.06 last 64.20) - open/close 3900 (time 1776201386.54 total 60.07 last 66.41) - open/close 4575 (time 1776201396.55 total 70.08 last 67.45) - open/close 5264 (time 1776201406.55 total 80.08 last 68.86) - open/close 5892 (time 1776201416.56 total 90.09 last 62.76) - open/close 6570 (time 1776201426.60 total 100.13 last 67.56) - open/close 7231 (time 1776201436.60 total 110.13 last 66.07) - open/close 7839 (time 1776201446.62 total 120.15 last 60.70) - open/close 8531 (time 1776201456.62 total 130.15 last 69.19) - open/close 9170 (time 1776201466.63 total 140.16 last 63.84) - open/close 9853 (time 1776201476.65 total 150.18 last 68.14) total: 10000 open/close in 152.53 seconds: 65.56 ops/second - unlinked 0 (time 1776201483 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 540 (time 1776201584.40 total 10.00 last 53.97) - open/close 1203 (time 1776201594.41 total 20.02 last 66.19) - open/close 1833 (time 1776201604.42 total 30.02 last 62.98) - open/close 2398 (time 1776201614.42 total 40.03 last 56.47) - open/close 3023 (time 1776201624.44 total 50.04 last 62.40) - open/close 3635 (time 1776201634.45 total 60.05 last 61.15) - open/close 4242 (time 1776201644.45 total 70.06 last 60.64) - open/close 4772 (time 1776201654.46 total 80.06 last 52.99) - open/close 5332 (time 1776201664.47 total 90.07 last 55.95) - open/close 5975 (time 1776201674.47 total 100.07 last 64.30) - open/close 6612 (time 1776201684.47 total 110.08 last 63.69) - open/close 7245 (time 1776201694.47 total 120.08 last 63.28) - open/close 7862 (time 1776201704.47 total 130.08 last 61.70) - open/close 8538 (time 1776201714.50 total 140.10 last 67.44) - open/close 9152 (time 1776201724.50 total 150.11 last 61.35) - open/close 9717 (time 1776201734.52 total 160.13 last 56.40) total: 10000 open/close in 164.83 seconds: 60.67 ops/second - unlinked 0 (time 1776201744 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg422-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg422-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg422-client.virtnet: -o user_xattr,flock oleg422-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg422-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing unload_modules_local oleg422-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg422-server: LNET ready to unload oleg422-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2