== conf-sanity test 69: replace an OST with the same index ========================================================== 08:56:31 (1783947391) start mds service on oleg232-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg232-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49665 inodes available. Want 99872. rc=0 - open/close 544 (time 1783947496.45 total 10.00 last 54.39) - open/close 1143 (time 1783947506.45 total 20.00 last 59.89) - open/close 1833 (time 1783947516.45 total 30.01 last 68.99) - open/close 2562 (time 1783947526.46 total 40.01 last 72.87) - open/close 3402 (time 1783947536.46 total 50.02 last 83.95) - open/close 4166 (time 1783947546.47 total 60.03 last 76.32) - open/close 4979 (time 1783947556.48 total 70.04 last 81.20) - open/close 5729 (time 1783947566.49 total 80.04 last 74.97) - open/close 6456 (time 1783947576.50 total 90.05 last 72.63) - open/close 7278 (time 1783947586.50 total 100.06 last 82.15) - open/close 8073 (time 1783947596.52 total 110.07 last 79.40) - open/close 8789 (time 1783947606.52 total 120.07 last 71.60) - open/close 9387 (time 1783947616.52 total 130.08 last 59.75) - open/close 9985 (time 1783947626.53 total 140.09 last 59.74) total: 10000 open/close in 140.51 seconds: 71.17 ops/second - unlinked 0 (time 1783947632 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 564 (time 1783947735.41 total 10.01 last 56.34) - open/close 1305 (time 1783947745.42 total 20.02 last 74.03) - open/close 2018 (time 1783947755.42 total 30.02 last 71.29) - open/close 2861 (time 1783947765.42 total 40.02 last 84.29) - open/close 3638 (time 1783947775.42 total 50.03 last 77.67) - open/close 4357 (time 1783947785.44 total 60.04 last 71.80) - open/close 5018 (time 1783947795.45 total 70.05 last 66.03) - open/close 5735 (time 1783947805.46 total 80.06 last 71.65) - open/close 6495 (time 1783947815.47 total 90.07 last 75.91) - open/close 7250 (time 1783947825.48 total 100.08 last 75.46) - open/close 7965 (time 1783947835.48 total 110.08 last 71.45) - open/close 8762 (time 1783947845.48 total 120.08 last 79.70) - open/close 9424 (time 1783947855.48 total 130.08 last 66.20) total: 10000 open/close in 138.86 seconds: 72.01 ops/second - unlinked 0 (time 1783947869 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 827 (time 1783947962.04 total 10.01 last 82.64) - open/close 1585 (time 1783947972.04 total 20.01 last 75.78) - open/close 2262 (time 1783947982.05 total 30.02 last 67.64) - open/close 3117 (time 1783947992.05 total 40.02 last 85.45) - open/close 3749 (time 1783948002.05 total 50.02 last 63.20) - open/close 4496 (time 1783948012.07 total 60.04 last 74.60) - open/close 5238 (time 1783948022.08 total 70.05 last 74.13) - open/close 5987 (time 1783948032.09 total 80.06 last 74.82) - open/close 6757 (time 1783948042.10 total 90.07 last 76.94) - open/close 7509 (time 1783948052.10 total 100.07 last 75.16) - open/close 8249 (time 1783948062.12 total 110.09 last 73.85) - open/close 8953 (time 1783948072.14 total 120.11 last 70.27) - open/close 9540 (time 1783948082.14 total 130.11 last 58.67) total: 10000 open/close in 135.89 seconds: 73.59 ops/second - unlinked 0 (time 1783948095 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 585 (time 1783948199.28 total 10.00 last 58.49) - open/close 1102 (time 1783948209.28 total 20.00 last 51.69) - open/close 1748 (time 1783948219.29 total 30.01 last 64.55) - open/close 2410 (time 1783948229.30 total 40.02 last 66.14) - open/close 3064 (time 1783948239.30 total 50.02 last 65.40) - open/close 3711 (time 1783948249.31 total 60.03 last 64.62) - open/close 4246 (time 1783948259.31 total 70.04 last 53.50) - open/close 4809 (time 1783948269.32 total 80.05 last 56.24) - open/close 5367 (time 1783948279.34 total 90.06 last 55.71) - open/close 5948 (time 1783948289.36 total 100.08 last 57.98) - open/close 6582 (time 1783948299.36 total 110.09 last 63.37) - open/close 7335 (time 1783948309.36 total 120.09 last 75.29) - open/close 8130 (time 1783948319.37 total 130.09 last 79.49) - open/close 8786 (time 1783948329.37 total 140.09 last 65.60) - open/close 9417 (time 1783948339.37 total 150.10 last 63.07) total: 10000 open/close in 158.35 seconds: 63.15 ops/second - unlinked 0 (time 1783948351 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 733 (time 1783948447.42 total 10.00 last 73.27) - open/close 1432 (time 1783948457.43 total 20.01 last 69.83) - open/close 2189 (time 1783948467.44 total 30.02 last 75.63) - open/close 3031 (time 1783948477.45 total 40.03 last 84.11) - open/close 3699 (time 1783948487.45 total 50.04 last 66.77) - open/close 4446 (time 1783948497.46 total 60.04 last 74.68) - open/close 5159 (time 1783948507.46 total 70.05 last 71.25) - open/close 5890 (time 1783948517.47 total 80.05 last 73.09) - open/close 6537 (time 1783948527.47 total 90.06 last 64.68) - open/close 7242 (time 1783948537.47 total 100.06 last 70.48) - open/close 7927 (time 1783948547.47 total 110.06 last 68.50) - open/close 8665 (time 1783948557.48 total 120.07 last 73.73) - open/close 9156 (time 1783948567.49 total 130.07 last 49.09) - open/close 9709 (time 1783948577.49 total 140.08 last 55.25) total: 10000 open/close in 145.45 seconds: 68.75 ops/second - unlinked 0 (time 1783948588 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 735 (time 1783948687.36 total 10.01 last 73.42) - open/close 1408 (time 1783948697.36 total 20.01 last 67.30) - open/close 2159 (time 1783948707.38 total 30.03 last 74.98) - open/close 2826 (time 1783948717.39 total 40.04 last 66.63) - open/close 3520 (time 1783948727.39 total 50.04 last 69.37) - open/close 4339 (time 1783948737.40 total 60.05 last 81.84) - open/close 5176 (time 1783948747.41 total 70.06 last 83.63) - open/close 5898 (time 1783948757.42 total 80.06 last 72.16) - open/close 6638 (time 1783948767.43 total 90.08 last 73.92) - open/close 7357 (time 1783948777.43 total 100.08 last 71.90) - open/close 8136 (time 1783948787.43 total 110.08 last 77.87) - open/close 8881 (time 1783948797.44 total 120.09 last 74.45) - open/close 9609 (time 1783948807.44 total 130.09 last 72.76) total: 10000 open/close in 136.38 seconds: 73.32 ops/second - unlinked 0 (time 1783948817 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 670 (time 1783948916.37 total 10.02 last 66.87) - open/close 1289 (time 1783948926.50 total 20.15 last 61.10) - open/close 1975 (time 1783948936.54 total 30.19 last 68.33) - open/close 2478 (time 1783948946.54 total 40.20 last 50.27) - open/close 2908 (time 1783948956.55 total 50.21 last 42.96) - open/close 3454 (time 1783948966.55 total 60.21 last 54.60) - open/close 4125 (time 1783948976.56 total 70.21 last 67.09) - open/close 4855 (time 1783948986.56 total 80.21 last 72.97) - open/close 5740 (time 1783948996.56 total 90.22 last 88.46) - open/close 6409 (time 1783949006.57 total 100.23 last 66.84) - open/close 7189 (time 1783949016.58 total 110.23 last 77.97) - open/close 8016 (time 1783949026.58 total 120.23 last 82.70) - open/close 8793 (time 1783949036.59 total 130.24 last 77.65) - open/close 9542 (time 1783949046.59 total 140.25 last 74.84) total: 10000 open/close in 146.28 seconds: 68.36 ops/second - unlinked 0 (time 1783949057 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 850 (time 1783949154.38 total 10.00 last 84.97) - open/close 1573 (time 1783949164.38 total 20.01 last 72.27) - open/close 2320 (time 1783949174.39 total 30.02 last 74.61) - open/close 3054 (time 1783949184.40 total 40.03 last 73.35) - open/close 3841 (time 1783949194.41 total 50.04 last 78.59) - open/close 4535 (time 1783949204.41 total 60.04 last 69.39) - open/close 5252 (time 1783949214.42 total 70.05 last 71.65) - open/close 5976 (time 1783949224.42 total 80.05 last 72.38) - open/close 6734 (time 1783949234.43 total 90.06 last 75.73) - open/close 7482 (time 1783949244.44 total 100.07 last 74.77) - open/close 8202 (time 1783949254.44 total 110.07 last 71.98) - open/close 8987 (time 1783949264.45 total 120.07 last 78.45) - open/close 9753 (time 1783949274.48 total 130.10 last 76.36) total: 10000 open/close in 133.39 seconds: 74.97 ops/second - unlinked 0 (time 1783949281 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 712 (time 1783949374.71 total 10.02 last 71.04) - open/close 1543 (time 1783949384.71 total 20.02 last 83.10) - open/close 2340 (time 1783949394.72 total 30.03 last 79.68) - open/close 3045 (time 1783949404.72 total 40.03 last 70.47) - open/close 3712 (time 1783949414.73 total 50.04 last 66.64) - open/close 4331 (time 1783949424.74 total 60.05 last 61.86) - open/close 5041 (time 1783949434.74 total 70.05 last 70.98) - open/close 5789 (time 1783949444.75 total 80.06 last 74.72) - open/close 6642 (time 1783949454.78 total 90.09 last 85.08) - open/close 7342 (time 1783949464.78 total 100.09 last 69.97) - open/close 7989 (time 1783949474.79 total 110.10 last 64.66) - open/close 8667 (time 1783949484.81 total 120.12 last 67.65) - open/close 9457 (time 1783949494.81 total 130.12 last 78.95) total: 10000 open/close in 138.01 seconds: 72.46 ops/second - unlinked 0 (time 1783949507 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 702 (time 1783949605.42 total 10.02 last 70.07) - open/close 1440 (time 1783949615.42 total 20.02 last 73.77) - open/close 2319 (time 1783949625.42 total 30.02 last 87.89) - open/close 3079 (time 1783949635.43 total 40.03 last 75.97) - open/close 3931 (time 1783949645.43 total 50.03 last 85.18) - open/close 4646 (time 1783949655.43 total 60.03 last 71.49) - open/close 5317 (time 1783949665.44 total 70.04 last 67.05) - open/close 6134 (time 1783949675.44 total 80.04 last 81.68) - open/close 6952 (time 1783949685.45 total 90.05 last 81.75) - open/close 7597 (time 1783949695.47 total 100.07 last 64.34) - open/close 8346 (time 1783949705.48 total 110.08 last 74.87) - open/close 9143 (time 1783949715.50 total 120.11 last 79.47) - open/close 9816 (time 1783949725.51 total 130.11 last 67.27) total: 10000 open/close in 133.07 seconds: 75.15 ops/second - unlinked 0 (time 1783949733 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre On OST0, 20236 used inodes rc=0 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local oleg232-server: modules unloaded. pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2