== conf-sanity test 69: replace an OST with the same index ========================================================== 13:55:13 (1776189313) start mds service on oleg224-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg224-server: detected 4 online CPUs by sysfs oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg224-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 929 (time 1776189412.20 total 10.00 last 92.88) - open/close 1857 (time 1776189422.20 total 20.01 last 92.75) - open/close 2667 (time 1776189432.21 total 30.01 last 80.96) - open/close 3537 (time 1776189442.21 total 40.02 last 86.94) - open/close 4396 (time 1776189452.22 total 50.02 last 85.88) - open/close 5211 (time 1776189462.22 total 60.02 last 81.50) - open/close 6053 (time 1776189472.23 total 70.04 last 84.05) - open/close 6838 (time 1776189482.24 total 80.04 last 78.47) - open/close 7697 (time 1776189492.25 total 90.06 last 85.80) - open/close 8362 (time 1776189502.26 total 100.06 last 66.46) - open/close 9279 (time 1776189512.26 total 110.07 last 91.67) - open/close 9919 (time 1776189522.28 total 120.09 last 63.87) total: 10000 open/close in 121.29 seconds: 82.44 ops/second - unlinked 0 (time 1776189529 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 809 (time 1776189619.93 total 10.00 last 80.90) - open/close 1668 (time 1776189629.93 total 20.00 last 85.89) - open/close 2607 (time 1776189639.93 total 30.00 last 93.88) - open/close 3597 (time 1776189649.94 total 40.01 last 98.96) - open/close 4511 (time 1776189659.95 total 50.02 last 91.32) - open/close 5395 (time 1776189669.96 total 60.03 last 88.32) - open/close 6379 (time 1776189679.96 total 70.03 last 98.38) - open/close 7318 (time 1776189689.96 total 80.03 last 93.85) - open/close 8299 (time 1776189699.97 total 90.04 last 98.05) - open/close 9250 (time 1776189709.98 total 100.05 last 95.01) total: 10000 open/close in 108.86 seconds: 91.86 ops/second - unlinked 0 (time 1776189723 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 626 (time 1776189806.43 total 10.01 last 62.54) - open/close 1494 (time 1776189816.44 total 20.01 last 86.77) - open/close 2233 (time 1776189826.45 total 30.03 last 73.80) - open/close 3016 (time 1776189836.47 total 40.05 last 78.13) - open/close 3850 (time 1776189846.48 total 50.06 last 83.34) - open/close 4373 (time 1776189856.52 total 60.09 last 52.10) - open/close 5013 (time 1776189866.52 total 70.10 last 63.97) - open/close 5844 (time 1776189876.55 total 80.12 last 82.91) - open/close 6655 (time 1776189886.55 total 90.13 last 81.07) - open/close 7457 (time 1776189896.56 total 100.13 last 80.12) - open/close 8134 (time 1776189906.56 total 110.14 last 67.67) - open/close 8844 (time 1776189916.57 total 120.14 last 70.98) - open/close 9595 (time 1776189926.57 total 130.15 last 75.05) total: 10000 open/close in 135.32 seconds: 73.90 ops/second - unlinked 0 (time 1776189936 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 724 (time 1776190018.96 total 10.01 last 72.36) - open/close 1516 (time 1776190028.96 total 20.01 last 79.18) - open/close 2315 (time 1776190038.97 total 30.01 last 79.87) - open/close 3230 (time 1776190048.98 total 40.02 last 91.38) - open/close 4060 (time 1776190058.98 total 50.03 last 82.96) - open/close 4880 (time 1776190069.00 total 60.04 last 81.87) - open/close 5564 (time 1776190079.01 total 70.05 last 68.34) - open/close 6252 (time 1776190089.01 total 80.06 last 68.78) - open/close 6936 (time 1776190099.02 total 90.06 last 68.36) - open/close 7730 (time 1776190109.02 total 100.07 last 79.36) - open/close 8556 (time 1776190119.03 total 110.07 last 82.57) - open/close 9255 (time 1776190129.04 total 120.08 last 69.81) total: 10000 open/close in 129.21 seconds: 77.39 ops/second - unlinked 0 (time 1776190142 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 689 (time 1776190245.49 total 10.02 last 68.79) - open/close 1421 (time 1776190255.50 total 20.02 last 73.15) - open/close 2082 (time 1776190265.51 total 30.03 last 66.04) - open/close 2829 (time 1776190275.52 total 40.04 last 74.64) - open/close 3549 (time 1776190285.52 total 50.04 last 72.00) - open/close 4337 (time 1776190295.53 total 60.06 last 78.68) - open/close 4999 (time 1776190305.54 total 70.07 last 66.14) - open/close 5656 (time 1776190315.55 total 80.07 last 65.66) - open/close 6406 (time 1776190325.55 total 90.07 last 74.99) - open/close 7312 (time 1776190335.56 total 100.08 last 90.54) - open/close 8152 (time 1776190345.57 total 110.09 last 83.91) - open/close 8944 (time 1776190355.57 total 120.09 last 79.18) - open/close 9546 (time 1776190365.58 total 130.11 last 60.12) total: 10000 open/close in 138.15 seconds: 72.39 ops/second - unlinked 0 (time 1776190377 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 765 (time 1776190472.82 total 10.03 last 76.30) - open/close 1466 (time 1776190482.82 total 20.03 last 70.06) - open/close 2173 (time 1776190492.84 total 30.05 last 70.59) - open/close 2727 (time 1776190502.88 total 40.09 last 55.17) - open/close 3398 (time 1776190512.89 total 50.10 last 67.04) - open/close 3946 (time 1776190522.90 total 60.11 last 54.75) - open/close 4749 (time 1776190532.90 total 70.11 last 80.25) - open/close 5437 (time 1776190542.91 total 80.12 last 68.74) - open/close 6116 (time 1776190552.91 total 90.12 last 67.89) - open/close 6929 (time 1776190562.92 total 100.13 last 81.26) - open/close 7651 (time 1776190572.93 total 110.14 last 72.13) - open/close 8276 (time 1776190582.93 total 120.14 last 62.46) - open/close 8867 (time 1776190592.94 total 130.15 last 59.06) - open/close 9627 (time 1776190602.95 total 140.16 last 75.94) total: 10000 open/close in 145.26 seconds: 68.84 ops/second - unlinked 0 (time 1776190612 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 607 (time 1776190721.41 total 10.01 last 60.66) - open/close 1252 (time 1776190731.42 total 20.02 last 64.43) - open/close 1879 (time 1776190741.44 total 30.04 last 62.56) - open/close 2422 (time 1776190751.44 total 40.04 last 54.29) - open/close 3114 (time 1776190761.45 total 50.05 last 69.16) - open/close 3730 (time 1776190771.45 total 60.05 last 61.57) - open/close 4430 (time 1776190781.46 total 70.06 last 69.92) - open/close 5159 (time 1776190791.48 total 80.08 last 72.78) - open/close 5930 (time 1776190801.49 total 90.09 last 76.99) - open/close 6500 (time 1776190811.51 total 100.11 last 56.93) - open/close 7187 (time 1776190821.51 total 110.11 last 68.68) - open/close 7941 (time 1776190831.52 total 120.12 last 75.33) - open/close 8745 (time 1776190841.52 total 130.12 last 80.36) - open/close 9440 (time 1776190851.53 total 140.13 last 69.45) total: 10000 open/close in 148.17 seconds: 67.49 ops/second - unlinked 0 (time 1776190863 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 634 (time 1776190966.88 total 10.00 last 63.39) - open/close 1125 (time 1776190976.89 total 20.01 last 49.04) - open/close 1841 (time 1776190986.89 total 30.02 last 71.56) - open/close 2459 (time 1776190996.89 total 40.02 last 61.80) - open/close 3072 (time 1776191006.91 total 50.04 last 61.19) - open/close 3684 (time 1776191016.92 total 60.04 last 61.17) - open/close 4307 (time 1776191026.92 total 70.04 last 62.29) - open/close 5004 (time 1776191036.93 total 80.05 last 69.65) - open/close 5726 (time 1776191046.93 total 90.06 last 72.15) - open/close 6559 (time 1776191056.94 total 100.06 last 83.28) - open/close 7185 (time 1776191066.95 total 110.07 last 62.54) - open/close 7763 (time 1776191076.95 total 120.07 last 57.80) - open/close 8582 (time 1776191086.95 total 130.07 last 81.89) - open/close 9315 (time 1776191096.95 total 140.08 last 73.29) total: 10000 open/close in 149.81 seconds: 66.75 ops/second - unlinked 0 (time 1776191111 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 665 (time 1776191206.84 total 10.00 last 66.47) - open/close 1272 (time 1776191216.85 total 20.01 last 60.66) - open/close 2018 (time 1776191226.86 total 30.02 last 74.52) - open/close 2799 (time 1776191236.87 total 40.03 last 78.06) - open/close 3517 (time 1776191246.87 total 50.03 last 71.77) - open/close 4145 (time 1776191256.88 total 60.04 last 62.73) - open/close 4905 (time 1776191266.89 total 70.05 last 75.92) - open/close 5752 (time 1776191276.90 total 80.06 last 84.65) - open/close 6555 (time 1776191286.90 total 90.06 last 80.25) - open/close 7417 (time 1776191296.91 total 100.07 last 86.16) - open/close 8241 (time 1776191306.91 total 110.07 last 82.37) - open/close 8965 (time 1776191316.92 total 120.08 last 72.37) - open/close 9839 (time 1776191326.93 total 130.09 last 87.27) total: 10000 open/close in 132.10 seconds: 75.70 ops/second - unlinked 0 (time 1776191333 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 686 (time 1776191420.50 total 10.01 last 68.55) - open/close 1370 (time 1776191430.54 total 20.05 last 68.09) - open/close 2116 (time 1776191440.55 total 30.06 last 74.54) - open/close 2992 (time 1776191450.55 total 40.06 last 87.59) - open/close 3754 (time 1776191460.55 total 50.06 last 76.20) - open/close 4580 (time 1776191470.56 total 60.07 last 82.55) - open/close 5252 (time 1776191480.56 total 70.07 last 67.16) - open/close 5914 (time 1776191490.59 total 80.10 last 66.03) - open/close 6660 (time 1776191500.59 total 90.10 last 74.60) - open/close 7576 (time 1776191510.60 total 100.11 last 91.49) - open/close 8458 (time 1776191520.61 total 110.13 last 88.06) - open/close 9203 (time 1776191530.63 total 120.14 last 74.41) - open/close 9807 (time 1776191540.63 total 130.14 last 60.38) total: 10000 open/close in 133.03 seconds: 75.17 ops/second - unlinked 0 (time 1776191549 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg224-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg224-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg224-server: LNET ready to unload oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2