== conf-sanity test 69: replace an OST with the same index ========================================================== 17:23:33 (1776201813) start mds service on oleg409-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg409-server: detected 4 online CPUs by sysfs oleg409-server: Force libcfs to create 2 CPU partitions oleg409-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg409-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg409-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg409-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 792 (time 1776201921.25 total 10.01 last 79.16) - open/close 1478 (time 1776201931.26 total 20.01 last 68.55) - open/close 2214 (time 1776201941.26 total 30.01 last 73.58) - open/close 2989 (time 1776201951.27 total 40.03 last 77.41) - open/close 3733 (time 1776201961.28 total 50.03 last 74.37) - open/close 4526 (time 1776201971.28 total 60.03 last 79.29) - open/close 5259 (time 1776201981.29 total 70.04 last 73.21) - open/close 6077 (time 1776201991.30 total 80.05 last 81.73) - open/close 6860 (time 1776202001.31 total 90.06 last 78.24) - open/close 7617 (time 1776202011.31 total 100.07 last 75.64) - open/close 8341 (time 1776202021.34 total 110.10 last 72.20) - open/close 9010 (time 1776202031.35 total 120.10 last 66.84) - open/close 9789 (time 1776202041.36 total 130.11 last 77.85) total: 10000 open/close in 132.78 seconds: 75.32 ops/second - unlinked 0 (time 1776202048 ; total 1 ; last 1) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 797 (time 1776202141.05 total 10.01 last 79.63) - open/close 1577 (time 1776202151.06 total 20.02 last 77.93) - open/close 2323 (time 1776202161.08 total 30.03 last 74.50) - open/close 3077 (time 1776202171.08 total 40.04 last 75.37) - open/close 3890 (time 1776202181.08 total 50.04 last 81.28) - open/close 4772 (time 1776202191.09 total 60.04 last 88.15) - open/close 5621 (time 1776202201.09 total 70.05 last 84.89) - open/close 6352 (time 1776202211.09 total 80.05 last 73.08) - open/close 7180 (time 1776202221.11 total 90.07 last 82.65) - open/close 7965 (time 1776202231.13 total 100.09 last 78.31) - open/close 8672 (time 1776202241.15 total 110.11 last 70.59) - open/close 9432 (time 1776202251.15 total 120.11 last 75.99) total: 10000 open/close in 127.14 seconds: 78.65 ops/second - unlinked 0 (time 1776202262 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 756 (time 1776202349.89 total 10.01 last 75.51) - open/close 1458 (time 1776202359.90 total 20.02 last 70.17) - open/close 2255 (time 1776202369.90 total 30.02 last 79.70) - open/close 3009 (time 1776202379.90 total 40.02 last 75.37) - open/close 3737 (time 1776202389.90 total 50.02 last 72.80) - open/close 4517 (time 1776202399.91 total 60.03 last 77.93) - open/close 5310 (time 1776202409.92 total 70.04 last 79.22) - open/close 6103 (time 1776202419.93 total 80.05 last 79.22) - open/close 6853 (time 1776202429.94 total 90.06 last 74.90) - open/close 7434 (time 1776202439.95 total 100.07 last 58.06) - open/close 8024 (time 1776202449.96 total 110.08 last 58.97) - open/close 8764 (time 1776202459.96 total 120.08 last 73.95) - open/close 9528 (time 1776202469.97 total 130.09 last 76.35) total: 10000 open/close in 136.40 seconds: 73.31 ops/second - unlinked 0 (time 1776202479 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 811 (time 1776202571.37 total 10.00 last 81.07) - open/close 1547 (time 1776202581.38 total 20.01 last 73.53) - open/close 2327 (time 1776202591.38 total 30.02 last 77.98) - open/close 3096 (time 1776202601.39 total 40.03 last 76.80) - open/close 3847 (time 1776202611.41 total 50.04 last 74.98) - open/close 4608 (time 1776202621.41 total 60.05 last 76.07) - open/close 5432 (time 1776202631.42 total 70.05 last 82.35) - open/close 6192 (time 1776202641.42 total 80.06 last 76.00) - open/close 6916 (time 1776202651.43 total 90.06 last 72.34) - open/close 7673 (time 1776202661.44 total 100.08 last 75.58) - open/close 8438 (time 1776202671.45 total 110.08 last 76.46) - open/close 9265 (time 1776202681.45 total 120.09 last 82.70) total: 10000 open/close in 129.60 seconds: 77.16 ops/second - unlinked 0 (time 1776202694 ; total 1 ; last 1) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 688 (time 1776202781.19 total 10.00 last 68.77) - open/close 1473 (time 1776202791.20 total 20.01 last 78.42) - open/close 2315 (time 1776202801.22 total 30.03 last 84.09) - open/close 2996 (time 1776202811.22 total 40.03 last 68.08) - open/close 3778 (time 1776202821.22 total 50.03 last 78.18) - open/close 4577 (time 1776202831.23 total 60.04 last 79.86) - open/close 5357 (time 1776202841.23 total 70.04 last 77.98) - open/close 6111 (time 1776202851.24 total 80.05 last 75.34) - open/close 6897 (time 1776202861.25 total 90.06 last 78.50) - open/close 7644 (time 1776202871.26 total 100.07 last 74.60) - open/close 8341 (time 1776202881.28 total 110.09 last 69.60) - open/close 9108 (time 1776202891.28 total 120.09 last 76.70) - open/close 9868 (time 1776202901.28 total 130.09 last 76.00) total: 10000 open/close in 131.83 seconds: 75.86 ops/second - unlinked 0 (time 1776202907 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 731 (time 1776202998.77 total 10.00 last 73.08) - open/close 1487 (time 1776203008.77 total 20.01 last 75.59) - open/close 2211 (time 1776203018.79 total 30.02 last 72.32) - open/close 2948 (time 1776203028.80 total 40.03 last 73.61) - open/close 3709 (time 1776203038.81 total 50.04 last 76.03) - open/close 4450 (time 1776203048.82 total 60.05 last 74.03) - open/close 5250 (time 1776203058.83 total 70.06 last 79.91) - open/close 5999 (time 1776203068.84 total 80.07 last 74.84) - open/close 6788 (time 1776203078.85 total 90.08 last 78.81) - open/close 7402 (time 1776203088.86 total 100.09 last 61.30) - open/close 8145 (time 1776203098.87 total 110.10 last 74.23) - open/close 8794 (time 1776203108.89 total 120.12 last 64.78) - open/close 9412 (time 1776203118.90 total 130.13 last 61.76) total: 10000 open/close in 138.19 seconds: 72.36 ops/second - unlinked 0 (time 1776203131 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 717 (time 1776203226.21 total 10.01 last 71.64) - open/close 1376 (time 1776203236.22 total 20.02 last 65.81) - open/close 2104 (time 1776203246.22 total 30.03 last 72.76) - open/close 2813 (time 1776203256.25 total 40.05 last 70.75) - open/close 3611 (time 1776203266.25 total 50.05 last 79.79) - open/close 4336 (time 1776203276.25 total 60.05 last 72.50) - open/close 4984 (time 1776203286.29 total 70.09 last 64.56) - open/close 5600 (time 1776203296.29 total 80.10 last 61.56) - open/close 6312 (time 1776203306.30 total 90.11 last 71.12) - open/close 7044 (time 1776203316.31 total 100.11 last 73.16) - open/close 7783 (time 1776203326.31 total 110.12 last 73.86) - open/close 8433 (time 1776203336.32 total 120.12 last 64.97) - open/close 9094 (time 1776203346.33 total 130.13 last 66.02) - open/close 9775 (time 1776203356.33 total 140.14 last 68.08) total: 10000 open/close in 143.28 seconds: 69.79 ops/second - unlinked 0 (time 1776203364 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 482 (time 1776203465.62 total 10.03 last 48.03) - open/close 1120 (time 1776203475.64 total 20.05 last 63.69) - open/close 1786 (time 1776203485.65 total 30.06 last 66.52) - open/close 2532 (time 1776203495.65 total 40.06 last 74.60) - open/close 3241 (time 1776203505.65 total 50.06 last 70.89) - open/close 3901 (time 1776203515.66 total 60.07 last 65.96) - open/close 4636 (time 1776203525.67 total 70.08 last 73.45) - open/close 5319 (time 1776203535.67 total 80.08 last 68.26) - open/close 5990 (time 1776203545.68 total 90.09 last 67.06) - open/close 6736 (time 1776203555.68 total 100.09 last 74.60) - open/close 7412 (time 1776203565.69 total 110.10 last 67.52) - open/close 8154 (time 1776203575.69 total 120.10 last 74.20) - open/close 8873 (time 1776203585.70 total 130.11 last 71.85) - open/close 9625 (time 1776203595.70 total 140.11 last 75.19) total: 10000 open/close in 145.15 seconds: 68.89 ops/second - unlinked 0 (time 1776203605 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 717 (time 1776203699.10 total 10.00 last 71.69) - open/close 1334 (time 1776203709.10 total 20.01 last 61.65) - open/close 1938 (time 1776203719.11 total 30.02 last 60.35) - open/close 2627 (time 1776203729.11 total 40.02 last 68.89) - open/close 3330 (time 1776203739.13 total 50.03 last 70.21) - open/close 4010 (time 1776203749.15 total 60.05 last 67.87) - open/close 4664 (time 1776203759.15 total 70.06 last 65.36) - open/close 5285 (time 1776203769.16 total 80.07 last 62.05) - open/close 5928 (time 1776203779.16 total 90.07 last 64.27) - open/close 6607 (time 1776203789.17 total 100.07 last 67.89) - open/close 7290 (time 1776203799.17 total 110.08 last 68.27) - open/close 8032 (time 1776203809.17 total 120.08 last 74.20) - open/close 8809 (time 1776203819.18 total 130.09 last 77.61) - open/close 9524 (time 1776203829.19 total 140.09 last 71.47) total: 10000 open/close in 145.95 seconds: 68.52 ops/second - unlinked 0 (time 1776203838 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 792 (time 1776203929.77 total 10.01 last 79.15) - open/close 1625 (time 1776203939.78 total 20.01 last 83.24) - open/close 2431 (time 1776203949.79 total 30.02 last 80.54) - open/close 3157 (time 1776203959.79 total 40.02 last 72.60) - open/close 3926 (time 1776203969.79 total 50.02 last 76.88) - open/close 4730 (time 1776203979.79 total 60.03 last 80.37) - open/close 5515 (time 1776203989.80 total 70.03 last 78.44) - open/close 6344 (time 1776203999.81 total 80.04 last 82.87) - open/close 7088 (time 1776204009.81 total 90.04 last 74.39) - open/close 7803 (time 1776204019.81 total 100.04 last 71.48) - open/close 8564 (time 1776204029.81 total 110.05 last 76.06) - open/close 9322 (time 1776204039.82 total 120.05 last 75.75) total: 10000 open/close in 129.19 seconds: 77.41 ops/second - unlinked 0 (time 1776204053 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.109@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg409-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg409-server: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg409-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg409-server: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg409-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg409-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local oleg409-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg409-server: LNET ready to unload oleg409-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2