== conf-sanity test 69: replace an OST with the same index ========================================================== 00:28:56 (1781584136) start mds service on oleg303-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions oleg303-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg303-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg303-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg303-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg303-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 674 (time 1781584247.67 total 10.01 last 67.35) - open/close 1356 (time 1781584257.67 total 20.01 last 68.20) - open/close 2042 (time 1781584267.70 total 30.03 last 68.44) - open/close 2743 (time 1781584277.70 total 40.04 last 70.05) - open/close 3496 (time 1781584287.71 total 50.04 last 75.27) - open/close 4243 (time 1781584297.72 total 60.06 last 74.60) - open/close 4937 (time 1781584307.73 total 70.07 last 69.32) - open/close 5648 (time 1781584317.74 total 80.07 last 71.06) - open/close 6338 (time 1781584327.74 total 90.08 last 68.99) - open/close 7040 (time 1781584337.74 total 100.08 last 70.20) - open/close 7745 (time 1781584347.74 total 110.08 last 70.49) - open/close 8447 (time 1781584357.75 total 120.08 last 70.17) - open/close 9174 (time 1781584367.79 total 130.12 last 72.39) - open/close 9875 (time 1781584377.81 total 140.15 last 69.94) total: 10000 open/close in 142.16 seconds: 70.34 ops/second - unlinked 0 (time 1781584384 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 656 (time 1781584477.38 total 10.00 last 65.59) - open/close 1403 (time 1781584487.38 total 20.01 last 74.63) - open/close 2182 (time 1781584497.39 total 30.02 last 77.83) - open/close 2950 (time 1781584507.40 total 40.03 last 76.75) - open/close 3785 (time 1781584517.42 total 50.04 last 83.37) - open/close 4580 (time 1781584527.43 total 60.05 last 79.43) - open/close 5347 (time 1781584537.44 total 70.06 last 76.61) - open/close 6133 (time 1781584547.45 total 80.07 last 78.51) - open/close 6913 (time 1781584557.46 total 90.09 last 77.91) - open/close 7716 (time 1781584567.48 total 100.11 last 80.13) - open/close 8487 (time 1781584577.49 total 110.11 last 77.06) - open/close 9278 (time 1781584587.51 total 120.13 last 78.94) total: 10000 open/close in 128.88 seconds: 77.59 ops/second - unlinked 0 (time 1781584600 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 588 (time 1781584690.51 total 10.01 last 58.73) - open/close 1205 (time 1781584700.52 total 20.02 last 61.67) - open/close 1932 (time 1781584710.53 total 30.03 last 72.60) - open/close 2628 (time 1781584720.53 total 40.03 last 69.59) - open/close 3344 (time 1781584730.55 total 50.05 last 71.45) - open/close 4091 (time 1781584740.56 total 60.06 last 74.61) - open/close 4871 (time 1781584750.57 total 70.07 last 77.96) - open/close 5579 (time 1781584760.57 total 80.08 last 70.76) - open/close 6240 (time 1781584770.58 total 90.08 last 66.04) - open/close 6909 (time 1781584780.58 total 100.08 last 66.89) - open/close 7628 (time 1781584790.59 total 110.09 last 71.83) - open/close 8370 (time 1781584800.59 total 120.10 last 74.19) - open/close 9088 (time 1781584810.60 total 130.10 last 71.77) - open/close 9791 (time 1781584820.61 total 140.11 last 70.19) total: 10000 open/close in 143.11 seconds: 69.88 ops/second - unlinked 0 (time 1781584828 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 728 (time 1781584922.91 total 10.00 last 72.78) - open/close 1411 (time 1781584932.92 total 20.01 last 68.27) - open/close 2159 (time 1781584942.92 total 30.01 last 74.76) - open/close 2852 (time 1781584952.93 total 40.02 last 69.26) - open/close 3564 (time 1781584962.95 total 50.04 last 71.03) - open/close 4251 (time 1781584972.96 total 60.05 last 68.66) - open/close 4890 (time 1781584982.97 total 70.06 last 63.83) - open/close 5585 (time 1781584992.97 total 80.06 last 69.47) - open/close 6235 (time 1781585002.99 total 90.08 last 64.89) - open/close 6931 (time 1781585013.00 total 100.09 last 69.52) - open/close 7611 (time 1781585023.01 total 110.10 last 67.94) - open/close 8312 (time 1781585033.02 total 120.11 last 70.03) - open/close 9012 (time 1781585043.02 total 130.11 last 69.99) - open/close 9712 (time 1781585053.03 total 140.11 last 69.97) total: 10000 open/close in 144.09 seconds: 69.40 ops/second - unlinked 0 (time 1781585061 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 668 (time 1781585163.70 total 10.00 last 66.78) - open/close 1333 (time 1781585173.71 total 20.01 last 66.47) - open/close 2070 (time 1781585183.72 total 30.02 last 73.64) - open/close 2681 (time 1781585193.73 total 40.03 last 61.04) - open/close 3313 (time 1781585203.74 total 50.04 last 63.12) - open/close 4017 (time 1781585213.75 total 60.05 last 70.31) - open/close 4768 (time 1781585223.77 total 70.07 last 75.00) - open/close 5423 (time 1781585233.78 total 80.08 last 65.43) - open/close 6051 (time 1781585243.78 total 90.08 last 62.77) - open/close 6686 (time 1781585253.78 total 100.08 last 63.49) - open/close 7329 (time 1781585263.79 total 110.09 last 64.24) - open/close 8000 (time 1781585273.81 total 120.11 last 67.01) - open/close 8623 (time 1781585283.81 total 130.11 last 62.26) - open/close 9288 (time 1781585293.85 total 140.14 last 66.28) - open/close 9969 (time 1781585303.86 total 150.15 last 68.03) total: 10000 open/close in 150.59 seconds: 66.40 ops/second - unlinked 0 (time 1781585308 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 691 (time 1781585405.51 total 10.00 last 69.08) - open/close 1348 (time 1781585415.53 total 20.02 last 65.60) - open/close 2085 (time 1781585425.53 total 30.02 last 73.66) - open/close 2887 (time 1781585435.54 total 40.03 last 80.18) - open/close 3545 (time 1781585445.54 total 50.03 last 65.76) - open/close 4274 (time 1781585455.55 total 60.04 last 72.86) - open/close 5029 (time 1781585465.56 total 70.05 last 75.42) - open/close 5778 (time 1781585475.58 total 80.07 last 74.76) - open/close 6488 (time 1781585485.58 total 90.07 last 70.99) - open/close 7208 (time 1781585495.58 total 100.07 last 71.97) - open/close 7941 (time 1781585505.60 total 110.09 last 73.20) - open/close 8605 (time 1781585515.61 total 120.10 last 66.34) - open/close 9270 (time 1781585525.61 total 130.10 last 66.47) - open/close 9826 (time 1781585535.63 total 140.12 last 55.49) total: 10000 open/close in 143.00 seconds: 69.93 ops/second - unlinked 0 (time 1781585544 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 594 (time 1781585641.31 total 10.01 last 59.32) - open/close 1189 (time 1781585651.32 total 20.02 last 59.44) - open/close 1850 (time 1781585661.33 total 30.04 last 66.01) - open/close 2533 (time 1781585671.34 total 40.04 last 68.27) - open/close 3194 (time 1781585681.35 total 50.05 last 66.05) - open/close 3922 (time 1781585691.35 total 60.06 last 72.74) - open/close 4579 (time 1781585701.36 total 70.06 last 65.67) - open/close 5396 (time 1781585711.36 total 80.07 last 81.67) - open/close 6132 (time 1781585721.36 total 90.07 last 73.60) - open/close 6900 (time 1781585731.38 total 100.08 last 76.69) - open/close 7608 (time 1781585741.39 total 110.09 last 70.71) - open/close 8419 (time 1781585751.40 total 120.10 last 81.02) - open/close 9150 (time 1781585761.41 total 130.11 last 73.05) - open/close 9939 (time 1781585771.41 total 140.11 last 78.89) total: 10000 open/close in 141.08 seconds: 70.88 ops/second - unlinked 0 (time 1781585776 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 663 (time 1781585866.27 total 10.02 last 66.17) - open/close 1342 (time 1781585876.28 total 20.04 last 67.77) - open/close 2077 (time 1781585886.29 total 30.05 last 73.44) - open/close 2807 (time 1781585896.31 total 40.06 last 72.89) - open/close 3589 (time 1781585906.31 total 50.07 last 78.17) - open/close 4320 (time 1781585916.32 total 60.08 last 73.03) - open/close 5048 (time 1781585926.33 total 70.08 last 72.75) - open/close 5794 (time 1781585936.33 total 80.09 last 74.57) - open/close 6566 (time 1781585946.34 total 90.10 last 77.12) - open/close 7287 (time 1781585956.35 total 100.10 last 72.07) - open/close 8127 (time 1781585966.35 total 110.11 last 83.94) - open/close 8899 (time 1781585976.35 total 120.11 last 77.19) - open/close 9679 (time 1781585986.37 total 130.12 last 77.92) total: 10000 open/close in 134.63 seconds: 74.28 ops/second - unlinked 0 (time 1781585995 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 749 (time 1781586084.16 total 10.01 last 74.85) - open/close 1431 (time 1781586094.16 total 20.01 last 68.18) - open/close 2246 (time 1781586104.17 total 30.01 last 81.46) - open/close 3047 (time 1781586114.17 total 40.02 last 80.07) - open/close 3859 (time 1781586124.19 total 50.03 last 81.06) - open/close 4695 (time 1781586134.20 total 60.04 last 83.54) - open/close 5495 (time 1781586144.20 total 70.04 last 79.99) - open/close 6366 (time 1781586154.20 total 80.05 last 87.05) - open/close 7203 (time 1781586164.20 total 90.05 last 83.69) - open/close 8018 (time 1781586174.21 total 100.06 last 81.44) - open/close 8866 (time 1781586184.22 total 110.06 last 84.74) - open/close 9675 (time 1781586194.22 total 120.07 last 80.85) total: 10000 open/close in 124.20 seconds: 80.52 ops/second - unlinked 0 (time 1781586201 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 725 (time 1781586291.67 total 10.01 last 72.43) - open/close 1420 (time 1781586301.68 total 20.02 last 69.43) - open/close 2090 (time 1781586311.70 total 30.03 last 66.91) - open/close 2802 (time 1781586321.71 total 40.04 last 71.14) - open/close 3590 (time 1781586331.71 total 50.04 last 78.78) - open/close 4402 (time 1781586341.72 total 60.05 last 81.11) - open/close 5181 (time 1781586351.73 total 70.06 last 77.85) - open/close 5995 (time 1781586361.73 total 80.07 last 81.37) - open/close 6794 (time 1781586371.73 total 90.07 last 79.89) - open/close 7559 (time 1781586381.73 total 100.07 last 76.49) - open/close 8268 (time 1781586391.76 total 110.09 last 70.72) - open/close 9011 (time 1781586401.76 total 120.10 last 74.25) - open/close 9775 (time 1781586411.76 total 130.10 last 76.39) total: 10000 open/close in 132.92 seconds: 75.23 ops/second - unlinked 0 (time 1781586418 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg303-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg303-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg303-server: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg303-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg303-server: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg303-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg303-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.103@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.103@tcp sys.timeout=20 start ost1 service on oleg303-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg303-server: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg303-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg303-server: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg303-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg303-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg303-server unloading modules via unload_modules_local on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing unload_modules_local oleg303-server: modules unloaded. pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2