== conf-sanity test 69: replace an OST with the same index ========================================================== 19:32:26 (1784935946) start mds service on oleg151-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg151-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg151-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg151-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 847 (time 1784936064.30 total 10.01 last 84.63) - open/close 1685 (time 1784936074.30 total 20.01 last 83.79) - open/close 2798 (time 1784936084.30 total 30.01 last 111.29) - open/close 3819 (time 1784936094.31 total 40.01 last 102.08) - open/close 4847 (time 1784936104.31 total 50.02 last 102.72) - open/close 5865 (time 1784936114.31 total 60.02 last 101.78) - open/close 6700 (time 1784936124.32 total 70.03 last 83.42) - open/close 7629 (time 1784936134.32 total 80.03 last 92.89) - open/close 8589 (time 1784936144.33 total 90.03 last 95.97) - open/close 9406 (time 1784936154.33 total 100.04 last 81.67) total: 10000 open/close in 106.41 seconds: 93.98 ops/second - unlinked 0 (time 1784936165 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 981 (time 1784936240.00 total 10.00 last 98.07) - open/close 1910 (time 1784936250.00 total 20.01 last 92.88) - open/close 2966 (time 1784936260.01 total 30.01 last 105.53) - open/close 3905 (time 1784936270.02 total 40.02 last 93.84) - open/close 4876 (time 1784936280.02 total 50.03 last 97.04) - open/close 5909 (time 1784936290.03 total 60.03 last 103.24) - open/close 6872 (time 1784936300.03 total 70.04 last 96.25) - open/close 7905 (time 1784936310.04 total 80.04 last 103.24) - open/close 8844 (time 1784936320.04 total 90.05 last 93.87) - open/close 9834 (time 1784936330.05 total 100.05 last 98.94) total: 10000 open/close in 101.93 seconds: 98.10 ops/second - unlinked 0 (time 1784936336 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1008 (time 1784936415.30 total 10.01 last 100.70) - open/close 2047 (time 1784936425.30 total 20.01 last 103.88) - open/close 3134 (time 1784936435.31 total 30.02 last 108.64) - open/close 4078 (time 1784936445.31 total 40.02 last 94.35) - open/close 5086 (time 1784936455.32 total 50.03 last 100.72) - open/close 5984 (time 1784936465.33 total 60.04 last 89.72) - open/close 6948 (time 1784936475.33 total 70.04 last 96.35) - open/close 7939 (time 1784936485.34 total 80.05 last 99.01) - open/close 9081 (time 1784936495.35 total 90.06 last 114.11) total: 10000 open/close in 98.82 seconds: 101.19 ops/second - unlinked 0 (time 1784936508 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 785 (time 1784936584.59 total 10.01 last 78.41) - open/close 1615 (time 1784936594.60 total 20.02 last 82.95) - open/close 2386 (time 1784936604.60 total 30.02 last 77.10) - open/close 3333 (time 1784936614.60 total 40.02 last 94.68) - open/close 4223 (time 1784936624.60 total 50.02 last 89.00) - open/close 5044 (time 1784936634.60 total 60.02 last 82.08) - open/close 6044 (time 1784936644.61 total 70.03 last 99.92) - open/close 7144 (time 1784936654.62 total 80.04 last 109.93) - open/close 7934 (time 1784936664.62 total 90.04 last 78.97) - open/close 8965 (time 1784936674.63 total 100.04 last 103.07) - open/close 9890 (time 1784936684.63 total 110.05 last 92.46) total: 10000 open/close in 111.38 seconds: 89.78 ops/second - unlinked 0 (time 1784936690 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 908 (time 1784936766.68 total 10.01 last 90.69) - open/close 1857 (time 1784936776.69 total 20.02 last 94.81) - open/close 2851 (time 1784936786.69 total 30.03 last 99.36) - open/close 3953 (time 1784936796.69 total 40.03 last 110.18) - open/close 4854 (time 1784936806.70 total 50.03 last 90.08) - open/close 5853 (time 1784936816.71 total 60.05 last 99.74) - open/close 6834 (time 1784936826.72 total 70.06 last 97.99) - open/close 7909 (time 1784936836.73 total 80.06 last 107.42) - open/close 8971 (time 1784936846.74 total 90.07 last 106.12) - open/close 9931 (time 1784936856.74 total 100.08 last 95.96) total: 10000 open/close in 100.70 seconds: 99.31 ops/second - unlinked 0 (time 1784936860 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 837 (time 1784936933.25 total 10.02 last 83.54) - open/close 1639 (time 1784936943.25 total 20.02 last 80.16) - open/close 2724 (time 1784936953.26 total 30.03 last 108.43) - open/close 3782 (time 1784936963.26 total 40.03 last 105.78) - open/close 4701 (time 1784936973.26 total 50.04 last 91.86) - open/close 5481 (time 1784936983.28 total 60.05 last 77.87) - open/close 6125 (time 1784936993.30 total 70.08 last 64.26) - open/close 6923 (time 1784937003.31 total 80.09 last 79.70) - open/close 7985 (time 1784937013.33 total 90.10 last 106.03) - open/close 8650 (time 1784937023.35 total 100.13 last 66.34) - open/close 9311 (time 1784937033.36 total 110.13 last 66.07) total: 10000 open/close in 119.34 seconds: 83.79 ops/second - unlinked 0 (time 1784937045 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 760 (time 1784937129.42 total 10.00 last 75.97) - open/close 1773 (time 1784937139.42 total 20.01 last 101.27) - open/close 2608 (time 1784937149.43 total 30.02 last 83.43) - open/close 3360 (time 1784937159.43 total 40.02 last 75.20) - open/close 4194 (time 1784937169.44 total 50.02 last 83.33) - open/close 5140 (time 1784937179.44 total 60.03 last 94.56) - open/close 6080 (time 1784937189.45 total 70.03 last 93.97) - open/close 7063 (time 1784937199.46 total 80.05 last 98.14) - open/close 8049 (time 1784937209.47 total 90.05 last 98.57) - open/close 9091 (time 1784937219.47 total 100.05 last 104.16) total: 10000 open/close in 108.45 seconds: 92.21 ops/second - unlinked 0 (time 1784937232 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 975 (time 1784937307.25 total 10.00 last 97.48) - open/close 1929 (time 1784937317.26 total 20.01 last 95.31) - open/close 2775 (time 1784937327.26 total 30.02 last 84.55) - open/close 3726 (time 1784937337.27 total 40.02 last 95.02) - open/close 4615 (time 1784937347.27 total 50.03 last 88.87) - open/close 5563 (time 1784937357.28 total 60.03 last 94.77) - open/close 6210 (time 1784937367.29 total 70.04 last 64.64) - open/close 7015 (time 1784937377.30 total 80.05 last 80.42) - open/close 7750 (time 1784937387.32 total 90.07 last 73.35) - open/close 8374 (time 1784937397.33 total 100.08 last 62.32) - open/close 8933 (time 1784937407.34 total 110.09 last 55.87) - open/close 9877 (time 1784937417.34 total 120.10 last 94.34) total: 10000 open/close in 121.10 seconds: 82.57 ops/second - unlinked 0 (time 1784937421 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 819 (time 1784937499.86 total 10.01 last 81.82) - open/close 1649 (time 1784937509.87 total 20.03 last 82.87) - open/close 2323 (time 1784937519.90 total 30.05 last 67.24) - open/close 3070 (time 1784937529.90 total 40.05 last 74.69) - open/close 3875 (time 1784937539.90 total 50.05 last 80.48) - open/close 4896 (time 1784937549.90 total 60.06 last 102.07) - open/close 5844 (time 1784937559.91 total 70.06 last 94.77) - open/close 6815 (time 1784937569.91 total 80.06 last 97.05) - open/close 7745 (time 1784937579.92 total 90.08 last 92.87) - open/close 8418 (time 1784937589.95 total 100.10 last 67.15) - open/close 9356 (time 1784937599.95 total 110.10 last 93.77) total: 10000 open/close in 116.70 seconds: 85.69 ops/second - unlinked 0 (time 1784937611 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 811 (time 1784937696.30 total 10.02 last 80.97) - open/close 1563 (time 1784937706.30 total 20.02 last 75.18) - open/close 2180 (time 1784937716.31 total 30.04 last 61.60) - open/close 2896 (time 1784937726.32 total 40.04 last 71.57) - open/close 3684 (time 1784937736.32 total 50.04 last 78.79) - open/close 4545 (time 1784937746.33 total 60.05 last 86.03) - open/close 5317 (time 1784937756.33 total 70.05 last 77.17) - open/close 6120 (time 1784937766.34 total 80.06 last 80.25) - open/close 6868 (time 1784937776.34 total 90.06 last 74.80) - open/close 7843 (time 1784937786.36 total 100.08 last 97.34) - open/close 8891 (time 1784937796.36 total 110.08 last 104.77) - open/close 9789 (time 1784937806.40 total 120.12 last 89.42) total: 10000 open/close in 122.66 seconds: 81.53 ops/second - unlinked 0 (time 1784937813 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg151-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 start ost1 service on oleg151-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server unloading modules via unload_modules_local on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local oleg151-server: modules unloaded. pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2