== conf-sanity test 69: replace an OST with the same index ========================================================== 18:44:04 (1787697844) start mds service on oleg603-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg603-server' oleg603-server: oleg603-server.virtnet: executing load_modules_local oleg603-server: Loading modules from /home/green/git/lustre-release/lustre oleg603-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg603-server: detected 4 online CPUs by sysfs oleg603-server: MODOPTS_LIBCFS= oleg603-server: Force libcfs to create 2 CPU partitions oleg603-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg603-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg603-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg603-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg603-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 978 (time 1787697944.68 total 10.01 last 97.73) - open/close 1968 (time 1787697954.69 total 20.02 last 98.89) - open/close 2950 (time 1787697964.69 total 30.02 last 98.15) - open/close 3998 (time 1787697974.70 total 40.03 last 104.70) - open/close 5074 (time 1787697984.70 total 50.03 last 107.59) - open/close 6147 (time 1787697994.70 total 60.04 last 107.28) - open/close 7167 (time 1787698004.71 total 70.04 last 101.94) - open/close 8100 (time 1787698014.72 total 80.05 last 93.25) - open/close 9019 (time 1787698024.73 total 90.06 last 91.78) - open/close 9882 (time 1787698034.74 total 100.07 last 86.18) total: 10000 open/close in 101.23 seconds: 98.78 ops/second - unlinked 0 (time 1787698040 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 907 (time 1787698110.10 total 10.00 last 90.66) - open/close 1800 (time 1787698120.11 total 20.01 last 89.22) - open/close 2552 (time 1787698130.12 total 30.02 last 75.16) - open/close 3376 (time 1787698140.13 total 40.03 last 82.34) - open/close 4386 (time 1787698150.14 total 50.04 last 100.86) - open/close 5322 (time 1787698160.15 total 60.05 last 93.52) - open/close 6305 (time 1787698170.15 total 70.05 last 98.30) - open/close 7414 (time 1787698180.16 total 80.06 last 110.81) - open/close 8340 (time 1787698190.16 total 90.06 last 92.54) - open/close 9363 (time 1787698200.17 total 100.07 last 102.28) total: 10000 open/close in 105.92 seconds: 94.41 ops/second - unlinked 0 (time 1787698210 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 731 (time 1787698295.75 total 10.01 last 73.03) - open/close 1522 (time 1787698305.75 total 20.01 last 79.10) - open/close 2416 (time 1787698315.75 total 30.01 last 89.39) - open/close 3367 (time 1787698325.76 total 40.02 last 95.02) - open/close 4321 (time 1787698335.77 total 50.03 last 95.27) - open/close 5221 (time 1787698345.77 total 60.04 last 89.98) - open/close 6168 (time 1787698355.78 total 70.05 last 94.60) - open/close 7185 (time 1787698365.79 total 80.06 last 101.59) - open/close 8160 (time 1787698375.80 total 90.06 last 97.46) - open/close 9162 (time 1787698385.82 total 100.08 last 99.96) total: 10000 open/close in 109.10 seconds: 91.66 ops/second - unlinked 0 (time 1787698397 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 983 (time 1787698471.28 total 10.00 last 98.29) - open/close 1999 (time 1787698481.28 total 20.00 last 101.60) - open/close 2965 (time 1787698491.29 total 30.01 last 96.52) - open/close 3867 (time 1787698501.29 total 40.02 last 90.15) - open/close 4822 (time 1787698511.31 total 50.04 last 95.32) - open/close 5732 (time 1787698521.35 total 60.07 last 90.67) - open/close 6631 (time 1787698531.35 total 70.07 last 89.90) - open/close 7617 (time 1787698541.36 total 80.08 last 98.51) - open/close 8587 (time 1787698551.36 total 90.08 last 96.96) - open/close 9545 (time 1787698561.37 total 100.09 last 95.78) total: 10000 open/close in 105.31 seconds: 94.96 ops/second - unlinked 0 (time 1787698571 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1008 (time 1787698645.97 total 10.01 last 100.74) - open/close 1944 (time 1787698655.97 total 20.01 last 93.58) - open/close 2909 (time 1787698665.98 total 30.02 last 96.36) - open/close 3894 (time 1787698675.99 total 40.03 last 98.44) - open/close 4910 (time 1787698685.99 total 50.03 last 101.55) - open/close 5883 (time 1787698696.00 total 60.04 last 97.23) - open/close 6755 (time 1787698706.01 total 70.05 last 87.13) - open/close 7691 (time 1787698716.01 total 80.05 last 93.54) - open/close 8562 (time 1787698726.02 total 90.06 last 87.07) - open/close 9586 (time 1787698736.02 total 100.06 last 102.35) total: 10000 open/close in 104.32 seconds: 95.86 ops/second - unlinked 0 (time 1787698744 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1005 (time 1787698815.57 total 10.01 last 100.42) - open/close 1959 (time 1787698825.58 total 20.02 last 95.25) - open/close 2980 (time 1787698835.59 total 30.03 last 102.04) - open/close 3903 (time 1787698845.59 total 40.03 last 92.25) - open/close 4901 (time 1787698855.61 total 50.05 last 99.67) - open/close 5851 (time 1787698865.61 total 60.06 last 94.92) - open/close 6798 (time 1787698875.64 total 70.08 last 94.46) - open/close 7796 (time 1787698885.64 total 80.09 last 99.76) - open/close 8704 (time 1787698895.66 total 90.10 last 90.65) - open/close 9723 (time 1787698905.66 total 100.11 last 101.86) total: 10000 open/close in 102.80 seconds: 97.28 ops/second - unlinked 0 (time 1787698911 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 941 (time 1787698980.62 total 10.01 last 94.04) - open/close 1934 (time 1787698990.62 total 20.01 last 99.29) - open/close 2943 (time 1787699000.62 total 30.01 last 100.87) - open/close 3929 (time 1787699010.63 total 40.02 last 98.50) - open/close 4906 (time 1787699020.64 total 50.03 last 97.65) - open/close 5910 (time 1787699030.65 total 60.04 last 100.31) - open/close 6817 (time 1787699040.65 total 70.04 last 90.67) - open/close 7780 (time 1787699050.65 total 80.04 last 96.25) - open/close 8697 (time 1787699060.65 total 90.04 last 91.70) - open/close 9732 (time 1787699070.66 total 100.05 last 103.44) total: 10000 open/close in 103.06 seconds: 97.03 ops/second - unlinked 0 (time 1787699077 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1045 (time 1787699146.51 total 10.00 last 104.48) - open/close 2121 (time 1787699156.51 total 20.00 last 107.59) - open/close 3158 (time 1787699166.51 total 30.01 last 103.62) - open/close 4173 (time 1787699176.52 total 40.01 last 101.49) - open/close 5332 (time 1787699186.53 total 50.02 last 115.76) - open/close 6374 (time 1787699196.53 total 60.03 last 104.17) - open/close 7502 (time 1787699206.54 total 70.03 last 112.72) - open/close 8608 (time 1787699216.55 total 80.04 last 110.48) - open/close 9782 (time 1787699226.55 total 90.05 last 117.35) total: 10000 open/close in 92.09 seconds: 108.59 ops/second - unlinked 0 (time 1787699231 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1253 (time 1787699296.50 total 10.01 last 125.15) - open/close 2302 (time 1787699306.50 total 20.01 last 104.90) - open/close 3331 (time 1787699316.50 total 30.02 last 102.87) - open/close 4406 (time 1787699326.51 total 40.02 last 107.46) - open/close 5577 (time 1787699336.51 total 50.02 last 117.07) - open/close 6790 (time 1787699346.51 total 60.03 last 121.22) - open/close 7990 (time 1787699356.52 total 70.03 last 119.93) - open/close 8976 (time 1787699366.53 total 80.04 last 98.53) total: 10000 open/close in 89.76 seconds: 111.41 ops/second - unlinked 0 (time 1787699379 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1003 (time 1787699449.80 total 10.00 last 100.29) - open/close 1999 (time 1787699459.81 total 20.01 last 99.53) - open/close 2941 (time 1787699469.82 total 30.02 last 94.07) - open/close 3936 (time 1787699479.82 total 40.02 last 99.50) - open/close 4923 (time 1787699489.82 total 50.03 last 98.65) - open/close 5850 (time 1787699499.83 total 60.03 last 92.66) - open/close 6904 (time 1787699509.83 total 70.03 last 105.38) - open/close 7875 (time 1787699519.84 total 80.04 last 97.01) - open/close 8822 (time 1787699529.84 total 90.04 last 94.68) - open/close 9849 (time 1787699539.85 total 100.05 last 102.60) total: 10000 open/close in 101.44 seconds: 98.58 ops/second - unlinked 0 (time 1787699544 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg603-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg603-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg603-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg603-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg603-server: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg603-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg603-server: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg603-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg603-client: oleg603-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg603-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg603-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg603-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.103@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.103@tcp sys.timeout=20 start ost1 service on oleg603-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg603-server: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg603-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg603-server: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg603-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg603-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg603-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg603-server stop mds service on oleg603-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg603-server stop mds service on oleg603-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg603-server unloading modules via unload_modules_local on: 'oleg603-server' oleg603-server: oleg603-server.virtnet: executing unload_modules_local oleg603-server: modules unloaded. pdsh@oleg603-client: oleg603-client: ssh exited with exit code 2 pdsh@oleg603-client: oleg603-server: ssh exited with exit code 2 pdsh@oleg603-client: oleg603-client: ssh exited with exit code 2 pdsh@oleg603-client: oleg603-server: ssh exited with exit code 2