== conf-sanity test 69: replace an OST with the same index ========================================================== 19:18:41 (1787440721) start mds service on oleg630-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg630-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg630-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg630-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg630-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 980 (time 1787440819.22 total 10.00 last 97.97) - open/close 2024 (time 1787440829.22 total 20.00 last 104.39) - open/close 2999 (time 1787440839.22 total 30.01 last 97.46) - open/close 3976 (time 1787440849.23 total 40.01 last 97.66) - open/close 4967 (time 1787440859.23 total 50.01 last 99.08) - open/close 6010 (time 1787440869.23 total 60.02 last 104.26) - open/close 7054 (time 1787440879.23 total 70.02 last 104.37) - open/close 7935 (time 1787440889.24 total 80.02 last 88.10) - open/close 8719 (time 1787440899.24 total 90.02 last 78.38) - open/close 9620 (time 1787440909.25 total 100.03 last 90.03) total: 10000 open/close in 103.99 seconds: 96.16 ops/second - unlinked 0 (time 1787440916 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 964 (time 1787440987.81 total 10.01 last 96.33) - open/close 1971 (time 1787440997.82 total 20.02 last 100.61) - open/close 2994 (time 1787441007.82 total 30.02 last 102.29) - open/close 3914 (time 1787441017.83 total 40.02 last 91.98) - open/close 4704 (time 1787441027.84 total 50.04 last 78.87) - open/close 5451 (time 1787441037.86 total 60.05 last 74.60) - open/close 6456 (time 1787441047.86 total 70.06 last 100.44) - open/close 7429 (time 1787441057.87 total 80.07 last 97.19) - open/close 8385 (time 1787441067.88 total 90.07 last 95.56) - open/close 9372 (time 1787441077.89 total 100.08 last 98.59) total: 10000 open/close in 106.83 seconds: 93.61 ops/second - unlinked 0 (time 1787441088 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 967 (time 1787441158.54 total 10.00 last 96.69) - open/close 1901 (time 1787441168.54 total 20.01 last 93.36) - open/close 2929 (time 1787441178.54 total 30.01 last 102.76) - open/close 3890 (time 1787441188.55 total 40.02 last 96.01) - open/close 4830 (time 1787441198.56 total 50.03 last 93.93) - open/close 5804 (time 1787441208.57 total 60.04 last 97.28) - open/close 6771 (time 1787441218.58 total 70.05 last 96.62) - open/close 7733 (time 1787441228.59 total 80.05 last 96.16) - open/close 8644 (time 1787441238.59 total 90.06 last 91.08) - open/close 9609 (time 1787441248.60 total 100.06 last 96.43) total: 10000 open/close in 104.38 seconds: 95.81 ops/second - unlinked 0 (time 1787441257 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 990 (time 1787441327.25 total 10.01 last 98.92) - open/close 1994 (time 1787441337.25 total 20.01 last 100.36) - open/close 2986 (time 1787441347.26 total 30.02 last 99.11) - open/close 3924 (time 1787441357.27 total 40.03 last 93.72) - open/close 4837 (time 1787441367.28 total 50.04 last 91.23) - open/close 5703 (time 1787441377.28 total 60.04 last 86.59) - open/close 6673 (time 1787441387.28 total 70.04 last 96.97) - open/close 7614 (time 1787441397.28 total 80.04 last 94.08) - open/close 8489 (time 1787441407.29 total 90.05 last 87.46) - open/close 9462 (time 1787441417.29 total 100.05 last 97.27) total: 10000 open/close in 105.59 seconds: 94.71 ops/second - unlinked 0 (time 1787441426 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1056 (time 1787441495.25 total 10.00 last 105.59) - open/close 2063 (time 1787441505.26 total 20.00 last 100.68) - open/close 3096 (time 1787441515.26 total 30.00 last 103.29) - open/close 4073 (time 1787441525.26 total 40.01 last 97.63) - open/close 5049 (time 1787441535.27 total 50.01 last 97.57) - open/close 6041 (time 1787441545.28 total 60.03 last 99.09) - open/close 6977 (time 1787441555.28 total 70.03 last 93.59) - open/close 8014 (time 1787441565.28 total 80.03 last 103.65) - open/close 9041 (time 1787441575.29 total 90.04 last 102.64) total: 10000 open/close in 99.73 seconds: 100.27 ops/second - unlinked 0 (time 1787441588 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1046 (time 1787441655.20 total 10.02 last 104.39) - open/close 2048 (time 1787441665.21 total 20.03 last 100.12) - open/close 3056 (time 1787441675.21 total 30.03 last 100.75) - open/close 4077 (time 1787441685.22 total 40.04 last 102.04) - open/close 5130 (time 1787441695.22 total 50.04 last 105.29) - open/close 6124 (time 1787441705.23 total 60.05 last 99.33) - open/close 7027 (time 1787441715.24 total 70.06 last 90.21) - open/close 7929 (time 1787441725.25 total 80.07 last 90.11) - open/close 8928 (time 1787441735.25 total 90.07 last 99.89) - open/close 9742 (time 1787441745.25 total 100.07 last 81.39) total: 10000 open/close in 102.94 seconds: 97.15 ops/second - unlinked 0 (time 1787441752 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 978 (time 1787441822.78 total 10.02 last 97.64) - open/close 2019 (time 1787441832.78 total 20.02 last 104.06) - open/close 2945 (time 1787441842.79 total 30.03 last 92.52) - open/close 3846 (time 1787441852.79 total 40.03 last 90.07) - open/close 4799 (time 1787441862.80 total 50.04 last 95.17) - open/close 5684 (time 1787441872.80 total 60.05 last 88.50) - open/close 6656 (time 1787441882.81 total 70.05 last 97.19) - open/close 7694 (time 1787441892.81 total 80.05 last 103.78) - open/close 8619 (time 1787441902.81 total 90.05 last 92.44) - open/close 9591 (time 1787441912.82 total 100.06 last 97.17) total: 10000 open/close in 105.40 seconds: 94.88 ops/second - unlinked 0 (time 1787441922 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 928 (time 1787441992.72 total 10.00 last 92.79) - open/close 1902 (time 1787442002.73 total 20.00 last 97.37) - open/close 2824 (time 1787442012.73 total 30.01 last 92.11) - open/close 3800 (time 1787442022.74 total 40.02 last 97.58) - open/close 4775 (time 1787442032.74 total 50.02 last 97.50) - open/close 5710 (time 1787442042.75 total 60.03 last 93.37) - open/close 6398 (time 1787442052.75 total 70.03 last 68.79) - open/close 7139 (time 1787442062.77 total 80.05 last 73.95) - open/close 8166 (time 1787442072.78 total 90.06 last 102.64) - open/close 9135 (time 1787442082.78 total 100.06 last 96.86) total: 10000 open/close in 109.02 seconds: 91.73 ops/second - unlinked 0 (time 1787442095 ; total 1 ; last 1) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 963 (time 1787442167.32 total 10.01 last 96.21) - open/close 1987 (time 1787442177.32 total 20.01 last 102.36) - open/close 2896 (time 1787442187.33 total 30.02 last 90.85) - open/close 3883 (time 1787442197.33 total 40.02 last 98.67) - open/close 4889 (time 1787442207.33 total 50.02 last 100.58) - open/close 5937 (time 1787442217.34 total 60.03 last 104.73) - open/close 6984 (time 1787442227.34 total 70.03 last 104.66) - open/close 7968 (time 1787442237.35 total 80.04 last 98.35) - open/close 8999 (time 1787442247.36 total 90.05 last 103.01) total: 10000 open/close in 99.73 seconds: 100.27 ops/second - unlinked 0 (time 1787442260 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 957 (time 1787442332.12 total 10.02 last 95.55) - open/close 1975 (time 1787442342.12 total 20.02 last 101.78) - open/close 2916 (time 1787442352.12 total 30.02 last 94.06) - open/close 3932 (time 1787442362.14 total 40.04 last 101.44) - open/close 4978 (time 1787442372.14 total 50.04 last 104.56) - open/close 5958 (time 1787442382.14 total 60.04 last 98.00) - open/close 6963 (time 1787442392.15 total 70.05 last 100.39) - open/close 7948 (time 1787442402.16 total 80.06 last 98.44) - open/close 8881 (time 1787442412.18 total 90.08 last 93.13) - open/close 9864 (time 1787442422.18 total 100.08 last 98.29) total: 10000 open/close in 101.35 seconds: 98.67 ops/second - unlinked 0 (time 1787442426 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.130@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg630-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg630-server: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg630-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg630-server: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg630-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.130@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.130@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.130@tcp sys.timeout=20 start ost1 service on oleg630-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg630-server: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg630-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg630-server: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg630-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg630-server unloading modules via unload_modules_local on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing unload_modules_local oleg630-server: modules unloaded. pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2