== conf-sanity test 69: replace an OST with the same index ========================================================== 07:49:08 (1775216948) start mds service on oleg323-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg323-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg323-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 736 (time 1775217047.54 total 10.01 last 73.55) - open/close 1473 (time 1775217057.56 total 20.02 last 73.59) - open/close 2282 (time 1775217067.56 total 30.02 last 80.88) - open/close 3102 (time 1775217077.57 total 40.03 last 81.92) - open/close 3975 (time 1775217087.59 total 50.06 last 87.08) - open/close 4744 (time 1775217097.60 total 60.06 last 76.87) - open/close 5555 (time 1775217107.60 total 70.07 last 81.08) - open/close 6344 (time 1775217117.60 total 80.07 last 78.89) - open/close 7209 (time 1775217127.61 total 90.07 last 86.46) - open/close 7967 (time 1775217137.61 total 100.07 last 75.79) - open/close 8761 (time 1775217147.61 total 110.08 last 79.37) - open/close 9535 (time 1775217157.61 total 120.08 last 77.37) total: 10000 open/close in 125.61 seconds: 79.61 ops/second - unlinked 0 (time 1775217167 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 841 (time 1775217254.81 total 10.00 last 84.07) - open/close 1803 (time 1775217264.81 total 20.00 last 96.19) - open/close 2761 (time 1775217274.83 total 30.02 last 95.69) - open/close 3559 (time 1775217284.83 total 40.02 last 79.74) - open/close 4398 (time 1775217294.85 total 50.05 last 83.73) - open/close 5188 (time 1775217304.85 total 60.05 last 78.99) - open/close 5969 (time 1775217314.86 total 70.05 last 78.09) - open/close 6818 (time 1775217324.86 total 80.06 last 84.83) - open/close 7679 (time 1775217334.87 total 90.06 last 86.03) - open/close 8529 (time 1775217344.87 total 100.06 last 84.99) - open/close 9375 (time 1775217354.88 total 110.07 last 84.57) total: 10000 open/close in 117.70 seconds: 84.96 ops/second - unlinked 0 (time 1775217366 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 851 (time 1775217453.60 total 10.00 last 85.08) - open/close 1677 (time 1775217463.61 total 20.01 last 82.54) - open/close 2480 (time 1775217473.63 total 30.03 last 80.17) - open/close 3267 (time 1775217483.63 total 40.03 last 78.65) - open/close 4066 (time 1775217493.64 total 50.04 last 79.87) - open/close 4850 (time 1775217503.64 total 60.04 last 78.38) - open/close 5681 (time 1775217513.65 total 70.05 last 83.04) - open/close 6480 (time 1775217523.65 total 80.05 last 79.88) - open/close 7253 (time 1775217533.65 total 90.05 last 77.27) - open/close 8090 (time 1775217543.66 total 100.06 last 83.66) - open/close 9012 (time 1775217553.66 total 110.06 last 92.17) - open/close 9835 (time 1775217563.66 total 120.06 last 82.27) total: 10000 open/close in 122.03 seconds: 81.95 ops/second - unlinked 0 (time 1775217569 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 891 (time 1775217651.97 total 10.00 last 89.10) - open/close 1658 (time 1775217661.98 total 20.01 last 76.60) - open/close 2505 (time 1775217671.99 total 30.02 last 84.61) - open/close 3425 (time 1775217682.00 total 40.03 last 91.98) - open/close 4337 (time 1775217692.00 total 50.03 last 91.19) - open/close 5208 (time 1775217702.00 total 60.03 last 87.08) - open/close 6070 (time 1775217712.01 total 70.04 last 86.07) - open/close 6950 (time 1775217722.03 total 80.06 last 87.90) - open/close 7756 (time 1775217732.03 total 90.06 last 80.59) - open/close 8499 (time 1775217742.03 total 100.06 last 74.29) - open/close 9244 (time 1775217752.04 total 110.07 last 74.45) total: 10000 open/close in 119.30 seconds: 83.82 ops/second - unlinked 0 (time 1775217765 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 865 (time 1775217847.74 total 10.00 last 86.49) - open/close 1674 (time 1775217857.74 total 20.01 last 80.85) - open/close 2413 (time 1775217867.75 total 30.01 last 73.84) - open/close 3184 (time 1775217877.76 total 40.03 last 77.02) - open/close 3887 (time 1775217887.77 total 50.04 last 70.22) - open/close 4639 (time 1775217897.78 total 60.04 last 75.15) - open/close 5388 (time 1775217907.78 total 70.05 last 74.86) - open/close 6189 (time 1775217917.79 total 80.05 last 80.06) - open/close 7015 (time 1775217927.80 total 90.07 last 82.47) - open/close 7889 (time 1775217937.80 total 100.07 last 87.38) - open/close 8821 (time 1775217947.81 total 110.08 last 93.15) - open/close 9694 (time 1775217957.81 total 120.08 last 87.27) total: 10000 open/close in 123.52 seconds: 80.96 ops/second - unlinked 0 (time 1775217965 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 838 (time 1775218047.74 total 10.00 last 83.76) - open/close 1647 (time 1775218057.74 total 20.00 last 80.90) - open/close 2539 (time 1775218067.75 total 30.01 last 89.14) - open/close 3369 (time 1775218077.75 total 40.01 last 83.00) - open/close 4259 (time 1775218087.76 total 50.02 last 88.88) - open/close 5151 (time 1775218097.76 total 60.03 last 89.17) - open/close 6029 (time 1775218107.77 total 70.03 last 87.76) - open/close 6911 (time 1775218117.77 total 80.04 last 88.14) - open/close 7762 (time 1775218127.79 total 90.05 last 85.00) - open/close 8585 (time 1775218137.80 total 100.06 last 82.22) - open/close 9430 (time 1775218147.80 total 110.07 last 84.43) total: 10000 open/close in 116.90 seconds: 85.55 ops/second - unlinked 0 (time 1775218158 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 780 (time 1775218244.26 total 10.01 last 77.95) - open/close 1581 (time 1775218254.26 total 20.01 last 80.08) - open/close 2378 (time 1775218264.26 total 30.01 last 79.66) - open/close 3242 (time 1775218274.27 total 40.02 last 86.34) - open/close 4127 (time 1775218284.29 total 50.04 last 88.31) - open/close 4979 (time 1775218294.29 total 60.04 last 85.19) - open/close 5798 (time 1775218304.31 total 70.06 last 81.75) - open/close 6553 (time 1775218314.31 total 80.06 last 75.49) - open/close 7322 (time 1775218324.32 total 90.07 last 76.86) - open/close 8166 (time 1775218334.32 total 100.07 last 84.35) - open/close 8857 (time 1775218344.33 total 110.08 last 69.09) - open/close 9758 (time 1775218354.33 total 120.08 last 90.10) total: 10000 open/close in 123.33 seconds: 81.09 ops/second - unlinked 0 (time 1775218362 ; total 1 ; last 1) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 880 (time 1775218446.33 total 10.01 last 87.91) - open/close 1781 (time 1775218456.35 total 20.03 last 89.95) - open/close 2629 (time 1775218466.35 total 30.03 last 84.80) - open/close 3521 (time 1775218476.36 total 40.04 last 89.12) - open/close 4436 (time 1775218486.36 total 50.04 last 91.45) - open/close 5283 (time 1775218496.38 total 60.06 last 84.56) - open/close 6094 (time 1775218506.39 total 70.06 last 81.05) - open/close 6939 (time 1775218516.39 total 80.07 last 84.49) - open/close 7797 (time 1775218526.39 total 90.07 last 85.76) - open/close 8686 (time 1775218536.40 total 100.08 last 88.82) - open/close 9533 (time 1775218546.41 total 110.09 last 84.61) total: 10000 open/close in 115.50 seconds: 86.58 ops/second - unlinked 0 (time 1775218555 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 846 (time 1775218638.64 total 10.00 last 84.57) - open/close 1644 (time 1775218648.64 total 20.01 last 79.78) - open/close 2502 (time 1775218658.65 total 30.01 last 85.74) - open/close 3386 (time 1775218668.66 total 40.02 last 88.30) - open/close 4201 (time 1775218678.67 total 50.03 last 81.49) - open/close 4996 (time 1775218688.67 total 60.03 last 79.45) - open/close 5839 (time 1775218698.68 total 70.04 last 84.22) - open/close 6627 (time 1775218708.69 total 80.05 last 78.74) - open/close 7471 (time 1775218718.70 total 90.06 last 84.30) - open/close 8366 (time 1775218728.71 total 100.07 last 89.45) - open/close 9365 (time 1775218738.71 total 110.07 last 99.88) total: 10000 open/close in 117.61 seconds: 85.02 ops/second - unlinked 0 (time 1775218749 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 886 (time 1775218830.75 total 10.00 last 88.58) - open/close 1714 (time 1775218840.75 total 20.00 last 82.80) - open/close 2633 (time 1775218850.76 total 30.01 last 91.83) - open/close 3551 (time 1775218860.77 total 40.02 last 91.69) - open/close 4411 (time 1775218870.77 total 50.02 last 85.97) - open/close 5263 (time 1775218880.79 total 60.04 last 85.09) - open/close 6166 (time 1775218890.79 total 70.04 last 90.28) - open/close 7030 (time 1775218900.80 total 80.05 last 86.32) - open/close 7927 (time 1775218910.80 total 90.05 last 89.68) - open/close 8791 (time 1775218920.81 total 100.06 last 86.31) - open/close 9698 (time 1775218930.82 total 110.07 last 90.66) total: 10000 open/close in 113.47 seconds: 88.13 ops/second - unlinked 0 (time 1775218937 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2