== conf-sanity test 69: replace an OST with the same index ========================================================== 17:23:44 (1786829024) start mds service on oleg347-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg347-server: detected 4 online CPUs by sysfs oleg347-server: MODOPTS_LIBCFS= oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg347-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg347-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg347-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 811 (time 1786829137.36 total 10.01 last 81.06) - open/close 1690 (time 1786829147.36 total 20.01 last 87.84) - open/close 2519 (time 1786829157.37 total 30.02 last 82.87) - open/close 3346 (time 1786829167.38 total 40.03 last 82.62) - open/close 4133 (time 1786829177.38 total 50.03 last 78.67) - open/close 4915 (time 1786829187.38 total 60.03 last 78.19) - open/close 5697 (time 1786829197.38 total 70.03 last 78.19) - open/close 6433 (time 1786829207.39 total 80.04 last 73.52) - open/close 7196 (time 1786829217.40 total 90.05 last 76.24) - open/close 8026 (time 1786829227.41 total 100.06 last 82.94) - open/close 8836 (time 1786829237.41 total 110.06 last 80.98) - open/close 9633 (time 1786829247.42 total 120.07 last 79.63) total: 10000 open/close in 125.08 seconds: 79.95 ops/second - unlinked 0 (time 1786829257 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 781 (time 1786829347.34 total 10.00 last 78.09) - open/close 1706 (time 1786829357.34 total 20.01 last 92.42) - open/close 2602 (time 1786829367.36 total 30.02 last 89.50) - open/close 3441 (time 1786829377.37 total 40.04 last 83.75) - open/close 4267 (time 1786829387.39 total 50.06 last 82.44) - open/close 5031 (time 1786829397.41 total 60.07 last 76.29) - open/close 5681 (time 1786829407.42 total 70.09 last 64.90) - open/close 6495 (time 1786829417.43 total 80.10 last 81.34) - open/close 7352 (time 1786829427.45 total 90.12 last 85.51) - open/close 8105 (time 1786829437.47 total 100.13 last 75.17) - open/close 8901 (time 1786829447.47 total 110.14 last 79.57) - open/close 9628 (time 1786829457.48 total 120.15 last 72.64) total: 10000 open/close in 125.51 seconds: 79.68 ops/second - unlinked 0 (time 1786829467 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 879 (time 1786829553.36 total 10.01 last 87.82) - open/close 1766 (time 1786829563.36 total 20.02 last 88.64) - open/close 2676 (time 1786829573.37 total 30.02 last 90.95) - open/close 3546 (time 1786829583.38 total 40.03 last 86.91) - open/close 4305 (time 1786829593.39 total 50.04 last 75.85) - open/close 5108 (time 1786829603.40 total 60.05 last 80.22) - open/close 5878 (time 1786829613.41 total 70.06 last 76.93) - open/close 6702 (time 1786829623.41 total 80.06 last 82.37) - open/close 7557 (time 1786829633.42 total 90.07 last 85.39) - open/close 8419 (time 1786829643.43 total 100.08 last 86.13) - open/close 9304 (time 1786829653.43 total 110.08 last 88.48) total: 10000 open/close in 118.86 seconds: 84.13 ops/second - unlinked 0 (time 1786829666 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 852 (time 1786829745.92 total 10.00 last 85.17) - open/close 1731 (time 1786829755.93 total 20.02 last 87.74) - open/close 2625 (time 1786829765.94 total 30.03 last 89.32) - open/close 3449 (time 1786829775.95 total 40.03 last 82.38) - open/close 4407 (time 1786829785.95 total 50.04 last 95.78) - open/close 5300 (time 1786829795.95 total 60.04 last 89.30) - open/close 6189 (time 1786829805.96 total 70.04 last 88.82) - open/close 7047 (time 1786829815.96 total 80.05 last 85.74) - open/close 7895 (time 1786829825.97 total 90.06 last 84.75) - open/close 8818 (time 1786829835.98 total 100.06 last 92.24) - open/close 9710 (time 1786829845.99 total 110.07 last 89.10) total: 10000 open/close in 113.84 seconds: 87.84 ops/second - unlinked 0 (time 1786829853 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 932 (time 1786829942.88 total 10.01 last 93.10) - open/close 1843 (time 1786829952.89 total 20.02 last 91.06) - open/close 2716 (time 1786829962.91 total 30.03 last 87.16) - open/close 3456 (time 1786829972.91 total 40.04 last 73.97) - open/close 4379 (time 1786829982.92 total 50.05 last 92.18) - open/close 5262 (time 1786829992.94 total 60.06 last 88.17) - open/close 6092 (time 1786830002.94 total 70.07 last 82.95) - open/close 6907 (time 1786830012.95 total 80.08 last 81.44) - open/close 7749 (time 1786830022.96 total 90.08 last 84.15) - open/close 8533 (time 1786830032.96 total 100.09 last 78.34) - open/close 9381 (time 1786830042.97 total 110.09 last 84.78) total: 10000 open/close in 117.27 seconds: 85.27 ops/second - unlinked 0 (time 1786830054 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 857 (time 1786830145.78 total 10.00 last 85.66) - open/close 1742 (time 1786830155.79 total 20.01 last 88.45) - open/close 2605 (time 1786830165.80 total 30.03 last 86.16) - open/close 3449 (time 1786830175.81 total 40.03 last 84.36) - open/close 4409 (time 1786830185.82 total 50.04 last 95.93) - open/close 5297 (time 1786830195.82 total 60.04 last 88.75) - open/close 6090 (time 1786830205.82 total 70.04 last 79.30) - open/close 6826 (time 1786830215.84 total 80.06 last 73.49) - open/close 7551 (time 1786830225.84 total 90.06 last 72.49) - open/close 8455 (time 1786830235.85 total 100.07 last 90.33) - open/close 9514 (time 1786830245.85 total 110.07 last 105.89) total: 10000 open/close in 115.77 seconds: 86.38 ops/second - unlinked 0 (time 1786830256 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 842 (time 1786830343.12 total 10.02 last 84.03) - open/close 1628 (time 1786830353.12 total 20.03 last 78.56) - open/close 2445 (time 1786830363.13 total 30.03 last 81.66) - open/close 3333 (time 1786830373.16 total 40.06 last 88.55) - open/close 4182 (time 1786830383.16 total 50.06 last 84.85) - open/close 4865 (time 1786830393.17 total 60.07 last 68.27) - open/close 5500 (time 1786830403.17 total 70.07 last 63.47) - open/close 6302 (time 1786830413.18 total 80.08 last 80.15) - open/close 7108 (time 1786830423.19 total 90.09 last 80.53) - open/close 7999 (time 1786830433.20 total 100.10 last 89.02) - open/close 8913 (time 1786830443.20 total 110.10 last 91.36) - open/close 9811 (time 1786830453.20 total 120.10 last 89.79) total: 10000 open/close in 122.40 seconds: 81.70 ops/second - unlinked 0 (time 1786830460 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 809 (time 1786830546.56 total 10.01 last 80.79) - open/close 1601 (time 1786830556.57 total 20.03 last 79.10) - open/close 2472 (time 1786830566.58 total 30.03 last 87.06) - open/close 3336 (time 1786830576.59 total 40.05 last 86.27) - open/close 4052 (time 1786830586.61 total 50.06 last 71.49) - open/close 4943 (time 1786830596.61 total 60.06 last 89.08) - open/close 5818 (time 1786830606.61 total 70.07 last 87.49) - open/close 6729 (time 1786830616.62 total 80.07 last 91.03) - open/close 7522 (time 1786830626.62 total 90.08 last 79.27) - open/close 8451 (time 1786830636.62 total 100.08 last 92.88) - open/close 9366 (time 1786830646.63 total 110.09 last 91.42) total: 10000 open/close in 117.10 seconds: 85.40 ops/second - unlinked 0 (time 1786830657 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 614 (time 1786830736.89 total 10.01 last 61.35) - open/close 1105 (time 1786830746.91 total 20.03 last 49.01) - open/close 1756 (time 1786830756.92 total 30.03 last 65.06) - open/close 2699 (time 1786830766.92 total 40.04 last 94.26) - open/close 3616 (time 1786830776.92 total 50.04 last 91.69) - open/close 4586 (time 1786830786.92 total 60.04 last 96.98) - open/close 5385 (time 1786830796.94 total 70.06 last 79.76) - open/close 6286 (time 1786830806.96 total 80.08 last 89.93) - open/close 7213 (time 1786830816.96 total 90.08 last 92.67) - open/close 7981 (time 1786830826.97 total 100.09 last 76.71) - open/close 8836 (time 1786830836.97 total 110.09 last 85.49) - open/close 9496 (time 1786830847.00 total 120.12 last 65.84) total: 10000 open/close in 127.56 seconds: 78.39 ops/second - unlinked 0 (time 1786830859 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 915 (time 1786830941.40 total 10.01 last 91.38) - open/close 1798 (time 1786830951.41 total 20.03 last 88.14) - open/close 2716 (time 1786830961.42 total 30.04 last 91.74) - open/close 3492 (time 1786830971.44 total 40.05 last 77.49) - open/close 4409 (time 1786830981.44 total 50.06 last 91.67) - open/close 5125 (time 1786830991.45 total 60.06 last 71.55) - open/close 5805 (time 1786831001.45 total 70.06 last 67.99) - open/close 6614 (time 1786831011.45 total 80.06 last 80.90) - open/close 7579 (time 1786831021.46 total 90.07 last 96.41) - open/close 8517 (time 1786831031.46 total 100.07 last 93.78) - open/close 9352 (time 1786831041.48 total 110.10 last 83.33) total: 10000 open/close in 117.80 seconds: 84.89 ops/second - unlinked 0 (time 1786831053 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg347-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 start ost1 service on oleg347-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules via unload_modules_local on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local oleg347-server: modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2