== conf-sanity test 69: replace an OST with the same index ========================================================== 15:44:04 (1775763844) start mds service on oleg606-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing load_modules_local oleg606-server: Loading modules from /home/green/git/lustre-release/lustre oleg606-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg606-server: detected 4 online CPUs by sysfs oleg606-server: MODOPTS_LIBCFS= oleg606-server: Force libcfs to create 2 CPU partitions oleg606-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg606-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg606-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg606-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg606-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 712 (time 1775763950.22 total 10.01 last 71.14) - open/close 1452 (time 1775763960.23 total 20.02 last 73.95) - open/close 2206 (time 1775763970.23 total 30.02 last 75.40) - open/close 2912 (time 1775763980.23 total 40.02 last 70.58) - open/close 3601 (time 1775763990.24 total 50.03 last 68.83) - open/close 4378 (time 1775764000.24 total 60.03 last 77.67) - open/close 5186 (time 1775764010.25 total 70.04 last 80.74) - open/close 5997 (time 1775764020.27 total 80.06 last 80.94) - open/close 6755 (time 1775764030.27 total 90.06 last 75.78) - open/close 7380 (time 1775764040.28 total 100.07 last 62.47) - open/close 8087 (time 1775764050.28 total 110.07 last 70.65) - open/close 8893 (time 1775764060.29 total 120.08 last 80.56) - open/close 9640 (time 1775764070.30 total 130.09 last 74.62) total: 10000 open/close in 135.27 seconds: 73.93 ops/second - unlinked 0 (time 1775764080 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 753 (time 1775764171.72 total 10.01 last 75.26) - open/close 1399 (time 1775764181.76 total 20.04 last 64.38) - open/close 2156 (time 1775764191.77 total 30.06 last 75.57) - open/close 2973 (time 1775764201.78 total 40.06 last 81.67) - open/close 3766 (time 1775764211.79 total 50.07 last 79.23) - open/close 4559 (time 1775764221.81 total 60.09 last 79.17) - open/close 5263 (time 1775764231.81 total 70.09 last 70.35) - open/close 6066 (time 1775764241.82 total 80.10 last 80.25) - open/close 6867 (time 1775764251.83 total 90.11 last 80.01) - open/close 7698 (time 1775764261.84 total 100.12 last 82.99) - open/close 8554 (time 1775764271.85 total 110.13 last 85.57) - open/close 9418 (time 1775764281.85 total 120.13 last 86.38) total: 10000 open/close in 126.88 seconds: 78.82 ops/second - unlinked 0 (time 1775764292 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 799 (time 1775764380.72 total 10.01 last 79.81) - open/close 1529 (time 1775764390.72 total 20.02 last 72.97) - open/close 2379 (time 1775764400.72 total 30.02 last 84.99) - open/close 3204 (time 1775764410.73 total 40.03 last 82.44) - open/close 4053 (time 1775764420.74 total 50.03 last 84.85) - open/close 4842 (time 1775764430.75 total 60.04 last 78.81) - open/close 5688 (time 1775764440.76 total 70.06 last 84.47) - open/close 6493 (time 1775764450.77 total 80.06 last 80.46) - open/close 7367 (time 1775764460.77 total 90.06 last 87.40) - open/close 8217 (time 1775764470.77 total 100.06 last 84.99) - open/close 9055 (time 1775764480.79 total 110.08 last 83.66) - open/close 9832 (time 1775764490.80 total 120.09 last 77.64) total: 10000 open/close in 122.12 seconds: 81.89 ops/second - unlinked 0 (time 1775764496 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 772 (time 1775764582.44 total 10.01 last 77.09) - open/close 1576 (time 1775764592.45 total 20.02 last 80.35) - open/close 2375 (time 1775764602.45 total 30.02 last 79.90) - open/close 3084 (time 1775764612.45 total 40.02 last 70.89) - open/close 3897 (time 1775764622.45 total 50.02 last 81.29) - open/close 4677 (time 1775764632.46 total 60.03 last 77.94) - open/close 5476 (time 1775764642.49 total 70.06 last 79.68) - open/close 6277 (time 1775764652.49 total 80.06 last 80.10) - open/close 6984 (time 1775764662.49 total 90.06 last 70.67) - open/close 7793 (time 1775764672.50 total 100.07 last 80.84) - open/close 8609 (time 1775764682.50 total 110.07 last 81.58) - open/close 9389 (time 1775764692.50 total 120.08 last 77.98) total: 10000 open/close in 127.34 seconds: 78.53 ops/second - unlinked 0 (time 1775764703 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 824 (time 1775764790.71 total 10.01 last 82.34) - open/close 1652 (time 1775764800.72 total 20.01 last 82.74) - open/close 2349 (time 1775764810.74 total 30.03 last 69.59) - open/close 3164 (time 1775764820.75 total 40.04 last 81.44) - open/close 4057 (time 1775764830.76 total 50.05 last 89.21) - open/close 4816 (time 1775764840.77 total 60.06 last 75.80) - open/close 5670 (time 1775764850.77 total 70.06 last 85.38) - open/close 6472 (time 1775764860.78 total 80.07 last 80.12) - open/close 7198 (time 1775764870.79 total 90.08 last 72.53) - open/close 8004 (time 1775764880.79 total 100.08 last 80.60) - open/close 8817 (time 1775764890.79 total 110.09 last 81.29) - open/close 9645 (time 1775764900.79 total 120.09 last 82.79) total: 10000 open/close in 124.71 seconds: 80.19 ops/second - unlinked 0 (time 1775764909 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 744 (time 1775764997.87 total 10.01 last 74.34) - open/close 1535 (time 1775765007.87 total 20.01 last 79.09) - open/close 2407 (time 1775765017.88 total 30.02 last 87.12) - open/close 3100 (time 1775765027.89 total 40.03 last 69.23) - open/close 4026 (time 1775765037.90 total 50.04 last 92.50) - open/close 4873 (time 1775765047.92 total 60.06 last 84.52) - open/close 5656 (time 1775765057.93 total 70.07 last 78.24) - open/close 6353 (time 1775765067.93 total 80.07 last 69.69) - open/close 7083 (time 1775765077.94 total 90.08 last 72.92) - open/close 7818 (time 1775765087.95 total 100.09 last 73.42) - open/close 8622 (time 1775765097.97 total 110.10 last 80.30) - open/close 9461 (time 1775765107.98 total 120.11 last 83.79) total: 10000 open/close in 126.34 seconds: 79.15 ops/second - unlinked 0 (time 1775765118 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 900 (time 1775765206.81 total 10.01 last 89.95) - open/close 1732 (time 1775765216.82 total 20.02 last 83.11) - open/close 2515 (time 1775765226.83 total 30.02 last 78.24) - open/close 3330 (time 1775765236.83 total 40.03 last 81.47) - open/close 4115 (time 1775765246.84 total 50.04 last 78.44) - open/close 5003 (time 1775765256.85 total 60.04 last 88.73) - open/close 5857 (time 1775765266.88 total 70.08 last 85.11) - open/close 6634 (time 1775765276.89 total 80.08 last 77.64) - open/close 7433 (time 1775765286.89 total 90.09 last 79.88) - open/close 8243 (time 1775765296.90 total 100.10 last 80.92) - open/close 9053 (time 1775765306.90 total 110.10 last 80.99) - open/close 9757 (time 1775765316.91 total 120.11 last 70.35) total: 10000 open/close in 122.52 seconds: 81.62 ops/second - unlinked 0 (time 1775765324 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 692 (time 1775765423.78 total 10.02 last 69.08) - open/close 1451 (time 1775765433.79 total 20.02 last 75.89) - open/close 2179 (time 1775765443.79 total 30.02 last 72.77) - open/close 3016 (time 1775765453.79 total 40.03 last 83.66) - open/close 3848 (time 1775765463.80 total 50.03 last 83.18) - open/close 4624 (time 1775765473.80 total 60.04 last 77.56) - open/close 5410 (time 1775765483.81 total 70.04 last 78.54) - open/close 6185 (time 1775765493.82 total 80.06 last 77.40) - open/close 6892 (time 1775765503.82 total 90.06 last 70.68) - open/close 7659 (time 1775765513.83 total 100.06 last 76.69) - open/close 8463 (time 1775765523.83 total 110.06 last 80.39) - open/close 9328 (time 1775765533.84 total 120.07 last 86.42) total: 10000 open/close in 128.55 seconds: 77.79 ops/second - unlinked 0 (time 1775765546 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 805 (time 1775765635.07 total 10.03 last 80.24) - open/close 1564 (time 1775765645.07 total 20.03 last 75.90) - open/close 2312 (time 1775765655.07 total 30.03 last 74.80) - open/close 3163 (time 1775765665.08 total 40.05 last 84.99) - open/close 3977 (time 1775765675.09 total 50.05 last 81.38) - open/close 4749 (time 1775765685.10 total 60.06 last 77.10) - open/close 5575 (time 1775765695.11 total 70.07 last 82.50) - open/close 6415 (time 1775765705.12 total 80.08 last 83.95) - open/close 7151 (time 1775765715.13 total 90.09 last 73.55) - open/close 7917 (time 1775765725.13 total 100.09 last 76.59) - open/close 8676 (time 1775765735.13 total 110.09 last 75.87) - open/close 9454 (time 1775765745.14 total 120.10 last 77.75) total: 10000 open/close in 127.20 seconds: 78.62 ops/second - unlinked 0 (time 1775765756 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 786 (time 1775765844.37 total 10.00 last 78.58) - open/close 1446 (time 1775765854.40 total 20.03 last 65.84) - open/close 2247 (time 1775765864.40 total 30.03 last 80.07) - open/close 3030 (time 1775765874.41 total 40.04 last 78.22) - open/close 3678 (time 1775765884.42 total 50.05 last 64.75) - open/close 4448 (time 1775765894.42 total 60.05 last 76.98) - open/close 5163 (time 1775765904.44 total 70.06 last 71.42) - open/close 5926 (time 1775765914.44 total 80.07 last 76.23) - open/close 6726 (time 1775765924.45 total 90.08 last 79.96) - open/close 7515 (time 1775765934.46 total 100.09 last 78.80) - open/close 8352 (time 1775765944.47 total 110.09 last 83.67) - open/close 9148 (time 1775765954.47 total 120.10 last 79.58) total: 10000 open/close in 129.72 seconds: 77.09 ops/second - unlinked 0 (time 1775765968 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg606-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg606-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg606-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.106@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg606-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg606-server: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg606-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg606-server: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg606-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg606-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg606-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg606-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.106@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.106@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.106@tcp sys.timeout=20 start ost1 service on oleg606-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg606-server: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg606-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg606-server: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg606-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg606-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg606-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg606-server stop mds service on oleg606-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg606-server stop mds service on oleg606-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg606-server unloading modules via unload_modules_local on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing unload_modules_local oleg606-server: modules unloaded. pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2