== conf-sanity test 69: replace an OST with the same index ========================================================== 12:16:59 (1774887419) start mds service on oleg119-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions oleg119-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg119-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg119-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg119-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 848 (time 1774887533.55 total 10.03 last 84.59) - open/close 1712 (time 1774887543.56 total 20.03 last 86.33) - open/close 2620 (time 1774887553.57 total 30.05 last 90.68) - open/close 3642 (time 1774887563.58 total 40.05 last 102.14) - open/close 4511 (time 1774887573.58 total 50.06 last 86.87) - open/close 5335 (time 1774887583.60 total 60.08 last 82.21) - open/close 6107 (time 1774887593.61 total 70.08 last 77.15) - open/close 7011 (time 1774887603.62 total 80.09 last 90.33) - open/close 8016 (time 1774887613.63 total 90.10 last 100.41) - open/close 8996 (time 1774887623.63 total 100.10 last 97.96) - open/close 9846 (time 1774887633.63 total 110.11 last 84.97) total: 10000 open/close in 112.06 seconds: 89.24 ops/second - unlinked 0 (time 1774887640 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 814 (time 1774887732.33 total 10.01 last 81.32) - open/close 1549 (time 1774887742.33 total 20.02 last 73.46) - open/close 2290 (time 1774887752.34 total 30.02 last 74.07) - open/close 3005 (time 1774887762.34 total 40.02 last 71.49) - open/close 3767 (time 1774887772.34 total 50.03 last 76.16) - open/close 4651 (time 1774887782.35 total 60.04 last 88.31) - open/close 5445 (time 1774887792.36 total 70.05 last 79.33) - open/close 6248 (time 1774887802.37 total 80.06 last 80.21) - open/close 6938 (time 1774887812.38 total 90.06 last 68.95) - open/close 7739 (time 1774887822.39 total 100.07 last 80.03) - open/close 8451 (time 1774887832.39 total 110.07 last 71.20) - open/close 9175 (time 1774887842.39 total 120.08 last 72.37) - open/close 9961 (time 1774887852.39 total 130.08 last 78.59) total: 10000 open/close in 130.60 seconds: 76.57 ops/second - unlinked 0 (time 1774887857 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 709 (time 1774887948.86 total 10.01 last 70.85) - open/close 1356 (time 1774887958.87 total 20.01 last 64.65) - open/close 2173 (time 1774887968.87 total 30.02 last 81.67) - open/close 3148 (time 1774887978.88 total 40.03 last 97.37) - open/close 4036 (time 1774887988.89 total 50.04 last 88.73) - open/close 4846 (time 1774887998.91 total 60.06 last 80.86) - open/close 5664 (time 1774888008.91 total 70.06 last 81.76) - open/close 6256 (time 1774888018.94 total 80.09 last 59.05) - open/close 6918 (time 1774888028.95 total 90.10 last 66.12) - open/close 7682 (time 1774888038.96 total 100.11 last 76.34) - open/close 8447 (time 1774888048.96 total 110.11 last 76.49) - open/close 9238 (time 1774888058.96 total 120.11 last 79.09) total: 10000 open/close in 128.81 seconds: 77.63 ops/second - unlinked 0 (time 1774888072 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 779 (time 1774888161.66 total 10.03 last 77.64) - open/close 1498 (time 1774888171.67 total 20.04 last 71.87) - open/close 2130 (time 1774888181.69 total 30.07 last 63.02) - open/close 2885 (time 1774888191.71 total 40.08 last 75.42) - open/close 3427 (time 1774888201.74 total 50.11 last 54.03) - open/close 4240 (time 1774888211.75 total 60.12 last 81.23) - open/close 5132 (time 1774888221.75 total 70.12 last 89.16) - open/close 5905 (time 1774888231.76 total 80.13 last 77.21) - open/close 6646 (time 1774888241.79 total 90.17 last 73.86) - open/close 7253 (time 1774888251.81 total 100.18 last 60.63) - open/close 8106 (time 1774888261.83 total 110.20 last 85.10) - open/close 8718 (time 1774888271.83 total 120.21 last 61.18) - open/close 9432 (time 1774888281.84 total 130.21 last 71.35) total: 10000 open/close in 138.02 seconds: 72.45 ops/second - unlinked 0 (time 1774888294 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 807 (time 1774888388.57 total 10.01 last 80.64) - open/close 1516 (time 1774888398.58 total 20.02 last 70.80) - open/close 2401 (time 1774888408.59 total 30.03 last 88.44) - open/close 3140 (time 1774888418.59 total 40.03 last 73.90) - open/close 3798 (time 1774888428.60 total 50.04 last 65.72) - open/close 4429 (time 1774888438.61 total 60.05 last 63.06) - open/close 5203 (time 1774888448.61 total 70.05 last 77.35) - open/close 5923 (time 1774888458.62 total 80.06 last 71.95) - open/close 6682 (time 1774888468.63 total 90.07 last 75.84) - open/close 7420 (time 1774888478.63 total 100.07 last 73.75) - open/close 8326 (time 1774888488.64 total 110.08 last 90.52) - open/close 9239 (time 1774888498.65 total 120.09 last 91.25) total: 10000 open/close in 128.18 seconds: 78.01 ops/second - unlinked 0 (time 1774888511 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 932 (time 1774888609.47 total 10.01 last 93.15) - open/close 1788 (time 1774888619.47 total 20.01 last 85.55) - open/close 2673 (time 1774888629.48 total 30.01 last 88.47) - open/close 3520 (time 1774888639.49 total 40.03 last 84.61) - open/close 4320 (time 1774888649.49 total 50.03 last 79.96) - open/close 5149 (time 1774888659.50 total 60.04 last 82.78) - open/close 5984 (time 1774888669.52 total 70.06 last 83.40) - open/close 6812 (time 1774888679.52 total 80.06 last 82.74) - open/close 7704 (time 1774888689.53 total 90.07 last 89.12) - open/close 8690 (time 1774888699.54 total 100.08 last 98.54) - open/close 9463 (time 1774888709.55 total 110.09 last 77.22) total: 10000 open/close in 117.97 seconds: 84.77 ops/second - unlinked 0 (time 1774888722 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 873 (time 1774888817.72 total 10.00 last 87.29) - open/close 1714 (time 1774888827.74 total 20.02 last 83.98) - open/close 2688 (time 1774888837.74 total 30.02 last 97.33) - open/close 3622 (time 1774888847.75 total 40.03 last 93.34) - open/close 4487 (time 1774888857.75 total 50.03 last 86.49) - open/close 5443 (time 1774888867.76 total 60.03 last 95.57) - open/close 6392 (time 1774888877.76 total 70.04 last 94.84) - open/close 7336 (time 1774888887.77 total 80.05 last 94.35) - open/close 8248 (time 1774888897.77 total 90.05 last 91.14) - open/close 9109 (time 1774888907.78 total 100.06 last 86.02) - open/close 9938 (time 1774888917.79 total 110.07 last 82.83) total: 10000 open/close in 111.16 seconds: 89.96 ops/second - unlinked 0 (time 1774888924 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 885 (time 1774889011.34 total 10.00 last 88.47) - open/close 1793 (time 1774889021.34 total 20.01 last 90.76) - open/close 2636 (time 1774889031.35 total 30.01 last 84.28) - open/close 3516 (time 1774889041.35 total 40.02 last 87.94) - open/close 4313 (time 1774889051.36 total 50.02 last 79.66) - open/close 5136 (time 1774889061.37 total 60.03 last 82.24) - open/close 5910 (time 1774889071.37 total 70.03 last 77.39) - open/close 6880 (time 1774889081.38 total 80.04 last 96.91) - open/close 7684 (time 1774889091.38 total 90.04 last 80.37) - open/close 8533 (time 1774889101.39 total 100.05 last 84.83) - open/close 9363 (time 1774889111.39 total 110.06 last 82.98) total: 10000 open/close in 116.99 seconds: 85.48 ops/second - unlinked 0 (time 1774889122 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 850 (time 1774889206.49 total 10.00 last 84.97) - open/close 1753 (time 1774889216.49 total 20.01 last 90.26) - open/close 2621 (time 1774889226.49 total 30.01 last 86.80) - open/close 3493 (time 1774889236.50 total 40.02 last 87.14) - open/close 4440 (time 1774889246.50 total 50.02 last 94.70) - open/close 5360 (time 1774889256.50 total 60.02 last 91.98) - open/close 6378 (time 1774889266.51 total 70.02 last 101.78) - open/close 7058 (time 1774889276.52 total 80.03 last 67.92) - open/close 8045 (time 1774889286.52 total 90.03 last 98.70) - open/close 8995 (time 1774889296.52 total 100.04 last 94.94) - open/close 9905 (time 1774889306.53 total 110.04 last 90.96) total: 10000 open/close in 111.01 seconds: 90.08 ops/second - unlinked 0 (time 1774889311 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 928 (time 1774889383.51 total 10.01 last 92.71) - open/close 1901 (time 1774889393.51 total 20.02 last 97.25) - open/close 2840 (time 1774889403.52 total 30.02 last 93.87) - open/close 3779 (time 1774889413.52 total 40.02 last 93.90) - open/close 4688 (time 1774889423.52 total 50.02 last 90.90) - open/close 5512 (time 1774889433.52 total 60.02 last 82.40) - open/close 6175 (time 1774889443.52 total 70.02 last 66.28) - open/close 6914 (time 1774889453.52 total 80.03 last 73.87) - open/close 7780 (time 1774889463.53 total 90.03 last 86.57) - open/close 8673 (time 1774889473.53 total 100.03 last 89.28) - open/close 9537 (time 1774889483.54 total 110.04 last 86.34) total: 10000 open/close in 114.55 seconds: 87.30 ops/second - unlinked 0 (time 1774889491 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local oleg119-server: modules unloaded. pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2