== conf-sanity test 69: replace an OST with the same index ========================================================== 06:27:38 (1786962458) start mds service on oleg428-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions oleg428-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg428-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg428-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg428-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 894 (time 1786962567.66 total 10.02 last 89.26) - open/close 1586 (time 1786962577.67 total 20.03 last 69.12) - open/close 2267 (time 1786962587.67 total 30.03 last 68.10) - open/close 2970 (time 1786962597.68 total 40.04 last 70.21) - open/close 3903 (time 1786962607.68 total 50.04 last 93.29) - open/close 4812 (time 1786962617.69 total 60.05 last 90.85) - open/close 5723 (time 1786962627.70 total 70.06 last 91.03) - open/close 6636 (time 1786962637.70 total 80.06 last 91.25) - open/close 7413 (time 1786962647.71 total 90.06 last 77.68) - open/close 8295 (time 1786962657.71 total 100.07 last 88.16) - open/close 9134 (time 1786962667.72 total 110.08 last 83.82) - open/close 9961 (time 1786962677.73 total 120.09 last 82.62) total: 10000 open/close in 120.44 seconds: 83.03 ops/second - unlinked 0 (time 1786962682 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 790 (time 1786962758.79 total 10.01 last 78.94) - open/close 1676 (time 1786962768.82 total 20.04 last 88.33) - open/close 2571 (time 1786962778.82 total 30.04 last 89.49) - open/close 3405 (time 1786962788.85 total 40.06 last 83.19) - open/close 4239 (time 1786962798.86 total 50.07 last 83.34) - open/close 5163 (time 1786962808.86 total 60.08 last 92.35) - open/close 6089 (time 1786962818.87 total 70.08 last 92.56) - open/close 7008 (time 1786962828.87 total 80.09 last 91.85) - open/close 7949 (time 1786962838.87 total 90.09 last 94.10) - open/close 8861 (time 1786962848.87 total 100.09 last 91.19) - open/close 9799 (time 1786962858.89 total 110.10 last 93.68) total: 10000 open/close in 112.14 seconds: 89.17 ops/second - unlinked 0 (time 1786962864 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 823 (time 1786962943.19 total 10.01 last 82.21) - open/close 1785 (time 1786962953.21 total 20.03 last 96.06) - open/close 2748 (time 1786962963.21 total 30.03 last 96.28) - open/close 3502 (time 1786962973.23 total 40.05 last 75.25) - open/close 4156 (time 1786962983.24 total 50.06 last 65.32) - open/close 5048 (time 1786962993.24 total 60.06 last 89.18) - open/close 5839 (time 1786963003.25 total 70.07 last 79.05) - open/close 6673 (time 1786963013.26 total 80.07 last 83.35) - open/close 7480 (time 1786963023.26 total 90.08 last 80.68) - open/close 8298 (time 1786963033.26 total 100.08 last 81.79) - open/close 9251 (time 1786963043.26 total 110.08 last 95.30) total: 10000 open/close in 119.52 seconds: 83.67 ops/second - unlinked 0 (time 1786963057 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 759 (time 1786963135.40 total 10.00 last 75.88) - open/close 1597 (time 1786963145.41 total 20.02 last 83.68) - open/close 2435 (time 1786963155.42 total 30.02 last 83.76) - open/close 3271 (time 1786963165.43 total 40.03 last 83.53) - open/close 4104 (time 1786963175.43 total 50.04 last 83.23) - open/close 4859 (time 1786963185.44 total 60.04 last 75.47) - open/close 5512 (time 1786963195.44 total 70.05 last 65.26) - open/close 6436 (time 1786963205.45 total 80.06 last 92.34) - open/close 7410 (time 1786963215.46 total 90.06 last 97.36) - open/close 8301 (time 1786963225.46 total 100.07 last 89.02) - open/close 9185 (time 1786963235.47 total 110.07 last 88.36) total: 10000 open/close in 119.57 seconds: 83.64 ops/second - unlinked 0 (time 1786963249 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 938 (time 1786963332.47 total 10.00 last 93.78) - open/close 1812 (time 1786963342.47 total 20.01 last 87.34) - open/close 2722 (time 1786963352.48 total 30.01 last 90.98) - open/close 3615 (time 1786963362.48 total 40.01 last 89.28) - open/close 4457 (time 1786963372.49 total 50.02 last 84.12) - open/close 5281 (time 1786963382.49 total 60.03 last 82.36) - open/close 6148 (time 1786963392.49 total 70.03 last 86.66) - open/close 6974 (time 1786963402.50 total 80.04 last 82.55) - open/close 7855 (time 1786963412.50 total 90.04 last 88.10) - open/close 8509 (time 1786963422.52 total 100.05 last 65.29) - open/close 9295 (time 1786963432.53 total 110.07 last 78.49) total: 10000 open/close in 118.77 seconds: 84.19 ops/second - unlinked 0 (time 1786963445 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 721 (time 1786963543.79 total 10.02 last 71.99) - open/close 1525 (time 1786963553.80 total 20.02 last 80.34) - open/close 2271 (time 1786963563.80 total 30.03 last 74.55) - open/close 3008 (time 1786963573.81 total 40.03 last 73.68) - open/close 3925 (time 1786963583.81 total 50.04 last 91.66) - open/close 4848 (time 1786963593.82 total 60.04 last 92.25) - open/close 5751 (time 1786963603.82 total 70.05 last 90.24) - open/close 6631 (time 1786963613.83 total 80.05 last 87.96) - open/close 7489 (time 1786963623.83 total 90.06 last 85.77) - open/close 8464 (time 1786963633.83 total 100.06 last 97.48) - open/close 9503 (time 1786963643.85 total 110.07 last 103.74) total: 10000 open/close in 115.62 seconds: 86.49 ops/second - unlinked 0 (time 1786963653 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 902 (time 1786963741.69 total 10.00 last 90.19) - open/close 1759 (time 1786963751.69 total 20.00 last 85.69) - open/close 2612 (time 1786963761.69 total 30.01 last 85.27) - open/close 3502 (time 1786963771.69 total 40.01 last 88.98) - open/close 4336 (time 1786963781.70 total 50.01 last 83.34) - open/close 5225 (time 1786963791.70 total 60.02 last 88.87) - open/close 5881 (time 1786963801.70 total 70.02 last 65.58) - open/close 6431 (time 1786963811.72 total 80.04 last 54.90) - open/close 7121 (time 1786963821.72 total 90.04 last 69.00) - open/close 8053 (time 1786963831.73 total 100.04 last 93.15) - open/close 8975 (time 1786963841.76 total 110.07 last 91.94) - open/close 9853 (time 1786963851.76 total 120.08 last 87.77) total: 10000 open/close in 121.67 seconds: 82.19 ops/second - unlinked 0 (time 1786963857 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 864 (time 1786963941.15 total 10.00 last 86.39) - open/close 1836 (time 1786963951.16 total 20.01 last 97.15) - open/close 2587 (time 1786963961.17 total 30.02 last 74.98) - open/close 3591 (time 1786963971.18 total 40.02 last 100.39) - open/close 4408 (time 1786963981.18 total 50.03 last 81.66) - open/close 5195 (time 1786963991.18 total 60.03 last 78.67) - open/close 6052 (time 1786964001.19 total 70.04 last 85.66) - open/close 6974 (time 1786964011.20 total 80.05 last 92.10) - open/close 7876 (time 1786964021.20 total 90.05 last 90.19) - open/close 8748 (time 1786964031.21 total 100.06 last 87.13) - open/close 9594 (time 1786964041.22 total 110.07 last 84.53) total: 10000 open/close in 115.01 seconds: 86.95 ops/second - unlinked 0 (time 1786964050 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 921 (time 1786964133.07 total 10.00 last 92.08) - open/close 1878 (time 1786964143.08 total 20.01 last 95.67) - open/close 2761 (time 1786964153.09 total 30.02 last 88.20) - open/close 3670 (time 1786964163.10 total 40.03 last 90.82) - open/close 4511 (time 1786964173.11 total 50.04 last 83.99) - open/close 5162 (time 1786964183.11 total 60.04 last 65.09) - open/close 6088 (time 1786964193.12 total 70.05 last 92.54) - open/close 6971 (time 1786964203.12 total 80.05 last 88.30) - open/close 7757 (time 1786964213.12 total 90.05 last 78.58) - open/close 8581 (time 1786964223.12 total 100.05 last 82.38) - open/close 9323 (time 1786964233.15 total 110.08 last 74.02) total: 10000 open/close in 117.42 seconds: 85.16 ops/second - unlinked 0 (time 1786964244 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 810 (time 1786964328.55 total 10.01 last 80.94) - open/close 1576 (time 1786964338.56 total 20.02 last 76.51) - open/close 2356 (time 1786964348.57 total 30.03 last 77.95) - open/close 2824 (time 1786964358.59 total 40.05 last 46.70) - open/close 3528 (time 1786964368.60 total 50.05 last 70.37) - open/close 4448 (time 1786964378.60 total 60.05 last 91.98) - open/close 5271 (time 1786964388.61 total 70.06 last 82.25) - open/close 6232 (time 1786964398.61 total 80.06 last 96.07) - open/close 7086 (time 1786964408.62 total 90.07 last 85.32) - open/close 8019 (time 1786964418.62 total 100.08 last 93.24) - open/close 8902 (time 1786964428.63 total 110.09 last 88.21) - open/close 9762 (time 1786964438.64 total 120.09 last 85.95) total: 10000 open/close in 123.20 seconds: 81.17 ops/second - unlinked 0 (time 1786964445 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.128@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg428-server: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg428-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg428-server: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg428-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.128@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.128@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.128@tcp sys.timeout=20 start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg428-server: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg428-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg428-server: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg428-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg428-server unloading modules via unload_modules_local on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing unload_modules_local oleg428-server: modules unloaded. pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2