== conf-sanity test 69: replace an OST with the same index ========================================================== 11:21:02 (1786720862) start mds service on oleg256-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg256-server: detected 4 online CPUs by sysfs oleg256-server: MODOPTS_LIBCFS= oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg256-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg256-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 699 (time 1786720983.09 total 10.03 last 69.68) - open/close 1388 (time 1786720993.09 total 20.03 last 68.89) - open/close 2013 (time 1786721003.10 total 30.05 last 62.42) - open/close 2592 (time 1786721013.11 total 40.06 last 57.83) - open/close 3303 (time 1786721023.13 total 50.07 last 71.02) - open/close 4057 (time 1786721033.13 total 60.07 last 75.38) - open/close 4735 (time 1786721043.14 total 70.09 last 67.71) - open/close 5541 (time 1786721053.16 total 80.10 last 80.45) - open/close 6323 (time 1786721063.17 total 90.11 last 78.15) - open/close 6902 (time 1786721073.18 total 100.12 last 57.85) - open/close 7417 (time 1786721083.18 total 110.13 last 51.47) - open/close 8086 (time 1786721093.19 total 120.13 last 66.87) - open/close 8726 (time 1786721103.19 total 130.14 last 63.97) - open/close 9314 (time 1786721113.20 total 140.14 last 58.76) total: 10000 open/close in 148.03 seconds: 67.55 ops/second - unlinked 0 (time 1786721125 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 820 (time 1786721210.59 total 10.01 last 81.93) - open/close 1572 (time 1786721220.60 total 20.02 last 75.08) - open/close 2400 (time 1786721230.62 total 30.04 last 82.70) - open/close 3010 (time 1786721240.62 total 40.04 last 60.98) - open/close 3651 (time 1786721250.63 total 50.05 last 64.02) - open/close 4372 (time 1786721260.63 total 60.05 last 72.09) - open/close 5122 (time 1786721270.67 total 70.10 last 74.69) - open/close 6080 (time 1786721280.68 total 80.10 last 95.76) - open/close 6904 (time 1786721290.69 total 90.11 last 82.31) - open/close 7660 (time 1786721300.70 total 100.12 last 75.55) - open/close 8369 (time 1786721310.70 total 110.12 last 70.86) - open/close 9104 (time 1786721320.71 total 120.13 last 73.47) - open/close 9973 (time 1786721330.71 total 130.13 last 86.90) total: 10000 open/close in 130.53 seconds: 76.61 ops/second - unlinked 0 (time 1786721335 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 882 (time 1786721424.12 total 10.00 last 88.20) - open/close 1627 (time 1786721434.12 total 20.00 last 74.48) - open/close 2450 (time 1786721444.12 total 30.01 last 82.28) - open/close 3241 (time 1786721454.13 total 40.02 last 79.02) - open/close 4001 (time 1786721464.14 total 50.02 last 75.96) - open/close 4834 (time 1786721474.14 total 60.03 last 83.26) - open/close 5734 (time 1786721484.15 total 70.03 last 89.94) - open/close 6634 (time 1786721494.17 total 80.06 last 89.76) - open/close 7346 (time 1786721504.18 total 90.07 last 71.14) - open/close 8105 (time 1786721514.18 total 100.07 last 75.89) - open/close 8879 (time 1786721524.19 total 110.07 last 77.37) - open/close 9674 (time 1786721534.20 total 120.09 last 79.38) total: 10000 open/close in 123.96 seconds: 80.67 ops/second - unlinked 0 (time 1786721542 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 817 (time 1786721640.14 total 10.00 last 81.67) - open/close 1587 (time 1786721650.14 total 20.01 last 76.98) - open/close 2374 (time 1786721660.15 total 30.02 last 78.62) - open/close 3228 (time 1786721670.15 total 40.02 last 85.39) - open/close 4083 (time 1786721680.16 total 50.03 last 85.44) - open/close 4933 (time 1786721690.16 total 60.03 last 84.96) - open/close 5833 (time 1786721700.17 total 70.04 last 89.91) - open/close 6641 (time 1786721710.18 total 80.05 last 80.74) - open/close 7391 (time 1786721720.19 total 90.06 last 74.95) - open/close 8255 (time 1786721730.20 total 100.07 last 86.25) - open/close 9038 (time 1786721740.21 total 110.07 last 78.29) - open/close 9898 (time 1786721750.21 total 120.08 last 85.95) total: 10000 open/close in 121.06 seconds: 82.60 ops/second - unlinked 0 (time 1786721755 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 796 (time 1786721835.29 total 10.00 last 79.58) - open/close 1620 (time 1786721845.29 total 20.00 last 82.39) - open/close 2409 (time 1786721855.29 total 30.01 last 78.88) - open/close 3070 (time 1786721865.30 total 40.02 last 66.04) - open/close 3694 (time 1786721875.30 total 50.02 last 62.40) - open/close 4292 (time 1786721885.31 total 60.02 last 59.76) - open/close 5270 (time 1786721895.31 total 70.03 last 97.78) - open/close 6065 (time 1786721905.32 total 80.04 last 79.41) - open/close 6954 (time 1786721915.35 total 90.06 last 88.70) - open/close 7804 (time 1786721925.35 total 100.06 last 84.97) - open/close 8453 (time 1786721935.35 total 110.07 last 64.87) - open/close 9198 (time 1786721945.36 total 120.07 last 74.46) - open/close 9809 (time 1786721955.37 total 130.09 last 61.00) total: 10000 open/close in 133.50 seconds: 74.91 ops/second - unlinked 0 (time 1786721963 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 814 (time 1786722048.37 total 10.01 last 81.33) - open/close 1563 (time 1786722058.38 total 20.02 last 74.85) - open/close 2465 (time 1786722068.38 total 30.02 last 90.19) - open/close 3311 (time 1786722078.38 total 40.02 last 84.59) - open/close 4126 (time 1786722088.39 total 50.03 last 81.43) - open/close 4937 (time 1786722098.39 total 60.03 last 81.08) - open/close 5796 (time 1786722108.40 total 70.04 last 85.79) - open/close 6627 (time 1786722118.41 total 80.05 last 82.99) - open/close 7494 (time 1786722128.43 total 90.07 last 86.60) - open/close 8335 (time 1786722138.46 total 100.10 last 83.83) - open/close 9122 (time 1786722148.46 total 110.10 last 78.66) - open/close 9996 (time 1786722158.47 total 120.11 last 87.30) total: 10000 open/close in 120.20 seconds: 83.19 ops/second - unlinked 0 (time 1786722163 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 708 (time 1786722247.65 total 10.00 last 70.77) - open/close 1189 (time 1786722257.67 total 20.01 last 48.05) - open/close 1832 (time 1786722267.67 total 30.02 last 64.26) - open/close 2603 (time 1786722277.68 total 40.03 last 77.00) - open/close 3440 (time 1786722287.69 total 50.04 last 83.65) - open/close 4203 (time 1786722297.69 total 60.04 last 76.30) - open/close 5070 (time 1786722307.70 total 70.05 last 86.62) - open/close 5895 (time 1786722317.72 total 80.07 last 82.29) - open/close 6545 (time 1786722327.73 total 90.08 last 64.97) - open/close 7357 (time 1786722337.73 total 100.08 last 81.17) - open/close 8238 (time 1786722347.74 total 110.09 last 88.08) - open/close 8988 (time 1786722357.74 total 120.09 last 74.94) - open/close 9874 (time 1786722367.75 total 130.10 last 88.56) total: 10000 open/close in 131.64 seconds: 75.96 ops/second - unlinked 0 (time 1786722374 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 857 (time 1786722462.14 total 10.00 last 85.69) - open/close 1498 (time 1786722472.16 total 20.02 last 63.96) - open/close 2014 (time 1786722482.17 total 30.03 last 51.58) - open/close 2545 (time 1786722492.17 total 40.03 last 53.10) - open/close 3156 (time 1786722502.17 total 50.03 last 61.09) - open/close 3886 (time 1786722512.17 total 60.03 last 73.00) - open/close 4711 (time 1786722522.18 total 70.04 last 82.44) - open/close 5478 (time 1786722532.19 total 80.05 last 76.61) - open/close 6133 (time 1786722542.20 total 90.06 last 65.41) - open/close 6842 (time 1786722552.20 total 100.06 last 70.88) - open/close 7591 (time 1786722562.20 total 110.07 last 74.90) - open/close 8375 (time 1786722572.21 total 120.07 last 78.36) - open/close 9189 (time 1786722582.22 total 130.08 last 81.33) - open/close 9883 (time 1786722592.22 total 140.08 last 69.38) total: 10000 open/close in 141.77 seconds: 70.54 ops/second - unlinked 0 (time 1786722598 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 841 (time 1786722681.95 total 10.00 last 84.07) - open/close 1796 (time 1786722691.96 total 20.01 last 95.46) - open/close 2597 (time 1786722701.96 total 30.01 last 80.05) - open/close 3352 (time 1786722711.97 total 40.02 last 75.48) - open/close 3945 (time 1786722721.97 total 50.02 last 59.27) - open/close 4631 (time 1786722731.98 total 60.03 last 68.57) - open/close 5498 (time 1786722741.99 total 70.04 last 86.62) - open/close 6225 (time 1786722751.99 total 80.04 last 72.66) - open/close 7010 (time 1786722762.00 total 90.05 last 78.44) - open/close 7798 (time 1786722772.00 total 100.05 last 78.78) - open/close 8574 (time 1786722782.02 total 110.07 last 77.43) - open/close 9425 (time 1786722792.03 total 120.08 last 85.05) total: 10000 open/close in 127.54 seconds: 78.40 ops/second - unlinked 0 (time 1786722803 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 640 (time 1786722892.20 total 10.00 last 63.99) - open/close 1233 (time 1786722902.21 total 20.01 last 59.23) - open/close 1979 (time 1786722912.22 total 30.02 last 74.58) - open/close 2781 (time 1786722922.23 total 40.03 last 80.10) - open/close 3553 (time 1786722932.24 total 50.04 last 77.14) - open/close 4215 (time 1786722942.25 total 60.05 last 66.11) - open/close 5093 (time 1786722952.25 total 70.05 last 87.78) - open/close 5964 (time 1786722962.26 total 80.07 last 86.99) - open/close 6815 (time 1786722972.27 total 90.07 last 85.07) - open/close 7690 (time 1786722982.27 total 100.07 last 87.45) - open/close 8399 (time 1786722992.29 total 110.10 last 70.75) - open/close 9162 (time 1786723002.30 total 120.10 last 76.23) - open/close 9986 (time 1786723012.31 total 130.11 last 82.36) total: 10000 open/close in 130.24 seconds: 76.78 ops/second - unlinked 0 (time 1786723017 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg256-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 start ost1 service on oleg256-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server unloading modules via unload_modules_local on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local oleg256-server: modules unloaded. pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2