== conf-sanity test 69: replace an OST with the same index ========================================================== 17:37:20 (1786311440) start mds service on oleg123-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions oleg123-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg123-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg123-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg123-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg123-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 668 (time 1786311577.59 total 10.01 last 66.74) - open/close 1244 (time 1786311587.59 total 20.01 last 57.60) - open/close 1945 (time 1786311597.61 total 30.02 last 70.00) - open/close 2570 (time 1786311607.62 total 40.04 last 62.42) - open/close 3299 (time 1786311617.62 total 50.04 last 72.88) - open/close 4029 (time 1786311627.63 total 60.05 last 72.96) - open/close 4801 (time 1786311637.64 total 70.06 last 77.12) - open/close 5492 (time 1786311647.67 total 80.09 last 68.88) - open/close 6190 (time 1786311657.68 total 90.10 last 69.72) - open/close 6797 (time 1786311667.69 total 100.10 last 60.68) - open/close 7490 (time 1786311677.69 total 110.10 last 69.29) - open/close 8188 (time 1786311687.69 total 120.11 last 69.79) - open/close 8784 (time 1786311697.69 total 130.11 last 59.58) - open/close 9494 (time 1786311707.69 total 140.11 last 71.00) total: 10000 open/close in 146.82 seconds: 68.11 ops/second - unlinked 0 (time 1786311719 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 705 (time 1786311821.34 total 10.00 last 70.48) - open/close 1543 (time 1786311831.35 total 20.01 last 83.73) - open/close 2278 (time 1786311841.36 total 30.02 last 73.42) - open/close 3150 (time 1786311851.36 total 40.03 last 87.18) - open/close 3848 (time 1786311861.37 total 50.03 last 69.77) - open/close 4585 (time 1786311871.37 total 60.03 last 73.68) - open/close 5249 (time 1786311881.39 total 70.05 last 66.29) - open/close 5921 (time 1786311891.40 total 80.06 last 67.11) - open/close 6569 (time 1786311901.40 total 90.07 last 64.77) - open/close 7309 (time 1786311911.41 total 100.07 last 73.98) - open/close 7929 (time 1786311921.43 total 110.09 last 61.87) - open/close 8571 (time 1786311931.44 total 120.11 last 64.10) - open/close 9197 (time 1786311941.45 total 130.11 last 62.58) - open/close 9855 (time 1786311951.45 total 140.11 last 65.78) total: 10000 open/close in 141.80 seconds: 70.52 ops/second - unlinked 0 (time 1786311959 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 610 (time 1786312057.99 total 10.01 last 60.91) - open/close 1207 (time 1786312067.99 total 20.02 last 59.69) - open/close 1831 (time 1786312078.01 total 30.03 last 62.30) - open/close 2356 (time 1786312088.02 total 40.05 last 52.41) - open/close 2749 (time 1786312098.02 total 50.05 last 39.30) - open/close 3290 (time 1786312108.04 total 60.06 last 54.04) - open/close 3980 (time 1786312118.04 total 70.06 last 68.99) - open/close 4659 (time 1786312128.04 total 80.06 last 67.90) - open/close 5334 (time 1786312138.05 total 90.07 last 67.42) - open/close 6034 (time 1786312148.05 total 100.08 last 69.96) - open/close 6843 (time 1786312158.06 total 110.08 last 80.87) - open/close 7533 (time 1786312168.07 total 120.09 last 68.94) - open/close 8266 (time 1786312178.08 total 130.10 last 73.24) - open/close 8971 (time 1786312188.10 total 140.12 last 70.35) - open/close 9728 (time 1786312198.10 total 150.13 last 75.66) total: 10000 open/close in 154.11 seconds: 64.89 ops/second - unlinked 0 (time 1786312208 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 643 (time 1786312306.40 total 10.02 last 64.17) - open/close 1208 (time 1786312316.41 total 20.02 last 56.47) - open/close 1896 (time 1786312326.42 total 30.04 last 68.70) - open/close 2734 (time 1786312336.43 total 40.04 last 83.77) - open/close 3375 (time 1786312346.43 total 50.04 last 64.09) - open/close 4012 (time 1786312356.43 total 60.05 last 63.66) - open/close 4704 (time 1786312366.45 total 70.06 last 69.11) - open/close 5338 (time 1786312376.46 total 80.07 last 63.33) - open/close 6129 (time 1786312386.47 total 90.09 last 78.97) - open/close 6851 (time 1786312396.47 total 100.09 last 72.20) - open/close 7484 (time 1786312406.47 total 110.09 last 63.29) - open/close 8150 (time 1786312416.49 total 120.11 last 66.50) - open/close 8923 (time 1786312426.51 total 130.12 last 77.18) - open/close 9768 (time 1786312436.51 total 140.13 last 84.45) total: 10000 open/close in 143.21 seconds: 69.83 ops/second - unlinked 0 (time 1786312445 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 754 (time 1786312549.70 total 10.01 last 75.33) - open/close 1428 (time 1786312559.71 total 20.02 last 67.35) - open/close 2197 (time 1786312569.71 total 30.02 last 76.87) - open/close 2882 (time 1786312579.71 total 40.02 last 68.49) - open/close 3529 (time 1786312589.72 total 50.03 last 64.63) - open/close 4192 (time 1786312599.74 total 60.05 last 66.20) - open/close 4875 (time 1786312609.74 total 70.05 last 68.28) - open/close 5427 (time 1786312619.74 total 80.05 last 55.20) - open/close 5944 (time 1786312629.75 total 90.06 last 51.65) - open/close 6473 (time 1786312639.76 total 100.07 last 52.85) - open/close 7222 (time 1786312649.76 total 110.07 last 74.89) - open/close 8044 (time 1786312659.78 total 120.10 last 82.01) - open/close 8755 (time 1786312669.79 total 130.10 last 71.07) - open/close 9445 (time 1786312679.79 total 140.10 last 68.96) total: 10000 open/close in 147.06 seconds: 68.00 ops/second - unlinked 0 (time 1786312691 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 646 (time 1786312787.23 total 10.01 last 64.56) - open/close 1330 (time 1786312797.25 total 20.02 last 68.29) - open/close 2065 (time 1786312807.25 total 30.02 last 73.50) - open/close 2936 (time 1786312817.26 total 40.03 last 87.03) - open/close 3752 (time 1786312827.26 total 50.04 last 81.54) - open/close 4559 (time 1786312837.27 total 60.04 last 80.66) - open/close 5305 (time 1786312847.27 total 70.05 last 74.56) - open/close 5819 (time 1786312857.29 total 80.06 last 51.34) - open/close 6478 (time 1786312867.29 total 90.06 last 65.87) - open/close 7151 (time 1786312877.29 total 100.07 last 67.27) - open/close 7831 (time 1786312887.30 total 110.07 last 67.98) - open/close 8414 (time 1786312897.30 total 120.08 last 58.28) - open/close 9031 (time 1786312907.32 total 130.10 last 61.56) - open/close 9736 (time 1786312917.33 total 140.10 last 70.49) total: 10000 open/close in 143.79 seconds: 69.54 ops/second - unlinked 0 (time 1786312926 ; total 1 ; last 1) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 778 (time 1786313015.52 total 10.01 last 77.76) - open/close 1625 (time 1786313025.52 total 20.01 last 84.66) - open/close 2465 (time 1786313035.54 total 30.03 last 83.85) - open/close 3228 (time 1786313045.55 total 40.04 last 76.24) - open/close 4068 (time 1786313055.55 total 50.04 last 83.97) - open/close 4918 (time 1786313065.56 total 60.04 last 84.97) - open/close 5673 (time 1786313075.57 total 70.05 last 75.43) - open/close 6363 (time 1786313085.57 total 80.06 last 68.96) - open/close 7068 (time 1786313095.59 total 90.07 last 70.41) - open/close 7907 (time 1786313105.60 total 100.08 last 83.82) - open/close 8651 (time 1786313115.60 total 110.08 last 74.40) - open/close 9477 (time 1786313125.60 total 120.09 last 82.56) total: 10000 open/close in 126.87 seconds: 78.82 ops/second - unlinked 0 (time 1786313138 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 890 (time 1786313226.23 total 10.01 last 88.95) - open/close 1777 (time 1786313236.23 total 20.01 last 88.69) - open/close 2585 (time 1786313246.23 total 30.01 last 80.77) - open/close 3426 (time 1786313256.24 total 40.02 last 84.04) - open/close 4338 (time 1786313266.24 total 50.02 last 91.19) - open/close 5245 (time 1786313276.25 total 60.02 last 90.65) - open/close 6117 (time 1786313286.25 total 70.02 last 87.20) - open/close 6804 (time 1786313296.29 total 80.06 last 68.44) - open/close 7550 (time 1786313306.29 total 90.07 last 74.55) - open/close 8285 (time 1786313316.31 total 100.08 last 73.40) - open/close 9013 (time 1786313326.31 total 110.08 last 72.79) - open/close 9726 (time 1786313336.31 total 120.09 last 71.27) total: 10000 open/close in 123.37 seconds: 81.06 ops/second - unlinked 0 (time 1786313344 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 739 (time 1786313433.52 total 10.00 last 73.87) - open/close 1431 (time 1786313443.52 total 20.01 last 69.15) - open/close 2188 (time 1786313453.53 total 30.02 last 75.64) - open/close 2894 (time 1786313463.54 total 40.03 last 70.52) - open/close 3622 (time 1786313473.54 total 50.03 last 72.78) - open/close 4330 (time 1786313483.55 total 60.04 last 70.77) - open/close 5051 (time 1786313493.55 total 70.04 last 72.06) - open/close 5861 (time 1786313503.55 total 80.04 last 81.00) - open/close 6678 (time 1786313513.56 total 90.05 last 81.65) - open/close 7429 (time 1786313523.56 total 100.05 last 75.08) - open/close 8357 (time 1786313533.57 total 110.05 last 92.77) - open/close 9086 (time 1786313543.57 total 120.06 last 72.88) - open/close 9799 (time 1786313553.60 total 130.09 last 71.09) total: 10000 open/close in 133.17 seconds: 75.09 ops/second - unlinked 0 (time 1786313562 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 859 (time 1786313650.35 total 10.00 last 85.87) - open/close 1736 (time 1786313660.35 total 20.00 last 87.69) - open/close 2529 (time 1786313670.35 total 30.01 last 79.29) - open/close 3375 (time 1786313680.36 total 40.01 last 84.52) - open/close 4152 (time 1786313690.36 total 50.02 last 77.66) - open/close 4810 (time 1786313700.36 total 60.02 last 65.80) - open/close 5338 (time 1786313710.37 total 70.02 last 52.79) - open/close 6135 (time 1786313720.38 total 80.04 last 79.59) - open/close 6832 (time 1786313730.38 total 90.04 last 69.68) - open/close 7580 (time 1786313740.39 total 100.05 last 74.74) - open/close 8414 (time 1786313750.39 total 110.05 last 83.37) - open/close 9201 (time 1786313760.40 total 120.06 last 78.67) - open/close 9939 (time 1786313770.41 total 130.07 last 73.72) total: 10000 open/close in 130.68 seconds: 76.53 ops/second - unlinked 0 (time 1786313775 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg123-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 start ost1 service on oleg123-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg123-server unloading modules via unload_modules_local on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local oleg123-server: modules unloaded. pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2