== conf-sanity test 69: replace an OST with the same index ========================================================== 02:53:07 (1787381587) start mds service on oleg435-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg435-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg435-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 611 (time 1787381705.33 total 10.00 last 61.09) - open/close 1220 (time 1787381715.33 total 20.00 last 60.90) - open/close 1832 (time 1787381725.35 total 30.02 last 61.12) - open/close 2435 (time 1787381735.36 total 40.03 last 60.24) - open/close 3034 (time 1787381745.37 total 50.04 last 59.84) - open/close 3642 (time 1787381755.37 total 60.04 last 60.77) - open/close 4146 (time 1787381765.37 total 70.04 last 50.40) - open/close 4709 (time 1787381775.37 total 80.04 last 56.29) - open/close 5215 (time 1787381785.38 total 90.05 last 50.54) - open/close 5860 (time 1787381795.40 total 100.06 last 64.43) - open/close 6478 (time 1787381805.42 total 110.09 last 61.65) - open/close 7132 (time 1787381815.42 total 120.09 last 65.38) - open/close 7705 (time 1787381825.43 total 130.10 last 57.28) - open/close 8309 (time 1787381835.43 total 140.10 last 60.39) - open/close 8833 (time 1787381845.43 total 150.10 last 52.38) - open/close 9419 (time 1787381855.47 total 160.14 last 58.39) total: 10000 open/close in 169.93 seconds: 58.85 ops/second - unlinked 0 (time 1787381870 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 721 (time 1787381969.52 total 10.00 last 72.10) - open/close 1453 (time 1787381979.53 total 20.01 last 73.15) - open/close 2160 (time 1787381989.53 total 30.02 last 70.64) - open/close 2844 (time 1787381999.54 total 40.02 last 68.36) - open/close 3548 (time 1787382009.57 total 50.05 last 70.21) - open/close 4184 (time 1787382019.57 total 60.05 last 63.58) - open/close 4835 (time 1787382029.57 total 70.05 last 65.07) - open/close 5448 (time 1787382039.58 total 80.06 last 61.28) - open/close 6089 (time 1787382049.58 total 90.07 last 64.05) - open/close 6753 (time 1787382059.62 total 100.10 last 66.16) - open/close 7364 (time 1787382069.63 total 110.11 last 61.04) - open/close 8070 (time 1787382079.64 total 120.12 last 70.56) - open/close 8803 (time 1787382089.64 total 130.12 last 73.24) - open/close 9480 (time 1787382099.65 total 140.13 last 67.65) total: 10000 open/close in 147.66 seconds: 67.72 ops/second - unlinked 0 (time 1787382112 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 720 (time 1787382213.09 total 10.00 last 71.98) - open/close 1360 (time 1787382223.10 total 20.01 last 63.96) - open/close 2000 (time 1787382233.13 total 30.04 last 63.78) - open/close 2721 (time 1787382243.14 total 40.05 last 72.06) - open/close 3415 (time 1787382253.16 total 50.07 last 69.27) - open/close 3932 (time 1787382263.17 total 60.08 last 51.65) - open/close 4597 (time 1787382273.19 total 70.10 last 66.36) - open/close 5275 (time 1787382283.20 total 80.10 last 67.76) - open/close 5972 (time 1787382293.20 total 90.11 last 69.66) - open/close 6657 (time 1787382303.22 total 100.13 last 68.38) - open/close 7324 (time 1787382313.22 total 110.13 last 66.67) - open/close 8001 (time 1787382323.22 total 120.13 last 67.68) - open/close 8729 (time 1787382333.23 total 130.14 last 72.78) - open/close 9462 (time 1787382343.24 total 140.15 last 73.21) total: 10000 open/close in 147.75 seconds: 67.68 ops/second - unlinked 0 (time 1787382355 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 540 (time 1787382453.04 total 10.01 last 53.96) - open/close 1177 (time 1787382463.05 total 20.01 last 63.69) - open/close 1772 (time 1787382473.06 total 30.02 last 59.41) - open/close 2510 (time 1787382483.07 total 40.04 last 73.73) - open/close 3184 (time 1787382493.09 total 50.06 last 67.24) - open/close 3839 (time 1787382503.10 total 60.07 last 65.46) - open/close 4532 (time 1787382513.10 total 70.07 last 69.28) - open/close 5252 (time 1787382523.12 total 80.08 last 71.91) - open/close 5921 (time 1787382533.13 total 90.10 last 66.79) - open/close 6437 (time 1787382543.14 total 100.10 last 51.56) - open/close 6933 (time 1787382553.15 total 110.11 last 49.57) - open/close 7676 (time 1787382563.16 total 120.12 last 74.24) - open/close 8327 (time 1787382573.17 total 130.13 last 65.02) - open/close 9026 (time 1787382583.19 total 140.15 last 69.75) - open/close 9780 (time 1787382593.21 total 150.17 last 75.27) total: 10000 open/close in 153.45 seconds: 65.17 ops/second - unlinked 0 (time 1787382602 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 622 (time 1787382703.84 total 10.01 last 62.16) - open/close 1443 (time 1787382713.84 total 20.01 last 82.09) - open/close 2173 (time 1787382723.85 total 30.02 last 72.94) - open/close 2869 (time 1787382733.86 total 40.03 last 69.53) - open/close 3633 (time 1787382743.87 total 50.04 last 76.31) - open/close 4276 (time 1787382753.88 total 60.05 last 64.23) - open/close 4949 (time 1787382763.88 total 70.05 last 67.27) - open/close 5652 (time 1787382773.90 total 80.07 last 70.22) - open/close 6404 (time 1787382783.90 total 90.07 last 75.17) - open/close 7072 (time 1787382793.91 total 100.08 last 66.75) - open/close 7804 (time 1787382803.91 total 110.08 last 73.20) - open/close 8504 (time 1787382813.91 total 120.08 last 69.97) - open/close 9212 (time 1787382823.91 total 130.08 last 70.78) - open/close 9903 (time 1787382833.92 total 140.09 last 69.09) total: 10000 open/close in 141.34 seconds: 70.75 ops/second - unlinked 0 (time 1787382840 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 682 (time 1787382936.28 total 10.01 last 68.15) - open/close 1418 (time 1787382946.29 total 20.02 last 73.54) - open/close 2118 (time 1787382956.31 total 30.03 last 69.89) - open/close 2839 (time 1787382966.32 total 40.04 last 72.04) - open/close 3563 (time 1787382976.32 total 50.05 last 72.35) - open/close 4304 (time 1787382986.33 total 60.05 last 74.07) - open/close 4980 (time 1787382996.33 total 70.06 last 67.57) - open/close 5665 (time 1787383006.35 total 80.07 last 68.41) - open/close 6292 (time 1787383016.36 total 90.08 last 62.64) - open/close 6985 (time 1787383026.36 total 100.09 last 69.26) - open/close 7569 (time 1787383036.37 total 110.09 last 58.37) - open/close 8209 (time 1787383046.37 total 120.10 last 63.96) - open/close 8730 (time 1787383056.38 total 130.11 last 52.05) - open/close 9212 (time 1787383066.39 total 140.11 last 48.17) - open/close 9930 (time 1787383076.39 total 150.11 last 71.78) total: 10000 open/close in 150.99 seconds: 66.23 ops/second - unlinked 0 (time 1787383082 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 693 (time 1787383178.85 total 10.02 last 69.16) - open/close 1356 (time 1787383188.86 total 20.03 last 66.23) - open/close 2066 (time 1787383198.88 total 30.05 last 70.87) - open/close 2770 (time 1787383208.88 total 40.06 last 70.35) - open/close 3527 (time 1787383218.89 total 50.07 last 75.62) - open/close 4276 (time 1787383228.91 total 60.08 last 74.80) - open/close 4958 (time 1787383238.91 total 70.08 last 68.19) - open/close 5677 (time 1787383248.91 total 80.08 last 71.87) - open/close 6388 (time 1787383258.92 total 90.09 last 71.06) - open/close 7085 (time 1787383268.92 total 100.10 last 69.65) - open/close 7806 (time 1787383278.93 total 110.10 last 72.08) - open/close 8572 (time 1787383288.94 total 120.11 last 76.54) - open/close 9321 (time 1787383298.94 total 130.11 last 74.89) total: 10000 open/close in 140.02 seconds: 71.42 ops/second - unlinked 0 (time 1787383312 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 633 (time 1787383413.97 total 10.01 last 63.22) - open/close 1298 (time 1787383423.97 total 20.01 last 66.49) - open/close 2046 (time 1787383433.97 total 30.02 last 74.80) - open/close 2770 (time 1787383443.99 total 40.03 last 72.27) - open/close 3517 (time 1787383454.00 total 50.04 last 74.65) - open/close 4208 (time 1787383464.01 total 60.05 last 69.00) - open/close 4918 (time 1787383474.02 total 70.06 last 70.93) - open/close 5619 (time 1787383484.03 total 80.07 last 70.08) - open/close 6306 (time 1787383494.03 total 90.08 last 68.64) - open/close 7053 (time 1787383504.05 total 100.09 last 74.58) - open/close 7732 (time 1787383514.05 total 110.10 last 67.87) - open/close 8347 (time 1787383524.06 total 120.11 last 61.44) - open/close 8986 (time 1787383534.07 total 130.11 last 63.89) - open/close 9591 (time 1787383544.07 total 140.11 last 60.48) total: 10000 open/close in 145.48 seconds: 68.74 ops/second - unlinked 0 (time 1787383554 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 604 (time 1787383649.08 total 10.01 last 60.35) - open/close 1207 (time 1787383659.08 total 20.01 last 60.29) - open/close 1848 (time 1787383669.08 total 30.01 last 64.09) - open/close 2513 (time 1787383679.09 total 40.02 last 66.42) - open/close 3178 (time 1787383689.10 total 50.03 last 66.46) - open/close 3859 (time 1787383699.10 total 60.03 last 68.07) - open/close 4584 (time 1787383709.11 total 70.04 last 72.49) - open/close 5370 (time 1787383719.11 total 80.04 last 78.57) - open/close 6084 (time 1787383729.11 total 90.04 last 71.38) - open/close 6785 (time 1787383739.11 total 100.04 last 70.09) - open/close 7510 (time 1787383749.12 total 110.05 last 72.45) - open/close 8187 (time 1787383759.13 total 120.06 last 67.63) - open/close 8823 (time 1787383769.14 total 130.07 last 63.56) - open/close 9500 (time 1787383779.14 total 140.07 last 67.67) total: 10000 open/close in 147.64 seconds: 67.73 ops/second - unlinked 0 (time 1787383791 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 723 (time 1787383884.53 total 10.01 last 72.21) - open/close 1442 (time 1787383894.54 total 20.02 last 71.87) - open/close 2142 (time 1787383904.54 total 30.02 last 69.99) - open/close 2840 (time 1787383914.54 total 40.02 last 69.80) - open/close 3521 (time 1787383924.55 total 50.03 last 68.04) - open/close 4212 (time 1787383934.55 total 60.03 last 69.08) - open/close 4813 (time 1787383944.56 total 70.04 last 60.06) - open/close 5405 (time 1787383954.58 total 80.06 last 59.09) - open/close 6026 (time 1787383964.58 total 90.06 last 62.07) - open/close 6716 (time 1787383974.59 total 100.07 last 68.95) - open/close 7359 (time 1787383984.59 total 110.07 last 64.28) - open/close 8011 (time 1787383994.60 total 120.08 last 65.17) - open/close 8676 (time 1787384004.61 total 130.09 last 66.42) - open/close 9391 (time 1787384014.63 total 140.11 last 71.37) total: 10000 open/close in 148.54 seconds: 67.32 ops/second - unlinked 0 (time 1787384028 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server unloading modules via unload_modules_local on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2