== conf-sanity test 69: replace an OST with the same index ========================================================== 22:16:49 (1787365009) start mds service on oleg411-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg411-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg411-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 685 (time 1787365142.69 total 10.00 last 68.50) - open/close 1342 (time 1787365152.70 total 20.00 last 65.68) - open/close 2050 (time 1787365162.70 total 30.01 last 70.77) - open/close 2737 (time 1787365172.72 total 40.03 last 68.55) - open/close 3442 (time 1787365182.74 total 50.05 last 70.36) - open/close 4105 (time 1787365192.75 total 60.06 last 66.26) - open/close 4810 (time 1787365202.75 total 70.06 last 70.50) - open/close 5490 (time 1787365212.76 total 80.07 last 67.92) - open/close 6053 (time 1787365222.77 total 90.08 last 56.25) - open/close 6711 (time 1787365232.77 total 100.08 last 65.80) - open/close 7384 (time 1787365242.78 total 110.09 last 67.22) - open/close 8181 (time 1787365252.80 total 120.11 last 79.56) - open/close 8884 (time 1787365262.80 total 130.11 last 70.27) - open/close 9613 (time 1787365272.81 total 140.12 last 72.85) total: 10000 open/close in 145.47 seconds: 68.74 ops/second - unlinked 0 (time 1787365283 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 671 (time 1787365377.20 total 10.00 last 67.08) - open/close 1351 (time 1787365387.20 total 20.01 last 67.95) - open/close 1952 (time 1787365397.21 total 30.01 last 60.08) - open/close 2524 (time 1787365407.21 total 40.02 last 57.19) - open/close 3078 (time 1787365417.21 total 50.02 last 55.37) - open/close 3750 (time 1787365427.22 total 60.03 last 67.17) - open/close 4437 (time 1787365437.22 total 70.03 last 68.69) - open/close 5159 (time 1787365447.23 total 80.04 last 72.11) - open/close 5842 (time 1787365457.24 total 90.04 last 68.26) - open/close 6557 (time 1787365467.24 total 100.05 last 71.48) - open/close 7233 (time 1787365477.25 total 110.06 last 67.50) - open/close 7981 (time 1787365487.26 total 120.07 last 74.78) - open/close 8688 (time 1787365497.27 total 130.08 last 70.63) - open/close 9252 (time 1787365507.27 total 140.08 last 56.39) - open/close 9920 (time 1787365517.28 total 150.09 last 66.73) total: 10000 open/close in 151.34 seconds: 66.08 ops/second - unlinked 0 (time 1787365523 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 767 (time 1787365616.01 total 10.00 last 76.70) - open/close 1462 (time 1787365626.01 total 20.01 last 69.46) - open/close 2199 (time 1787365636.01 total 30.01 last 73.70) - open/close 2874 (time 1787365646.02 total 40.01 last 67.45) - open/close 3587 (time 1787365656.03 total 50.02 last 71.26) - open/close 4195 (time 1787365666.05 total 60.05 last 60.64) - open/close 4750 (time 1787365676.05 total 70.05 last 55.50) - open/close 5211 (time 1787365686.06 total 80.05 last 46.06) - open/close 5792 (time 1787365696.06 total 90.06 last 58.07) - open/close 6374 (time 1787365706.07 total 100.06 last 58.20) - open/close 7143 (time 1787365716.08 total 110.08 last 76.78) - open/close 7884 (time 1787365726.09 total 120.08 last 74.04) - open/close 8656 (time 1787365736.11 total 130.10 last 77.05) - open/close 9347 (time 1787365746.11 total 140.11 last 69.09) total: 10000 open/close in 149.02 seconds: 67.11 ops/second - unlinked 0 (time 1787365758 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 677 (time 1787365856.01 total 10.01 last 67.62) - open/close 1381 (time 1787365866.01 total 20.01 last 70.39) - open/close 2030 (time 1787365876.02 total 30.02 last 64.84) - open/close 2735 (time 1787365886.03 total 40.03 last 70.42) - open/close 3431 (time 1787365896.05 total 50.05 last 69.47) - open/close 4072 (time 1787365906.05 total 60.06 last 64.08) - open/close 4762 (time 1787365916.06 total 70.06 last 68.97) - open/close 5488 (time 1787365926.07 total 80.07 last 72.50) - open/close 6128 (time 1787365936.08 total 90.08 last 63.96) - open/close 6849 (time 1787365946.08 total 100.08 last 72.10) - open/close 7579 (time 1787365956.10 total 110.10 last 72.86) - open/close 8306 (time 1787365966.10 total 120.10 last 72.67) - open/close 9056 (time 1787365976.12 total 130.12 last 74.90) - open/close 9760 (time 1787365986.12 total 140.12 last 70.36) total: 10000 open/close in 142.88 seconds: 69.99 ops/second - unlinked 0 (time 1787365993 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 730 (time 1787366083.61 total 10.00 last 72.99) - open/close 1470 (time 1787366093.61 total 20.00 last 73.99) - open/close 2164 (time 1787366103.61 total 30.01 last 69.35) - open/close 2869 (time 1787366113.63 total 40.02 last 70.42) - open/close 3586 (time 1787366123.64 total 50.04 last 71.58) - open/close 4295 (time 1787366133.66 total 60.05 last 70.80) - open/close 5085 (time 1787366143.67 total 70.06 last 78.92) - open/close 5840 (time 1787366153.67 total 80.07 last 75.46) - open/close 6448 (time 1787366163.68 total 90.08 last 60.74) - open/close 7126 (time 1787366173.69 total 100.08 last 67.76) - open/close 7891 (time 1787366183.71 total 110.10 last 76.36) - open/close 8654 (time 1787366193.71 total 120.11 last 76.28) - open/close 9339 (time 1787366203.71 total 130.11 last 68.47) total: 10000 open/close in 138.46 seconds: 72.23 ops/second - unlinked 0 (time 1787366216 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 724 (time 1787366306.14 total 10.01 last 72.35) - open/close 1417 (time 1787366316.14 total 20.01 last 69.30) - open/close 2230 (time 1787366326.15 total 30.02 last 81.23) - open/close 3028 (time 1787366336.16 total 40.02 last 79.75) - open/close 3818 (time 1787366346.17 total 50.03 last 78.93) - open/close 4582 (time 1787366356.17 total 60.03 last 76.39) - open/close 5305 (time 1787366366.18 total 70.04 last 72.24) - open/close 5990 (time 1787366376.19 total 80.05 last 68.43) - open/close 6739 (time 1787366386.20 total 90.06 last 74.81) - open/close 7327 (time 1787366396.21 total 100.07 last 58.75) - open/close 7995 (time 1787366406.21 total 110.07 last 66.78) - open/close 8527 (time 1787366416.23 total 120.09 last 53.12) - open/close 9190 (time 1787366426.23 total 130.09 last 66.28) - open/close 9874 (time 1787366436.24 total 140.10 last 68.32) total: 10000 open/close in 141.75 seconds: 70.55 ops/second - unlinked 0 (time 1787366442 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 734 (time 1787366531.36 total 10.01 last 73.32) - open/close 1380 (time 1787366541.37 total 20.02 last 64.55) - open/close 2118 (time 1787366551.37 total 30.02 last 73.79) - open/close 2840 (time 1787366561.38 total 40.02 last 72.17) - open/close 3640 (time 1787366571.38 total 50.03 last 79.99) - open/close 4407 (time 1787366581.39 total 60.04 last 76.61) - open/close 5105 (time 1787366591.39 total 70.04 last 69.77) - open/close 5810 (time 1787366601.41 total 80.06 last 70.40) - open/close 6580 (time 1787366611.41 total 90.06 last 77.00) - open/close 7325 (time 1787366621.41 total 100.06 last 74.48) - open/close 8044 (time 1787366631.42 total 110.06 last 71.87) - open/close 8837 (time 1787366641.42 total 120.07 last 79.27) - open/close 9533 (time 1787366651.43 total 130.08 last 69.51) total: 10000 open/close in 136.93 seconds: 73.03 ops/second - unlinked 0 (time 1787366662 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 652 (time 1787366757.20 total 10.03 last 64.98) - open/close 1356 (time 1787366767.22 total 20.05 last 70.27) - open/close 2122 (time 1787366777.23 total 30.06 last 76.53) - open/close 2861 (time 1787366787.25 total 40.08 last 73.79) - open/close 3603 (time 1787366797.27 total 50.10 last 74.02) - open/close 4316 (time 1787366807.28 total 60.11 last 71.25) - open/close 5068 (time 1787366817.28 total 70.11 last 75.18) - open/close 5814 (time 1787366827.30 total 80.13 last 74.48) - open/close 6461 (time 1787366837.31 total 90.14 last 64.60) - open/close 7160 (time 1787366847.32 total 100.15 last 69.85) - open/close 7872 (time 1787366857.32 total 110.15 last 71.16) - open/close 8606 (time 1787366867.32 total 120.16 last 73.39) - open/close 9306 (time 1787366877.33 total 130.16 last 69.95) total: 10000 open/close in 139.59 seconds: 71.64 ops/second - unlinked 0 (time 1787366892 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 709 (time 1787366987.03 total 10.01 last 70.82) - open/close 1330 (time 1787366997.04 total 20.02 last 62.05) - open/close 1959 (time 1787367007.06 total 30.04 last 62.79) - open/close 2570 (time 1787367017.07 total 40.05 last 61.04) - open/close 3267 (time 1787367027.08 total 50.06 last 69.61) - open/close 3920 (time 1787367037.09 total 60.07 last 65.24) - open/close 4471 (time 1787367047.09 total 70.07 last 55.10) - open/close 5219 (time 1787367057.09 total 80.07 last 74.79) - open/close 5932 (time 1787367067.10 total 90.08 last 71.24) - open/close 6625 (time 1787367077.11 total 100.08 last 69.26) - open/close 7302 (time 1787367087.11 total 110.09 last 67.69) - open/close 7972 (time 1787367097.12 total 120.10 last 66.94) - open/close 8660 (time 1787367107.13 total 130.11 last 68.70) - open/close 9381 (time 1787367117.14 total 140.12 last 72.04) total: 10000 open/close in 147.99 seconds: 67.57 ops/second - unlinked 0 (time 1787367129 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 748 (time 1787367229.68 total 10.01 last 74.75) - open/close 1524 (time 1787367239.69 total 20.01 last 77.56) - open/close 2188 (time 1787367249.72 total 30.04 last 66.19) - open/close 2858 (time 1787367259.73 total 40.05 last 66.94) - open/close 3506 (time 1787367269.73 total 50.06 last 64.78) - open/close 4199 (time 1787367279.74 total 60.07 last 69.22) - open/close 4946 (time 1787367289.75 total 70.07 last 74.67) - open/close 5705 (time 1787367299.75 total 80.08 last 75.85) - open/close 6495 (time 1787367309.77 total 90.09 last 78.88) - open/close 7191 (time 1787367319.78 total 100.10 last 69.55) - open/close 7957 (time 1787367329.78 total 110.11 last 76.54) - open/close 8658 (time 1787367339.79 total 120.12 last 70.05) - open/close 9355 (time 1787367349.80 total 130.12 last 69.67) total: 10000 open/close in 139.75 seconds: 71.55 ops/second - unlinked 0 (time 1787367364 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server unloading modules via unload_modules_local on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: modules unloaded. pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2