== conf-sanity test 69: replace an OST with the same index ========================================================== 06:07:17 (1787825237) start mds service on oleg226-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg226-server: detected 4 online CPUs by sysfs oleg226-server: MODOPTS_LIBCFS= oleg226-server: Force libcfs to create 2 CPU partitions oleg226-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg226-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg226-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg226-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 679 (time 1787825364.48 total 10.02 last 67.78) - open/close 1394 (time 1787825374.49 total 20.02 last 71.45) - open/close 2085 (time 1787825384.50 total 30.03 last 69.04) - open/close 2830 (time 1787825394.50 total 40.04 last 74.49) - open/close 3717 (time 1787825404.50 total 50.04 last 88.68) - open/close 4572 (time 1787825414.52 total 60.06 last 85.30) - open/close 5497 (time 1787825424.53 total 70.07 last 92.42) - open/close 6325 (time 1787825434.55 total 80.08 last 82.67) - open/close 7047 (time 1787825444.55 total 90.09 last 72.19) - open/close 7825 (time 1787825454.55 total 100.09 last 77.77) - open/close 8684 (time 1787825464.57 total 110.10 last 85.78) - open/close 9543 (time 1787825474.58 total 120.12 last 85.77) total: 10000 open/close in 126.47 seconds: 79.07 ops/second - unlinked 0 (time 1787825485 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 758 (time 1787825568.47 total 10.02 last 75.68) - open/close 1669 (time 1787825578.47 total 20.02 last 91.07) - open/close 2474 (time 1787825588.48 total 30.02 last 80.49) - open/close 3223 (time 1787825598.49 total 40.04 last 74.79) - open/close 3953 (time 1787825608.49 total 50.04 last 72.99) - open/close 4699 (time 1787825618.50 total 60.04 last 74.55) - open/close 5543 (time 1787825628.50 total 70.04 last 84.39) - open/close 6266 (time 1787825638.50 total 80.05 last 72.28) - open/close 7062 (time 1787825648.51 total 90.06 last 79.53) - open/close 7807 (time 1787825658.53 total 100.07 last 74.39) - open/close 8548 (time 1787825668.53 total 110.07 last 74.06) - open/close 9340 (time 1787825678.53 total 120.08 last 79.20) total: 10000 open/close in 128.34 seconds: 77.92 ops/second - unlinked 0 (time 1787825691 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 663 (time 1787825779.78 total 10.00 last 66.29) - open/close 1505 (time 1787825789.80 total 20.02 last 84.04) - open/close 2234 (time 1787825799.82 total 30.04 last 72.74) - open/close 2937 (time 1787825809.82 total 40.05 last 70.24) - open/close 3723 (time 1787825819.83 total 50.05 last 78.58) - open/close 4496 (time 1787825829.84 total 60.06 last 77.22) - open/close 5349 (time 1787825839.84 total 70.07 last 85.25) - open/close 6235 (time 1787825849.85 total 80.07 last 88.55) - open/close 7122 (time 1787825859.85 total 90.07 last 88.70) - open/close 8054 (time 1787825869.85 total 100.08 last 93.18) - open/close 8797 (time 1787825879.86 total 110.08 last 74.26) - open/close 9596 (time 1787825889.86 total 120.08 last 79.89) total: 10000 open/close in 125.82 seconds: 79.48 ops/second - unlinked 0 (time 1787825900 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 719 (time 1787825982.83 total 10.01 last 71.84) - open/close 1535 (time 1787825992.83 total 20.01 last 81.59) - open/close 2260 (time 1787826002.84 total 30.01 last 72.49) - open/close 2957 (time 1787826012.85 total 40.02 last 69.63) - open/close 3801 (time 1787826022.85 total 50.03 last 84.34) - open/close 4359 (time 1787826032.86 total 60.04 last 55.74) - open/close 4913 (time 1787826042.87 total 70.04 last 55.38) - open/close 5658 (time 1787826052.88 total 80.05 last 74.43) - open/close 6391 (time 1787826062.88 total 90.06 last 73.25) - open/close 7366 (time 1787826072.88 total 100.06 last 97.48) - open/close 8188 (time 1787826082.89 total 110.06 last 82.18) - open/close 8998 (time 1787826092.89 total 120.07 last 80.96) - open/close 9804 (time 1787826102.90 total 130.08 last 80.54) total: 10000 open/close in 131.94 seconds: 75.79 ops/second - unlinked 0 (time 1787826109 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 796 (time 1787826193.53 total 10.01 last 79.51) - open/close 1519 (time 1787826203.53 total 20.01 last 72.28) - open/close 2444 (time 1787826213.54 total 30.01 last 92.49) - open/close 3139 (time 1787826223.54 total 40.02 last 69.49) - open/close 3899 (time 1787826233.54 total 50.02 last 75.97) - open/close 4675 (time 1787826243.54 total 60.02 last 77.58) - open/close 5203 (time 1787826253.56 total 70.04 last 52.74) - open/close 6002 (time 1787826263.56 total 80.04 last 79.85) - open/close 6626 (time 1787826273.57 total 90.05 last 62.33) - open/close 7381 (time 1787826283.58 total 100.06 last 75.41) - open/close 8061 (time 1787826293.60 total 110.08 last 67.91) - open/close 8869 (time 1787826303.60 total 120.08 last 80.79) - open/close 9729 (time 1787826313.61 total 130.09 last 85.88) total: 10000 open/close in 134.32 seconds: 74.45 ops/second - unlinked 0 (time 1787826322 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 905 (time 1787826413.39 total 10.00 last 90.46) - open/close 2084 (time 1787826423.39 total 20.01 last 117.86) - open/close 2891 (time 1787826433.41 total 30.03 last 80.56) - open/close 3516 (time 1787826443.43 total 40.05 last 62.36) - open/close 4484 (time 1787826453.44 total 50.05 last 96.75) - open/close 5277 (time 1787826463.44 total 60.05 last 79.30) - open/close 5973 (time 1787826473.44 total 70.06 last 69.58) - open/close 6796 (time 1787826483.46 total 80.07 last 82.17) - open/close 7550 (time 1787826493.46 total 90.08 last 75.37) - open/close 8337 (time 1787826503.47 total 100.08 last 78.65) - open/close 9070 (time 1787826513.48 total 110.09 last 73.23) - open/close 9938 (time 1787826523.48 total 120.10 last 86.72) total: 10000 open/close in 120.97 seconds: 82.66 ops/second - unlinked 0 (time 1787826529 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 859 (time 1787826618.85 total 10.01 last 85.85) - open/close 1625 (time 1787826628.86 total 20.01 last 76.55) - open/close 2419 (time 1787826638.86 total 30.02 last 79.36) - open/close 3338 (time 1787826648.87 total 40.03 last 91.77) - open/close 4105 (time 1787826658.90 total 50.05 last 76.54) - open/close 4887 (time 1787826668.90 total 60.06 last 78.13) - open/close 5802 (time 1787826678.91 total 70.06 last 91.47) - open/close 6560 (time 1787826688.91 total 80.07 last 75.77) - open/close 7331 (time 1787826698.92 total 90.08 last 77.04) - open/close 8210 (time 1787826708.93 total 100.08 last 87.83) - open/close 9044 (time 1787826718.93 total 110.08 last 83.40) - open/close 9805 (time 1787826728.94 total 120.09 last 76.03) total: 10000 open/close in 123.04 seconds: 81.28 ops/second - unlinked 0 (time 1787826737 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 871 (time 1787826832.34 total 10.00 last 87.07) - open/close 1555 (time 1787826842.35 total 20.01 last 68.33) - open/close 2427 (time 1787826852.36 total 30.02 last 87.15) - open/close 3308 (time 1787826862.36 total 40.02 last 88.08) - open/close 3945 (time 1787826872.37 total 50.03 last 63.64) - open/close 4611 (time 1787826882.38 total 60.04 last 66.52) - open/close 5403 (time 1787826892.39 total 70.05 last 79.14) - open/close 6106 (time 1787826902.39 total 80.05 last 70.27) - open/close 6975 (time 1787826912.40 total 90.06 last 86.84) - open/close 7674 (time 1787826922.41 total 100.07 last 69.85) - open/close 8449 (time 1787826932.41 total 110.07 last 77.47) - open/close 9095 (time 1787826942.41 total 120.08 last 64.59) - open/close 9955 (time 1787826952.42 total 130.08 last 85.94) total: 10000 open/close in 130.64 seconds: 76.55 ops/second - unlinked 0 (time 1787826958 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 818 (time 1787827045.99 total 10.01 last 81.75) - open/close 1477 (time 1787827056.02 total 20.03 last 65.71) - open/close 2023 (time 1787827066.03 total 30.04 last 54.55) - open/close 2770 (time 1787827076.04 total 40.05 last 74.65) - open/close 3609 (time 1787827086.04 total 50.06 last 83.86) - open/close 4514 (time 1787827096.05 total 60.06 last 90.46) - open/close 5215 (time 1787827106.05 total 70.06 last 70.07) - open/close 5934 (time 1787827116.06 total 80.07 last 71.84) - open/close 6749 (time 1787827126.08 total 90.09 last 81.35) - open/close 7472 (time 1787827136.10 total 100.11 last 72.18) - open/close 8092 (time 1787827146.11 total 110.13 last 61.89) - open/close 8703 (time 1787827156.12 total 120.13 last 61.08) - open/close 9409 (time 1787827166.12 total 130.13 last 70.58) total: 10000 open/close in 138.08 seconds: 72.42 ops/second - unlinked 0 (time 1787827179 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 750 (time 1787827269.57 total 10.00 last 74.99) - open/close 1610 (time 1787827279.57 total 20.00 last 86.00) - open/close 2428 (time 1787827289.58 total 30.01 last 81.72) - open/close 3224 (time 1787827299.59 total 40.02 last 79.56) - open/close 4114 (time 1787827309.59 total 50.02 last 88.93) - open/close 5006 (time 1787827319.59 total 60.02 last 89.20) - open/close 5718 (time 1787827329.61 total 70.04 last 71.10) - open/close 6333 (time 1787827339.61 total 80.04 last 61.48) - open/close 6998 (time 1787827349.61 total 90.04 last 66.50) - open/close 7680 (time 1787827359.63 total 100.06 last 68.08) - open/close 8333 (time 1787827369.64 total 110.07 last 65.23) - open/close 9123 (time 1787827379.65 total 120.08 last 78.90) - open/close 9962 (time 1787827389.66 total 130.09 last 83.81) total: 10000 open/close in 130.97 seconds: 76.35 ops/second - unlinked 0 (time 1787827396 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg226-server unloading modules via unload_modules_local on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local oleg226-server: modules unloaded. pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2