== conf-sanity test 69: replace an OST with the same index ========================================================== 11:23:16 (1780327396) start mds service on oleg455-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg455-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg455-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg455-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg455-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1099 (time 1780327465.53 total 10.00 last 109.85) - open/close 2284 (time 1780327475.53 total 20.01 last 118.46) - open/close 3254 (time 1780327485.55 total 30.03 last 96.83) - open/close 4225 (time 1780327495.56 total 40.03 last 97.04) - open/close 5213 (time 1780327505.56 total 50.04 last 98.76) - open/close 6233 (time 1780327515.57 total 60.04 last 101.94) - open/close 7164 (time 1780327525.57 total 70.04 last 93.07) - open/close 8167 (time 1780327535.58 total 80.05 last 100.24) - open/close 9267 (time 1780327545.59 total 90.06 last 109.89) total: 10000 open/close in 97.09 seconds: 103.00 ops/second - unlinked 0 (time 1780327555 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1060 (time 1780327620.41 total 10.00 last 105.97) - open/close 2166 (time 1780327630.42 total 20.01 last 110.48) - open/close 3315 (time 1780327640.43 total 30.02 last 114.80) - open/close 4558 (time 1780327650.44 total 40.03 last 124.26) - open/close 5633 (time 1780327660.44 total 50.03 last 107.42) - open/close 6749 (time 1780327670.45 total 60.04 last 111.55) - open/close 7842 (time 1780327680.45 total 70.04 last 109.30) - open/close 9019 (time 1780327690.45 total 80.04 last 117.68) total: 10000 open/close in 88.23 seconds: 113.35 ops/second - unlinked 0 (time 1780327700 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1354 (time 1780327756.20 total 10.00 last 135.35) - open/close 2797 (time 1780327766.20 total 20.01 last 144.28) - open/close 4379 (time 1780327776.20 total 30.01 last 158.16) - open/close 5719 (time 1780327786.20 total 40.01 last 133.96) - open/close 6871 (time 1780327796.21 total 50.02 last 115.14) - open/close 8021 (time 1780327806.22 total 60.02 last 114.89) - open/close 9258 (time 1780327816.22 total 70.03 last 123.67) total: 10000 open/close in 75.61 seconds: 132.25 ops/second - unlinked 0 (time 1780327823 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 558 (time 1780327926.84 total 10.00 last 55.79) - open/close 1127 (time 1780327936.86 total 20.02 last 56.79) - open/close 1657 (time 1780327946.87 total 30.03 last 52.96) - open/close 2405 (time 1780327956.88 total 40.04 last 74.75) - open/close 3136 (time 1780327966.88 total 50.04 last 73.03) - open/close 3718 (time 1780327976.89 total 60.05 last 58.15) - open/close 4262 (time 1780327986.90 total 70.06 last 54.36) - open/close 4962 (time 1780327996.91 total 80.07 last 69.91) - open/close 5646 (time 1780328006.92 total 90.08 last 68.35) - open/close 6438 (time 1780328016.94 total 100.10 last 79.06) - open/close 7225 (time 1780328026.94 total 110.10 last 78.67) - open/close 7953 (time 1780328036.96 total 120.12 last 72.70) - open/close 8769 (time 1780328046.96 total 130.12 last 81.58) - open/close 9583 (time 1780328056.97 total 140.13 last 81.30) total: 10000 open/close in 145.47 seconds: 68.74 ops/second - unlinked 0 (time 1780328066 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 781 (time 1780328150.39 total 10.00 last 78.06) - open/close 1590 (time 1780328160.39 total 20.01 last 80.89) - open/close 2219 (time 1780328170.40 total 30.01 last 62.87) - open/close 2959 (time 1780328180.41 total 40.03 last 73.88) - open/close 3442 (time 1780328190.44 total 50.06 last 48.15) - open/close 4214 (time 1780328200.46 total 60.07 last 77.08) - open/close 4960 (time 1780328210.48 total 70.10 last 74.41) - open/close 5461 (time 1780328220.52 total 80.13 last 49.92) - open/close 6004 (time 1780328230.54 total 90.15 last 54.20) - open/close 6735 (time 1780328240.54 total 100.16 last 73.06) - open/close 7276 (time 1780328250.55 total 110.17 last 54.06) - open/close 7919 (time 1780328260.55 total 120.17 last 64.29) - open/close 8719 (time 1780328270.55 total 130.17 last 80.00) - open/close 9328 (time 1780328280.56 total 140.17 last 60.89) - open/close 9968 (time 1780328290.56 total 150.18 last 63.96) total: 10000 open/close in 150.54 seconds: 66.43 ops/second - unlinked 0 (time 1780328297 ; total 1 ; last 1) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 881 (time 1780328389.75 total 10.01 last 88.06) - open/close 1786 (time 1780328399.75 total 20.01 last 90.46) - open/close 2654 (time 1780328409.75 total 30.01 last 86.78) - open/close 3564 (time 1780328419.76 total 40.01 last 90.99) - open/close 4389 (time 1780328429.76 total 50.02 last 82.44) - open/close 5149 (time 1780328439.77 total 60.02 last 75.97) - open/close 6014 (time 1780328449.77 total 70.03 last 86.48) - open/close 6792 (time 1780328459.78 total 80.03 last 77.74) - open/close 7590 (time 1780328469.78 total 90.03 last 79.79) - open/close 8447 (time 1780328479.79 total 100.04 last 85.62) - open/close 9209 (time 1780328489.79 total 110.05 last 76.16) total: 10000 open/close in 119.80 seconds: 83.47 ops/second - unlinked 0 (time 1780328504 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 793 (time 1780328594.24 total 10.00 last 79.28) - open/close 1518 (time 1780328604.26 total 20.02 last 72.40) - open/close 2300 (time 1780328614.27 total 30.03 last 78.07) - open/close 3063 (time 1780328624.28 total 40.04 last 76.26) - open/close 3876 (time 1780328634.28 total 50.04 last 81.27) - open/close 4598 (time 1780328644.28 total 60.04 last 72.19) - open/close 5322 (time 1780328654.29 total 70.05 last 72.33) - open/close 6037 (time 1780328664.30 total 80.06 last 71.43) - open/close 6629 (time 1780328674.31 total 90.07 last 59.13) - open/close 7472 (time 1780328684.33 total 100.09 last 84.21) - open/close 8261 (time 1780328694.34 total 110.10 last 78.82) - open/close 9042 (time 1780328704.34 total 120.10 last 78.09) - open/close 9863 (time 1780328714.34 total 130.10 last 82.07) total: 10000 open/close in 131.86 seconds: 75.84 ops/second - unlinked 0 (time 1780328720 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 881 (time 1780328812.84 total 10.01 last 87.99) - open/close 1762 (time 1780328822.85 total 20.02 last 88.04) - open/close 2606 (time 1780328832.87 total 30.03 last 84.27) - open/close 3421 (time 1780328842.88 total 40.05 last 81.35) - open/close 4253 (time 1780328852.89 total 50.06 last 83.13) - open/close 5127 (time 1780328862.90 total 60.06 last 87.37) - open/close 6058 (time 1780328872.90 total 70.07 last 93.05) - open/close 6874 (time 1780328882.91 total 80.08 last 81.55) - open/close 7710 (time 1780328892.91 total 90.08 last 83.57) - open/close 8620 (time 1780328902.92 total 100.09 last 90.90) - open/close 9592 (time 1780328912.93 total 110.09 last 97.16) total: 10000 open/close in 115.00 seconds: 86.95 ops/second - unlinked 0 (time 1780328920 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1286 (time 1780328982.98 total 10.00 last 128.58) - open/close 2672 (time 1780328992.99 total 20.01 last 138.53) - open/close 3777 (time 1780329003.00 total 30.01 last 110.44) - open/close 5233 (time 1780329013.00 total 40.02 last 145.55) - open/close 6763 (time 1780329023.00 total 50.02 last 152.96) - open/close 8296 (time 1780329033.00 total 60.02 last 153.28) - open/close 9906 (time 1780329043.01 total 70.02 last 160.96) total: 10000 open/close in 70.54 seconds: 141.76 ops/second - unlinked 0 (time 1780329045 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1931 (time 1780329090.33 total 10.01 last 192.99) - open/close 3688 (time 1780329100.34 total 20.01 last 175.62) - open/close 5276 (time 1780329110.34 total 30.01 last 158.75) - open/close 6815 (time 1780329120.35 total 40.02 last 153.84) - open/close 8608 (time 1780329130.35 total 50.02 last 179.26) total: 10000 open/close in 59.26 seconds: 168.76 ops/second - unlinked 0 (time 1780329141 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.155@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg455-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg455-server: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg455-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec oleg455-server: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg455-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre On OST0, 15995 used inodes rc=0 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.155@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.155@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.155@tcp sys.timeout=20 start ost1 service on oleg455-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg455-server: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg455-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg455-server: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg455-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg455-server unloading modules via unload_modules_local on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local oleg455-server: modules unloaded. pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2