== conf-sanity test 69: replace an OST with the same index ========================================================== 04:34:13 (1788165253) start mds service on oleg451-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg451-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg451-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 774 (time 1788165385.57 total 10.01 last 77.33) - open/close 1596 (time 1788165395.57 total 20.02 last 82.15) - open/close 2350 (time 1788165405.58 total 30.02 last 75.35) - open/close 3123 (time 1788165415.58 total 40.02 last 77.29) - open/close 3828 (time 1788165425.59 total 50.03 last 70.45) - open/close 4610 (time 1788165435.60 total 60.04 last 78.14) - open/close 5307 (time 1788165445.60 total 70.04 last 69.69) - open/close 5947 (time 1788165455.60 total 80.04 last 63.99) - open/close 6772 (time 1788165465.60 total 90.04 last 82.50) - open/close 7463 (time 1788165475.61 total 100.05 last 69.02) - open/close 8213 (time 1788165485.62 total 110.06 last 74.97) - open/close 8945 (time 1788165495.62 total 120.06 last 73.17) - open/close 9699 (time 1788165505.62 total 130.06 last 75.38) total: 10000 open/close in 133.54 seconds: 74.89 ops/second - unlinked 0 (time 1788165512 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 736 (time 1788165606.81 total 10.00 last 73.58) - open/close 1482 (time 1788165616.82 total 20.01 last 74.56) - open/close 2200 (time 1788165626.82 total 30.01 last 71.80) - open/close 2930 (time 1788165636.83 total 40.02 last 72.90) - open/close 3738 (time 1788165646.84 total 50.03 last 80.72) - open/close 4450 (time 1788165656.86 total 60.05 last 71.09) - open/close 5173 (time 1788165666.86 total 70.06 last 72.24) - open/close 5937 (time 1788165676.87 total 80.06 last 76.39) - open/close 6618 (time 1788165686.87 total 90.06 last 68.09) - open/close 7278 (time 1788165696.88 total 100.07 last 65.94) - open/close 7941 (time 1788165706.90 total 110.09 last 66.14) - open/close 8683 (time 1788165716.90 total 120.10 last 74.17) - open/close 9418 (time 1788165726.90 total 130.10 last 73.49) total: 10000 open/close in 137.07 seconds: 72.95 ops/second - unlinked 0 (time 1788165737 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 729 (time 1788165830.25 total 10.02 last 72.78) - open/close 1378 (time 1788165840.25 total 20.02 last 64.87) - open/close 2078 (time 1788165850.25 total 30.02 last 69.99) - open/close 2796 (time 1788165860.26 total 40.03 last 71.74) - open/close 3498 (time 1788165870.27 total 50.04 last 70.17) - open/close 4230 (time 1788165880.27 total 60.04 last 73.15) - open/close 4962 (time 1788165890.28 total 70.05 last 73.13) - open/close 5709 (time 1788165900.29 total 80.06 last 74.64) - open/close 6349 (time 1788165910.30 total 90.07 last 63.94) - open/close 6945 (time 1788165920.30 total 100.07 last 59.59) - open/close 7541 (time 1788165930.30 total 110.07 last 59.59) - open/close 8243 (time 1788165940.31 total 120.08 last 70.18) - open/close 8966 (time 1788165950.31 total 130.08 last 72.25) - open/close 9646 (time 1788165960.32 total 140.09 last 67.97) total: 10000 open/close in 146.08 seconds: 68.46 ops/second - unlinked 0 (time 1788165972 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 610 (time 1788166076.30 total 10.01 last 60.96) - open/close 1242 (time 1788166086.31 total 20.01 last 63.20) - open/close 1944 (time 1788166096.31 total 30.01 last 70.19) - open/close 2522 (time 1788166106.32 total 40.02 last 57.75) - open/close 3137 (time 1788166116.32 total 50.02 last 61.49) - open/close 3818 (time 1788166126.33 total 60.03 last 68.03) - open/close 4538 (time 1788166136.33 total 70.03 last 71.98) - open/close 5281 (time 1788166146.33 total 80.04 last 74.27) - open/close 5895 (time 1788166156.35 total 90.06 last 61.28) - open/close 6560 (time 1788166166.36 total 100.06 last 66.48) - open/close 7249 (time 1788166176.36 total 110.07 last 68.85) - open/close 7971 (time 1788166186.37 total 120.07 last 72.16) - open/close 8660 (time 1788166196.38 total 130.08 last 68.86) - open/close 9386 (time 1788166206.38 total 140.08 last 72.56) total: 10000 open/close in 148.55 seconds: 67.32 ops/second - unlinked 0 (time 1788166219 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 579 (time 1788166321.14 total 10.01 last 57.85) - open/close 1337 (time 1788166331.16 total 20.02 last 75.69) - open/close 2030 (time 1788166341.17 total 30.03 last 69.23) - open/close 2669 (time 1788166351.18 total 40.05 last 63.81) - open/close 3336 (time 1788166361.18 total 50.05 last 66.68) - open/close 3886 (time 1788166371.20 total 60.06 last 54.92) - open/close 4524 (time 1788166381.21 total 70.08 last 63.73) - open/close 5213 (time 1788166391.22 total 80.08 last 68.84) - open/close 5859 (time 1788166401.23 total 90.10 last 64.52) - open/close 6534 (time 1788166411.23 total 100.10 last 67.49) - open/close 7190 (time 1788166421.24 total 110.11 last 65.55) - open/close 7806 (time 1788166431.26 total 120.13 last 61.46) - open/close 8549 (time 1788166441.28 total 130.14 last 74.18) - open/close 9263 (time 1788166451.30 total 140.17 last 71.21) - open/close 9959 (time 1788166461.32 total 150.19 last 69.49) total: 10000 open/close in 150.71 seconds: 66.35 ops/second - unlinked 0 (time 1788166467 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 736 (time 1788166559.99 total 10.00 last 73.57) - open/close 1475 (time 1788166570.00 total 20.01 last 73.86) - open/close 2248 (time 1788166580.00 total 30.01 last 77.27) - open/close 2976 (time 1788166590.01 total 40.02 last 72.78) - open/close 3647 (time 1788166600.01 total 50.02 last 67.06) - open/close 4363 (time 1788166610.01 total 60.02 last 71.60) - open/close 4979 (time 1788166620.02 total 70.03 last 61.56) - open/close 5708 (time 1788166630.03 total 80.04 last 72.83) - open/close 6387 (time 1788166640.03 total 90.04 last 67.89) - open/close 7007 (time 1788166650.04 total 100.05 last 61.94) - open/close 7567 (time 1788166660.05 total 110.06 last 55.93) - open/close 8123 (time 1788166670.08 total 120.09 last 55.43) - open/close 8661 (time 1788166680.09 total 130.10 last 53.77) - open/close 9437 (time 1788166690.09 total 140.10 last 77.57) total: 10000 open/close in 148.09 seconds: 67.53 ops/second - unlinked 0 (time 1788166702 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 642 (time 1788166798.38 total 10.02 last 64.06) - open/close 1335 (time 1788166808.39 total 20.03 last 69.26) - open/close 2111 (time 1788166818.42 total 30.06 last 77.38) - open/close 2874 (time 1788166828.43 total 40.06 last 76.24) - open/close 3577 (time 1788166838.44 total 50.08 last 70.21) - open/close 4228 (time 1788166848.46 total 60.10 last 64.96) - open/close 4833 (time 1788166858.46 total 70.10 last 60.48) - open/close 5499 (time 1788166868.48 total 80.12 last 66.46) - open/close 6114 (time 1788166878.49 total 90.13 last 61.47) - open/close 6700 (time 1788166888.49 total 100.13 last 58.58) - open/close 7379 (time 1788166898.49 total 110.13 last 67.89) - open/close 8032 (time 1788166908.50 total 120.14 last 65.28) - open/close 8675 (time 1788166918.50 total 130.14 last 64.27) - open/close 9365 (time 1788166928.52 total 140.16 last 68.88) total: 10000 open/close in 148.71 seconds: 67.24 ops/second - unlinked 0 (time 1788166941 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 632 (time 1788167044.04 total 10.02 last 63.08) - open/close 1395 (time 1788167054.04 total 20.02 last 76.29) - open/close 2162 (time 1788167064.05 total 30.03 last 76.62) - open/close 2874 (time 1788167074.06 total 40.04 last 71.13) - open/close 3613 (time 1788167084.06 total 50.04 last 73.87) - open/close 4337 (time 1788167094.08 total 60.06 last 72.28) - open/close 5114 (time 1788167104.08 total 70.07 last 77.65) - open/close 5830 (time 1788167114.09 total 80.08 last 71.53) - open/close 6528 (time 1788167124.10 total 90.09 last 69.73) - open/close 7312 (time 1788167134.11 total 100.09 last 78.36) - open/close 8076 (time 1788167144.11 total 110.10 last 76.38) - open/close 8794 (time 1788167154.12 total 120.10 last 71.74) - open/close 9470 (time 1788167164.13 total 130.11 last 67.56) total: 10000 open/close in 137.74 seconds: 72.60 ops/second - unlinked 0 (time 1788167175 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 673 (time 1788167270.25 total 10.00 last 67.27) - open/close 1447 (time 1788167280.26 total 20.01 last 77.37) - open/close 2173 (time 1788167290.28 total 30.03 last 72.45) - open/close 2859 (time 1788167300.28 total 40.03 last 68.57) - open/close 3564 (time 1788167310.29 total 50.04 last 70.44) - open/close 4250 (time 1788167320.29 total 60.05 last 68.57) - open/close 4912 (time 1788167330.31 total 70.06 last 66.10) - open/close 5654 (time 1788167340.31 total 80.06 last 74.18) - open/close 6505 (time 1788167350.31 total 90.07 last 85.07) - open/close 7263 (time 1788167360.33 total 100.08 last 75.69) - open/close 7961 (time 1788167370.34 total 110.09 last 69.73) - open/close 8658 (time 1788167380.36 total 120.12 last 69.54) - open/close 9291 (time 1788167390.37 total 130.12 last 63.24) total: 10000 open/close in 140.08 seconds: 71.39 ops/second - unlinked 0 (time 1788167404 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 750 (time 1788167496.12 total 10.01 last 74.94) - open/close 1537 (time 1788167506.12 total 20.01 last 78.70) - open/close 2229 (time 1788167516.12 total 30.01 last 69.19) - open/close 3007 (time 1788167526.12 total 40.01 last 77.79) - open/close 3708 (time 1788167536.13 total 50.02 last 70.04) - open/close 4425 (time 1788167546.14 total 60.03 last 71.65) - open/close 5185 (time 1788167556.14 total 70.03 last 75.99) - open/close 5899 (time 1788167566.16 total 80.05 last 71.27) - open/close 6661 (time 1788167576.17 total 90.06 last 76.12) - open/close 7389 (time 1788167586.17 total 100.06 last 72.75) - open/close 8180 (time 1788167596.18 total 110.06 last 79.10) - open/close 8897 (time 1788167606.18 total 120.07 last 71.67) - open/close 9600 (time 1788167616.19 total 130.08 last 70.20) total: 10000 open/close in 134.39 seconds: 74.41 ops/second - unlinked 0 (time 1788167625 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local oleg451-server: modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2