== conf-sanity test 69: replace an OST with the same index ========================================================== 09:23:06 (1783948986) start mds service on oleg243-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg243-server: detected 4 online CPUs by sysfs oleg243-server: MODOPTS_LIBCFS= oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg243-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg243-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 755 (time 1783949115.52 total 10.00 last 75.47) - open/close 1487 (time 1783949125.52 total 20.01 last 73.16) - open/close 2303 (time 1783949135.53 total 30.02 last 81.52) - open/close 3083 (time 1783949145.54 total 40.03 last 77.92) - open/close 3766 (time 1783949155.55 total 50.03 last 68.27) - open/close 4570 (time 1783949165.57 total 60.06 last 80.22) - open/close 5219 (time 1783949175.58 total 70.07 last 64.82) - open/close 5923 (time 1783949185.60 total 80.09 last 70.27) - open/close 6710 (time 1783949195.62 total 90.10 last 78.58) - open/close 7431 (time 1783949205.62 total 100.11 last 72.05) - open/close 8200 (time 1783949215.63 total 110.12 last 76.86) - open/close 9088 (time 1783949225.64 total 120.12 last 88.73) - open/close 9901 (time 1783949235.65 total 130.14 last 81.19) total: 10000 open/close in 131.64 seconds: 75.96 ops/second - unlinked 0 (time 1783949241 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 811 (time 1783949331.39 total 10.02 last 80.94) - open/close 1653 (time 1783949341.40 total 20.03 last 84.12) - open/close 2526 (time 1783949351.41 total 30.04 last 87.20) - open/close 3296 (time 1783949361.42 total 40.05 last 76.92) - open/close 3958 (time 1783949371.44 total 50.07 last 66.08) - open/close 4757 (time 1783949381.44 total 60.07 last 79.90) - open/close 5496 (time 1783949391.47 total 70.09 last 73.73) - open/close 6190 (time 1783949401.47 total 80.09 last 69.39) - open/close 6872 (time 1783949411.47 total 90.10 last 68.17) - open/close 7482 (time 1783949421.48 total 100.11 last 60.94) - open/close 8091 (time 1783949431.49 total 110.12 last 60.84) - open/close 8782 (time 1783949441.50 total 120.12 last 69.07) - open/close 9528 (time 1783949451.50 total 130.13 last 74.55) total: 10000 open/close in 136.88 seconds: 73.06 ops/second - unlinked 0 (time 1783949463 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 775 (time 1783949546.15 total 10.01 last 77.43) - open/close 1730 (time 1783949556.16 total 20.02 last 95.43) - open/close 2691 (time 1783949566.17 total 30.02 last 96.05) - open/close 3804 (time 1783949576.17 total 40.03 last 111.23) - open/close 4884 (time 1783949586.18 total 50.04 last 107.87) - open/close 5955 (time 1783949596.19 total 60.05 last 107.01) - open/close 6569 (time 1783949606.20 total 70.05 last 61.38) - open/close 7428 (time 1783949616.20 total 80.05 last 85.88) - open/close 8204 (time 1783949626.21 total 90.07 last 77.48) - open/close 8858 (time 1783949636.22 total 100.07 last 65.38) - open/close 9692 (time 1783949646.22 total 110.07 last 83.40) total: 10000 open/close in 113.31 seconds: 88.25 ops/second - unlinked 0 (time 1783949653 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 887 (time 1783949730.62 total 10.01 last 88.60) - open/close 1877 (time 1783949740.62 total 20.01 last 98.98) - open/close 2702 (time 1783949750.62 total 30.01 last 82.49) - open/close 3375 (time 1783949760.63 total 40.02 last 67.25) - open/close 4247 (time 1783949770.64 total 50.03 last 87.16) - open/close 5033 (time 1783949780.65 total 60.04 last 78.52) - open/close 5793 (time 1783949790.66 total 70.05 last 75.90) - open/close 6666 (time 1783949800.66 total 80.05 last 87.29) - open/close 7551 (time 1783949810.66 total 90.05 last 88.48) - open/close 8403 (time 1783949820.67 total 100.06 last 85.13) - open/close 9120 (time 1783949830.68 total 110.07 last 71.66) total: 10000 open/close in 118.70 seconds: 84.24 ops/second - unlinked 0 (time 1783949844 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 950 (time 1783949931.42 total 10.00 last 94.95) - open/close 1900 (time 1783949941.42 total 20.01 last 94.96) - open/close 2906 (time 1783949951.43 total 30.02 last 100.51) - open/close 3902 (time 1783949961.43 total 40.02 last 99.58) - open/close 4840 (time 1783949971.44 total 50.03 last 93.75) - open/close 5771 (time 1783949981.44 total 60.03 last 93.07) - open/close 6563 (time 1783949991.44 total 70.03 last 79.18) - open/close 7299 (time 1783950001.44 total 80.03 last 73.59) - open/close 8204 (time 1783950011.45 total 90.03 last 90.49) - open/close 9110 (time 1783950021.46 total 100.04 last 90.50) total: 10000 open/close in 109.01 seconds: 91.73 ops/second - unlinked 0 (time 1783950034 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 895 (time 1783950125.51 total 10.01 last 89.37) - open/close 1778 (time 1783950135.51 total 20.02 last 88.25) - open/close 2582 (time 1783950145.52 total 30.03 last 80.31) - open/close 3366 (time 1783950155.54 total 40.04 last 78.29) - open/close 3895 (time 1783950165.55 total 50.06 last 52.84) - open/close 4501 (time 1783950175.55 total 60.06 last 60.56) - open/close 5150 (time 1783950185.55 total 70.06 last 64.89) - open/close 5602 (time 1783950195.57 total 80.07 last 45.14) - open/close 6032 (time 1783950205.57 total 90.08 last 42.99) - open/close 6622 (time 1783950215.59 total 100.09 last 58.91) - open/close 7221 (time 1783950225.59 total 110.10 last 59.88) - open/close 8119 (time 1783950235.60 total 120.11 last 89.68) - open/close 9147 (time 1783950245.61 total 130.11 last 102.77) total: 10000 open/close in 138.89 seconds: 72.00 ops/second - unlinked 0 (time 1783950259 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 589 (time 1783950342.70 total 10.01 last 58.86) - open/close 1352 (time 1783950352.71 total 20.02 last 76.19) - open/close 2227 (time 1783950362.71 total 30.02 last 87.49) - open/close 3069 (time 1783950372.72 total 40.03 last 84.13) - open/close 3816 (time 1783950382.73 total 50.03 last 74.68) - open/close 4773 (time 1783950392.73 total 60.04 last 95.66) - open/close 5603 (time 1783950402.73 total 70.04 last 82.98) - open/close 6521 (time 1783950412.75 total 80.06 last 91.61) - open/close 7430 (time 1783950422.76 total 90.06 last 90.86) - open/close 8524 (time 1783950432.76 total 100.07 last 109.39) - open/close 9513 (time 1783950442.76 total 110.07 last 98.85) total: 10000 open/close in 114.50 seconds: 87.33 ops/second - unlinked 0 (time 1783950451 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 852 (time 1783950528.77 total 10.01 last 85.11) - open/close 1968 (time 1783950538.78 total 20.01 last 111.58) - open/close 3100 (time 1783950548.78 total 30.02 last 113.10) - open/close 4090 (time 1783950558.79 total 40.03 last 98.96) - open/close 4980 (time 1783950568.80 total 50.04 last 88.91) - open/close 6071 (time 1783950578.80 total 60.04 last 109.04) - open/close 7091 (time 1783950588.81 total 70.05 last 101.96) - open/close 8069 (time 1783950598.81 total 80.05 last 97.75) - open/close 9146 (time 1783950608.82 total 90.06 last 107.64) total: 10000 open/close in 98.38 seconds: 101.65 ops/second - unlinked 0 (time 1783950621 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 970 (time 1783950696.86 total 10.00 last 96.98) - open/close 1753 (time 1783950706.86 total 20.00 last 78.29) - open/close 2669 (time 1783950716.86 total 30.01 last 91.55) - open/close 3495 (time 1783950726.87 total 40.02 last 82.50) - open/close 4536 (time 1783950736.88 total 50.03 last 104.05) - open/close 5613 (time 1783950746.89 total 60.03 last 107.61) - open/close 6548 (time 1783950756.89 total 70.04 last 93.45) - open/close 7357 (time 1783950766.91 total 80.06 last 80.75) - open/close 8058 (time 1783950776.92 total 90.07 last 70.03) - open/close 9052 (time 1783950786.92 total 100.07 last 99.40) total: 10000 open/close in 109.49 seconds: 91.33 ops/second - unlinked 0 (time 1783950800 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 921 (time 1783950875.71 total 10.03 last 91.86) - open/close 1834 (time 1783950885.72 total 20.03 last 91.24) - open/close 2595 (time 1783950895.72 total 30.03 last 76.09) - open/close 3700 (time 1783950905.72 total 40.03 last 110.48) - open/close 4756 (time 1783950915.72 total 50.04 last 105.58) - open/close 5753 (time 1783950925.73 total 60.05 last 99.60) - open/close 6823 (time 1783950935.74 total 70.05 last 106.97) - open/close 7853 (time 1783950945.75 total 80.06 last 102.89) - open/close 8969 (time 1783950955.75 total 90.06 last 111.58) total: 10000 open/close in 99.96 seconds: 100.04 ops/second - unlinked 0 (time 1783950969 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg243-server unloading modules via unload_modules_local on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: modules unloaded. pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2