== conf-sanity test 69: replace an OST with the same index ========================================================== 03:06:59 (1784185619) start mds service on oleg157-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing load_modules_local oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg157-server: Loading modules from /home/green/git/lustre-release/lustre oleg157-server: detected 4 online CPUs by sysfs oleg157-server: MODOPTS_LIBCFS= oleg157-server: Force libcfs to create 2 CPU partitions oleg157-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg157-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg157-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg157-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 779 (time 1784185720.19 total 10.00 last 77.86) - open/close 1427 (time 1784185730.19 total 20.00 last 64.80) - open/close 2195 (time 1784185740.19 total 30.01 last 76.77) - open/close 3095 (time 1784185750.19 total 40.01 last 89.98) - open/close 4164 (time 1784185760.20 total 50.01 last 106.86) - open/close 5227 (time 1784185770.20 total 60.02 last 106.21) - open/close 6272 (time 1784185780.21 total 70.03 last 104.46) - open/close 7153 (time 1784185790.22 total 80.04 last 87.98) - open/close 7999 (time 1784185800.22 total 90.04 last 84.60) - open/close 8900 (time 1784185810.23 total 100.05 last 90.04) - open/close 9847 (time 1784185820.23 total 110.05 last 94.67) total: 10000 open/close in 112.66 seconds: 88.76 ops/second - unlinked 0 (time 1784185827 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1042 (time 1784185901.81 total 10.01 last 104.12) - open/close 2080 (time 1784185911.82 total 20.01 last 103.77) - open/close 2734 (time 1784185921.83 total 30.02 last 65.35) - open/close 3704 (time 1784185931.83 total 40.02 last 96.96) - open/close 4546 (time 1784185941.84 total 50.03 last 84.15) - open/close 5508 (time 1784185951.84 total 60.03 last 96.18) - open/close 6485 (time 1784185961.84 total 70.03 last 97.69) - open/close 7549 (time 1784185971.85 total 80.04 last 106.30) - open/close 8456 (time 1784185981.85 total 90.05 last 90.67) - open/close 9558 (time 1784185991.86 total 100.05 last 110.11) total: 10000 open/close in 103.79 seconds: 96.35 ops/second - unlinked 0 (time 1784185999 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 993 (time 1784186082.56 total 10.01 last 99.20) - open/close 2089 (time 1784186092.57 total 20.02 last 109.50) - open/close 3148 (time 1784186102.57 total 30.02 last 105.86) - open/close 4109 (time 1784186112.58 total 40.03 last 96.06) - open/close 4792 (time 1784186122.59 total 50.04 last 68.24) - open/close 5452 (time 1784186132.60 total 60.05 last 65.94) - open/close 6398 (time 1784186142.60 total 70.05 last 94.53) - open/close 7503 (time 1784186152.61 total 80.05 last 110.48) - open/close 8625 (time 1784186162.61 total 90.06 last 112.12) - open/close 9301 (time 1784186172.62 total 100.06 last 67.58) total: 10000 open/close in 107.88 seconds: 92.70 ops/second - unlinked 0 (time 1784186184 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1037 (time 1784186263.73 total 10.00 last 103.68) - open/close 1972 (time 1784186273.74 total 20.01 last 93.42) - open/close 2871 (time 1784186283.74 total 30.02 last 89.84) - open/close 3710 (time 1784186293.75 total 40.02 last 83.86) - open/close 4460 (time 1784186303.79 total 50.07 last 74.66) - open/close 5381 (time 1784186313.80 total 60.08 last 92.00) - open/close 6462 (time 1784186323.81 total 70.09 last 107.99) - open/close 7583 (time 1784186333.82 total 80.09 last 112.05) - open/close 8702 (time 1784186343.82 total 90.10 last 111.86) - open/close 9613 (time 1784186353.83 total 100.10 last 91.08) total: 10000 open/close in 104.88 seconds: 95.35 ops/second - unlinked 0 (time 1784186364 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 990 (time 1784186447.91 total 10.01 last 98.92) - open/close 1999 (time 1784186457.91 total 20.01 last 100.85) - open/close 2956 (time 1784186467.93 total 30.03 last 95.56) - open/close 4019 (time 1784186477.93 total 40.03 last 106.25) - open/close 5077 (time 1784186487.94 total 50.04 last 105.71) - open/close 5925 (time 1784186497.95 total 60.06 last 84.68) - open/close 6955 (time 1784186507.96 total 70.06 last 102.94) - open/close 7945 (time 1784186517.97 total 80.07 last 98.95) - open/close 8951 (time 1784186527.97 total 90.07 last 100.54) - open/close 9994 (time 1784186537.98 total 100.08 last 104.26) total: 10000 open/close in 100.11 seconds: 99.89 ops/second - unlinked 0 (time 1784186542 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 856 (time 1784186618.94 total 10.00 last 85.59) - open/close 1802 (time 1784186628.95 total 20.01 last 94.55) - open/close 2747 (time 1784186638.95 total 30.01 last 94.44) - open/close 3685 (time 1784186648.96 total 40.02 last 93.74) - open/close 4548 (time 1784186658.96 total 50.03 last 86.25) - open/close 5473 (time 1784186668.97 total 60.03 last 92.47) - open/close 6423 (time 1784186678.97 total 70.04 last 94.93) - open/close 7248 (time 1784186688.98 total 80.04 last 82.45) - open/close 8142 (time 1784186698.99 total 90.05 last 89.37) - open/close 9208 (time 1784186708.99 total 100.05 last 106.60) total: 10000 open/close in 106.85 seconds: 93.59 ops/second - unlinked 0 (time 1784186719 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 795 (time 1784186812.59 total 10.01 last 79.39) - open/close 1501 (time 1784186822.59 total 20.02 last 70.56) - open/close 2352 (time 1784186832.60 total 30.02 last 85.08) - open/close 3138 (time 1784186842.61 total 40.03 last 78.53) - open/close 3850 (time 1784186852.62 total 50.04 last 71.13) - open/close 4582 (time 1784186862.62 total 60.05 last 73.15) - open/close 5603 (time 1784186872.63 total 70.05 last 102.05) - open/close 6618 (time 1784186882.63 total 80.06 last 101.46) - open/close 7656 (time 1784186892.63 total 90.06 last 103.78) - open/close 8574 (time 1784186902.65 total 100.07 last 91.69) - open/close 9560 (time 1784186912.65 total 110.08 last 98.53) total: 10000 open/close in 114.04 seconds: 87.69 ops/second - unlinked 0 (time 1784186921 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 919 (time 1784187000.57 total 10.01 last 91.84) - open/close 2002 (time 1784187010.58 total 20.01 last 108.26) - open/close 3008 (time 1784187020.58 total 30.01 last 100.58) - open/close 3928 (time 1784187030.59 total 40.02 last 91.94) - open/close 4992 (time 1784187040.60 total 50.03 last 106.26) - open/close 5972 (time 1784187050.61 total 60.04 last 97.91) - open/close 6939 (time 1784187060.62 total 70.05 last 96.61) - open/close 7779 (time 1784187070.62 total 80.05 last 83.97) - open/close 8958 (time 1784187080.62 total 90.06 last 117.89) total: 10000 open/close in 99.89 seconds: 100.11 ops/second - unlinked 0 (time 1784187094 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 848 (time 1784187169.02 total 10.01 last 84.75) - open/close 1877 (time 1784187179.03 total 20.01 last 102.86) - open/close 2875 (time 1784187189.03 total 30.01 last 99.76) - open/close 3916 (time 1784187199.04 total 40.02 last 104.00) - open/close 4833 (time 1784187209.04 total 50.03 last 91.69) - open/close 5660 (time 1784187219.05 total 60.03 last 82.63) - open/close 6628 (time 1784187229.06 total 70.04 last 96.71) - open/close 7634 (time 1784187239.07 total 80.05 last 100.54) - open/close 8595 (time 1784187249.08 total 90.06 last 96.00) - open/close 9464 (time 1784187259.08 total 100.06 last 86.86) total: 10000 open/close in 105.45 seconds: 94.84 ops/second - unlinked 0 (time 1784187268 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1016 (time 1784187351.89 total 10.00 last 101.59) - open/close 2053 (time 1784187361.89 total 20.00 last 103.70) - open/close 2980 (time 1784187371.90 total 30.01 last 92.60) - open/close 3887 (time 1784187381.91 total 40.02 last 90.64) - open/close 4586 (time 1784187391.92 total 50.03 last 69.81) - open/close 5554 (time 1784187401.93 total 60.04 last 96.74) - open/close 6579 (time 1784187411.93 total 70.04 last 102.47) - open/close 7486 (time 1784187421.94 total 80.05 last 90.59) - open/close 8352 (time 1784187431.95 total 90.06 last 86.54) - open/close 9336 (time 1784187441.95 total 100.06 last 98.37) total: 10000 open/close in 105.75 seconds: 94.57 ops/second - unlinked 0 (time 1784187451 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg157-server: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg157-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg157-server: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg157-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.157@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.157@tcp sys.timeout=20 start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg157-server: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg157-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg157-server: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg157-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg157-server unloading modules via unload_modules_local on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing unload_modules_local oleg157-server: modules unloaded. pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2