== conf-sanity test 69: replace an OST with the same index ========================================================== 23:24:55 (1786418695) start mds service on oleg659-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing load_modules_local oleg659-server: Loading modules from /home/green/git/lustre-release/lustre oleg659-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg659-server: detected 4 online CPUs by sysfs oleg659-server: MODOPTS_LIBCFS= oleg659-server: Force libcfs to create 2 CPU partitions oleg659-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg659-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg659-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg659-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg659-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-OST0000 oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg659-client.virtnet: -o user_xattr,flock 192.168.206.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 763 (time 1786418807.37 total 10.01 last 76.21) - open/close 1375 (time 1786418817.37 total 20.02 last 61.18) - open/close 2023 (time 1786418827.38 total 30.02 last 64.75) - open/close 2618 (time 1786418837.38 total 40.03 last 59.48) - open/close 3309 (time 1786418847.39 total 50.03 last 69.05) - open/close 4036 (time 1786418857.39 total 60.04 last 72.65) - open/close 4768 (time 1786418867.40 total 70.04 last 73.18) - open/close 5561 (time 1786418877.40 total 80.05 last 79.25) - open/close 6334 (time 1786418887.41 total 90.06 last 77.25) - open/close 7178 (time 1786418897.41 total 100.06 last 84.39) - open/close 8025 (time 1786418907.43 total 110.07 last 84.57) - open/close 8779 (time 1786418917.45 total 120.09 last 75.23) - open/close 9481 (time 1786418927.46 total 130.10 last 70.16) total: 10000 open/close in 139.02 seconds: 71.93 ops/second - unlinked 0 (time 1786418941 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 824 (time 1786419032.85 total 10.01 last 82.29) - open/close 1658 (time 1786419042.85 total 20.02 last 83.34) - open/close 2393 (time 1786419052.86 total 30.03 last 73.43) - open/close 3142 (time 1786419062.87 total 40.04 last 74.85) - open/close 3954 (time 1786419072.88 total 50.05 last 81.10) - open/close 4785 (time 1786419082.90 total 60.07 last 82.95) - open/close 5557 (time 1786419092.90 total 70.07 last 77.19) - open/close 6303 (time 1786419102.91 total 80.07 last 74.54) - open/close 7106 (time 1786419112.91 total 90.08 last 80.28) - open/close 7856 (time 1786419122.92 total 100.09 last 74.94) - open/close 8640 (time 1786419132.93 total 110.10 last 78.30) - open/close 9348 (time 1786419142.93 total 120.10 last 70.79) total: 10000 open/close in 129.87 seconds: 77.00 ops/second - unlinked 0 (time 1786419158 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 742 (time 1786419246.50 total 10.01 last 74.10) - open/close 1524 (time 1786419256.51 total 20.03 last 78.10) - open/close 2345 (time 1786419266.52 total 30.03 last 82.04) - open/close 3189 (time 1786419276.53 total 40.04 last 84.31) - open/close 3882 (time 1786419286.54 total 50.05 last 69.23) - open/close 4557 (time 1786419296.55 total 60.07 last 67.42) - open/close 5212 (time 1786419306.58 total 70.10 last 65.28) - open/close 6004 (time 1786419316.59 total 80.11 last 79.15) - open/close 6772 (time 1786419326.60 total 90.12 last 76.71) - open/close 7574 (time 1786419336.62 total 100.13 last 80.08) - open/close 8316 (time 1786419346.63 total 110.14 last 74.13) - open/close 9039 (time 1786419356.63 total 120.15 last 72.26) - open/close 9825 (time 1786419366.63 total 130.15 last 78.60) total: 10000 open/close in 132.48 seconds: 75.48 ops/second - unlinked 0 (time 1786419373 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 739 (time 1786419462.76 total 10.00 last 73.88) - open/close 1585 (time 1786419472.77 total 20.00 last 84.58) - open/close 2403 (time 1786419482.77 total 30.01 last 81.74) - open/close 3209 (time 1786419492.79 total 40.03 last 80.49) - open/close 4022 (time 1786419502.79 total 50.03 last 81.28) - open/close 4762 (time 1786419512.80 total 60.04 last 73.93) - open/close 5565 (time 1786419522.81 total 70.05 last 80.20) - open/close 6369 (time 1786419532.82 total 80.06 last 80.35) - open/close 7007 (time 1786419542.84 total 90.08 last 63.66) - open/close 7700 (time 1786419552.86 total 100.10 last 69.16) - open/close 8423 (time 1786419562.86 total 110.10 last 72.28) - open/close 9255 (time 1786419572.87 total 120.11 last 83.17) total: 10000 open/close in 129.53 seconds: 77.20 ops/second - unlinked 0 (time 1786419586 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 786 (time 1786419674.23 total 10.00 last 78.60) - open/close 1541 (time 1786419684.24 total 20.01 last 75.42) - open/close 2284 (time 1786419694.25 total 30.02 last 74.24) - open/close 3064 (time 1786419704.26 total 40.03 last 77.95) - open/close 3807 (time 1786419714.26 total 50.03 last 74.26) - open/close 4671 (time 1786419724.27 total 60.03 last 86.38) - open/close 5482 (time 1786419734.27 total 70.03 last 81.10) - open/close 6239 (time 1786419744.27 total 80.04 last 75.63) - open/close 7041 (time 1786419754.29 total 90.05 last 80.11) - open/close 7845 (time 1786419764.31 total 100.07 last 80.24) - open/close 8612 (time 1786419774.31 total 110.08 last 76.69) - open/close 9401 (time 1786419784.31 total 120.08 last 78.89) total: 10000 open/close in 128.09 seconds: 78.07 ops/second - unlinked 0 (time 1786419795 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 792 (time 1786419880.04 total 10.01 last 79.13) - open/close 1572 (time 1786419890.05 total 20.02 last 77.94) - open/close 2342 (time 1786419900.06 total 30.03 last 76.94) - open/close 3166 (time 1786419910.07 total 40.03 last 82.36) - open/close 3960 (time 1786419920.07 total 50.03 last 79.38) - open/close 4741 (time 1786419930.08 total 60.04 last 78.04) - open/close 5543 (time 1786419940.08 total 70.05 last 80.15) - open/close 6353 (time 1786419950.10 total 80.07 last 80.85) - open/close 7151 (time 1786419960.11 total 90.07 last 79.75) - open/close 7928 (time 1786419970.11 total 100.08 last 77.66) - open/close 8559 (time 1786419980.11 total 110.08 last 63.10) - open/close 9319 (time 1786419990.12 total 120.08 last 75.97) total: 10000 open/close in 129.04 seconds: 77.50 ops/second - unlinked 0 (time 1786420002 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 694 (time 1786420089.04 total 10.01 last 69.35) - open/close 1475 (time 1786420099.04 total 20.01 last 78.10) - open/close 2226 (time 1786420109.04 total 30.02 last 75.04) - open/close 2992 (time 1786420119.05 total 40.02 last 76.55) - open/close 3742 (time 1786420129.09 total 50.06 last 74.72) - open/close 4475 (time 1786420139.09 total 60.07 last 73.27) - open/close 5163 (time 1786420149.10 total 70.07 last 68.79) - open/close 5869 (time 1786420159.10 total 80.07 last 70.59) - open/close 6636 (time 1786420169.10 total 90.07 last 76.68) - open/close 7446 (time 1786420179.10 total 100.07 last 80.99) - open/close 8278 (time 1786420189.11 total 110.08 last 83.13) - open/close 9127 (time 1786420199.12 total 120.09 last 84.79) - open/close 9872 (time 1786420209.13 total 130.11 last 74.40) total: 10000 open/close in 131.54 seconds: 76.02 ops/second - unlinked 0 (time 1786420215 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 807 (time 1786420303.24 total 10.01 last 80.61) - open/close 1622 (time 1786420313.25 total 20.02 last 81.46) - open/close 2516 (time 1786420323.25 total 30.02 last 89.36) - open/close 3309 (time 1786420333.26 total 40.02 last 79.29) - open/close 4088 (time 1786420343.27 total 50.03 last 77.82) - open/close 4929 (time 1786420353.27 total 60.04 last 84.07) - open/close 5808 (time 1786420363.28 total 70.04 last 87.84) - open/close 6610 (time 1786420373.29 total 80.05 last 80.12) - open/close 7479 (time 1786420383.29 total 90.06 last 86.85) - open/close 8279 (time 1786420393.30 total 100.07 last 79.93) - open/close 9091 (time 1786420403.32 total 110.09 last 81.05) - open/close 9946 (time 1786420413.32 total 120.09 last 85.47) total: 10000 open/close in 120.81 seconds: 82.77 ops/second - unlinked 0 (time 1786420418 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 737 (time 1786420504.12 total 10.01 last 73.64) - open/close 1554 (time 1786420514.13 total 20.01 last 81.70) - open/close 2320 (time 1786420524.13 total 30.02 last 76.52) - open/close 3065 (time 1786420534.14 total 40.02 last 74.49) - open/close 3879 (time 1786420544.14 total 50.02 last 81.39) - open/close 4653 (time 1786420554.14 total 60.02 last 77.39) - open/close 5248 (time 1786420564.14 total 70.02 last 59.48) - open/close 5934 (time 1786420574.15 total 80.03 last 68.54) - open/close 6638 (time 1786420584.16 total 90.04 last 70.34) - open/close 7323 (time 1786420594.16 total 100.05 last 68.46) - open/close 8032 (time 1786420604.17 total 110.05 last 70.88) - open/close 8793 (time 1786420614.17 total 120.06 last 76.03) - open/close 9583 (time 1786420624.18 total 130.07 last 78.92) total: 10000 open/close in 135.57 seconds: 73.76 ops/second - unlinked 0 (time 1786420634 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 855 (time 1786420724.42 total 10.00 last 85.46) - open/close 1680 (time 1786420734.43 total 20.01 last 82.45) - open/close 2492 (time 1786420744.45 total 30.03 last 81.06) - open/close 3301 (time 1786420754.45 total 40.03 last 80.85) - open/close 4139 (time 1786420764.47 total 50.04 last 83.71) - open/close 4926 (time 1786420774.48 total 60.06 last 78.60) - open/close 5689 (time 1786420784.49 total 70.07 last 76.23) - open/close 6468 (time 1786420794.49 total 80.07 last 77.85) - open/close 7273 (time 1786420804.50 total 90.08 last 80.48) - open/close 8073 (time 1786420814.50 total 100.08 last 79.97) - open/close 8839 (time 1786420824.51 total 110.09 last 76.54) - open/close 9649 (time 1786420834.53 total 120.11 last 80.82) total: 10000 open/close in 123.97 seconds: 80.66 ops/second - unlinked 0 (time 1786420842 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg659-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg659-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg659-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.159@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg659-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-OST0000 oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg659-server: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg659-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg659-server: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg659-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg659-client.virtnet: -o user_xattr,flock 192.168.206.159@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg659-client: oleg659-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg659-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg659-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg659-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.159@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.159@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.159@tcp sys.timeout=20 start ost1 service on oleg659-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-OST0000 oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg659-server: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg659-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg659-server: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg659-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg659-client.virtnet: -o user_xattr,flock 192.168.206.159@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg659-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg659-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg659-server stop mds service on oleg659-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg659-server stop mds service on oleg659-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg659-server unloading modules via unload_modules_local on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing unload_modules_local oleg659-server: modules unloaded. pdsh@oleg659-client: oleg659-client: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-server: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-client: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-server: ssh exited with exit code 2