== conf-sanity test 69: replace an OST with the same index ========================================================== 07:30:31 (1786015831) start mds service on oleg335-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg335-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg335-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 519 (time 1786015968.74 total 10.01 last 51.83) - open/close 976 (time 1786015978.77 total 20.04 last 45.56) - open/close 1536 (time 1786015988.79 total 30.06 last 55.92) - open/close 1998 (time 1786015998.80 total 40.07 last 46.15) - open/close 2499 (time 1786016008.80 total 50.07 last 50.10) - open/close 3039 (time 1786016018.80 total 60.07 last 53.97) - open/close 3613 (time 1786016028.81 total 70.08 last 57.35) - open/close 4162 (time 1786016038.82 total 80.09 last 54.87) - open/close 4684 (time 1786016048.83 total 90.11 last 52.11) - open/close 5089 (time 1786016058.83 total 100.11 last 40.50) - open/close 5633 (time 1786016068.85 total 110.13 last 54.29) - open/close 6186 (time 1786016078.87 total 120.14 last 55.22) - open/close 6765 (time 1786016088.89 total 130.16 last 57.79) - open/close 7325 (time 1786016098.91 total 140.18 last 55.88) - open/close 7881 (time 1786016108.94 total 150.21 last 55.42) - open/close 8272 (time 1786016118.95 total 160.22 last 39.07) - open/close 8705 (time 1786016128.98 total 170.25 last 43.18) - open/close 9266 (time 1786016138.99 total 180.26 last 56.03) - open/close 9772 (time 1786016149.00 total 190.27 last 50.57) total: 10000 open/close in 194.75 seconds: 51.35 ops/second - unlinked 0 (time 1786016158 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 439 (time 1786016274.97 total 10.01 last 43.86) - open/close 863 (time 1786016284.97 total 20.01 last 42.39) - open/close 1364 (time 1786016295.00 total 30.04 last 49.95) - open/close 1866 (time 1786016305.00 total 40.04 last 50.19) - open/close 2399 (time 1786016315.03 total 50.07 last 53.18) - open/close 2905 (time 1786016325.05 total 60.09 last 50.49) - open/close 3581 (time 1786016335.05 total 70.09 last 67.57) - open/close 4172 (time 1786016345.07 total 80.11 last 59.02) - open/close 4843 (time 1786016355.08 total 90.12 last 66.99) - open/close 5437 (time 1786016365.09 total 100.13 last 59.35) - open/close 6070 (time 1786016375.11 total 110.15 last 63.16) - open/close 6614 (time 1786016385.12 total 120.16 last 54.37) - open/close 7138 (time 1786016395.12 total 130.16 last 52.39) - open/close 7685 (time 1786016405.13 total 140.17 last 54.64) - open/close 8221 (time 1786016415.13 total 150.17 last 53.58) - open/close 8836 (time 1786016425.15 total 160.19 last 61.42) - open/close 9553 (time 1786016435.15 total 170.20 last 71.63) total: 10000 open/close in 178.33 seconds: 56.07 ops/second - unlinked 0 (time 1786016448 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 609 (time 1786016558.60 total 10.03 last 60.74) - open/close 1125 (time 1786016568.61 total 20.04 last 51.55) - open/close 1635 (time 1786016578.62 total 30.04 last 50.96) - open/close 2157 (time 1786016588.64 total 40.06 last 52.09) - open/close 2685 (time 1786016598.68 total 50.10 last 52.60) - open/close 3264 (time 1786016608.68 total 60.11 last 57.87) - open/close 4110 (time 1786016618.70 total 70.12 last 84.46) - open/close 4649 (time 1786016628.70 total 80.13 last 53.88) - open/close 5353 (time 1786016638.74 total 90.16 last 70.17) - open/close 5895 (time 1786016648.75 total 100.18 last 54.11) - open/close 6449 (time 1786016658.76 total 110.19 last 55.35) - open/close 7084 (time 1786016668.76 total 120.19 last 63.48) - open/close 8023 (time 1786016678.77 total 130.19 last 93.88) - open/close 8490 (time 1786016688.79 total 140.22 last 46.57) - open/close 8982 (time 1786016698.80 total 150.22 last 49.20) - open/close 9503 (time 1786016708.80 total 160.23 last 52.06) total: 10000 open/close in 169.82 seconds: 58.88 ops/second - unlinked 0 (time 1786016724 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 559 (time 1786016830.85 total 10.01 last 55.87) - open/close 1007 (time 1786016840.86 total 20.02 last 44.75) - open/close 1560 (time 1786016850.87 total 30.03 last 55.24) - open/close 2041 (time 1786016860.87 total 40.03 last 48.10) - open/close 2568 (time 1786016870.88 total 50.04 last 52.62) - open/close 3055 (time 1786016880.89 total 60.05 last 48.66) - open/close 3512 (time 1786016890.92 total 70.08 last 45.56) - open/close 4039 (time 1786016900.93 total 80.09 last 52.66) - open/close 4553 (time 1786016910.97 total 90.13 last 51.18) - open/close 5104 (time 1786016920.97 total 100.13 last 55.10) - open/close 5597 (time 1786016930.99 total 110.15 last 49.23) - open/close 6109 (time 1786016941.00 total 120.16 last 51.12) - open/close 6641 (time 1786016951.00 total 130.16 last 53.19) - open/close 7076 (time 1786016961.01 total 140.17 last 43.46) - open/close 7724 (time 1786016971.04 total 150.20 last 64.62) - open/close 8266 (time 1786016981.05 total 160.21 last 54.17) - open/close 8758 (time 1786016991.05 total 170.21 last 49.19) - open/close 9319 (time 1786017001.05 total 180.21 last 56.07) - open/close 9825 (time 1786017011.06 total 190.22 last 50.58) total: 10000 open/close in 194.18 seconds: 51.50 ops/second - unlinked 0 (time 1786017023 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 546 (time 1786017144.25 total 10.00 last 54.58) - open/close 1435 (time 1786017154.26 total 20.01 last 88.83) - open/close 2031 (time 1786017164.28 total 30.03 last 59.47) - open/close 2620 (time 1786017174.29 total 40.04 last 58.87) - open/close 3150 (time 1786017184.31 total 50.06 last 52.90) - open/close 3645 (time 1786017194.31 total 60.06 last 49.49) - open/close 4155 (time 1786017204.31 total 70.06 last 51.00) - open/close 4873 (time 1786017214.33 total 80.08 last 71.66) - open/close 5479 (time 1786017224.34 total 90.09 last 60.52) - open/close 6130 (time 1786017234.34 total 100.10 last 65.08) - open/close 6662 (time 1786017244.36 total 110.11 last 53.13) - open/close 7316 (time 1786017254.36 total 120.11 last 65.38) - open/close 8044 (time 1786017264.36 total 130.12 last 72.75) - open/close 8763 (time 1786017274.38 total 140.13 last 71.82) - open/close 9365 (time 1786017284.39 total 150.15 last 60.09) - open/close 9974 (time 1786017294.40 total 160.15 last 60.87) total: 10000 open/close in 160.68 seconds: 62.24 ops/second - unlinked 0 (time 1786017301 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 536 (time 1786017418.28 total 10.01 last 53.57) - open/close 1180 (time 1786017428.29 total 20.01 last 64.35) - open/close 1848 (time 1786017438.30 total 30.03 last 66.71) - open/close 2489 (time 1786017448.32 total 40.04 last 64.01) - open/close 3019 (time 1786017458.35 total 50.07 last 52.85) - open/close 3598 (time 1786017468.35 total 60.07 last 57.90) - open/close 4180 (time 1786017478.37 total 70.09 last 58.07) - open/close 4835 (time 1786017488.38 total 80.10 last 65.45) - open/close 5383 (time 1786017498.39 total 90.11 last 54.74) - open/close 5915 (time 1786017508.40 total 100.12 last 53.15) - open/close 6438 (time 1786017518.41 total 110.13 last 52.22) - open/close 7106 (time 1786017528.42 total 120.14 last 66.74) - open/close 7704 (time 1786017538.42 total 130.14 last 59.79) - open/close 8440 (time 1786017548.43 total 140.15 last 73.53) - open/close 9006 (time 1786017558.45 total 150.18 last 56.48) - open/close 9510 (time 1786017568.46 total 160.18 last 50.37) total: 10000 open/close in 168.59 seconds: 59.32 ops/second - unlinked 0 (time 1786017582 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 483 (time 1786017694.21 total 10.02 last 48.21) - open/close 937 (time 1786017704.22 total 20.03 last 45.37) - open/close 1558 (time 1786017714.22 total 30.03 last 62.10) - open/close 2037 (time 1786017724.24 total 40.04 last 47.83) - open/close 2452 (time 1786017734.25 total 50.06 last 41.44) - open/close 2890 (time 1786017744.25 total 60.06 last 43.79) - open/close 3340 (time 1786017754.26 total 70.06 last 44.98) - open/close 3904 (time 1786017764.26 total 80.07 last 56.38) - open/close 4522 (time 1786017774.28 total 90.08 last 61.71) - open/close 5033 (time 1786017784.30 total 100.10 last 50.99) - open/close 5767 (time 1786017794.30 total 110.11 last 73.37) - open/close 6162 (time 1786017804.83 total 120.64 last 37.50) - open/close 6523 (time 1786017814.84 total 130.64 last 36.08) - open/close 7039 (time 1786017824.85 total 140.65 last 51.54) - open/close 7604 (time 1786017834.85 total 150.66 last 56.48) - open/close 8073 (time 1786017844.87 total 160.67 last 46.83) - open/close 8569 (time 1786017854.88 total 170.69 last 49.54) - open/close 9234 (time 1786017864.89 total 180.69 last 66.46) - open/close 9727 (time 1786017874.89 total 190.69 last 49.30) total: 10000 open/close in 194.38 seconds: 51.45 ops/second - unlinked 0 (time 1786017885 ; total 0 ; last 0) total: 10000 unlinks in 122 seconds: 81.967216 unlinks/second - open/close 596 (time 1786018023.76 total 10.01 last 59.54) - open/close 1289 (time 1786018033.76 total 20.01 last 69.30) - open/close 1874 (time 1786018043.76 total 30.02 last 58.47) - open/close 2561 (time 1786018053.79 total 40.04 last 68.54) - open/close 3363 (time 1786018063.81 total 50.06 last 80.05) - open/close 3891 (time 1786018073.82 total 60.07 last 52.74) - open/close 4348 (time 1786018083.86 total 70.11 last 45.51) - open/close 4805 (time 1786018093.87 total 80.12 last 45.66) - open/close 5353 (time 1786018103.87 total 90.13 last 54.76) - open/close 5903 (time 1786018113.88 total 100.13 last 54.98) - open/close 6398 (time 1786018123.89 total 110.14 last 49.46) - open/close 6945 (time 1786018133.89 total 120.15 last 54.65) - open/close 7496 (time 1786018143.90 total 130.16 last 55.04) - open/close 8049 (time 1786018153.91 total 140.16 last 55.27) - open/close 8547 (time 1786018163.94 total 150.19 last 49.67) - open/close 9219 (time 1786018173.94 total 160.19 last 67.18) - open/close 9822 (time 1786018183.95 total 170.20 last 60.22) total: 10000 open/close in 173.59 seconds: 57.61 ops/second - unlinked 0 (time 1786018192 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 479 (time 1786018300.97 total 10.00 last 47.90) - open/close 1301 (time 1786018310.98 total 20.00 last 82.18) - open/close 2461 (time 1786018320.98 total 30.00 last 116.00) - open/close 3234 (time 1786018330.98 total 40.01 last 77.24) - open/close 3943 (time 1786018341.01 total 50.04 last 70.73) - open/close 4566 (time 1786018351.01 total 60.04 last 62.29) - open/close 5367 (time 1786018361.01 total 70.04 last 80.09) - open/close 6275 (time 1786018371.04 total 80.07 last 90.53) - open/close 6828 (time 1786018381.04 total 90.07 last 55.29) - open/close 7392 (time 1786018391.07 total 100.10 last 56.22) - open/close 8011 (time 1786018401.08 total 110.11 last 61.85) - open/close 8538 (time 1786018411.09 total 120.12 last 52.66) - open/close 9013 (time 1786018421.10 total 130.13 last 47.46) - open/close 9520 (time 1786018431.12 total 140.15 last 50.58) total: 10000 open/close in 147.60 seconds: 67.75 ops/second - unlinked 0 (time 1786018443 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 539 (time 1786018552.12 total 10.02 last 53.79) - open/close 1030 (time 1786018562.14 total 20.04 last 49.00) - open/close 1555 (time 1786018572.15 total 30.05 last 52.47) - open/close 2185 (time 1786018582.15 total 40.05 last 62.98) - open/close 2890 (time 1786018592.18 total 50.08 last 70.29) - open/close 3481 (time 1786018602.19 total 60.09 last 59.03) - open/close 4066 (time 1786018612.20 total 70.09 last 58.49) - open/close 4720 (time 1786018622.20 total 80.10 last 65.39) - open/close 5248 (time 1786018632.21 total 90.10 last 52.76) - open/close 5726 (time 1786018642.21 total 100.11 last 47.79) - open/close 6325 (time 1786018652.21 total 110.11 last 59.87) - open/close 6911 (time 1786018662.22 total 120.12 last 58.57) - open/close 7665 (time 1786018672.22 total 130.12 last 75.39) - open/close 8311 (time 1786018682.22 total 140.12 last 64.58) - open/close 8830 (time 1786018692.26 total 150.16 last 51.69) - open/close 9400 (time 1786018702.26 total 160.16 last 56.98) - open/close 9766 (time 1786018712.27 total 170.17 last 36.57) total: 10000 open/close in 174.93 seconds: 57.17 ops/second - unlinked 0 (time 1786018724 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2