== conf-sanity test 69: replace an OST with the same index ========================================================== 10:59:13 (1779202753) start mds service on oleg148-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions oleg148-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg148-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg148-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg148-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg148-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 668 (time 1779202878.38 total 10.00 last 66.78) - open/close 1299 (time 1779202888.38 total 20.00 last 63.09) - open/close 2007 (time 1779202898.38 total 30.01 last 70.77) - open/close 2828 (time 1779202908.39 total 40.01 last 82.07) - open/close 3604 (time 1779202918.39 total 50.02 last 77.55) - open/close 4292 (time 1779202928.40 total 60.02 last 68.76) - open/close 4892 (time 1779202938.41 total 70.03 last 59.96) - open/close 5355 (time 1779202948.42 total 80.05 last 46.23) - open/close 5924 (time 1779202958.43 total 90.06 last 56.82) - open/close 6507 (time 1779202968.44 total 100.07 last 58.27) - open/close 7185 (time 1779202978.46 total 110.09 last 67.67) - open/close 7816 (time 1779202988.47 total 120.09 last 63.05) - open/close 8463 (time 1779202998.47 total 130.09 last 64.69) - open/close 9221 (time 1779203008.47 total 140.09 last 75.80) - open/close 9844 (time 1779203018.47 total 150.10 last 62.30) total: 10000 open/close in 152.60 seconds: 65.53 ops/second - unlinked 0 (time 1779203026 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 648 (time 1779203127.21 total 10.00 last 64.77) - open/close 1333 (time 1779203137.21 total 20.01 last 68.45) - open/close 1970 (time 1779203147.24 total 30.03 last 63.55) - open/close 2574 (time 1779203157.25 total 40.05 last 60.30) - open/close 3311 (time 1779203167.27 total 50.07 last 73.56) - open/close 3960 (time 1779203177.28 total 60.08 last 64.85) - open/close 4742 (time 1779203187.28 total 70.08 last 78.19) - open/close 5475 (time 1779203197.29 total 80.09 last 73.25) - open/close 6210 (time 1779203207.30 total 90.09 last 73.45) - open/close 7048 (time 1779203217.30 total 100.09 last 83.79) - open/close 7554 (time 1779203227.31 total 110.11 last 50.54) - open/close 8143 (time 1779203237.31 total 120.11 last 58.89) - open/close 8894 (time 1779203247.31 total 130.11 last 75.08) - open/close 9578 (time 1779203257.32 total 140.11 last 68.38) total: 10000 open/close in 146.12 seconds: 68.44 ops/second - unlinked 0 (time 1779203269 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 726 (time 1779203370.14 total 10.00 last 72.60) - open/close 1344 (time 1779203380.14 total 20.01 last 61.76) - open/close 2067 (time 1779203390.14 total 30.01 last 72.30) - open/close 2729 (time 1779203400.14 total 40.01 last 66.19) - open/close 3448 (time 1779203410.16 total 50.02 last 71.80) - open/close 4059 (time 1779203420.17 total 60.04 last 61.02) - open/close 4893 (time 1779203430.18 total 70.04 last 83.34) - open/close 5609 (time 1779203440.19 total 80.06 last 71.52) - open/close 6156 (time 1779203450.20 total 90.07 last 54.65) - open/close 6653 (time 1779203460.24 total 100.11 last 49.50) - open/close 7291 (time 1779203470.24 total 110.11 last 63.80) - open/close 7900 (time 1779203480.25 total 120.12 last 60.83) - open/close 8506 (time 1779203490.26 total 130.12 last 60.57) - open/close 9105 (time 1779203500.29 total 140.15 last 59.71) - open/close 9786 (time 1779203510.30 total 150.16 last 68.06) total: 10000 open/close in 153.86 seconds: 64.99 ops/second - unlinked 0 (time 1779203520 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 760 (time 1779203616.85 total 10.01 last 75.95) - open/close 1451 (time 1779203626.87 total 20.03 last 68.97) - open/close 2186 (time 1779203636.88 total 30.04 last 73.40) - open/close 2920 (time 1779203646.89 total 40.04 last 73.37) - open/close 3580 (time 1779203656.89 total 50.04 last 65.99) - open/close 4252 (time 1779203666.90 total 60.06 last 67.11) - open/close 4912 (time 1779203676.93 total 70.09 last 65.82) - open/close 5602 (time 1779203686.93 total 80.09 last 68.98) - open/close 6368 (time 1779203696.94 total 90.10 last 76.49) - open/close 7073 (time 1779203706.96 total 100.11 last 70.42) - open/close 7632 (time 1779203716.97 total 110.13 last 55.82) - open/close 8214 (time 1779203726.97 total 120.13 last 58.20) - open/close 8866 (time 1779203736.99 total 130.15 last 65.07) - open/close 9475 (time 1779203747.05 total 140.21 last 60.54) total: 10000 open/close in 149.17 seconds: 67.04 ops/second - unlinked 0 (time 1779203761 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 398 (time 1779203870.76 total 10.02 last 39.72) - open/close 816 (time 1779203880.76 total 20.02 last 41.80) - open/close 1344 (time 1779203890.78 total 30.03 last 52.74) - open/close 1911 (time 1779203900.78 total 40.04 last 56.66) - open/close 2404 (time 1779203910.79 total 50.04 last 49.27) - open/close 3000 (time 1779203920.79 total 60.05 last 59.59) - open/close 3559 (time 1779203930.80 total 70.05 last 55.85) - open/close 4383 (time 1779203940.80 total 80.06 last 82.37) - open/close 5049 (time 1779203950.81 total 90.06 last 66.57) - open/close 5768 (time 1779203960.81 total 100.07 last 71.85) - open/close 6559 (time 1779203970.84 total 110.09 last 78.93) - open/close 7267 (time 1779203980.84 total 120.10 last 70.75) - open/close 7950 (time 1779203990.85 total 130.10 last 68.27) - open/close 8714 (time 1779204000.85 total 140.11 last 76.38) - open/close 9410 (time 1779204010.85 total 150.11 last 69.59) total: 10000 open/close in 158.43 seconds: 63.12 ops/second - unlinked 0 (time 1779204024 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 553 (time 1779204129.64 total 10.01 last 55.22) - open/close 1036 (time 1779204139.64 total 20.02 last 48.29) - open/close 1698 (time 1779204149.64 total 30.02 last 66.16) - open/close 2412 (time 1779204159.65 total 40.03 last 71.35) - open/close 3131 (time 1779204169.66 total 50.04 last 71.81) - open/close 3839 (time 1779204179.67 total 60.05 last 70.75) - open/close 4529 (time 1779204189.68 total 70.05 last 68.97) - open/close 5156 (time 1779204199.68 total 80.06 last 62.66) - open/close 5794 (time 1779204209.69 total 90.07 last 63.74) - open/close 6468 (time 1779204219.70 total 100.07 last 67.38) - open/close 7175 (time 1779204229.70 total 110.08 last 70.66) - open/close 7651 (time 1779204239.70 total 120.08 last 47.60) - open/close 8234 (time 1779204249.70 total 130.08 last 58.29) - open/close 8962 (time 1779204259.71 total 140.08 last 72.76) - open/close 9612 (time 1779204269.71 total 150.09 last 64.99) total: 10000 open/close in 154.91 seconds: 64.55 ops/second - unlinked 0 (time 1779204279 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 651 (time 1779204385.20 total 10.00 last 65.09) - open/close 1501 (time 1779204395.21 total 20.01 last 84.95) - open/close 2137 (time 1779204405.21 total 30.01 last 63.59) - open/close 2846 (time 1779204415.22 total 40.02 last 70.83) - open/close 3478 (time 1779204425.22 total 50.02 last 63.19) - open/close 4008 (time 1779204435.22 total 60.02 last 53.00) - open/close 4504 (time 1779204445.24 total 70.04 last 49.52) - open/close 4992 (time 1779204455.25 total 80.05 last 48.75) - open/close 5482 (time 1779204465.25 total 90.05 last 48.99) - open/close 6039 (time 1779204475.26 total 100.06 last 55.67) - open/close 6682 (time 1779204485.26 total 110.06 last 64.29) - open/close 7233 (time 1779204495.27 total 120.07 last 55.03) - open/close 7761 (time 1779204505.28 total 130.08 last 52.75) - open/close 8395 (time 1779204515.32 total 140.12 last 63.18) - open/close 8922 (time 1779204525.32 total 150.12 last 52.67) - open/close 9529 (time 1779204535.34 total 160.14 last 60.58) total: 10000 open/close in 167.58 seconds: 59.67 ops/second - unlinked 0 (time 1779204548 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 662 (time 1779204661.94 total 10.01 last 66.17) - open/close 1360 (time 1779204671.96 total 20.02 last 69.66) - open/close 1995 (time 1779204681.97 total 30.04 last 63.43) - open/close 2604 (time 1779204691.97 total 40.04 last 60.85) - open/close 3184 (time 1779204701.98 total 50.05 last 57.96) - open/close 3924 (time 1779204711.99 total 60.06 last 73.95) - open/close 4602 (time 1779204721.99 total 70.06 last 67.76) - open/close 5159 (time 1779204732.02 total 80.09 last 55.57) - open/close 5693 (time 1779204742.02 total 90.09 last 53.40) - open/close 6313 (time 1779204752.04 total 100.11 last 61.88) - open/close 6941 (time 1779204762.05 total 110.12 last 62.73) - open/close 7535 (time 1779204772.06 total 120.13 last 59.32) - open/close 8095 (time 1779204782.07 total 130.14 last 55.94) - open/close 8657 (time 1779204792.08 total 140.14 last 56.18) - open/close 9312 (time 1779204802.08 total 150.15 last 65.49) total: 10000 open/close in 159.87 seconds: 62.55 ops/second - unlinked 0 (time 1779204818 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 690 (time 1779204927.81 total 10.00 last 68.99) - open/close 1368 (time 1779204937.81 total 20.01 last 67.75) - open/close 2119 (time 1779204947.83 total 30.02 last 75.01) - open/close 2880 (time 1779204957.83 total 40.03 last 76.04) - open/close 3511 (time 1779204967.85 total 50.05 last 62.99) - open/close 4170 (time 1779204977.88 total 60.07 last 65.74) - open/close 4841 (time 1779204987.88 total 70.07 last 67.08) - open/close 5534 (time 1779204997.89 total 80.08 last 69.26) - open/close 6207 (time 1779205007.89 total 90.08 last 67.30) - open/close 6840 (time 1779205017.89 total 100.09 last 63.27) - open/close 7365 (time 1779205027.90 total 110.09 last 52.48) - open/close 7924 (time 1779205037.91 total 120.10 last 55.82) - open/close 8405 (time 1779205047.93 total 130.13 last 48.00) - open/close 8910 (time 1779205057.94 total 140.13 last 50.47) - open/close 9552 (time 1779205067.95 total 150.15 last 64.09) total: 10000 open/close in 157.36 seconds: 63.55 ops/second - unlinked 0 (time 1779205080 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 686 (time 1779205189.70 total 10.03 last 68.37) - open/close 1265 (time 1779205199.70 total 20.04 last 57.89) - open/close 1643 (time 1779205209.71 total 30.04 last 37.78) - open/close 2133 (time 1779205219.72 total 40.05 last 48.96) - open/close 2635 (time 1779205229.72 total 50.06 last 50.15) - open/close 3271 (time 1779205239.74 total 60.07 last 63.53) - open/close 3991 (time 1779205249.74 total 70.08 last 71.96) - open/close 4694 (time 1779205259.74 total 80.08 last 70.28) - open/close 5305 (time 1779205269.76 total 90.09 last 61.01) - open/close 5911 (time 1779205279.76 total 100.10 last 60.58) - open/close 6504 (time 1779205289.77 total 110.11 last 59.24) - open/close 7052 (time 1779205299.77 total 120.11 last 54.78) - open/close 7762 (time 1779205309.78 total 130.12 last 70.94) - open/close 8460 (time 1779205319.79 total 140.13 last 69.75) - open/close 9097 (time 1779205329.79 total 150.13 last 63.70) - open/close 9722 (time 1779205339.80 total 160.14 last 62.44) total: 10000 open/close in 165.90 seconds: 60.28 ops/second - unlinked 0 (time 1779205352 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.148@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg148-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg148-server: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg148-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg148-server: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg148-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.148@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.148@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.148@tcp sys.timeout=20 start ost1 service on oleg148-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg148-server: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg148-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg148-server: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg148-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg148-server unloading modules via unload_modules_local on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing unload_modules_local oleg148-server: modules unloaded. pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2