== conf-sanity test 69: replace an OST with the same index ========================================================== 01:42:23 (1775194943) start mds service on oleg410-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg410-server: detected 4 online CPUs by sysfs oleg410-server: MODOPTS_LIBCFS= oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg410-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg410-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg410-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 518 (time 1775195064.10 total 10.01 last 51.73) - open/close 1059 (time 1775195074.11 total 20.03 last 54.02) - open/close 1671 (time 1775195084.13 total 30.05 last 61.07) - open/close 2340 (time 1775195094.14 total 40.05 last 66.88) - open/close 2835 (time 1775195104.15 total 50.07 last 49.43) - open/close 3345 (time 1775195114.15 total 60.07 last 51.00) - open/close 3934 (time 1775195124.15 total 70.07 last 58.88) - open/close 4538 (time 1775195134.16 total 80.07 last 60.38) - open/close 5057 (time 1775195144.18 total 90.10 last 51.78) - open/close 5621 (time 1775195154.19 total 100.11 last 56.33) - open/close 6164 (time 1775195164.21 total 110.13 last 54.21) - open/close 6778 (time 1775195174.21 total 120.13 last 61.40) - open/close 7404 (time 1775195184.21 total 130.13 last 62.58) - open/close 8014 (time 1775195194.22 total 140.14 last 60.97) - open/close 8584 (time 1775195204.23 total 150.15 last 56.93) - open/close 9084 (time 1775195214.23 total 160.15 last 49.99) - open/close 9561 (time 1775195224.24 total 170.15 last 47.67) total: 10000 open/close in 178.17 seconds: 56.13 ops/second - unlinked 0 (time 1775195237 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 526 (time 1775195358.42 total 10.00 last 52.59) - open/close 1038 (time 1775195368.43 total 20.02 last 51.11) - open/close 1595 (time 1775195378.45 total 30.04 last 55.61) - open/close 2210 (time 1775195388.45 total 40.04 last 61.50) - open/close 2767 (time 1775195398.48 total 50.06 last 55.55) - open/close 3374 (time 1775195408.49 total 60.08 last 60.62) - open/close 3895 (time 1775195418.50 total 70.09 last 52.04) - open/close 4361 (time 1775195428.51 total 80.10 last 46.55) - open/close 4841 (time 1775195438.54 total 90.13 last 47.86) - open/close 5349 (time 1775195448.55 total 100.13 last 50.77) - open/close 6017 (time 1775195458.56 total 110.15 last 66.71) - open/close 6584 (time 1775195468.57 total 120.15 last 56.68) - open/close 7143 (time 1775195478.57 total 130.16 last 55.87) - open/close 7741 (time 1775195488.58 total 140.16 last 59.76) - open/close 8333 (time 1775195498.60 total 150.19 last 59.04) - open/close 8928 (time 1775195508.62 total 160.20 last 59.42) - open/close 9475 (time 1775195518.63 total 170.22 last 54.61) total: 10000 open/close in 179.45 seconds: 55.73 ops/second - unlinked 0 (time 1775195533 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 566 (time 1775195645.14 total 10.00 last 56.58) - open/close 1098 (time 1775195655.14 total 20.01 last 53.16) - open/close 1663 (time 1775195665.16 total 30.02 last 56.43) - open/close 2226 (time 1775195675.16 total 40.03 last 56.27) - open/close 2870 (time 1775195685.19 total 50.06 last 64.20) - open/close 3431 (time 1775195695.19 total 60.06 last 56.09) - open/close 4057 (time 1775195705.20 total 70.07 last 62.55) - open/close 4693 (time 1775195715.23 total 80.09 last 63.44) - open/close 5270 (time 1775195725.23 total 90.10 last 57.68) - open/close 5821 (time 1775195735.23 total 100.10 last 55.08) - open/close 6346 (time 1775195745.24 total 110.11 last 52.45) - open/close 6937 (time 1775195755.24 total 120.11 last 59.09) - open/close 7660 (time 1775195765.26 total 130.13 last 72.18) - open/close 8206 (time 1775195775.26 total 140.13 last 54.58) - open/close 8812 (time 1775195785.26 total 150.13 last 60.59) - open/close 9409 (time 1775195795.28 total 160.15 last 59.61) - open/close 9992 (time 1775195805.28 total 170.15 last 58.29) total: 10000 open/close in 170.33 seconds: 58.71 ops/second - unlinked 0 (time 1775195810 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 670 (time 1775195919.06 total 10.01 last 66.94) - open/close 1266 (time 1775195929.06 total 20.01 last 59.60) - open/close 1883 (time 1775195939.07 total 30.02 last 61.60) - open/close 2494 (time 1775195949.09 total 40.04 last 60.98) - open/close 3222 (time 1775195959.11 total 50.06 last 72.71) - open/close 3925 (time 1775195969.11 total 60.06 last 70.27) - open/close 4557 (time 1775195979.12 total 70.07 last 63.12) - open/close 5238 (time 1775195989.13 total 80.08 last 68.07) - open/close 5868 (time 1775195999.13 total 90.08 last 62.96) - open/close 6476 (time 1775196009.14 total 100.09 last 60.74) - open/close 7088 (time 1775196019.16 total 110.11 last 61.11) - open/close 7744 (time 1775196029.16 total 120.11 last 65.58) - open/close 8391 (time 1775196039.17 total 130.12 last 64.66) - open/close 9002 (time 1775196049.18 total 140.13 last 61.04) - open/close 9668 (time 1775196059.19 total 150.14 last 66.51) total: 10000 open/close in 155.25 seconds: 64.41 ops/second - unlinked 0 (time 1775196068 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 647 (time 1775196175.99 total 10.02 last 64.55) - open/close 1297 (time 1775196186.01 total 20.04 last 64.90) - open/close 1925 (time 1775196196.02 total 30.05 last 62.76) - open/close 2494 (time 1775196206.02 total 40.05 last 56.86) - open/close 3145 (time 1775196216.03 total 50.06 last 65.04) - open/close 3812 (time 1775196226.04 total 60.07 last 66.66) - open/close 4513 (time 1775196236.04 total 70.07 last 70.07) - open/close 5142 (time 1775196246.05 total 80.08 last 62.85) - open/close 5755 (time 1775196256.06 total 90.08 last 61.27) - open/close 6400 (time 1775196266.07 total 100.10 last 64.38) - open/close 7080 (time 1775196276.09 total 110.11 last 67.92) - open/close 7640 (time 1775196286.09 total 120.12 last 55.98) - open/close 8272 (time 1775196296.09 total 130.12 last 63.19) - open/close 8889 (time 1775196306.09 total 140.12 last 61.69) - open/close 9520 (time 1775196316.10 total 150.13 last 63.03) total: 10000 open/close in 158.24 seconds: 63.20 ops/second - unlinked 0 (time 1775196329 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 461 (time 1775196436.24 total 10.01 last 46.04) - open/close 963 (time 1775196446.25 total 20.03 last 50.12) - open/close 1525 (time 1775196456.26 total 30.03 last 56.18) - open/close 2082 (time 1775196466.27 total 40.04 last 55.66) - open/close 2684 (time 1775196476.29 total 50.06 last 60.07) - open/close 3436 (time 1775196486.29 total 60.07 last 75.16) - open/close 3987 (time 1775196496.29 total 70.07 last 55.09) - open/close 4565 (time 1775196506.33 total 80.10 last 57.60) - open/close 5032 (time 1775196516.34 total 90.11 last 46.67) - open/close 5777 (time 1775196526.35 total 100.12 last 74.43) - open/close 6268 (time 1775196536.35 total 110.13 last 49.07) - open/close 6682 (time 1775196546.37 total 120.14 last 41.34) - open/close 7051 (time 1775196556.40 total 130.17 last 36.78) - open/close 7460 (time 1775196566.41 total 140.19 last 40.84) - open/close 7869 (time 1775196576.43 total 150.21 last 40.82) - open/close 8330 (time 1775196586.44 total 160.21 last 46.06) - open/close 8862 (time 1775196596.45 total 170.22 last 53.17) - open/close 9502 (time 1775196606.45 total 180.23 last 63.95) total: 10000 open/close in 187.53 seconds: 53.33 ops/second - unlinked 0 (time 1775196618 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 609 (time 1775196726.07 total 10.04 last 60.67) - open/close 1230 (time 1775196736.07 total 20.04 last 62.09) - open/close 1927 (time 1775196746.07 total 30.05 last 69.66) - open/close 2577 (time 1775196756.09 total 40.06 last 64.90) - open/close 3221 (time 1775196766.10 total 50.07 last 64.34) - open/close 3871 (time 1775196776.11 total 60.08 last 64.95) - open/close 4499 (time 1775196786.11 total 70.08 last 62.76) - open/close 5143 (time 1775196796.13 total 80.10 last 64.28) - open/close 5809 (time 1775196806.14 total 90.11 last 66.53) - open/close 6472 (time 1775196816.15 total 100.12 last 66.24) - open/close 7128 (time 1775196826.17 total 110.14 last 65.51) - open/close 7757 (time 1775196836.17 total 120.15 last 62.85) - open/close 8379 (time 1775196846.18 total 130.15 last 62.16) - open/close 9039 (time 1775196856.19 total 140.16 last 65.95) - open/close 9692 (time 1775196866.21 total 150.18 last 65.17) total: 10000 open/close in 155.61 seconds: 64.26 ops/second - unlinked 0 (time 1775196877 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 572 (time 1775196987.44 total 10.00 last 57.18) - open/close 1189 (time 1775196997.48 total 20.05 last 61.43) - open/close 1770 (time 1775197007.48 total 30.05 last 58.09) - open/close 2319 (time 1775197017.49 total 40.06 last 54.85) - open/close 2900 (time 1775197027.51 total 50.07 last 58.01) - open/close 3470 (time 1775197037.51 total 60.08 last 56.96) - open/close 4118 (time 1775197047.52 total 70.09 last 64.75) - open/close 4652 (time 1775197057.53 total 80.10 last 53.35) - open/close 5308 (time 1775197067.53 total 90.10 last 65.58) - open/close 5980 (time 1775197077.55 total 100.11 last 67.12) - open/close 6529 (time 1775197087.56 total 110.12 last 54.85) - open/close 7178 (time 1775197097.56 total 120.12 last 64.88) - open/close 7796 (time 1775197107.57 total 130.13 last 61.74) - open/close 8379 (time 1775197117.58 total 140.14 last 58.24) - open/close 8986 (time 1775197127.59 total 150.15 last 60.64) - open/close 9600 (time 1775197137.61 total 160.18 last 61.26) total: 10000 open/close in 167.87 seconds: 59.57 ops/second - unlinked 0 (time 1775197150 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 645 (time 1775197262.85 total 10.00 last 64.47) - open/close 1187 (time 1775197272.87 total 20.02 last 54.13) - open/close 1859 (time 1775197282.87 total 30.02 last 67.16) - open/close 2489 (time 1775197292.87 total 40.02 last 63.00) - open/close 3108 (time 1775197302.88 total 50.03 last 61.87) - open/close 3808 (time 1775197312.88 total 60.03 last 69.96) - open/close 4442 (time 1775197322.89 total 70.04 last 63.36) - open/close 5077 (time 1775197332.90 total 80.06 last 63.40) - open/close 5671 (time 1775197342.91 total 90.06 last 59.39) - open/close 6248 (time 1775197352.91 total 100.06 last 57.70) - open/close 6806 (time 1775197362.93 total 110.09 last 55.65) - open/close 7411 (time 1775197372.96 total 120.12 last 60.31) - open/close 7958 (time 1775197382.98 total 130.13 last 54.64) - open/close 8536 (time 1775197392.98 total 140.13 last 57.76) - open/close 9087 (time 1775197402.99 total 150.15 last 55.03) - open/close 9768 (time 1775197412.99 total 160.15 last 68.10) total: 10000 open/close in 164.16 seconds: 60.92 ops/second - unlinked 0 (time 1775197422 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 598 (time 1775197524.65 total 10.00 last 59.79) - open/close 1136 (time 1775197534.67 total 20.01 last 53.73) - open/close 1626 (time 1775197544.68 total 30.03 last 48.94) - open/close 2207 (time 1775197554.69 total 40.03 last 58.06) - open/close 2795 (time 1775197564.70 total 50.05 last 58.70) - open/close 3393 (time 1775197574.72 total 60.07 last 59.68) - open/close 3957 (time 1775197584.73 total 70.08 last 56.35) - open/close 4586 (time 1775197594.73 total 80.08 last 62.89) - open/close 5111 (time 1775197604.74 total 90.09 last 52.46) - open/close 5698 (time 1775197614.74 total 100.09 last 58.70) - open/close 6206 (time 1775197624.75 total 110.10 last 50.73) - open/close 6789 (time 1775197634.76 total 120.11 last 58.27) - open/close 7285 (time 1775197644.77 total 130.12 last 49.54) - open/close 7850 (time 1775197654.77 total 140.12 last 56.49) - open/close 8463 (time 1775197664.78 total 150.13 last 61.23) - open/close 9146 (time 1775197674.80 total 160.14 last 68.23) - open/close 9793 (time 1775197684.81 total 170.16 last 64.61) total: 10000 open/close in 173.14 seconds: 57.76 ops/second - unlinked 0 (time 1775197693 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre On OST0, 17218 used inodes rc=0 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg410-server unloading modules via unload_modules_local on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: modules unloaded. pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2