== conf-sanity test 69: replace an OST with the same index ========================================================== 17:27:14 (1777498034) start mds service on oleg136-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg136-server: detected 4 online CPUs by sysfs oleg136-server: MODOPTS_LIBCFS= oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg136-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg136-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg136-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 482 (time 1777498180.76 total 10.01 last 48.16) - open/close 971 (time 1777498190.78 total 20.03 last 48.80) - open/close 1475 (time 1777498200.81 total 30.05 last 50.28) - open/close 1914 (time 1777498210.81 total 40.06 last 43.88) - open/close 2417 (time 1777498220.82 total 50.07 last 50.23) - open/close 2872 (time 1777498230.83 total 60.08 last 45.49) - open/close 3410 (time 1777498240.85 total 70.10 last 53.68) - open/close 3953 (time 1777498250.85 total 80.10 last 54.28) - open/close 4531 (time 1777498260.86 total 90.11 last 57.76) - open/close 5056 (time 1777498270.87 total 100.12 last 52.46) - open/close 5562 (time 1777498280.88 total 110.13 last 50.53) - open/close 6115 (time 1777498290.89 total 120.14 last 55.26) - open/close 6668 (time 1777498300.90 total 130.15 last 55.22) - open/close 7210 (time 1777498310.90 total 140.15 last 54.19) - open/close 7850 (time 1777498320.91 total 150.16 last 63.94) - open/close 8455 (time 1777498330.92 total 160.17 last 60.47) - open/close 9029 (time 1777498340.93 total 170.18 last 57.33) - open/close 9689 (time 1777498350.93 total 180.18 last 65.99) total: 10000 open/close in 185.72 seconds: 53.84 ops/second - unlinked 0 (time 1777498362 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 576 (time 1777498474.90 total 10.00 last 57.59) - open/close 1176 (time 1777498484.92 total 20.02 last 59.89) - open/close 1780 (time 1777498494.93 total 30.03 last 60.36) - open/close 2448 (time 1777498504.93 total 40.03 last 66.79) - open/close 3072 (time 1777498514.94 total 50.04 last 62.33) - open/close 3589 (time 1777498524.95 total 60.05 last 51.64) - open/close 4174 (time 1777498534.96 total 70.07 last 58.42) - open/close 4752 (time 1777498544.98 total 80.09 last 57.69) - open/close 5356 (time 1777498554.99 total 90.09 last 60.34) - open/close 5929 (time 1777498565.02 total 100.12 last 57.14) - open/close 6561 (time 1777498575.03 total 110.13 last 63.15) - open/close 7104 (time 1777498585.03 total 120.13 last 54.29) - open/close 7668 (time 1777498595.05 total 130.15 last 56.32) - open/close 8249 (time 1777498605.06 total 140.16 last 58.05) - open/close 8853 (time 1777498615.09 total 150.19 last 60.19) - open/close 9449 (time 1777498625.09 total 160.19 last 59.59) total: 10000 open/close in 168.97 seconds: 59.18 ops/second - unlinked 0 (time 1777498639 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 666 (time 1777498740.21 total 10.00 last 66.59) - open/close 1347 (time 1777498750.21 total 20.00 last 68.10) - open/close 2097 (time 1777498760.21 total 30.00 last 74.98) - open/close 2850 (time 1777498770.22 total 40.01 last 75.26) - open/close 3610 (time 1777498780.23 total 50.02 last 75.93) - open/close 4338 (time 1777498790.24 total 60.03 last 72.70) - open/close 5124 (time 1777498800.26 total 70.05 last 78.46) - open/close 5787 (time 1777498810.27 total 80.06 last 66.22) - open/close 6501 (time 1777498820.29 total 90.08 last 71.30) - open/close 7279 (time 1777498830.30 total 100.08 last 77.73) - open/close 7925 (time 1777498840.34 total 110.13 last 64.32) - open/close 8364 (time 1777498850.36 total 120.15 last 43.80) - open/close 8661 (time 1777498860.38 total 130.17 last 29.65) - open/close 8967 (time 1777498870.38 total 140.17 last 30.59) - open/close 9323 (time 1777498880.39 total 150.18 last 35.57) - open/close 9674 (time 1777498890.43 total 160.21 last 34.98) total: 10000 open/close in 168.33 seconds: 59.41 ops/second - unlinked 0 (time 1777498906 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 574 (time 1777499022.19 total 10.02 last 57.31) - open/close 1173 (time 1777499032.22 total 20.04 last 59.75) - open/close 1764 (time 1777499042.23 total 30.06 last 59.01) - open/close 2316 (time 1777499052.25 total 40.08 last 55.08) - open/close 2863 (time 1777499062.26 total 50.09 last 54.65) - open/close 3476 (time 1777499072.27 total 60.09 last 61.25) - open/close 3976 (time 1777499082.28 total 70.10 last 49.96) - open/close 4477 (time 1777499092.28 total 80.10 last 50.10) - open/close 4984 (time 1777499102.30 total 90.12 last 50.61) - open/close 5588 (time 1777499112.31 total 100.13 last 60.32) - open/close 6170 (time 1777499122.32 total 110.15 last 58.13) - open/close 6698 (time 1777499132.33 total 120.15 last 52.76) - open/close 7264 (time 1777499142.33 total 130.16 last 56.57) - open/close 7853 (time 1777499152.38 total 140.21 last 58.62) - open/close 8508 (time 1777499162.40 total 150.22 last 65.41) - open/close 9154 (time 1777499172.40 total 160.23 last 64.54) - open/close 9841 (time 1777499182.42 total 170.24 last 68.61) total: 10000 open/close in 172.59 seconds: 57.94 ops/second - unlinked 0 (time 1777499190 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 613 (time 1777499308.02 total 10.00 last 61.27) - open/close 1272 (time 1777499318.02 total 20.01 last 65.87) - open/close 1858 (time 1777499328.03 total 30.02 last 58.55) - open/close 2459 (time 1777499338.05 total 40.03 last 60.00) - open/close 3024 (time 1777499348.06 total 50.04 last 56.45) - open/close 3534 (time 1777499358.06 total 60.05 last 50.97) - open/close 4148 (time 1777499368.07 total 70.06 last 61.33) - open/close 4766 (time 1777499378.08 total 80.07 last 61.75) - open/close 5349 (time 1777499388.08 total 90.07 last 58.30) - open/close 5897 (time 1777499398.09 total 100.07 last 54.78) - open/close 6518 (time 1777499408.10 total 110.08 last 62.04) - open/close 7116 (time 1777499418.11 total 120.09 last 59.73) - open/close 7638 (time 1777499428.11 total 130.10 last 52.17) - open/close 8215 (time 1777499438.13 total 140.12 last 57.60) - open/close 8794 (time 1777499448.13 total 150.12 last 57.89) - open/close 9433 (time 1777499458.14 total 160.13 last 63.86) - open/close 9933 (time 1777499468.14 total 170.13 last 49.99) total: 10000 open/close in 171.01 seconds: 58.48 ops/second - unlinked 0 (time 1777499474 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 531 (time 1777499592.19 total 10.02 last 52.99) - open/close 1095 (time 1777499602.19 total 20.03 last 56.38) - open/close 1744 (time 1777499612.21 total 30.04 last 64.79) - open/close 2374 (time 1777499622.22 total 40.05 last 62.95) - open/close 2914 (time 1777499632.27 total 50.11 last 53.69) - open/close 3458 (time 1777499642.30 total 60.13 last 54.28) - open/close 3988 (time 1777499652.31 total 70.14 last 52.95) - open/close 4436 (time 1777499662.31 total 80.14 last 44.78) - open/close 5054 (time 1777499672.31 total 90.14 last 61.80) - open/close 5634 (time 1777499682.32 total 100.16 last 57.93) - open/close 6181 (time 1777499692.33 total 110.16 last 54.67) - open/close 6783 (time 1777499702.35 total 120.18 last 60.09) - open/close 7292 (time 1777499712.35 total 130.18 last 50.88) - open/close 7798 (time 1777499722.35 total 140.19 last 50.59) - open/close 8390 (time 1777499732.36 total 150.19 last 59.16) - open/close 8958 (time 1777499742.36 total 160.20 last 56.77) - open/close 9583 (time 1777499752.41 total 170.24 last 62.24) total: 10000 open/close in 177.59 seconds: 56.31 ops/second - unlinked 0 (time 1777499765 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 604 (time 1777499880.59 total 10.01 last 60.36) - open/close 1145 (time 1777499890.63 total 20.04 last 53.91) - open/close 1756 (time 1777499900.65 total 30.06 last 60.98) - open/close 2337 (time 1777499910.65 total 40.06 last 58.10) - open/close 3017 (time 1777499920.65 total 50.06 last 67.99) - open/close 3610 (time 1777499930.66 total 60.07 last 59.25) - open/close 4234 (time 1777499940.66 total 70.08 last 62.37) - open/close 4828 (time 1777499950.67 total 80.09 last 59.32) - open/close 5434 (time 1777499960.68 total 90.09 last 60.58) - open/close 6065 (time 1777499970.69 total 100.10 last 63.04) - open/close 6645 (time 1777499980.69 total 110.11 last 57.96) - open/close 7276 (time 1777499990.71 total 120.12 last 63.01) - open/close 7932 (time 1777500000.71 total 130.13 last 65.56) - open/close 8524 (time 1777500010.72 total 140.14 last 59.15) - open/close 9073 (time 1777500020.73 total 150.15 last 54.85) - open/close 9651 (time 1777500030.74 total 160.15 last 57.76) total: 10000 open/close in 165.21 seconds: 60.53 ops/second - unlinked 0 (time 1777500040 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 674 (time 1777500150.63 total 10.01 last 67.37) - open/close 1327 (time 1777500160.64 total 20.01 last 65.26) - open/close 2001 (time 1777500170.65 total 30.02 last 67.32) - open/close 2699 (time 1777500180.66 total 40.03 last 69.74) - open/close 3286 (time 1777500190.66 total 50.03 last 58.68) - open/close 3978 (time 1777500200.68 total 60.05 last 69.10) - open/close 4636 (time 1777500210.70 total 70.07 last 65.67) - open/close 5312 (time 1777500220.70 total 80.07 last 67.59) - open/close 5998 (time 1777500230.70 total 90.07 last 68.57) - open/close 6619 (time 1777500240.73 total 100.10 last 61.92) - open/close 7277 (time 1777500250.73 total 110.11 last 65.78) - open/close 7937 (time 1777500260.74 total 120.11 last 65.94) - open/close 8648 (time 1777500270.75 total 130.12 last 71.05) - open/close 9357 (time 1777500280.77 total 140.14 last 70.76) total: 10000 open/close in 149.79 seconds: 66.76 ops/second - unlinked 0 (time 1777500294 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 633 (time 1777500393.21 total 10.01 last 63.24) - open/close 1292 (time 1777500403.22 total 20.02 last 65.84) - open/close 1941 (time 1777500413.23 total 30.03 last 64.85) - open/close 2571 (time 1777500423.23 total 40.03 last 62.99) - open/close 3284 (time 1777500433.26 total 50.06 last 71.11) - open/close 4011 (time 1777500443.26 total 60.06 last 72.68) - open/close 4706 (time 1777500453.29 total 70.08 last 69.32) - open/close 5374 (time 1777500463.30 total 80.09 last 66.77) - open/close 6078 (time 1777500473.30 total 90.09 last 70.38) - open/close 6782 (time 1777500483.30 total 100.10 last 70.38) - open/close 7464 (time 1777500493.30 total 110.10 last 68.19) - open/close 8141 (time 1777500503.30 total 120.10 last 67.69) - open/close 8854 (time 1777500513.30 total 130.10 last 71.30) - open/close 9560 (time 1777500523.31 total 140.10 last 70.57) total: 10000 open/close in 146.26 seconds: 68.37 ops/second - unlinked 0 (time 1777500534 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 703 (time 1777500633.55 total 10.01 last 70.24) - open/close 1389 (time 1777500643.55 total 20.01 last 68.59) - open/close 2087 (time 1777500653.56 total 30.03 last 69.68) - open/close 2837 (time 1777500663.57 total 40.03 last 74.98) - open/close 3593 (time 1777500673.57 total 50.03 last 75.58) - open/close 4307 (time 1777500683.59 total 60.05 last 71.29) - open/close 5002 (time 1777500693.61 total 70.08 last 69.31) - open/close 5745 (time 1777500703.63 total 80.09 last 74.19) - open/close 6463 (time 1777500713.63 total 90.10 last 71.75) - open/close 7173 (time 1777500723.64 total 100.10 last 70.98) - open/close 7983 (time 1777500733.65 total 110.11 last 80.92) - open/close 8772 (time 1777500743.66 total 120.13 last 78.75) - open/close 9448 (time 1777500753.67 total 130.13 last 67.59) total: 10000 open/close in 137.78 seconds: 72.58 ops/second - unlinked 0 (time 1777500765 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg136-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 start ost1 service on oleg136-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server unloading modules via unload_modules_local on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local oleg136-server: modules unloaded. pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2