== conf-sanity test 69: replace an OST with the same index ========================================================== 08:06:00 (1787832360) start mds service on oleg426-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg426-server' oleg426-server: oleg426-server.virtnet: executing load_modules_local oleg426-server: Loading modules from /home/green/git/lustre-release/lustre oleg426-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg426-server: detected 4 online CPUs by sysfs oleg426-server: MODOPTS_LIBCFS= oleg426-server: Force libcfs to create 2 CPU partitions oleg426-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg426-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg426-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg426-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg426-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 776 (time 1787832477.52 total 10.01 last 77.53) - open/close 1510 (time 1787832487.53 total 20.01 last 73.36) - open/close 2421 (time 1787832497.54 total 30.03 last 90.97) - open/close 3206 (time 1787832507.54 total 40.03 last 78.48) - open/close 3999 (time 1787832517.55 total 50.03 last 79.27) - open/close 4826 (time 1787832527.55 total 60.04 last 82.66) - open/close 5618 (time 1787832537.56 total 70.04 last 79.18) - open/close 6405 (time 1787832547.56 total 80.04 last 78.69) - open/close 7122 (time 1787832557.56 total 90.05 last 71.65) - open/close 7770 (time 1787832567.57 total 100.05 last 64.78) - open/close 8442 (time 1787832577.59 total 110.08 last 67.05) - open/close 9185 (time 1787832587.60 total 120.08 last 74.24) - open/close 9840 (time 1787832597.61 total 130.10 last 65.40) total: 10000 open/close in 132.29 seconds: 75.59 ops/second - unlinked 0 (time 1787832604 ; total 1 ; last 1) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 763 (time 1787832699.35 total 10.00 last 76.28) - open/close 1465 (time 1787832709.36 total 20.02 last 70.10) - open/close 2113 (time 1787832719.37 total 30.03 last 64.74) - open/close 2916 (time 1787832729.38 total 40.03 last 80.25) - open/close 3729 (time 1787832739.38 total 50.04 last 81.26) - open/close 4573 (time 1787832749.39 total 60.05 last 84.30) - open/close 5339 (time 1787832759.40 total 70.05 last 76.55) - open/close 6196 (time 1787832769.40 total 80.06 last 85.69) - open/close 6985 (time 1787832779.41 total 90.06 last 78.85) - open/close 7849 (time 1787832789.42 total 100.07 last 86.33) - open/close 8768 (time 1787832799.42 total 110.08 last 91.83) - open/close 9607 (time 1787832809.43 total 120.08 last 83.87) total: 10000 open/close in 125.68 seconds: 79.57 ops/second - unlinked 0 (time 1787832820 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 723 (time 1787832910.23 total 10.00 last 72.28) - open/close 1450 (time 1787832920.24 total 20.01 last 72.65) - open/close 2200 (time 1787832930.27 total 30.04 last 74.79) - open/close 2944 (time 1787832940.29 total 40.06 last 74.23) - open/close 3703 (time 1787832950.30 total 50.07 last 75.81) - open/close 4543 (time 1787832960.31 total 60.08 last 83.97) - open/close 5402 (time 1787832970.31 total 70.08 last 85.89) - open/close 6151 (time 1787832980.31 total 80.08 last 74.87) - open/close 6993 (time 1787832990.32 total 90.09 last 84.11) - open/close 7738 (time 1787833000.33 total 100.10 last 74.45) - open/close 8522 (time 1787833010.34 total 110.11 last 78.31) - open/close 9215 (time 1787833020.34 total 120.11 last 69.30) total: 10000 open/close in 129.85 seconds: 77.01 ops/second - unlinked 0 (time 1787833034 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 670 (time 1787833123.35 total 10.01 last 66.91) - open/close 1497 (time 1787833133.36 total 20.02 last 82.63) - open/close 2390 (time 1787833143.36 total 30.03 last 89.26) - open/close 3286 (time 1787833153.37 total 40.03 last 89.55) - open/close 4091 (time 1787833163.37 total 50.03 last 80.49) - open/close 4890 (time 1787833173.37 total 60.03 last 79.89) - open/close 5687 (time 1787833183.37 total 70.04 last 79.67) - open/close 6568 (time 1787833193.38 total 80.04 last 88.04) - open/close 7350 (time 1787833203.38 total 90.05 last 78.17) - open/close 8187 (time 1787833213.39 total 100.05 last 83.64) - open/close 9002 (time 1787833223.39 total 110.05 last 81.50) - open/close 9787 (time 1787833233.41 total 120.07 last 78.37) total: 10000 open/close in 122.83 seconds: 81.41 ops/second - unlinked 0 (time 1787833240 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 848 (time 1787833321.29 total 10.03 last 84.59) - open/close 1597 (time 1787833331.30 total 20.03 last 74.88) - open/close 2329 (time 1787833341.34 total 30.07 last 72.89) - open/close 3010 (time 1787833351.36 total 40.09 last 67.93) - open/close 3702 (time 1787833361.41 total 50.15 last 68.84) - open/close 4452 (time 1787833371.44 total 60.17 last 74.79) - open/close 5204 (time 1787833381.45 total 70.18 last 75.17) - open/close 5944 (time 1787833391.45 total 80.18 last 73.97) - open/close 6713 (time 1787833401.46 total 90.19 last 76.85) - open/close 7444 (time 1787833411.46 total 100.19 last 73.07) - open/close 7997 (time 1787833421.48 total 110.21 last 55.19) - open/close 8744 (time 1787833431.49 total 120.22 last 74.62) - open/close 9516 (time 1787833441.49 total 130.22 last 77.20) total: 10000 open/close in 135.96 seconds: 73.55 ops/second - unlinked 0 (time 1787833451 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 847 (time 1787833533.93 total 10.00 last 84.69) - open/close 1679 (time 1787833543.93 total 20.00 last 83.17) - open/close 2450 (time 1787833553.94 total 30.01 last 77.04) - open/close 3167 (time 1787833563.96 total 40.03 last 71.59) - open/close 4020 (time 1787833573.96 total 50.04 last 85.24) - open/close 4883 (time 1787833583.96 total 60.04 last 86.30) - open/close 5707 (time 1787833593.97 total 70.04 last 82.34) - open/close 6546 (time 1787833603.98 total 80.05 last 83.86) - open/close 7381 (time 1787833614.00 total 90.07 last 83.30) - open/close 8209 (time 1787833624.01 total 100.08 last 82.75) - open/close 8946 (time 1787833634.04 total 110.11 last 73.45) - open/close 9686 (time 1787833644.05 total 120.12 last 73.92) total: 10000 open/close in 124.38 seconds: 80.40 ops/second - unlinked 0 (time 1787833652 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 651 (time 1787833742.43 total 10.01 last 65.06) - open/close 1218 (time 1787833752.43 total 20.01 last 56.70) - open/close 2087 (time 1787833762.44 total 30.02 last 86.82) - open/close 2885 (time 1787833772.44 total 40.02 last 79.78) - open/close 3663 (time 1787833782.44 total 50.02 last 77.78) - open/close 4499 (time 1787833792.45 total 60.03 last 83.50) - open/close 5288 (time 1787833802.46 total 70.04 last 78.85) - open/close 6089 (time 1787833812.46 total 80.04 last 80.09) - open/close 6915 (time 1787833822.46 total 90.04 last 82.58) - open/close 7649 (time 1787833832.47 total 100.05 last 73.38) - open/close 8487 (time 1787833842.47 total 110.05 last 83.76) - open/close 9314 (time 1787833852.50 total 120.08 last 82.48) total: 10000 open/close in 129.26 seconds: 77.36 ops/second - unlinked 0 (time 1787833866 ; total 1 ; last 1) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 838 (time 1787833950.53 total 10.00 last 83.78) - open/close 1669 (time 1787833960.54 total 20.01 last 83.04) - open/close 2543 (time 1787833970.55 total 30.02 last 87.32) - open/close 3351 (time 1787833980.55 total 40.02 last 80.80) - open/close 4183 (time 1787833990.55 total 50.02 last 83.17) - open/close 4959 (time 1787834000.55 total 60.03 last 77.57) - open/close 5830 (time 1787834010.56 total 70.04 last 87.01) - open/close 6719 (time 1787834020.58 total 80.06 last 88.71) - open/close 7469 (time 1787834030.59 total 90.06 last 74.95) - open/close 8191 (time 1787834040.62 total 100.09 last 72.03) - open/close 8903 (time 1787834050.63 total 110.10 last 71.12) - open/close 9599 (time 1787834060.64 total 120.11 last 69.53) total: 10000 open/close in 126.28 seconds: 79.19 ops/second - unlinked 0 (time 1787834071 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 553 (time 1787834171.82 total 10.00 last 55.29) - open/close 1238 (time 1787834181.85 total 20.03 last 68.29) - open/close 1730 (time 1787834191.86 total 30.04 last 49.15) - open/close 2286 (time 1787834201.88 total 40.06 last 55.49) - open/close 2890 (time 1787834211.88 total 50.06 last 60.39) - open/close 3492 (time 1787834221.89 total 60.07 last 60.15) - open/close 4202 (time 1787834231.90 total 70.08 last 70.91) - open/close 4996 (time 1787834241.91 total 80.09 last 79.37) - open/close 5692 (time 1787834251.92 total 90.11 last 69.47) - open/close 6300 (time 1787834261.93 total 100.11 last 60.79) - open/close 6939 (time 1787834271.94 total 110.12 last 63.84) - open/close 7470 (time 1787834281.94 total 120.13 last 53.07) - open/close 8051 (time 1787834291.95 total 130.13 last 58.05) - open/close 8665 (time 1787834301.98 total 140.16 last 61.21) - open/close 9260 (time 1787834311.99 total 150.18 last 59.43) - open/close 9877 (time 1787834322.01 total 160.19 last 61.61) total: 10000 open/close in 161.76 seconds: 61.82 ops/second - unlinked 0 (time 1787834329 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 599 (time 1787834437.27 total 10.01 last 59.86) - open/close 1265 (time 1787834447.27 total 20.01 last 66.58) - open/close 1728 (time 1787834457.28 total 30.02 last 46.25) - open/close 2289 (time 1787834467.29 total 40.03 last 56.07) - open/close 2891 (time 1787834477.29 total 50.03 last 60.17) - open/close 3617 (time 1787834487.30 total 60.04 last 72.56) - open/close 4288 (time 1787834497.31 total 70.05 last 67.05) - open/close 4993 (time 1787834507.31 total 80.05 last 70.49) - open/close 5696 (time 1787834517.32 total 90.06 last 70.20) - open/close 6295 (time 1787834527.33 total 100.07 last 59.88) - open/close 6925 (time 1787834537.33 total 110.07 last 62.99) - open/close 7636 (time 1787834547.33 total 120.07 last 71.06) - open/close 8364 (time 1787834557.34 total 130.08 last 72.73) - open/close 9080 (time 1787834567.34 total 140.09 last 71.58) - open/close 9870 (time 1787834577.35 total 150.09 last 78.99) total: 10000 open/close in 151.40 seconds: 66.05 ops/second - unlinked 0 (time 1787834583 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg426-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg426-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg426-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg426-server: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg426-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg426-server: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg426-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg426-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg426-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.126@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.126@tcp sys.timeout=20 start ost1 service on oleg426-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg426-server: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg426-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg426-server: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg426-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg426-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg426-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server stop mds service on oleg426-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg426-server stop mds service on oleg426-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg426-server unloading modules via unload_modules_local on: 'oleg426-server' oleg426-server: oleg426-server.virtnet: executing unload_modules_local oleg426-server: modules unloaded. pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2