== conf-sanity test 69: replace an OST with the same index ========================================================== 17:45:03 (1788903903) start mds service on oleg207-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg207-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg207-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 2249 (time 1788903966.01 total 10.00 last 224.84) - open/close 4672 (time 1788903976.01 total 20.01 last 242.24) - open/close 6823 (time 1788903986.02 total 30.02 last 214.81) - open/close 8973 (time 1788903996.02 total 40.02 last 214.99) total: 10000 open/close in 44.38 seconds: 225.31 ops/second - unlinked 0 (time 1788904001 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2246 (time 1788904036.83 total 10.00 last 224.53) - open/close 4280 (time 1788904046.83 total 20.00 last 203.39) - open/close 5978 (time 1788904056.83 total 30.01 last 169.77) - open/close 7998 (time 1788904066.83 total 40.01 last 201.96) total: 10000 open/close in 48.76 seconds: 205.07 ops/second - unlinked 0 (time 1788904077 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2441 (time 1788904112.15 total 10.01 last 243.89) - open/close 4433 (time 1788904122.15 total 20.01 last 199.18) - open/close 6981 (time 1788904132.15 total 30.01 last 254.69) - open/close 9391 (time 1788904142.15 total 40.01 last 240.98) total: 10000 open/close in 42.70 seconds: 234.22 ops/second - unlinked 0 (time 1788904146 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1342 (time 1788904197.12 total 10.00 last 134.15) - open/close 2631 (time 1788904207.13 total 20.01 last 128.84) - open/close 4088 (time 1788904217.13 total 30.01 last 145.64) - open/close 5195 (time 1788904227.14 total 40.02 last 110.62) - open/close 6002 (time 1788904237.15 total 50.02 last 80.66) - open/close 6690 (time 1788904247.16 total 60.04 last 68.72) - open/close 7371 (time 1788904257.17 total 70.05 last 68.04) - open/close 7666 (time 1788904267.17 total 80.05 last 29.49) - open/close 7969 (time 1788904277.20 total 90.08 last 30.20) - open/close 8228 (time 1788904287.21 total 100.09 last 25.89) - open/close 8490 (time 1788904297.22 total 110.10 last 26.16) - open/close 8890 (time 1788904307.24 total 120.12 last 39.95) - open/close 9332 (time 1788904317.24 total 130.12 last 44.19) total: 10000 open/close in 137.48 seconds: 72.74 ops/second - unlinked 0 (time 1788904328 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 931 (time 1788904403.80 total 10.00 last 93.09) - open/close 1702 (time 1788904413.80 total 20.00 last 77.07) - open/close 2508 (time 1788904423.81 total 30.01 last 80.52) - open/close 3488 (time 1788904433.82 total 40.02 last 97.97) - open/close 4317 (time 1788904443.82 total 50.02 last 82.84) - open/close 5131 (time 1788904453.83 total 60.03 last 81.37) - open/close 6130 (time 1788904463.84 total 70.04 last 99.82) - open/close 7101 (time 1788904473.84 total 80.04 last 97.04) - open/close 7879 (time 1788904483.85 total 90.05 last 77.74) - open/close 8661 (time 1788904493.85 total 100.05 last 78.20) - open/close 9517 (time 1788904503.85 total 110.05 last 85.60) total: 10000 open/close in 115.73 seconds: 86.41 ops/second - unlinked 0 (time 1788904514 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 787 (time 1788904598.04 total 10.02 last 78.58) - open/close 1619 (time 1788904608.05 total 20.03 last 83.11) - open/close 2471 (time 1788904618.05 total 30.03 last 85.18) - open/close 3266 (time 1788904628.06 total 40.03 last 79.47) - open/close 4135 (time 1788904638.07 total 50.04 last 86.79) - open/close 5099 (time 1788904648.08 total 60.05 last 96.31) - open/close 6018 (time 1788904658.09 total 70.06 last 91.84) - open/close 6847 (time 1788904668.09 total 80.06 last 82.90) - open/close 7784 (time 1788904678.09 total 90.06 last 93.67) - open/close 8676 (time 1788904688.11 total 100.09 last 88.99) - open/close 9334 (time 1788904698.13 total 110.11 last 65.68) total: 10000 open/close in 117.12 seconds: 85.38 ops/second - unlinked 0 (time 1788904708 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1200 (time 1788904782.38 total 10.00 last 119.98) - open/close 2277 (time 1788904792.38 total 20.00 last 107.70) - open/close 3337 (time 1788904802.38 total 30.00 last 105.97) - open/close 4448 (time 1788904812.39 total 40.01 last 111.02) - open/close 5597 (time 1788904822.39 total 50.02 last 114.84) - open/close 6725 (time 1788904832.39 total 60.02 last 112.80) - open/close 7922 (time 1788904842.40 total 70.03 last 119.60) - open/close 9043 (time 1788904852.41 total 80.03 last 112.00) total: 10000 open/close in 88.64 seconds: 112.81 ops/second - unlinked 0 (time 1788904863 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1268 (time 1788904924.22 total 10.00 last 126.75) - open/close 2562 (time 1788904934.22 total 20.01 last 129.39) - open/close 3704 (time 1788904944.22 total 30.01 last 114.15) - open/close 4954 (time 1788904954.23 total 40.02 last 124.87) - open/close 6314 (time 1788904964.24 total 50.02 last 135.98) - open/close 7616 (time 1788904974.24 total 60.02 last 130.18) - open/close 8858 (time 1788904984.24 total 70.03 last 124.13) total: 10000 open/close in 78.10 seconds: 128.04 ops/second - unlinked 0 (time 1788904995 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1438 (time 1788905043.99 total 10.00 last 143.80) - open/close 3015 (time 1788905054.00 total 20.01 last 157.61) - open/close 4708 (time 1788905064.00 total 30.01 last 169.24) - open/close 6744 (time 1788905074.00 total 40.01 last 203.59) - open/close 9604 (time 1788905084.01 total 50.01 last 285.97) total: 10000 open/close in 51.06 seconds: 195.83 ops/second - unlinked 0 (time 1788905086 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2445 (time 1788905117.63 total 10.00 last 244.47) - open/close 5014 (time 1788905127.64 total 20.00 last 256.81) - open/close 7928 (time 1788905137.64 total 30.01 last 291.22) total: 10000 open/close in 35.91 seconds: 278.47 ops/second - unlinked 0 (time 1788905144 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre On OST0, 15078 used inodes rc=0 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 13 sec oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2