== conf-sanity test 69: replace an OST with the same index ========================================================== 10:59:39 (1776869979) start mds service on oleg232-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg232-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg232-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1716 (time 1776870036.65 total 10.01 last 171.44) - open/close 3531 (time 1776870046.65 total 20.01 last 181.47) - open/close 5330 (time 1776870056.66 total 30.02 last 179.81) - open/close 7730 (time 1776870066.66 total 40.02 last 239.98) total: 10000 open/close in 49.68 seconds: 201.30 ops/second - unlinked 0 (time 1776870078 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 1634 (time 1776870114.46 total 10.00 last 163.38) - open/close 3857 (time 1776870124.46 total 20.00 last 222.26) - open/close 5750 (time 1776870134.46 total 30.00 last 189.28) - open/close 8035 (time 1776870144.46 total 40.00 last 228.49) total: 10000 open/close in 47.20 seconds: 211.85 ops/second - unlinked 0 (time 1776870153 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2432 (time 1776870183.91 total 10.00 last 243.19) - open/close 5003 (time 1776870193.91 total 20.00 last 257.04) - open/close 7743 (time 1776870203.91 total 30.00 last 273.97) total: 10000 open/close in 36.71 seconds: 272.41 ops/second - unlinked 0 (time 1776870212 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3308 (time 1776870240.89 total 10.00 last 330.78) - open/close 7001 (time 1776870250.89 total 20.00 last 369.23) total: 10000 open/close in 29.82 seconds: 335.34 ops/second - unlinked 0 (time 1776870262 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3289 (time 1776870291.18 total 10.00 last 328.89) - open/close 7664 (time 1776870301.18 total 20.00 last 437.46) total: 10000 open/close in 25.39 seconds: 393.79 ops/second - unlinked 0 (time 1776870307 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 3989 (time 1776870331.70 total 10.00 last 398.85) - open/close 8081 (time 1776870341.70 total 20.00 last 409.12) total: 10000 open/close in 24.57 seconds: 406.98 ops/second - unlinked 0 (time 1776870347 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4186 (time 1776870371.51 total 10.00 last 418.57) - open/close 8912 (time 1776870381.51 total 20.00 last 472.56) total: 10000 open/close in 22.18 seconds: 450.83 ops/second - unlinked 0 (time 1776870385 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 5066 (time 1776870407.81 total 10.00 last 506.56) - open/close 9743 (time 1776870417.82 total 20.00 last 467.65) total: 10000 open/close in 20.50 seconds: 487.74 ops/second - unlinked 0 (time 1776870419 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 4546 (time 1776870441.66 total 10.00 last 454.41) - open/close 8948 (time 1776870451.66 total 20.00 last 440.19) total: 10000 open/close in 22.35 seconds: 447.47 ops/second - unlinked 0 (time 1776870455 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 4583 (time 1776870477.98 total 10.00 last 458.27) - open/close 9701 (time 1776870487.98 total 20.00 last 511.74) total: 10000 open/close in 20.61 seconds: 485.09 ops/second - unlinked 0 (time 1776870489 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre On OST0, 15618 used inodes rc=0 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 15 sec oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg232-server unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local oleg232-server: modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2