== conf-sanity test 69: replace an OST with the same index ========================================================== 16:22:04 (1784233324) start mds service on oleg260-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg260-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg260-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 943 (time 1784233425.08 total 10.01 last 94.18) - open/close 1949 (time 1784233435.08 total 20.01 last 100.59) - open/close 2951 (time 1784233445.09 total 30.02 last 100.13) - open/close 3862 (time 1784233455.10 total 40.03 last 91.03) - open/close 4794 (time 1784233465.11 total 50.04 last 93.13) - open/close 5721 (time 1784233475.11 total 60.05 last 92.62) - open/close 6617 (time 1784233485.12 total 70.05 last 89.58) - open/close 7516 (time 1784233495.12 total 80.05 last 89.85) - open/close 8362 (time 1784233505.14 total 90.07 last 84.46) - open/close 9162 (time 1784233515.15 total 100.08 last 79.94) total: 10000 open/close in 109.94 seconds: 90.95 ops/second - unlinked 0 (time 1784233528 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 971 (time 1784233609.19 total 10.01 last 97.02) - open/close 2035 (time 1784233619.19 total 20.01 last 106.38) - open/close 3043 (time 1784233629.20 total 30.02 last 100.74) - open/close 4071 (time 1784233639.21 total 40.02 last 102.73) - open/close 5130 (time 1784233649.21 total 50.03 last 105.87) - open/close 6272 (time 1784233659.21 total 60.03 last 114.19) - open/close 7365 (time 1784233669.22 total 70.03 last 109.23) - open/close 8610 (time 1784233679.22 total 80.04 last 124.47) - open/close 9833 (time 1784233689.23 total 90.04 last 122.23) total: 10000 open/close in 91.31 seconds: 109.52 ops/second - unlinked 0 (time 1784233693 ; total 1 ; last 1) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1227 (time 1784233750.20 total 10.01 last 122.62) - open/close 2379 (time 1784233760.20 total 20.01 last 115.20) - open/close 3668 (time 1784233770.21 total 30.01 last 128.80) - open/close 4854 (time 1784233780.22 total 40.02 last 118.53) - open/close 6086 (time 1784233790.22 total 50.02 last 123.17) - open/close 7377 (time 1784233800.22 total 60.02 last 129.09) - open/close 8974 (time 1784233810.22 total 70.03 last 159.64) total: 10000 open/close in 77.59 seconds: 128.88 ops/second - unlinked 0 (time 1784233820 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1314 (time 1784233863.75 total 10.00 last 131.39) - open/close 2590 (time 1784233873.75 total 20.00 last 127.59) - open/close 4214 (time 1784233883.76 total 30.00 last 162.35) - open/close 5634 (time 1784233893.76 total 40.01 last 141.91) - open/close 6954 (time 1784233903.76 total 50.01 last 131.97) - open/close 8667 (time 1784233913.76 total 60.01 last 171.28) - open/close 9978 (time 1784233923.77 total 70.02 last 131.02) total: 10000 open/close in 70.18 seconds: 142.50 ops/second - unlinked 0 (time 1784233926 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1209 (time 1784233983.48 total 10.00 last 120.90) - open/close 2561 (time 1784233993.49 total 20.01 last 135.08) - open/close 3858 (time 1784234003.49 total 30.01 last 129.66) - open/close 5196 (time 1784234013.50 total 40.02 last 133.68) - open/close 6528 (time 1784234023.52 total 50.03 last 133.03) - open/close 7878 (time 1784234033.52 total 60.04 last 134.90) - open/close 9078 (time 1784234043.52 total 70.04 last 119.99) total: 10000 open/close in 78.03 seconds: 128.16 ops/second - unlinked 0 (time 1784234053 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1279 (time 1784234107.09 total 10.00 last 127.88) - open/close 2570 (time 1784234117.10 total 20.01 last 129.01) - open/close 4181 (time 1784234127.11 total 30.01 last 161.03) - open/close 5852 (time 1784234137.11 total 40.02 last 167.02) - open/close 7691 (time 1784234147.11 total 50.02 last 183.89) - open/close 9281 (time 1784234157.11 total 60.02 last 158.93) total: 10000 open/close in 65.44 seconds: 152.81 ops/second - unlinked 0 (time 1784234164 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 2033 (time 1784234209.33 total 10.01 last 203.16) - open/close 3559 (time 1784234219.33 total 20.01 last 152.59) - open/close 5425 (time 1784234229.34 total 30.01 last 186.53) - open/close 7892 (time 1784234239.34 total 40.01 last 246.69) - open/close 9838 (time 1784234249.34 total 50.01 last 194.58) total: 10000 open/close in 51.04 seconds: 195.93 ops/second - unlinked 0 (time 1784234252 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1963 (time 1784234293.33 total 10.01 last 196.13) - open/close 3821 (time 1784234303.33 total 20.01 last 185.78) - open/close 6421 (time 1784234313.34 total 30.02 last 259.85) - open/close 8487 (time 1784234323.34 total 40.02 last 206.43) total: 10000 open/close in 47.01 seconds: 212.74 ops/second - unlinked 0 (time 1784234332 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 2175 (time 1784234374.34 total 10.00 last 217.42) - open/close 3614 (time 1784234384.34 total 20.01 last 143.80) - open/close 5389 (time 1784234394.35 total 30.02 last 177.42) - open/close 7162 (time 1784234404.35 total 40.02 last 177.28) - open/close 8718 (time 1784234414.35 total 50.02 last 155.55) total: 10000 open/close in 56.78 seconds: 176.11 ops/second - unlinked 0 (time 1784234422 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 2182 (time 1784234468.71 total 10.00 last 218.19) - open/close 4583 (time 1784234478.71 total 20.00 last 240.09) - open/close 6890 (time 1784234488.72 total 30.00 last 230.64) - open/close 8943 (time 1784234498.72 total 40.01 last 205.19) total: 10000 open/close in 45.92 seconds: 217.77 ops/second - unlinked 0 (time 1784234506 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 13 sec oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2