== conf-sanity test 69: replace an OST with the same index ========================================================== 21:04:48 (1787101488) start mds service on oleg148-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions oleg148-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg148-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg148-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg148-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg148-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1032 (time 1787101563.46 total 10.01 last 103.14) - open/close 2168 (time 1787101573.46 total 20.01 last 113.59) - open/close 3372 (time 1787101583.46 total 30.01 last 120.34) - open/close 4625 (time 1787101593.47 total 40.01 last 125.27) - open/close 5865 (time 1787101603.49 total 50.04 last 123.69) - open/close 7108 (time 1787101613.49 total 60.04 last 124.29) - open/close 8383 (time 1787101623.50 total 70.04 last 127.44) - open/close 9548 (time 1787101633.51 total 80.06 last 116.37) total: 10000 open/close in 83.73 seconds: 119.43 ops/second - unlinked 0 (time 1787101639 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1215 (time 1787101694.45 total 10.01 last 121.43) - open/close 2473 (time 1787101704.45 total 20.01 last 125.77) - open/close 3760 (time 1787101714.46 total 30.01 last 128.61) - open/close 5118 (time 1787101724.46 total 40.02 last 135.74) - open/close 6434 (time 1787101734.47 total 50.02 last 131.53) - open/close 7846 (time 1787101744.47 total 60.03 last 141.14) - open/close 9081 (time 1787101754.47 total 70.03 last 123.45) total: 10000 open/close in 77.96 seconds: 128.27 ops/second - unlinked 0 (time 1787101764 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1274 (time 1787101817.91 total 10.00 last 127.36) - open/close 2575 (time 1787101827.91 total 20.00 last 130.09) - open/close 3977 (time 1787101837.92 total 30.01 last 140.11) - open/close 5291 (time 1787101847.92 total 40.01 last 131.35) - open/close 6638 (time 1787101857.93 total 50.02 last 134.60) - open/close 8020 (time 1787101867.93 total 60.02 last 138.20) - open/close 9394 (time 1787101877.93 total 70.02 last 137.39) total: 10000 open/close in 73.73 seconds: 135.64 ops/second - unlinked 0 (time 1787101883 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1569 (time 1787101934.52 total 10.00 last 156.90) - open/close 3096 (time 1787101944.53 total 20.00 last 152.64) - open/close 4334 (time 1787101954.53 total 30.01 last 123.79) - open/close 5569 (time 1787101964.54 total 40.01 last 123.42) - open/close 6936 (time 1787101974.54 total 50.01 last 136.66) - open/close 8311 (time 1787101984.54 total 60.01 last 137.49) - open/close 9734 (time 1787101994.55 total 70.03 last 142.12) total: 10000 open/close in 72.25 seconds: 138.41 ops/second - unlinked 0 (time 1787101998 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1543 (time 1787102047.88 total 10.00 last 154.27) - open/close 3003 (time 1787102057.89 total 20.01 last 145.82) - open/close 4499 (time 1787102067.90 total 30.02 last 149.57) - open/close 6103 (time 1787102077.90 total 40.02 last 160.33) - open/close 8069 (time 1787102087.90 total 50.02 last 196.58) total: 10000 open/close in 57.66 seconds: 173.42 ops/second - unlinked 0 (time 1787102097 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1240 (time 1787102141.04 total 10.00 last 123.96) - open/close 2552 (time 1787102151.04 total 20.01 last 131.11) - open/close 4412 (time 1787102161.04 total 30.01 last 185.96) - open/close 6461 (time 1787102171.05 total 40.01 last 204.85) - open/close 8240 (time 1787102181.05 total 50.01 last 177.90) total: 10000 open/close in 57.83 seconds: 172.93 ops/second - unlinked 0 (time 1787102190 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 1949 (time 1787102224.09 total 10.00 last 194.81) - open/close 3696 (time 1787102234.09 total 20.01 last 174.67) - open/close 5399 (time 1787102244.10 total 30.01 last 170.25) - open/close 7285 (time 1787102254.10 total 40.02 last 188.47) - open/close 9542 (time 1787102264.10 total 50.02 last 225.66) total: 10000 open/close in 51.69 seconds: 193.46 ops/second - unlinked 0 (time 1787102267 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1602 (time 1787102309.74 total 10.00 last 160.17) - open/close 3083 (time 1787102319.74 total 20.01 last 148.04) - open/close 5106 (time 1787102329.75 total 30.01 last 202.26) - open/close 6935 (time 1787102339.75 total 40.01 last 182.90) - open/close 8325 (time 1787102349.76 total 50.02 last 138.83) - open/close 9843 (time 1787102359.76 total 60.03 last 151.72) total: 10000 open/close in 60.79 seconds: 164.49 ops/second - unlinked 0 (time 1787102362 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2793 (time 1787102392.97 total 10.00 last 279.26) - open/close 5099 (time 1787102402.97 total 20.01 last 230.50) - open/close 7779 (time 1787102412.97 total 30.01 last 267.97) total: 10000 open/close in 38.08 seconds: 262.61 ops/second - unlinked 0 (time 1787102422 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3215 (time 1787102453.81 total 10.00 last 321.40) - open/close 5644 (time 1787102463.81 total 20.01 last 242.82) - open/close 8172 (time 1787102473.82 total 30.01 last 252.72) total: 10000 open/close in 35.84 seconds: 279.04 ops/second - unlinked 0 (time 1787102481 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.148@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg148-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg148-server: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg148-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg148-server: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg148-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.148@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.148@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.148@tcp sys.timeout=20 start ost1 service on oleg148-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg148-server: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg148-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg148-server: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg148-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg148-server unloading modules via unload_modules_local on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing unload_modules_local oleg148-server: modules unloaded. pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2