== conf-sanity test 69: replace an OST with the same index ========================================================== 09:31:10 (1786627870) start mds service on oleg311-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg311-server: detected 4 online CPUs by sysfs oleg311-server: MODOPTS_LIBCFS= oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg311-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg311-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg311-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg311-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 744 (time 1786627987.96 total 10.03 last 74.21) - open/close 1421 (time 1786627997.97 total 20.03 last 67.64) - open/close 2144 (time 1786628007.97 total 30.04 last 72.26) - open/close 2890 (time 1786628017.98 total 40.05 last 74.51) - open/close 3592 (time 1786628027.99 total 50.06 last 70.16) - open/close 4339 (time 1786628038.00 total 60.07 last 74.62) - open/close 5079 (time 1786628048.00 total 70.07 last 73.99) - open/close 5831 (time 1786628058.01 total 80.08 last 75.13) - open/close 6594 (time 1786628068.01 total 90.08 last 76.29) - open/close 7041 (time 1786628078.04 total 100.10 last 44.59) - open/close 7458 (time 1786628088.05 total 110.11 last 41.66) - open/close 8112 (time 1786628098.06 total 120.12 last 65.35) - open/close 8888 (time 1786628108.06 total 130.12 last 77.59) - open/close 9646 (time 1786628118.07 total 140.13 last 75.71) total: 10000 open/close in 144.70 seconds: 69.11 ops/second - unlinked 0 (time 1786628126 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 727 (time 1786628219.00 total 10.00 last 72.70) - open/close 1504 (time 1786628229.01 total 20.01 last 77.62) - open/close 2299 (time 1786628239.02 total 30.02 last 79.40) - open/close 3045 (time 1786628249.02 total 40.02 last 74.60) - open/close 3775 (time 1786628259.04 total 50.04 last 72.88) - open/close 4524 (time 1786628269.04 total 60.04 last 74.86) - open/close 5339 (time 1786628279.05 total 70.05 last 81.43) - open/close 6009 (time 1786628289.06 total 80.06 last 66.96) - open/close 6540 (time 1786628299.07 total 90.07 last 53.05) - open/close 7073 (time 1786628309.07 total 100.07 last 53.28) - open/close 7655 (time 1786628319.08 total 110.08 last 58.14) - open/close 8366 (time 1786628329.08 total 120.09 last 71.08) - open/close 9140 (time 1786628339.10 total 130.10 last 77.31) - open/close 9871 (time 1786628349.10 total 140.10 last 73.05) total: 10000 open/close in 141.74 seconds: 70.55 ops/second - unlinked 0 (time 1786628354 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 684 (time 1786628448.69 total 10.00 last 68.37) - open/close 1431 (time 1786628458.70 total 20.01 last 74.64) - open/close 2231 (time 1786628468.71 total 30.02 last 79.96) - open/close 3071 (time 1786628478.72 total 40.03 last 83.90) - open/close 3765 (time 1786628488.72 total 50.03 last 69.37) - open/close 4556 (time 1786628498.72 total 60.04 last 79.09) - open/close 5308 (time 1786628508.73 total 70.04 last 75.17) - open/close 6090 (time 1786628518.74 total 80.05 last 78.12) - open/close 6849 (time 1786628528.74 total 90.05 last 75.87) - open/close 7620 (time 1786628538.75 total 100.06 last 77.04) - open/close 8426 (time 1786628548.75 total 110.06 last 80.59) - open/close 9126 (time 1786628558.76 total 120.07 last 69.92) - open/close 9884 (time 1786628568.77 total 130.08 last 75.74) total: 10000 open/close in 131.98 seconds: 75.77 ops/second - unlinked 0 (time 1786628574 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 722 (time 1786628664.75 total 10.01 last 72.15) - open/close 1526 (time 1786628674.76 total 20.01 last 80.37) - open/close 2328 (time 1786628684.77 total 30.02 last 80.09) - open/close 3219 (time 1786628694.78 total 40.04 last 88.97) - open/close 3918 (time 1786628704.79 total 50.05 last 69.85) - open/close 4673 (time 1786628714.79 total 60.05 last 75.48) - open/close 5438 (time 1786628724.80 total 70.05 last 76.48) - open/close 6263 (time 1786628734.80 total 80.05 last 82.48) - open/close 7048 (time 1786628744.81 total 90.06 last 78.41) - open/close 7800 (time 1786628754.81 total 100.07 last 75.19) - open/close 8637 (time 1786628764.82 total 110.07 last 83.63) - open/close 9432 (time 1786628774.82 total 120.08 last 79.47) total: 10000 open/close in 127.20 seconds: 78.61 ops/second - unlinked 0 (time 1786628785 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 701 (time 1786628872.19 total 10.02 last 69.99) - open/close 1380 (time 1786628882.20 total 20.02 last 67.85) - open/close 2084 (time 1786628892.21 total 30.03 last 70.37) - open/close 2835 (time 1786628902.22 total 40.04 last 75.00) - open/close 3698 (time 1786628912.23 total 50.05 last 86.24) - open/close 4454 (time 1786628922.23 total 60.06 last 75.55) - open/close 5249 (time 1786628932.24 total 70.06 last 79.46) - open/close 6053 (time 1786628942.24 total 80.06 last 80.38) - open/close 6826 (time 1786628952.25 total 90.07 last 77.23) - open/close 7573 (time 1786628962.26 total 100.08 last 74.63) - open/close 8339 (time 1786628972.26 total 110.09 last 76.56) - open/close 9104 (time 1786628982.28 total 120.10 last 76.42) - open/close 9874 (time 1786628992.28 total 130.10 last 76.98) total: 10000 open/close in 131.92 seconds: 75.80 ops/second - unlinked 0 (time 1786628998 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 691 (time 1786629087.22 total 10.02 last 68.93) - open/close 1329 (time 1786629097.23 total 20.03 last 63.74) - open/close 2015 (time 1786629107.25 total 30.05 last 68.47) - open/close 2743 (time 1786629117.25 total 40.06 last 72.78) - open/close 3417 (time 1786629127.28 total 50.08 last 67.24) - open/close 4249 (time 1786629137.28 total 60.08 last 83.19) - open/close 5084 (time 1786629147.28 total 70.09 last 83.46) - open/close 5812 (time 1786629157.29 total 80.09 last 72.74) - open/close 6589 (time 1786629167.30 total 90.10 last 77.64) - open/close 7331 (time 1786629177.31 total 100.11 last 74.14) - open/close 8082 (time 1786629187.31 total 110.12 last 75.05) - open/close 8866 (time 1786629197.32 total 120.13 last 78.33) - open/close 9591 (time 1786629207.36 total 130.16 last 72.26) total: 10000 open/close in 135.69 seconds: 73.70 ops/second - unlinked 0 (time 1786629217 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 710 (time 1786629310.35 total 10.00 last 70.98) - open/close 1479 (time 1786629320.36 total 20.01 last 76.83) - open/close 2208 (time 1786629330.36 total 30.01 last 72.88) - open/close 2955 (time 1786629340.37 total 40.02 last 74.63) - open/close 3703 (time 1786629350.38 total 50.03 last 74.74) - open/close 4476 (time 1786629360.40 total 60.05 last 77.17) - open/close 5258 (time 1786629370.41 total 70.06 last 78.11) - open/close 6056 (time 1786629380.43 total 80.08 last 79.63) - open/close 6827 (time 1786629390.44 total 90.09 last 77.06) - open/close 7509 (time 1786629400.44 total 100.09 last 68.16) - open/close 8316 (time 1786629410.45 total 110.10 last 80.67) - open/close 9133 (time 1786629420.45 total 120.10 last 81.69) - open/close 9907 (time 1786629430.45 total 130.10 last 77.39) total: 10000 open/close in 131.40 seconds: 76.10 ops/second - unlinked 0 (time 1786629436 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 676 (time 1786629527.68 total 10.01 last 67.55) - open/close 1348 (time 1786629537.68 total 20.01 last 67.20) - open/close 2161 (time 1786629547.69 total 30.02 last 81.21) - open/close 2919 (time 1786629557.69 total 40.02 last 75.77) - open/close 3686 (time 1786629567.70 total 50.04 last 76.60) - open/close 4400 (time 1786629577.72 total 60.05 last 71.32) - open/close 5108 (time 1786629587.72 total 70.05 last 70.76) - open/close 5873 (time 1786629597.73 total 80.06 last 76.45) - open/close 6611 (time 1786629607.74 total 90.07 last 73.74) - open/close 7430 (time 1786629617.74 total 100.07 last 81.88) - open/close 8192 (time 1786629627.74 total 110.08 last 76.16) - open/close 8921 (time 1786629637.75 total 120.09 last 72.83) - open/close 9685 (time 1786629647.76 total 130.09 last 76.35) total: 10000 open/close in 133.89 seconds: 74.69 ops/second - unlinked 0 (time 1786629655 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 773 (time 1786629743.71 total 10.01 last 77.25) - open/close 1582 (time 1786629753.72 total 20.02 last 80.81) - open/close 2320 (time 1786629763.73 total 30.02 last 73.75) - open/close 3078 (time 1786629773.76 total 40.05 last 75.60) - open/close 3858 (time 1786629783.77 total 50.06 last 77.92) - open/close 4556 (time 1786629793.77 total 60.07 last 69.77) - open/close 5257 (time 1786629803.78 total 70.08 last 70.02) - open/close 6059 (time 1786629813.79 total 80.09 last 80.10) - open/close 6860 (time 1786629823.80 total 90.09 last 80.05) - open/close 7606 (time 1786629833.81 total 100.10 last 74.53) - open/close 8396 (time 1786629843.81 total 110.11 last 78.98) - open/close 9158 (time 1786629853.82 total 120.12 last 76.12) - open/close 9881 (time 1786629863.83 total 130.13 last 72.23) total: 10000 open/close in 131.73 seconds: 75.91 ops/second - unlinked 0 (time 1786629869 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 777 (time 1786629958.90 total 10.00 last 77.69) - open/close 1528 (time 1786629968.91 total 20.02 last 74.98) - open/close 2280 (time 1786629978.92 total 30.03 last 75.12) - open/close 3062 (time 1786629988.93 total 40.04 last 78.12) - open/close 3826 (time 1786629998.95 total 50.06 last 76.26) - open/close 4582 (time 1786630008.95 total 60.06 last 75.58) - open/close 5345 (time 1786630018.96 total 70.06 last 76.26) - open/close 6121 (time 1786630028.97 total 80.08 last 77.50) - open/close 6821 (time 1786630038.98 total 90.08 last 69.95) - open/close 7576 (time 1786630048.99 total 100.09 last 75.45) - open/close 8377 (time 1786630058.99 total 110.10 last 80.07) - open/close 9153 (time 1786630069.01 total 120.11 last 77.49) - open/close 9860 (time 1786630079.02 total 130.12 last 70.63) total: 10000 open/close in 132.02 seconds: 75.74 ops/second - unlinked 0 (time 1786630085 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.111@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg311-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg311-server: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg311-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg311-server: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg311-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.111@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.111@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.111@tcp sys.timeout=20 start ost1 service on oleg311-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg311-server: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg311-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg311-server: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg311-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg311-server unloading modules via unload_modules_local on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local oleg311-server: modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2