== conf-sanity test 69: replace an OST with the same index ========================================================== 15:44:31 (1775763871) start mds service on oleg618-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg618-server' oleg618-server: oleg618-server.virtnet: executing load_modules_local oleg618-server: Loading modules from /home/green/git/lustre-release/lustre oleg618-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg618-server: detected 4 online CPUs by sysfs oleg618-server: MODOPTS_LIBCFS= oleg618-server: Force libcfs to create 2 CPU partitions oleg618-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg618-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg618-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg618-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg618-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-OST0000 oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg618-client.virtnet: -o user_xattr,flock 192.168.206.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 677 (time 1775763987.62 total 10.00 last 67.69) - open/close 1307 (time 1775763997.64 total 20.02 last 62.90) - open/close 1960 (time 1775764007.64 total 30.02 last 65.30) - open/close 2652 (time 1775764017.65 total 40.03 last 69.12) - open/close 3330 (time 1775764027.68 total 50.06 last 67.61) - open/close 3996 (time 1775764037.70 total 60.08 last 66.44) - open/close 4636 (time 1775764047.71 total 70.09 last 63.96) - open/close 5302 (time 1775764057.71 total 80.09 last 66.57) - open/close 5976 (time 1775764067.72 total 90.10 last 67.34) - open/close 6545 (time 1775764077.74 total 100.11 last 56.82) - open/close 7140 (time 1775764087.75 total 110.13 last 59.39) - open/close 7803 (time 1775764097.77 total 120.14 last 66.22) - open/close 8464 (time 1775764107.77 total 130.15 last 66.09) - open/close 9171 (time 1775764117.77 total 140.15 last 70.66) - open/close 9875 (time 1775764127.77 total 150.15 last 70.40) total: 10000 open/close in 151.79 seconds: 65.88 ops/second - unlinked 0 (time 1775764133 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 608 (time 1775764232.13 total 10.03 last 60.64) - open/close 1227 (time 1775764242.13 total 20.03 last 61.88) - open/close 1890 (time 1775764252.14 total 30.04 last 66.25) - open/close 2621 (time 1775764262.16 total 40.06 last 72.94) - open/close 3309 (time 1775764272.16 total 50.06 last 68.80) - open/close 4022 (time 1775764282.17 total 60.07 last 71.25) - open/close 4779 (time 1775764292.17 total 70.07 last 75.67) - open/close 5456 (time 1775764302.18 total 80.08 last 67.67) - open/close 6161 (time 1775764312.18 total 90.08 last 70.46) - open/close 6792 (time 1775764322.21 total 100.10 last 62.95) - open/close 7389 (time 1775764332.21 total 110.11 last 59.68) - open/close 7981 (time 1775764342.22 total 120.12 last 59.14) - open/close 8622 (time 1775764352.23 total 130.12 last 64.07) - open/close 9229 (time 1775764362.24 total 140.14 last 60.61) - open/close 9873 (time 1775764372.24 total 150.14 last 64.39) total: 10000 open/close in 152.18 seconds: 65.71 ops/second - unlinked 0 (time 1775764378 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 718 (time 1775764472.40 total 10.01 last 71.73) - open/close 1382 (time 1775764482.41 total 20.02 last 66.31) - open/close 2020 (time 1775764492.41 total 30.03 last 63.78) - open/close 2670 (time 1775764502.43 total 40.04 last 64.92) - open/close 3402 (time 1775764512.43 total 50.04 last 73.15) - open/close 4089 (time 1775764522.45 total 60.06 last 68.60) - open/close 4790 (time 1775764532.47 total 70.08 last 69.95) - open/close 5423 (time 1775764542.48 total 80.09 last 63.22) - open/close 6085 (time 1775764552.49 total 90.10 last 66.17) - open/close 6812 (time 1775764562.49 total 100.10 last 72.68) - open/close 7459 (time 1775764572.49 total 110.10 last 64.69) - open/close 8158 (time 1775764582.51 total 120.12 last 69.77) - open/close 8848 (time 1775764592.51 total 130.12 last 68.98) - open/close 9523 (time 1775764602.53 total 140.14 last 67.40) total: 10000 open/close in 147.62 seconds: 67.74 ops/second - unlinked 0 (time 1775764614 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 724 (time 1775764708.64 total 10.01 last 72.33) - open/close 1468 (time 1775764718.64 total 20.01 last 74.40) - open/close 2203 (time 1775764728.65 total 30.02 last 73.40) - open/close 2834 (time 1775764738.67 total 40.04 last 62.99) - open/close 3592 (time 1775764748.67 total 50.05 last 75.76) - open/close 4240 (time 1775764758.67 total 60.05 last 64.80) - open/close 4847 (time 1775764768.68 total 70.06 last 60.64) - open/close 5539 (time 1775764778.68 total 80.06 last 69.18) - open/close 6238 (time 1775764788.69 total 90.06 last 69.87) - open/close 6933 (time 1775764798.70 total 100.07 last 69.43) - open/close 7583 (time 1775764808.71 total 110.08 last 64.92) - open/close 8326 (time 1775764818.72 total 120.09 last 74.23) - open/close 9132 (time 1775764828.72 total 130.10 last 80.56) - open/close 9780 (time 1775764838.73 total 140.11 last 64.74) total: 10000 open/close in 143.06 seconds: 69.90 ops/second - unlinked 0 (time 1775764846 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 624 (time 1775764944.57 total 10.00 last 62.39) - open/close 1312 (time 1775764954.58 total 20.01 last 68.71) - open/close 1975 (time 1775764964.59 total 30.02 last 66.25) - open/close 2628 (time 1775764974.60 total 40.03 last 65.25) - open/close 3245 (time 1775764984.61 total 50.04 last 61.63) - open/close 3861 (time 1775764994.63 total 60.06 last 61.48) - open/close 4597 (time 1775765004.63 total 70.07 last 73.56) - open/close 5344 (time 1775765014.63 total 80.07 last 74.69) - open/close 5995 (time 1775765024.64 total 90.07 last 65.07) - open/close 6641 (time 1775765034.65 total 100.08 last 64.56) - open/close 7307 (time 1775765044.66 total 110.09 last 66.52) - open/close 8009 (time 1775765054.67 total 120.10 last 70.11) - open/close 8648 (time 1775765064.68 total 130.11 last 63.82) - open/close 9324 (time 1775765074.68 total 140.11 last 67.59) total: 10000 open/close in 150.04 seconds: 66.65 ops/second - unlinked 0 (time 1775765088 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 682 (time 1775765189.82 total 10.01 last 68.11) - open/close 1310 (time 1775765199.82 total 20.01 last 62.80) - open/close 1934 (time 1775765209.83 total 30.02 last 62.35) - open/close 2644 (time 1775765219.83 total 40.03 last 70.95) - open/close 3301 (time 1775765229.85 total 50.05 last 65.59) - open/close 4014 (time 1775765239.85 total 60.05 last 71.29) - open/close 4697 (time 1775765249.85 total 70.05 last 68.30) - open/close 5382 (time 1775765259.87 total 80.06 last 68.40) - open/close 6034 (time 1775765269.89 total 90.08 last 65.07) - open/close 6806 (time 1775765279.89 total 100.09 last 77.18) - open/close 7498 (time 1775765289.90 total 110.09 last 69.15) - open/close 8215 (time 1775765299.90 total 120.10 last 71.66) - open/close 8836 (time 1775765309.90 total 130.10 last 62.10) - open/close 9498 (time 1775765319.91 total 140.10 last 66.17) total: 10000 open/close in 148.42 seconds: 67.37 ops/second - unlinked 0 (time 1775765333 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 634 (time 1775765444.08 total 10.01 last 63.33) - open/close 1273 (time 1775765454.09 total 20.02 last 63.82) - open/close 1946 (time 1775765464.10 total 30.03 last 67.26) - open/close 2612 (time 1775765474.10 total 40.04 last 66.57) - open/close 3211 (time 1775765484.11 total 50.04 last 59.88) - open/close 3818 (time 1775765494.13 total 60.06 last 60.58) - open/close 4529 (time 1775765504.14 total 70.07 last 70.98) - open/close 5142 (time 1775765514.17 total 80.10 last 61.14) - open/close 5756 (time 1775765524.18 total 90.11 last 61.32) - open/close 6417 (time 1775765534.19 total 100.12 last 66.07) - open/close 7096 (time 1775765544.20 total 110.14 last 67.78) - open/close 7797 (time 1775765554.21 total 120.15 last 70.03) - open/close 8389 (time 1775765564.22 total 130.16 last 59.15) - open/close 9095 (time 1775765574.23 total 140.16 last 70.56) - open/close 9705 (time 1775765584.23 total 150.17 last 60.97) total: 10000 open/close in 154.87 seconds: 64.57 ops/second - unlinked 0 (time 1775765593 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 729 (time 1775765693.39 total 10.00 last 72.90) - open/close 1435 (time 1775765703.40 total 20.00 last 70.57) - open/close 2173 (time 1775765713.40 total 30.01 last 73.76) - open/close 2864 (time 1775765723.41 total 40.02 last 69.04) - open/close 3536 (time 1775765733.42 total 50.02 last 67.16) - open/close 4252 (time 1775765743.42 total 60.03 last 71.56) - open/close 4920 (time 1775765753.43 total 70.04 last 66.71) - open/close 5537 (time 1775765763.45 total 80.06 last 61.59) - open/close 6202 (time 1775765773.46 total 90.06 last 66.47) - open/close 6925 (time 1775765783.47 total 100.08 last 72.20) - open/close 7599 (time 1775765793.48 total 110.09 last 67.36) - open/close 8318 (time 1775765803.48 total 120.09 last 71.88) - open/close 9038 (time 1775765813.50 total 130.11 last 71.85) - open/close 9723 (time 1775765823.51 total 140.11 last 68.47) total: 10000 open/close in 144.32 seconds: 69.29 ops/second - unlinked 0 (time 1775765833 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 679 (time 1775765932.54 total 10.04 last 67.65) - open/close 1364 (time 1775765942.55 total 20.04 last 68.49) - open/close 2091 (time 1775765952.55 total 30.04 last 72.66) - open/close 2737 (time 1775765962.56 total 40.05 last 64.55) - open/close 3336 (time 1775765972.57 total 50.06 last 59.84) - open/close 3959 (time 1775765982.59 total 60.08 last 62.19) - open/close 4594 (time 1775765992.59 total 70.08 last 63.47) - open/close 5272 (time 1775766002.59 total 80.09 last 67.78) - open/close 5932 (time 1775766012.61 total 90.10 last 65.90) - open/close 6462 (time 1775766022.61 total 100.10 last 53.00) - open/close 6907 (time 1775766032.61 total 110.11 last 44.48) - open/close 7496 (time 1775766042.63 total 120.12 last 58.82) - open/close 8085 (time 1775766052.64 total 130.13 last 58.85) - open/close 8794 (time 1775766062.64 total 140.13 last 70.87) - open/close 9529 (time 1775766072.65 total 150.15 last 73.41) total: 10000 open/close in 158.66 seconds: 63.03 ops/second - unlinked 0 (time 1775766085 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 602 (time 1775766190.25 total 10.00 last 60.19) - open/close 1255 (time 1775766200.27 total 20.01 last 65.22) - open/close 1910 (time 1775766210.27 total 30.02 last 65.45) - open/close 2518 (time 1775766220.28 total 40.02 last 60.79) - open/close 3168 (time 1775766230.28 total 50.03 last 64.97) - open/close 3825 (time 1775766240.28 total 60.03 last 65.67) - open/close 4530 (time 1775766250.31 total 70.06 last 70.30) - open/close 5100 (time 1775766260.32 total 80.07 last 56.95) - open/close 5640 (time 1775766270.33 total 90.08 last 53.93) - open/close 6265 (time 1775766280.35 total 100.09 last 62.43) - open/close 6875 (time 1775766290.36 total 110.10 last 60.93) - open/close 7559 (time 1775766300.37 total 120.11 last 68.33) - open/close 8269 (time 1775766310.37 total 130.12 last 70.95) - open/close 8883 (time 1775766320.38 total 140.13 last 61.33) - open/close 9508 (time 1775766330.39 total 150.14 last 62.47) total: 10000 open/close in 158.46 seconds: 63.11 ops/second - unlinked 0 (time 1775766344 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg618-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg618-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg618-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg618-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-OST0000 oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg618-server: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg618-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg618-server: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg618-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg618-client.virtnet: -o user_xattr,flock 192.168.206.118@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg618-client: oleg618-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg618-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg618-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg618-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.118@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.118@tcp sys.timeout=20 start ost1 service on oleg618-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-OST0000 oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg618-server: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg618-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg618-server: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg618-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg618-client.virtnet: -o user_xattr,flock 192.168.206.118@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg618-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg618-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg618-server stop mds service on oleg618-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg618-server stop mds service on oleg618-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg618-server unloading modules via unload_modules_local on: 'oleg618-server' oleg618-server: oleg618-server.virtnet: executing unload_modules_local oleg618-server: modules unloaded. pdsh@oleg618-client: oleg618-client: ssh exited with exit code 2 pdsh@oleg618-client: oleg618-server: ssh exited with exit code 2 pdsh@oleg618-client: oleg618-client: ssh exited with exit code 2 pdsh@oleg618-client: oleg618-server: ssh exited with exit code 2