== conf-sanity test 69: replace an OST with the same index ========================================================== 12:42:35 (1786812155) start mds service on oleg212-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing load_modules_local oleg212-server: Loading modules from /home/green/git/lustre-release/lustre oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg212-server: detected 4 online CPUs by sysfs oleg212-server: MODOPTS_LIBCFS= oleg212-server: Force libcfs to create 2 CPU partitions oleg212-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg212-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg212-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg212-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg212-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 703 (time 1786812270.49 total 10.00 last 70.29) - open/close 1429 (time 1786812280.50 total 20.01 last 72.56) - open/close 2233 (time 1786812290.54 total 30.05 last 80.09) - open/close 2975 (time 1786812300.54 total 40.05 last 74.20) - open/close 3855 (time 1786812310.54 total 50.05 last 87.97) - open/close 4584 (time 1786812320.56 total 60.06 last 72.79) - open/close 5351 (time 1786812330.56 total 70.07 last 76.66) - open/close 6294 (time 1786812340.56 total 80.07 last 94.29) - open/close 7276 (time 1786812350.57 total 90.08 last 98.14) - open/close 8118 (time 1786812360.58 total 100.09 last 84.10) - open/close 8934 (time 1786812370.59 total 110.10 last 81.52) - open/close 9919 (time 1786812380.60 total 120.11 last 98.39) total: 10000 open/close in 121.51 seconds: 82.30 ops/second - unlinked 0 (time 1786812387 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 760 (time 1786812469.19 total 10.01 last 75.93) - open/close 1583 (time 1786812479.20 total 20.01 last 82.26) - open/close 2496 (time 1786812489.20 total 30.01 last 91.30) - open/close 3284 (time 1786812499.21 total 40.03 last 78.70) - open/close 3894 (time 1786812509.22 total 50.03 last 60.97) - open/close 4759 (time 1786812519.22 total 60.04 last 86.44) - open/close 5587 (time 1786812529.22 total 70.04 last 82.80) - open/close 6553 (time 1786812539.23 total 80.05 last 96.55) - open/close 7671 (time 1786812549.24 total 90.05 last 111.70) - open/close 8614 (time 1786812559.24 total 100.05 last 94.30) - open/close 9369 (time 1786812569.24 total 110.06 last 75.45) total: 10000 open/close in 117.34 seconds: 85.22 ops/second - unlinked 0 (time 1786812580 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 792 (time 1786812669.60 total 10.02 last 79.07) - open/close 1641 (time 1786812679.60 total 20.02 last 84.90) - open/close 2453 (time 1786812689.60 total 30.02 last 81.18) - open/close 3437 (time 1786812699.60 total 40.02 last 98.35) - open/close 4444 (time 1786812709.61 total 50.03 last 100.66) - open/close 5384 (time 1786812719.62 total 60.04 last 93.89) - open/close 6479 (time 1786812729.62 total 70.04 last 109.47) - open/close 7409 (time 1786812739.62 total 80.04 last 92.99) - open/close 8241 (time 1786812749.63 total 90.05 last 83.13) - open/close 9032 (time 1786812759.64 total 100.06 last 79.04) - open/close 9862 (time 1786812769.65 total 110.07 last 82.89) total: 10000 open/close in 111.39 seconds: 89.77 ops/second - unlinked 0 (time 1786812775 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 964 (time 1786812849.53 total 10.00 last 96.38) - open/close 1712 (time 1786812859.54 total 20.01 last 74.75) - open/close 2564 (time 1786812869.54 total 30.01 last 85.18) - open/close 3417 (time 1786812879.55 total 40.02 last 85.26) - open/close 4315 (time 1786812889.55 total 50.02 last 89.79) - open/close 5196 (time 1786812899.55 total 60.02 last 88.08) - open/close 5950 (time 1786812909.55 total 70.02 last 75.39) - open/close 6674 (time 1786812919.56 total 80.03 last 72.36) - open/close 7627 (time 1786812929.57 total 90.04 last 95.15) - open/close 8353 (time 1786812939.59 total 100.06 last 72.45) - open/close 9004 (time 1786812949.61 total 110.08 last 64.99) - open/close 9682 (time 1786812959.62 total 120.08 last 67.77) total: 10000 open/close in 124.36 seconds: 80.41 ops/second - unlinked 0 (time 1786812969 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 777 (time 1786813057.48 total 10.01 last 77.66) - open/close 1733 (time 1786813067.49 total 20.01 last 95.58) - open/close 2492 (time 1786813077.49 total 30.01 last 75.89) - open/close 3231 (time 1786813087.49 total 40.01 last 73.87) - open/close 4134 (time 1786813097.50 total 50.02 last 90.22) - open/close 4926 (time 1786813107.50 total 60.02 last 79.20) - open/close 5804 (time 1786813117.52 total 70.04 last 87.62) - open/close 6651 (time 1786813127.52 total 80.05 last 84.67) - open/close 7468 (time 1786813137.54 total 90.06 last 81.61) - open/close 8227 (time 1786813147.54 total 100.06 last 75.85) - open/close 9153 (time 1786813157.54 total 110.07 last 92.58) total: 10000 open/close in 118.43 seconds: 84.44 ops/second - unlinked 0 (time 1786813170 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 514 (time 1786813253.86 total 10.01 last 51.33) - open/close 1281 (time 1786813263.88 total 20.02 last 76.61) - open/close 2066 (time 1786813273.92 total 30.07 last 78.15) - open/close 2775 (time 1786813283.93 total 40.08 last 70.85) - open/close 3537 (time 1786813293.93 total 50.08 last 76.18) - open/close 4298 (time 1786813303.94 total 60.09 last 76.04) - open/close 5257 (time 1786813313.95 total 70.10 last 95.75) - open/close 6049 (time 1786813323.96 total 80.11 last 79.15) - open/close 6707 (time 1786813333.97 total 90.12 last 65.72) - open/close 7566 (time 1786813343.99 total 100.14 last 85.76) - open/close 8463 (time 1786813353.99 total 110.14 last 89.64) - open/close 9266 (time 1786813364.00 total 120.15 last 80.28) - open/close 9849 (time 1786813374.00 total 130.15 last 58.29) total: 10000 open/close in 133.87 seconds: 74.70 ops/second - unlinked 0 (time 1786813385 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 913 (time 1786813472.64 total 10.01 last 91.25) - open/close 1829 (time 1786813482.64 total 20.01 last 91.57) - open/close 2632 (time 1786813492.65 total 30.02 last 80.22) - open/close 3435 (time 1786813502.65 total 40.02 last 80.29) - open/close 4237 (time 1786813512.67 total 50.04 last 80.07) - open/close 5025 (time 1786813522.68 total 60.05 last 78.73) - open/close 5955 (time 1786813532.68 total 70.05 last 92.97) - open/close 6951 (time 1786813542.68 total 80.05 last 99.58) - open/close 7782 (time 1786813552.69 total 90.06 last 83.06) - open/close 8507 (time 1786813562.69 total 100.06 last 72.50) - open/close 9145 (time 1786813572.70 total 110.06 last 63.75) total: 10000 open/close in 119.17 seconds: 83.91 ops/second - unlinked 0 (time 1786813586 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 939 (time 1786813669.98 total 10.01 last 93.82) - open/close 1892 (time 1786813679.98 total 20.01 last 95.25) - open/close 2912 (time 1786813689.99 total 30.02 last 101.90) - open/close 3905 (time 1786813699.99 total 40.03 last 99.27) - open/close 4950 (time 1786813710.00 total 50.03 last 104.44) - open/close 5973 (time 1786813720.00 total 60.04 last 102.28) - open/close 6978 (time 1786813730.01 total 70.04 last 100.41) - open/close 7928 (time 1786813740.02 total 80.05 last 94.94) - open/close 8840 (time 1786813750.02 total 90.05 last 91.16) - open/close 9459 (time 1786813760.04 total 100.07 last 61.80) total: 10000 open/close in 108.46 seconds: 92.20 ops/second - unlinked 0 (time 1786813774 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1075 (time 1786813848.67 total 10.01 last 107.41) - open/close 2135 (time 1786813858.68 total 20.01 last 105.98) - open/close 3140 (time 1786813868.69 total 30.03 last 100.35) - open/close 4199 (time 1786813878.69 total 40.03 last 105.90) - open/close 5317 (time 1786813888.69 total 50.03 last 111.77) - open/close 6393 (time 1786813898.73 total 60.06 last 107.24) - open/close 7275 (time 1786813908.73 total 70.07 last 88.16) - open/close 8183 (time 1786813918.74 total 80.08 last 90.69) - open/close 9141 (time 1786813928.75 total 90.08 last 95.75) total: 10000 open/close in 98.98 seconds: 101.03 ops/second - unlinked 0 (time 1786813941 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1048 (time 1786814010.87 total 10.00 last 104.78) - open/close 2028 (time 1786814020.87 total 20.01 last 97.92) - open/close 2776 (time 1786814030.87 total 30.01 last 74.80) - open/close 3478 (time 1786814040.89 total 40.02 last 70.11) - open/close 4241 (time 1786814050.89 total 50.03 last 76.26) - open/close 5138 (time 1786814060.90 total 60.03 last 89.68) - open/close 5978 (time 1786814070.90 total 70.03 last 83.99) - open/close 6717 (time 1786814080.90 total 80.04 last 73.85) - open/close 7643 (time 1786814090.93 total 90.06 last 92.40) - open/close 8708 (time 1786814100.93 total 100.07 last 106.44) - open/close 9691 (time 1786814110.94 total 110.07 last 98.27) total: 10000 open/close in 113.29 seconds: 88.27 ops/second - unlinked 0 (time 1786814119 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.112@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg212-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg212-server: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg212-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg212-server: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg212-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.112@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.112@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.112@tcp sys.timeout=20 start ost1 service on oleg212-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg212-server: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg212-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg212-server: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg212-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg212-server unloading modules via unload_modules_local on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing unload_modules_local oleg212-server: modules unloaded. pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2