== conf-sanity test 69: replace an OST with the same index ========================================================== 19:29:32 (1763339372) start mds service on oleg235-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions oleg235-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg235-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg235-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 697 (time 1763339472.08 total 10.01 last 69.64) - open/close 1402 (time 1763339482.10 total 20.03 last 70.38) - open/close 2036 (time 1763339492.12 total 30.05 last 63.25) - open/close 2761 (time 1763339502.13 total 40.05 last 72.48) - open/close 3505 (time 1763339512.14 total 50.06 last 74.32) - open/close 4289 (time 1763339522.15 total 60.07 last 78.35) - open/close 4981 (time 1763339532.16 total 70.09 last 69.08) - open/close 5805 (time 1763339542.18 total 80.10 last 82.30) - open/close 6617 (time 1763339552.18 total 90.11 last 81.15) - open/close 7362 (time 1763339562.19 total 100.12 last 74.40) - open/close 8084 (time 1763339572.20 total 110.13 last 72.15) - open/close 8853 (time 1763339582.21 total 120.13 last 76.84) - open/close 9624 (time 1763339592.22 total 130.14 last 77.03) total: 10000 open/close in 135.06 seconds: 74.04 ops/second - unlinked 0 (time 1763339601 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 776 (time 1763339692.74 total 10.00 last 77.57) - open/close 1527 (time 1763339702.74 total 20.01 last 75.05) - open/close 2304 (time 1763339712.74 total 30.01 last 77.69) - open/close 3017 (time 1763339722.75 total 40.02 last 71.23) - open/close 3727 (time 1763339732.77 total 50.04 last 70.91) - open/close 4454 (time 1763339742.77 total 60.04 last 72.67) - open/close 5031 (time 1763339752.78 total 70.05 last 57.65) - open/close 5723 (time 1763339762.79 total 80.05 last 69.15) - open/close 6384 (time 1763339772.80 total 90.06 last 66.04) - open/close 7104 (time 1763339782.81 total 100.07 last 71.93) - open/close 7808 (time 1763339792.82 total 110.09 last 70.29) - open/close 8526 (time 1763339802.83 total 120.10 last 71.74) - open/close 9081 (time 1763339812.83 total 130.10 last 55.49) - open/close 9679 (time 1763339822.85 total 140.12 last 59.68) total: 10000 open/close in 144.88 seconds: 69.02 ops/second - unlinked 0 (time 1763339831 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 615 (time 1763339923.75 total 10.00 last 61.47) - open/close 1124 (time 1763339933.76 total 20.02 last 50.85) - open/close 1774 (time 1763339943.79 total 30.05 last 64.81) - open/close 2437 (time 1763339953.81 total 40.06 last 66.22) - open/close 3031 (time 1763339963.81 total 50.06 last 59.40) - open/close 3618 (time 1763339973.81 total 60.06 last 58.68) - open/close 4192 (time 1763339983.81 total 70.07 last 57.38) - open/close 4859 (time 1763339993.82 total 80.07 last 66.65) - open/close 5618 (time 1763340003.83 total 90.08 last 75.86) - open/close 6362 (time 1763340013.85 total 100.10 last 74.25) - open/close 7169 (time 1763340023.86 total 110.11 last 80.55) - open/close 7929 (time 1763340033.87 total 120.12 last 75.99) - open/close 8707 (time 1763340043.87 total 130.12 last 77.78) - open/close 9427 (time 1763340053.88 total 140.13 last 71.92) total: 10000 open/close in 148.70 seconds: 67.25 ops/second - unlinked 0 (time 1763340065 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 707 (time 1763340153.91 total 10.01 last 70.62) - open/close 1444 (time 1763340163.92 total 20.02 last 73.64) - open/close 2184 (time 1763340173.94 total 30.04 last 73.84) - open/close 2830 (time 1763340183.95 total 40.05 last 64.55) - open/close 3557 (time 1763340193.96 total 50.05 last 72.66) - open/close 4236 (time 1763340203.96 total 60.06 last 67.88) - open/close 4993 (time 1763340213.97 total 70.07 last 75.59) - open/close 5645 (time 1763340223.98 total 80.08 last 65.13) - open/close 6364 (time 1763340233.98 total 90.08 last 71.89) - open/close 7124 (time 1763340243.99 total 100.09 last 75.97) - open/close 7920 (time 1763340254.00 total 110.10 last 79.53) - open/close 8702 (time 1763340264.01 total 120.11 last 78.10) - open/close 9307 (time 1763340274.01 total 130.11 last 60.48) total: 10000 open/close in 139.93 seconds: 71.46 ops/second - unlinked 0 (time 1763340288 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 766 (time 1763340380.85 total 10.01 last 76.52) - open/close 1552 (time 1763340390.85 total 20.01 last 78.56) - open/close 2364 (time 1763340400.86 total 30.02 last 81.16) - open/close 3172 (time 1763340410.86 total 40.02 last 80.79) - open/close 3853 (time 1763340420.86 total 50.02 last 68.08) - open/close 4529 (time 1763340430.88 total 60.04 last 67.51) - open/close 5256 (time 1763340440.88 total 70.04 last 72.65) - open/close 5912 (time 1763340450.89 total 80.05 last 65.57) - open/close 6562 (time 1763340460.89 total 90.06 last 64.96) - open/close 7206 (time 1763340470.90 total 100.06 last 64.36) - open/close 7852 (time 1763340480.90 total 110.06 last 64.59) - open/close 8599 (time 1763340490.91 total 120.07 last 74.66) - open/close 9357 (time 1763340500.92 total 130.08 last 75.71) total: 10000 open/close in 139.07 seconds: 71.91 ops/second - unlinked 0 (time 1763340514 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 767 (time 1763340606.41 total 10.01 last 76.59) - open/close 1603 (time 1763340616.42 total 20.02 last 83.56) - open/close 2395 (time 1763340626.44 total 30.04 last 79.03) - open/close 3135 (time 1763340636.44 total 40.04 last 73.99) - open/close 3839 (time 1763340646.46 total 50.06 last 70.25) - open/close 4580 (time 1763340656.47 total 60.07 last 74.03) - open/close 5304 (time 1763340666.48 total 70.08 last 72.33) - open/close 6078 (time 1763340676.49 total 80.10 last 77.29) - open/close 6839 (time 1763340686.49 total 90.10 last 76.09) - open/close 7623 (time 1763340696.51 total 100.11 last 78.28) - open/close 8432 (time 1763340706.51 total 110.12 last 80.88) - open/close 9226 (time 1763340716.53 total 120.13 last 79.25) - open/close 9995 (time 1763340726.53 total 130.14 last 76.88) total: 10000 open/close in 130.18 seconds: 76.82 ops/second - unlinked 0 (time 1763340729 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 707 (time 1763340822.17 total 10.01 last 70.66) - open/close 1445 (time 1763340832.19 total 20.02 last 73.67) - open/close 2230 (time 1763340842.20 total 30.03 last 78.47) - open/close 2972 (time 1763340852.20 total 40.03 last 74.18) - open/close 3732 (time 1763340862.20 total 50.04 last 75.95) - open/close 4464 (time 1763340872.21 total 60.04 last 73.14) - open/close 5186 (time 1763340882.22 total 70.05 last 72.15) - open/close 5869 (time 1763340892.22 total 80.06 last 68.27) - open/close 6568 (time 1763340902.23 total 90.07 last 69.83) - open/close 7312 (time 1763340912.24 total 100.07 last 74.38) - open/close 8003 (time 1763340922.25 total 110.08 last 69.00) - open/close 8644 (time 1763340932.27 total 120.10 last 63.97) - open/close 9396 (time 1763340942.27 total 130.11 last 75.19) total: 10000 open/close in 139.27 seconds: 71.80 ops/second - unlinked 0 (time 1763340955 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 715 (time 1763341044.37 total 10.01 last 71.46) - open/close 1430 (time 1763341054.38 total 20.01 last 71.44) - open/close 2137 (time 1763341064.38 total 30.02 last 70.64) - open/close 2884 (time 1763341074.38 total 40.02 last 74.70) - open/close 3630 (time 1763341084.39 total 50.02 last 74.59) - open/close 4367 (time 1763341094.40 total 60.03 last 73.62) - open/close 5111 (time 1763341104.41 total 70.05 last 74.31) - open/close 5853 (time 1763341114.41 total 80.05 last 74.16) - open/close 6568 (time 1763341124.43 total 90.07 last 71.40) - open/close 7298 (time 1763341134.43 total 100.07 last 72.95) - open/close 8030 (time 1763341144.44 total 110.08 last 73.13) - open/close 8738 (time 1763341154.44 total 120.08 last 70.79) - open/close 9525 (time 1763341164.45 total 130.08 last 78.69) total: 10000 open/close in 136.58 seconds: 73.22 ops/second - unlinked 0 (time 1763341173 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 724 (time 1763341265.46 total 10.00 last 72.39) - open/close 1444 (time 1763341275.47 total 20.01 last 71.95) - open/close 2230 (time 1763341285.47 total 30.01 last 78.55) - open/close 2957 (time 1763341295.49 total 40.03 last 72.56) - open/close 3664 (time 1763341305.49 total 50.03 last 70.70) - open/close 4324 (time 1763341315.50 total 60.04 last 65.96) - open/close 5022 (time 1763341325.51 total 70.05 last 69.74) - open/close 5749 (time 1763341335.51 total 80.05 last 72.70) - open/close 6493 (time 1763341345.52 total 90.06 last 74.29) - open/close 7162 (time 1763341355.52 total 100.07 last 66.89) - open/close 7891 (time 1763341365.54 total 110.08 last 72.81) - open/close 8607 (time 1763341375.54 total 120.08 last 71.57) - open/close 9315 (time 1763341385.54 total 130.09 last 70.76) total: 10000 open/close in 139.57 seconds: 71.65 ops/second - unlinked 0 (time 1763341398 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 785 (time 1763341491.11 total 10.01 last 78.41) - open/close 1587 (time 1763341501.12 total 20.02 last 80.15) - open/close 2312 (time 1763341511.13 total 30.03 last 72.43) - open/close 3049 (time 1763341521.13 total 40.04 last 73.65) - open/close 3900 (time 1763341531.14 total 50.04 last 85.04) - open/close 4651 (time 1763341541.14 total 60.04 last 75.10) - open/close 5395 (time 1763341551.16 total 70.06 last 74.28) - open/close 6157 (time 1763341561.16 total 80.06 last 76.19) - open/close 6950 (time 1763341571.17 total 90.07 last 79.23) - open/close 7765 (time 1763341581.17 total 100.07 last 81.48) - open/close 8489 (time 1763341591.17 total 110.07 last 72.38) - open/close 9218 (time 1763341601.18 total 120.08 last 72.83) - open/close 9907 (time 1763341611.19 total 130.10 last 68.80) total: 10000 open/close in 131.51 seconds: 76.04 ops/second - unlinked 0 (time 1763341616 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg235-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing unload_modules_local oleg235-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg235-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2