== conf-sanity test 69: replace an OST with the same index ========================================================== 14:37:20 (1776191840) start mds service on oleg308-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 916 (time 1776191937.56 total 10.01 last 91.49) - open/close 1823 (time 1776191947.56 total 20.02 last 90.64) - open/close 2689 (time 1776191957.57 total 30.02 last 86.55) - open/close 3576 (time 1776191967.58 total 40.03 last 88.65) - open/close 4400 (time 1776191977.58 total 50.03 last 82.38) - open/close 5276 (time 1776191987.58 total 60.04 last 87.58) - open/close 6154 (time 1776191997.58 total 70.04 last 87.80) - open/close 6893 (time 1776192007.59 total 80.04 last 73.87) - open/close 7639 (time 1776192017.61 total 90.07 last 74.39) - open/close 8517 (time 1776192027.62 total 100.07 last 87.78) - open/close 9400 (time 1776192037.62 total 110.07 last 88.27) total: 10000 open/close in 119.20 seconds: 83.89 ops/second - unlinked 0 (time 1776192050 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 687 (time 1776192137.98 total 10.01 last 68.66) - open/close 1509 (time 1776192147.98 total 20.01 last 82.19) - open/close 2367 (time 1776192157.99 total 30.02 last 85.71) - open/close 3217 (time 1776192168.00 total 40.03 last 84.93) - open/close 4072 (time 1776192178.00 total 50.03 last 85.48) - open/close 4795 (time 1776192188.00 total 60.03 last 72.28) - open/close 5553 (time 1776192198.01 total 70.04 last 75.76) - open/close 6462 (time 1776192208.01 total 80.04 last 90.86) - open/close 7335 (time 1776192218.02 total 90.05 last 87.25) - open/close 8143 (time 1776192228.02 total 100.05 last 80.77) - open/close 8932 (time 1776192238.03 total 110.06 last 78.84) - open/close 9749 (time 1776192248.03 total 120.06 last 81.67) total: 10000 open/close in 123.58 seconds: 80.92 ops/second - unlinked 0 (time 1776192255 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 706 (time 1776192341.15 total 10.01 last 70.52) - open/close 1207 (time 1776192351.15 total 20.02 last 50.08) - open/close 1943 (time 1776192361.16 total 30.02 last 73.57) - open/close 2686 (time 1776192371.16 total 40.02 last 74.29) - open/close 3455 (time 1776192381.17 total 50.03 last 76.86) - open/close 4308 (time 1776192391.17 total 60.03 last 85.24) - open/close 5161 (time 1776192401.19 total 70.05 last 85.17) - open/close 5861 (time 1776192411.20 total 80.06 last 69.92) - open/close 6510 (time 1776192421.20 total 90.06 last 64.90) - open/close 7269 (time 1776192431.21 total 100.07 last 75.83) - open/close 8123 (time 1776192441.21 total 110.08 last 85.35) - open/close 8985 (time 1776192451.22 total 120.08 last 86.12) - open/close 9838 (time 1776192461.23 total 130.09 last 85.27) total: 10000 open/close in 132.01 seconds: 75.75 ops/second - unlinked 0 (time 1776192466 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 542 (time 1776192552.27 total 10.01 last 54.14) - open/close 1268 (time 1776192562.28 total 20.02 last 72.57) - open/close 2057 (time 1776192572.28 total 30.02 last 78.85) - open/close 2779 (time 1776192582.29 total 40.03 last 72.15) - open/close 3584 (time 1776192592.29 total 50.03 last 80.48) - open/close 4389 (time 1776192602.30 total 60.04 last 80.45) - open/close 5121 (time 1776192612.31 total 70.05 last 73.13) - open/close 5633 (time 1776192622.33 total 80.07 last 51.10) - open/close 6364 (time 1776192632.33 total 90.07 last 73.06) - open/close 7068 (time 1776192642.33 total 100.07 last 70.40) - open/close 7802 (time 1776192652.35 total 110.09 last 73.29) - open/close 8579 (time 1776192662.35 total 120.09 last 77.65) - open/close 9349 (time 1776192672.36 total 130.10 last 76.95) total: 10000 open/close in 139.56 seconds: 71.65 ops/second - unlinked 0 (time 1776192686 ; total 1 ; last 1) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 822 (time 1776192776.07 total 10.02 last 82.06) - open/close 1687 (time 1776192786.08 total 20.03 last 86.43) - open/close 2559 (time 1776192796.08 total 30.03 last 87.14) - open/close 3440 (time 1776192806.10 total 40.05 last 87.96) - open/close 4317 (time 1776192816.10 total 50.05 last 87.69) - open/close 5113 (time 1776192826.11 total 60.06 last 79.54) - open/close 5828 (time 1776192836.11 total 70.06 last 71.48) - open/close 6565 (time 1776192846.13 total 80.07 last 73.59) - open/close 7285 (time 1776192856.13 total 90.08 last 71.97) - open/close 8029 (time 1776192866.14 total 100.09 last 74.35) - open/close 8832 (time 1776192876.15 total 110.09 last 80.23) - open/close 9662 (time 1776192886.15 total 120.10 last 82.98) total: 10000 open/close in 124.12 seconds: 80.57 ops/second - unlinked 0 (time 1776192893 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 778 (time 1776192982.51 total 10.01 last 77.70) - open/close 1579 (time 1776192992.51 total 20.02 last 80.07) - open/close 2266 (time 1776193002.53 total 30.03 last 68.59) - open/close 2942 (time 1776193012.54 total 40.05 last 67.51) - open/close 3636 (time 1776193022.56 total 50.06 last 69.31) - open/close 4288 (time 1776193032.57 total 60.07 last 65.13) - open/close 4920 (time 1776193042.57 total 70.08 last 63.15) - open/close 5676 (time 1776193052.58 total 80.08 last 75.59) - open/close 6479 (time 1776193062.58 total 90.08 last 80.29) - open/close 7316 (time 1776193072.58 total 100.09 last 83.63) - open/close 8126 (time 1776193082.59 total 110.09 last 80.96) - open/close 8952 (time 1776193092.60 total 120.11 last 82.49) - open/close 9663 (time 1776193102.60 total 130.11 last 71.09) total: 10000 open/close in 133.90 seconds: 74.68 ops/second - unlinked 0 (time 1776193110 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 682 (time 1776193206.67 total 10.01 last 68.13) - open/close 1496 (time 1776193216.67 total 20.01 last 81.37) - open/close 2357 (time 1776193226.69 total 30.03 last 85.96) - open/close 3194 (time 1776193236.71 total 40.05 last 83.55) - open/close 3936 (time 1776193246.72 total 50.06 last 74.13) - open/close 4765 (time 1776193256.72 total 60.06 last 82.85) - open/close 5482 (time 1776193266.73 total 70.07 last 71.67) - open/close 6120 (time 1776193276.73 total 80.07 last 63.77) - open/close 6759 (time 1776193286.74 total 90.08 last 63.87) - open/close 7501 (time 1776193296.74 total 100.08 last 74.13) - open/close 8262 (time 1776193306.77 total 110.11 last 75.93) - open/close 8763 (time 1776193316.77 total 120.11 last 50.07) - open/close 9654 (time 1776193326.78 total 130.12 last 89.01) total: 10000 open/close in 134.62 seconds: 74.28 ops/second - unlinked 0 (time 1776193335 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 810 (time 1776193423.46 total 10.01 last 80.93) - open/close 1460 (time 1776193433.47 total 20.01 last 64.97) - open/close 2189 (time 1776193443.49 total 30.03 last 72.77) - open/close 2964 (time 1776193453.49 total 40.03 last 77.48) - open/close 3622 (time 1776193463.50 total 50.04 last 65.73) - open/close 4421 (time 1776193473.51 total 60.05 last 79.86) - open/close 5232 (time 1776193483.51 total 70.05 last 81.08) - open/close 6098 (time 1776193493.51 total 80.06 last 86.54) - open/close 6899 (time 1776193503.52 total 90.06 last 80.06) - open/close 7726 (time 1776193513.52 total 100.07 last 82.67) - open/close 8557 (time 1776193523.52 total 110.07 last 83.09) - open/close 9393 (time 1776193533.53 total 120.07 last 83.56) total: 10000 open/close in 127.62 seconds: 78.36 ops/second - unlinked 0 (time 1776193545 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 840 (time 1776193629.79 total 10.00 last 83.97) - open/close 1705 (time 1776193639.80 total 20.01 last 86.48) - open/close 2510 (time 1776193649.80 total 30.01 last 80.45) - open/close 3240 (time 1776193659.81 total 40.02 last 72.94) - open/close 3948 (time 1776193669.81 total 50.02 last 70.79) - open/close 4817 (time 1776193679.82 total 60.03 last 86.83) - open/close 5556 (time 1776193689.84 total 70.05 last 73.77) - open/close 6369 (time 1776193699.85 total 80.06 last 81.21) - open/close 7108 (time 1776193709.85 total 90.06 last 73.89) - open/close 7616 (time 1776193719.86 total 100.07 last 50.74) - open/close 8461 (time 1776193729.87 total 110.08 last 84.47) - open/close 9384 (time 1776193739.87 total 120.08 last 92.23) total: 10000 open/close in 127.50 seconds: 78.43 ops/second - unlinked 0 (time 1776193751 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 778 (time 1776193835.01 total 10.01 last 77.75) - open/close 1661 (time 1776193845.01 total 20.01 last 88.25) - open/close 2512 (time 1776193855.03 total 30.03 last 84.97) - open/close 3298 (time 1776193865.04 total 40.04 last 78.49) - open/close 4131 (time 1776193875.05 total 50.05 last 83.24) - open/close 5010 (time 1776193885.05 total 60.05 last 87.89) - open/close 5852 (time 1776193895.06 total 70.05 last 84.18) - open/close 6683 (time 1776193905.06 total 80.06 last 83.07) - open/close 7592 (time 1776193915.07 total 90.06 last 90.83) - open/close 8483 (time 1776193925.07 total 100.07 last 89.06) - open/close 9377 (time 1776193935.08 total 110.07 last 89.35) total: 10000 open/close in 117.45 seconds: 85.14 ops/second - unlinked 0 (time 1776193945 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg308-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg308-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg308-server: LNET ready to unload oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2