== conf-sanity test 69: replace an OST with the same index ========================================================== 09:53:32 (1761314012) start mds service on oleg440-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 786 (time 1761314090.42 total 10.00 last 78.60) - open/close 1594 (time 1761314100.44 total 20.02 last 80.66) - open/close 2455 (time 1761314110.44 total 30.02 last 86.09) - open/close 3224 (time 1761314120.45 total 40.03 last 76.81) - open/close 3966 (time 1761314130.46 total 50.04 last 74.14) - open/close 4712 (time 1761314140.46 total 60.04 last 74.58) - open/close 5505 (time 1761314150.47 total 70.05 last 79.22) - open/close 6281 (time 1761314160.49 total 80.07 last 77.48) - open/close 7083 (time 1761314170.49 total 90.07 last 80.16) - open/close 7830 (time 1761314180.50 total 100.08 last 74.63) - open/close 8584 (time 1761314190.51 total 110.09 last 75.33) - open/close 9389 (time 1761314200.52 total 120.10 last 80.46) total: 10000 open/close in 127.48 seconds: 78.45 ops/second - unlinked 0 (time 1761314211 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 945 (time 1761314289.53 total 10.00 last 94.49) - open/close 1772 (time 1761314299.54 total 20.01 last 82.61) - open/close 2561 (time 1761314309.55 total 30.02 last 78.87) - open/close 3376 (time 1761314319.55 total 40.02 last 81.45) - open/close 4325 (time 1761314329.56 total 50.03 last 94.86) - open/close 5254 (time 1761314339.56 total 60.03 last 92.90) - open/close 6134 (time 1761314349.57 total 70.04 last 87.92) - open/close 6952 (time 1761314359.57 total 80.04 last 81.73) - open/close 7793 (time 1761314369.58 total 90.05 last 84.04) - open/close 8739 (time 1761314379.58 total 100.05 last 94.58) - open/close 9613 (time 1761314389.59 total 110.06 last 87.36) total: 10000 open/close in 114.81 seconds: 87.10 ops/second - unlinked 0 (time 1761314397 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 800 (time 1761314476.12 total 10.01 last 79.90) - open/close 1684 (time 1761314486.12 total 20.02 last 88.37) - open/close 2486 (time 1761314496.13 total 30.02 last 80.15) - open/close 3387 (time 1761314506.13 total 40.03 last 90.07) - open/close 4194 (time 1761314516.14 total 50.03 last 80.63) - open/close 4971 (time 1761314526.14 total 60.04 last 77.67) - open/close 5822 (time 1761314536.15 total 70.05 last 85.03) - open/close 6641 (time 1761314546.16 total 80.06 last 81.82) - open/close 7376 (time 1761314556.18 total 90.07 last 73.37) - open/close 8084 (time 1761314566.18 total 100.08 last 70.77) - open/close 8805 (time 1761314576.19 total 110.08 last 72.05) - open/close 9520 (time 1761314586.19 total 120.09 last 71.48) total: 10000 open/close in 125.84 seconds: 79.47 ops/second - unlinked 0 (time 1761314595 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 866 (time 1761314680.32 total 10.01 last 86.55) - open/close 1799 (time 1761314690.33 total 20.01 last 93.24) - open/close 2603 (time 1761314700.37 total 30.05 last 80.11) - open/close 3320 (time 1761314710.37 total 40.05 last 71.70) - open/close 4060 (time 1761314720.38 total 50.06 last 73.92) - open/close 4822 (time 1761314730.38 total 60.07 last 76.14) - open/close 5556 (time 1761314740.39 total 70.07 last 73.37) - open/close 6281 (time 1761314750.39 total 80.07 last 72.48) - open/close 7047 (time 1761314760.41 total 90.09 last 76.48) - open/close 7806 (time 1761314770.42 total 100.10 last 75.84) - open/close 8584 (time 1761314780.43 total 110.12 last 77.67) - open/close 9357 (time 1761314790.44 total 120.13 last 77.22) total: 10000 open/close in 128.25 seconds: 77.97 ops/second - unlinked 0 (time 1761314801 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 847 (time 1761314890.45 total 10.01 last 84.65) - open/close 1625 (time 1761314900.46 total 20.01 last 77.76) - open/close 2378 (time 1761314910.46 total 30.01 last 75.28) - open/close 3143 (time 1761314920.47 total 40.02 last 76.45) - open/close 3822 (time 1761314930.49 total 50.04 last 67.74) - open/close 4543 (time 1761314940.49 total 60.04 last 72.10) - open/close 5260 (time 1761314950.49 total 70.04 last 71.69) - open/close 6044 (time 1761314960.50 total 80.05 last 78.32) - open/close 6899 (time 1761314970.50 total 90.06 last 85.49) - open/close 7655 (time 1761314980.51 total 100.06 last 75.54) - open/close 8301 (time 1761314990.51 total 110.07 last 64.57) - open/close 8912 (time 1761315000.52 total 120.07 last 61.09) - open/close 9512 (time 1761315010.52 total 130.07 last 60.00) total: 10000 open/close in 137.12 seconds: 72.93 ops/second - unlinked 0 (time 1761315021 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 786 (time 1761315113.15 total 10.01 last 78.54) - open/close 1575 (time 1761315123.15 total 20.01 last 78.89) - open/close 2382 (time 1761315133.17 total 30.03 last 80.56) - open/close 3078 (time 1761315143.17 total 40.03 last 69.56) - open/close 3821 (time 1761315153.19 total 50.04 last 74.22) - open/close 4425 (time 1761315163.19 total 60.05 last 60.38) - open/close 5121 (time 1761315173.19 total 70.05 last 69.59) - open/close 5865 (time 1761315183.20 total 80.06 last 74.30) - open/close 6605 (time 1761315193.21 total 90.07 last 73.94) - open/close 7392 (time 1761315203.21 total 100.07 last 78.68) - open/close 8098 (time 1761315213.22 total 110.08 last 70.57) - open/close 8856 (time 1761315223.23 total 120.09 last 75.73) - open/close 9588 (time 1761315233.24 total 130.10 last 73.13) total: 10000 open/close in 135.75 seconds: 73.66 ops/second - unlinked 0 (time 1761315241 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 680 (time 1761315331.64 total 10.01 last 67.91) - open/close 1381 (time 1761315341.65 total 20.02 last 70.05) - open/close 2125 (time 1761315351.66 total 30.03 last 74.30) - open/close 2870 (time 1761315361.66 total 40.04 last 74.49) - open/close 3579 (time 1761315371.67 total 50.04 last 70.86) - open/close 4315 (time 1761315381.69 total 60.06 last 73.45) - open/close 4984 (time 1761315391.69 total 70.07 last 66.87) - open/close 5626 (time 1761315401.70 total 80.07 last 64.17) - open/close 6306 (time 1761315411.72 total 90.09 last 67.87) - open/close 6978 (time 1761315421.72 total 100.09 last 67.19) - open/close 7607 (time 1761315431.72 total 110.09 last 62.89) - open/close 8224 (time 1761315441.74 total 120.11 last 61.56) - open/close 8854 (time 1761315451.74 total 130.12 last 62.99) - open/close 9512 (time 1761315461.76 total 140.13 last 65.72) total: 10000 open/close in 148.02 seconds: 67.56 ops/second - unlinked 0 (time 1761315474 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 682 (time 1761315575.94 total 10.02 last 68.08) - open/close 1353 (time 1761315585.95 total 20.03 last 67.03) - open/close 2048 (time 1761315595.96 total 30.04 last 69.45) - open/close 2714 (time 1761315605.98 total 40.06 last 66.46) - open/close 3439 (time 1761315615.99 total 50.07 last 72.38) - open/close 4182 (time 1761315626.00 total 60.08 last 74.27) - open/close 4905 (time 1761315636.01 total 70.09 last 72.21) - open/close 5637 (time 1761315646.02 total 80.10 last 73.14) - open/close 6354 (time 1761315656.03 total 90.11 last 71.65) - open/close 7070 (time 1761315666.04 total 100.12 last 71.52) - open/close 7726 (time 1761315676.04 total 110.12 last 65.57) - open/close 8442 (time 1761315686.04 total 120.12 last 71.59) - open/close 9094 (time 1761315696.05 total 130.13 last 65.15) - open/close 9748 (time 1761315706.05 total 140.13 last 65.38) total: 10000 open/close in 144.06 seconds: 69.42 ops/second - unlinked 0 (time 1761315714 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 749 (time 1761315813.48 total 10.01 last 74.81) - open/close 1497 (time 1761315823.49 total 20.02 last 74.74) - open/close 2219 (time 1761315833.49 total 30.02 last 72.17) - open/close 2855 (time 1761315843.51 total 40.04 last 63.50) - open/close 3571 (time 1761315853.51 total 50.05 last 71.56) - open/close 4285 (time 1761315863.52 total 60.05 last 71.36) - open/close 5051 (time 1761315873.53 total 70.06 last 76.52) - open/close 5736 (time 1761315883.53 total 80.06 last 68.49) - open/close 6346 (time 1761315893.54 total 90.07 last 60.94) - open/close 6896 (time 1761315903.56 total 100.09 last 54.91) - open/close 7444 (time 1761315913.56 total 110.10 last 54.77) - open/close 8113 (time 1761315923.57 total 120.11 last 66.83) - open/close 8728 (time 1761315933.58 total 130.11 last 61.46) - open/close 9409 (time 1761315943.58 total 140.12 last 68.08) total: 10000 open/close in 148.91 seconds: 67.15 ops/second - unlinked 0 (time 1761315956 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 694 (time 1761316050.02 total 10.00 last 69.37) - open/close 1442 (time 1761316060.03 total 20.01 last 74.74) - open/close 2198 (time 1761316070.03 total 30.01 last 75.60) - open/close 2977 (time 1761316080.03 total 40.02 last 77.87) - open/close 3743 (time 1761316090.04 total 50.02 last 76.54) - open/close 4481 (time 1761316100.04 total 60.03 last 73.77) - open/close 5213 (time 1761316110.05 total 70.04 last 73.12) - open/close 5955 (time 1761316120.06 total 80.04 last 74.17) - open/close 6690 (time 1761316130.07 total 90.05 last 73.43) - open/close 7389 (time 1761316140.07 total 100.06 last 69.85) - open/close 8099 (time 1761316150.08 total 110.07 last 70.95) - open/close 8751 (time 1761316160.09 total 120.07 last 65.16) - open/close 9472 (time 1761316170.10 total 130.09 last 72.01) total: 10000 open/close in 137.74 seconds: 72.60 ops/second - unlinked 0 (time 1761316181 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg440-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg440-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg440-server: LNET ready to unload oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2