== conf-sanity test 69: replace an OST with the same index ========================================================== 09:20:01 (1761312001) start mds service on oleg348-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg348-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg348-server: detected 4 online CPUs by sysfs oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg348-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg348-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg348-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 765 (time 1761312097.26 total 10.00 last 76.49) - open/close 1553 (time 1761312107.27 total 20.01 last 78.73) - open/close 2269 (time 1761312117.27 total 30.01 last 71.59) - open/close 3023 (time 1761312127.28 total 40.02 last 75.34) - open/close 3783 (time 1761312137.28 total 50.02 last 75.96) - open/close 4583 (time 1761312147.30 total 60.04 last 79.85) - open/close 5368 (time 1761312157.30 total 70.05 last 78.47) - open/close 6184 (time 1761312167.32 total 80.07 last 81.45) - open/close 6957 (time 1761312177.32 total 90.07 last 77.28) - open/close 7737 (time 1761312187.33 total 100.08 last 77.93) - open/close 8539 (time 1761312197.33 total 110.08 last 80.20) - open/close 9272 (time 1761312207.34 total 120.08 last 73.28) total: 10000 open/close in 129.31 seconds: 77.33 ops/second - unlinked 0 (time 1761312219 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 806 (time 1761312307.66 total 10.00 last 80.57) - open/close 1633 (time 1761312317.68 total 20.02 last 82.54) - open/close 2409 (time 1761312327.69 total 30.03 last 77.58) - open/close 3209 (time 1761312337.69 total 40.03 last 79.99) - open/close 3961 (time 1761312347.71 total 50.05 last 75.05) - open/close 4631 (time 1761312357.72 total 60.06 last 66.93) - open/close 5344 (time 1761312367.73 total 70.07 last 71.18) - open/close 6110 (time 1761312377.74 total 80.08 last 76.57) - open/close 6953 (time 1761312387.74 total 90.09 last 84.25) - open/close 7764 (time 1761312397.75 total 100.09 last 81.08) - open/close 8662 (time 1761312407.75 total 110.09 last 89.76) - open/close 9609 (time 1761312417.76 total 120.10 last 94.59) total: 10000 open/close in 124.20 seconds: 80.52 ops/second - unlinked 0 (time 1761312424 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 851 (time 1761312500.37 total 10.01 last 85.04) - open/close 1600 (time 1761312510.38 total 20.02 last 74.83) - open/close 2466 (time 1761312520.38 total 30.02 last 86.60) - open/close 3364 (time 1761312530.38 total 40.02 last 89.76) - open/close 4185 (time 1761312540.40 total 50.04 last 81.98) - open/close 5039 (time 1761312550.40 total 60.04 last 85.35) - open/close 5792 (time 1761312560.42 total 70.05 last 75.21) - open/close 6626 (time 1761312570.42 total 80.06 last 83.36) - open/close 7423 (time 1761312580.43 total 90.07 last 79.65) - open/close 8125 (time 1761312590.43 total 100.07 last 70.16) - open/close 8933 (time 1761312600.44 total 110.08 last 80.71) - open/close 9740 (time 1761312610.45 total 120.09 last 80.66) total: 10000 open/close in 123.14 seconds: 81.21 ops/second - unlinked 0 (time 1761312616 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 867 (time 1761312697.14 total 10.00 last 86.67) - open/close 1664 (time 1761312707.14 total 20.01 last 79.66) - open/close 2473 (time 1761312717.15 total 30.01 last 80.86) - open/close 3308 (time 1761312727.15 total 40.01 last 83.50) - open/close 4151 (time 1761312737.15 total 50.02 last 84.24) - open/close 4959 (time 1761312747.19 total 60.06 last 80.50) - open/close 5790 (time 1761312757.19 total 70.06 last 83.09) - open/close 6750 (time 1761312767.20 total 80.07 last 95.89) - open/close 7614 (time 1761312777.20 total 90.07 last 86.40) - open/close 8579 (time 1761312787.21 total 100.07 last 96.48) - open/close 9427 (time 1761312797.22 total 110.08 last 84.72) total: 10000 open/close in 117.01 seconds: 85.46 ops/second - unlinked 0 (time 1761312806 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1141 (time 1761312884.48 total 10.01 last 114.04) - open/close 2149 (time 1761312894.48 total 20.01 last 100.78) - open/close 3068 (time 1761312904.48 total 30.01 last 91.90) - open/close 3997 (time 1761312914.49 total 40.01 last 92.85) - open/close 4944 (time 1761312924.50 total 50.02 last 94.60) - open/close 5946 (time 1761312934.50 total 60.03 last 100.16) - open/close 6841 (time 1761312944.50 total 70.03 last 89.48) - open/close 7756 (time 1761312954.52 total 80.04 last 91.39) - open/close 8645 (time 1761312964.52 total 90.05 last 88.86) - open/close 9581 (time 1761312974.52 total 100.05 last 93.56) total: 10000 open/close in 104.19 seconds: 95.98 ops/second - unlinked 0 (time 1761312981 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 895 (time 1761313058.43 total 10.00 last 89.49) - open/close 1700 (time 1761313068.43 total 20.00 last 80.50) - open/close 2535 (time 1761313078.45 total 30.02 last 83.31) - open/close 3446 (time 1761313088.45 total 40.02 last 91.09) - open/close 4262 (time 1761313098.47 total 50.04 last 81.45) - open/close 5172 (time 1761313108.47 total 60.05 last 90.96) - open/close 6083 (time 1761313118.48 total 70.05 last 91.05) - open/close 6943 (time 1761313128.48 total 80.05 last 86.00) - open/close 7790 (time 1761313138.49 total 90.06 last 84.60) - open/close 8635 (time 1761313148.50 total 100.07 last 84.42) - open/close 9470 (time 1761313158.51 total 110.08 last 83.43) total: 10000 open/close in 116.78 seconds: 85.63 ops/second - unlinked 0 (time 1761313168 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 834 (time 1761313251.43 total 10.02 last 83.24) - open/close 1673 (time 1761313261.44 total 20.03 last 83.84) - open/close 2567 (time 1761313271.45 total 30.03 last 89.37) - open/close 3461 (time 1761313281.45 total 40.03 last 89.38) - open/close 4507 (time 1761313291.46 total 50.04 last 104.50) - open/close 5340 (time 1761313301.46 total 60.04 last 83.30) - open/close 6201 (time 1761313311.46 total 70.04 last 86.09) - open/close 7068 (time 1761313321.47 total 80.05 last 86.64) - open/close 7837 (time 1761313331.47 total 90.05 last 76.86) - open/close 8590 (time 1761313341.47 total 100.06 last 75.30) - open/close 9426 (time 1761313351.48 total 110.07 last 83.52) total: 10000 open/close in 117.81 seconds: 84.88 ops/second - unlinked 0 (time 1761313362 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 846 (time 1761313447.31 total 10.01 last 84.48) - open/close 1658 (time 1761313457.32 total 20.03 last 81.11) - open/close 2441 (time 1761313467.32 total 30.03 last 78.27) - open/close 3272 (time 1761313477.33 total 40.03 last 83.08) - open/close 4091 (time 1761313487.33 total 50.03 last 81.88) - open/close 4840 (time 1761313497.33 total 60.04 last 74.89) - open/close 5612 (time 1761313507.34 total 70.04 last 77.16) - open/close 6415 (time 1761313517.34 total 80.05 last 80.25) - open/close 7274 (time 1761313527.35 total 90.05 last 85.86) - open/close 8083 (time 1761313537.36 total 100.06 last 80.80) - open/close 8887 (time 1761313547.39 total 110.09 last 80.18) - open/close 9591 (time 1761313557.39 total 120.09 last 70.39) total: 10000 open/close in 125.97 seconds: 79.38 ops/second - unlinked 0 (time 1761313567 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 875 (time 1761313653.42 total 10.00 last 87.48) - open/close 1719 (time 1761313663.42 total 20.00 last 84.39) - open/close 2616 (time 1761313673.43 total 30.01 last 89.64) - open/close 3606 (time 1761313683.44 total 40.02 last 98.90) - open/close 4655 (time 1761313693.44 total 50.03 last 104.84) - open/close 5606 (time 1761313703.46 total 60.04 last 94.99) - open/close 6561 (time 1761313713.47 total 70.06 last 95.33) - open/close 7396 (time 1761313723.47 total 80.06 last 83.50) - open/close 8318 (time 1761313733.48 total 90.06 last 92.16) - open/close 9209 (time 1761313743.48 total 100.06 last 89.09) total: 10000 open/close in 108.58 seconds: 92.10 ops/second - unlinked 0 (time 1761313754 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 837 (time 1761313831.76 total 10.01 last 83.61) - open/close 1777 (time 1761313841.76 total 20.01 last 94.00) - open/close 2717 (time 1761313851.76 total 30.02 last 93.95) - open/close 3533 (time 1761313861.77 total 40.02 last 81.56) - open/close 4306 (time 1761313871.77 total 50.02 last 77.29) - open/close 5176 (time 1761313881.77 total 60.02 last 86.99) - open/close 6029 (time 1761313891.77 total 70.02 last 85.29) - open/close 6881 (time 1761313901.78 total 80.03 last 85.13) - open/close 7799 (time 1761313911.79 total 90.04 last 91.72) - open/close 8696 (time 1761313921.79 total 100.05 last 89.67) - open/close 9555 (time 1761313931.79 total 110.05 last 85.89) total: 10000 open/close in 115.03 seconds: 86.93 ops/second - unlinked 0 (time 1761313939 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.148@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg348-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg348-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg348-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre On OST0, 10386 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local oleg348-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg348-server: LNET ready to unload oleg348-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2