== conf-sanity test 69: replace an OST with the same index ========================================================== 09:01:05 (1761310865) start mds service on oleg323-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg323-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg323-server: detected 4 online CPUs by sysfs oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg323-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 1038 (time 1761310944.68 total 10.00 last 103.77) - open/close 2140 (time 1761310954.68 total 20.00 last 110.18) - open/close 3292 (time 1761310964.68 total 30.01 last 115.19) - open/close 4299 (time 1761310974.69 total 40.01 last 100.64) - open/close 5097 (time 1761310984.70 total 50.02 last 79.76) - open/close 5914 (time 1761310994.70 total 60.02 last 81.65) - open/close 6757 (time 1761311004.71 total 70.03 last 84.20) - open/close 7632 (time 1761311014.72 total 80.04 last 87.42) - open/close 8561 (time 1761311024.74 total 90.06 last 92.79) - open/close 9481 (time 1761311034.74 total 100.06 last 91.99) total: 10000 open/close in 105.40 seconds: 94.88 ops/second - unlinked 0 (time 1761311042 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1382 (time 1761311103.74 total 10.01 last 138.11) - open/close 2791 (time 1761311113.74 total 20.01 last 140.81) - open/close 4411 (time 1761311123.75 total 30.02 last 161.92) - open/close 6136 (time 1761311133.75 total 40.02 last 172.47) - open/close 7995 (time 1761311143.75 total 50.02 last 185.85) - open/close 9626 (time 1761311153.76 total 60.03 last 162.99) total: 10000 open/close in 62.29 seconds: 160.55 ops/second - unlinked 0 (time 1761311157 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1613 (time 1761311202.79 total 10.00 last 161.22) - open/close 3555 (time 1761311212.79 total 20.01 last 194.15) - open/close 5089 (time 1761311222.80 total 30.01 last 153.34) - open/close 6430 (time 1761311232.80 total 40.01 last 134.09) - open/close 7603 (time 1761311242.80 total 50.01 last 117.26) - open/close 8701 (time 1761311252.81 total 60.02 last 109.72) total: 10000 open/close in 70.00 seconds: 142.85 ops/second - unlinked 0 (time 1761311264 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 727 (time 1761311330.77 total 10.01 last 72.60) - open/close 1630 (time 1761311340.78 total 20.02 last 90.27) - open/close 2723 (time 1761311350.78 total 30.02 last 109.25) - open/close 3798 (time 1761311360.79 total 40.03 last 107.44) - open/close 4785 (time 1761311370.80 total 50.04 last 98.61) - open/close 5738 (time 1761311380.81 total 60.05 last 95.17) - open/close 6717 (time 1761311390.82 total 70.06 last 97.82) - open/close 7468 (time 1761311400.83 total 80.07 last 74.99) - open/close 7958 (time 1761311410.86 total 90.10 last 48.85) - open/close 8417 (time 1761311420.88 total 100.12 last 45.81) - open/close 8956 (time 1761311430.90 total 110.14 last 53.82) - open/close 9513 (time 1761311440.90 total 120.14 last 55.68) total: 10000 open/close in 127.58 seconds: 78.38 ops/second - unlinked 0 (time 1761311452 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 771 (time 1761311534.33 total 10.00 last 77.07) - open/close 1509 (time 1761311544.33 total 20.01 last 73.74) - open/close 2227 (time 1761311554.34 total 30.02 last 71.74) - open/close 3060 (time 1761311564.35 total 40.02 last 83.28) - open/close 3910 (time 1761311574.36 total 50.03 last 84.91) - open/close 4789 (time 1761311584.36 total 60.04 last 87.85) - open/close 5723 (time 1761311594.37 total 70.05 last 93.34) - open/close 6532 (time 1761311604.38 total 80.06 last 80.82) - open/close 7398 (time 1761311614.38 total 90.06 last 86.56) - open/close 8276 (time 1761311624.40 total 100.08 last 87.62) - open/close 9147 (time 1761311634.41 total 110.09 last 87.06) - open/close 9991 (time 1761311644.43 total 120.10 last 84.24) total: 10000 open/close in 120.19 seconds: 83.20 ops/second - unlinked 0 (time 1761311648 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 840 (time 1761311730.69 total 10.00 last 83.97) - open/close 1654 (time 1761311740.70 total 20.02 last 81.30) - open/close 2484 (time 1761311750.71 total 30.03 last 82.88) - open/close 3277 (time 1761311760.72 total 40.04 last 79.25) - open/close 4127 (time 1761311770.72 total 50.04 last 84.99) - open/close 4961 (time 1761311780.72 total 60.04 last 83.37) - open/close 5849 (time 1761311790.73 total 70.04 last 88.77) - open/close 6788 (time 1761311800.73 total 80.04 last 93.89) - open/close 7694 (time 1761311810.73 total 90.05 last 90.58) - open/close 8588 (time 1761311820.73 total 100.05 last 89.36) - open/close 9543 (time 1761311830.74 total 110.06 last 95.46) total: 10000 open/close in 114.99 seconds: 86.96 ops/second - unlinked 0 (time 1761311839 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 896 (time 1761311922.93 total 10.00 last 89.59) - open/close 1722 (time 1761311932.94 total 20.01 last 82.55) - open/close 2582 (time 1761311942.95 total 30.02 last 85.86) - open/close 3453 (time 1761311952.96 total 40.03 last 87.02) - open/close 4360 (time 1761311962.97 total 50.04 last 90.65) - open/close 5291 (time 1761311972.97 total 60.04 last 93.10) - open/close 6249 (time 1761311982.97 total 70.05 last 95.73) - open/close 7152 (time 1761311992.98 total 80.05 last 90.29) - open/close 8011 (time 1761312002.99 total 90.06 last 85.81) - open/close 8901 (time 1761312012.99 total 100.06 last 88.97) - open/close 9809 (time 1761312023.01 total 110.08 last 90.66) total: 10000 open/close in 112.17 seconds: 89.15 ops/second - unlinked 0 (time 1761312028 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 893 (time 1761312106.36 total 10.01 last 89.24) - open/close 1777 (time 1761312116.36 total 20.01 last 88.37) - open/close 2651 (time 1761312126.36 total 30.01 last 87.39) - open/close 3528 (time 1761312136.36 total 40.01 last 87.70) - open/close 4463 (time 1761312146.37 total 50.02 last 93.42) - open/close 5388 (time 1761312156.37 total 60.02 last 92.46) - open/close 6313 (time 1761312166.38 total 70.03 last 92.42) - open/close 7219 (time 1761312176.39 total 80.04 last 90.55) - open/close 8107 (time 1761312186.40 total 90.05 last 88.74) - open/close 9056 (time 1761312196.40 total 100.05 last 94.89) - open/close 9990 (time 1761312206.41 total 110.06 last 93.29) total: 10000 open/close in 110.14 seconds: 90.79 ops/second - unlinked 0 (time 1761312209 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 878 (time 1761312287.70 total 10.01 last 87.72) - open/close 1716 (time 1761312297.71 total 20.01 last 83.77) - open/close 2653 (time 1761312307.72 total 30.02 last 93.62) - open/close 3545 (time 1761312317.72 total 40.03 last 89.15) - open/close 4366 (time 1761312327.72 total 50.03 last 82.09) - open/close 5227 (time 1761312337.73 total 60.03 last 86.09) - open/close 6109 (time 1761312347.73 total 70.04 last 88.15) - open/close 6921 (time 1761312357.74 total 80.05 last 81.10) - open/close 7734 (time 1761312367.75 total 90.05 last 81.25) - open/close 8624 (time 1761312377.75 total 100.06 last 88.96) - open/close 9559 (time 1761312387.75 total 110.06 last 93.50) total: 10000 open/close in 115.11 seconds: 86.87 ops/second - unlinked 0 (time 1761312395 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 930 (time 1761312462.72 total 10.00 last 92.99) - open/close 1725 (time 1761312472.73 total 20.01 last 79.45) - open/close 2683 (time 1761312482.74 total 30.02 last 95.70) - open/close 3647 (time 1761312492.75 total 40.03 last 96.33) - open/close 4583 (time 1761312502.75 total 50.03 last 93.55) - open/close 5513 (time 1761312512.76 total 60.04 last 92.93) - open/close 6481 (time 1761312522.77 total 70.05 last 96.73) - open/close 7460 (time 1761312532.77 total 80.05 last 97.85) - open/close 8409 (time 1761312542.78 total 90.06 last 94.85) - open/close 9342 (time 1761312552.78 total 100.06 last 93.26) total: 10000 open/close in 107.04 seconds: 93.42 ops/second - unlinked 0 (time 1761312562 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg323-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg323-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg323-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg323-server: LNET ready to unload oleg323-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2