== conf-sanity test 69: replace an OST with the same index ========================================================== 10:02:12 (1761314532) start mds service on oleg318-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg318-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg318-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 831 (time 1761314629.05 total 10.02 last 82.92) - open/close 1598 (time 1761314639.06 total 20.03 last 76.65) - open/close 2387 (time 1761314649.06 total 30.04 last 78.85) - open/close 3265 (time 1761314659.07 total 40.04 last 87.78) - open/close 4144 (time 1761314669.07 total 50.04 last 87.87) - open/close 5047 (time 1761314679.09 total 60.06 last 90.09) - open/close 5838 (time 1761314689.10 total 70.07 last 79.02) - open/close 6644 (time 1761314699.11 total 80.08 last 80.58) - open/close 7455 (time 1761314709.11 total 90.08 last 81.05) - open/close 8279 (time 1761314719.12 total 100.09 last 82.34) - open/close 9076 (time 1761314729.13 total 110.10 last 79.61) - open/close 9834 (time 1761314739.14 total 120.11 last 75.74) total: 10000 open/close in 122.19 seconds: 81.84 ops/second - unlinked 0 (time 1761314744 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 779 (time 1761314823.05 total 10.01 last 77.86) - open/close 1566 (time 1761314833.06 total 20.01 last 78.64) - open/close 2373 (time 1761314843.06 total 30.02 last 80.64) - open/close 3197 (time 1761314853.07 total 40.03 last 82.35) - open/close 4046 (time 1761314863.08 total 50.04 last 84.79) - open/close 4897 (time 1761314873.09 total 60.04 last 85.06) - open/close 5680 (time 1761314883.09 total 70.05 last 78.25) - open/close 6501 (time 1761314893.10 total 80.06 last 82.01) - open/close 7352 (time 1761314903.11 total 90.06 last 85.07) - open/close 8181 (time 1761314913.11 total 100.07 last 82.87) - open/close 9066 (time 1761314923.12 total 110.08 last 88.41) - open/close 9877 (time 1761314933.12 total 120.08 last 81.08) total: 10000 open/close in 121.56 seconds: 82.26 ops/second - unlinked 0 (time 1761314937 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 904 (time 1761315004.55 total 10.01 last 90.29) - open/close 1806 (time 1761315014.55 total 20.01 last 90.19) - open/close 2659 (time 1761315024.55 total 30.01 last 85.29) - open/close 3458 (time 1761315034.56 total 40.02 last 79.87) - open/close 4258 (time 1761315044.58 total 50.04 last 79.84) - open/close 5101 (time 1761315054.58 total 60.04 last 84.27) - open/close 5953 (time 1761315064.59 total 70.05 last 85.12) - open/close 6775 (time 1761315074.59 total 80.05 last 82.19) - open/close 7593 (time 1761315084.60 total 90.06 last 81.72) - open/close 8457 (time 1761315094.61 total 100.07 last 86.37) - open/close 9248 (time 1761315104.61 total 110.07 last 79.03) total: 10000 open/close in 118.90 seconds: 84.10 ops/second - unlinked 0 (time 1761315116 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 813 (time 1761315187.90 total 10.01 last 81.21) - open/close 1748 (time 1761315197.91 total 20.02 last 93.38) - open/close 2616 (time 1761315207.92 total 30.03 last 86.74) - open/close 3436 (time 1761315217.92 total 40.04 last 81.94) - open/close 4348 (time 1761315227.92 total 50.04 last 91.20) - open/close 5294 (time 1761315237.93 total 60.04 last 94.55) - open/close 6260 (time 1761315247.94 total 70.05 last 96.53) - open/close 7156 (time 1761315257.94 total 80.05 last 89.60) - open/close 7967 (time 1761315267.94 total 90.05 last 81.07) - open/close 8849 (time 1761315277.95 total 100.06 last 88.11) - open/close 9678 (time 1761315287.95 total 110.07 last 82.88) total: 10000 open/close in 113.97 seconds: 87.74 ops/second - unlinked 0 (time 1761315294 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 865 (time 1761315377.62 total 10.00 last 86.48) - open/close 1757 (time 1761315387.63 total 20.01 last 89.14) - open/close 2584 (time 1761315397.63 total 30.01 last 82.66) - open/close 3372 (time 1761315407.63 total 40.02 last 78.78) - open/close 4191 (time 1761315417.64 total 50.03 last 81.83) - open/close 5028 (time 1761315427.66 total 60.04 last 83.57) - open/close 5917 (time 1761315437.67 total 70.05 last 88.80) - open/close 6880 (time 1761315447.67 total 80.05 last 96.29) - open/close 7816 (time 1761315457.67 total 90.06 last 93.58) - open/close 8726 (time 1761315467.68 total 100.06 last 90.94) - open/close 9505 (time 1761315477.69 total 110.07 last 77.83) total: 10000 open/close in 116.59 seconds: 85.77 ops/second - unlinked 0 (time 1761315487 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 788 (time 1761315564.79 total 10.00 last 78.80) - open/close 1615 (time 1761315574.80 total 20.01 last 82.63) - open/close 2483 (time 1761315584.81 total 30.02 last 86.67) - open/close 3425 (time 1761315594.82 total 40.03 last 94.17) - open/close 4285 (time 1761315604.82 total 50.03 last 85.95) - open/close 5150 (time 1761315614.83 total 60.04 last 86.42) - open/close 5887 (time 1761315624.84 total 70.06 last 73.60) - open/close 6701 (time 1761315634.85 total 80.06 last 81.34) - open/close 7546 (time 1761315644.86 total 90.07 last 84.45) - open/close 8399 (time 1761315654.86 total 100.07 last 85.26) - open/close 9237 (time 1761315664.87 total 110.09 last 83.71) total: 10000 open/close in 118.55 seconds: 84.35 ops/second - unlinked 0 (time 1761315676 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 869 (time 1761315760.76 total 10.00 last 86.88) - open/close 1726 (time 1761315770.76 total 20.00 last 85.69) - open/close 2542 (time 1761315780.77 total 30.01 last 81.52) - open/close 3439 (time 1761315790.78 total 40.02 last 89.61) - open/close 4226 (time 1761315800.79 total 50.03 last 78.63) - open/close 5071 (time 1761315810.79 total 60.04 last 84.46) - open/close 5840 (time 1761315820.80 total 70.04 last 76.86) - open/close 6645 (time 1761315830.80 total 80.04 last 80.49) - open/close 7481 (time 1761315840.80 total 90.04 last 83.59) - open/close 8295 (time 1761315850.81 total 100.05 last 81.34) - open/close 9088 (time 1761315860.81 total 110.06 last 79.27) - open/close 9874 (time 1761315870.82 total 120.06 last 78.54) total: 10000 open/close in 121.61 seconds: 82.23 ops/second - unlinked 0 (time 1761315875 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 769 (time 1761315957.95 total 10.01 last 76.86) - open/close 1560 (time 1761315967.96 total 20.01 last 79.05) - open/close 2304 (time 1761315977.97 total 30.03 last 74.29) - open/close 3106 (time 1761315987.99 total 40.04 last 80.08) - open/close 3904 (time 1761315998.00 total 50.06 last 79.69) - open/close 4724 (time 1761316008.02 total 60.08 last 81.84) - open/close 5559 (time 1761316018.02 total 70.08 last 83.50) - open/close 6320 (time 1761316028.03 total 80.08 last 76.07) - open/close 7130 (time 1761316038.04 total 90.09 last 80.91) - open/close 7903 (time 1761316048.04 total 100.10 last 77.25) - open/close 8713 (time 1761316058.06 total 110.11 last 80.91) - open/close 9558 (time 1761316068.06 total 120.11 last 84.48) total: 10000 open/close in 125.79 seconds: 79.50 ops/second - unlinked 0 (time 1761316077 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1013 (time 1761316158.29 total 10.01 last 101.15) - open/close 2004 (time 1761316168.30 total 20.02 last 99.01) - open/close 2908 (time 1761316178.31 total 30.03 last 90.31) - open/close 3745 (time 1761316188.32 total 40.04 last 83.67) - open/close 4620 (time 1761316198.32 total 50.04 last 87.45) - open/close 5611 (time 1761316208.33 total 60.05 last 99.04) - open/close 6556 (time 1761316218.33 total 70.05 last 94.48) - open/close 7372 (time 1761316228.34 total 80.06 last 81.54) - open/close 8264 (time 1761316238.34 total 90.06 last 89.19) - open/close 9115 (time 1761316248.34 total 100.06 last 85.06) total: 10000 open/close in 110.00 seconds: 90.91 ops/second - unlinked 0 (time 1761316261 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 840 (time 1761316337.74 total 10.00 last 84.00) - open/close 1645 (time 1761316347.75 total 20.00 last 80.48) - open/close 2495 (time 1761316357.75 total 30.01 last 84.92) - open/close 3338 (time 1761316367.77 total 40.02 last 84.21) - open/close 4138 (time 1761316377.78 total 50.03 last 79.92) - open/close 4959 (time 1761316387.78 total 60.04 last 82.07) - open/close 5795 (time 1761316397.80 total 70.05 last 83.47) - open/close 6628 (time 1761316407.80 total 80.05 last 83.30) - open/close 7441 (time 1761316417.81 total 90.06 last 81.22) - open/close 8301 (time 1761316427.81 total 100.07 last 85.95) - open/close 9106 (time 1761316437.81 total 110.07 last 80.48) - open/close 9925 (time 1761316447.82 total 120.08 last 81.82) total: 10000 open/close in 121.12 seconds: 82.56 ops/second - unlinked 0 (time 1761316451 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg318-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg318-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg318-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg318-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg318-server: LNET ready to unload oleg318-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2