== conf-sanity test 69: replace an OST with the same index ========================================================== 07:20:39 (1764937239) start mds service on oleg617-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing load_modules_local oleg617-server: Loading modules from /home/green/git/lustre-release/lustre oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg617-server: detected 4 online CPUs by sysfs oleg617-server: Force libcfs to create 2 CPU partitions oleg617-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg617-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg617-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg617-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 750 (time 1764937320.65 total 10.00 last 74.97) - open/close 1635 (time 1764937330.66 total 20.02 last 88.39) - open/close 2457 (time 1764937340.67 total 30.02 last 82.14) - open/close 3318 (time 1764937350.67 total 40.03 last 86.05) - open/close 4134 (time 1764937360.68 total 50.04 last 81.52) - open/close 4953 (time 1764937370.69 total 60.04 last 81.86) - open/close 5744 (time 1764937380.69 total 70.05 last 79.06) - open/close 6589 (time 1764937390.69 total 80.05 last 84.49) - open/close 7370 (time 1764937400.70 total 90.05 last 78.07) - open/close 8152 (time 1764937410.71 total 100.07 last 78.11) - open/close 8891 (time 1764937420.72 total 110.07 last 73.85) - open/close 9757 (time 1764937430.72 total 120.08 last 86.55) total: 10000 open/close in 122.85 seconds: 81.40 ops/second - unlinked 0 (time 1764937436 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 833 (time 1764937517.71 total 10.00 last 83.30) - open/close 1652 (time 1764937527.71 total 20.00 last 81.90) - open/close 2528 (time 1764937537.71 total 30.00 last 87.60) - open/close 3334 (time 1764937547.71 total 40.00 last 80.59) - open/close 4068 (time 1764937557.72 total 50.01 last 73.36) - open/close 4806 (time 1764937567.72 total 60.01 last 73.75) - open/close 5563 (time 1764937577.72 total 70.02 last 75.70) - open/close 6438 (time 1764937587.73 total 80.02 last 87.43) - open/close 7307 (time 1764937597.74 total 90.03 last 86.87) - open/close 8144 (time 1764937607.75 total 100.04 last 83.62) - open/close 8817 (time 1764937617.75 total 110.04 last 67.27) - open/close 9475 (time 1764937627.76 total 120.05 last 65.72) total: 10000 open/close in 125.27 seconds: 79.83 ops/second - unlinked 0 (time 1764937636 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 827 (time 1764937722.36 total 10.00 last 82.67) - open/close 1699 (time 1764937732.36 total 20.01 last 87.17) - open/close 2504 (time 1764937742.37 total 30.01 last 80.46) - open/close 3357 (time 1764937752.37 total 40.02 last 85.25) - open/close 4194 (time 1764937762.38 total 50.03 last 83.61) - open/close 5068 (time 1764937772.39 total 60.04 last 87.32) - open/close 5913 (time 1764937782.40 total 70.05 last 84.45) - open/close 6797 (time 1764937792.40 total 80.05 last 88.37) - open/close 7616 (time 1764937802.41 total 90.05 last 81.85) - open/close 8517 (time 1764937812.43 total 100.08 last 89.89) - open/close 9301 (time 1764937822.44 total 110.09 last 78.34) - open/close 9992 (time 1764937832.46 total 120.10 last 68.97) total: 10000 open/close in 120.24 seconds: 83.16 ops/second - unlinked 0 (time 1764937836 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 868 (time 1764937921.36 total 10.01 last 86.75) - open/close 1723 (time 1764937931.37 total 20.01 last 85.48) - open/close 2572 (time 1764937941.38 total 30.02 last 84.80) - open/close 3310 (time 1764937951.38 total 40.02 last 73.80) - open/close 4168 (time 1764937961.38 total 50.02 last 85.79) - open/close 4938 (time 1764937971.40 total 60.04 last 76.83) - open/close 5786 (time 1764937981.41 total 70.05 last 84.73) - open/close 6621 (time 1764937991.43 total 80.07 last 83.32) - open/close 7485 (time 1764938001.44 total 90.08 last 86.31) - open/close 8361 (time 1764938011.44 total 100.08 last 87.59) - open/close 9171 (time 1764938021.44 total 110.09 last 80.99) - open/close 9929 (time 1764938031.45 total 120.09 last 75.78) total: 10000 open/close in 121.08 seconds: 82.59 ops/second - unlinked 0 (time 1764938036 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 838 (time 1764938122.48 total 10.01 last 83.72) - open/close 1649 (time 1764938132.49 total 20.03 last 80.96) - open/close 2388 (time 1764938142.50 total 30.03 last 73.88) - open/close 3184 (time 1764938152.50 total 40.04 last 79.55) - open/close 3830 (time 1764938162.51 total 50.04 last 64.56) - open/close 4626 (time 1764938172.51 total 60.05 last 79.56) - open/close 5455 (time 1764938182.53 total 70.06 last 82.80) - open/close 6283 (time 1764938192.54 total 80.07 last 82.70) - open/close 7034 (time 1764938202.55 total 90.08 last 75.04) - open/close 7595 (time 1764938212.55 total 100.09 last 56.06) - open/close 8178 (time 1764938222.56 total 110.10 last 58.22) - open/close 8811 (time 1764938232.59 total 120.12 last 63.17) - open/close 9566 (time 1764938242.59 total 130.12 last 75.48) total: 10000 open/close in 135.55 seconds: 73.77 ops/second - unlinked 0 (time 1764938252 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 904 (time 1764938340.68 total 10.00 last 90.37) - open/close 1650 (time 1764938350.69 total 20.01 last 74.54) - open/close 2479 (time 1764938360.69 total 30.01 last 82.89) - open/close 3264 (time 1764938370.69 total 40.02 last 78.45) - open/close 4181 (time 1764938380.70 total 50.03 last 91.62) - open/close 4956 (time 1764938390.70 total 60.03 last 77.49) - open/close 5817 (time 1764938400.70 total 70.03 last 86.08) - open/close 6666 (time 1764938410.71 total 80.04 last 84.85) - open/close 7525 (time 1764938420.72 total 90.04 last 85.85) - open/close 8390 (time 1764938430.72 total 100.05 last 86.46) - open/close 9164 (time 1764938440.73 total 110.05 last 77.35) - open/close 9929 (time 1764938450.73 total 120.05 last 76.50) total: 10000 open/close in 120.91 seconds: 82.71 ops/second - unlinked 0 (time 1764938455 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 913 (time 1764938535.40 total 10.01 last 91.20) - open/close 1812 (time 1764938545.40 total 20.01 last 89.89) - open/close 2698 (time 1764938555.40 total 30.01 last 88.57) - open/close 3524 (time 1764938565.40 total 40.02 last 82.59) - open/close 4359 (time 1764938575.41 total 50.02 last 83.45) - open/close 5254 (time 1764938585.41 total 60.02 last 89.48) - open/close 6117 (time 1764938595.41 total 70.03 last 86.29) - open/close 7007 (time 1764938605.41 total 80.03 last 88.97) - open/close 7721 (time 1764938615.42 total 90.03 last 71.37) - open/close 8624 (time 1764938625.43 total 100.04 last 90.23) - open/close 9554 (time 1764938635.43 total 110.04 last 93.00) total: 10000 open/close in 115.04 seconds: 86.93 ops/second - unlinked 0 (time 1764938644 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 911 (time 1764938728.09 total 10.00 last 91.10) - open/close 1775 (time 1764938738.11 total 20.02 last 86.21) - open/close 2666 (time 1764938748.12 total 30.03 last 89.03) - open/close 3518 (time 1764938758.12 total 40.03 last 85.19) - open/close 4396 (time 1764938768.13 total 50.04 last 87.73) - open/close 5231 (time 1764938778.13 total 60.04 last 83.49) - open/close 5924 (time 1764938788.14 total 70.04 last 69.28) - open/close 6646 (time 1764938798.16 total 80.07 last 72.00) - open/close 7343 (time 1764938808.17 total 90.08 last 69.65) - open/close 8154 (time 1764938818.18 total 100.09 last 81.03) - open/close 8950 (time 1764938828.19 total 110.09 last 79.56) - open/close 9825 (time 1764938838.19 total 120.09 last 87.49) total: 10000 open/close in 122.14 seconds: 81.87 ops/second - unlinked 0 (time 1764938843 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 794 (time 1764938933.23 total 10.02 last 79.28) - open/close 1618 (time 1764938943.23 total 20.02 last 82.37) - open/close 2387 (time 1764938953.24 total 30.02 last 76.89) - open/close 3275 (time 1764938963.25 total 40.03 last 88.71) - open/close 4178 (time 1764938973.25 total 50.03 last 90.27) - open/close 5049 (time 1764938983.27 total 60.05 last 86.93) - open/close 5874 (time 1764938993.27 total 70.06 last 82.48) - open/close 6836 (time 1764939003.27 total 80.06 last 96.17) - open/close 7707 (time 1764939013.28 total 90.06 last 87.05) - open/close 8613 (time 1764939023.29 total 100.07 last 90.52) - open/close 9362 (time 1764939033.30 total 110.09 last 74.78) total: 10000 open/close in 118.02 seconds: 84.73 ops/second - unlinked 0 (time 1764939044 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 825 (time 1764939127.27 total 10.01 last 82.42) - open/close 1556 (time 1764939137.27 total 20.01 last 73.08) - open/close 2421 (time 1764939147.27 total 30.02 last 86.47) - open/close 3188 (time 1764939157.29 total 40.03 last 76.60) - open/close 3935 (time 1764939167.29 total 50.03 last 74.69) - open/close 4747 (time 1764939177.30 total 60.04 last 81.13) - open/close 5512 (time 1764939187.30 total 70.04 last 76.48) - open/close 6343 (time 1764939197.31 total 80.05 last 83.03) - open/close 7186 (time 1764939207.32 total 90.06 last 84.22) - open/close 8065 (time 1764939217.33 total 100.07 last 87.79) - open/close 8940 (time 1764939227.34 total 110.08 last 87.45) - open/close 9586 (time 1764939237.34 total 120.08 last 64.57) total: 10000 open/close in 125.29 seconds: 79.81 ops/second - unlinked 0 (time 1764939246 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg617-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.206.117@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg617-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg617-server: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg617-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg617-server: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg617-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg617-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg617-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing unload_modules_local oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg617-server: LNET ready to unload oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg617-client: oleg617-server: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-client: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-client: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 2