== conf-sanity test 69: replace an OST with the same index ========================================================== 10:33:55 (1761316435) start mds service on oleg134-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg134-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 884 (time 1761316529.65 total 10.01 last 88.31) - open/close 1664 (time 1761316539.66 total 20.02 last 77.96) - open/close 2547 (time 1761316549.67 total 30.03 last 88.19) - open/close 3386 (time 1761316559.69 total 40.05 last 83.72) - open/close 4211 (time 1761316569.70 total 50.06 last 82.42) - open/close 5044 (time 1761316579.71 total 60.07 last 83.25) - open/close 5979 (time 1761316589.73 total 70.09 last 93.31) - open/close 6809 (time 1761316599.74 total 80.10 last 82.92) - open/close 7684 (time 1761316609.74 total 90.10 last 87.46) - open/close 8657 (time 1761316619.75 total 100.11 last 97.19) - open/close 9573 (time 1761316629.76 total 110.12 last 91.56) total: 10000 open/close in 114.71 seconds: 87.18 ops/second - unlinked 0 (time 1761316637 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 935 (time 1761316713.06 total 10.00 last 93.49) - open/close 2092 (time 1761316723.07 total 20.00 last 115.66) - open/close 3178 (time 1761316733.07 total 30.01 last 108.52) - open/close 4186 (time 1761316743.08 total 40.02 last 100.74) - open/close 5070 (time 1761316753.09 total 50.03 last 88.28) - open/close 5910 (time 1761316763.11 total 60.04 last 83.89) - open/close 6786 (time 1761316773.11 total 70.05 last 87.53) - open/close 7660 (time 1761316783.12 total 80.05 last 87.38) - open/close 8551 (time 1761316793.12 total 90.06 last 89.04) - open/close 9542 (time 1761316803.13 total 100.06 last 99.06) total: 10000 open/close in 104.84 seconds: 95.38 ops/second - unlinked 0 (time 1761316810 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 820 (time 1761316887.86 total 10.00 last 81.96) - open/close 1678 (time 1761316897.87 total 20.01 last 85.73) - open/close 2493 (time 1761316907.87 total 30.01 last 81.49) - open/close 3364 (time 1761316917.88 total 40.02 last 87.06) - open/close 4307 (time 1761316927.88 total 50.02 last 94.27) - open/close 5244 (time 1761316937.88 total 60.02 last 93.69) - open/close 6068 (time 1761316947.89 total 70.03 last 82.33) - open/close 6930 (time 1761316957.89 total 80.04 last 86.17) - open/close 7768 (time 1761316967.90 total 90.04 last 83.74) - open/close 8604 (time 1761316977.91 total 100.05 last 83.50) - open/close 9412 (time 1761316987.92 total 110.06 last 80.76) total: 10000 open/close in 117.27 seconds: 85.28 ops/second - unlinked 0 (time 1761316997 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 795 (time 1761317080.34 total 10.00 last 79.46) - open/close 1615 (time 1761317090.35 total 20.01 last 81.96) - open/close 2407 (time 1761317100.35 total 30.01 last 79.18) - open/close 3267 (time 1761317110.35 total 40.02 last 85.97) - open/close 4084 (time 1761317120.36 total 50.02 last 81.65) - open/close 4922 (time 1761317130.36 total 60.03 last 83.76) - open/close 5820 (time 1761317140.37 total 70.03 last 89.73) - open/close 6618 (time 1761317150.38 total 80.04 last 79.72) - open/close 7404 (time 1761317160.38 total 90.05 last 78.55) - open/close 8209 (time 1761317170.40 total 100.07 last 80.35) - open/close 8967 (time 1761317180.41 total 110.07 last 75.79) - open/close 9791 (time 1761317190.41 total 120.07 last 82.40) total: 10000 open/close in 122.62 seconds: 81.55 ops/second - unlinked 0 (time 1761317196 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 777 (time 1761317286.12 total 10.00 last 77.67) - open/close 1560 (time 1761317296.13 total 20.01 last 78.25) - open/close 2352 (time 1761317306.13 total 30.01 last 79.18) - open/close 3130 (time 1761317316.14 total 40.02 last 77.72) - open/close 3878 (time 1761317326.15 total 50.03 last 74.76) - open/close 4638 (time 1761317336.16 total 60.04 last 75.95) - open/close 5489 (time 1761317346.16 total 70.04 last 85.05) - open/close 6376 (time 1761317356.17 total 80.05 last 88.60) - open/close 7262 (time 1761317366.18 total 90.06 last 88.53) - open/close 8089 (time 1761317376.18 total 100.06 last 82.68) - open/close 8998 (time 1761317386.19 total 110.07 last 90.85) - open/close 9904 (time 1761317396.19 total 120.07 last 90.55) total: 10000 open/close in 121.38 seconds: 82.39 ops/second - unlinked 0 (time 1761317400 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 852 (time 1761317484.36 total 10.00 last 85.16) - open/close 1645 (time 1761317494.36 total 20.01 last 79.29) - open/close 2244 (time 1761317504.37 total 30.02 last 59.80) - open/close 3053 (time 1761317514.39 total 40.03 last 80.80) - open/close 3894 (time 1761317524.40 total 50.05 last 84.01) - open/close 4709 (time 1761317534.40 total 60.05 last 81.48) - open/close 5524 (time 1761317544.40 total 70.05 last 81.49) - open/close 6315 (time 1761317554.41 total 80.05 last 79.06) - open/close 7259 (time 1761317564.41 total 90.05 last 94.39) - open/close 8080 (time 1761317574.43 total 100.07 last 81.94) - open/close 8957 (time 1761317584.43 total 110.08 last 87.65) - open/close 9861 (time 1761317594.44 total 120.08 last 90.36) total: 10000 open/close in 121.66 seconds: 82.20 ops/second - unlinked 0 (time 1761317598 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 792 (time 1761317669.45 total 10.01 last 79.15) - open/close 1721 (time 1761317679.45 total 20.01 last 92.89) - open/close 2563 (time 1761317689.45 total 30.01 last 84.15) - open/close 3418 (time 1761317699.46 total 40.02 last 85.48) - open/close 4303 (time 1761317709.47 total 50.03 last 88.42) - open/close 5202 (time 1761317719.50 total 60.06 last 89.62) - open/close 6008 (time 1761317729.50 total 70.06 last 80.55) - open/close 6902 (time 1761317739.52 total 80.08 last 89.28) - open/close 7741 (time 1761317749.52 total 90.08 last 83.88) - open/close 8728 (time 1761317759.52 total 100.08 last 98.66) - open/close 9678 (time 1761317769.53 total 110.09 last 94.92) total: 10000 open/close in 113.65 seconds: 87.99 ops/second - unlinked 0 (time 1761317776 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 892 (time 1761317851.04 total 10.01 last 89.11) - open/close 1812 (time 1761317861.05 total 20.02 last 91.90) - open/close 2736 (time 1761317871.06 total 30.03 last 92.36) - open/close 3710 (time 1761317881.06 total 40.03 last 97.35) - open/close 4608 (time 1761317891.06 total 50.03 last 89.80) - open/close 5457 (time 1761317901.08 total 60.04 last 84.79) - open/close 6318 (time 1761317911.08 total 70.04 last 86.10) - open/close 7183 (time 1761317921.08 total 80.05 last 86.49) - open/close 7971 (time 1761317931.09 total 90.05 last 78.72) - open/close 8896 (time 1761317941.10 total 100.07 last 92.39) - open/close 9834 (time 1761317951.11 total 110.08 last 93.71) total: 10000 open/close in 111.94 seconds: 89.33 ops/second - unlinked 0 (time 1761317955 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 831 (time 1761318033.31 total 10.00 last 83.09) - open/close 1694 (time 1761318043.31 total 20.01 last 86.26) - open/close 2691 (time 1761318053.31 total 30.01 last 99.69) - open/close 3601 (time 1761318063.31 total 40.01 last 90.98) - open/close 4564 (time 1761318073.33 total 50.02 last 96.16) - open/close 5384 (time 1761318083.33 total 60.03 last 81.96) - open/close 6210 (time 1761318093.34 total 70.04 last 82.54) - open/close 7116 (time 1761318103.35 total 80.05 last 90.50) - open/close 7994 (time 1761318113.36 total 90.05 last 87.77) - open/close 8862 (time 1761318123.37 total 100.06 last 86.72) - open/close 9760 (time 1761318133.37 total 110.06 last 89.76) total: 10000 open/close in 112.39 seconds: 88.97 ops/second - unlinked 0 (time 1761318138 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 887 (time 1761318204.03 total 10.00 last 88.69) - open/close 1813 (time 1761318214.04 total 20.01 last 92.52) - open/close 2781 (time 1761318224.04 total 30.01 last 96.79) - open/close 3719 (time 1761318234.04 total 40.01 last 93.79) - open/close 4601 (time 1761318244.04 total 50.02 last 88.16) - open/close 5463 (time 1761318254.06 total 60.03 last 86.04) - open/close 6327 (time 1761318264.07 total 70.04 last 86.35) - open/close 7211 (time 1761318274.07 total 80.04 last 88.38) - open/close 8115 (time 1761318284.08 total 90.05 last 90.31) - open/close 8963 (time 1761318294.08 total 100.05 last 84.78) - open/close 9860 (time 1761318304.09 total 110.07 last 89.58) total: 10000 open/close in 111.58 seconds: 89.63 ops/second - unlinked 0 (time 1761318308 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg134-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg134-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg134-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg134-server: LNET ready to unload oleg134-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2