== conf-sanity test 69: replace an OST with the same index ========================================================== 16:31:16 (1764106276) start mds service on oleg131-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg131-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 2472 (time 1764106334.45 total 10.00 last 247.18) - open/close 4503 (time 1764106344.45 total 20.00 last 203.05) - open/close 6018 (time 1764106354.45 total 30.01 last 151.45) - open/close 7607 (time 1764106364.46 total 40.01 last 158.84) - open/close 9279 (time 1764106374.46 total 50.01 last 167.16) total: 10000 open/close in 54.15 seconds: 184.67 ops/second - unlinked 0 (time 1764106380 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1981 (time 1764106426.31 total 10.00 last 198.04) - open/close 4113 (time 1764106436.31 total 20.01 last 213.16) - open/close 5973 (time 1764106446.31 total 30.01 last 185.98) - open/close 7682 (time 1764106456.32 total 40.01 last 170.79) - open/close 9449 (time 1764106466.32 total 50.02 last 176.60) total: 10000 open/close in 54.24 seconds: 184.36 ops/second - unlinked 0 (time 1764106472 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1474 (time 1764106514.80 total 10.01 last 147.32) - open/close 2810 (time 1764106524.80 total 20.01 last 133.54) - open/close 4526 (time 1764106534.81 total 30.02 last 171.50) - open/close 6148 (time 1764106544.81 total 40.02 last 162.19) - open/close 7867 (time 1764106554.81 total 50.02 last 171.83) - open/close 9554 (time 1764106564.82 total 60.02 last 168.63) total: 10000 open/close in 62.48 seconds: 160.06 ops/second - unlinked 0 (time 1764106569 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1557 (time 1764106616.18 total 10.00 last 155.67) - open/close 2976 (time 1764106626.18 total 20.01 last 141.82) - open/close 4748 (time 1764106636.18 total 30.01 last 177.18) - open/close 6897 (time 1764106646.18 total 40.01 last 214.89) - open/close 8890 (time 1764106656.19 total 50.01 last 199.22) total: 10000 open/close in 57.59 seconds: 173.64 ops/second - unlinked 0 (time 1764106665 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1814 (time 1764106707.72 total 10.00 last 181.37) - open/close 3904 (time 1764106717.72 total 20.00 last 208.95) - open/close 5947 (time 1764106727.72 total 30.01 last 204.27) - open/close 7775 (time 1764106737.73 total 40.01 last 182.74) - open/close 9567 (time 1764106747.73 total 50.01 last 179.14) total: 10000 open/close in 52.50 seconds: 190.48 ops/second - unlinked 0 (time 1764106752 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1547 (time 1764106804.37 total 10.00 last 154.70) - open/close 2824 (time 1764106814.37 total 20.00 last 127.68) - open/close 4197 (time 1764106824.38 total 30.01 last 137.25) - open/close 5644 (time 1764106834.38 total 40.01 last 144.70) - open/close 6995 (time 1764106844.38 total 50.01 last 135.05) - open/close 8718 (time 1764106854.38 total 60.01 last 172.23) total: 10000 open/close in 67.17 seconds: 148.88 ops/second - unlinked 0 (time 1764106863 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2389 (time 1764106899.85 total 10.00 last 238.83) - open/close 4813 (time 1764106909.85 total 20.01 last 242.30) - open/close 7271 (time 1764106919.86 total 30.01 last 245.65) - open/close 8415 (time 1764106929.88 total 40.03 last 114.16) - open/close 9197 (time 1764106939.89 total 50.04 last 78.13) - open/close 9590 (time 1764106949.89 total 60.05 last 39.27) - open/close 9935 (time 1764106959.93 total 70.08 last 34.39) total: 10000 open/close in 72.08 seconds: 138.74 ops/second - unlinked 0 (time 1764106971 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 808 (time 1764107066.02 total 10.00 last 80.78) - open/close 1569 (time 1764107076.02 total 20.01 last 76.08) - open/close 2375 (time 1764107086.02 total 30.01 last 80.58) - open/close 3181 (time 1764107096.03 total 40.02 last 80.54) - open/close 4009 (time 1764107106.05 total 50.03 last 82.67) - open/close 4832 (time 1764107116.05 total 60.03 last 82.28) - open/close 5705 (time 1764107126.05 total 70.04 last 87.28) - open/close 6544 (time 1764107136.06 total 80.04 last 83.86) - open/close 7454 (time 1764107146.07 total 90.05 last 90.92) - open/close 8317 (time 1764107156.07 total 100.05 last 86.26) - open/close 9131 (time 1764107166.08 total 110.06 last 81.35) - open/close 9994 (time 1764107176.08 total 120.07 last 86.25) total: 10000 open/close in 120.13 seconds: 83.24 ops/second - unlinked 0 (time 1764107178 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 801 (time 1764107255.04 total 10.01 last 80.04) - open/close 1386 (time 1764107265.05 total 20.02 last 58.44) - open/close 1958 (time 1764107275.06 total 30.02 last 57.18) - open/close 2751 (time 1764107285.08 total 40.04 last 79.12) - open/close 3819 (time 1764107295.08 total 50.05 last 106.79) - open/close 4891 (time 1764107305.08 total 60.05 last 107.19) - open/close 5919 (time 1764107315.09 total 70.05 last 102.75) - open/close 7150 (time 1764107325.09 total 80.05 last 123.09) - open/close 8024 (time 1764107335.09 total 90.05 last 87.38) - open/close 8797 (time 1764107345.10 total 100.06 last 77.23) - open/close 9592 (time 1764107355.11 total 110.07 last 79.43) total: 10000 open/close in 114.81 seconds: 87.10 ops/second - unlinked 0 (time 1764107362 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1207 (time 1764107417.67 total 10.00 last 120.70) - open/close 2355 (time 1764107427.68 total 20.01 last 114.72) - open/close 3344 (time 1764107437.69 total 30.02 last 98.82) - open/close 4565 (time 1764107447.69 total 40.02 last 122.09) - open/close 5860 (time 1764107457.69 total 50.02 last 129.42) - open/close 7798 (time 1764107467.69 total 60.02 last 193.80) total: 10000 open/close in 68.46 seconds: 146.07 ops/second - unlinked 0 (time 1764107477 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg131-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg131-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg131-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2