== conf-sanity test 69: replace an OST with the same index ========================================================== 09:35:04 (1766414104) start mds service on oleg230-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions oleg230-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg230-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg230-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg230-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 891 (time 1766414198.65 total 10.00 last 89.10) - open/close 1819 (time 1766414208.65 total 20.00 last 92.80) - open/close 2794 (time 1766414218.65 total 30.00 last 97.50) - open/close 3762 (time 1766414228.66 total 40.01 last 96.73) - open/close 4690 (time 1766414238.67 total 50.01 last 92.74) - open/close 5649 (time 1766414248.67 total 60.02 last 95.88) - open/close 6626 (time 1766414258.67 total 70.02 last 97.66) - open/close 7561 (time 1766414268.68 total 80.02 last 93.47) - open/close 8468 (time 1766414278.69 total 90.03 last 90.62) - open/close 9396 (time 1766414288.69 total 100.04 last 92.75) total: 10000 open/close in 106.87 seconds: 93.57 ops/second - unlinked 0 (time 1766414299 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 921 (time 1766414374.39 total 10.01 last 92.03) - open/close 1798 (time 1766414384.41 total 20.02 last 87.58) - open/close 2686 (time 1766414394.41 total 30.03 last 88.75) - open/close 3620 (time 1766414404.41 total 40.03 last 93.39) - open/close 4505 (time 1766414414.42 total 50.03 last 88.47) - open/close 5415 (time 1766414424.42 total 60.03 last 91.00) - open/close 6329 (time 1766414434.43 total 70.04 last 91.29) - open/close 7246 (time 1766414444.44 total 80.05 last 91.59) - open/close 8046 (time 1766414454.44 total 90.06 last 79.98) - open/close 8899 (time 1766414464.45 total 100.06 last 85.25) - open/close 9805 (time 1766414474.45 total 110.06 last 90.59) total: 10000 open/close in 112.02 seconds: 89.27 ops/second - unlinked 0 (time 1766414479 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 759 (time 1766414562.34 total 10.02 last 75.77) - open/close 1571 (time 1766414572.35 total 20.03 last 81.13) - open/close 2505 (time 1766414582.36 total 30.03 last 93.34) - open/close 3434 (time 1766414592.36 total 40.04 last 92.86) - open/close 4462 (time 1766414602.37 total 50.04 last 102.72) - open/close 5388 (time 1766414612.37 total 60.05 last 92.57) - open/close 6257 (time 1766414622.39 total 70.06 last 86.79) - open/close 7257 (time 1766414632.39 total 80.07 last 99.93) - open/close 8189 (time 1766414642.40 total 90.07 last 93.17) - open/close 9113 (time 1766414652.40 total 100.08 last 92.34) - open/close 9915 (time 1766414662.42 total 110.09 last 80.06) total: 10000 open/close in 111.02 seconds: 90.07 ops/second - unlinked 0 (time 1766414667 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 843 (time 1766414743.48 total 10.00 last 84.27) - open/close 1672 (time 1766414753.50 total 20.02 last 82.74) - open/close 2475 (time 1766414763.51 total 30.03 last 80.25) - open/close 3301 (time 1766414773.51 total 40.03 last 82.59) - open/close 4131 (time 1766414783.51 total 50.03 last 82.97) - open/close 5078 (time 1766414793.52 total 60.04 last 94.62) - open/close 6054 (time 1766414803.53 total 70.05 last 97.51) - open/close 6954 (time 1766414813.54 total 80.06 last 89.89) - open/close 7949 (time 1766414823.55 total 90.07 last 99.45) - open/close 8973 (time 1766414833.56 total 100.08 last 102.24) - open/close 9881 (time 1766414843.57 total 110.09 last 90.76) total: 10000 open/close in 111.58 seconds: 89.62 ops/second - unlinked 0 (time 1766414848 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 957 (time 1766414923.53 total 10.01 last 95.63) - open/close 1830 (time 1766414933.54 total 20.01 last 87.28) - open/close 2699 (time 1766414943.54 total 30.01 last 86.87) - open/close 3575 (time 1766414953.54 total 40.02 last 87.57) - open/close 4611 (time 1766414963.54 total 50.02 last 103.58) - open/close 5489 (time 1766414973.55 total 60.03 last 87.72) - open/close 6329 (time 1766414983.56 total 70.03 last 83.94) - open/close 7216 (time 1766414993.58 total 80.05 last 88.53) - open/close 8165 (time 1766415003.59 total 90.06 last 94.82) - open/close 9097 (time 1766415013.60 total 100.07 last 93.08) total: 10000 open/close in 109.29 seconds: 91.50 ops/second - unlinked 0 (time 1766415025 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 964 (time 1766415099.17 total 10.01 last 96.30) - open/close 1928 (time 1766415109.18 total 20.02 last 96.29) - open/close 2865 (time 1766415119.19 total 30.03 last 93.64) - open/close 3865 (time 1766415129.19 total 40.03 last 99.97) - open/close 4873 (time 1766415139.20 total 50.04 last 100.73) - open/close 5839 (time 1766415149.20 total 60.04 last 96.60) - open/close 6805 (time 1766415159.21 total 70.05 last 96.53) - open/close 7786 (time 1766415169.22 total 80.06 last 98.01) - open/close 8739 (time 1766415179.23 total 90.07 last 95.17) - open/close 9706 (time 1766415189.24 total 100.07 last 96.65) total: 10000 open/close in 102.94 seconds: 97.15 ops/second - unlinked 0 (time 1766415195 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1007 (time 1766415266.55 total 10.01 last 100.61) - open/close 1993 (time 1766415276.55 total 20.01 last 98.55) - open/close 2991 (time 1766415286.55 total 30.02 last 99.78) - open/close 3999 (time 1766415296.55 total 40.02 last 100.80) - open/close 5034 (time 1766415306.56 total 50.02 last 103.46) - open/close 6118 (time 1766415316.56 total 60.02 last 108.37) - open/close 7127 (time 1766415326.57 total 70.03 last 100.83) - open/close 8152 (time 1766415336.58 total 80.04 last 102.40) - open/close 9226 (time 1766415346.58 total 90.05 last 107.34) total: 10000 open/close in 97.43 seconds: 102.64 ops/second - unlinked 0 (time 1766415356 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 923 (time 1766415430.21 total 10.01 last 92.25) - open/close 1918 (time 1766415440.22 total 20.02 last 99.40) - open/close 2900 (time 1766415450.23 total 30.02 last 98.14) - open/close 3873 (time 1766415460.23 total 40.03 last 97.22) - open/close 4879 (time 1766415470.24 total 50.04 last 100.54) - open/close 5927 (time 1766415480.24 total 60.04 last 104.78) - open/close 6935 (time 1766415490.25 total 70.04 last 100.74) - open/close 7975 (time 1766415500.25 total 80.05 last 103.98) - open/close 9177 (time 1766415510.25 total 90.05 last 120.16) total: 10000 open/close in 98.24 seconds: 101.79 ops/second - unlinked 0 (time 1766415521 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1073 (time 1766415583.69 total 10.00 last 107.28) - open/close 2016 (time 1766415593.70 total 20.02 last 94.17) - open/close 2980 (time 1766415603.71 total 30.02 last 96.36) - open/close 4071 (time 1766415613.71 total 40.03 last 109.04) - open/close 5098 (time 1766415623.73 total 50.04 last 102.57) - open/close 6086 (time 1766415633.73 total 60.04 last 98.78) - open/close 7150 (time 1766415643.74 total 70.05 last 106.33) - open/close 8283 (time 1766415653.74 total 80.05 last 113.24) - open/close 9295 (time 1766415663.75 total 90.07 last 101.08) total: 10000 open/close in 97.06 seconds: 103.02 ops/second - unlinked 0 (time 1766415673 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1119 (time 1766415736.20 total 10.00 last 111.88) - open/close 2366 (time 1766415746.20 total 20.00 last 124.66) - open/close 3621 (time 1766415756.20 total 30.01 last 125.50) - open/close 4885 (time 1766415766.20 total 40.01 last 126.40) - open/close 6115 (time 1766415776.21 total 50.01 last 122.92) - open/close 7232 (time 1766415786.21 total 60.02 last 111.64) - open/close 8394 (time 1766415796.22 total 70.02 last 116.14) - open/close 9436 (time 1766415806.22 total 80.03 last 104.15) total: 10000 open/close in 85.31 seconds: 117.23 ops/second - unlinked 0 (time 1766415813 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.130@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg230-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg230-server: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg230-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg230-server: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg230-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre On OST0, 14720 used inodes rc=0 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.130@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.130@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.130@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg230-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg230-server: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg230-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg230-server: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg230-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg230-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing unload_modules_local oleg230-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2