== conf-sanity test 69: replace an OST with the same index ========================================================== 04:24:35 (1769073875) start mds service on oleg157-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing load_modules_local oleg157-server: Loading modules from /home/green/git/lustre-release/lustre oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg157-server: detected 4 online CPUs by sysfs oleg157-server: MODOPTS_LIBCFS= oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg157-server: Force libcfs to create 2 CPU partitions oleg157-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg157-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg157-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 707 (time 1769074000.68 total 10.01 last 70.60) - open/close 1506 (time 1769074010.68 total 20.02 last 79.89) - open/close 2308 (time 1769074020.68 total 30.02 last 80.20) - open/close 3094 (time 1769074030.68 total 40.02 last 78.59) - open/close 3818 (time 1769074040.70 total 50.03 last 72.30) - open/close 4556 (time 1769074050.71 total 60.04 last 73.73) - open/close 5418 (time 1769074060.73 total 70.06 last 86.04) - open/close 6119 (time 1769074070.73 total 80.06 last 70.10) - open/close 6944 (time 1769074080.73 total 90.06 last 82.47) - open/close 7665 (time 1769074090.73 total 100.07 last 72.08) - open/close 8348 (time 1769074100.74 total 110.08 last 68.24) - open/close 9192 (time 1769074110.75 total 120.08 last 84.34) - open/close 9968 (time 1769074120.76 total 130.10 last 77.49) total: 10000 open/close in 130.42 seconds: 76.68 ops/second - unlinked 0 (time 1769074126 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 756 (time 1769074215.98 total 10.00 last 75.59) - open/close 1487 (time 1769074225.98 total 20.01 last 73.05) - open/close 2294 (time 1769074235.99 total 30.01 last 80.66) - open/close 3015 (time 1769074245.99 total 40.01 last 72.08) - open/close 3738 (time 1769074255.99 total 50.02 last 72.27) - open/close 4638 (time 1769074266.00 total 60.02 last 89.96) - open/close 5592 (time 1769074276.00 total 70.03 last 95.36) - open/close 6595 (time 1769074286.01 total 80.04 last 100.19) - open/close 7502 (time 1769074296.02 total 90.04 last 90.65) - open/close 8311 (time 1769074306.03 total 100.05 last 80.81) - open/close 9351 (time 1769074316.04 total 110.06 last 103.93) total: 10000 open/close in 117.02 seconds: 85.46 ops/second - unlinked 0 (time 1769074327 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 588 (time 1769074408.89 total 10.01 last 58.72) - open/close 1412 (time 1769074418.89 total 20.02 last 82.39) - open/close 2103 (time 1769074428.89 total 30.02 last 69.09) - open/close 2876 (time 1769074438.90 total 40.02 last 77.25) - open/close 3762 (time 1769074448.91 total 50.04 last 88.49) - open/close 4540 (time 1769074458.93 total 60.05 last 77.66) - open/close 5381 (time 1769074468.94 total 70.07 last 83.98) - open/close 6249 (time 1769074478.95 total 80.07 last 86.77) - open/close 7020 (time 1769074488.98 total 90.10 last 76.86) - open/close 7703 (time 1769074498.98 total 100.11 last 68.27) - open/close 8512 (time 1769074508.99 total 110.11 last 80.89) - open/close 9380 (time 1769074518.99 total 120.12 last 86.73) total: 10000 open/close in 128.02 seconds: 78.11 ops/second - unlinked 0 (time 1769074531 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 647 (time 1769074621.60 total 10.02 last 64.59) - open/close 1339 (time 1769074631.60 total 20.03 last 69.14) - open/close 1848 (time 1769074641.62 total 30.04 last 50.84) - open/close 2628 (time 1769074651.62 total 40.04 last 77.94) - open/close 3563 (time 1769074661.63 total 50.05 last 93.42) - open/close 4419 (time 1769074671.63 total 60.05 last 85.60) - open/close 5395 (time 1769074681.64 total 70.06 last 97.57) - open/close 6370 (time 1769074691.64 total 80.06 last 97.46) - open/close 7336 (time 1769074701.65 total 90.07 last 96.48) - open/close 8337 (time 1769074711.66 total 100.08 last 100.03) - open/close 9400 (time 1769074721.66 total 110.08 last 106.29) total: 10000 open/close in 117.23 seconds: 85.30 ops/second - unlinked 0 (time 1769074733 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 895 (time 1769074812.58 total 10.00 last 89.47) - open/close 1663 (time 1769074822.59 total 20.01 last 76.74) - open/close 2744 (time 1769074832.60 total 30.02 last 107.98) - open/close 3738 (time 1769074842.60 total 40.02 last 99.38) - open/close 4575 (time 1769074852.61 total 50.04 last 83.60) - open/close 5518 (time 1769074862.61 total 60.04 last 94.27) - open/close 6533 (time 1769074872.62 total 70.04 last 101.47) - open/close 7374 (time 1769074882.62 total 80.05 last 84.08) - open/close 8257 (time 1769074892.63 total 90.05 last 88.24) - open/close 9129 (time 1769074902.64 total 100.07 last 87.08) - open/close 9964 (time 1769074912.65 total 110.08 last 83.40) total: 10000 open/close in 110.70 seconds: 90.33 ops/second - unlinked 0 (time 1769074918 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 848 (time 1769074996.28 total 10.00 last 84.79) - open/close 1665 (time 1769075006.28 total 20.01 last 81.66) - open/close 2700 (time 1769075016.29 total 30.01 last 103.48) - open/close 3649 (time 1769075026.29 total 40.01 last 94.88) - open/close 4575 (time 1769075036.29 total 50.01 last 92.55) - open/close 5483 (time 1769075046.29 total 60.02 last 90.77) - open/close 6467 (time 1769075056.30 total 70.02 last 98.39) - open/close 7241 (time 1769075066.30 total 80.02 last 77.38) - open/close 7889 (time 1769075076.31 total 90.03 last 64.74) - open/close 8828 (time 1769075086.31 total 100.03 last 93.87) - open/close 9671 (time 1769075096.33 total 110.05 last 84.18) total: 10000 open/close in 115.80 seconds: 86.35 ops/second - unlinked 0 (time 1769075106 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 749 (time 1769075190.35 total 10.00 last 74.87) - open/close 1356 (time 1769075200.36 total 20.01 last 60.68) - open/close 2211 (time 1769075210.37 total 30.02 last 85.43) - open/close 2854 (time 1769075220.37 total 40.02 last 64.26) - open/close 3754 (time 1769075230.37 total 50.02 last 89.98) - open/close 4642 (time 1769075240.38 total 60.03 last 88.75) - open/close 5534 (time 1769075250.38 total 70.03 last 89.20) - open/close 6499 (time 1769075260.39 total 80.04 last 96.43) - open/close 7475 (time 1769075270.40 total 90.05 last 97.52) - open/close 8304 (time 1769075280.40 total 100.05 last 82.85) - open/close 9216 (time 1769075290.42 total 110.07 last 91.07) total: 10000 open/close in 119.96 seconds: 83.36 ops/second - unlinked 0 (time 1769075305 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 867 (time 1769075385.46 total 10.01 last 86.63) - open/close 1705 (time 1769075395.47 total 20.01 last 83.77) - open/close 2743 (time 1769075405.47 total 30.02 last 103.74) - open/close 3739 (time 1769075415.48 total 40.02 last 99.54) - open/close 4726 (time 1769075425.48 total 50.03 last 98.66) - open/close 5666 (time 1769075435.50 total 60.04 last 93.86) - open/close 6551 (time 1769075445.51 total 70.05 last 88.42) - open/close 7437 (time 1769075455.51 total 80.06 last 88.53) - open/close 8163 (time 1769075465.52 total 90.06 last 72.58) - open/close 8965 (time 1769075475.52 total 100.07 last 80.17) - open/close 9784 (time 1769075485.53 total 110.07 last 81.83) total: 10000 open/close in 113.20 seconds: 88.34 ops/second - unlinked 0 (time 1769075494 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 847 (time 1769075584.36 total 10.00 last 84.69) - open/close 1595 (time 1769075594.38 total 20.01 last 74.71) - open/close 2656 (time 1769075604.38 total 30.02 last 106.04) - open/close 3524 (time 1769075614.39 total 40.03 last 86.74) - open/close 4321 (time 1769075624.40 total 50.03 last 79.65) - open/close 5123 (time 1769075634.40 total 60.04 last 80.15) - open/close 5966 (time 1769075644.40 total 70.04 last 84.30) - open/close 6754 (time 1769075654.41 total 80.04 last 78.76) - open/close 7493 (time 1769075664.43 total 90.06 last 73.76) - open/close 8310 (time 1769075674.44 total 100.07 last 81.62) - open/close 9062 (time 1769075684.44 total 110.08 last 75.17) - open/close 9800 (time 1769075694.45 total 120.09 last 73.74) total: 10000 open/close in 122.13 seconds: 81.88 ops/second - unlinked 0 (time 1769075701 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 770 (time 1769075785.67 total 10.01 last 76.92) - open/close 1586 (time 1769075795.69 total 20.02 last 81.50) - open/close 2437 (time 1769075805.69 total 30.03 last 85.03) - open/close 3319 (time 1769075815.70 total 40.04 last 88.15) - open/close 4204 (time 1769075825.71 total 50.04 last 88.46) - open/close 5206 (time 1769075835.71 total 60.05 last 100.16) - open/close 6218 (time 1769075845.71 total 70.05 last 101.16) - open/close 7089 (time 1769075855.72 total 80.05 last 87.08) - open/close 7932 (time 1769075865.73 total 90.07 last 84.19) - open/close 8844 (time 1769075875.73 total 100.07 last 91.18) - open/close 9668 (time 1769075885.73 total 110.07 last 82.38) total: 10000 open/close in 114.81 seconds: 87.10 ops/second - unlinked 0 (time 1769075894 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg157-server: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg157-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg157-server: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg157-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.157@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.157@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg157-server: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg157-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg157-server: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg157-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg157-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing unload_modules_local oleg157-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg157-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2