== conf-sanity test 69: replace an OST with the same index ========================================================== 17:36:40 (1763332600) start mds service on oleg201-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg201-server: detected 4 online CPUs by sysfs oleg201-server: MODOPTS_LIBCFS= oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg201-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 693 (time 1763332692.50 total 10.01 last 69.26) - open/close 1484 (time 1763332702.51 total 20.02 last 78.99) - open/close 2212 (time 1763332712.51 total 30.02 last 72.80) - open/close 2842 (time 1763332722.52 total 40.03 last 62.94) - open/close 3548 (time 1763332732.53 total 50.04 last 70.50) - open/close 4244 (time 1763332742.54 total 60.05 last 69.56) - open/close 4974 (time 1763332752.54 total 70.05 last 72.97) - open/close 5754 (time 1763332762.55 total 80.06 last 77.94) - open/close 6486 (time 1763332772.57 total 90.07 last 73.09) - open/close 7308 (time 1763332782.57 total 100.08 last 82.19) - open/close 7996 (time 1763332792.58 total 110.09 last 68.72) - open/close 8808 (time 1763332802.58 total 120.09 last 81.18) - open/close 9552 (time 1763332812.59 total 130.10 last 74.32) total: 10000 open/close in 135.85 seconds: 73.61 ops/second - unlinked 0 (time 1763332822 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 729 (time 1763332920.31 total 10.00 last 72.90) - open/close 1492 (time 1763332930.32 total 20.01 last 76.26) - open/close 2235 (time 1763332940.32 total 30.01 last 74.26) - open/close 3099 (time 1763332950.32 total 40.01 last 86.38) - open/close 3835 (time 1763332960.33 total 50.02 last 73.54) - open/close 4524 (time 1763332970.34 total 60.03 last 68.85) - open/close 5158 (time 1763332980.34 total 70.03 last 63.38) - open/close 5926 (time 1763332990.36 total 80.05 last 76.71) - open/close 6746 (time 1763333000.37 total 90.06 last 81.91) - open/close 7543 (time 1763333010.38 total 100.07 last 79.57) - open/close 8289 (time 1763333020.39 total 110.08 last 74.52) - open/close 8987 (time 1763333030.40 total 120.09 last 69.78) - open/close 9941 (time 1763333040.40 total 130.09 last 95.40) total: 10000 open/close in 130.59 seconds: 76.57 ops/second - unlinked 0 (time 1763333043 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 869 (time 1763333121.91 total 10.01 last 86.81) - open/close 1732 (time 1763333131.92 total 20.02 last 86.25) - open/close 2648 (time 1763333141.92 total 30.02 last 91.55) - open/close 3616 (time 1763333151.93 total 40.03 last 96.74) - open/close 4489 (time 1763333161.93 total 50.03 last 87.27) - open/close 5340 (time 1763333171.94 total 60.04 last 84.99) - open/close 6221 (time 1763333181.94 total 70.04 last 88.10) - open/close 7147 (time 1763333191.95 total 80.05 last 92.54) - open/close 8003 (time 1763333201.95 total 90.05 last 85.60) - open/close 8796 (time 1763333211.97 total 100.07 last 79.15) - open/close 9707 (time 1763333221.98 total 110.07 last 91.06) total: 10000 open/close in 113.41 seconds: 88.18 ops/second - unlinked 0 (time 1763333227 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 939 (time 1763333299.28 total 10.00 last 93.89) - open/close 1815 (time 1763333309.29 total 20.01 last 87.53) - open/close 2730 (time 1763333319.30 total 30.01 last 91.47) - open/close 3632 (time 1763333329.31 total 40.02 last 90.11) - open/close 4556 (time 1763333339.31 total 50.03 last 92.31) - open/close 5533 (time 1763333349.33 total 60.04 last 97.58) - open/close 6444 (time 1763333359.34 total 70.05 last 91.02) - open/close 7333 (time 1763333369.34 total 80.06 last 88.84) - open/close 8191 (time 1763333379.36 total 90.08 last 85.67) - open/close 9049 (time 1763333389.37 total 100.09 last 85.66) - open/close 9866 (time 1763333399.38 total 110.10 last 81.62) total: 10000 open/close in 111.97 seconds: 89.31 ops/second - unlinked 0 (time 1763333405 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 790 (time 1763333492.67 total 10.00 last 78.97) - open/close 1617 (time 1763333502.67 total 20.01 last 82.65) - open/close 2476 (time 1763333512.68 total 30.02 last 85.81) - open/close 3303 (time 1763333522.69 total 40.02 last 82.67) - open/close 4095 (time 1763333532.69 total 50.03 last 79.19) - open/close 4842 (time 1763333542.69 total 60.03 last 74.66) - open/close 5631 (time 1763333552.70 total 70.04 last 78.84) - open/close 6481 (time 1763333562.70 total 80.04 last 84.97) - open/close 7226 (time 1763333572.71 total 90.05 last 74.45) - open/close 7841 (time 1763333582.72 total 100.06 last 61.42) - open/close 8463 (time 1763333592.72 total 110.06 last 62.20) - open/close 9289 (time 1763333602.73 total 120.07 last 82.55) total: 10000 open/close in 129.48 seconds: 77.23 ops/second - unlinked 0 (time 1763333615 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 965 (time 1763333690.91 total 10.01 last 96.43) - open/close 1854 (time 1763333700.92 total 20.01 last 88.85) - open/close 2739 (time 1763333710.93 total 30.02 last 88.43) - open/close 3612 (time 1763333720.93 total 40.02 last 87.27) - open/close 4480 (time 1763333730.93 total 50.03 last 86.78) - open/close 5282 (time 1763333740.95 total 60.04 last 80.05) - open/close 5982 (time 1763333750.97 total 70.06 last 69.88) - open/close 6729 (time 1763333760.97 total 80.07 last 74.66) - open/close 7563 (time 1763333770.98 total 90.07 last 83.35) - open/close 8434 (time 1763333780.99 total 100.08 last 87.03) - open/close 9241 (time 1763333791.00 total 110.09 last 80.63) total: 10000 open/close in 119.53 seconds: 83.66 ops/second - unlinked 0 (time 1763333803 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 848 (time 1763333882.48 total 10.01 last 84.71) - open/close 1713 (time 1763333892.48 total 20.02 last 86.43) - open/close 2618 (time 1763333902.48 total 30.02 last 90.50) - open/close 3576 (time 1763333912.49 total 40.02 last 95.75) - open/close 4441 (time 1763333922.49 total 50.02 last 86.50) - open/close 5337 (time 1763333932.50 total 60.03 last 89.53) - open/close 6133 (time 1763333942.50 total 70.04 last 79.54) - open/close 6989 (time 1763333952.51 total 80.05 last 85.55) - open/close 7806 (time 1763333962.51 total 90.05 last 81.69) - open/close 8650 (time 1763333972.51 total 100.05 last 84.39) - open/close 9561 (time 1763333982.52 total 110.05 last 91.06) total: 10000 open/close in 114.92 seconds: 87.02 ops/second - unlinked 0 (time 1763333990 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 775 (time 1763334069.33 total 10.00 last 77.48) - open/close 1616 (time 1763334079.34 total 20.01 last 84.07) - open/close 2476 (time 1763334089.34 total 30.01 last 85.94) - open/close 3270 (time 1763334099.35 total 40.01 last 79.39) - open/close 4164 (time 1763334109.35 total 50.02 last 89.34) - open/close 5055 (time 1763334119.36 total 60.03 last 89.01) - open/close 6035 (time 1763334129.37 total 70.04 last 97.90) - open/close 6976 (time 1763334139.38 total 80.05 last 94.03) - open/close 7889 (time 1763334149.38 total 90.05 last 91.29) - open/close 8683 (time 1763334159.38 total 100.05 last 79.38) - open/close 9438 (time 1763334169.40 total 110.07 last 75.35) total: 10000 open/close in 117.50 seconds: 85.10 ops/second - unlinked 0 (time 1763334180 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 922 (time 1763334255.80 total 10.00 last 92.17) - open/close 1758 (time 1763334265.81 total 20.01 last 83.52) - open/close 2667 (time 1763334275.82 total 30.02 last 90.83) - open/close 3425 (time 1763334285.82 total 40.02 last 75.80) - open/close 4197 (time 1763334295.83 total 50.04 last 77.07) - open/close 5038 (time 1763334305.84 total 60.04 last 84.05) - open/close 5867 (time 1763334315.84 total 70.04 last 82.90) - open/close 6787 (time 1763334325.85 total 80.05 last 91.93) - open/close 7744 (time 1763334335.85 total 90.06 last 95.64) - open/close 8636 (time 1763334345.85 total 100.06 last 89.18) - open/close 9548 (time 1763334355.86 total 110.06 last 91.15) total: 10000 open/close in 114.80 seconds: 87.11 ops/second - unlinked 0 (time 1763334363 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 906 (time 1763334439.05 total 10.01 last 90.50) - open/close 1806 (time 1763334449.05 total 20.01 last 89.98) - open/close 2786 (time 1763334459.05 total 30.01 last 97.99) - open/close 3886 (time 1763334469.06 total 40.02 last 109.96) - open/close 5026 (time 1763334479.07 total 50.03 last 113.81) - open/close 5895 (time 1763334489.07 total 60.04 last 86.88) - open/close 6769 (time 1763334499.09 total 70.05 last 87.26) - open/close 7648 (time 1763334509.10 total 80.06 last 87.82) - open/close 8493 (time 1763334519.10 total 90.06 last 84.48) - open/close 9278 (time 1763334529.10 total 100.06 last 78.50) total: 10000 open/close in 108.87 seconds: 91.85 ops/second - unlinked 0 (time 1763334541 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.101@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.101@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg201-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2