== conf-sanity test 69: replace an OST with the same index ========================================================== 19:52:15 (1763340735) start mds service on oleg139-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg139-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg139-server: detected 4 online CPUs by sysfs oleg139-server: MODOPTS_LIBCFS= oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg139-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg139-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 801 (time 1763340824.72 total 10.01 last 80.04) - open/close 1567 (time 1763340834.73 total 20.02 last 76.51) - open/close 2369 (time 1763340844.74 total 30.03 last 80.14) - open/close 3103 (time 1763340854.74 total 40.03 last 73.38) - open/close 3889 (time 1763340864.76 total 50.04 last 78.50) - open/close 4615 (time 1763340874.76 total 60.05 last 72.58) - open/close 5399 (time 1763340884.77 total 70.06 last 78.31) - open/close 6215 (time 1763340894.78 total 80.06 last 81.54) - open/close 7019 (time 1763340904.78 total 90.07 last 80.37) - open/close 7801 (time 1763340914.78 total 100.07 last 78.19) - open/close 8593 (time 1763340924.79 total 110.07 last 79.15) - open/close 9428 (time 1763340934.80 total 120.08 last 83.42) total: 10000 open/close in 127.24 seconds: 78.59 ops/second - unlinked 0 (time 1763340945 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 782 (time 1763341029.09 total 10.01 last 78.15) - open/close 1584 (time 1763341039.09 total 20.01 last 80.14) - open/close 2337 (time 1763341049.10 total 30.02 last 75.27) - open/close 3070 (time 1763341059.10 total 40.02 last 73.24) - open/close 3853 (time 1763341069.12 total 50.04 last 78.19) - open/close 4635 (time 1763341079.13 total 60.05 last 78.11) - open/close 5399 (time 1763341089.14 total 70.06 last 76.33) - open/close 6130 (time 1763341099.15 total 80.07 last 73.06) - open/close 6956 (time 1763341109.15 total 90.07 last 82.57) - open/close 7687 (time 1763341119.16 total 100.08 last 73.03) - open/close 8442 (time 1763341129.17 total 110.09 last 75.40) - open/close 9187 (time 1763341139.18 total 120.10 last 74.46) - open/close 9873 (time 1763341149.20 total 130.12 last 68.48) total: 10000 open/close in 132.34 seconds: 75.57 ops/second - unlinked 0 (time 1763341155 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 737 (time 1763341243.19 total 10.00 last 73.70) - open/close 1480 (time 1763341253.20 total 20.01 last 74.26) - open/close 2215 (time 1763341263.21 total 30.02 last 73.43) - open/close 2917 (time 1763341273.22 total 40.02 last 70.15) - open/close 3659 (time 1763341283.22 total 50.03 last 74.15) - open/close 4432 (time 1763341293.23 total 60.04 last 77.23) - open/close 5147 (time 1763341303.24 total 70.05 last 71.42) - open/close 5876 (time 1763341313.25 total 80.05 last 72.88) - open/close 6638 (time 1763341323.25 total 90.06 last 76.18) - open/close 7445 (time 1763341333.27 total 100.07 last 80.56) - open/close 8168 (time 1763341343.27 total 110.07 last 72.29) - open/close 8877 (time 1763341353.27 total 120.07 last 70.89) - open/close 9587 (time 1763341363.27 total 130.08 last 70.97) total: 10000 open/close in 136.24 seconds: 73.40 ops/second - unlinked 0 (time 1763341373 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 593 (time 1763341469.02 total 10.00 last 59.29) - open/close 1277 (time 1763341479.02 total 20.01 last 68.36) - open/close 2055 (time 1763341489.03 total 30.01 last 77.78) - open/close 2793 (time 1763341499.03 total 40.02 last 73.76) - open/close 3514 (time 1763341509.03 total 50.02 last 72.09) - open/close 4183 (time 1763341519.04 total 60.02 last 66.87) - open/close 4904 (time 1763341529.05 total 70.04 last 71.98) - open/close 5683 (time 1763341539.06 total 80.04 last 77.86) - open/close 6399 (time 1763341549.06 total 90.05 last 71.55) - open/close 7133 (time 1763341559.08 total 100.06 last 73.31) - open/close 7935 (time 1763341569.08 total 110.07 last 80.16) - open/close 8695 (time 1763341579.14 total 120.12 last 75.58) - open/close 9404 (time 1763341589.14 total 130.13 last 70.86) total: 10000 open/close in 137.64 seconds: 72.65 ops/second - unlinked 0 (time 1763341601 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 693 (time 1763341693.80 total 10.01 last 69.25) - open/close 1451 (time 1763341703.80 total 20.01 last 75.79) - open/close 2230 (time 1763341713.80 total 30.01 last 77.90) - open/close 2980 (time 1763341723.82 total 40.02 last 74.90) - open/close 3716 (time 1763341733.82 total 50.02 last 73.58) - open/close 4497 (time 1763341743.84 total 60.04 last 77.97) - open/close 5296 (time 1763341753.84 total 70.04 last 79.88) - open/close 6097 (time 1763341763.84 total 80.04 last 80.10) - open/close 6946 (time 1763341773.84 total 90.05 last 84.86) - open/close 7755 (time 1763341783.85 total 100.05 last 80.85) - open/close 8564 (time 1763341793.86 total 110.06 last 80.85) - open/close 9491 (time 1763341803.86 total 120.06 last 92.69) total: 10000 open/close in 125.78 seconds: 79.50 ops/second - unlinked 0 (time 1763341812 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 755 (time 1763341900.10 total 10.01 last 75.42) - open/close 1532 (time 1763341910.10 total 20.01 last 77.67) - open/close 2321 (time 1763341920.11 total 30.02 last 78.85) - open/close 3072 (time 1763341930.12 total 40.03 last 75.04) - open/close 3861 (time 1763341940.12 total 50.03 last 78.86) - open/close 4672 (time 1763341950.13 total 60.04 last 81.06) - open/close 5463 (time 1763341960.14 total 70.05 last 79.04) - open/close 6284 (time 1763341970.14 total 80.05 last 82.06) - open/close 7097 (time 1763341980.14 total 90.05 last 81.28) - open/close 7844 (time 1763341990.15 total 100.06 last 74.67) - open/close 8632 (time 1763342000.16 total 110.07 last 78.70) - open/close 9387 (time 1763342010.16 total 120.07 last 75.48) total: 10000 open/close in 128.34 seconds: 77.92 ops/second - unlinked 0 (time 1763342021 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 883 (time 1763342109.47 total 10.01 last 88.23) - open/close 1663 (time 1763342119.48 total 20.02 last 77.92) - open/close 2422 (time 1763342129.51 total 30.04 last 75.70) - open/close 3197 (time 1763342139.52 total 40.05 last 77.42) - open/close 3964 (time 1763342149.53 total 50.07 last 76.61) - open/close 4745 (time 1763342159.55 total 60.08 last 77.99) - open/close 5441 (time 1763342169.55 total 70.08 last 69.59) - open/close 6138 (time 1763342179.56 total 80.09 last 69.64) - open/close 6828 (time 1763342189.56 total 90.10 last 68.94) - open/close 7513 (time 1763342199.58 total 100.12 last 68.38) - open/close 8237 (time 1763342209.59 total 110.13 last 72.33) - open/close 9007 (time 1763342219.60 total 120.13 last 76.96) - open/close 9716 (time 1763342229.61 total 130.15 last 70.77) total: 10000 open/close in 134.66 seconds: 74.26 ops/second - unlinked 0 (time 1763342237 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 731 (time 1763342335.70 total 10.01 last 73.04) - open/close 1428 (time 1763342345.71 total 20.02 last 69.66) - open/close 2144 (time 1763342355.72 total 30.03 last 71.53) - open/close 2840 (time 1763342365.72 total 40.03 last 69.56) - open/close 3592 (time 1763342375.73 total 50.03 last 75.18) - open/close 4320 (time 1763342385.73 total 60.04 last 72.76) - open/close 5068 (time 1763342395.74 total 70.04 last 74.77) - open/close 5775 (time 1763342405.74 total 80.05 last 70.69) - open/close 6423 (time 1763342415.74 total 90.05 last 64.77) - open/close 7087 (time 1763342425.76 total 100.07 last 66.29) - open/close 7833 (time 1763342435.76 total 110.07 last 74.57) - open/close 8585 (time 1763342445.77 total 120.07 last 75.17) - open/close 9275 (time 1763342455.77 total 130.08 last 68.99) - open/close 9939 (time 1763342465.78 total 140.09 last 66.31) total: 10000 open/close in 141.05 seconds: 70.90 ops/second - unlinked 0 (time 1763342470 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 812 (time 1763342563.75 total 10.01 last 81.11) - open/close 1568 (time 1763342573.77 total 20.03 last 75.45) - open/close 2363 (time 1763342583.77 total 30.03 last 79.49) - open/close 3135 (time 1763342593.78 total 40.04 last 77.14) - open/close 3905 (time 1763342603.80 total 50.06 last 76.86) - open/close 4660 (time 1763342613.80 total 60.06 last 75.48) - open/close 5428 (time 1763342623.80 total 70.07 last 76.76) - open/close 6195 (time 1763342633.81 total 80.07 last 76.69) - open/close 6962 (time 1763342643.81 total 90.08 last 76.64) - open/close 7717 (time 1763342653.82 total 100.08 last 75.47) - open/close 8481 (time 1763342663.83 total 110.09 last 76.34) - open/close 9246 (time 1763342673.85 total 120.11 last 76.33) - open/close 9993 (time 1763342683.85 total 130.11 last 74.69) total: 10000 open/close in 130.20 seconds: 76.80 ops/second - unlinked 0 (time 1763342687 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 777 (time 1763342775.72 total 10.00 last 77.67) - open/close 1536 (time 1763342785.72 total 20.00 last 75.90) - open/close 2295 (time 1763342795.73 total 30.01 last 75.84) - open/close 2973 (time 1763342805.74 total 40.02 last 67.73) - open/close 3714 (time 1763342815.75 total 50.03 last 74.03) - open/close 4474 (time 1763342825.76 total 60.04 last 75.92) - open/close 5239 (time 1763342835.76 total 70.04 last 76.47) - open/close 5994 (time 1763342845.77 total 80.06 last 75.40) - open/close 6725 (time 1763342855.78 total 90.07 last 73.05) - open/close 7453 (time 1763342865.79 total 100.07 last 72.74) - open/close 8145 (time 1763342875.81 total 110.10 last 69.04) - open/close 8864 (time 1763342885.83 total 120.12 last 71.76) - open/close 9525 (time 1763342895.83 total 130.12 last 66.07) total: 10000 open/close in 136.98 seconds: 73.00 ops/second - unlinked 0 (time 1763342906 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg139-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg139-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg139-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg139-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg139-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local oleg139-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg139-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2