== conf-sanity test 69: replace an OST with the same index ========================================================== 06:54:18 (1753440858) start mds service on oleg147-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing load_modules_local oleg147-server: Loading modules from /home/green/git/lustre-release/lustre oleg147-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg147-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg147-server: detected 4 online CPUs by sysfs oleg147-server: MODOPTS_LIBCFS= oleg147-server: Force libcfs to create 2 CPU partitions oleg147-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg147-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg147-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg147-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1390 (time 1753440914.25 total 10.00 last 138.94) - open/close 3133 (time 1753440924.25 total 20.01 last 174.25) - open/close 4614 (time 1753440934.25 total 30.01 last 148.07) - open/close 5924 (time 1753440944.26 total 40.01 last 130.94) - open/close 7439 (time 1753440954.26 total 50.02 last 151.42) - open/close 9000 (time 1753440964.26 total 60.02 last 156.07) total: 10000 open/close in 65.77 seconds: 152.04 ops/second - unlinked 0 (time 1753440972 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 632 (time 1753441052.36 total 10.01 last 63.13) - open/close 1228 (time 1753441062.37 total 20.02 last 59.55) - open/close 1820 (time 1753441072.37 total 30.02 last 59.18) - open/close 2578 (time 1753441082.38 total 40.03 last 75.73) - open/close 3375 (time 1753441092.39 total 50.04 last 79.64) - open/close 4156 (time 1753441102.40 total 60.06 last 77.97) - open/close 4925 (time 1753441112.41 total 70.07 last 76.83) - open/close 5730 (time 1753441122.42 total 80.07 last 80.45) - open/close 6516 (time 1753441132.42 total 90.07 last 78.59) - open/close 7296 (time 1753441142.42 total 100.08 last 77.98) - open/close 8078 (time 1753441152.42 total 110.08 last 78.19) - open/close 8837 (time 1753441162.44 total 120.09 last 75.81) - open/close 9603 (time 1753441172.44 total 130.10 last 76.54) total: 10000 open/close in 134.92 seconds: 74.12 ops/second - unlinked 0 (time 1753441180 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 763 (time 1753441271.34 total 10.01 last 76.26) - open/close 1471 (time 1753441281.36 total 20.02 last 70.69) - open/close 2189 (time 1753441291.38 total 30.04 last 71.66) - open/close 2950 (time 1753441301.38 total 40.04 last 76.08) - open/close 3696 (time 1753441311.39 total 50.05 last 74.55) - open/close 4441 (time 1753441321.40 total 60.06 last 74.40) - open/close 5173 (time 1753441331.41 total 70.07 last 73.14) - open/close 5974 (time 1753441341.41 total 80.08 last 80.07) - open/close 6761 (time 1753441351.42 total 90.08 last 78.69) - open/close 7554 (time 1753441361.42 total 100.08 last 79.29) - open/close 8325 (time 1753441371.42 total 110.09 last 77.04) - open/close 9099 (time 1753441381.43 total 120.09 last 77.33) - open/close 9841 (time 1753441391.44 total 130.10 last 74.17) total: 10000 open/close in 132.26 seconds: 75.61 ops/second - unlinked 0 (time 1753441396 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 887 (time 1753441480.77 total 10.00 last 88.66) - open/close 1673 (time 1753441490.79 total 20.02 last 78.50) - open/close 2600 (time 1753441500.79 total 30.02 last 92.68) - open/close 3689 (time 1753441510.79 total 40.02 last 108.85) - open/close 4682 (time 1753441520.80 total 50.03 last 99.25) - open/close 5552 (time 1753441530.80 total 60.03 last 86.98) - open/close 6414 (time 1753441540.80 total 70.03 last 86.19) - open/close 7454 (time 1753441550.80 total 80.03 last 104.00) - open/close 8356 (time 1753441560.82 total 90.05 last 90.10) - open/close 9356 (time 1753441570.82 total 100.05 last 99.96) total: 10000 open/close in 106.93 seconds: 93.52 ops/second - unlinked 0 (time 1753441579 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1178 (time 1753441641.08 total 10.01 last 117.70) - open/close 2553 (time 1753441651.09 total 20.02 last 137.40) - open/close 3758 (time 1753441661.10 total 30.03 last 120.38) - open/close 4846 (time 1753441671.11 total 40.04 last 108.70) - open/close 5895 (time 1753441681.12 total 50.05 last 104.77) - open/close 7112 (time 1753441691.13 total 60.06 last 121.61) - open/close 8481 (time 1753441701.13 total 70.06 last 136.86) - open/close 9738 (time 1753441711.14 total 80.07 last 125.58) total: 10000 open/close in 81.95 seconds: 122.02 ops/second - unlinked 0 (time 1753441714 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1399 (time 1753441771.76 total 10.01 last 139.81) - open/close 2730 (time 1753441781.77 total 20.01 last 133.06) - open/close 3987 (time 1753441791.77 total 30.01 last 125.70) - open/close 5020 (time 1753441801.77 total 40.01 last 103.29) - open/close 6114 (time 1753441811.77 total 50.02 last 109.32) - open/close 7487 (time 1753441821.78 total 60.02 last 137.22) - open/close 8913 (time 1753441831.79 total 70.03 last 142.52) total: 10000 open/close in 78.06 seconds: 128.11 ops/second - unlinked 0 (time 1753441841 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1644 (time 1753441895.11 total 10.01 last 164.29) - open/close 3270 (time 1753441905.11 total 20.01 last 162.59) - open/close 4836 (time 1753441915.11 total 30.01 last 156.51) - open/close 6451 (time 1753441925.11 total 40.02 last 161.45) - open/close 8357 (time 1753441935.12 total 50.02 last 190.51) total: 10000 open/close in 59.83 seconds: 167.14 ops/second - unlinked 0 (time 1753441946 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1592 (time 1753441998.13 total 10.00 last 159.15) - open/close 2910 (time 1753442008.13 total 20.00 last 131.80) - open/close 4144 (time 1753442018.13 total 30.01 last 123.34) - open/close 5422 (time 1753442028.13 total 40.01 last 127.80) - open/close 6674 (time 1753442038.14 total 50.02 last 125.09) - open/close 7669 (time 1753442048.15 total 60.02 last 99.43) - open/close 8790 (time 1753442058.15 total 70.03 last 112.03) total: 10000 open/close in 79.91 seconds: 125.13 ops/second - unlinked 0 (time 1753442070 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1574 (time 1753442123.13 total 10.00 last 157.40) - open/close 2889 (time 1753442133.14 total 20.01 last 131.42) - open/close 3943 (time 1753442143.15 total 30.02 last 105.26) - open/close 5154 (time 1753442153.15 total 40.02 last 121.06) - open/close 6548 (time 1753442163.16 total 50.03 last 139.28) - open/close 7775 (time 1753442173.17 total 60.03 last 122.66) - open/close 9104 (time 1753442183.17 total 70.03 last 132.90) total: 10000 open/close in 76.10 seconds: 131.41 ops/second - unlinked 0 (time 1753442191 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1679 (time 1753442232.40 total 10.00 last 167.82) - open/close 3371 (time 1753442242.40 total 20.01 last 169.13) - open/close 5089 (time 1753442252.41 total 30.01 last 171.74) - open/close 6766 (time 1753442262.41 total 40.02 last 167.60) - open/close 8524 (time 1753442272.42 total 50.02 last 175.73) total: 10000 open/close in 56.97 seconds: 175.53 ops/second - unlinked 0 (time 1753442280 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg147-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg147-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.147@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg147-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg147-server: oleg147-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg147-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg147-server: oleg147-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg147-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg147-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg147-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server stop mds service on oleg147-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg147-server stop mds service on oleg147-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg147-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing unload_modules_local oleg147-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg147-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2