== conf-sanity test 69: replace an OST with the same index ========================================================== 08:57:56 (1761310676) start mds service on oleg238-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: Force libcfs to create 2 CPU partitions oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg238-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg238-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 1365 (time 1761310743.31 total 10.01 last 136.42) - open/close 2731 (time 1761310753.31 total 20.01 last 136.57) - open/close 4244 (time 1761310763.32 total 30.01 last 151.27) - open/close 5499 (time 1761310773.32 total 40.02 last 125.39) - open/close 6936 (time 1761310783.33 total 50.03 last 143.57) - open/close 8211 (time 1761310793.34 total 60.03 last 127.44) - open/close 9173 (time 1761310803.34 total 70.04 last 96.17) - open/close 9825 (time 1761310813.34 total 80.04 last 65.18) total: 10000 open/close in 82.81 seconds: 120.75 ops/second - unlinked 0 (time 1761310822 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 826 (time 1761310906.60 total 10.01 last 82.51) - open/close 1729 (time 1761310916.60 total 20.01 last 90.30) - open/close 2648 (time 1761310926.61 total 30.02 last 91.82) - open/close 3575 (time 1761310936.62 total 40.02 last 92.66) - open/close 4455 (time 1761310946.62 total 50.02 last 87.99) - open/close 5393 (time 1761310956.62 total 60.03 last 93.75) - open/close 6322 (time 1761310966.64 total 70.04 last 92.78) - open/close 7236 (time 1761310976.64 total 80.05 last 91.37) - open/close 8162 (time 1761310986.64 total 90.05 last 92.57) - open/close 9138 (time 1761310996.65 total 100.06 last 97.51) total: 10000 open/close in 108.70 seconds: 92.00 ops/second - unlinked 0 (time 1761311008 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1114 (time 1761311081.99 total 10.00 last 111.38) - open/close 2098 (time 1761311092.00 total 20.01 last 98.28) - open/close 3140 (time 1761311102.01 total 30.02 last 104.13) - open/close 4198 (time 1761311112.01 total 40.02 last 105.79) - open/close 5207 (time 1761311122.01 total 50.03 last 100.85) - open/close 6269 (time 1761311132.02 total 60.03 last 106.17) - open/close 7481 (time 1761311142.02 total 70.03 last 121.12) - open/close 8626 (time 1761311152.02 total 80.04 last 114.48) - open/close 9701 (time 1761311162.04 total 90.05 last 107.35) total: 10000 open/close in 92.59 seconds: 108.01 ops/second - unlinked 0 (time 1761311166 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1159 (time 1761311229.05 total 10.00 last 115.86) - open/close 2336 (time 1761311239.06 total 20.01 last 117.68) - open/close 3561 (time 1761311249.06 total 30.01 last 122.50) - open/close 4904 (time 1761311259.06 total 40.01 last 134.20) - open/close 6519 (time 1761311269.07 total 50.02 last 161.45) - open/close 7730 (time 1761311279.08 total 60.03 last 120.97) - open/close 8781 (time 1761311289.08 total 70.03 last 105.08) - open/close 9492 (time 1761311299.09 total 80.04 last 71.04) total: 10000 open/close in 87.37 seconds: 114.46 ops/second - unlinked 0 (time 1761311311 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1005 (time 1761311385.41 total 10.01 last 100.41) - open/close 2017 (time 1761311395.43 total 20.02 last 101.06) - open/close 2744 (time 1761311405.43 total 30.03 last 72.68) - open/close 3297 (time 1761311415.45 total 40.04 last 55.21) - open/close 3864 (time 1761311425.46 total 50.05 last 56.63) - open/close 4543 (time 1761311435.46 total 60.06 last 67.88) - open/close 5409 (time 1761311445.47 total 70.06 last 86.57) - open/close 6268 (time 1761311455.47 total 80.06 last 85.88) - open/close 7087 (time 1761311465.47 total 90.07 last 81.87) - open/close 7747 (time 1761311475.48 total 100.08 last 65.93) - open/close 8534 (time 1761311485.49 total 110.08 last 78.66) - open/close 9272 (time 1761311495.49 total 120.09 last 73.79) total: 10000 open/close in 128.35 seconds: 77.91 ops/second - unlinked 0 (time 1761311506 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 754 (time 1761311585.13 total 10.01 last 75.33) - open/close 1555 (time 1761311595.13 total 20.01 last 80.10) - open/close 2455 (time 1761311605.13 total 30.02 last 89.94) - open/close 3390 (time 1761311615.14 total 40.02 last 93.46) - open/close 4279 (time 1761311625.14 total 50.03 last 88.83) - open/close 5096 (time 1761311635.15 total 60.03 last 81.69) - open/close 5969 (time 1761311645.16 total 70.04 last 87.20) - open/close 6780 (time 1761311655.16 total 80.05 last 81.06) - open/close 7592 (time 1761311665.16 total 90.05 last 81.18) - open/close 8285 (time 1761311675.17 total 100.05 last 69.28) - open/close 8980 (time 1761311685.18 total 110.06 last 69.43) - open/close 9804 (time 1761311695.18 total 120.07 last 82.35) total: 10000 open/close in 122.19 seconds: 81.84 ops/second - unlinked 0 (time 1761311701 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 851 (time 1761311780.47 total 10.00 last 85.10) - open/close 1810 (time 1761311790.47 total 20.00 last 95.89) - open/close 2726 (time 1761311800.47 total 30.00 last 91.59) - open/close 3743 (time 1761311810.48 total 40.01 last 101.67) - open/close 4629 (time 1761311820.48 total 50.01 last 88.52) - open/close 5441 (time 1761311830.50 total 60.03 last 81.08) - open/close 6192 (time 1761311840.50 total 70.03 last 75.06) - open/close 6907 (time 1761311850.52 total 80.05 last 71.42) - open/close 7723 (time 1761311860.52 total 90.05 last 81.59) - open/close 8583 (time 1761311870.52 total 100.05 last 85.94) - open/close 9410 (time 1761311880.53 total 110.06 last 82.66) total: 10000 open/close in 117.23 seconds: 85.30 ops/second - unlinked 0 (time 1761311891 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 732 (time 1761311970.33 total 10.00 last 73.18) - open/close 1532 (time 1761311980.33 total 20.00 last 79.99) - open/close 2375 (time 1761311990.33 total 30.00 last 84.29) - open/close 3328 (time 1761312000.35 total 40.02 last 95.15) - open/close 4246 (time 1761312010.35 total 50.02 last 91.77) - open/close 5194 (time 1761312020.36 total 60.03 last 94.77) - open/close 6176 (time 1761312030.36 total 70.03 last 98.15) - open/close 7056 (time 1761312040.37 total 80.04 last 87.92) - open/close 8062 (time 1761312050.38 total 90.05 last 100.53) - open/close 8949 (time 1761312060.38 total 100.05 last 88.68) - open/close 9816 (time 1761312070.38 total 110.05 last 86.65) total: 10000 open/close in 111.96 seconds: 89.32 ops/second - unlinked 0 (time 1761312075 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 959 (time 1761312148.90 total 10.00 last 95.86) - open/close 1881 (time 1761312158.91 total 20.01 last 92.15) - open/close 2796 (time 1761312168.92 total 30.02 last 91.44) - open/close 3799 (time 1761312178.92 total 40.02 last 100.25) - open/close 4778 (time 1761312188.92 total 50.02 last 97.87) - open/close 5722 (time 1761312198.93 total 60.03 last 94.32) - open/close 6680 (time 1761312208.94 total 70.04 last 95.76) - open/close 7610 (time 1761312218.94 total 80.04 last 92.96) - open/close 8516 (time 1761312228.94 total 90.04 last 90.58) - open/close 9457 (time 1761312238.95 total 100.05 last 94.00) total: 10000 open/close in 106.34 seconds: 94.03 ops/second - unlinked 0 (time 1761312248 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 833 (time 1761312325.42 total 10.01 last 83.20) - open/close 1703 (time 1761312335.42 total 20.02 last 86.93) - open/close 2685 (time 1761312345.42 total 30.02 last 98.19) - open/close 3649 (time 1761312355.43 total 40.02 last 96.36) - open/close 4659 (time 1761312365.43 total 50.03 last 100.96) - open/close 5599 (time 1761312375.44 total 60.04 last 93.92) - open/close 6463 (time 1761312385.44 total 70.04 last 86.39) - open/close 7304 (time 1761312395.46 total 80.05 last 84.00) - open/close 8171 (time 1761312405.46 total 90.05 last 86.69) - open/close 9050 (time 1761312415.46 total 100.05 last 87.89) - open/close 9933 (time 1761312425.46 total 110.06 last 88.25) total: 10000 open/close in 110.79 seconds: 90.26 ops/second - unlinked 0 (time 1761312428 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg238-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg238-server: LNET ready to unload oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2