== conf-sanity test 69: replace an OST with the same index ========================================================== 08:51:02 (1747399862) start mds service on oleg336-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg336-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg336-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg336-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock oleg336-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 1825 (time 1747399921.78 total 10.01 last 182.34) - open/close 3618 (time 1747399931.78 total 20.01 last 179.19) - open/close 5466 (time 1747399941.78 total 30.02 last 184.78) - open/close 7102 (time 1747399951.78 total 40.02 last 163.58) - open/close 8892 (time 1747399961.78 total 50.02 last 178.99) total: 10000 open/close in 56.32 seconds: 177.56 ops/second - unlinked 0 (time 1747399969 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1797 (time 1747400010.77 total 10.00 last 179.61) - open/close 3407 (time 1747400020.77 total 20.01 last 160.97) - open/close 4939 (time 1747400030.77 total 30.01 last 153.20) - open/close 6706 (time 1747400040.78 total 40.01 last 176.56) - open/close 8203 (time 1747400050.78 total 50.02 last 149.68) - open/close 9845 (time 1747400060.78 total 60.02 last 164.16) total: 10000 open/close in 60.86 seconds: 164.32 ops/second - unlinked 0 (time 1747400063 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1897 (time 1747400102.85 total 10.00 last 189.66) - open/close 3620 (time 1747400112.85 total 20.01 last 172.23) - open/close 5276 (time 1747400122.85 total 30.01 last 165.56) - open/close 7049 (time 1747400132.86 total 40.01 last 177.24) - open/close 8790 (time 1747400142.86 total 50.01 last 174.08) total: 10000 open/close in 56.49 seconds: 177.03 ops/second - unlinked 0 (time 1747400150 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2560 (time 1747400187.38 total 10.00 last 255.91) - open/close 5155 (time 1747400197.38 total 20.00 last 259.49) - open/close 7982 (time 1747400207.38 total 30.00 last 282.67) total: 10000 open/close in 36.66 seconds: 272.76 ops/second - unlinked 0 (time 1747400215 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2285 (time 1747400245.97 total 10.00 last 228.43) - open/close 4667 (time 1747400255.97 total 20.01 last 238.14) - open/close 7096 (time 1747400265.97 total 30.01 last 242.85) - open/close 9304 (time 1747400275.98 total 40.01 last 220.73) total: 10000 open/close in 43.46 seconds: 230.07 ops/second - unlinked 0 (time 1747400280 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2786 (time 1747400314.88 total 10.00 last 278.58) - open/close 5158 (time 1747400324.88 total 20.00 last 237.14) - open/close 7674 (time 1747400334.88 total 30.00 last 251.57) total: 10000 open/close in 38.90 seconds: 257.07 ops/second - unlinked 0 (time 1747400344 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2972 (time 1747400374.22 total 10.00 last 297.18) - open/close 6033 (time 1747400384.22 total 20.00 last 306.08) - open/close 9208 (time 1747400394.23 total 30.00 last 317.45) total: 10000 open/close in 32.34 seconds: 309.23 ops/second - unlinked 0 (time 1747400397 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3695 (time 1747400422.43 total 10.00 last 369.46) - open/close 7163 (time 1747400432.43 total 20.00 last 346.78) total: 10000 open/close in 28.75 seconds: 347.86 ops/second - unlinked 0 (time 1747400442 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4085 (time 1747400467.58 total 10.00 last 408.42) - open/close 8551 (time 1747400477.58 total 20.00 last 446.58) total: 10000 open/close in 23.79 seconds: 420.38 ops/second - unlinked 0 (time 1747400482 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3500 (time 1747400509.26 total 10.00 last 349.88) - open/close 6687 (time 1747400519.26 total 20.00 last 318.69) total: 10000 open/close in 29.31 seconds: 341.17 ops/second - unlinked 0 (time 1747400529 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.136@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg336-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg336-server: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg336-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg336-server: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg336-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock oleg336-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server LNET ready to unload unloading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: LNET ready to unload modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2