== conf-sanity test 69: replace an OST with the same index ========================================================== 21:08:11 (1746148091) start mds service on oleg225-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg225-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg225-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1297 (time 1746148150.63 total 10.00 last 129.68) - open/close 2675 (time 1746148160.64 total 20.01 last 137.72) - open/close 3880 (time 1746148170.64 total 30.01 last 120.49) - open/close 4901 (time 1746148180.64 total 40.01 last 102.10) - open/close 6039 (time 1746148190.64 total 50.01 last 113.74) - open/close 7294 (time 1746148200.65 total 60.02 last 125.45) - open/close 8661 (time 1746148210.65 total 70.02 last 136.66) total: 10000 open/close in 78.85 seconds: 126.82 ops/second - unlinked 0 (time 1746148221 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1223 (time 1746148273.39 total 10.00 last 122.25) - open/close 2791 (time 1746148283.39 total 20.01 last 156.71) - open/close 4396 (time 1746148293.40 total 30.01 last 160.45) - open/close 5908 (time 1746148303.40 total 40.02 last 151.17) - open/close 7333 (time 1746148313.40 total 50.02 last 142.47) - open/close 8791 (time 1746148323.41 total 60.03 last 145.66) total: 10000 open/close in 67.37 seconds: 148.42 ops/second - unlinked 0 (time 1746148332 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1529 (time 1746148373.54 total 10.00 last 152.84) - open/close 3144 (time 1746148383.54 total 20.01 last 161.46) - open/close 4654 (time 1746148393.54 total 30.01 last 150.97) - open/close 6448 (time 1746148403.55 total 40.02 last 179.28) - open/close 7906 (time 1746148413.55 total 50.02 last 145.78) - open/close 9418 (time 1746148423.55 total 60.02 last 151.18) total: 10000 open/close in 63.98 seconds: 156.29 ops/second - unlinked 0 (time 1746148429 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1246 (time 1746148484.95 total 10.01 last 124.48) - open/close 2636 (time 1746148494.97 total 20.03 last 138.65) - open/close 3999 (time 1746148504.98 total 30.04 last 136.26) - open/close 5417 (time 1746148514.99 total 40.05 last 141.65) - open/close 6763 (time 1746148524.99 total 50.05 last 134.54) - open/close 8075 (time 1746148535.01 total 60.07 last 131.01) - open/close 9474 (time 1746148545.01 total 70.08 last 139.78) total: 10000 open/close in 73.45 seconds: 136.15 ops/second - unlinked 0 (time 1746148549 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1550 (time 1746148600.87 total 10.01 last 154.86) - open/close 3026 (time 1746148610.87 total 20.02 last 147.50) - open/close 4293 (time 1746148620.88 total 30.02 last 126.67) - open/close 5555 (time 1746148630.88 total 40.03 last 126.09) - open/close 6846 (time 1746148640.89 total 50.03 last 129.07) - open/close 8145 (time 1746148650.90 total 60.04 last 129.77) - open/close 9320 (time 1746148660.90 total 70.05 last 117.44) total: 10000 open/close in 75.50 seconds: 132.45 ops/second - unlinked 0 (time 1746148668 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1602 (time 1746148725.11 total 10.00 last 160.13) - open/close 3021 (time 1746148735.12 total 20.01 last 141.79) - open/close 4283 (time 1746148745.12 total 30.01 last 126.19) - open/close 5394 (time 1746148755.12 total 40.02 last 111.03) - open/close 6516 (time 1746148765.14 total 50.03 last 112.06) - open/close 7697 (time 1746148775.14 total 60.03 last 118.08) - open/close 9014 (time 1746148785.14 total 70.03 last 131.69) total: 10000 open/close in 76.88 seconds: 130.07 ops/second - unlinked 0 (time 1746148793 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1241 (time 1746148850.45 total 10.00 last 124.04) - open/close 2439 (time 1746148860.46 total 20.01 last 119.76) - open/close 3819 (time 1746148870.46 total 30.01 last 137.94) - open/close 5123 (time 1746148880.46 total 40.01 last 130.40) - open/close 6423 (time 1746148890.46 total 50.02 last 129.95) - open/close 7912 (time 1746148900.47 total 60.02 last 148.88) - open/close 9385 (time 1746148910.47 total 70.02 last 147.26) total: 10000 open/close in 74.95 seconds: 133.42 ops/second - unlinked 0 (time 1746148917 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1440 (time 1746148972.48 total 10.00 last 143.96) - open/close 2847 (time 1746148982.48 total 20.00 last 140.69) - open/close 4197 (time 1746148992.48 total 30.01 last 134.95) - open/close 5648 (time 1746149002.49 total 40.01 last 145.05) - open/close 7041 (time 1746149012.50 total 50.02 last 139.11) - open/close 8400 (time 1746149022.51 total 60.03 last 135.83) - open/close 9897 (time 1746149032.51 total 70.03 last 149.68) total: 10000 open/close in 70.70 seconds: 141.45 ops/second - unlinked 0 (time 1746149034 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1624 (time 1746149082.88 total 10.01 last 162.32) - open/close 2992 (time 1746149092.88 total 20.01 last 136.77) - open/close 4431 (time 1746149102.88 total 30.01 last 143.85) - open/close 5793 (time 1746149112.89 total 40.01 last 136.17) - open/close 7322 (time 1746149122.89 total 50.01 last 152.89) - open/close 8584 (time 1746149132.89 total 60.02 last 126.17) - open/close 9887 (time 1746149142.89 total 70.02 last 130.26) total: 10000 open/close in 70.78 seconds: 141.29 ops/second - unlinked 0 (time 1746149145 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1278 (time 1746149200.48 total 10.00 last 127.74) - open/close 2542 (time 1746149210.48 total 20.01 last 126.36) - open/close 4019 (time 1746149220.49 total 30.01 last 147.66) - open/close 5620 (time 1746149230.49 total 40.01 last 160.09) - open/close 6848 (time 1746149240.49 total 50.01 last 122.80) - open/close 8103 (time 1746149250.49 total 60.01 last 125.47) - open/close 9293 (time 1746149260.50 total 70.03 last 118.85) total: 10000 open/close in 75.80 seconds: 131.92 ops/second - unlinked 0 (time 1746149268 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg225-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg225-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre On OST0, 16616 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2