== conf-sanity test 69: replace an OST with the same index ========================================================== 01:18:31 (1734675511) start mds service on oleg145-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: MODOPTS_LIBCFS= oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg145-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1191 (time 1734675573.21 total 10.00 last 119.08) - open/close 2439 (time 1734675583.21 total 20.00 last 124.78) - open/close 3830 (time 1734675593.21 total 30.00 last 139.09) - open/close 5089 (time 1734675603.22 total 40.01 last 125.85) - open/close 6295 (time 1734675613.22 total 50.01 last 120.57) - open/close 7832 (time 1734675623.22 total 60.01 last 153.66) - open/close 9404 (time 1734675633.22 total 70.01 last 157.17) total: 10000 open/close in 74.83 seconds: 133.63 ops/second - unlinked 0 (time 1734675639 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1350 (time 1734675692.08 total 10.00 last 134.94) - open/close 2779 (time 1734675702.09 total 20.01 last 142.82) - open/close 4184 (time 1734675712.09 total 30.01 last 140.45) - open/close 5495 (time 1734675722.10 total 40.02 last 131.02) - open/close 6851 (time 1734675732.10 total 50.02 last 135.54) - open/close 8287 (time 1734675742.11 total 60.03 last 143.51) - open/close 9861 (time 1734675752.11 total 70.03 last 157.36) total: 10000 open/close in 70.73 seconds: 141.39 ops/second - unlinked 0 (time 1734675753 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1553 (time 1734675796.96 total 10.00 last 155.25) - open/close 3211 (time 1734675806.97 total 20.01 last 165.72) - open/close 4838 (time 1734675816.97 total 30.01 last 162.68) - open/close 6405 (time 1734675826.97 total 40.01 last 156.68) - open/close 8159 (time 1734675836.97 total 50.01 last 175.38) - open/close 9874 (time 1734675846.98 total 60.02 last 171.38) total: 10000 open/close in 60.80 seconds: 164.47 ops/second - unlinked 0 (time 1734675849 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1721 (time 1734675887.57 total 10.00 last 172.09) - open/close 3523 (time 1734675897.57 total 20.00 last 180.18) - open/close 5476 (time 1734675907.57 total 30.00 last 195.27) - open/close 7395 (time 1734675917.57 total 40.01 last 191.85) - open/close 9718 (time 1734675927.57 total 50.01 last 232.25) total: 10000 open/close in 51.55 seconds: 193.99 ops/second - unlinked 0 (time 1734675930 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 2283 (time 1734675969.79 total 10.00 last 228.22) - open/close 4117 (time 1734675979.79 total 20.00 last 183.38) - open/close 6047 (time 1734675989.79 total 30.00 last 192.99) - open/close 8058 (time 1734675999.80 total 40.01 last 201.06) total: 10000 open/close in 49.95 seconds: 200.20 ops/second - unlinked 0 (time 1734676011 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2050 (time 1734676047.91 total 10.00 last 204.97) - open/close 4525 (time 1734676057.91 total 20.00 last 247.47) - open/close 6656 (time 1734676067.91 total 30.01 last 213.01) - open/close 8612 (time 1734676077.92 total 40.01 last 195.54) total: 10000 open/close in 48.02 seconds: 208.24 ops/second - unlinked 0 (time 1734676087 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1868 (time 1734676124.86 total 10.00 last 186.73) - open/close 3740 (time 1734676134.86 total 20.01 last 187.10) - open/close 5550 (time 1734676144.86 total 30.01 last 180.98) - open/close 7372 (time 1734676154.87 total 40.01 last 182.19) - open/close 9466 (time 1734676164.87 total 50.02 last 209.28) total: 10000 open/close in 53.20 seconds: 187.98 ops/second - unlinked 0 (time 1734676169 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1702 (time 1734676213.71 total 10.01 last 170.11) - open/close 3421 (time 1734676223.71 total 20.01 last 171.82) - open/close 5226 (time 1734676233.72 total 30.01 last 180.45) - open/close 7061 (time 1734676243.72 total 40.02 last 183.41) - open/close 8929 (time 1734676253.73 total 50.02 last 186.72) total: 10000 open/close in 55.13 seconds: 181.39 ops/second - unlinked 0 (time 1734676260 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1734 (time 1734676299.71 total 10.00 last 173.37) - open/close 3379 (time 1734676309.71 total 20.01 last 164.40) - open/close 4924 (time 1734676319.71 total 30.01 last 154.50) - open/close 6474 (time 1734676329.72 total 40.01 last 154.95) - open/close 8038 (time 1734676339.72 total 50.01 last 156.39) - open/close 9573 (time 1734676349.72 total 60.01 last 153.47) total: 10000 open/close in 62.77 seconds: 159.30 ops/second - unlinked 0 (time 1734676354 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1663 (time 1734676400.22 total 10.00 last 166.26) - open/close 3116 (time 1734676410.22 total 20.01 last 145.25) - open/close 4611 (time 1734676420.23 total 30.01 last 149.44) - open/close 6073 (time 1734676430.23 total 40.01 last 146.15) - open/close 7748 (time 1734676440.23 total 50.02 last 167.47) - open/close 9698 (time 1734676450.24 total 60.02 last 194.89) total: 10000 open/close in 62.17 seconds: 160.84 ops/second - unlinked 0 (time 1734676454 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre On OST0, 13948 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg145-server unloading modules via unload_modules_local on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2