[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 633912380 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003173] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.006019] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.007011] pid_max: default: 32768 minimum: 301 [ 0.009021] LSM: Security Framework initializing [ 0.010040] Yama: becoming mindful. [ 0.011031] SELinux: Initializing. [ 0.012063] *** VALIDATE selinux *** [ 0.021042] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026609] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.030460] *** VALIDATE proc *** [ 0.032103] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.035102] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038044] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042664] debug: unmapping init [mem 0xffffffff8fc59000-0xffffffff8fc60fff] [ 0.044207] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045840] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046031] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051027] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053376] rcu: Hierarchical SRCU implementation. [ 0.055762] smp: Bringing up secondary CPUs ... [ 0.056665] x86: Booting SMP configuration: [ 0.057033] .... node #0, CPUs: #1 #2 #3 [ 0.060299] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.196969] node 0 deferred pages initialised in 132ms [ 0.201159] devtmpfs: initialized [ 0.203315] x86/mm: Memory block size: 128MB [ 0.206377] gcov: version magic: 0x41383552 [ 0.209295] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.213099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.217360] pinctrl core: initialized pinctrl subsystem [ 0.219349] [ 0.219853] ************************************************************* [ 0.222013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.225014] ** ** [ 0.228016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.232031] ** ** [ 0.234012] ** This means that this kernel is built to expose internal ** [ 0.237143] ** IOMMU data structures, which may compromise security on ** [ 0.241016] ** your system. ** [ 0.243019] ** ** [ 0.246017] ** If you see this message and you are not debugging the ** [ 0.248011] ** kernel, report this immediately to your vendor! ** [ 0.251014] ** ** [ 0.253034] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256015] ************************************************************* [ 0.259714] NET: Registered protocol family 16 [ 0.262296] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.264113] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.272158] cpuidle: using governor menu [ 0.275013] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.278686] PCI: Using configuration type 1 for base access [ 0.280153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.291062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.292000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.293298] cryptd: max_cpu_qlen set to 1000 [ 0.296431] ACPI: Added _OSI(Module Device) [ 0.298018] ACPI: Added _OSI(Processor Device) [ 0.300081] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.302021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.307843] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.313583] ACPI: Interpreter enabled [ 0.315053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.317015] ACPI: Using IOAPIC for interrupt routing [ 0.318101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.321546] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.334591] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.337031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.339020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.342125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.347603] acpiphp: Slot [2] registered [ 0.349117] acpiphp: Slot [5] registered [ 0.351088] acpiphp: Slot [6] registered [ 0.352219] acpiphp: Slot [7] registered [ 0.354093] acpiphp: Slot [8] registered [ 0.355091] acpiphp: Slot [9] registered [ 0.356119] acpiphp: Slot [10] registered [ 0.358121] acpiphp: Slot [3] registered [ 0.359085] acpiphp: Slot [4] registered [ 0.360092] acpiphp: Slot [11] registered [ 0.362186] acpiphp: Slot [12] registered [ 0.363178] acpiphp: Slot [13] registered [ 0.364123] acpiphp: Slot [14] registered [ 0.367183] acpiphp: Slot [15] registered [ 0.369116] acpiphp: Slot [16] registered [ 0.371122] acpiphp: Slot [17] registered [ 0.373132] acpiphp: Slot [18] registered [ 0.374114] acpiphp: Slot [19] registered [ 0.376165] acpiphp: Slot [20] registered [ 0.378203] acpiphp: Slot [21] registered [ 0.379102] acpiphp: Slot [22] registered [ 0.381116] acpiphp: Slot [23] registered [ 0.383220] acpiphp: Slot [24] registered [ 0.385116] acpiphp: Slot [25] registered [ 0.387120] acpiphp: Slot [26] registered [ 0.389130] acpiphp: Slot [27] registered [ 0.390116] acpiphp: Slot [28] registered [ 0.392131] acpiphp: Slot [29] registered [ 0.394106] acpiphp: Slot [30] registered [ 0.395110] acpiphp: Slot [31] registered [ 0.397062] PCI host bridge to bus 0000:00 [ 0.398016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.401025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.403021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.406029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.409038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.412042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.414258] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.417108] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.420000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.429945] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.436072] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.439034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.444025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.448183] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.452133] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.455086] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.458878] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.465018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.476026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.484016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.490000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.519035] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.551038] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.603033] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.631412] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.642018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.654024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.699022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.715552] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.726021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.737021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.780022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.800749] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.811020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.821020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.844018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.856631] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.870018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.877014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.890016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.903741] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.912016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.936015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.974022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.987091] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.989378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.993535] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.995350] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.997205] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.002659] iommu: Default domain type: Passthrough [ 1.004324] SCSI subsystem initialized [ 1.006144] ACPI: bus type USB registered [ 1.007100] usbcore: registered new interface driver usbfs [ 1.009119] usbcore: registered new interface driver hub [ 1.011087] usbcore: registered new device driver usb [ 1.012233] pps_core: LinuxPPS API ver. 1 registered [ 1.014014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.017082] PTP clock support registered [ 1.019143] EDAC MC: Ver: 3.0.0 [ 1.021068] PCI: Using ACPI for IRQ routing [ 1.023025] NetLabel: Initializing [ 1.024010] NetLabel: domain hash size = 128 [ 1.026072] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.029119] NetLabel: unlabeled traffic allowed by default [ 1.031387] vgaarb: loaded [ 1.033389] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.034015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.040231] clocksource: Switched to clocksource kvm-clock [ 1.161263] VFS: Disk quotas dquot_6.6.0 [ 1.163223] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.165944] *** VALIDATE ramfs *** [ 1.167383] *** VALIDATE hugetlbfs *** [ 1.169214] pnp: PnP ACPI init [ 1.172043] pnp: PnP ACPI: found 6 devices [ 1.193416] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.197965] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.200311] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.202420] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.205154] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.207191] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.210037] NET: Registered protocol family 2 [ 1.213936] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.219583] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.223304] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.228128] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.231245] TCP: Hash tables configured (established 65536 bind 65536) [ 1.233892] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.236870] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.239377] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.242210] NET: Registered protocol family 1 [ 1.246445] RPC: Registered named UNIX socket transport module. [ 1.248817] RPC: Registered udp transport module. [ 1.250596] RPC: Registered tcp transport module. [ 1.252581] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.255444] NET: Registered protocol family 44 [ 1.257268] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.259960] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.262383] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.265204] PCI: CLS 0 bytes, default 64 [ 1.266962] Unpacking initramfs... [ 2.719925] debug: unmapping init [mem 0xffff9b50fcc54000-0xffff9b50fffbffff] [ 2.723835] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.727254] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.730140] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 3.252970] Initialise system trusted keyrings [ 3.254970] Key type blacklist registered [ 3.257423] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.268969] zbud: loaded [ 3.272754] *** VALIDATE nfs *** [ 3.274574] *** VALIDATE nfs4 *** [ 3.277656] pstore: using deflate compression [ 3.283368] Platform Keyring initialized [ 3.421728] NET: Registered protocol family 38 [ 3.424046] Key type asymmetric registered [ 3.425896] Asymmetric key parser 'x509' registered [ 3.428162] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.432188] io scheduler mq-deadline registered [ 3.434266] io scheduler kyber registered [ 3.436588] io scheduler bfq registered [ 3.440030] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.443521] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.446749] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.450297] ACPI: Power Button [PWRF] [ 3.456961] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.465286] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.514257] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.535616] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.550673] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.579833] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.610677] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.616311] Non-volatile memory driver v1.3 [ 3.618267] Linux agpgart interface v0.103 [ 3.645302] virtio_blk virtio1: [vda] 134848 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.649974] vda: detected capacity change from 0 to 69042176 [ 3.671055] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.673860] vdb: detected capacity change from 0 to 1073741824 [ 3.703623] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.706857] vdc: detected capacity change from 0 to 2621440000 [ 3.741110] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.744786] vdd: detected capacity change from 0 to 2621440000 [ 3.766473] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.770467] vde: detected capacity change from 0 to 4294967296 [ 3.791603] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.795370] vdf: detected capacity change from 0 to 4294967296 [ 3.806440] libphy: Fixed MDIO Bus: probed [ 3.812361] usbcore: registered new interface driver usbserial_generic [ 3.814992] usbserial: USB Serial support registered for generic [ 3.817857] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.822193] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.824043] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.826580] mousedev: PS/2 mouse device common for all mice [ 3.830331] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.831588] rtc_cmos 00:05: RTC can wake from S4 [ 3.840609] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.844163] rtc_cmos 00:05: registered as rtc0 [ 3.846679] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.851150] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.851206] intel_pstate: CPU model not supported [ 3.860126] hid: raw HID events driver (C) Jiri Kosina [ 3.866624] usbcore: registered new interface driver usbhid [ 3.868119] usbhid: USB HID core driver [ 3.869317] drop_monitor: Initializing network drop monitor service [ 3.871163] Initializing XFRM netlink socket [ 3.872802] NET: Registered protocol family 10 [ 3.875452] Segment Routing with IPv6 [ 3.876583] NET: Registered protocol family 17 [ 3.878338] mpls_gso: MPLS GSO support [ 3.884563] RAS: Correctable Errors collector initialized. [ 3.886326] AVX version of gcm_enc/dec engaged. [ 3.888206] AES CTR mode by8 optimization enabled [ 3.966974] sched_clock: Marking stable (3966894438, 0)->(4907890441, -940996003) [ 3.972792] registered taskstats version 1 [ 3.974499] Loading compiled-in X.509 certificates [ 3.978295] zswap: loaded using pool lzo/zbud [ 4.004738] Key type big_key registered [ 4.017257] Key type encrypted registered [ 4.018510] ima: No TPM chip found, activating TPM-bypass! [ 4.020244] ima: Allocated hash algorithm: sha1 [ 4.021987] ima: No architecture policies found [ 4.023375] evm: Initialising EVM extended attributes: [ 4.024870] evm: security.selinux [ 4.028169] evm: security.ima [ 4.028985] evm: security.capability [ 4.030212] evm: HMAC attrs: 0x1 [ 4.032621] rtc_cmos 00:05: setting system clock to 2026-06-01 14:23:27 UTC (1780323807) [ 4.038486] debug: unmapping init [mem 0xffffffff90c03000-0xffffffff90dfffff] [ 4.041138] debug: unmapping init [mem 0xffffffff8f982000-0xffffffff8fc58fff] [ 4.048185] Write protecting the kernel read-only data: 28672k [ 4.051440] debug: unmapping init [mem 0xffffffff8e003000-0xffffffff8e1fffff] [ 4.053899] debug: unmapping init [mem 0xffffffff8e914000-0xffffffff8e9fffff] [ 4.087579] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.098433] systemd[1]: Detected virtualization kvm. [ 4.099709] systemd[1]: Detected architecture x86-64. [ 4.101512] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.125140] systemd[1]: No hostname configured. [ 4.126574] systemd[1]: Set hostname to . [ 4.128264] random: systemd: uninitialized urandom read (16 bytes read) [ 4.130276] systemd[1]: Initializing machine ID from random generator. [ 4.185603] random: ln: uninitialized urandom read (6 bytes read) [ 4.275985] random: systemd: uninitialized urandom read (16 bytes read) [ 4.278046] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.282147] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.286917] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.959223] device-mapper: uevent: version 1.0.3 [ 4.961986] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook...[ 5.291252] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.696683] virtio_net virtio0 ens2: renamed from eth0 [ 5.840347] scsi host0: ata_piix [ 5.898398] scsi host1: ata_piix [ 5.904770] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.914443] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.541411] random: crng init done [ 10.542783] random: 7 urandom warning(s) missed due to ratelimiting [ 10.683616] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.981086] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.541991] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.896228] SELinux: Disabled at runtime. [ 13.954885] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.963683] systemd[1]: Detected virtualization kvm. [ 13.965174] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.066074] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.073548] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.082897] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.089202] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.092718] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.102049] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.112429] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 15.267362] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 15.763213] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.514329] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.548239] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.775401] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.810868] EDAC sbridge: Ver: 1.1.2 [ 19.862314] Key type dns_resolver registered [ 20.282871] NFS: Registering the id_resolver key type [ 20.284607] Key type id_resolver registered [ 20.286017] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg429-server login: [ 79.680122] libcfs: loading out-of-tree module taints kernel. [ 79.715517] Key type ._llcrypt registered [ 79.719494] Key type .llcrypt registered [ 79.820614] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 91.883074] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 92.812682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.833091] alg: No test for adler32 (adler32-zlib) [ 94.063879] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 94.644790] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 96.327181] Key type lgssc registered [ 97.580694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.910133] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 133.558426] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 142.737377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.755026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.913735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.948681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 144.000324] Lustre: lustre-MDT0000: new disk, initializing [ 144.048343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.066370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.730490] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.422520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.470649] Lustre: 6488:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 155.507458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 155.510707] Lustre: Skipped 1 previous similar message [ 155.562143] Lustre: lustre-MDT0001: new disk, initializing [ 155.597338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 155.617826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 155.626092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 158.258848] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.845992] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.689933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.868972] Lustre: lustre-OST0000: new disk, initializing [ 168.873406] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.877813] Lustre: 8392:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.921084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.348787] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.076705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.084633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.132172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 181.373252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 181.507482] Lustre: lustre-OST0001: new disk, initializing [ 181.511784] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 181.522717] Lustre: 9447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 181.566323] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.845874] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.907667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 187.917098] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 187.961832] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 194.682490] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 199.778377] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 203.547012] hrtimer: interrupt took 6002678 ns [ 204.072466] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing check_logdir /tmp/testlogs/ [ 207.131593] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing yml_node [ 210.122284] Lustre: DEBUG MARKER: Client: 2.17.53.29 [ 211.939951] Lustre: DEBUG MARKER: MDS: 2.17.53.29 [ 213.819342] Lustre: DEBUG MARKER: OSS: 2.17.53.29 [ 214.956878] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 1 10:26:57 EDT 2026 [ 225.965915] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 227.021084] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 242.656029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 242.663734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 242.685272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 244.194449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.198926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 244.206994] Lustre: Skipped 2 previous similar messages [ 244.214165] Lustre: Skipped 3 previous similar messages [ 247.291647] Lustre: server umount lustre-MDT0000 complete [ 250.207406] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324053 with bad export cookie 6271911515544672250 [ 250.209664] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 250.217926] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 254.433195] LustreError: 12079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 254.437751] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.441509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 254.445549] LustreError: 12079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 254.467865] Lustre: Skipped 1 previous similar message [ 255.457683] LustreError: 6498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.465717] LustreError: 6498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 256.565021] Lustre: server umount lustre-MDT0001 complete [ 259.727531] Lustre: server umount lustre-OST0000 complete [ 262.890654] Lustre: server umount lustre-OST0001 complete [ 267.369722] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 273.295452] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 296.744257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 296.982432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 297.014448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 297.072151] Lustre: lustre-MDT0000: new disk, initializing [ 297.137187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 297.158626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 300.014467] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 308.335312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 308.438205] Lustre: 14953:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 308.478653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 308.482206] Lustre: Skipped 1 previous similar message [ 308.534658] Lustre: lustre-MDT0001: new disk, initializing [ 308.615159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 308.663065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 308.677747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 311.982725] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 317.422694] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 319.901890] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 324.335445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 324.541952] Lustre: lustre-OST0000: new disk, initializing [ 324.544422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 324.549858] Lustre: 16693:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 324.621455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.262385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 326.273025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 326.354516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 328.372386] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 333.381285] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 336.294377] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 336.449700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 339.256869] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 339.397848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 341.986876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 341.988630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.992215] LustreError: Skipped 1 previous similar message [ 342.004982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 342.010935] Lustre: Skipped 4 previous similar messages [ 346.818206] Lustre: server umount lustre-OST0000 complete [ 349.664629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 349.671519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.688287] Lustre: Skipped 1 previous similar message [ 349.698233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.481931] Lustre: server umount lustre-MDT0000 complete [ 357.345820] LustreError: 14963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.357976] LustreError: 14963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 357.763855] LustreError: 15731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324161 with bad export cookie 6271911515544673783 [ 357.768078] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.772020] LustreError: 15731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 358.000885] Lustre: server umount lustre-MDT0001 complete [ 360.707996] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:29:23 (1780324163) [ 362.221562] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 363.285304] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:29:25 (1780324165) [ 367.544787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.899623] LustreError: 18954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.955247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.604692] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 373.217851] LustreError: 18955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.829749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.534774] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 381.741726] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 384.047333] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 387.987602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 391.868496] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 395.301161] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 412.131542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 412.131603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 412.153618] Lustre: Skipped 1 previous similar message [ 412.172361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 412.178625] Lustre: Skipped 2 previous similar messages [ 415.477188] Lustre: server umount lustre-OST0000 complete [ 422.377767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 431.584678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 431.588579] Lustre: Skipped 6 previous similar messages [ 432.095213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 432.224947] Lustre: server umount lustre-MDT0000 complete [ 434.534065] LustreError: 18936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324238 with bad export cookie 6271911515544674658 [ 434.537839] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 434.542761] LustreError: 18936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 434.744205] Lustre: server umount lustre-MDT0001 complete [ 441.565155] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 443.551337] Key type lgssc unregistered [ 443.743904] LNet: 22147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.748459] LNetError: 22147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 443.759694] LNet: Removed LNI 192.168.204.129@tcp [ 444.359162] Key type .llcrypt unregistered [ 444.362708] Key type ._llcrypt unregistered [ 458.753173] Key type ._llcrypt registered [ 458.754894] Key type .llcrypt registered [ 458.827347] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 459.361469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 459.446414] alg: No test for adler32 (adler32-zlib) [ 460.422917] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 460.588552] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 462.223150] Key type lgssc registered [ 462.950629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 468.734529] loop: module loaded [ 468.791483] loop0: detected capacity change from 0 to 204800000 [ 468.971540] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 468.984839] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 469.237679] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 473.314383] loop0: detected capacity change from 0 to 204800000 [ 473.439798] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 473.449946] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 473.459878] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 476.972843] loop0: detected capacity change from 0 to 204800000 [ 481.031994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 481.049132] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.208716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 482.260929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 482.320391] Lustre: lustre-MDT0000: new disk, initializing [ 482.387206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.400952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 485.296812] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 491.568925] loop1: detected capacity change from 0 to 204800000 [ 495.307613] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.470254] Lustre: lustre-OST07c6: new disk, initializing [ 495.476036] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 495.481063] Lustre: Skipped 1 previous similar message [ 495.485873] Lustre: 25276:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.493151] Lustre: lustre-OST07c6: Not available for connect from 0@lo (not set up) [ 495.539420] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 499.640888] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 500.741841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 500.748561] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 500.782926] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 504.144750] Lustre: 24279:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 504.152033] Lustre: 24279:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 504.156983] Lustre: 24279:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 504.163804] Lustre: 24279:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 504.171921] Lustre: 24279:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 504.175379] Lustre: 24279:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 521.360112] Lustre: 24278:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 521.365526] Lustre: 24278:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 521.369892] Lustre: 24278:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 521.375748] Lustre: 24278:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 521.380618] Lustre: 24278:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 521.384212] Lustre: 24278:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 525.725817] Lustre: 24278:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 525.732337] Lustre: 24278:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 525.736161] Lustre: 24278:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 525.738792] Lustre: 24278:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 525.743105] Lustre: 24278:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 525.747708] Lustre: 24278:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 525.752574] Lustre: 24278:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 525.758952] Lustre: 24278:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 525.768598] Lustre: 24278:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 525.773029] Lustre: 24278:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 525.777283] Lustre: 24278:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 525.780950] Lustre: 24278:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 528.923768] Lustre: 24279:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 528.933995] Lustre: 24279:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 528.939514] Lustre: 24279:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 528.943746] Lustre: 24279:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 528.948714] Lustre: 24279:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 528.953616] Lustre: 24279:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 528.958662] Lustre: 24279:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 528.964339] Lustre: 24279:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 528.969030] Lustre: 24279:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 528.974149] Lustre: 24279:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 528.978915] Lustre: 24279:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 528.983778] Lustre: 24279:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.022484] Lustre: 25861:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 533.029926] Lustre: 25861:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 533.034305] Lustre: 25861:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 533.039212] Lustre: 25861:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.043739] Lustre: 25861:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 533.048231] Lustre: 25861:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.053884] Lustre: 25861:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 533.062299] Lustre: 25861:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.070126] Lustre: 25861:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 533.074037] Lustre: 25861:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.082222] Lustre: 25861:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 533.087297] Lustre: 25861:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 543.283453] Lustre: 24277:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 543.287872] Lustre: 24277:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 543.291270] Lustre: 24277:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 543.296123] Lustre: 24277:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 543.299745] Lustre: 24277:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 543.302156] Lustre: 24277:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 543.305256] Lustre: 24277:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 543.309139] Lustre: 24277:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 543.312468] Lustre: 24277:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 543.315585] Lustre: 24277:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 543.319348] Lustre: 24277:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 543.322554] Lustre: 24277:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 592.870543] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.871808] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 592.881544] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 595.648666] Lustre: server umount lustre-OST07c6 complete [ 601.166854] Lustre: server umount lustre-MDT0000 complete [ 609.180520] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 610.759498] Key type lgssc unregistered [ 610.962990] LNet: 27792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.972951] LNetError: 27792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.981898] LNet: Removed LNI 192.168.204.129@tcp [ 611.482778] Key type .llcrypt unregistered [ 611.484725] Key type ._llcrypt unregistered [ 623.450352] Key type ._llcrypt registered [ 623.452372] Key type .llcrypt registered [ 623.516676] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 624.085080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.100798] alg: No test for adler32 (adler32-zlib) [ 625.052489] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 625.176880] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 626.799185] Key type lgssc registered [ 627.460447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.673356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 631.695099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.988977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.207706] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 640.069539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.275618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 642.378828] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 645.317311] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 647.323316] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 651.100713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 651.313475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 654.564786] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 657.310556] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 658.474042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 674.785402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.796098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 674.797866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 674.807341] Lustre: Skipped 1 previous similar message [ 676.787537] Lustre: server umount lustre-OST0000 complete [ 679.906599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 679.909896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 679.916815] Lustre: Skipped 2 previous similar messages [ 685.024145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 685.029598] Lustre: Skipped 2 previous similar messages [ 690.147754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 690.157714] Lustre: Skipped 1 previous similar message [ 693.215198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 693.312488] Lustre: server umount lustre-MDT0000 complete [ 695.269432] LustreError: 31173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.278930] LustreError: 31173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 695.511083] LustreError: 29005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324498 with bad export cookie 309824653195749892 [ 695.516248] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 695.518739] LustreError: 29005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 695.684560] Lustre: server umount lustre-MDT0001 complete [ 701.380971] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 702.992777] Key type lgssc unregistered [ 703.193086] LNet: 32424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.197401] LNetError: 32424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.206820] LNet: Removed LNI 192.168.204.129@tcp [ 703.647939] Key type .llcrypt unregistered [ 703.650148] Key type ._llcrypt unregistered [ 714.517960] Key type ._llcrypt registered [ 714.519825] Key type .llcrypt registered [ 714.589641] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 715.084673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 715.136425] alg: No test for adler32 (adler32-zlib) [ 716.065736] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 716.187216] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 717.807171] Key type lgssc registered [ 718.440986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 722.798461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 722.810231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.093353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 726.717772] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 731.562894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.763917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 733.967269] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 737.115738] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 739.222827] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 743.411538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 743.593882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 747.027506] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 748.717058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 750.057618] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 762.341486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.345256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 762.352290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 762.361857] Lustre: Skipped 1 previous similar message [ 766.862472] Lustre: server umount lustre-OST0000 complete [ 772.581081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 772.583676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.585744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 772.602752] Lustre: Skipped 2 previous similar messages [ 777.698137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 777.703015] Lustre: Skipped 2 previous similar messages [ 781.792907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 781.797241] Lustre: Skipped 1 previous similar message [ 783.327152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 783.415667] Lustre: server umount lustre-MDT0000 complete [ 785.490579] LustreError: 33637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324588 with bad export cookie 14790984422235204204 [ 785.493996] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.499302] LustreError: 33637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 785.693083] Lustre: server umount lustre-MDT0001 complete [ 791.681911] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 793.379993] Key type lgssc unregistered [ 793.560471] LNet: 36829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.565686] LNetError: 36829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.573449] LNet: Removed LNI 192.168.204.129@tcp [ 794.017641] Key type .llcrypt unregistered [ 794.019154] Key type ._llcrypt unregistered [ 807.276723] Key type ._llcrypt registered [ 807.280037] Key type .llcrypt registered [ 807.331830] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 807.754838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 807.764946] alg: No test for adler32 (adler32-zlib) [ 808.682399] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 808.812916] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 810.431175] Key type lgssc registered [ 811.101812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 815.301375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 815.314904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.594502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.039612] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 823.468195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.687497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 825.749514] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 828.552846] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 830.403097] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 834.185737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 834.401224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.597692] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 840.915598] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 841.582481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 858.845139] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 859.106880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.117226] Lustre: Skipped 2 previous similar messages [ 861.234590] Lustre: server umount lustre-MDT0000 complete [ 863.342340] LustreError: 38187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324666 with bad export cookie 11592836671157716581 [ 863.345298] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 863.352300] LustreError: 38187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 863.951279] LustreError: 38201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.966532] LustreError: 38201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 863.968944] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [ 863.975157] Lustre: Skipped 3 previous similar messages [ 864.226053] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.256812] LustreError: 38202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.256844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 868.270459] LustreError: 38202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 868.277619] Lustre: Skipped 1 previous similar message [ 869.606907] Lustre: server umount lustre-MDT0001 complete [ 888.293613] Lustre: server umount lustre-OST0000 complete [ 895.751260] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 897.096914] Key type lgssc unregistered [ 897.260689] LNet: 41724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.270418] LNetError: 41724:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.284223] LNet: Removed LNI 192.168.204.129@tcp [ 897.669430] Key type .llcrypt unregistered [ 897.671333] Key type ._llcrypt unregistered [ 909.168487] Key type ._llcrypt registered [ 909.170181] Key type .llcrypt registered [ 909.218176] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 909.729355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.747775] alg: No test for adler32 (adler32-zlib) [ 910.650788] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 910.759670] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 912.359171] Key type lgssc registered [ 912.876652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.846865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 916.861954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 918.121848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.318383] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 925.088957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.274665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 927.201445] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 929.706804] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 931.267045] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 934.046237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.210574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.589614] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 939.007106] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 943.466117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 948.709291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 948.709555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.714292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.721563] Lustre: Skipped 1 previous similar message [ 950.488166] Lustre: lustre-OST0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 950.494108] Lustre: Skipped 1 previous similar message [ 953.826165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.832208] Lustre: Skipped 1 previous similar message [ 954.005592] Lustre: server umount lustre-OST0000 complete [ 967.653596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.655698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 967.659889] Lustre: Skipped 1 previous similar message [ 967.665176] Lustre: Skipped 1 previous similar message [ 972.769436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 972.773134] Lustre: Skipped 1 previous similar message [ 973.889757] Lustre: server umount lustre-MDT0000 complete [ 975.549056] LustreError: 44717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324779 with bad export cookie 5810069908482183961 [ 975.551876] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 975.554383] LustreError: 44717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 981.790039] Lustre: server umount lustre-MDT0001 complete [ 986.513147] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 987.706408] Key type lgssc unregistered [ 987.858478] LNet: 46454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.864419] LNetError: 46454:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.874350] LNet: Removed LNI 192.168.204.129@tcp [ 988.231576] Key type .llcrypt unregistered [ 988.233245] Key type ._llcrypt unregistered [ 998.501820] Key type ._llcrypt registered [ 998.503602] Key type .llcrypt registered [ 998.552870] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 998.913840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.921022] alg: No test for adler32 (adler32-zlib) [ 999.804954] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 999.911412] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1001.511244] Key type lgssc registered [ 1001.941194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1004.873798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1004.886092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.069534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.423635] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1010.727127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.896360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1012.435740] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1014.501131] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1015.910592] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1018.516192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.647246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.842661] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1023.027447] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1030.952427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1036.789392] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1038.132171] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1049.053545] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-0110h40m51s [ 1049.692927] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1051.618714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.620050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.625398] Lustre: Skipped 2 previous similar messages [ 1051.628403] Lustre: Skipped 1 previous similar message [ 1052.641548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1052.645157] Lustre: Skipped 3 previous similar messages [ 1054.936875] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 1056.740769] Lustre: server umount lustre-MDT0000 complete [ 1058.173692] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1058.787269] LustreError: 47830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.795077] LustreError: 47830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1061.007697] LustreError: 49084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.099772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.173878] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.286855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.321794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1062.311275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1062.912304] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1064.296406] Lustre: DEBUG MARKER: Wait for df (35973) ... [ 1064.908216] Lustre: DEBUG MARKER: done [ 1065.286508] LustreError: 47811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.29@tcp arrived at 1780324868 with bad export cookie 9134282754095403815 [ 1065.293605] LustreError: 47811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1066.464158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1066.467617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1066.473206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1066.475878] Lustre: Skipped 1 previous similar message [ 1066.975913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1068.126897] Lustre: server umount lustre-OST0000 complete [ 1072.608801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1072.609533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.615295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.620909] Lustre: Skipped 2 previous similar messages [ 1072.623643] Lustre: Skipped 3 previous similar messages [ 1075.761211] Lustre: server umount lustre-MDT0000 complete [ 1077.445460] LustreError: 50138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324880 with bad export cookie 9134282754095404284 [ 1077.447404] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.452742] LustreError: 50138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1077.593350] Lustre: server umount lustre-MDT0001 complete [ 1081.754520] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1082.920688] Key type lgssc unregistered [ 1083.058496] LNet: 52389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.062506] LNetError: 52389:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.072528] LNet: Removed LNI 192.168.204.129@tcp [ 1083.423824] Key type .llcrypt unregistered [ 1083.426648] Key type ._llcrypt unregistered [ 1100.462508] Key type ._llcrypt registered [ 1100.464179] Key type .llcrypt registered [ 1100.511677] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1101.037538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1101.047977] alg: No test for adler32 (adler32-zlib) [ 1101.922163] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1102.021785] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1103.615716] Key type lgssc registered [ 1104.128849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1107.294984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1107.303403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.428339] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1108.441213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1108.511890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.082469] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1113.570304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.618082] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1113.636145] Lustre: 54558:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1113.775896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1115.558906] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1117.913108] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1119.482491] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1122.506326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1122.602156] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1122.689368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.131855] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1127.456134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1127.624703] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1132.390714] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-0110h42m15s [ 1133.131898] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1136.486756] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1137.693508] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1137.716795] LustreError: 54563:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1157.858394] Lustre: lustre-MDT0000: Client d9d2056f-f43e-4c81-a4f7-658a4e80bd5c (at 192.168.204.29@tcp) reconnecting [ 1177.759428] LustreError: 54563:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1178.850917] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1179.986890] Lustre: DEBUG MARKER: done [ 1183.715855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.716760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1183.721958] Lustre: Skipped 1 previous similar message [ 1183.725868] LustreError: Skipped 1 previous similar message [ 1183.727670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1183.735912] Lustre: Skipped 1 previous similar message [ 1188.377861] Lustre: server umount lustre-OST0000 complete [ 1190.879974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1190.883683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.890188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.906388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.910278] Lustre: Skipped 2 previous similar messages [ 1194.976986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1194.980170] Lustre: Skipped 1 previous similar message [ 1201.125849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.129627] Lustre: Skipped 1 previous similar message [ 1204.191150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1204.287119] Lustre: server umount lustre-MDT0000 complete [ 1205.942668] LustreError: 55279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325009 with bad export cookie 15822229398606578296 [ 1205.945220] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.947573] LustreError: 55279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1206.100598] Lustre: server umount lustre-MDT0001 complete [ 1210.731626] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1211.952934] Key type lgssc unregistered [ 1212.119066] LNet: 58523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.123931] LNetError: 58523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.135784] LNet: Removed LNI 192.168.204.129@tcp [ 1212.580396] Key type .llcrypt unregistered [ 1212.582384] Key type ._llcrypt unregistered [ 1233.094601] Key type ._llcrypt registered [ 1233.096491] Key type .llcrypt registered [ 1233.148544] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1233.558753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1233.567161] alg: No test for adler32 (adler32-zlib) [ 1234.471462] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1234.587785] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1236.191177] Key type lgssc registered [ 1236.752488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1238.372571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1238.384714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.576340] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1239.593742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1239.689546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.173873] Lustre: Failing over lustre-MDT0000 [ 1240.326855] Lustre: server umount lustre-MDT0000 complete [ 1243.530913] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:44:06 (1780325046) [ 1246.670510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.899404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.743661] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1252.804711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.868773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1252.886477] Lustre: 61295:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1253.038719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1254.868944] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1257.488712] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1259.234318] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1262.398732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1262.496062] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1262.588937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.943945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1265.137713] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1267.593584] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1272.176152] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1278.433334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.434730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1278.443620] Lustre: Skipped 2 previous similar messages [ 1278.446644] Lustre: Skipped 1 previous similar message [ 1280.686208] Lustre: server umount lustre-MDT0000 complete [ 1282.445724] LustreError: 61289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325085 with bad export cookie 3697994017771037175 [ 1282.449456] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.451332] LustreError: 61289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1282.577831] Lustre: server umount lustre-MDT0001 complete [ 1284.286276] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1286.546419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1290.645439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.684359] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3351e9f762c32ad3 [ 1291.692165] Lustre: MGC192.168.204.129@tcp: Connection restored to 0@lo (at 0@lo) [ 1291.875204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.933853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1293.746264] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1297.376441] LustreError: 62918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.377216] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.390000] LustreError: 62918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1297.939638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1299.180056] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1299.944354] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1300.191205] Lustre: 60221:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780325087/real 1780325087] req@000000009b24f558 x1866806107078016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780325103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1302.387207] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1304.081918] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1304.543141] Lustre: 60219:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780325092/real 1780325092] req@00000000516a7443 x1866806107078272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780325108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1304.562744] Lustre: 60219:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1307.743387] Lustre: 60220:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780325095/real 1780325095] req@00000000f47906e1 x1866806107078528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780325111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1308.641499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1308.648764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.663112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.668609] Lustre: Skipped 1 previous similar message [ 1309.667625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1309.670432] Lustre: Skipped 1 previous similar message [ 1312.896098] Lustre: server umount lustre-MDT0000 complete [ 1314.784380] LustreError: 64725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.798009] LustreError: 64725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1315.107881] LustreError: 61289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325118 with bad export cookie 3697994017771039443 [ 1315.109060] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1315.114763] LustreError: 61289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1315.326871] Lustre: server umount lustre-MDT0001 complete [ 1319.167788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1325.053791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.287292] Lustre: 60219:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780325123/real 1780325123] req@0000000071ce6d92 x1866806107099136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780325139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1336.307703] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.316236] Lustre: Skipped 2 previous similar messages [ 1340.450839] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3351e9f762c33036 [ 1340.460981] Lustre: MGC192.168.204.129@tcp: Connection restored to 0@lo (at 0@lo) [ 1340.468724] Lustre: Skipped 1 previous similar message [ 1340.635942] LustreError: 65440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.673267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.678019] Lustre: Skipped 1 previous similar message [ 1340.709863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1341.471196] Lustre: 60221:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780325128/real 1780325128] req@000000000eaf4f25 x1866806107099392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780325144 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1341.485854] Lustre: 60221:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1343.216949] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1345.763642] LustreError: 65440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.781178] LustreError: 65440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1348.661175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.008348] Lustre: 60220:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780325138/real 1780325138] req@000000007d41abe4 x1866806107099904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780325154 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1351.024341] Lustre: 60220:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1351.435871] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1354.220745] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1354.683837] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1356.727930] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1364.449180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.449746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1364.450343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.455174] Lustre: Skipped 1 previous similar message [ 1365.934673] Lustre: server umount lustre-MDT0000 complete [ 1367.613338] LustreError: 62925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325171 with bad export cookie 3697994017771040822 [ 1367.616283] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.619991] LustreError: 62925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1367.747141] Lustre: server umount lustre-MDT0001 complete [ 1370.861698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1372.436986] Lustre: DEBUG MARKER: files compared the same [ 1383.629962] Lustre: server umount lustre-OST0000 complete [ 1390.999178] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1392.306576] Key type lgssc unregistered [ 1392.460427] LNet: 70208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.465124] LNetError: 70208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.478073] LNet: Removed LNI 192.168.204.129@tcp [ 1392.844367] Key type .llcrypt unregistered [ 1392.846360] Key type ._llcrypt unregistered [ 1404.173701] Key type ._llcrypt registered [ 1404.175724] Key type .llcrypt registered [ 1404.252211] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1404.749712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1404.849256] alg: No test for adler32 (adler32-zlib) [ 1405.744805] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1405.880325] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1407.479235] Key type lgssc registered [ 1407.997658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1411.727849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1411.745357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.973632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.657356] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug +malloc all [ 1418.586364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1418.737226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1420.924049] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug +malloc all [ 1423.743684] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1425.608534] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1428.868901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1428.960598] LustreError: 73177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.969768] LustreError: 73177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1429.030945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.613789] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug +malloc all [ 1433.825520] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1438.184559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1443.297209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.297826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.302159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1443.310030] Lustre: Skipped 1 previous similar message [ 1443.314948] Lustre: Skipped 1 previous similar message [ 1445.272845] Lustre: server umount lustre-OST0000 complete [ 1448.417493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.421361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1448.424899] Lustre: Skipped 1 previous similar message [ 1448.432360] Lustre: Skipped 1 previous similar message [ 1449.353235] Lustre: server umount lustre-MDT0000 complete [ 1451.446370] LustreError: 71566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325254 with bad export cookie 6341497363283734152 [ 1451.447630] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1451.452753] LustreError: 71566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1451.610979] Lustre: server umount lustre-MDT0001 complete [ 1456.506339] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1457.875972] Key type lgssc unregistered [ 1458.049254] LNet: 74755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.055619] LNetError: 74755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.066756] LNet: Removed LNI 192.168.204.129@tcp [ 1458.429553] Key type .llcrypt unregistered [ 1458.431521] Key type ._llcrypt unregistered [ 1470.784856] Key type ._llcrypt registered [ 1470.787740] Key type .llcrypt registered [ 1470.842423] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1471.417286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1471.438695] alg: No test for adler32 (adler32-zlib) [ 1472.334294] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1472.447759] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1474.048286] Key type lgssc registered [ 1474.668963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1478.596223] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1478.605882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1494.111514] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1499.231270] LustreError: 76107:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 1525.855211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.785221] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1531.291127] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1535.268488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.336114] Lustre: *** cfs_fail_loc=706, val=0*** [ 1535.519347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.562303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1537.584659] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1540.585424] LustreError: 76133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.593685] LustreError: 76133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1541.606538] LustreError: 77078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.614731] LustreError: 77078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1542.145739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.391456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1544.655566] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1547.345850] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1549.304503] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1550.819491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1550.826326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.839762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1552.868587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1552.870844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1552.873912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.654082] Lustre: server umount lustre-OST0000 complete [ 1563.106478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.106560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1563.107504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1563.107509] Lustre: Skipped 1 previous similar message [ 1563.114779] Lustre: Skipped 1 previous similar message [ 1564.738560] Lustre: server umount lustre-MDT0000 complete [ 1566.819561] LustreError: 76115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325370 with bad export cookie 17159505526276432785 [ 1566.822222] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1566.832814] LustreError: 76115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1567.026217] Lustre: server umount lustre-MDT0001 complete [ 1572.121401] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1573.589462] Key type lgssc unregistered [ 1573.761691] LNet: 79362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.764811] LNetError: 79362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.776645] LNet: Removed LNI 192.168.204.129@tcp [ 1574.184095] Key type .llcrypt unregistered [ 1574.185757] Key type ._llcrypt unregistered [ 1587.352442] Key type ._llcrypt registered [ 1587.353969] Key type .llcrypt registered [ 1587.420329] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1587.965705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1587.977484] alg: No test for adler32 (adler32-zlib) [ 1588.889131] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1589.030554] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1590.639146] Key type lgssc registered [ 1591.219054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1595.107081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1595.120147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.851751] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1601.189454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.465635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1603.092648] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1606.624493] LustreError: 81409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.631165] LustreError: 81409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1606.643954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.802889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.202219] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1610.913890] LustreError: 81413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.924186] LustreError: 81413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1612.937250] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1612.944938] Lustre: MGS: Client 580be3d9-17ab-45a2-b67a-a1c003dc5684 (at 0@lo) reconnecting [ 1612.955270] LustreError: 81423:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1613.102744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.165820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1615.036972] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1628.644674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1628.646074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.656096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1628.664167] Lustre: Skipped 1 previous similar message [ 1628.671287] Lustre: Skipped 1 previous similar message [ 1631.006206] Lustre: server umount lustre-OST0000 complete [ 1633.760837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.761110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1633.763122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.767063] Lustre: Skipped 1 previous similar message [ 1638.880202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.883859] Lustre: Skipped 2 previous similar messages [ 1644.001456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.006887] Lustre: Skipped 1 previous similar message [ 1647.071159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1647.122761] Lustre: server umount lustre-MDT0000 complete [ 1648.959061] Lustre: server umount lustre-MDT0001 complete [ 1650.594218] Lustre: server umount MGS complete [ 1655.595144] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:50:58 (1780325458) [ 1668.454649] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 1672.319552] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1688.069622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.012229] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1695.406473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.582829] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1695.613446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1695.661154] Lustre: lustre-MDT0001: new disk, initializing [ 1695.705667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1695.721386] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1695.726220] Lustre: 88464:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1695.731717] LustreError: 88464:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1698.080703] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1703.781635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1703.945544] Lustre: lustre-OST0000: new disk, initializing [ 1703.953804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1703.963658] Lustre: 89215:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1704.014269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.215566] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1708.896056] Lustre: 89621:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000086746cc x1866806478278016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780325528 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1710.990875] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1714.057445] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1714.064113] Lustre: MGS: Client ac4e5911-0c0d-4b84-ba87-15d0639eb00a (at 0@lo) reconnecting [ 1714.080554] Lustre: 87749:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1714.090953] LustreError: 88449:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1714.096419] LustreError: 88449:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1714.104901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1714.145830] Lustre: lustre-MDT0000: new disk, initializing [ 1714.214959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.225384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1715.297203] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1715.297572] Lustre: 90170:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000b3dd0580 x1866806478284160/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1780325534 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1715.321507] Lustre: 90170:0:(client.c:1638:after_reply()) Skipped 1 previous similar message [ 1716.024688] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1717.348847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1717.355097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1717.361684] Lustre: Skipped 1 previous similar message [ 1717.376082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1736.162778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.164048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1736.171507] Lustre: Skipped 1 previous similar message [ 1741.221396] Lustre: server umount lustre-OST0000 complete [ 1746.403375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.403532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1746.410707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.417283] Lustre: Skipped 2 previous similar messages [ 1757.151178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1757.203941] Lustre: server umount lustre-MDT0000 complete [ 1765.319124] Lustre: server umount lustre-MDT0001 complete [ 1767.283134] Lustre: server umount MGS complete [ 1772.006588] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:52:54 (1780325574) [ 1778.588716] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1779.756651] Key type lgssc unregistered [ 1779.921546] LNet: 92628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.924337] LNetError: 92628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.933313] LNet: Removed LNI 192.168.204.129@tcp [ 1780.282568] Key type .llcrypt unregistered [ 1780.284687] Key type ._llcrypt unregistered [ 1788.715107] Key type ._llcrypt registered [ 1788.716696] Key type .llcrypt registered [ 1788.761288] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1789.268778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1789.277859] alg: No test for adler32 (adler32-zlib) [ 1790.177352] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1790.292235] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1791.895210] Key type lgssc registered [ 1792.373725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1795.825939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1795.840254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.100109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1798.733891] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1802.190190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.356443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1803.943433] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1806.179890] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1807.598645] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1810.641326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1810.784529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1813.321195] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1815.702088] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1822.056176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1832.418187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.418694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1832.419660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1832.424914] Lustre: Skipped 1 previous similar message [ 1836.068099] Lustre: server umount lustre-OST0000 complete [ 1838.559839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1838.563173] LustreError: Skipped 1 previous similar message [ 1838.565479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.571144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.573796] Lustre: Skipped 1 previous similar message [ 1842.656673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.656699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.756703] Lustre: server umount lustre-MDT0000 complete [ 1845.391618] LustreError: 93790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325648 with bad export cookie 17108932126838678088 [ 1845.397203] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1845.398454] LustreError: 93790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1845.582533] Lustre: server umount lustre-MDT0001 complete [ 1850.253873] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1851.510935] Key type lgssc unregistered [ 1851.665430] LNet: 97003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.668928] LNetError: 97003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.683190] LNet: Removed LNI 192.168.204.129@tcp [ 1852.092941] Key type .llcrypt unregistered [ 1852.094820] Key type ._llcrypt unregistered [ 1859.782265] Key type ._llcrypt registered [ 1859.783935] Key type .llcrypt registered [ 1859.829892] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1860.242625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1860.280287] alg: No test for adler32 (adler32-zlib) [ 1861.166658] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1861.288551] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1862.887637] Key type lgssc registered [ 1863.353789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1866.008637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1866.008678] LustreError: 98047:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1866.012677] Lustre: Skipped 1 previous similar message [ 1866.019015] LustreError: 98048:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1866.023063] LustreError: 98047:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 1866.029473] LustreError: 98047:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1866.035442] LustreError: 98047:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1866.040504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.275038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.334112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.465180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1872.048993] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1874.095123] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing lsmod [ 1875.286478] LustreError: 99218:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1875.286481] LustreError: 99217:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1875.295078] LustreError: 99218:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 1875.295454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1875.299188] LustreError: 99218:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1875.308838] LustreError: 99218:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1875.464615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.649600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.652255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.658470] Lustre: Skipped 2 previous similar messages [ 1879.662545] Lustre: Skipped 1 previous similar message [ 1882.719388] Lustre: server umount lustre-MDT0000 complete [ 1884.308299] LustreError: 98066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325687 with bad export cookie 6443090657502207376 [ 1884.311051] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1884.313046] LustreError: 98066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1885.152326] LustreError: 98081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.152440] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.159358] LustreError: 98081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1885.167256] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1885.167264] Lustre: Skipped 1 previous similar message [ 1885.792549] LustreError: 98516:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004144dd4e x1866806763470080/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1885.877457] Lustre: server umount lustre-MDT0001 complete [ 1887.748078] Lustre: server umount lustre-OST0000 complete [ 1891.583315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.815549] LustreError: 100282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.851403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.660501] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1896.928062] LustreError: 100283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.604637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.787808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1899.696631] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1902.188292] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1903.985318] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1907.329496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1907.489907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.227121] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1912.993688] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1916.712691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1922.017818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1922.019258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.020295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1922.034379] Lustre: Skipped 1 previous similar message [ 1926.821411] Lustre: server umount lustre-OST0000 complete [ 1928.675117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1928.679288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.685344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.688548] Lustre: Skipped 1 previous similar message [ 1934.840854] Lustre: server umount lustre-MDT0000 complete [ 1936.525201] LustreError: 100262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325739 with bad export cookie 6443090657502208048 [ 1936.527887] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1936.534544] LustreError: 100262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1936.664973] Lustre: server umount lustre-MDT0001 complete [ 1941.215699] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1942.592473] Key type lgssc unregistered [ 1942.775477] LNet: 103453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.780547] LNetError: 103453:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.798647] LNet: Removed LNI 192.168.204.129@tcp [ 1943.180717] Key type .llcrypt unregistered [ 1943.183884] Key type ._llcrypt unregistered [ 1954.604102] Key type ._llcrypt registered [ 1954.605559] Key type .llcrypt registered [ 1954.660217] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1955.078418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.146788] alg: No test for adler32 (adler32-zlib) [ 1956.066968] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1956.170721] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1957.783226] Key type lgssc registered [ 1958.321937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1962.253875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1962.267798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.555106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.586966] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1969.486953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.686399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1971.586250] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1974.297826] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1975.996514] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1979.132443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1979.330978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.032190] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1984.516644] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1988.522861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1993.048754] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1994.430181] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1998.817326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.819964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1998.832151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1998.840077] Lustre: Skipped 1 previous similar message [ 2002.343944] Lustre: server umount lustre-OST0000 complete [ 2005.472909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2005.475890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.487298] Lustre: Skipped 1 previous similar message [ 2005.490440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.056927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.057445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.471131] Lustre: server umount lustre-MDT0000 complete [ 2012.585352] LustreError: 107019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325816 with bad export cookie 17908174603734356380 [ 2012.588260] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2012.589910] LustreError: 107019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2012.764901] Lustre: server umount lustre-MDT0001 complete [ 2018.669267] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2020.293796] Key type lgssc unregistered [ 2020.470688] LNet: 108111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.474976] LNetError: 108111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2020.483755] LNet: Removed LNI 192.168.204.129@tcp [ 2020.935693] Key type .llcrypt unregistered [ 2020.938636] Key type ._llcrypt unregistered [ 2029.544989] Key type ._llcrypt registered [ 2029.551405] Key type .llcrypt registered [ 2029.607091] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2030.074483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2030.138224] alg: No test for adler32 (adler32-zlib) [ 2031.068754] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 2031.184918] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2032.792094] Key type lgssc registered [ 2033.299234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2036.777942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2036.786642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.021310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.733072] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2043.761390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.958649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2045.674527] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2047.969934] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2049.452745] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2052.419045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2052.583914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.920342] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2057.106345] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2062.829699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2073.058176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.060224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2073.063060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2073.068648] Lustre: Skipped 1 previous similar message [ 2076.951459] Lustre: server umount lustre-OST0000 complete [ 2079.712655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2079.717272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.723093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.726326] Lustre: Skipped 1 previous similar message [ 2083.297195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.298776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.567797] Lustre: server umount lustre-MDT0000 complete [ 2086.200042] LustreError: 109275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325889 with bad export cookie 14773745317327818176 [ 2086.202219] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2086.206066] LustreError: 109275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2086.369663] Lustre: server umount lustre-MDT0001 complete [ 2090.897892] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2092.100770] Key type lgssc unregistered [ 2092.258536] LNet: 112490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.261844] LNetError: 112490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.270474] LNet: Removed LNI 192.168.204.129@tcp [ 2092.589431] Key type .llcrypt unregistered [ 2092.591120] Key type ._llcrypt unregistered [ 2103.303139] Key type ._llcrypt registered [ 2103.304324] Key type .llcrypt registered [ 2103.347841] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2103.644138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2103.686316] alg: No test for adler32 (adler32-zlib) [ 2104.559398] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 2104.654928] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2106.255223] Key type lgssc registered [ 2106.758627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2110.054029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2110.065223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.275682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2112.975089] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2116.874119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.136922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2119.004316] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2121.550359] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2123.097533] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2125.665068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2125.822700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.958703] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2129.902258] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2136.102733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2138.820286] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2140.263894] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2140.267952] Lustre: Skipped 2 previous similar messages [ 2141.960146] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2141.964198] Lustre: Skipped 2 previous similar messages [ 2145.278740] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2145.290253] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2148.552624] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.29@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2148.556779] Lustre: Skipped 2 previous similar messages [ 2153.990839] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.29@tcp 192.168.0.[2,10]@tcp [ 2155.627598] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.29/24@tcp in log lustre-MDT0000 [ 2155.630948] Lustre: Skipped 2 previous similar messages [ 2159.547137] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.29/24@tcp [ 2159.549468] Lustre: Skipped 1 previous similar message [ 2162.656445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2162.663115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2162.670089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2166.754236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2166.757282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2166.758296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.759056] LustreError: Skipped 1 previous similar message [ 2166.764404] Lustre: Skipped 1 previous similar message [ 2168.471171] Lustre: server umount lustre-OST0000 complete [ 2171.872601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.873411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2171.878060] Lustre: Skipped 1 previous similar message [ 2176.027214] Lustre: server umount lustre-MDT0000 complete [ 2176.992072] LustreError: 113867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.998224] LustreError: 113867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2177.380162] LustreError: 113847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325980 with bad export cookie 4936391614185438301 [ 2177.382438] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.384947] LustreError: 113847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2177.519918] Lustre: server umount lustre-MDT0001 complete [ 2181.144253] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2182.206744] Key type lgssc unregistered [ 2182.336283] LNet: 118535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.338816] LNetError: 118535:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.350441] LNet: Removed LNI 192.168.204.129@tcp [ 2182.643613] Key type .llcrypt unregistered [ 2182.644698] Key type ._llcrypt unregistered [ 2192.527072] Key type ._llcrypt registered [ 2192.528468] Key type .llcrypt registered [ 2192.574112] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2192.913739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2192.947540] alg: No test for adler32 (adler32-zlib) [ 2193.806497] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 2193.891452] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2195.471183] Key type lgssc registered [ 2195.843728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2198.483438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2198.491544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.654437] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2199.662408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.665440] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.29/24@tcp [ 2201.071417] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2204.017229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2204.136125] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2204.146821] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.29/24@tcp [ 2204.164729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2205.577626] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2207.555612] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2208.863642] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2211.367621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2211.501816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.557834] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2215.446212] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2224.804376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2229.215728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2229.218722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.223634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.241067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.241527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.242986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2230.995214] Lustre: server umount lustre-OST0000 complete [ 2234.386837] Lustre: server umount lustre-MDT0000 complete [ 2234.847928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2234.850929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.856434] LustreError: 120055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.863048] LustreError: 120055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2235.360447] LustreError: 122113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.366969] LustreError: 122113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2235.703079] LustreError: 120035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326039 with bad export cookie 14553464415128973368 [ 2235.706638] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2235.707432] LustreError: 120035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2235.812835] Lustre: server umount lustre-MDT0001 complete [ 2239.289801] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2240.331442] Key type lgssc unregistered [ 2240.467368] LNet: 123303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.469584] LNetError: 123303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.477381] LNet: Removed LNI 192.168.204.129@tcp [ 2240.791767] Key type .llcrypt unregistered [ 2240.792787] Key type ._llcrypt unregistered [ 2249.443540] Key type ._llcrypt registered [ 2249.445212] Key type .llcrypt registered [ 2249.486970] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 2255.590202] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2255.994457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2256.006203] alg: No test for adler32 (adler32-zlib) [ 2256.866139] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 2256.949051] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2258.527178] Key type lgssc registered [ 2258.955383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2271.085742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2271.092119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.191806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2272.210582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2272.253832] Lustre: lustre-MDT0000: new disk, initializing [ 2272.288615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.299450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2273.708954] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2278.108744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.152921] Lustre: 126655:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2278.177503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2278.179675] Lustre: Skipped 1 previous similar message [ 2278.212988] Lustre: lustre-MDT0001: new disk, initializing [ 2278.246226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.260663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2278.265206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2279.626637] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2282.842611] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2284.171505] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2286.586539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.686299] Lustre: lustre-OST0000: new disk, initializing [ 2286.689359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2286.692693] Lustre: 128396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2286.720741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.585783] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2288.888741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2288.891948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2288.927180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2291.550371] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2292.868438] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2292.940831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2294.185670] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2294.258436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2296.976655] Lustre: server umount lustre-OST0000 complete [ 2298.847966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2298.852339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.859350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2299.360491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2299.360854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.480193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.482689] Lustre: Skipped 2 previous similar messages [ 2307.579532] Lustre: server umount lustre-MDT0000 complete [ 2309.076697] LustreError: 126647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326112 with bad export cookie 12100726459328532907 [ 2309.080091] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2309.081783] LustreError: 126647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2319.382373] Lustre: server umount lustre-MDT0001 complete [ 2322.592966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.794588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.132624] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2327.386321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.507895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.791652] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2330.560410] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2331.798344] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2334.058552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2336.157867] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2338.170289] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2339.711570] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2348.989494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2350.453991] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2350.529333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2354.199422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2354.263539] Lustre: lustre-OST0001: new disk, initializing [ 2354.265767] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2354.269498] Lustre: 133523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2354.300782] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2354.304322] Lustre: Skipped 1 previous similar message [ 2356.218452] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2359.620658] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2359.808958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2359.813843] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2359.844198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2360.726702] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2362.081449] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2362.161648] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2363.475308] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dc005179800.ost_server_uuid 50 [ 2364.052578] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dc005179800.ost_server_uuid in FULL state after 0 sec [ 2370.020850] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2370.020950] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.023852] LustreError: Skipped 1 previous similar message [ 2370.028227] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2370.031790] Lustre: Skipped 1 previous similar message [ 2372.224658] Lustre: server umount lustre-OST0001 complete [ 2374.112056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2374.115332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.119814] Lustre: Skipped 1 previous similar message [ 2374.122250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2375.137454] LustreError: 132889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.143172] LustreError: 132889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2379.599910] Lustre: server umount lustre-OST0000 complete [ 2383.839697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2383.842288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.846110] Lustre: Skipped 1 previous similar message [ 2383.847851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2383.850057] Lustre: Skipped 2 previous similar messages [ 2395.103108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2395.167801] Lustre: server umount lustre-MDT0000 complete [ 2396.639282] LustreError: 130445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326200 with bad export cookie 12100726459328533775 [ 2396.643122] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2396.645872] LustreError: 130445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2396.797062] Lustre: server umount lustre-MDT0001 complete [ 2403.948496] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2405.049390] Key type lgssc unregistered [ 2405.190482] LNet: 136372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.194477] LNetError: 136372:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2405.206530] LNet: Removed LNI 192.168.204.129@tcp [ 2405.532188] Key type .llcrypt unregistered [ 2405.534060] Key type ._llcrypt unregistered [ 2419.773711] Key type ._llcrypt registered [ 2419.775181] Key type .llcrypt registered [ 2419.818326] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 2426.111730] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2426.487030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2426.561493] alg: No test for adler32 (adler32-zlib) [ 2427.425829] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 2427.509322] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2429.087144] Key type lgssc registered [ 2429.492610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2441.147992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2441.154111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.244495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2442.257066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2442.298067] Lustre: lustre-MDT0000: new disk, initializing [ 2442.324249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.332236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2443.756801] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2447.960687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.997258] Lustre: 140464:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2448.019559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2448.022077] Lustre: Skipped 1 previous similar message [ 2448.062171] Lustre: lustre-MDT0001: new disk, initializing [ 2448.097858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2448.111395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2448.115826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2449.359384] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2452.532382] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2453.708900] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2455.964508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2456.078814] Lustre: lustre-OST0000: new disk, initializing [ 2456.081057] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2456.083720] Lustre: 142206:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2456.125105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.148956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2457.153045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2457.195125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2458.079327] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2461.203266] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2463.113869] Lustre: Failing over lustre-OST0000 [ 2463.170967] Lustre: server umount lustre-OST0000 complete [ 2476.082884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2476.199863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.204767] Lustre: Skipped 1 previous similar message [ 2476.208299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.213695] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2477.588495] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2478.200856] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2479.458300] Lustre: Failing over lustre-MDT0000 [ 2479.548327] Lustre: server umount lustre-MDT0000 complete [ 2481.633365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.638153] Lustre: Skipped 2 previous similar messages [ 2482.655155] Lustre: 138692:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780326270/real 1780326270] req@00000000c24cdac3 x1866807356976128/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780326286 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2482.665497] Lustre: 138692:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2486.752615] LustreError: 140470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.760406] LustreError: 140470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2487.775168] Lustre: 138694:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780326275/real 1780326275] req@00000000c9f4a231 x1866807356977408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780326291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2491.873063] LustreError: 140475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.880131] LustreError: 140475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2492.516425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.566402] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2492.571612] Lustre: 140467:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.129@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2492.659632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2492.676943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.694097] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2493.973276] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2497.228628] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2498.021063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2498.024055] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2498.049731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2503.138489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.138831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2503.142909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.144677] LustreError: Skipped 1 previous similar message [ 2503.148191] Lustre: Skipped 1 previous similar message [ 2505.164238] Lustre: server umount lustre-OST0000 complete [ 2508.256781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.257228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2508.257439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.262564] Lustre: Skipped 2 previous similar messages [ 2512.537165] Lustre: server umount lustre-MDT0000 complete [ 2513.376171] LustreError: 140475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.381764] LustreError: 140475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2513.838290] LustreError: 141241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326317 with bad export cookie 13492820926844747667 [ 2513.841698] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2513.843814] LustreError: 141241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2513.947785] Lustre: server umount lustre-MDT0001 complete [ 2517.287933] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2518.247717] Key type lgssc unregistered [ 2518.364325] LNet: 145686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2518.367544] LNetError: 145686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2518.378288] LNet: Removed LNI 192.168.204.129@tcp [ 2518.636639] Key type .llcrypt unregistered [ 2518.637691] Key type ._llcrypt unregistered [ 2538.579214] Key type ._llcrypt registered [ 2538.580465] Key type .llcrypt registered [ 2538.625171] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2539.123090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2539.162365] alg: No test for adler32 (adler32-zlib) [ 2540.019469] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 2540.102870] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2541.679191] Key type lgssc registered [ 2542.030530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2544.510319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2544.516789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.663834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.910852] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2549.987234] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2551.973815] LustreError: 147216:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2551.978020] LustreError: 147216:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2551.980930] LustreError: 147216:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2552.097345] Lustre: server umount lustre-MDT0000 complete [ 2556.921184] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 2560.157637] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2571.162686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.248610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2571.258210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2571.295692] Lustre: lustre-MDT0000: new disk, initializing [ 2571.318593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.324392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2572.555214] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2576.595513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.637718] Lustre: 150367:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2576.658865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2576.661141] Lustre: Skipped 1 previous similar message [ 2576.697852] Lustre: lustre-MDT0001: new disk, initializing [ 2576.729102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2576.740693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2576.744342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2578.035427] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2581.087821] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2582.239834] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2584.553051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2584.649789] Lustre: lustre-OST0000: new disk, initializing [ 2584.652114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2584.655712] Lustre: 152106:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2584.689571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.493725] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2589.702854] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2593.275184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2593.278790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2593.318046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2596.738320] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2597.302891] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:06:40 (1780326400) [ 2663.903559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2663.906388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.910658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.927398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.927757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.929972] Lustre: Skipped 1 previous similar message [ 2664.935964] Lustre: Skipped 1 previous similar message [ 2666.850929] Lustre: server umount lustre-MDT0000 complete [ 2668.056615] LustreError: 150360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326471 with bad export cookie 13668278092022545334 [ 2668.057444] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.061404] LustreError: 150360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2670.047443] LustreError: 150403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.047506] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2670.053511] LustreError: 150403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2670.059988] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2670.062377] Lustre: Skipped 1 previous similar message [ 2673.055625] LustreError: 153752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.055639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2678.175591] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2678.175592] LustreError: 150374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.335095] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2682.400958] Lustre: server umount lustre-MDT0001 complete [ 2689.668683] Lustre: server umount lustre-OST0000 complete [ 2692.454589] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 2695.466792] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2706.616639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.699676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2706.711242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2706.749249] Lustre: lustre-MDT0000: new disk, initializing [ 2706.774471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.781683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2708.074268] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2712.102717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.143746] Lustre: 156588:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2712.165716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2712.167905] Lustre: Skipped 1 previous similar message [ 2712.208542] Lustre: lustre-MDT0001: new disk, initializing [ 2712.258063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2712.262594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2713.504199] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2716.456828] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2717.569044] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2719.676024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2719.766459] Lustre: 158325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2719.790278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.793084] Lustre: Skipped 1 previous similar message [ 2721.485242] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2724.418499] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2725.680502] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2727.411750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2727.415891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2727.437973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2727.782812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2728.983889] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2729.049527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2732.513580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.513850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.516895] LustreError: Skipped 1 previous similar message [ 2732.523239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.526504] Lustre: Skipped 1 previous similar message [ 2735.758989] Lustre: server umount lustre-OST0000 complete [ 2737.632624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.636791] Lustre: Skipped 2 previous similar messages [ 2743.060636] Lustre: server umount lustre-MDT0000 complete [ 2744.292129] LustreError: 159735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326547 with bad export cookie 13668278092022704150 [ 2744.294829] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2744.296196] LustreError: 159735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2744.422883] Lustre: server umount lustre-MDT0001 complete [ 2747.162705] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:09:09 (1780326549) [ 2752.121996] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 2755.078798] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2765.903705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.984175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2765.994622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2765.996121] Lustre: Skipped 1 previous similar message [ 2766.029851] Lustre: lustre-MDT0000: new disk, initializing [ 2766.031717] Lustre: Skipped 1 previous similar message [ 2766.052084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2766.057700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2767.237994] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2771.293498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.333821] Lustre: 162910:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2771.424518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2772.630920] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2775.504902] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2776.679374] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2778.880378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.977132] Lustre: 164658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2780.777163] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2783.836405] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2785.783041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2785.816114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2790.881397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2790.884056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.888420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2790.890084] Lustre: Skipped 5 previous similar messages [ 2792.716290] Lustre: server umount lustre-OST0000 complete [ 2800.026419] Lustre: server umount lustre-MDT0000 complete [ 2801.119862] LustreError: 162919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.126248] LustreError: 162919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2801.297121] LustreError: 162901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326604 with bad export cookie 13668278092022705032 [ 2801.299597] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2801.301963] LustreError: 162901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2801.409454] Lustre: server umount lustre-MDT0001 complete [ 2804.254770] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:10:07 (1780326607) [ 2808.976152] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 2811.699298] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2823.123181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.239257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2823.252852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2823.254756] Lustre: Skipped 3 previous similar messages [ 2823.292947] Lustre: lustre-MDT0000: new disk, initializing [ 2823.295059] Lustre: Skipped 2 previous similar messages [ 2823.322529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.325619] Lustre: Skipped 2 previous similar messages [ 2823.334546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2823.336915] Lustre: Skipped 2 previous similar messages [ 2824.559438] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2828.622176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.662235] Lustre: 168824:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2828.774770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2830.059531] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2833.151033] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2834.259062] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2836.425044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2836.543693] Lustre: 170572:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2838.410512] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2840.347660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2841.483555] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2845.665285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2845.665976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.668437] LustreError: Skipped 2 previous similar messages [ 2845.672961] Lustre: Skipped 3 previous similar messages [ 2845.674969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2845.677558] Lustre: Skipped 3 previous similar messages [ 2849.485157] Lustre: server umount lustre-OST0000 complete [ 2856.790911] Lustre: server umount lustre-MDT0000 complete [ 2858.007994] LustreError: 168816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326661 with bad export cookie 13668278092022706103 [ 2858.011288] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2858.012550] LustreError: 168816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2858.113370] Lustre: server umount lustre-MDT0001 complete [ 2861.211422] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2862.169722] Key type lgssc unregistered [ 2862.298290] LNet: 172295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2862.300531] LNetError: 172295:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2862.309269] LNet: Removed LNI 192.168.204.129@tcp [ 2862.588238] Key type .llcrypt unregistered [ 2862.589335] Key type ._llcrypt unregistered [ 2871.240563] Key type ._llcrypt registered [ 2871.241639] Key type .llcrypt registered [ 2871.271274] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2871.634328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2871.641511] alg: No test for adler32 (adler32-zlib) [ 2872.486562] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 2872.559019] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2874.135139] Key type lgssc registered [ 2874.470459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2876.823822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2876.830181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.982922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.261972] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2881.913790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.031866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2883.259208] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2885.033506] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2886.131112] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2888.370837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.499232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.396818] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2891.984027] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2902.824608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2922.463844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2922.466456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.470724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2923.488846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2923.489087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2923.490081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2925.390304] Lustre: server umount lustre-OST0000 complete [ 2928.095972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2928.099043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.104473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.106097] Lustre: Skipped 1 previous similar message [ 2932.634300] Lustre: server umount lustre-MDT0000 complete [ 2933.728533] LustreError: 173670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.734015] LustreError: 173670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2933.837435] LustreError: 173650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326737 with bad export cookie 294443062410245266 [ 2933.841275] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2933.841443] LustreError: 173650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2933.966521] Lustre: server umount lustre-MDT0001 complete [ 2937.238716] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2938.170461] Key type lgssc unregistered [ 2938.289347] LNet: 176865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2938.291661] LNetError: 176865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2938.298223] LNet: Removed LNI 192.168.204.129@tcp [ 2938.544658] Key type .llcrypt unregistered [ 2938.545688] Key type ._llcrypt unregistered [ 2946.859414] Key type ._llcrypt registered [ 2946.860553] Key type .llcrypt registered [ 2946.895746] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2947.304749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2947.331947] alg: No test for adler32 (adler32-zlib) [ 2948.190448] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 2948.281242] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2949.871215] Key type lgssc registered [ 2950.279983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2952.931948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2952.938973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.105377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2955.357481] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2958.906377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.100555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2961.445180] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2964.909454] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2967.236675] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2970.817734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.011413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.172827] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2977.407535] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2979.180820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2989.543534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.544582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2989.546027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2989.561384] Lustre: Skipped 1 previous similar message [ 2990.303898] Lustre: lustre-OST0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 2990.307929] Lustre: Skipped 1 previous similar message [ 2994.660130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2999.779588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2999.781963] Lustre: Skipped 2 previous similar messages [ 3002.335128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3002.392644] Lustre: server umount lustre-OST0000 complete [ 3004.565454] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8dc005c8f800.ost_server_uuid 50 [ 3005.212733] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8dc005c8f800.ost_server_uuid in DISCONN state after 0 sec [ 3005.907320] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3018.629846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.637737] Lustre: Skipped 1 previous similar message [ 3018.640393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.644508] Lustre: Skipped 2 previous similar messages [ 3025.382878] Lustre: server umount lustre-MDT0000 complete [ 3028.745610] LustreError: 178221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326832 with bad export cookie 3603192217980490889 [ 3028.751383] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3028.755107] LustreError: 178221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3029.054891] Lustre: server umount lustre-MDT0001 complete [ 3036.759221] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:13:58 (1780326838) [ 3043.431190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3043.899373] LustreError: 181766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.922218] LustreError: 181766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3044.052689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.365825] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3049.443591] LustreError: 181767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.458595] LustreError: 181767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3053.541535] LustreError: 181766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.448081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.734571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3058.664539] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3063.249350] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3066.085896] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3071.128655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3071.393152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.314164] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3078.972520] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3083.140422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.272929] Lustre: lustre-OST0001: new disk, initializing [ 3083.275551] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3083.278568] Lustre: 184263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3086.858278] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3089.512311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3091.716226] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3092.496717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3092.501430] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3092.548250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3095.009111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3095.014874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.033564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.036203] Lustre: Skipped 3 previous similar messages [ 3099.297693] Lustre: server umount lustre-OST0000 complete [ 3102.419289] LustreError: 183359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3102.937131] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3103.055569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3105.291933] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3105.408988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3107.535515] LustreError: 183360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.545957] LustreError: 183360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3118.048869] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3118.049253] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.053516] LustreError: Skipped 2 previous similar messages [ 3118.054932] LustreError: 183360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.055734] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3118.055739] Lustre: Skipped 3 previous similar messages [ 3118.063739] Lustre: Skipped 2 previous similar messages [ 3118.081945] LustreError: 183360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3123.183967] Lustre: server umount lustre-OST0001 complete [ 3126.240435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3126.246237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.815289] Lustre: server umount lustre-MDT0000 complete [ 3133.914594] LustreError: 184262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326937 with bad export cookie 3603192217980491946 [ 3133.917019] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3133.924760] LustreError: 184262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3134.120693] Lustre: server umount lustre-MDT0001 complete [ 3149.734886] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:15:52 (1780326952) [ 3153.783396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.994034] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3154.013118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3154.153147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.157942] Lustre: Skipped 1 previous similar message [ 3156.438615] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3161.288821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.369767] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3161.406341] Lustre: 187551:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3163.898905] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3166.954150] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3168.889417] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3172.564736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3172.694370] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3175.832413] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3178.037423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3178.618962] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3182.774251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.820100] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3186.559792] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3189.852890] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3192.292037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.295190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.302698] Lustre: Skipped 2 previous similar messages [ 3192.308172] Lustre: Skipped 5 previous similar messages [ 3197.719650] Lustre: server umount lustre-OST0000 complete [ 3200.719623] LustreError: 189171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3200.731511] LustreError: 189171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3212.771531] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3212.774572] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3212.783741] LustreError: Skipped 1 previous similar message [ 3212.796220] Lustre: Skipped 1 previous similar message [ 3216.887855] Lustre: server umount lustre-OST0001 complete [ 3223.008126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3225.287316] Lustre: server umount lustre-MDT0000 complete [ 3227.513459] LustreError: 187542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327030 with bad export cookie 3603192217980493346 [ 3227.516294] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3227.518973] LustreError: 187542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3227.707751] Lustre: server umount lustre-MDT0001 complete [ 3241.595858] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:17:24 (1780327044) [ 3250.918690] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3257.730530] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3286.201571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.434972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3286.482642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3286.567485] Lustre: lustre-MDT0000: new disk, initializing [ 3286.630388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3286.633863] Lustre: Skipped 3 previous similar messages [ 3286.647523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3289.837438] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3298.116868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.216790] Lustre: 195152:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3298.269313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3298.275923] Lustre: Skipped 1 previous similar message [ 3298.295772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3298.302234] Lustre: Skipped 7 previous similar messages [ 3298.334193] Lustre: lustre-MDT0001: new disk, initializing [ 3298.427159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3298.435835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3301.631376] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3307.647923] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3310.534537] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3316.101349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3316.285612] Lustre: lustre-OST0000: new disk, initializing [ 3316.289888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3316.302293] Lustre: 196892:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3321.702829] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3324.504619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3324.518269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3324.659726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3328.398950] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3332.023573] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3332.248019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3335.593759] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3335.764947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3340.261865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3340.264936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.274946] LustreError: Skipped 1 previous similar message [ 3340.299881] Lustre: Skipped 3 previous similar messages [ 3343.743377] Lustre: server umount lustre-OST0000 complete [ 3353.835634] Lustre: server umount lustre-MDT0000 complete [ 3355.619865] LustreError: 195161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.640217] LustreError: 195161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3357.668056] LustreError: 195145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327161 with bad export cookie 3603192217980494886 [ 3357.680405] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.082664] Lustre: server umount lustre-MDT0001 complete [ 3365.334079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3365.756308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.764309] Lustre: Skipped 2 previous similar messages [ 3369.559616] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3376.985075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3381.469812] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3385.495599] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3388.472774] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3393.958517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3398.960781] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3403.331309] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3406.522272] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3411.000388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3414.253931] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3414.442186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3416.039835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3416.048485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.052930] LustreError: Skipped 1 previous similar message [ 3416.069725] Lustre: Skipped 3 previous similar messages [ 3418.274083] Lustre: server umount lustre-OST0000 complete [ 3423.063118] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3426.562877] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3453.245779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3456.552235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3456.562895] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3456.564160] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3456.579027] Lustre: Skipped 1 previous similar message [ 3457.322982] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3461.189813] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3474.426782] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 11 sec [ 3479.260458] Lustre: server umount lustre-OST0000 complete [ 3482.082143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.087266] Lustre: Skipped 8 previous similar messages [ 3492.365537] Lustre: server umount lustre-MDT0000 complete [ 3494.785347] LustreError: 198941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327298 with bad export cookie 3603192217980495775 [ 3494.788589] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3494.793592] LustreError: 198941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3505.192853] Lustre: server umount lustre-MDT0001 complete [ 3510.722789] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:21:53 (1780327313) [ 3514.172077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.406253] LustreError: 204453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.420481] LustreError: 204453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3514.451490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.458937] Lustre: Skipped 3 previous similar messages [ 3516.637766] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3521.439700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.965639] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3527.034420] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3529.029429] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3532.618499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.901079] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3539.300298] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3541.826622] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3541.939607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3544.309994] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3547.567552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3551.019865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3551.128109] Lustre: lustre-OST0001: new disk, initializing [ 3551.131757] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3551.136074] Lustre: 207469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3554.778504] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3559.440094] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3561.505073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3561.512834] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3561.593199] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3562.062722] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3562.229925] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3564.877866] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3565.001218] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3566.562741] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3566.569346] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.576677] Lustre: Skipped 2 previous similar messages [ 3568.181762] Lustre: server umount lustre-OST0001 complete [ 3572.170591] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3572.286546] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3574.769596] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3574.900498] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3600.873702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3604.577868] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3606.504621] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3606.514756] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3606.516135] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3606.527928] Lustre: Skipped 1 previous similar message [ 3607.781378] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3608.671247] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3615.976970] Lustre: server umount lustre-OST0001 complete [ 3636.337785] Lustre: server umount lustre-OST0000 complete [ 3644.640962] Lustre: server umount lustre-MDT0000 complete [ 3646.843634] LustreError: 205172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327450 with bad export cookie 3603192217980496741 [ 3646.846529] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3646.853794] LustreError: 205172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3661.279366] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3661.438521] Lustre: server umount lustre-MDT0001 complete [ 3674.570449] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:24:36 (1780327476) [ 3679.107912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3679.279913] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3679.297452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3681.918072] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3687.346239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.415441] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3687.432255] Lustre: 212413:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3690.410899] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3693.775394] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3696.129270] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3700.297488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3700.432938] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3704.299175] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3707.648944] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3713.943974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3714.001709] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3717.866502] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3721.640464] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3722.660760] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3725.537418] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3725.665901] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3728.200811] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3728.361364] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3730.501892] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dc0035b1800.ost_server_uuid 50 [ 3731.400709] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dc0035b1800.ost_server_uuid in FULL state after 0 sec [ 3732.102749] Lustre: Permanently deactivating lustre-OST0001 [ 3732.113535] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3734.222845] Lustre: Permanently reactivating lustre-OST0001 [ 3740.641656] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3740.646511] LustreError: Skipped 6 previous similar messages [ 3740.646577] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3740.655871] Lustre: Skipped 16 previous similar messages [ 3742.686479] Lustre: server umount lustre-OST0001 complete [ 3745.763517] LustreError: 214028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3745.781884] LustreError: 214028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 3751.089897] Lustre: server umount lustre-OST0000 complete [ 3759.638954] Lustre: server umount lustre-MDT0000 complete [ 3762.076739] LustreError: 212403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327565 with bad export cookie 3603192217980498029 [ 3762.077339] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3762.085248] LustreError: 212403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3762.264663] Lustre: server umount lustre-MDT0001 complete [ 3776.375810] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:26:18 (1780327578) [ 3781.242449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.483120] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3781.517543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3781.523730] Lustre: Skipped 5 previous similar messages [ 3781.663854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.670904] Lustre: Skipped 8 previous similar messages [ 3784.021417] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3789.153553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.219734] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3789.238531] Lustre: 218513:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3791.759838] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3795.044315] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3797.326969] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3801.756733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3801.904704] Lustre: Permanently deactivating lustre-OST0000 [ 3801.914148] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3803.528190] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3805.557814] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3809.222647] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3813.618606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3817.236970] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3820.521914] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3823.175026] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3826.673820] Lustre: Permanently reactivating lustre-OST0000 [ 3826.690077] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3826.697251] Lustre: Skipped 2 previous similar messages [ 3831.321290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.328674] Lustre: Skipped 13 previous similar messages [ 3831.331515] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3831.337259] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3831.345709] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3831.374919] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3831.389165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3843.608724] Lustre: server umount lustre-OST0001 complete [ 3851.714821] Lustre: server umount lustre-OST0000 complete [ 3860.305895] Lustre: server umount lustre-MDT0000 complete [ 3862.539648] LustreError: 220155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327666 with bad export cookie 3603192217980499674 [ 3862.541866] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3862.549039] LustreError: 220155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3862.736096] Lustre: server umount lustre-MDT0001 complete [ 3868.486272] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3870.438599] Key type lgssc unregistered [ 3870.702669] LNet: 222898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3870.706875] LNetError: 222898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3870.726775] LNet: Removed LNI 192.168.204.129@tcp [ 3871.223489] Key type .llcrypt unregistered [ 3871.225281] Key type ._llcrypt unregistered [ 3884.255077] Key type ._llcrypt registered [ 3884.257105] Key type .llcrypt registered [ 3884.320688] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3884.950933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3884.977758] alg: No test for adler32 (adler32-zlib) [ 3885.893507] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 3886.026461] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3887.639195] Key type lgssc registered [ 3888.345264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3894.042088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3894.061849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.346235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.510756] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3902.388845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3902.471432] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3902.494297] Lustre: 224313:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3902.502560] Lustre: Permanently deactivating lustre-MDT0001 [ 3902.511767] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3902.698429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3902.878738] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3902.885516] LustreError: 224357:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3902.893446] LustreError: 224357:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3902.900852] LustreError: 224357:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 3904.989778] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3907.649082] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3909.489794] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3913.254280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3913.458794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3916.407153] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3919.201947] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3919.590681] LustreError: 225937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3919.596088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3919.600888] LustreError: 225937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3922.884304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3923.003614] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3925.860703] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3928.045771] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3928.882902] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3929.071664] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3929.076131] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3931.997064] Lustre: Permanently reactivating lustre-MDT0001 [ 3932.011744] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3932.015378] Lustre: Skipped 2 previous similar messages [ 3934.371159] Lustre: lustre-MDT0001: Client 630f1cdc-bec1-435b-9056-7c0fd2a386c9 (at 192.168.204.29@tcp) reconnecting [ 3937.116994] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3938.056153] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3940.784208] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3940.903090] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3943.164814] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3943.272745] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3944.656430] Lustre: Permanently deactivating lustre-MDT0001 [ 3944.669893] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3944.673306] Lustre: Skipped 2 previous similar messages [ 3946.842451] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3959.264485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.270252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.279102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3959.783955] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.784772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3959.796237] Lustre: Skipped 2 previous similar messages [ 3959.804326] Lustre: Skipped 2 previous similar messages [ 3963.773820] Lustre: server umount lustre-MDT0000 complete [ 3964.900451] LustreError: 224326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.911276] LustreError: 224326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3965.446591] LustreError: 224307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327768 with bad export cookie 4544905897040210518 [ 3965.450285] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.453456] LustreError: 224307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3965.598219] Lustre: server umount lustre-MDT0001 complete [ 3973.380725] Lustre: server umount lustre-OST0000 complete [ 3978.702769] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:29:41 (1780327781) [ 3983.836661] Lustre: server umount lustre-OST0001 complete [ 3986.996476] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3992.735259] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4011.442874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.571095] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4011.576030] Lustre: Skipped 2 previous similar messages [ 4011.589383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4011.638965] Lustre: lustre-MDT0000: new disk, initializing [ 4011.695217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.704394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4013.763990] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4019.355198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.415705] Lustre: 231978:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4019.455594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4019.461495] Lustre: Skipped 1 previous similar message [ 4019.512385] Lustre: lustre-MDT0001: new disk, initializing [ 4019.586096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4019.593133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4021.762884] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4028.059316] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4033.659437] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4041.199489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.372190] Lustre: lustre-OST0000: new disk, initializing [ 4041.374558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4041.378400] Lustre: 233718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4041.419291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4041.421654] Lustre: Skipped 1 previous similar message [ 4048.681567] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4049.508139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4049.516414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4049.634058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4059.657313] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4062.352685] LustreError: 231985:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4071.767730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4072.039394] Lustre: lustre-OST0001: new disk, initializing [ 4072.044283] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4072.050212] Lustre: 234716:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4072.187092] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4072.439101] LustreError: 231985:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4072.461289] LustreError: 232762:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4078.267697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4078.275162] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4078.353886] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4080.070485] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4082.512080] LustreError: 232762:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4087.743871] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4090.045124] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4093.925797] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4093.938886] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.945838] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4093.958743] Lustre: Skipped 1 previous similar message [ 4093.977139] Lustre: Skipped 1 previous similar message [ 4099.042927] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4099.624574] Lustre: server umount lustre-OST0001 complete [ 4104.173720] LustreError: 233702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.203408] LustreError: 233702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4109.285104] LustreError: 234940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4109.308607] LustreError: 234940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4114.408539] LustreError: 233701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.450409] LustreError: 233701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4124.643859] LustreError: 235584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4124.650438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.702987] LustreError: 235584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4124.759897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4124.777407] Lustre: Skipped 2 previous similar messages [ 4128.013703] Lustre: server umount lustre-OST0000 complete [ 4134.881730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.898673] Lustre: Skipped 1 previous similar message [ 4134.914499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4134.928526] Lustre: Skipped 1 previous similar message [ 4140.606921] Lustre: server umount lustre-MDT0000 complete [ 4145.120623] LustreError: 231989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.141110] LustreError: 231989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4145.341720] LustreError: 232756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327948 with bad export cookie 4544905897040213332 [ 4145.367805] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4145.378772] LustreError: 232756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4151.937874] Lustre: server umount lustre-MDT0001 complete [ 4161.754769] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4165.041694] Key type lgssc unregistered [ 4165.608069] LNet: 236622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4165.634975] LNetError: 236622:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4165.668803] LNet: Removed LNI 192.168.204.129@tcp [ 4167.308401] Key type .llcrypt unregistered [ 4167.316545] Key type ._llcrypt unregistered [ 4212.633960] Key type ._llcrypt registered [ 4212.636231] Key type .llcrypt registered [ 4212.758132] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4214.041679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4214.083114] alg: No test for adler32 (adler32-zlib) [ 4215.197621] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 4215.465687] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4217.159407] Key type lgssc registered [ 4218.395705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4225.963969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4226.013935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.302579] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4227.325482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4227.469729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.554473] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4241.094330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.314134] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4241.357749] Lustre: 238814:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4241.685815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4246.255780] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4251.864479] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4255.588079] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4261.881440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4262.006516] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4262.201839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.174132] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4271.736866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4273.460979] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4323.301188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.308582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4323.309116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4323.321538] Lustre: Skipped 1 previous similar message [ 4323.332619] Lustre: Skipped 1 previous similar message [ 4328.423665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4328.433941] Lustre: Skipped 1 previous similar message [ 4329.585594] Lustre: server umount lustre-OST0000 complete [ 4334.038373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4346.772359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.062275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.072784] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4354.218184] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4354.233907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4354.234086] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4354.255209] Lustre: Skipped 1 previous similar message [ 4359.529672] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4400.611176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4400.615795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.625441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4400.647772] Lustre: Skipped 1 previous similar message [ 4405.729708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4405.733421] Lustre: Skipped 2 previous similar messages [ 4405.964466] Lustre: server umount lustre-OST0000 complete [ 4415.969242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4415.976995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4415.982218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4415.993485] Lustre: Skipped 1 previous similar message [ 4416.012911] Lustre: Skipped 1 previous similar message [ 4417.699884] Lustre: server umount lustre-MDT0000 complete [ 4423.957199] LustreError: 238806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328227 with bad export cookie 2158583413170256899 [ 4423.962836] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4423.993568] LustreError: 238806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4424.764906] Lustre: server umount lustre-MDT0001 complete [ 4440.316074] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4443.421928] Key type lgssc unregistered [ 4443.798906] LNet: 243769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4443.806686] LNetError: 243769:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4443.825488] LNet: Removed LNI 192.168.204.129@tcp [ 4445.079597] Key type .llcrypt unregistered [ 4445.086768] Key type ._llcrypt unregistered [ 4475.198662] Key type ._llcrypt registered [ 4475.202694] Key type .llcrypt registered [ 4475.378875] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4476.892824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4476.949058] alg: No test for adler32 (adler32-zlib) [ 4478.163729] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 4478.355325] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4480.031277] Key type lgssc registered [ 4481.703537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4493.578862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4493.592512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.618717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4502.045641] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4505.067869] LustreError: 245170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.112823] LustreError: 245170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4510.176063] LustreError: 245171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.061255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4513.449895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4518.034554] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4524.145709] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4528.162514] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4535.855772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4536.283881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4540.422805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4543.589374] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4549.462786] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4571.107891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4571.110413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4571.117218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4571.118361] Lustre: Skipped 1 previous similar message [ 4575.415437] Lustre: server umount lustre-OST0000 complete [ 4580.321080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4580.332324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.344191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4580.358038] Lustre: Skipped 1 previous similar message [ 4581.344345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4581.346171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.942681] Lustre: server umount lustre-MDT0000 complete [ 4586.465616] LustreError: 245167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.488388] LustreError: 245167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4587.876423] LustreError: 245152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328391 with bad export cookie 14644871084580534150 [ 4587.881091] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4587.885916] LustreError: 245152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4588.206986] Lustre: server umount lustre-MDT0001 complete [ 4597.424810] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4599.695803] Key type lgssc unregistered [ 4599.931205] LNet: 249121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4599.941611] LNetError: 249121:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4599.959974] LNet: Removed LNI 192.168.204.129@tcp [ 4600.658941] Key type .llcrypt unregistered [ 4600.663078] Key type ._llcrypt unregistered [ 4614.187306] Key type ._llcrypt registered [ 4614.189430] Key type .llcrypt registered [ 4614.252195] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4615.066442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4615.181975] alg: No test for adler32 (adler32-zlib) [ 4616.225582] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 4616.435699] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4618.079188] Key type lgssc registered [ 4618.928847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4624.923607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4624.952771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.361736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.955062] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4635.622588] LustreError: 250306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4635.644239] LustreError: 250306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4637.333665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.716620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4641.734839] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4646.949402] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4650.745047] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4657.572341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4658.155352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.226934] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4670.028500] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4671.488214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4686.822146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.822273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4686.842132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4686.866784] Lustre: Skipped 1 previous similar message [ 4688.363236] Lustre: server umount lustre-OST0000 complete [ 4693.988463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4693.995073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4694.009451] Lustre: Skipped 1 previous similar message [ 4694.034497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.059221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.068472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.747567] Lustre: server umount lustre-MDT0000 complete [ 4702.178626] LustreError: 250306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.201165] LustreError: 250306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4702.674164] LustreError: 250286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328506 with bad export cookie 1319026904290080870 [ 4702.679034] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4702.685602] LustreError: 250286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4703.113692] Lustre: server umount lustre-MDT0001 complete [ 4715.165656] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4718.024530] Key type lgssc unregistered [ 4718.313277] LNet: 253626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4718.332091] LNetError: 253626:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4718.353125] LNet: Removed LNI 192.168.204.129@tcp [ 4719.253877] Key type .llcrypt unregistered [ 4719.257251] Key type ._llcrypt unregistered [ 4744.554847] Key type ._llcrypt registered [ 4744.558600] Key type .llcrypt registered [ 4744.674774] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4746.060023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4746.142438] alg: No test for adler32 (adler32-zlib) [ 4747.246386] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 4747.565796] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4749.281197] Key type lgssc registered [ 4751.203274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4760.431581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4760.468493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.056023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.309221] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4772.323811] LustreError: 255028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.351223] LustreError: 255028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4775.467573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.851920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4780.522821] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4786.646596] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4790.194537] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4797.351626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4797.760185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4803.958000] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4806.960231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4809.588711] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4832.739718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.743376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4832.760560] Lustre: Skipped 1 previous similar message [ 4837.867733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4837.872724] Lustre: Skipped 2 previous similar messages [ 4838.182815] Lustre: server umount lustre-OST0000 complete [ 4842.979670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4843.015177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4847.589686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4847.598740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.621590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4853.217521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4853.224996] Lustre: Skipped 3 previous similar messages [ 4857.311169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4857.498145] Lustre: server umount lustre-MDT0000 complete [ 4858.340836] LustreError: 256321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.360116] LustreError: 256321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4861.060022] LustreError: 255744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328664 with bad export cookie 8385524137335140770 [ 4861.065741] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4861.081432] LustreError: 255744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4861.595576] Lustre: server umount lustre-MDT0001 complete [ 4873.969030] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4876.351831] Key type lgssc unregistered [ 4876.636967] LNet: 258974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4876.647989] LNetError: 258974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4876.668803] LNet: Removed LNI 192.168.204.129@tcp [ 4877.372642] Key type .llcrypt unregistered [ 4877.375586] Key type ._llcrypt unregistered [ 4893.725352] Key type ._llcrypt registered [ 4893.730108] Key type .llcrypt registered [ 4893.873959] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4894.989058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4895.029916] alg: No test for adler32 (adler32-zlib) [ 4896.293974] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 4896.539836] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4898.263183] Key type lgssc registered [ 4899.529803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4907.412737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4907.444362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.075943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.569610] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4919.269220] LustreError: 260195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4919.299108] LustreError: 260195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4922.293456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.620702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4926.394650] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4930.826590] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4933.588776] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4940.980164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4941.726751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.633598] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4954.131534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4956.067547] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4972.513579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4972.527658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4972.560132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4974.565971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.566456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4974.585665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4974.600658] Lustre: Skipped 1 previous similar message [ 4979.681332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4979.698089] Lustre: Skipped 1 previous similar message [ 4984.287865] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4984.469875] Lustre: server umount lustre-OST0000 complete [ 4989.410436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4989.417699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.434069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4994.277516] Lustre: server umount lustre-MDT0000 complete [ 4995.046804] LustreError: 260197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.070423] LustreError: 260197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4997.580089] LustreError: 260162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328801 with bad export cookie 3921003024465143551 [ 4997.589686] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4997.939954] Lustre: server umount lustre-MDT0001 complete [ 5007.054752] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5009.695453] Key type lgssc unregistered [ 5010.030232] LNet: 263538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5010.038573] LNetError: 263538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5010.069068] LNet: Removed LNI 192.168.204.129@tcp [ 5010.965580] Key type .llcrypt unregistered [ 5010.967615] Key type ._llcrypt unregistered [ 5021.114199] Key type ._llcrypt registered [ 5021.121448] Key type .llcrypt registered [ 5021.220937] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5036.312645] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 5048.512722] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5049.380480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5049.400343] alg: No test for adler32 (adler32-zlib) [ 5050.409415] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 5050.573800] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5052.207274] Key type lgssc registered [ 5053.305152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5083.001997] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5083.046097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5084.259686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5084.281939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5084.351906] Lustre: lustre-MDT0000: new disk, initializing [ 5084.466394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.495296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5087.858574] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5096.103368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5096.212794] Lustre: 267018:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5096.269485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5096.278471] Lustre: Skipped 1 previous similar message [ 5096.328866] Lustre: lustre-MDT0001: new disk, initializing [ 5096.427617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5096.472765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5096.484130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5099.406772] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5105.027511] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5107.913639] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5113.233329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5113.483047] Lustre: lustre-OST0000: new disk, initializing [ 5113.486794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5113.492566] Lustre: 268756:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5113.581636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5117.877196] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5119.505608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5119.510923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5119.580100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5123.102758] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5125.928143] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5126.081377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5128.532769] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5128.674852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5134.818121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5134.819376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.822559] LustreError: Skipped 1 previous similar message [ 5134.823918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5134.837577] Lustre: Skipped 1 previous similar message [ 5139.937647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5139.941742] Lustre: Skipped 2 previous similar messages [ 5144.031165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5144.124307] Lustre: server umount lustre-OST0000 complete [ 5147.616502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5147.623311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.639844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5150.178392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5150.179052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.629755] Lustre: server umount lustre-MDT0000 complete [ 5155.337553] LustreError: 268755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328958 with bad export cookie 16047176932603170323 [ 5155.340438] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5155.349954] LustreError: 268755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5155.584823] Lustre: server umount lustre-MDT0001 complete [ 5160.617854] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:49:23 (1780328963) [ 5164.428324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.701591] LustreError: 271019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.715646] LustreError: 271019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5164.740359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.950761] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5170.146483] LustreError: 271020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5171.720408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5171.906843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5174.075621] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5176.893368] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5178.918359] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5182.515841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5182.699758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.444512] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5188.182327] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5204.600240] Lustre: server umount lustre-OST0000 complete [ 5207.521310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5207.521949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5207.527649] Lustre: Skipped 1 previous similar message [ 5207.531644] Lustre: Skipped 2 previous similar messages [ 5216.764825] Lustre: server umount lustre-MDT0000 complete [ 5217.760602] LustreError: 272062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.767240] LustreError: 272062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5218.498755] LustreError: 271001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329021 with bad export cookie 16047176932603171205 [ 5218.503237] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5218.503425] LustreError: 271001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5228.075603] Lustre: server umount lustre-MDT0001 complete [ 5232.654134] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5234.010996] Key type lgssc unregistered [ 5234.165313] LNet: 274224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5234.169627] LNetError: 274224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5234.180698] LNet: Removed LNI 192.168.204.129@tcp [ 5234.565413] Key type .llcrypt unregistered [ 5234.567414] Key type ._llcrypt unregistered [ 5245.873495] Key type ._llcrypt registered [ 5245.874850] Key type .llcrypt registered [ 5245.922888] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 5253.711475] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5254.212524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5254.226986] alg: No test for adler32 (adler32-zlib) [ 5255.113648] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 5255.223847] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5256.831619] Key type lgssc registered [ 5257.377753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5277.980871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5277.994771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.112234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5279.127314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5279.169210] Lustre: lustre-MDT0000: new disk, initializing [ 5279.219083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5279.230155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5280.898753] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5285.885847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5285.941635] Lustre: 277988:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5285.976625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5285.980100] Lustre: Skipped 1 previous similar message [ 5286.028409] Lustre: lustre-MDT0001: new disk, initializing [ 5286.072563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5286.090929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5286.096191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5287.781170] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5291.471604] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5293.001804] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5296.083661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5296.219694] Lustre: lustre-OST2710: new disk, initializing [ 5296.222368] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5296.226732] Lustre: 279731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5296.285995] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5299.048324] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5299.393813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5299.402174] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5299.443565] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5303.112271] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5306.104104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5306.180981] Lustre: lustre-OST03e8: new disk, initializing [ 5306.183439] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5306.187158] Lustre: 280680:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5306.228552] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5306.487617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5306.492728] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5306.539491] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5308.718640] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5311.011783] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5314.750261] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5314.896639] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5316.790994] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5316.886851] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5318.609576] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5318.701813] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5320.541934] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5320.642297] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5322.211154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5322.212286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.214392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5322.225763] Lustre: Skipped 2 previous similar messages [ 5327.330292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5327.334401] Lustre: Skipped 6 previous similar messages [ 5328.241361] Lustre: server umount lustre-MDT0000 complete [ 5330.005142] LustreError: 277979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329133 with bad export cookie 15395813420003315866 [ 5330.008088] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5330.012581] LustreError: 277979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5330.166445] Lustre: server umount lustre-MDT0001 complete [ 5341.659227] Lustre: server umount lustre-OST2710 complete [ 5352.750940] Lustre: server umount lustre-OST03e8 complete [ 5360.534630] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 5364.446339] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5379.752952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.861863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5379.879383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5379.931966] Lustre: lustre-MDT0000: new disk, initializing [ 5379.980719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.990975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5381.626637] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5386.778548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5386.833629] Lustre: 285702:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5386.963380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5388.747265] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5392.345442] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5393.862773] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5396.973950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5397.106688] Lustre: lustre-OST0000: new disk, initializing [ 5397.108783] Lustre: Skipped 1 previous similar message [ 5397.111324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5397.114206] Lustre: Skipped 2 previous similar messages [ 5397.117191] Lustre: 287442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5397.158400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.161588] Lustre: Skipped 1 previous similar message [ 5399.495379] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5401.091997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5401.095791] Lustre: Skipped 1 previous similar message [ 5401.100338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5401.148755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5403.282161] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5405.088936] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5405.181403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5406.861479] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5406.953099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5411.298605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.298760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5411.302065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5411.303333] Lustre: Skipped 2 previous similar messages [ 5413.788287] Lustre: server umount lustre-OST0000 complete [ 5416.418153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5416.418931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5416.423750] Lustre: Skipped 1 previous similar message [ 5416.426914] Lustre: Skipped 1 previous similar message [ 5421.474903] Lustre: server umount lustre-MDT0000 complete [ 5422.936156] LustreError: 286478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329226 with bad export cookie 15395813420003317546 [ 5422.938541] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5422.941250] LustreError: 286478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5423.080533] Lustre: server umount lustre-MDT0001 complete [ 5426.266338] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:53:49 (1780329229) [ 5426.877468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5427.524402] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:53:50 (1780329230) [ 5428.627001] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5439.017437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5439.140809] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5439.152143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5439.233330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5440.674151] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5443.781656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.826275] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5443.838551] Lustre: 291130:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5445.389633] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5461.667816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5461.774783] Lustre: Denying initial registration attempt from nid 192.168.204.129@tcp, specified as failover [ 5461.779454] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5461.783890] LustreError: 292543:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 5461.787881] LustreError: 292543:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5461.791200] LustreError: 292543:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5461.825984] Lustre: server umount lustre-OST0000 complete [ 5461.828460] LustreError: 292543:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5464.544134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5464.544272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.544712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5464.544716] Lustre: Skipped 1 previous similar message [ 5464.548477] LustreError: Skipped 1 previous similar message [ 5464.554879] Lustre: Skipped 1 previous similar message [ 5470.429385] Lustre: server umount lustre-MDT0000 complete [ 5471.937705] LustreError: 291854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329275 with bad export cookie 15395813420003318428 [ 5471.942448] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5471.943778] LustreError: 291854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5472.088496] Lustre: server umount lustre-MDT0001 complete [ 5475.989408] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5477.005925] Key type lgssc unregistered [ 5477.133654] LNet: 293650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5477.136908] LNetError: 293650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5477.149431] LNet: Removed LNI 192.168.204.129@tcp [ 5477.459061] Key type .llcrypt unregistered [ 5477.460336] Key type ._llcrypt unregistered [ 5481.204922] Key type ._llcrypt registered [ 5481.206161] Key type .llcrypt registered [ 5481.245723] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5481.648324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5481.693228] alg: No test for adler32 (adler32-zlib) [ 5482.568590] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 5482.670411] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5484.271295] Key type lgssc registered [ 5484.736388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5511.843636] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5515.950483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5515.958828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5517.097788] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5517.109731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5517.195352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.845051] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5522.339890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.379504] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5522.390221] Lustre: 296354:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5522.494714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5523.963619] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5526.093501] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5527.375283] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5542.819400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5542.900221] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5543.017362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.197718] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5547.206796] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5548.055214] Lustre: server umount lustre-OST0000 complete [ 5550.585141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.586325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5550.591087] Lustre: Skipped 1 previous similar message [ 5555.678146] Lustre: server umount lustre-MDT0000 complete [ 5557.176260] LustreError: 296346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329360 with bad export cookie 3404583215720640755 [ 5557.180581] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5557.181473] LustreError: 296346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5557.314698] Lustre: server umount lustre-MDT0001 complete [ 5561.437273] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5562.694513] Key type lgssc unregistered [ 5562.845422] LNet: 299562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5562.848817] LNetError: 299562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5562.857316] LNet: Removed LNI 192.168.204.129@tcp [ 5563.155663] Key type .llcrypt unregistered [ 5563.157212] Key type ._llcrypt unregistered [ 5573.397343] Key type ._llcrypt registered [ 5573.399221] Key type .llcrypt registered [ 5573.443498] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5573.807471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5573.860395] alg: No test for adler32 (adler32-zlib) [ 5574.771324] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 5574.897905] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5576.513858] Key type lgssc registered [ 5577.019735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5580.197058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5580.213493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.457888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5583.067596] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5586.607234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5586.772414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5588.345521] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5590.516394] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5591.816781] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5596.195256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5596.320663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.239073] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5600.097029] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5610.978713] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.980260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5610.985648] Lustre: Skipped 1 previous similar message [ 5611.737454] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 5611.740396] Lustre: Skipped 2 previous similar messages [ 5615.199166] Lustre: server umount lustre-MDT0000 complete [ 5616.096471] LustreError: 300938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.103209] LustreError: 300938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5616.693594] LustreError: 300918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329420 with bad export cookie 2296387895822327321 [ 5616.697676] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5616.698874] LustreError: 300918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5616.845779] LustreError: 300934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.853854] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [ 5621.216422] LustreError: 300933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.216669] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.228479] Lustre: Skipped 1 previous similar message [ 5621.230985] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5622.907535] Lustre: server umount lustre-MDT0001 complete [ 5625.323076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5628.359807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.631589] LustreError: 300248:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000508ab468 x1866810657889280/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5641.759892] LustreError: 302541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.767826] LustreError: 302541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5641.789487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.814616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5643.258977] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5645.791984] LustreError: 302542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.383232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.523686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5647.978943] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5650.111259] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5650.664367] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5653.810145] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5655.192351] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5655.858988] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5661.159306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5661.160320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5661.164831] Lustre: Skipped 1 previous similar message [ 5661.171388] Lustre: Skipped 2 previous similar messages [ 5663.514921] Lustre: server umount lustre-OST0000 complete [ 5666.272638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5666.278507] Lustre: Skipped 1 previous similar message [ 5671.160349] Lustre: server umount lustre-MDT0000 complete [ 5671.392220] LustreError: 304190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5671.398790] LustreError: 304190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5672.602381] LustreError: 300919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329476 with bad export cookie 2296387895822334475 [ 5672.605280] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5672.607421] LustreError: 300919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5672.728174] Lustre: server umount lustre-MDT0001 complete [ 5677.037926] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5678.218673] Key type lgssc unregistered [ 5678.351348] LNet: 306610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5678.354263] LNetError: 306610:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5678.363448] LNet: Removed LNI 192.168.204.129@tcp [ 5678.685473] Key type .llcrypt unregistered [ 5678.686666] Key type ._llcrypt unregistered [ 5688.590019] Key type ._llcrypt registered [ 5688.591330] Key type .llcrypt registered [ 5688.630767] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5688.913941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5688.933590] alg: No test for adler32 (adler32-zlib) [ 5689.816221] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 5689.914686] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5691.503155] Key type lgssc registered [ 5691.944981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5694.926583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5694.934718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5696.123902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5697.603412] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5700.855633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.000085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5702.518281] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5704.592228] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5706.078276] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5711.329202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.329732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5711.329885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.335302] Lustre: Skipped 1 previous similar message [ 5713.374771] Lustre: server umount lustre-MDT0000 complete [ 5714.834449] LustreError: 308702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329518 with bad export cookie 13052269424549873800 [ 5714.837297] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5714.838817] LustreError: 308702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5714.974731] Lustre: server umount lustre-MDT0001 complete [ 5720.814602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5720.956946] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5720.970695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5721.064492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.707959] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5726.546036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.623154] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5726.640517] Lustre: 310343:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5726.780810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5728.518817] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5730.612897] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5731.985089] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5737.123919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5737.214749] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5737.219249] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5737.222968] LustreError: 312002:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5737.226221] LustreError: 312002:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5737.228813] LustreError: 312002:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5737.260222] Lustre: server umount lustre-OST0000 complete [ 5737.262236] LustreError: 312002:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5742.372453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5742.456820] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5742.533717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5744.661374] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5746.702047] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5749.800357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5749.996278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5750.039684] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5750.079289] Lustre: lustre-OST0001: new disk, initializing [ 5750.081734] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5750.085398] Lustre: 313427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5752.470305] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5754.615257] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5755.718331] Lustre: server umount lustre-OST0001 complete [ 5757.935575] LustreError: 312469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.937534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.941627] LustreError: 312469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5757.950559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5757.954021] Lustre: Skipped 2 previous similar messages [ 5763.043464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.044312] LustreError: 312467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.046431] Lustre: Skipped 1 previous similar message [ 5763.054122] LustreError: 312467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5763.291438] Lustre: server umount lustre-OST0000 complete [ 5767.648126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5767.651924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.657435] Lustre: Skipped 1 previous similar message [ 5767.659896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5771.110706] Lustre: server umount lustre-MDT0000 complete [ 5772.743625] LustreError: 310335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329576 with bad export cookie 13052269424549874339 [ 5772.746453] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.748280] LustreError: 310335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5772.895902] Lustre: server umount lustre-MDT0001 complete [ 5777.017441] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5778.280918] Key type lgssc unregistered [ 5778.425659] LNet: 315249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5778.428456] LNetError: 315249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5778.440398] LNet: Removed LNI 192.168.204.129@tcp [ 5778.793774] Key type .llcrypt unregistered [ 5778.794944] Key type ._llcrypt unregistered [ 5811.584678] Key type ._llcrypt registered [ 5811.585775] Key type .llcrypt registered [ 5811.621776] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 5817.703487] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5818.053485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5818.096074] alg: No test for adler32 (adler32-zlib) [ 5818.954255] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 5819.037395] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5820.623204] Key type lgssc registered [ 5820.996771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5832.625471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5832.630655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.731653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5833.746532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5833.785242] Lustre: lustre-MDT0000: new disk, initializing [ 5833.810088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.816958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5835.208395] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5839.602491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.647059] Lustre: 320121:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5839.674371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5839.677069] Lustre: Skipped 1 previous similar message [ 5839.715605] Lustre: lustre-MDT0001: new disk, initializing [ 5839.755586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.770296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5839.775243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5841.180569] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5844.482692] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5845.787139] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5848.210321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5848.308678] Lustre: lustre-OST0000: new disk, initializing [ 5848.311838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5848.316071] Lustre: 321857:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5848.351567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5849.144408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5849.148764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5849.187898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5850.291614] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5853.323455] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5854.591462] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5854.668144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5856.040459] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5856.122025] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5859.299639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5859.299995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.300838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.303277] LustreError: Skipped 1 previous similar message [ 5859.311766] Lustre: Skipped 1 previous similar message [ 5864.419948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.422454] Lustre: Skipped 2 previous similar messages [ 5869.537629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5869.541339] Lustre: Skipped 1 previous similar message [ 5871.071151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5871.202545] Lustre: server umount lustre-OST0000 complete [ 5874.658786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.661835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5874.667115] Lustre: Skipped 1 previous similar message [ 5879.780534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5879.783173] Lustre: Skipped 2 previous similar messages [ 5880.675720] Lustre: server umount lustre-MDT0000 complete [ 5883.698983] LustreError: 320112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329687 with bad export cookie 11773704250024770828 [ 5883.703021] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5883.726027] LustreError: 320112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5884.264319] Lustre: server umount lustre-MDT0001 complete [ 5891.112709] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:01:33 (1780329693) [ 5898.558732] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:01:39 (1780329699) [ 5918.787423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.046950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5925.397605] LustreError: 324483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.435949] LustreError: 324483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5930.490522] LustreError: 324484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.564068] LustreError: 324483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.533235] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5936.614775] LustreError: 324484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5941.734179] LustreError: 324483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.053762] LustreError: 324484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.138276] LustreError: 324484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5954.459458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5954.765110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5962.474463] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5970.349243] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5976.741829] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5988.156100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.441361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.474153] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6007.286689] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6016.122049] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6017.961922] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6019.598952] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6021.564309] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6023.424911] Lustre: Failing over lustre-MDT0000 [ 6023.708746] Lustre: server umount lustre-MDT0000 complete [ 6024.945707] LustreError: 326370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.702689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6025.722067] Lustre: Skipped 2 previous similar messages [ 6042.080276] Lustre: 318348:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780329829/real 1780329829] req@00000000444cf771 x1866810913783040/t0(0) o400->MGC192.168.204.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780329845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6042.123613] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6043.888118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6052.321971] LustreError: 318344:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000007513673 x1866810913789184/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6052.675796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6052.737829] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6056.674036] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6057.973557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6058.006460] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6058.011871] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6058.096319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6066.391581] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6068.320673] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6072.614494] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6074.846440] Lustre: Failing over lustre-MDT0000 [ 6075.109101] Lustre: server umount lustre-MDT0000 complete [ 6077.180507] LustreError: 326370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6077.219776] LustreError: 326370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 6078.433751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.434614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6094.247545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.434542] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6094.890529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6094.972582] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6098.641829] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6099.625886] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6099.963789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6099.973036] Lustre: Skipped 2 previous similar messages [ 6100.024540] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6100.085398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6107.526032] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6109.031232] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6120.417057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6120.425502] Lustre: Skipped 3 previous similar messages [ 6120.429064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6122.054975] Lustre: server umount lustre-MDT0000 complete [ 6126.764033] LustreError: 324463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329930 with bad export cookie 11773704250024773467 [ 6126.769280] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6126.782056] LustreError: 324463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6127.297450] Lustre: server umount lustre-MDT0001 complete [ 6142.065692] Lustre: server umount lustre-OST0000 complete [ 6158.478961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.886743] LustreError: 330261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.901476] LustreError: 330261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 6158.942791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.126685] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6171.174367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.589986] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6181.275323] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6185.158532] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6191.470368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.004746] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6204.280254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6204.362857] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6211.554855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6211.565510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.581651] Lustre: Skipped 1 previous similar message [ 6211.590829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6211.595126] Lustre: Skipped 2 previous similar messages [ 6214.633524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6225.380646] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6225.797395] Lustre: server umount lustre-OST0000 complete [ 6233.055954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6233.060115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.081136] Lustre: Skipped 1 previous similar message [ 6245.345050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6245.356978] Lustre: Skipped 12 previous similar messages [ 6246.367290] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6246.751742] Lustre: server umount lustre-MDT0000 complete [ 6251.253179] LustreError: 330241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330054 with bad export cookie 11773704250024774216 [ 6251.279956] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6251.282716] LustreError: 330241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6251.796684] Lustre: server umount lustre-MDT0001 complete [ 6265.959542] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6269.841638] Key type lgssc unregistered [ 6270.246172] LNet: 333440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6270.264656] LNetError: 333440:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6271.350195] LNet: Removed LNI 192.168.204.129@tcp [ 6272.449247] Key type .llcrypt unregistered [ 6272.451876] Key type ._llcrypt unregistered [ 6312.424280] Key type ._llcrypt registered [ 6312.425695] Key type .llcrypt registered [ 6312.615681] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6314.393980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6314.542817] alg: No test for adler32 (adler32-zlib) [ 6315.758290] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 6316.152824] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6317.943283] Key type lgssc registered [ 6319.877986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6328.649092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6328.688129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.116840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.174143] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6339.557521] LustreError: 334940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6339.585478] LustreError: 334940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6343.241568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.591594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6347.706468] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6352.701837] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6355.752551] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6360.994612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6361.268735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.513043] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6369.521407] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6374.574889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6384.613841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6384.623676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6384.643609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6385.130465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.140399] Lustre: Skipped 1 previous similar message [ 6385.152240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6385.158515] Lustre: Skipped 2 previous similar messages [ 6385.429281] Lustre: server umount lustre-MDT0000 complete [ 6388.635895] LustreError: 334922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330192 with bad export cookie 8301895736450578391 [ 6388.646382] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6388.652877] LustreError: 334922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6388.899440] Lustre: server umount lustre-MDT0001 complete [ 6402.310294] Lustre: server umount lustre-OST0000 complete [ 6412.001442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.410321] LustreError: 338332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.465647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.315709] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6417.893786] LustreError: 338333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.871554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6421.987294] LustreError: 338332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6422.140794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6424.757364] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6428.319313] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6430.916771] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6435.484875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6435.757342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6436.783845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6439.608267] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6443.188555] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6452.704318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6452.709343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.725586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6457.316215] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.318992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6457.334744] Lustre: Skipped 1 previous similar message [ 6457.341048] Lustre: Skipped 1 previous similar message [ 6458.704868] Lustre: server umount lustre-MDT0000 complete [ 6460.797644] LustreError: 338314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330264 with bad export cookie 8301895736450579504 [ 6460.799035] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6460.804243] LustreError: 338314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6460.982922] Lustre: server umount lustre-MDT0001 complete [ 6472.721989] Lustre: server umount lustre-OST0000 complete [ 6481.837239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6482.124898] LustreError: 341900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.171383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6484.555865] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6489.455907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6492.035076] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6494.927442] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6497.056566] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6501.103266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6504.741634] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6507.436403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6507.771288] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6512.608595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6512.620348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6512.628334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6516.259480] Lustre: server umount lustre-OST0000 complete [ 6524.896542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6524.900823] Lustre: Skipped 5 previous similar messages [ 6533.087278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6533.194413] Lustre: server umount lustre-MDT0000 complete [ 6535.137297] LustreError: 341896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.152922] LustreError: 341896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6535.705603] LustreError: 341882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330339 with bad export cookie 8301895736450580533 [ 6535.709603] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6535.713941] LustreError: 341882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6535.921161] Lustre: server umount lustre-MDT0001 complete [ 6541.849455] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6543.480540] Key type lgssc unregistered [ 6543.667549] LNet: 345072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6543.671412] LNetError: 345072:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6543.683268] LNet: Removed LNI 192.168.204.129@tcp [ 6544.089129] Key type .llcrypt unregistered [ 6544.091075] Key type ._llcrypt unregistered [ 6559.105476] Key type ._llcrypt registered [ 6559.108709] Key type .llcrypt registered [ 6559.192213] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6559.854326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6559.876915] alg: No test for adler32 (adler32-zlib) [ 6560.792877] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 6560.954410] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6562.591472] Key type lgssc registered [ 6563.260189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6567.945837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6567.957260] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.963839] LustreError: 346471:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6567.978084] LustreError: 346471:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6567.981725] LustreError: 346471:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6567.984619] LustreError: 346471:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6567.988735] LustreError: 346471:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6572.409967] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6572.416479] LustreError: 346868:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6572.429251] LustreError: 346868:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6572.433747] LustreError: 346868:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6572.438605] LustreError: 346868:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6572.443979] LustreError: 346868:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6583.808248] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6585.310604] Key type lgssc unregistered [ 6585.476544] LNet: 347854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6585.482551] LNetError: 347854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6585.495584] LNet: Removed LNI 192.168.204.129@tcp [ 6585.915597] Key type .llcrypt unregistered [ 6585.917444] Key type ._llcrypt unregistered [ 6595.267098] Key type ._llcrypt registered [ 6595.268866] Key type .llcrypt registered [ 6595.322594] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 6603.984543] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6604.482124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6604.521079] alg: No test for adler32 (adler32-zlib) [ 6605.459862] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 6605.606499] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6607.223181] Key type lgssc registered [ 6607.937782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6627.816293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6627.829853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.942658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6628.958973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6629.004293] Lustre: lustre-MDT0000: new disk, initializing [ 6629.051286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.062218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6630.851090] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6636.373529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6636.439429] Lustre: 350949:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6636.473702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6636.477267] Lustre: Skipped 1 previous similar message [ 6636.523406] Lustre: lustre-MDT0001: new disk, initializing [ 6636.586218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6636.612616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6636.618346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6638.374468] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6641.976467] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6643.439394] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6645.875926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6645.976846] Lustre: lustre-OST0000: new disk, initializing [ 6645.979484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6645.983366] Lustre: 352688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6646.018132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6648.203811] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6651.682669] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6653.377425] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6655.491085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6655.495537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6655.532798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6656.543642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6658.106523] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6658.195261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6660.578355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6660.578750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6660.579467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6660.582669] LustreError: Skipped 1 previous similar message [ 6660.588852] Lustre: Skipped 1 previous similar message [ 6665.049659] Lustre: server umount lustre-OST0000 complete [ 6667.231820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6667.234702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.241679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.244119] Lustre: Skipped 1 previous similar message [ 6670.816702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6670.817047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.619321] Lustre: server umount lustre-MDT0000 complete [ 6673.983328] LustreError: 350941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330477 with bad export cookie 2748652301936124838 [ 6673.985767] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6673.989115] LustreError: 350941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6674.112628] Lustre: server umount lustre-MDT0001 complete [ 6677.399723] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6678.046407] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:14:40 (1780330480) [ 6680.421459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.596390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6682.018019] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6685.318834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6685.460120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6686.935802] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6688.960520] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6690.383578] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6693.013579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6695.369547] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6697.644800] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6700.199356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6700.260888] Lustre: lustre-OST0001: new disk, initializing [ 6700.262984] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6700.265984] Lustre: 357516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6700.297794] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.300720] Lustre: Skipped 1 previous similar message [ 6702.283070] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6704.063313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6704.067117] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6704.108521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6705.403846] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6709.216153] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6709.216238] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6709.218440] LustreError: Skipped 1 previous similar message [ 6709.222324] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6709.225830] Lustre: Skipped 2 previous similar messages [ 6713.410451] Lustre: server umount lustre-OST0001 complete [ 6716.388197] LustreError: 356616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6716.394530] LustreError: 356616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6721.511509] LustreError: 356614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.517475] LustreError: 356614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6726.112661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6726.115911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6726.121107] Lustre: Skipped 1 previous similar message [ 6726.123646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.125804] Lustre: Skipped 3 previous similar messages [ 6726.626730] LustreError: 356616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6726.632799] LustreError: 356616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6731.727997] Lustre: server umount lustre-OST0000 complete [ 6736.865679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.865841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6736.866813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6736.866821] Lustre: Skipped 1 previous similar message [ 6736.869815] Lustre: Skipped 2 previous similar messages [ 6736.873220] LustreError: Skipped 1 previous similar message [ 6739.289363] Lustre: server umount lustre-MDT0000 complete [ 6740.669365] LustreError: 357514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330544 with bad export cookie 2748652301936125706 [ 6740.672677] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6740.674862] LustreError: 357514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6746.917880] Lustre: server umount lustre-MDT0001 complete [ 6753.045960] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6755.322140] Key type lgssc unregistered [ 6755.590886] LNet: 359369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6755.600789] LNetError: 359369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6755.615551] LNet: Removed LNI 192.168.204.129@tcp [ 6756.229216] Key type .llcrypt unregistered [ 6756.231097] Key type ._llcrypt unregistered [ 6797.848510] Key type ._llcrypt registered [ 6797.850616] Key type .llcrypt registered [ 6797.997118] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6799.747105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6799.797650] alg: No test for adler32 (adler32-zlib) [ 6800.753293] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 6800.894144] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6802.527199] Key type lgssc registered [ 6803.405663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6817.377610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6817.441668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6819.080563] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6819.161602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6819.509956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6826.791077] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6842.122561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.359812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6842.450365] Lustre: 362135:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6842.991763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6851.459401] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6862.646932] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6868.674265] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6873.582885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.584380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6873.616866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6873.632838] Lustre: Skipped 1 previous similar message [ 6878.855373] Lustre: server umount lustre-MDT0000 complete [ 6883.830649] LustreError: 362142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6883.869850] LustreError: 362142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6885.171687] LustreError: 362126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330688 with bad export cookie 13351129336696398165 [ 6885.174473] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6885.183449] LustreError: 362126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6885.610633] Lustre: server umount lustre-MDT0001 complete [ 6899.511459] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:18:21 (1780330701) [ 6907.488950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.884642] LustreError: 364426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.933598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6912.124902] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6913.001332] LustreError: 364426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6918.114194] LustreError: 364427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.028564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.531127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6927.554592] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6933.401266] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6937.670715] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6944.685340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.933105] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6945.204236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.165203] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6956.727228] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6963.654984] Lustre: Permanently deactivating lustre-OST0000 [ 6963.674212] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6970.922083] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6973.494484] LustreError: 367074:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6973.501919] LustreError: 367074:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6973.506606] LustreError: 367074:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6976.962187] LustreError: 367122:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6976.967226] LustreError: 367122:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6985.449313] Lustre: server umount lustre-OST0000 complete [ 6988.063036] LustreError: 367372:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6988.092160] LustreError: 367372:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6990.819131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.820827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6990.830188] Lustre: Skipped 1 previous similar message [ 6990.845132] Lustre: Skipped 2 previous similar messages [ 6995.937068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6995.945535] Lustre: Skipped 1 previous similar message [ 7000.249075] Lustre: server umount lustre-MDT0000 complete [ 7001.059953] LustreError: 364422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7001.078886] LustreError: 364422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7004.633566] LustreError: 364408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330808 with bad export cookie 13351129336696398746 [ 7004.644864] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7004.647979] LustreError: 364408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7015.099452] Lustre: server umount lustre-MDT0001 complete [ 7024.386805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.195652] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7032.339614] LustreError: 368514:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7038.265891] Lustre: 368853:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.129@tcp [ 7040.376811] Lustre: server umount MGS complete [ 7052.436144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.858954] LustreError: 369578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7052.880788] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7052.886763] Lustre: Skipped 1 previous similar message [ 7052.977056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7057.188265] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7066.385528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7066.644979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7071.869881] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7078.656385] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7083.347416] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7085.383920] Lustre: Permanently reactivating lustre-OST0000 [ 7085.413353] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7085.423338] Lustre: Skipped 2 previous similar messages [ 7116.255662] LustreError: 371185:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7126.083929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.665370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7126.673700] Lustre: Skipped 1 previous similar message [ 7134.071150] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7140.881889] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7151.585572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7151.604118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.632484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7151.643487] Lustre: Skipped 1 previous similar message [ 7156.706337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7156.716609] Lustre: Skipped 1 previous similar message [ 7161.311537] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7161.594177] Lustre: server umount lustre-OST0000 complete [ 7166.944791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.948568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7166.970797] Lustre: Skipped 2 previous similar messages [ 7166.993185] Lustre: Skipped 1 previous similar message [ 7180.255381] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7180.465756] Lustre: server umount lustre-MDT0000 complete [ 7182.304919] LustreError: 369573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7182.327615] LustreError: 369573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7184.561137] LustreError: 369558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330988 with bad export cookie 13351129336696399796 [ 7184.563170] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7184.578173] LustreError: 369558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7185.099291] Lustre: server umount lustre-MDT0001 complete [ 7198.099696] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7201.156415] Key type lgssc unregistered [ 7201.593759] LNet: 373688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7201.604106] LNetError: 373688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7201.616450] LNet: Removed LNI 192.168.204.129@tcp [ 7202.499066] Key type .llcrypt unregistered [ 7202.502114] Key type ._llcrypt unregistered [ 7223.629955] Key type ._llcrypt registered [ 7223.642916] Key type .llcrypt registered [ 7223.801450] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 7241.832647] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7243.421315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7243.454101] alg: No test for adler32 (adler32-zlib) [ 7244.565266] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 7244.836153] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7246.562247] Key type lgssc registered [ 7247.985888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7286.821555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7286.927867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.222824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7288.284462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7288.447197] Lustre: lustre-MDT0000: new disk, initializing [ 7288.559145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.596852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7293.411143] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7306.642855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.858627] Lustre: 376814:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7306.961841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7306.968265] Lustre: Skipped 1 previous similar message [ 7307.042511] Lustre: lustre-MDT0001: new disk, initializing [ 7307.132107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7307.192703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7307.207521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7312.124301] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7320.963455] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7325.748231] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7332.992958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.247790] Lustre: lustre-OST0000: new disk, initializing [ 7333.251590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7333.257736] Lustre: 378554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7333.356174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.666058] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7340.679905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7340.684038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7340.881237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7348.017379] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7354.136912] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7354.337548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7360.230973] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7360.532448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7366.625890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7366.629172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.646016] LustreError: Skipped 1 previous similar message [ 7366.647586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7366.705027] Lustre: Skipped 1 previous similar message [ 7371.760988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7371.774986] Lustre: Skipped 1 previous similar message [ 7376.893548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7376.901535] Lustre: Skipped 1 previous similar message [ 7377.375213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7377.702272] Lustre: server umount lustre-OST0000 complete [ 7384.032433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7384.047552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7384.071597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7384.080539] Lustre: Skipped 1 previous similar message [ 7387.104798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7388.593575] Lustre: server umount lustre-MDT0000 complete [ 7393.015326] LustreError: 376805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331196 with bad export cookie 8195778771953989521 [ 7393.017383] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7393.027772] LustreError: 376805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7393.481831] Lustre: server umount lustre-MDT0001 complete [ 7404.165365] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7405.890415] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:26:47 (1780331207) [ 7413.041467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.565295] LustreError: 380869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.592963] LustreError: 380869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7413.664476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.498645] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7418.860496] LustreError: 380870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7423.977168] LustreError: 380869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7427.276667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7427.757901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7433.104345] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7439.848532] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7444.351218] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7451.441902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7451.907172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7458.773347] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7465.105315] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7467.331306] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7470.260750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7478.764332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7478.771374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7478.796046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7478.805143] Lustre: Skipped 2 previous similar messages [ 7482.040564] Lustre: server umount lustre-OST0000 complete [ 7488.992958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7489.002721] LustreError: Skipped 1 previous similar message [ 7489.010362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.033990] Lustre: Skipped 1 previous similar message [ 7489.052519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7489.055485] Lustre: Skipped 2 previous similar messages [ 7492.506965] Lustre: server umount lustre-MDT0000 complete [ 7495.139897] LustreError: 381990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7495.160937] LustreError: 381990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7497.211905] LustreError: 380849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331300 with bad export cookie 8195778771953990403 [ 7497.213818] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7497.223847] LustreError: 380849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7497.729717] Lustre: server umount lustre-MDT0001 complete [ 7510.290512] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7513.743589] Key type lgssc unregistered [ 7514.119263] LNet: 384237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7514.137680] LNetError: 384237:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7514.173281] LNet: Removed LNI 192.168.204.129@tcp [ 7515.218769] Key type .llcrypt unregistered [ 7515.223460] Key type ._llcrypt unregistered [ 7541.849113] Key type ._llcrypt registered [ 7541.850727] Key type .llcrypt registered [ 7542.068820] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7542.551306] Key type .llcrypt unregistered [ 7542.554259] Key type ._llcrypt unregistered [ 7564.901692] Key type ._llcrypt registered [ 7564.903433] Key type .llcrypt registered [ 7565.006444] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7566.939072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7566.968365] alg: No test for adler32 (adler32-zlib) [ 7568.099259] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 7568.545631] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7570.295816] Key type lgssc registered [ 7571.983100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7580.473456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7580.504467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7582.465401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.711334] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7591.904986] LustreError: 386420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.935625] LustreError: 386420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7597.032403] LustreError: 386419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7597.693790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7598.129657] LustreError: 386420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7598.170536] LustreError: 386420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7598.185853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7603.173659] LustreError: 386419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.194208] LustreError: 386419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7604.833224] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7607.271126] LustreError: 386420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7607.299657] LustreError: 386420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7607.316396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7611.968046] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7617.506533] LustreError: 386416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.532910] LustreError: 386416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7619.879923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.259140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7626.242568] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7637.989836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7638.003244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.035046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7640.553252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7640.574448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.597314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7643.878301] Lustre: server umount lustre-OST0000 complete [ 7650.457045] Lustre: server umount lustre-MDT0000 complete [ 7650.792851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7650.821655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.843960] LustreError: 386420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7654.696590] LustreError: 386400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331458 with bad export cookie 10912819952384371368 [ 7654.712389] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7654.722783] LustreError: 386400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7655.087254] Lustre: server umount lustre-MDT0001 complete [ 7667.568993] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7671.825674] Key type lgssc unregistered [ 7672.205790] LNet: 389404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7672.220832] LNetError: 389404:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7672.236041] LNet: Removed LNI 192.168.204.129@tcp [ 7673.245583] Key type .llcrypt unregistered [ 7673.247583] Key type ._llcrypt unregistered [ 7698.400438] Key type ._llcrypt registered [ 7698.403526] Key type .llcrypt registered [ 7698.543203] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7699.752578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7699.789770] alg: No test for adler32 (adler32-zlib) [ 7700.993983] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 7701.285670] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7703.007219] Key type lgssc registered [ 7704.360344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7712.067151] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7712.088658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7727.647928] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7732.767261] LustreError: 390778:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 7759.524161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.827559] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7772.101886] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7779.843232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7780.470196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7780.552422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7785.562091] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7785.953550] LustreError: 390803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.970930] LustreError: 390803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7791.074057] LustreError: 390804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.147739] LustreError: 391704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.171399] LustreError: 391704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7794.559483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7795.011394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7799.594339] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7806.225502] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7809.672433] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7819.267728] Lustre: server umount lustre-OST0000 complete [ 7824.870264] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7824.879396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7824.905506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.892372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.898342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.907887] Lustre: Skipped 1 previous similar message [ 7831.011176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7831.029104] Lustre: Skipped 1 previous similar message [ 7833.151103] Lustre: server umount lustre-MDT0000 complete [ 7836.130122] LustreError: 391707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7836.152307] LustreError: 391707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7837.057858] LustreError: 390785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331640 with bad export cookie 4057266644820753690 [ 7837.069520] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7847.625189] Lustre: server umount lustre-MDT0001 complete [ 7857.728261] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7860.408913] Key type lgssc unregistered [ 7860.678452] LNet: 393992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7860.686275] LNetError: 393992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7860.710263] LNet: Removed LNI 192.168.204.129@tcp [ 7861.504515] Key type .llcrypt unregistered [ 7861.508438] Key type ._llcrypt unregistered [ 7888.430667] Key type ._llcrypt registered [ 7888.433572] Key type .llcrypt registered [ 7888.596779] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7889.629579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7889.852086] alg: No test for adler32 (adler32-zlib) [ 7891.066838] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 7891.443527] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7893.167191] Key type lgssc registered [ 7894.528557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7902.622823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7902.653218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7904.293611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7909.377645] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7914.468255] LustreError: 395397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7914.495801] LustreError: 395397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7918.229840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.651572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7922.842403] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7928.357816] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7931.733300] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7938.040033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7938.433667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7944.319407] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7948.664076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7949.005705] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7953.888711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.901558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7953.903932] Lustre: Skipped 2 previous similar messages [ 7953.913049] Lustre: Skipped 2 previous similar messages [ 7955.707431] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 7958.736345] Lustre: server umount lustre-MDT0000 complete [ 7959.009902] LustreError: 396107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7959.025757] LustreError: 396107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7960.792076] LustreError: 395393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7961.801510] LustreError: 397013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.29@tcp arrived at 1780331765 with bad export cookie 3948395612196528810 [ 7961.814294] LustreError: 397013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7963.889183] LustreError: 395378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331767 with bad export cookie 3948395612196527914 [ 7963.897542] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7963.912958] LustreError: 395378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7964.128942] LustreError: 395397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7964.131861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.162374] LustreError: 395397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7964.191163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7968.161089] LustreError: 395394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.166128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7968.173386] LustreError: 395394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7970.124526] Lustre: server umount lustre-OST0000 complete [ 7977.904896] Lustre: server umount lustre-MDT0001 complete [ 7988.564956] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7991.287569] Key type lgssc unregistered [ 7991.590106] LNet: 398724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7991.606963] LNetError: 398724:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7991.635083] LNet: Removed LNI 192.168.204.129@tcp [ 7992.527343] Key type .llcrypt unregistered [ 7992.531988] Key type ._llcrypt unregistered [ 8016.432122] Key type ._llcrypt registered [ 8016.435330] Key type .llcrypt registered [ 8016.520628] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8017.903674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8017.932137] alg: No test for adler32 (adler32-zlib) [ 8018.979922] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 8019.196517] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 8020.882715] Key type lgssc registered [ 8022.206186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8029.389447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8029.425979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8030.846104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.433489] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8041.441731] LustreError: 400125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8041.467842] LustreError: 400125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8042.830807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8043.303922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8046.765265] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8051.520074] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8055.481787] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8062.340700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8062.753105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.615564] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8074.253719] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8076.094236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8084.449283] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8084.459479] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.476928] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8085.235788] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [ 8086.499895] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8086.509503] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.767913] Lustre: server umount lustre-MDT0001 complete [ 8090.331343] LustreError: 400120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8091.618332] LustreError: 400125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8091.644181] LustreError: 400125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8096.746689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8096.748488] LustreError: 400835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8096.765736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8096.766717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8096.766723] Lustre: Skipped 1 previous similar message [ 8101.859316] LustreError: 400125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8101.867220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8102.022589] Lustre: server umount lustre-OST0000 complete [ 8106.759636] Lustre: server umount lustre-MDT0000 complete [ 8121.735690] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8124.396940] Key type lgssc unregistered [ 8124.724987] LNet: 403454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8124.732118] LNetError: 403454:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8124.755116] LNet: Removed LNI 192.168.204.129@tcp [ 8125.597150] Key type .llcrypt unregistered [ 8125.601618] Key type ._llcrypt unregistered [ 8150.907379] Key type ._llcrypt registered [ 8150.909269] Key type .llcrypt registered [ 8151.076872] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 8165.707418] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8166.572921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8166.641897] alg: No test for adler32 (adler32-zlib) [ 8167.676960] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 8167.919219] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 8169.623433] Key type lgssc registered [ 8170.836336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8205.993245] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8219.685698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8219.707260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.026864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8221.102859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8221.224960] Lustre: lustre-MDT0000: new disk, initializing [ 8221.314787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8221.333150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8225.458567] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8240.823591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.108409] Lustre: lustre-OST0000: new disk, initializing [ 8241.116329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8241.125638] Lustre: Skipped 1 previous similar message [ 8241.138313] Lustre: 408422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8241.219085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.184056] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8248.425588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8248.439317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8248.499535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8255.327236] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8269.386752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.516338] Lustre: 407421:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8269.575583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8269.673375] Lustre: lustre-MDT0001: new disk, initializing [ 8269.790639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8269.834447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8269.848491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8274.079765] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8283.379879] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8283.644097] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8288.600900] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8288.829553] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8297.191252] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [ 8297.952573] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8297.959549] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.967415] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8297.995509] Lustre: Skipped 1 previous similar message [ 8300.096787] Lustre: server umount lustre-MDT0001 complete [ 8308.199690] LustreError: 407428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.229378] LustreError: 407428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8313.324904] LustreError: 408430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8318.434474] LustreError: 407433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8318.434626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.451264] LustreError: 407433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8318.474569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8318.481734] Lustre: Skipped 1 previous similar message [ 8323.554549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.558199] LustreError: 409635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8328.680522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8328.682672] LustreError: 407433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8331.231168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8331.420852] Lustre: server umount lustre-OST0000 complete [ 8341.199914] Lustre: server umount lustre-MDT0000 complete [ 8352.942455] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8355.359968] Key type lgssc unregistered [ 8355.564933] LNet: 411986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8355.574135] LNetError: 411986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8355.587306] LNet: Removed LNI 192.168.204.129@tcp [ 8356.288848] Key type .llcrypt unregistered [ 8356.291329] Key type ._llcrypt unregistered [ 8392.372801] Key type ._llcrypt registered [ 8392.381318] Key type .llcrypt registered [ 8392.517544] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8392.883431] Key type .llcrypt unregistered [ 8392.895751] Key type ._llcrypt unregistered [ 8407.757078] Key type ._llcrypt registered [ 8407.759291] Key type .llcrypt registered [ 8407.852786] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8408.517639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8408.531655] alg: No test for adler32 (adler32-zlib) [ 8409.530345] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 8409.763528] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 8411.423162] Key type lgssc registered [ 8412.400536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8437.366755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8437.410690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8438.578992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8438.600292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8438.651148] Lustre: lustre-MDT0000: new disk, initializing [ 8438.703273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8438.718706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8442.161384] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8451.295028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8451.463843] Lustre: 415338:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8451.512308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8451.515678] Lustre: Skipped 1 previous similar message [ 8451.582827] Lustre: lustre-MDT0001: new disk, initializing [ 8451.707921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8451.753214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8451.762034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8455.354442] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8462.889338] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8467.596904] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8475.112260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8475.340626] Lustre: lustre-OST0000: new disk, initializing [ 8475.346692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8475.358225] Lustre: 417081:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8475.503162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8481.286244] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8484.442869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8484.464383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8484.673815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8490.020810] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8497.632451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8497.646519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8497.658716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.194805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.195460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.205866] Lustre: Skipped 1 previous similar message [ 8500.226249] Lustre: Skipped 1 previous similar message [ 8503.295626] Lustre: server umount lustre-MDT0000 complete [ 8507.112584] LustreError: 415373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332310 with bad export cookie 6508510350385501866 [ 8507.124832] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8507.137070] LustreError: 415373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8507.509180] Lustre: server umount lustre-MDT0001 complete [ 8521.780406] Lustre: server umount lustre-OST0000 complete [ 8531.990945] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:45:33 (1780332333) [ 8539.229631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8539.877952] LustreError: 418995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8539.908491] LustreError: 418995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8539.966162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.557422] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8545.270176] LustreError: 418996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8550.376502] LustreError: 418995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8553.084160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8553.510976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8557.325913] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8562.660956] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8566.486520] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8586.165483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8586.441476] Lustre: Found index 0 for lustre-OST0000, updating log [ 8586.622076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8592.021125] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8597.547662] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8611.200658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8619.488887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8619.501446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8619.520659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8619.524724] Lustre: Skipped 1 previous similar message [ 8624.718918] Lustre: server umount lustre-OST0000 complete [ 8642.025530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.028471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.050974] Lustre: Skipped 2 previous similar messages [ 8642.060016] Lustre: Skipped 3 previous similar messages [ 8646.312170] LNet: 1 local NIs in recovery (showing 1): 192.168.204.129@tcp [ 8647.141053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8647.150735] Lustre: Skipped 2 previous similar messages [ 8655.327894] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8655.617218] Lustre: server umount lustre-MDT0000 complete [ 8659.180763] LustreError: 418976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332462 with bad export cookie 6508510350385503259 [ 8659.184448] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8659.188674] LustreError: 418976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8659.510203] Lustre: server umount lustre-MDT0001 complete [ 8667.415148] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:47:48 (1780332468) [ 8710.879198] LNet: 1 local NIs in recovery (showing 1): 192.168.204.129@tcp [ 9285.240692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9285.802242] LustreError: 425089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9285.897090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9291.070873] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9291.257618] LustreError: 425090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9296.353825] LustreError: 425089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9300.974896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9301.456202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9305.932718] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9311.702587] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9316.074428] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9337.397704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9337.571541] LNet: 1 local NIs in recovery (showing 1): 192.168.204.129@tcp [ 9337.636128] Lustre: Found index 0 for lustre-OST0000, updating log [ 9337.927868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9343.785483] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9349.904949] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9352.304865] Lustre: server umount lustre-OST0000 complete [ 9356.897605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9356.910540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9356.913290] Lustre: Skipped 1 previous similar message [ 9356.926565] Lustre: Skipped 7 previous similar messages [ 9357.926224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9357.940511] Lustre: Skipped 1 previous similar message [ 9359.969192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9359.980172] Lustre: Skipped 3 previous similar messages [ 9362.287957] Lustre: server umount lustre-MDT0000 complete [ 9363.046030] LustreError: 425801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.067905] LustreError: 425801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9366.093692] LustreError: 425069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333169 with bad export cookie 6508510350385504344 [ 9366.094459] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9366.100253] LustreError: 425069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9366.396504] Lustre: server umount lustre-MDT0001 complete [ 9397.359891] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 9397.983175] LNet: 1 local NIs in recovery (showing 1): 192.168.204.129@tcp [ 9407.570065] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9447.051506] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:00:48 (1780333248) [ 9466.523366] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 9469.867252] Key type lgssc unregistered [ 9470.195843] LNet: 431318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9470.206721] LNetError: 431318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9598.254290] LNet: Removed LNI 192.168.204.129@tcp [ 9598.968667] Key type .llcrypt unregistered [ 9598.971751] Key type ._llcrypt unregistered [ 9612.893290] Key type ._llcrypt registered [ 9612.895189] Key type .llcrypt registered [ 9612.964764] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9613.679571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9613.751290] alg: No test for adler32 (adler32-zlib) [ 9614.813490] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 9615.071138] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9616.711454] Key type lgssc registered [ 9617.700708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9621.536367] LNet: Added LNI 192.168.204.129@tcp100 [8/256/0/180] [ 9622.402711] LNet: Added LNI 192.168.204.129@tcp101 [8/256/0/180] [ 9623.167960] LNet: Added LNI 192.168.204.129@tcp102 [8/256/0/180] [ 9623.969421] LNet: Added LNI 192.168.204.129@tcp103 [8/256/0/180] [ 9624.853902] LNet: Added LNI 192.168.204.129@tcp104 [8/256/0/180] [ 9625.527595] LNet: Added LNI 192.168.204.129@tcp105 [8/256/0/180] [ 9626.201272] LNet: Added LNI 192.168.204.129@tcp106 [8/256/0/180] [ 9627.006622] LNet: Added LNI 192.168.204.129@tcp107 [8/256/0/180] [ 9627.841195] LNet: Added LNI 192.168.204.129@tcp108 [8/256/0/180] [ 9628.590370] LNet: Added LNI 192.168.204.129@tcp109 [8/256/0/180] [ 9629.390253] LNet: Added LNI 192.168.204.129@tcp110 [8/256/0/180] [ 9630.193545] LNet: Added LNI 192.168.204.129@tcp111 [8/256/0/180] [ 9630.997146] LNet: Added LNI 192.168.204.129@tcp112 [8/256/0/180] [ 9631.902130] LNet: Added LNI 192.168.204.129@tcp113 [8/256/0/180] [ 9632.581436] LNet: Added LNI 192.168.204.129@tcp114 [8/256/0/180] [ 9633.505823] LNet: Added LNI 192.168.204.129@tcp115 [8/256/0/180] [ 9634.358410] LNet: Added LNI 192.168.204.129@tcp116 [8/256/0/180] [ 9635.163137] LNet: Added LNI 192.168.204.129@tcp117 [8/256/0/180] [ 9636.131974] LNet: Added LNI 192.168.204.129@tcp118 [8/256/0/180] [ 9637.268414] LNet: Added LNI 192.168.204.129@tcp119 [8/256/0/180] [ 9638.066099] LNet: Added LNI 192.168.204.129@tcp120 [8/256/0/180] [ 9639.312555] LNet: Added LNI 192.168.204.129@tcp121 [8/256/0/180] [ 9640.387844] LNet: Added LNI 192.168.204.129@tcp122 [8/256/0/180] [ 9641.663393] LNet: Added LNI 192.168.204.129@tcp123 [8/256/0/180] [ 9643.214959] LNet: Added LNI 192.168.204.129@tcp124 [8/256/0/180] [ 9644.784409] LNet: Added LNI 192.168.204.129@tcp125 [8/256/0/180] [ 9645.716143] LNet: Added LNI 192.168.204.129@tcp126 [8/256/0/180] [ 9647.016703] LNet: Added LNI 192.168.204.129@tcp127 [8/256/0/180] [ 9648.239932] LNet: Added LNI 192.168.204.129@tcp128 [8/256/0/180] [ 9649.076375] LNet: Added LNI 192.168.204.129@tcp129 [8/256/0/180] [ 9650.067595] LNet: Added LNI 192.168.204.129@tcp130 [8/256/0/180] [ 9650.965737] LNet: Added LNI 192.168.204.129@tcp131 [8/256/0/180] [ 9651.761973] LNet: Added LNI 192.168.204.129@tcp132 [8/256/0/180] [ 9652.496042] LNet: Added LNI 192.168.204.129@tcp133 [8/256/0/180] [ 9653.246148] LNet: Added LNI 192.168.204.129@tcp134 [8/256/0/180] [ 9654.156365] LNet: Added LNI 192.168.204.129@tcp135 [8/256/0/180] [ 9660.665660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9660.700488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9661.840106] Lustre: 434328:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [ 9662.043878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9662.089160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9662.194975] Lustre: lustre-MDT0000: new disk, initializing [ 9662.323600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9662.348511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9666.917842] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9678.160768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.291373] Lustre: 434345:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9678.340393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9678.345655] Lustre: Skipped 1 previous similar message [ 9678.419962] Lustre: lustre-MDT0001: new disk, initializing [ 9678.465478] Lustre: 435127:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [ 9678.473149] Lustre: 435127:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9678.585441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9678.637330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9678.651197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9680.232211] Lustre: 435214:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [ 9680.249303] Lustre: 435214:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9682.546607] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9688.947948] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9692.251978] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9697.960227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.192389] Lustre: lustre-OST0000: new disk, initializing [ 9698.195668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9698.200892] Lustre: 436091:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9698.303495] Lustre: 436096:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [ 9698.313714] Lustre: 436096:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 9698.395138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9704.215339] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9704.520100] Lustre: 436442:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [ 9704.534294] Lustre: 436442:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9704.612132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9704.616965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9704.726042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9711.662047] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9720.289027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9720.292466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9720.298265] LustreError: Skipped 1 previous similar message [ 9720.306966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9720.320882] Lustre: Skipped 1 previous similar message [ 9720.340522] Lustre: Skipped 1 previous similar message [ 9724.459980] Lustre: server umount lustre-OST0000 complete [ 9730.016774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9730.026070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9730.043627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9734.040591] Lustre: server umount lustre-MDT0000 complete [ 9737.441221] LustreError: 434337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333540 with bad export cookie 12289175956827776150 [ 9737.443566] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9737.453155] LustreError: 434337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9737.755570] Lustre: server umount lustre-MDT0001 complete [ 9746.288240] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 9748.831170] Key type lgssc unregistered [ 9749.131304] LNet: 437872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9749.137187] LNetError: 437872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9750.175229] LNet: Removed LNI 192.168.204.129@tcp [ 9750.177225] LNetError: 437872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9750.181587] LNet: Removed LNI 192.168.204.129@tcp100 [ 9751.199166] LNet: Removed LNI 192.168.204.129@tcp101 [ 9751.215791] LNetError: 437872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9751.228874] LNetError: 437872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9751.234469] LNet: Removed LNI 192.168.204.129@tcp102 [ 9752.287265] LNet: Removed LNI 192.168.204.129@tcp103 [ 9753.311401] LNet: Removed LNI 192.168.204.129@tcp104 [ 9753.314464] LNetError: 437872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9753.329561] LNetError: 437872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9753.334276] LNet: Removed LNI 192.168.204.129@tcp105 [ 9754.402518] LNet: Removed LNI 192.168.204.129@tcp106 [ 9755.425078] LNet: Removed LNI 192.168.204.129@tcp107 [ 9756.448064] LNet: Removed LNI 192.168.204.129@tcp108 [ 9757.471695] LNet: Removed LNI 192.168.204.129@tcp109 [ 9757.475369] LNetError: 437872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9757.493206] LNetError: 437872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9757.509305] LNet: Removed LNI 192.168.204.129@tcp110 [ 9758.559156] LNet: Removed LNI 192.168.204.129@tcp111 [ 9759.583147] LNet: Removed LNI 192.168.204.129@tcp112 [ 9760.607187] LNet: Removed LNI 192.168.204.129@tcp113 [ 9760.615328] LNet: Removed LNI 192.168.204.129@tcp114 [ 9760.620381] LNet: Removed LNI 192.168.204.129@tcp115 [ 9760.623165] LNet: Removed LNI 192.168.204.129@tcp116 [ 9760.626760] LNet: Removed LNI 192.168.204.129@tcp117 [ 9760.633820] LNet: Removed LNI 192.168.204.129@tcp118 [ 9760.636814] LNet: Removed LNI 192.168.204.129@tcp119 [ 9760.639463] LNet: Removed LNI 192.168.204.129@tcp120 [ 9760.643776] LNet: Removed LNI 192.168.204.129@tcp121 [ 9760.647137] LNet: Removed LNI 192.168.204.129@tcp122 [ 9760.649804] LNet: Removed LNI 192.168.204.129@tcp123 [ 9760.652447] LNet: Removed LNI 192.168.204.129@tcp124 [ 9760.654761] LNet: Removed LNI 192.168.204.129@tcp125 [ 9760.657255] LNet: Removed LNI 192.168.204.129@tcp126 [ 9760.658767] LNet: Removed LNI 192.168.204.129@tcp127 [ 9760.661642] LNet: Removed LNI 192.168.204.129@tcp128 [ 9760.664632] LNet: Removed LNI 192.168.204.129@tcp129 [ 9760.667815] LNet: Removed LNI 192.168.204.129@tcp130 [ 9760.670786] LNet: Removed LNI 192.168.204.129@tcp131 [ 9760.673941] LNet: Removed LNI 192.168.204.129@tcp132 [ 9760.676601] LNet: Removed LNI 192.168.204.129@tcp133 [ 9760.678583] LNet: Removed LNI 192.168.204.129@tcp134 [ 9760.694454] LNet: Removed LNI 192.168.204.129@tcp135 [ 9761.511561] Key type .llcrypt unregistered [ 9761.514208] Key type ._llcrypt unregistered [ 9784.545180] Key type ._llcrypt registered [ 9784.547625] Key type .llcrypt registered [ 9784.642971] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 9797.878813] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9799.177255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9799.272769] alg: No test for adler32 (adler32-zlib) [ 9800.356859] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 9800.611964] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9802.273053] Key type lgssc registered [ 9803.344117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9841.111710] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:07:22 (1780333642) [ 9857.219204] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 9859.649917] Key type lgssc unregistered [ 9859.905154] LNet: 441900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9859.911819] LNetError: 441900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9859.934556] LNet: Removed LNI 192.168.204.129@tcp [ 9860.638684] Key type .llcrypt unregistered [ 9860.642151] Key type ._llcrypt unregistered [ 9878.157077] Key type ._llcrypt registered [ 9878.163389] Key type .llcrypt registered [ 9878.305054] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9879.297198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9879.313775] alg: No test for adler32 (adler32-zlib) [ 9880.456982] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 9880.779689] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9882.431466] Key type lgssc registered [ 9883.963172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9894.324136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9894.351320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9895.731546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9895.771952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9895.858722] Lustre: lustre-MDT0000: new disk, initializing [ 9896.004699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9896.048595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9900.723812] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9911.661948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.792829] Lustre: 443094:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9911.849122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9911.853547] Lustre: Skipped 1 previous similar message [ 9911.949993] Lustre: lustre-MDT0001: new disk, initializing [ 9912.096583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9912.158555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9912.178939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9916.911913] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9925.560978] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9929.852080] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9937.013839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9937.471954] Lustre: lustre-OST0000: new disk, initializing [ 9937.481684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9937.491809] Lustre: 444838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9937.643876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9944.549948] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9945.753744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9945.764560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9945.915029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9953.347443] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9956.899609] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing lnet_if_list [ 9963.152955] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9963.190685] LNet: Added LNI 192.168.204.129@tcp10 [8/256/0/180] [ 9970.403863] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9970.458679] LNet: Added LNI 192.168.204.129@tcp11 [8/256/0/180] [ 9974.782293] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9974.809686] LNet: Added LNI 192.168.204.129@tcp12 [8/256/0/180] [ 9980.283384] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9980.325376] LNet: Added LNI 192.168.204.129@tcp13 [8/256/0/180] [ 9985.043695] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9985.077363] LNet: Added LNI 192.168.204.129@tcp14 [8/256/0/180] [ 9990.019389] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9990.053876] LNet: Added LNI 192.168.204.129@tcp15 [8/256/0/180] [ 9995.609479] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9995.654031] LNet: Added LNI 192.168.204.129@tcp16 [8/256/0/180] [10000.988303] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10001.011747] LNet: Added LNI 192.168.204.129@tcp17 [8/256/0/180] [10006.324802] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10006.355489] LNet: Added LNI 192.168.204.129@tcp18 [8/256/0/180] [10011.919222] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10011.983530] LNet: Added LNI 192.168.204.129@tcp19 [8/256/0/180] [10018.243839] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10018.285245] LNet: Added LNI 192.168.204.129@tcp20 [8/256/0/180] [10025.046865] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10025.077846] LNet: Added LNI 192.168.204.129@tcp21 [8/256/0/180] [10032.027473] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10032.083603] LNet: Added LNI 192.168.204.129@tcp22 [8/256/0/180] [10039.451483] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10039.475043] LNet: Added LNI 192.168.204.129@tcp23 [8/256/0/180] [10046.499274] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10046.528430] LNet: Added LNI 192.168.204.129@tcp24 [8/256/0/180] [10053.476694] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10053.523467] LNet: Added LNI 192.168.204.129@tcp25 [8/256/0/180] [10060.947800] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10060.995867] LNet: Added LNI 192.168.204.129@tcp26 [8/256/0/180] [10067.394432] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10067.473845] LNet: Added LNI 192.168.204.129@tcp27 [8/256/0/180] [10074.474513] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10074.510111] LNet: Added LNI 192.168.204.129@tcp28 [8/256/0/180] [10080.941024] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10080.996257] LNet: Added LNI 192.168.204.129@tcp29 [8/256/0/180] [10088.358139] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10088.419660] LNet: Added LNI 192.168.204.129@tcp30 [8/256/0/180] [10095.184581] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10095.232470] LNet: Added LNI 192.168.204.129@tcp31 [8/256/0/180] [10101.289701] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10101.326168] LNet: Added LNI 192.168.204.129@tcp32 [8/256/0/180] [10107.317382] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10107.365035] LNet: Added LNI 192.168.204.129@tcp33 [8/256/0/180] [10114.722195] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10114.760342] LNet: Added LNI 192.168.204.129@tcp34 [8/256/0/180] [10122.740586] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10122.770025] LNet: Added LNI 192.168.204.129@tcp35 [8/256/0/180] [10132.459005] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10132.511502] LNet: Added LNI 192.168.204.129@tcp36 [8/256/0/180] [10140.762227] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10140.794960] LNet: Added LNI 192.168.204.129@tcp37 [8/256/0/180] [10149.619557] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10149.646493] LNet: Added LNI 192.168.204.129@tcp38 [8/256/0/180] [10158.220511] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10158.293937] LNet: Added LNI 192.168.204.129@tcp39 [8/256/0/180] [10167.496950] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10167.564456] LNet: Added LNI 192.168.204.129@tcp40 [8/256/0/180] [10178.093902] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10178.140645] LNet: Added LNI 192.168.204.129@tcp41 [8/256/0/180] [10189.006886] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10189.039360] LNet: Added LNI 192.168.204.129@tcp42 [8/256/0/180] [10198.740855] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10198.772667] LNet: Added LNI 192.168.204.129@tcp43 [8/256/0/180] [10208.013921] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10208.076788] LNet: Added LNI 192.168.204.129@tcp44 [8/256/0/180] [10216.343211] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10216.372901] LNet: Added LNI 192.168.204.129@tcp45 [8/256/0/180] [10225.912905] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10225.964664] LNet: Added LNI 192.168.204.129@tcp46 [8/256/0/180] [10235.292597] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10235.319290] LNet: Added LNI 192.168.204.129@tcp47 [8/256/0/180] [10243.548822] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10243.578422] LNet: Added LNI 192.168.204.129@tcp48 [8/256/0/180] [10252.586677] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10252.632879] LNet: Added LNI 192.168.204.129@tcp49 [8/256/0/180] [10261.967515] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10262.009227] LNet: Added LNI 192.168.204.129@tcp50 [8/256/0/180] [10284.011614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10284.015940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10284.030671] Lustre: Skipped 2 previous similar messages [10284.040296] Lustre: Skipped 1 previous similar message [10287.012201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10287.033892] Lustre: Skipped 3 previous similar messages [10288.572923] Lustre: server umount lustre-MDT0000 complete [10293.173665] LustreError: 443089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334096 with bad export cookie 12867108490217735209 [10293.184229] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10293.199344] LustreError: 443089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10293.552256] Lustre: server umount lustre-MDT0001 complete [10312.674993] Lustre: 442371:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780334100/real 1780334100] req@000000009fff4bf3 x1866815173214208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780334116 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10312.709101] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10313.762378] Lustre: 442371:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780334101/real 1780334101] req@0000000017b555e2 x1866815173214336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780334117 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10315.245574] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10317.791118] Lustre: 442368:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780334105/real 1780334105] req@000000002765dc7b x1866815173214464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780334121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10324.453486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.959580] Lustre: 442371:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780334112/real 1780334112] req@00000000fd981ec5 x1866815173214848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780334128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10324.983330] Lustre: 442371:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10338.817901] LustreError: 444832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10338.848909] LustreError: 444832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10338.940593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10344.119536] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10344.420513] LustreError: 444833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10347.492656] LustreError: 444832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10347.504363] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10352.619308] LustreError: 444832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10353.291626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.748448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10357.867330] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10358.682757] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10364.555223] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10367.928462] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10388.961238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10388.969061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10388.978452] Lustre: Skipped 1 previous similar message [10388.987164] Lustre: Skipped 1 previous similar message [10393.949243] Lustre: server umount lustre-OST0000 complete [10399.209798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10399.218583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10399.222568] Lustre: Skipped 1 previous similar message [10399.241977] Lustre: Skipped 1 previous similar message [10403.940764] Lustre: server umount lustre-MDT0000 complete [10404.324165] LustreError: 468242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10404.349353] LustreError: 468242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10407.545446] LustreError: 443088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334211 with bad export cookie 12867108490217737750 [10407.553694] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10407.555672] LustreError: 443088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10407.938439] Lustre: server umount lustre-MDT0001 complete [10418.679615] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [10421.387535] Key type lgssc unregistered [10421.796431] LNet: 470799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10421.817043] LNetError: 470799:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10421.823078] LNet: Removed LNI 192.168.204.129@tcp [10421.830767] LNet: Removed LNI 192.168.204.129@tcp10 [10421.834621] LNet: Removed LNI 192.168.204.129@tcp11 [10421.841075] LNet: Removed LNI 192.168.204.129@tcp12 [10421.846443] LNet: Removed LNI 192.168.204.129@tcp13 [10421.848622] LNet: Removed LNI 192.168.204.129@tcp14 [10421.850266] LNet: Removed LNI 192.168.204.129@tcp15 [10421.862497] LNet: Removed LNI 192.168.204.129@tcp16 [10421.870777] LNet: Removed LNI 192.168.204.129@tcp17 [10421.874885] LNet: Removed LNI 192.168.204.129@tcp18 [10421.881357] LNet: Removed LNI 192.168.204.129@tcp19 [10421.888814] LNet: Removed LNI 192.168.204.129@tcp20 [10421.891359] LNet: Removed LNI 192.168.204.129@tcp21 [10421.893958] LNet: Removed LNI 192.168.204.129@tcp22 [10421.899494] LNet: Removed LNI 192.168.204.129@tcp23 [10421.902254] LNet: Removed LNI 192.168.204.129@tcp24 [10421.908381] LNet: Removed LNI 192.168.204.129@tcp25 [10421.911652] LNet: Removed LNI 192.168.204.129@tcp26 [10421.916497] LNet: Removed LNI 192.168.204.129@tcp27 [10421.920255] LNet: Removed LNI 192.168.204.129@tcp28 [10421.923826] LNet: Removed LNI 192.168.204.129@tcp29 [10421.927401] LNet: Removed LNI 192.168.204.129@tcp30 [10421.931201] LNet: Removed LNI 192.168.204.129@tcp31 [10421.934868] LNet: Removed LNI 192.168.204.129@tcp32 [10421.937992] LNet: Removed LNI 192.168.204.129@tcp33 [10421.941600] LNet: Removed LNI 192.168.204.129@tcp34 [10421.945885] LNet: Removed LNI 192.168.204.129@tcp35 [10421.950960] LNet: Removed LNI 192.168.204.129@tcp36 [10421.954599] LNet: Removed LNI 192.168.204.129@tcp37 [10421.958854] LNet: Removed LNI 192.168.204.129@tcp38 [10421.963526] LNet: Removed LNI 192.168.204.129@tcp39 [10421.966856] LNet: Removed LNI 192.168.204.129@tcp40 [10421.972596] LNet: Removed LNI 192.168.204.129@tcp41 [10421.976628] LNet: Removed LNI 192.168.204.129@tcp42 [10421.979560] LNet: Removed LNI 192.168.204.129@tcp43 [10421.982848] LNet: Removed LNI 192.168.204.129@tcp44 [10421.985621] LNet: Removed LNI 192.168.204.129@tcp45 [10421.988795] LNet: Removed LNI 192.168.204.129@tcp46 [10421.991519] LNet: Removed LNI 192.168.204.129@tcp47 [10421.995248] LNet: Removed LNI 192.168.204.129@tcp48 [10421.997863] LNet: Removed LNI 192.168.204.129@tcp49 [10422.009652] LNet: Removed LNI 192.168.204.129@tcp50 [10422.821633] Key type .llcrypt unregistered [10422.826045] Key type ._llcrypt unregistered [10441.461564] Key type ._llcrypt registered [10441.464464] Key type .llcrypt registered [10441.580308] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [10454.978246] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10455.839112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10455.903121] alg: No test for adler32 (adler32-zlib) [10456.991316] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [10457.231533] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [10458.959216] Key type lgssc registered [10460.173600] Lustre: Echo OBD driver; http://www.lustre.org/ [10494.037694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10494.077836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.513265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10495.583821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10495.730756] Lustre: lustre-MDT0000: new disk, initializing [10495.832361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10495.866289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10500.667059] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10511.715032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10511.887730] Lustre: 473961:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10511.950405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10511.954390] Lustre: Skipped 1 previous similar message [10512.049530] Lustre: lustre-MDT0001: new disk, initializing [10512.179202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10512.225132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10512.241584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10516.503754] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10524.248097] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10527.552754] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10534.489465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10534.661854] Lustre: *** cfs_fail_loc=90f, val=0*** [10534.730221] Lustre: lustre-OST0000: new disk, initializing [10534.734980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10534.744747] Lustre: 475753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10534.843640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10540.962071] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10543.199479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10543.215108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10543.322362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10548.673582] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10558.950784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10558.953360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10558.955736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10558.964490] Lustre: Skipped 1 previous similar message [10558.986584] LustreError: Skipped 1 previous similar message [10562.065861] Lustre: server umount lustre-OST0000 complete [10568.673037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10568.690839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10568.704552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10568.710454] Lustre: Skipped 1 previous similar message [10572.247188] Lustre: server umount lustre-MDT0000 complete [10575.695082] LustreError: 473955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334379 with bad export cookie 5276978816474320148 [10575.698762] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10575.714915] LustreError: 473955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10576.186617] Lustre: server umount lustre-MDT0001 complete [10587.144874] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [10590.179601] Key type lgssc unregistered [10590.534980] LNet: 477526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10590.555464] LNetError: 477526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10590.585361] LNet: Removed LNI 192.168.204.129@tcp [10591.483259] Key type .llcrypt unregistered [10591.485761] Key type ._llcrypt unregistered [10609.621458] Key type ._llcrypt registered [10609.625772] Key type .llcrypt registered [10609.719085] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10610.931358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10610.982070] alg: No test for adler32 (adler32-zlib) [10612.336974] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [10612.691310] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [10614.383127] Key type lgssc registered [10615.431390] Lustre: Echo OBD driver; http://www.lustre.org/ [10622.869047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10622.897988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10624.588956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10629.339313] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10633.699859] LustreError: 478782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.726731] LustreError: 478782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10638.823596] LustreError: 478781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10639.219669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10639.768873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10644.804413] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10650.564886] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10654.971939] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10664.036324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10664.646059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10671.991710] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10679.371073] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10690.056467] loop0: detected capacity change from 0 to 204800000 [10690.230650] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10690.244890] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10690.262985] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10695.996592] loop0: detected capacity change from 0 to 204800000 [10696.180645] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10696.194155] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10696.207763] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10701.724845] loop0: detected capacity change from 0 to 204800000 [10707.541787] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10707.711789] Lustre: fs15246-OST0000: new disk, initializing [10707.716944] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10707.724601] Lustre: 482038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10707.775880] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10713.661537] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10722.390390] loop1: detected capacity change from 0 to 204800000 [10727.457606] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10727.568971] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10727.642535] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10727.732501] Lustre: fs15246-MDT0000: new disk, initializing [10727.911711] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10727.938560] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10728.019414] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10729.060565] Lustre: 483144:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000000abaf63 x1866815940638976/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780334548 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10732.094570] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10738.146063] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10738.167042] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10738.202035] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10756.580825] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10756.596460] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10759.857499] Lustre: server umount fs15246-MDT0000 complete [10761.703614] LustreError: 478776:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.819713] LustreError: 479741:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.377501] Lustre: server umount fs15246-OST0000 complete [10781.666039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10781.675753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10781.701794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10786.404327] Lustre: server umount lustre-OST0000 complete [10792.416443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10792.430257] Lustre: Skipped 1 previous similar message [10792.439421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10792.447289] Lustre: Skipped 3 previous similar messages [10797.539871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10797.551755] Lustre: Skipped 1 previous similar message [10801.642301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10801.661525] Lustre: Skipped 1 previous similar message [10804.194427] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10804.589570] Lustre: server umount lustre-MDT0000 complete [10806.753843] LustreError: 478778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10806.771317] LustreError: 478778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10808.956304] LustreError: 478762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334612 with bad export cookie 4281438665301218574 [10808.975494] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10808.980160] LustreError: 478762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10809.428812] Lustre: server umount lustre-MDT0001 complete [10820.503227] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [10823.766542] Key type lgssc unregistered [10824.156083] LNet: 486207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10824.172949] LNetError: 486207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10825.255518] LNet: Removed LNI 192.168.204.129@tcp [10826.230932] Key type .llcrypt unregistered [10826.236476] Key type ._llcrypt unregistered [10846.316722] Key type ._llcrypt registered [10846.322563] Key type .llcrypt registered [10846.501778] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10847.582041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10847.614703] alg: No test for adler32 (adler32-zlib) [10848.634387] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [10848.902879] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [10850.618939] Key type lgssc registered [10851.908259] Lustre: Echo OBD driver; http://www.lustre.org/ [10888.666702] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [10899.324509] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10934.791779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10934.857210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10936.163470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10936.208529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10936.329310] Lustre: lustre-MDT0000: new disk, initializing [10936.452737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10936.476338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10941.071308] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10952.982147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10953.119647] Lustre: 490623:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10953.189825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10953.199785] Lustre: Skipped 1 previous similar message [10953.263795] Lustre: lustre-MDT0001: new disk, initializing [10953.382846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10953.441376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10953.455937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10958.443931] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10966.133679] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10970.391654] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10977.802718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10978.165984] Lustre: lustre-OST0000: new disk, initializing [10978.170225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10978.179242] Lustre: 492365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10978.288717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10985.214097] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10986.080279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10986.094954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10986.223530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10993.267599] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10998.388797] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10998.622710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11003.616519] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11003.854303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11006.949930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11006.951121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11006.962045] LustreError: Skipped 1 previous similar message [11006.963414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11007.012943] Lustre: Skipped 1 previous similar message [11012.074331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11012.082597] Lustre: Skipped 2 previous similar messages [11012.428894] Lustre: server umount lustre-OST0000 complete [11017.184608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11017.186615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11017.210418] Lustre: Skipped 1 previous similar message [11017.229906] Lustre: Skipped 1 previous similar message [11022.308420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11022.320741] Lustre: Skipped 1 previous similar message [11022.872877] Lustre: server umount lustre-MDT0000 complete [11026.913789] LustreError: 490613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334830 with bad export cookie 7502632497444677428 [11026.920592] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11026.927342] LustreError: 490613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11027.453274] Lustre: server umount lustre-MDT0001 complete [11057.485133] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [11069.330594] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [11105.525721] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10889 sec ========= 13:28:27 (1780334907) [11107.299813] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:28:28 (1780334908) === [11110.737841] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:28:32 (1780334912) === [11152.830270] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [11156.697729] Key type lgssc unregistered [11157.061052] LNet: 498816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11157.070170] LNetError: 498816:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11157.086929] LNet: Removed LNI 192.168.204.129@tcp [11158.180529] Key type .llcrypt unregistered [11158.187933] Key type ._llcrypt unregistered